1 Big Data and Advanced Analytics Technologies for the Smart Grid Arnie de Castro, PhD SAS Institute IEEE PES 2014 General Meeting July 27-31, 2014 Panel Session: Using Smart Grid Data to Improve Planning, Analytics, and Operation of the US Capital region T&D Systems
BIG DATA Big Data is Relative, not Absolute When volume, velocity and variety of data exceeds an organization s storage or compute capacity for accurate and timely decision-making Meter Traditional AMI Meter PMU Reads/month 1 2,880 77,760,000
Analytics Across the Energy Value Chain
5 Technologies for the Smart Grid Enterprise Analytics Situational Awareness, Descriptive to Predictive, Visualization Grid Operations Analytics Predictive Asset Maintenance, Outage Management, PMU Monitoring and Analytics, Smart Meter Analytics, Distribution Optimization Consumer Analytics Energy Forecasting, Consumption Analysis, Revenue Protection
ENTERPRISE ANALYTICS
Innovative Strategies for Big Data Analytics A flexible enterprise architecture that supports many data types and usage patterns Upstream use of analytics to optimize data relevance Real-time visualization and advanced analytics to accelerate understanding and action Common analytical framework across the enterprise
GRID OPTIMIZATION ANALYTICS
Predictive Asset Maintenance Identify equipment that is likely to fail and/or determine its remaining lifetime Prioritize problems based on business impact Determine root cause more quickly Provide automated reporting and alerting Provide a collaborative environment
The Problem: humidity sensor failure Pi graph: Indicates key parameters during the humidity sensor failure. NOTE: Humidity Sensor failed High NOx limit is 20ppm (we are seeing ~30 ppm below). Magenta line is instantaneous NOx ppm. Copyr i g ht 2012, SAS Ins titut e Inc. All rights res er ve d.
Root Cause Analysis Copyr i g ht 2012, SAS Ins titut e Inc. All rights res er ve d.
Diagnostic Sequence Diagram:
OUTAGE OPTIMIZATION
Outage Optimization- Balancing Customer Satisfaction and Reliability: SAIDI, SAIFI 1500 1000 500 0 Hospital High Business High Residential High Residential Low Residential Medium Police High Emergency Services High School Medium Outages 30-60 Outage 60-90 Outage > 90
16 Traditional and Advanced Analytical Impact Methods
Outages by City during Hurricane Irene 2011 17
18 Predicting Outages Narrowing Down the Variables Affecting Asset Failure Survival Analyses Modeling Asset Potential Storm Failure Outage Prediction Model Scoring
Outage Prediction Model 19
20 After the Storm Analytics Travel time calculation Modify MILP Framework for Customer Restoration Constraints MILP Solver to Create Optimal Solutions versus Standard Utility Routing
Constrained Customer Restoration Problem 21
PMU MONITORING AND ANALYTICS
Phasor Measurement Units (PMUs) Issue: A REAL WORLD EXAMPLE FROM THE POWER GRID Latency; a delay of 3 seconds or more may be too late to take action to control system stability, leading to a blackout. Background: With Phasor Measurement Units (PMUs), measurements taken are precisely time-synchronized and taken many times a second (i.e. 30 to 60 samples/second) offering dynamic visibility into the power system. Approach: Develop analytics to: Understand Steady State operation Detect events on the network Categorize the event on the network Direct appropriate action based on the event Capture data for post event analysis
WHAT ARE PHASOR MEASUREMENT UNITS? Phasor Measurement Units Next-Gen measurement devices for power grid Collect measurements (frequency, voltage, current, phase angle) at 30 meas/second Synchronized across locations by GPS clock Courtesy: US Dept of Energy
PMU Analytics Process Pi Server Data Quality/ Transform Event Detection Event Identification Event Quantification Notifications
DETAIL CHARTS FOR EVENT PMU Event Analysis Current oscillates after event, but then dampens down to normal
SIMILARITY ANALYSIS Event Identification Reference time series for various events Incoming data stream is compared to reference time series
SIMILARITY ANALYSIS Event Identification Similarity between incoming stream and reference time series is measured and quantified
SMART METER ANALYTICS
Smart Meter Analytics 30
Customer Analysis 31
Load Analysis 32
DISTRIBUTION OPTIMIZATION
Distribution Optimization GIS, OMS SCADA/DMS, Meter Data, Sensor Data Distribution Network Model Tap Changing Transformers Capacitors Regulators Distributed Generation Energy Storage Network Operations Model Load Forecasts Load Models Load Analytics Measurement and Verification Distribution Optimization Conservation Voltage Reduction Loss Minimization Direct Load Control Cost Optimization Distributed Intelligence Connectivity (Static) Data Operational (Dynamic) Data Optimization Software
ENERGY FORECASTING
Energy Forecasting Spatial load forecasting Outlier detection Demand response forecasting Weather forecasting Hydro/wind/solar generation forecasting Price forecasting
CONSUMPTION ANALYSIS
Load Profile Comparisons via Segmentation
ENABLING TECHNOLOGIES
HIGH PERFORMANCE ANALYTICS
HIGH- PERFORMANCE ANALYTICS
Analytics Server Architecture Massively Parallel Processing ( MPP ) in the context of SAS Visual Analytics SAS VA Server LASR Cluster LASR Cluster LASR Cluster Workspace Server SAS LASR Analytic Server SAS LASR Analytic Server SAS LASR Analytic Server MEMORY Mid-Tier Co-Located Data Storage Co-Located Data Storage Co-Located Data Storage STORAGE Metadata PROCESSING RDBMS Nonrelational ERP Hadoop unstructured PC Files DATA SOURCES
HIGH PERFORMANCE ANALYTICS TECHNIQUES
ANALYTICS FORECASTING Leveraging historical data to drive better insight into decision-making for the future TEXT ANALYTICS Finding treasures in unstructured data like social media or survey tools that could uncover insights about consumer sentiment DATA MINING Mine transaction databases for usage patterns that indicate abnormalities INFORMATION MANAGEMENT STATISTICS OPTIMIZATION Analyze massive amounts of data in order to accurately identify areas likely to produce the most profitable results Copyright 2012, SAS Institute Inc. All rights reserved.
EVENT STREAM PROCESSING
Event Stream Processing (ESP) ESP is a subcategory of Complex Event Processing (CEP) focused on analyzing/processing events in motion called Event Streams.* The SAS ESP is an embeddable engine that can be integrated into or front-end SAS solutions. * This is the definition provided by the Event Processing Technical Society Copyr i g ht 2012, SAS Ins titut e Inc. All rights res er ve d.
TYPICAL CHARACTERISTICS OF EVENT STREAM PROCESSING APPLICATIONS: Continuous queries on data in motion (with incremental results) Moves analytics from centralized data warehouse to edge analytics (closer to the occurrence of the events) Very low (max) event processing latencies (i.e., µsecs-msecs) High volumes (>100k events/sec) Derived event windows with retention policies Memory constrained for performance (i.e., Bounded state) Predetermined data mining, decision making, alerting, position management, scoring, profiling, Copyr i g ht 2012, SAS Ins titut e Inc. All rights res er ve d.
Hybrid (Multi-staged) Analytics: Streaming Analytics front-ending historical/predictive analytics ESP RDBMS ESPs store the queries and continuously stream data through the queries Databases store the data and periodically run queries against the stored data EVENTS INCREMENTAL RESULTS QUERIES RESULTS Copyr i g ht 2012, SAS Ins titut e Inc. All rights res er ve d.
ESP Utilities Power Grid Management Monitor the Power Grid for Concerning Meter Reading Patterns Suggestive of Less Than Optimal Health Event Stream Processing Server Data Flow Model: Substations Meter Readings Slot1: vmin>vmax Readings (source) vmingtvmax (copy) volchk1 (filter) volchk2 (filter) sigma1 (filter & pattern) sigma2 (filter & pattern) Meter readings are continuously published into Readings source window Readings window uses output slot feature with exvolion vmin>vmax to send bad readings to vmingtvmax window & good readings to cleanse readings window DataFlux data quality functions are used to cleanse the readings. Null fields are fixed via procedural window using prior state. Aggregate window adds meter stats to readings: count, ave ave vmax, ave vmin, stdev, stdev vmax, stdev vmin volchk1: vmin<minthresh1 or vmax>maxthresh1 Cleanse readings (compute& procedural) Slot 0: vmin<=vmax readingswstats (aggregate) volchk3 (filter) integral (pattern) volchk2: vmin<miinthresh2or vmax>maxthresh2 volchk3: vmin<=0 or vmax<=0 volchk4: vmin>avevmin+2*stdvmin or vmin<avevmin-2*stdvmin or : vmax>avevmax+2*stdvmax or vmax<avevmax- 2*stdVMAX sigma1: 4 out of 5 consecutive points fall beyond 1σ, on the same side of the centerline (mean) Grid Management Console volchk4 (filter) downtrend (pattern) Sigma2: 2 out of 3 consecutive points beyond 2σ, on the same side of the centerline (mean) Integral: 9 consecutive points either above or below the centerline (mean) Downtrend: trend of 6 points in a row either increasing or decreasing
Connected Device PROCESS REFERENCE ARCHITECTURE SOURCE DATA EVENT STREAM PROCESSING Batch Processing Data MODEL DEVELOPMENT / BATCH ANALYSIS / ALERT / REPORT / ROOT CAUSE / ADJUDICATE Trade/ Financial Feeds Sensor Data/ Smart Device Threshold Models Patterns Queries Model Deployment ACCESS SERVER Maintenance/ Quality Dashboard/Alerts Telemetry 10011 01 100111 Network Traffic Databases Low Latency ACCESS ENGINES Streaming Data Access/Cleanse Data In-Memory Extreme Parallelism Distribution of Analytics Processes Customer Seg/ Next Best Offer Network Security/ Management Fraud & Compliance Mobile Dashboard/ Alerts Data Visualization Routers, switches Data Management Batch Processing Workflows/ Case Management
Analytics Solutions Across the Energy Value Chain