REAL-TIME PRICE FORECAST WITH BIG DATA A STATE SPACE APPROACH Lang Tong (PI), Robert J. Thomas, Yuting Ji, and Jinsub Kim School of Electrical and Computer Engineering, Cornell University Jie Mei, Georgia Institute of Technology August 7, 2013 CERTS Review
DATA QUALITY AND ITS EFFECTS ON MARKET OPERATIONS DENY OF SERVICE ATTACK ON REAL-TIME ELECTRICITY MARKET COVER UP PROTECTION AGAINST TOPOLOGY ATTACK Lang Tong (PI), Robert J. Thomas, and Jinsub Kim Cornell University August 8, 2013 CERTS Review
Project overview Objectives Accurate short-term probabilistic forecasting of real-time LMP. Incorporate real-time measurements (e.g. SCADA/PMU). Scalable computation techniques. Summary of results A real-time LMP model with forecasting and measurement uncertainties. A Markov chain abstraction of real-time LMP computation. Monte Carlo sampling techniques. Preliminary simulations.
Outline Motivation Load and real-time LMP as random processes Benchmark techniques Comments on the state of the art. Probabilistic forecasting of real-time LMP Simulation studies Summary and future work
Sample paths of load and LMP
Day ahead vs. real-time
Benchmark techniques Time series ARMA, ARIMA, ARMAX, GARCH Machine learning Neural networks, support vector machines (SVM) Hybrid techniques Jump (switching) models
Load forecasting Forecasting Method ANN [1] ANN [2] MAPE 1.46% 1.24% SVM [3] 2.09% [1] P. Mandal, T. Senjyu, N. Urasaki, and T. Funabashi, A neural network based several-hour-ahead electric load forecasting using similar days approach, Int. Journal. of Electric Power and Energy System, vol. 28, no. 6, pp. 367 373, July 2006. [2] P. Mandal, T. Senjyu, N. Urasaki, and T. Funabashi, Short-Term Price Forecasting for Competitive Electricity Market, Power Symposium, 2006. NAPS 2006. 38 th North. [3] S. Fan, C. Mao and L. Chen, Next-day electricity-price forecasting using a hybrid network, IET Generation, Transmission & Distribution, Volume 1, Issue 1, January, 2007.
Day ahead LMP forecasting Forecasting Method ANN [1] ANN [2] ANN [4] MAPE 15.96% 12.92% 8.67% SVM [3] 11.31% GARCH [5] 11.55% ARMAX [6] 9.01% [4] P. Mandal, T. Senjyu, N. Urasaki, and T. Funabashi, A neural network based several-hour-ahead electric load forecasting using similar days approach, Int. Journal. of Electric Power and Energy System, vol. 28, no. 6, pp. 367 373, July 2006. [5] R. Garcia, J. Contreras, A GARCH Forecasting Model to Predict Day-Ahead Electricity Prices, Int. Journal of Electric Power and Energy System, vol. 20, no. 2, MAY 2005. [6] J. Zhang, C. Yu, G. Hou, Application of Chaotic Particle Swarm Optimization in the Short-term Electricity Price Forecasting, Int. Journal of Electric Power and Energy System, vol. 28, no. 6, pp.
Real-time LMP forecasting Actual and forecast six-hours-ahead electricity price for Victorian electricity market (Nov 1 st to 7 th, 2003). Actual and forecast six-hours-ahead electricity price for PJM market (Jan 8 th to 14 th, 2006). [4] F. Sarafraz, H. Ghasemi, H. Monsef, Locational Marginal Price Forecasting by Locally
Summary of the state-of-the-art Extensive literature: Mostly black box techniques Primarily providing point forecasting Rarely deal with on LMP and network effects Extremely accurate load forecasting (1-3%) Relatively poor price forecasting (10-20%)
Outline Motivation Probabilistic forecasting of real-time LMP A stylized real-time ex post LMP model Information structure LMP states and Markov chain representations Probabilistic forecast Preliminary simulations Summary and future work
A stylized real-time ex post LMP model
State estimation
Ex ante dispatch
Ex post eligible generators
Ex post LMP
Ex post LMP via IDCOPF
Information structure and Markov chain
Probabilistic forecasting
Outline Motivation Probabilistic forecasting of real-time LMP Preliminary simulations IEEE 16 bus with NYISO sample load profiles Varying load errors and correlations Monte Carlo techniques to estimate transition matrices Summary and future work
Load and price distributions 1 LMP distribution at time 15 38 36 34 Probability 0.5 0 1 2 3 4 5 6 7 8 9 1011 Markov State (LMP) index Scaled average actual load profile (upper/lower bounds for predicted load profile) from NYISO June 1 st actual load profile upper bound lower bound 32 1 LMP distribution at time 42 MW 30 28 26 24 22 Probability 0.5 0 1 2 3 4 5 6 7 8 9 1011 Markov State (LMP) index Probability 1 0.5 0 LMP distribution at time 119 1 2 3 4 5 6 7 8 9 1011 Markov State (LMP) index Probability 1 0.5 0 LMP distribution at time 220 1 2 3 4 5 6 7 8 9 1011 Markov State (LMP) index 20 0 15 42 119 200 288 time index (5 mins interval in 24 hours)
LMP trajectories $/MW 50 45 40 35 30 25 load bus1 bus2 bus3 bus4 bus5 bus6 bus7 bus8 bus9 bus10 bus11 bus12 bus13 bus14 LMP trajectories 20 15 0 50 100 150 200 250 time index (5 mins interval in 24 hours)
Time inhomogeneous Markov Chain
Transition Matrices Starting Markov State (LMP) index Transition matrix at time 15 2 4-1 6-2 8 10-3 12-4 2 4 6 8 10 12 Ending Markov State (LMP) index Starting Markov State (LMP) index 2 4 6 8 10 12 Transition matrix at time 29 2 4 6 8 10 12 Ending Markov State (LMP) index -1-2 -3 Starting Markov State (LMP) inde Transition matrix at time 117 2 0 4-2 6-4 8 10-6 12-8 2 4 6 8 10 12 Ending Markov State (LMP) index Starting Markov State (LMP) index 2 4 6 8 10 12 Transition matrix at time 220 2 4 6 8 10 12 Ending Markov State (LMP) index 0-1 -2-3
6 hours ahead prediction error Probability of incorrect prediction Absolute Percentage Error (MAPE 10 0 10-2 10-4 10-6 10 8 6 4 2 0 Prediction error for best and second best prediction Best prediction 2 nd best prediction 20 40 60 80 100 120 140 160 180 200 time index (5 mins interval in 12 hours) Starting Markov State (LMP) index Prediction transition matrix at time 21 0 2-0.5 4-1 6-1.5 8-2 10-2.5 12-3 2 4 6 8 10 12 Ending Markov State (LMP) index MAPE of best (most likely) prediction 20 40 60 80 100 120 140 160 180 200 time index (5 mins interval in 12 hours) Starting Markov State (LMP) index Prediction transition matrix at time 21 0 2-0.5 4-1 6-1.5 8-2 10-2.5 12-3 2 4 6 8 10 12 Ending Markov State (LMP) index
LMP prediction for different time horizon $/MW 50 40 30 20 Actual Best prediction 2 nd best LMP prediction for 5 mins ahead $/MW 50 40 30 20 LMP prediction for 1 hour ahead Actual Best prediction 2 nd best 10 10 $/MW 0 50 40 30 20 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Bus index Actual Best prediction 2 nd best LMP prediction for 3 hours ahead $/MW 0 50 40 30 20 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Bus index LMP prediction for 6 hours ahead Actual Best prediction 2 nd best 10 10 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Bus index 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Bus index
LMP distribution for different prediction horizons LMP distribution for 5 mins ahead LMP distribution for 1 hour ahead 10 0 10 0 Probability (Log) 10-2 Probability (Log) 10-2 10-4 10 0 1 2 3 4 5 6 7 8 9 10 11 12 13 Markov State (LMP) index LMP distribution for 3 hours ahead 10-4 10 0 1 2 3 4 5 6 7 8 9 10 11 12 13 Markov State (LMP) index LMP distribution for 6 hours ahead Probability (Log) 10-2 Probability (Log) 10-2 10-4 1 2 3 4 5 6 7 8 9 10 11 12 13 Markov State (LMP) index 10-4 1 2 3 4 5 6 7 8 9 10 11 12 13 Markov State (LMP) index
Planned research Forecasting techniques Incorporating load models More sophisticated Monte Carlo techniques (important sampling, MCMC) Generator behavior models Forecasting with renewable sources Demand response Extensive simulation studies and comparisons
DATA QUALITY AND ITS EFFECTS ON MARKET OPERATIONS DENY OF SERVICE ATTACK ON REAL-TIME ELECTRICITY MARKET Lang Tong (PI), Robert J. Thomas, and Jinsub Kim Cornell University Presented at CERTS Review August 7-8, 2013
Observability
A graph theoretic condition (KCD 80)
Undetectable attack (KJTT 11)
Only half of the meters are needed!
Bad data identification and removal
Framing attack via QCQP
DoS attack on real-time operations
Simulation examples (AC)
Conclusion Attacks on cyber physical systems (power systems) are real: State attacks Topology attacks Dispatch attacks A key step toward protection is deploy authentication mechanisms.