Electricity market price spike forecast with data mining techniques

Transcription

1 Electric Power Systems Research 73 (2005) Electricity market price spike forecast with data mining techniques Xin Lu a,1, Zhao Yang Dong b,, Xue Li c a School of Computer, University of Electronic Science and Technology of China, China b Z.Y. Dong is with the School of Information Technology and Electrical Engineering, The University of Queensland, St Lucia, QLD 4072 Australia c X. Li is with the School of Information Technology and Electrical Engineering, The University of Queensland, St Lucia,QLD4072 Australia Received 8 March 2004; received in revised form 2 June 2004; accepted 13 June 2004 Available online 8 September 2004 Abstract Electricity market price forecast is a changeling yet very important task for electricity market managers and participants. Due to the complexity and uncertainties in the power grid, electricity prices are highly volatile and normally carry with spikes, which may be tens or even hundreds of times higher than the normal price. Such electricity spikes are very difficult to be predicted. So far, most of the research on electricity price forecast is based on the normal range electricity prices. This paper proposes a data mining based electricity price forecast framework, which can predict the normal price as well as the price spikes. The normal price can be predicted by a previously proposed wavelet and neural network based forecast model, while the spikes are forecasted based on a data mining approach. This paper focuses on the spike prediction and explores the reasons for price spikes based on the measurement of a proposed composite supply demand balance index (SDI) and relative demand index (RDI). These indices are able to reflect the relationship among electricity demand, electricity supply and electricity reserve capacity. The proposed model is based on a mining database including market clearing price, trading hour, electricity demand, electricity supply and reserve. Bayesian classification and similarity searching techniques are used to mine the database to find out the internal relationships between electricity price spikes and these proposed. The mining results are used to form the price spike forecast model. This proposed model is able to generate forecasted price spike, level of spike and associated forecast confidence level. The model is tested with the Queensland electricity market data with promising results. Crown Copyright 2004 Published by Elsevier B.V. All rights reserved. Keywords: Electricity price forecast; Price spike; Data mining; Statistical analysis 1. Introduction The electricity market clearing price is the most important information in a competitive electricity market to all the market participants including generation companies, retail companies, transmission network providers and market managers. Market participants have been continuously attempting to be able to forecast the market clearing prices in order to stay competitive in a competitive market. For a generation company, the ability to accurately forecast the market Corresponding author. Tel.: ; fax: addresses: luxinmail@uestc.edu.cn (X. Lu), zdong@itee.uq.edu.au (Z.Y. Dong), xueli@itee.uq.edu.au (X. Li). 1 This work is performed while visiting the School of Information Technology and Electrical Engineering, The University of Queensland, St Lucia, Qld. 4072, Australia. clearing price means that the company is able to bid into the market strategically as well as to optimally manage its assets. A retailer company is able to buy at lower prices with prior knowledge. The market manager is able to provide better management and planning for the electricity market with better forecasting prices. It has been continuous efforts for electricity price forecast research in recent years [1 14]. Various techniques have been used in literature for price forecasting. An efficient way of forecast is by using regression models including ARIMA and fuzzy-neural autoregressive model [1,2]. Recently, many of the forecast techniques are based on the learning capability of artificial neural networks (ANNs) trained with historical price data [3,4]. Time series analysis is another important attempt in this direction with promising results [5,6]. Signal processing techniques have been used in time series based price forecast with the ANN /$ see front matter. Crown Copyright 2004 Published by Elsevier B.V. All rights reserved. doi: /j.epsr

2 20 X. Lu et al. / Electric Power Systems Research 73 (2005) learning modules to discover the hidden dynamics and trends in the price data series in order to improve the forecast performance [7]. It is pointed out in [8] that stochastic regime switching models are able to describe electricity price behavior to suit the needs for financial analyses. The author of [8] also proposed an idea of representing the probability of switching to a high-price regime as a function of load that provides a quantitative measure for anticipating price spikes. However, most of the existing techniques are effective only on the normal price signals only. In those approaches electricity price spikes have to be removed before applying such forecasting models; otherwise a very large or even completely wrong forecasted price will be generated at price spike occasions. There is currently a lack of forecasting models that can predict not only the normal price but also the volatile price spikes. This is more important in a pool type market such as the National Electricity Market (NEM) of Australia where the Value of Lost Load (VOLL) is as high as $ 10,000 per MWh [15]. Under such market, the price spikes are more influential to market participants, because a price spike could be hundreds of times higher than the normal price and a short period of time in such high price spike could eliminate the annual profit of a company. This paper aims at building a forecasting model especially for the price spikes. The model should be able to handle the uncertainty of spike occurrence and difficulty to predict characteristics. It is able to generate satisfactory price spike forecasts, level of spikes and confidence level. The paper is organized as follows: Section 2 describes the framework of price spike forecast model; Section 3 defines the price spikes and data pre-processing; Section 4 discusses the main factors resulting in price spikes in an electricity market; Section 5 describes the price spike forecast model; Section 6 details the case study of the proposed forecast model applied to the Queensland market of the Australian NEM; and finally Section 7 concludes the paper. 2. Framework of electricity price forecasting Electricity price forecasting is to predict the market clearing prices based on historical data. To the best of the authors knowledge, almost all the existing electricity price forecast techniques require filtering out the price spike signals in order to produce the normal electricity price forecast. By doing so, normal prices can be forecasted with rather high accuracy; however these techniques generally are unable to predict the price spike signals. This is mainly because of the lack of historical data of price spikes and the fact that a single model is hardly able to represent two distinctive trends in time series. Consequently, these existing forecast models have very poor price spike forecast capability or do not have such capability at all. In this paper, the proposed comprehensive price forecast model is composed of a normal price forecast module and a price spike forecast module see Fig. 1. The electricity market clearing price is separated into two parts, i.e. normal Fig. 1. Flow chart of the comprehensive electricity price forecast model. prices and price spikes which are processed by the normal price forecast module and the price spike forecast module, respectively. The normal price forecast module is based on the authors previous work on neural network wavelet decomposition time series load and price forecast module [7]. The normal price forecast module is well established and tested to be effective. This paper focuses on the price spike forecast module, which uses the historical price spike information for forecasting. This module is able to explore the two distinctive trends in normal and price spikes to generate satisfactory and comprehensive forecast results. Historical electricity price data including both the normal prices and the price spikes are fed into the forecast model. The pre-processing module separates the two price signals. The normal prices are further processed by a wavelet decomposition based neural network (NN wavelet) time series forecasting module. The NN wavelet module also predicts the possibility of price spikes at specific occasions. If a specific occasion is forecasted to have a price spike signal, then the price spike forecast module is activated with all other historical price spike data to estimate the values of the price spikes, stage of the price spikes and level of the forecast confidence. Both the forecasted normal prices and price spikes are then reconstructed to form the overall comprehensive electricity price. 3. Price spike and data pre-processing The NEM of Australia is composed of states of Queensland, New South Wales, Victoria, South Australia and the snowy region with Tasmania to join soon. The National Electricity Market Management Company Limited (NEMMCO) publishes the historical and real time data of the NEM regional reference price (RRP) through its website. With some study of the published historical data, it is easy to identify some special RRPs as high as several thousands of Australian

3 X. Lu et al. / Electric Power Systems Research 73 (2005) dollars per MWH, which is several hundred times higher than the normal price around $ per MWH. For example, at on 31 July 2003 the RRP in NSW electricity market is as high as $ per MWH. Such abnormal prices are referred as price spikes. These price spikes occur following the market bidding process and are results of unexpected events such as transmission network contingencies, transmission congestion and generation contingencies. Price spikes are highly uncertain and randomized, and are difficult to predict. Their existence also has considerable impact on normal price forecast [3]. In order to get better normal price forecast, the impact from price spikes should be carefully eliminated from historical price data. Nevertheless, price spikes carry with very important information and should not be simply replaced by average values from neighboring data pointes. Special processing techniques are required for such processing stage Categorization of the price spikes In the proposed forecast model, we further categorise price spikes to be those abnormal RRP data. They are usually much higher than the mean RRP value, but they may also be very low or negative value. Queensland electricity market RRP data on 18 January 2003 are given in Fig. 2. It can be seen that there are several price spike data during the day peak demand hours or 15:00, 18:00 19:00 while the mean RRP value is around $ 28 per MWH [16]. In our proposed model, price spikes are determined by the following approaches: (1) Based on statistics the outlier price is calculated from the historical data set. Let µ be the mean value of the historical data set, δ be the standard deviation of the data set, and P ν be the abnormal data threshold value of the sample data set. P ν can be calculated by [17,18]: P ν = µ ± 2δ (1) Any RRP > P ν is regarded as outlier price type price spike. (2) Based on the experience with electricity market prices, abnormal high price can be determined. An abnormal high price threshold value P τ can be calculated based on the probabilistic distribution of electricity prices of each electricity market. If RRP >P τ (2) then this particular RRP is regarded as an abnormal high price type price spike. (3) Abnormal jump price can be calculated by examining the price difference between two neighboring time prices. Let current time price be RRP(i) and the previous time price be RRP(i 1), the difference at time i is RRP(i) = RRP(i) RRP(i 1). Let P be the maximum jump different price in the normal price range, then for any RRP(i) > P (3) The current time price RRP(i) is regarded as abnormal price jump type price spike. (4) Negative price refers to negative RRPs, i.e. any RRP < 0 (4) is regarded as negative price type price spike. Among all these four price spikes, the simple and most commonly used abnormal high price type price spike is used for analysis in this paper. The threshold value P t is different depending on the different electricity market. In CA market, the high price threshold value P t is US$ 50 per MWh and all prices higher than this value are regarded as price spikes. In this paper, after statistical analysis, a threshold value P t of $ 75 per MWh is used for Queensland electricity market analysis. All RRPs higher than P t = $ 75 per MWh are regarded as price spikes in this market. It should be noted that the price spike should be different at different time even in the same electricity market to truly reflect the characteristics of the particular market. A simple experience based threshold value is straightforward in price spike definition; however it can only identify large price spikes from the price data set. It fails to identify those smaller price spikes. For example, in Fig. 2, there are considerable jump difference at 12:20 and 15:00 compared with there priori prices. However, these prices are still within the threshold value of $ 75 per MWh and are not identified as price spikes based on this simple criteria. Nevertheless, these Fig. 2. RRP of Queensland electricity market on 18 January 2003.

4 22 X. Lu et al. / Electric Power Systems Research 73 (2005) abnormal jump prices are important in price analysis and are regarded as abnormal jump type price spikes for further analysis Data pre-processing The first step for price forecast is data pre-processing. We perform data mining on the large amount of historical data to identify information relevant to price spikes. Such information includes date, time, RRP, electricity demand, electricity supply capacity, electricity reserve capacity and the difference from their previous values such as RRP variation difference, demand variation difference, supply capacity difference and reserve capacity difference. Because of the unavailability of weather data they are not included in this case study even though they do have impact on price spikes. These mined data are stored into a database for further usage at the forecast stage. The following symbols are used to represent the individual information at time i: Date(i) date, e.g. 1 Sep 2003 Time(i) time, e.g. 18:30 RRP(i) regional reference price at time i, e.g. $ 9.28 per MWh Demand(i) electricity demand at time i, e.g MWh Supply(i) electricity supply capacity at time i, e.g MWh Reserve(i) electricity reserve at time i, e.g MWh RRP(i) RRP different at time i as compared with previous time i 1, i.e. RRP(i) RRP(i 1), e.g. $ 3.21 per MWh, $ 4.35 per MWh Demand(i) demand difference at time i as compared with the previous time, e.g. 452 MWh, 75 MWh Supply(i) supply difference at time i as compared with the previous time, e.g. 200 MWh, MWh Reserve(i) generation reserve difference at time i as compared with the previous time, e.g. 80 MWh. The relationship holds among electricity reserve, supply capacity and demand: Reserve(i) = Supply(i) Demand(i) (5) 4. Factors affecting the market price Although determined by the independent system operator (ISO), the electricity market price is a result of many factors including current demand, supply capacity, reserve capacity, network conditions, generation bids and seasonal impacts. Among these factors, the demand and supply relationship is the essential one. Other factors will ultimately reflect in demand and demand supply balance reserve. We will start the analysis with this determining factor to find out their impacts on the price signal RRP. Then the time series characteristics of RRP and its probabilistic distribution characteristics at different stages will be analyzed. Finally, we will locate the reasons resulting for price spikes ARRP related factors RRP versus demand In almost every market, the price is nearly proportional to the demand. The higher is the demand the higher is the price; however this relationship is hard to formulate accurately in the electricity market. As shown in Fig. 3, the distribution of the RRP versus demand roughly indicates such relationship RRP versus reserve The relationship between reserve and the electricity price of Queensland electricity market of the Australian NEM is given in Fig. 4. The figure clearly indicates that the reserve capacity is approximately proportional in a nonlinear way to the inverse of the RRP. When the reserve capacity is sufficient enough, the electricity price is more likely to be in lower value ranges, otherwise is more likely to be in the higher value ranges. Statistical analysis also shows that when the reserve capacity is less than 20 30% of the total demand, the electricity price RRP is more likely to increase at a higher rate Time series analysis of RRP data The price data is naturally a time series. The Queensland electricity market RRP signal is given in Figs. 5 and 6, which exhibits the seasonal weekly and daily trends as time series [19]. It reflects the demand and supply balance of an electricity market. In Australian NEM it is defined as reserve = available generation capacity demand network transmission capacity. Each electricity market within the Australian NEM defines its own low reserve condition (LRC) as an index for alarm signals to the available reserve capacity for the market. Currently, the LRC values are [18] given in Table 1. Table 1 List of LRCs of the four major NEM regions Market NSW VIC SA QLD LRC 660 MW 540 MW 260 MW 450 MW Fig. 3. Demand vs. RRP of Queensland electricity market in January September 2003, where x axes represents the demand in MW and y axes represents the RRP in AU$/MWh.

5 X. Lu et al. / Electric Power Systems Research 73 (2005) The RRP distribution of Queensland market has very low price spike probability (<1%), and with over 89% probability in $ per MWh range The main reasons causing price spike Fig. 4. Electricity reserve vs. RRP of Queensland electricity market in January September 2003, where x axes represents the demand in MW and y axes represents the RRP in AU$/MWh. It can also be seen from Figs. 5 and 6 that the price signal has higher volatility in summer and winter peak times. The high prices are more likely to happen during peak times. There are many time series trends in the price data set together with some price spikes Probabilistic distribution of different RRP ranges Cumulative distribution of the regional reference price data shows the distribution of the price in different price ranges. Such distribution of Queensland electricity market during January September 2003 is given in Table 2. In an ideal competitive electricity market, price spikes only happen when the demand exceeds supply. In fact most of the electricity markets are not ideal competitive market; consequently, price spikes happen even when there is sufficient supply to meet the demand. Both abnormal high price type and negative price spikes have significant impact on the market participants. The main reasons causing price spikes are listed as follows: (1) Practice of market power by generation companies at higher demands and with transmission limits. (2) A large number of generation companies have high generation reserve will cause the demand supply curve to shift and cause possible price spikes. (3) Contingency conditions such as unexpected extreme weather conditions may cause price spikes. (4) Insufficient transmission or generation infrastructure due to either under estimated demand forecast or lack of market stimulation for new market entries may also cause price spikes. Fig. 5. RRP of Queensland electricity market as time series with weekly trends from 9 September to 15 September Fig. 6. RRP of Queensland electricity market as time series with monthly trends from January to September 2003.

6 24 X. Lu et al. / Electric Power Systems Research 73 (2005) Table 2 Cumulative probability distribution of RRP of Queensland electricity market in January September 2003 RRP range ($/MWh) < Frequency Probability (%) RRP range ($/MWh) >5000 Total Frequency Probability (%) (5) Other conditions such as transmission congestion, supply scarcity and market power can also contribute to price spikes. Based on the general analysis, considering the situation of Queensland electricity market, the probability of price spikes has close relationship with demand, reserve and time. It can be summarized as follows: (1) The probability of price spikes is high when the demand is high, otherwise is low. (2) The spike probability is high when generation reserve is smaller than a certain level; the reserve level has closer relation with price spikes. (3) Price spike probability is higher at daily peak hours, and is lower otherwise. (4) Price spike probability is higher at working days than that of weekends and public holidays. Generally, every electricity market has its own economic characteristics in view of its demands, generation service providers, transmission services and market management rules. Accordingly, the factors affecting market prices are different to some extent and the main factors contributing to price spikes should be analyzed individually to consider the specific characteristics of each market. 5. Price spike forecast model The fundamental characteristic of an electricity market is the demand supply balance. Most of other market factors are directly or indirectly reflected by demand supply balance. The price spike forecast model is based on data mining on the information from the spot market, including the historical market price, demand, supply and reserve data. From previous analysis, it is clear that the market price has close relationship with such factors as demand, supply and reserve. However, a single relationship is able to reflect only a fraction of the overall complex relationship with prices. As a result, we propose a composite relationship between RRP price and demand, reserve and relative demand levels. We first define the following two important terms which will be used in the forecast model in order to have a better insight into the factors affecting RRPs Supply demand balance index (SDF) SDI refers to the balance between electricity demand and supply, and is defined by Eq. (6): [ ] (Supply(i) Demand(i)) SDI = 100% Demand(i) [ ] Reserve(i) = 100% (6) Demand(i) where Demand(i) is the market demand at time/occasion i, Supply(i) is the electricity supply capacity at occasion i and Reserve(i) is the electricity supply reserve capacity at i Relative demand index (RDI) RDI refers to the relative degrees of current time demand with the initial demand of the trading day, and is defined by Eq. (7): RDI = Demand(i) (7) Demand b where Demand b is the demand value at the beginning of the trading day, which is 4:30 for Australian NEM. It is well known that moving average model is a useful forecast tool for time series. In our approach, moving average can be used in the base price forecast as time series. However, in price spike forecast, a fixed moving window is needed to for application of a moving average model to reflect the relative changes in the price signals for predicting the spikes. This will increase the complexity of the method especially with no priori knowledge on the price spike information and is therefore not employed here. As a result, we define the RDI Fig. 7. Scatter plot of RRP vs. SDI for Queensland electricity market in January September 2003, where x axes is the SDI and y axes is the RRP in AU$/MWh.

7 X. Lu et al. / Electric Power Systems Research 73 (2005) Fig. 8. Line Graph of RRP vs. SDI for Queensland electricity market from 7 to 14 June A. Forecasting the occurrence of price spikes: the first step in price forecasting is to predict if price spike is going to happen at certain occasion. Basically, this is achieved by normal price forecast on RRP i. If the forecasted normal price RRP i is greater than a threshold price value then a price spike may happen at occasion I: { True, RRPi > $ 75 per MWh, Spike(RRP i ) = False, RRP i $ 75 per MWh. (8) Fig. 9. Scatter plot of RRP vs. RDI for Queensland electricity market in January September 2003, where x axes is the RDI and y axes is the RRP in AU$/MWh. based on the initial demand of the trading day instead of on a moving average basis. The relationship between RRP and SDI is shown in Figs. 7 and 8. The relationship between RRP and RDI is given in Figs. 9 and 10. It can be seen from Figs that SDI and RDI are able to describe their relationship with RRP more accurately than their original demand and reserve data. They also reflect the composite impact on market prices from demand and reserve. There are three steps to predict the electricity price by the proposed forecast model: The occurrence of price spike can also be determined by comparing the trading information with that of the previous occasion.b. Forecasting the range of price spikes: as shown Figs. 7 10, both the relationships either RRP SDI or RRP RDI are all nonlinear. In addition the price spikes are naturally highly stochastic and largely randomly distributed. It is very difficult to predict both the normal prices and price spikes. We use two approaches to deal with forecasting of these two price signals respectively in order to achieve higher accuracy and reliability. We are only interested in the relationship of RRP SDI and RRP RDI during the price spike periods. From the historical data the probabilistic distribution of RRP versus SDI and RDI can be computed to further assist in price forecast. The range of price spikes can be forecasted based on the probabilistic distribution of RRP versus SDI and Fig. 10. Line Graph of RRP vs. RDI for Queensland Electricity Market from 7 Jun to 14 Jun 2003.

8 26 X. Lu et al. / Electric Power Systems Research 73 (2005) RDI through data mining techniques such as categorization algorithms. More specifically, the range of RRP can be predicted based on the SDI and RDI values which will be shown in the sequel. Currently, there are many data mining techniques that can be used for this purpose such as, Judgment Tree categorization method, Bayesian categorization method, Neural Network categorization, Correlation based categorization, closest k-neighborhood categorization, reasoning based categorization, Genetic Algorithm based categorization, rough set and fuzzy set based categorization [20]. Bayesian categorization is a statistic analysis based approach. It is able to predict the probability of the relationships among different members, i.e. it is able to produce the reliability of the forecast. It is also easy to use and is therefore employed in this paper for the price range forecast module. For completion, Bayesian method is summarized in the following [20]: (1) Let an n-dimensional vector X = {x 1, x 2,..., x n } to represent each data sample where x i, is the value of the sample s ith dimensional property. Assuming that there are m categories C 1,C 2,...,C m in each data sample. For a given data sample X with unknown categories. (2) Categorization method can be used to predict the category C, where X belongs to, if and only if P(C i X) >P(C j X), I j m, j i (9) According to Bayesian theorem: P(C i X) = P(X C i)p(c i ) (10) P(X) (3) If the data sample set has too many properties, i.e. if n is very large, the computational cost of computing P(X C i ) can be very high. In order to reduce the computational cost, we assume that the properties are independent, then we have P(X C i ) = n P(X k C i ) (11) k=1 The data samples properties are corresponding range of values are given below: SDI [< 2, 2...3, 3...4, 4...5, 5...6,>6], RDI [< 1.2, , , , , >1.6], RRP [ , , , , , >2000] where SDI and RDI are the input properties of the data sample; RRP is category identification property. The number of properties is small in the data sample for the proposed model. The data sample input can also use the joint information from SDI and RDI as well, i.e. SDI RDI < SDI, RDI > (12) Consequently, Bayesian method can be used directly without introducing possible errors caused by the property independency assumption. (4) The training data sample is then collected through data mining from the database. With the training data sample, Bayesian method can be used to categorize those new unknown data samples, i.e. to predict the price spike ranges and associated level of confidence based on SDI and RDI. C. Forecasting the price spike values: after having determined the range of price spikes following the 4 steps in stage (B), it is of considerable interest for market participants to be able to further predict the actual values of the price spike within the predicted range. We use the k-closest point approach for this task. From the training data samples, k neighboring samples closest to the unknown sample are selected. Then the average value of the k-closest samples is computed as the unknown sample s value. The distance to the neighboring samples is determined by the root mean square value as given in Eq. (13): d(x, Y) = n ( ) X i Y 2 i (13) MAX i MIN i i=1 where X = {x 1, x 2,..., x n } and Y = {y 1, y 2,..., y n } are two neighboring samples, MAX i and MIN i represent the maximum and minimal values of samples X and Y. Let ε R + be a threshold value defining the neighbors in the sample space, then all the points in this space with a distance to the unknown sample d(x, Y)<ε are regarded as neighbors of this unknown sample. Assuming there are k (k 1) such neighboring samples, then the unknown sample s RRP value can be predicted as RR P = 1 k k RRP(i) (14) i=1 6. Price spike forecast case study The proposed price forecast model is used to predict the price spikes of the Queensland electricity market of Australian NEM. The market RRP values, demand, generation reserve and other historical data during January June 2003 are used to establish the training data sample sets. The data during July September 2003 are used as test data. The overall performance is limited due to the fact that there are only 9 months data available at the time of this paper is written. The forecast model s performance will be enhanced should more data available in the future. Some of the typical case study results are given in Tables 3 5.

9 X. Lu et al. / Electric Power Systems Research 73 (2005) Table 3 Frequency and probability of price spike of Queensland electricity market during January June 2003 Range of price spike Total >2000 Frequency Probability (%) P(C 1 ) = 28.2 P(C 2 ) = 28.2 P(C 3 ) = 14.1 P(C 4 ) = 12.8 P(C 5 ) = 14.1 P(C 6 ) = Table 4 Confidence level of RRP estimated from SDI RDI relationship SDI, RDI Range of spikes , P(X C 1 ) , P(X C 2 ) , P(X C 3 ) , P(X C 4 ) , P(X C 5 ) >2000, P(X C 6 ) SDI 2, 0 < RDI C = 1/1 0 2 < SDI 3, 1.2 < RDI 1.3 C = 4/6 0 C = 2/ < SDI 3, 1.3 < RDI 1.4 C = 6/12 C = 2/12 C = 4/ < SDI 3, 1.4 < RDI 1.5 C = 3/6 C = 3/ < SDI 3, 1.5 < RDI 1.6 C = 4/9 C = 2/9 C = l/9 C = 2/ < SDI 4, 1.3 < RDI 1.4 C = 2/ < SDI 4, 1.4 < RDI 1.5 C = l/7 C = 4/7 C = l/7 C = l/ < SDI 4, 1.5 < RDI C = 6/17 C = 3/17 C = 4/17 C = 4/ < SDI 5, 1.4 < RDI 1.5 C = l/3 C = l/3 0 C = l/ < SDI 5, 15 < RDI 1.6 C = 1/11 C = 4/11 0 C = 2/11 C = 2/11 C = 2/11 4 < SDI 5, 1.6 < RDI C = 3/ < SDI 6, 1.5 < RDI C = 1/1 0 From Table 5 it can be seen that the price spike forecast has an accurate rate over 50%. Most of the error rate is less than 30% with only one close to 49% on 31 July 2003 where the forecasted RRP is $ per MWh with 100% confidence and the actual RRP is $ per MWh. It can be seen that even if the forecast error is bit higher for this particular entry, but the forecasted confidence level of 100% and the actual RRP of $ are already extremely useful for any market participants making their decisions based on the price forecast. The forecast error for this particular day can easily be understood given that the actual price is much higher than the average price spikes and the forecasted RRP value is already much higher than the forecasted average values. In fact other forecasted RRPs are also close enough to the real values to provided useful information for the users. In addition, give the high volatility of this particular point, most price forecast techniques will have it removed from forecasting analysis, while the proposed price spike forecast method can provide very useful and close enough forecast for market applications. Overall, the seemingly not high rate in forecasting is mainly because of the very limited availability of price spike data in the historical data (with less than 1% probability of price spikes during January September 2003 in Queensland Electricity Market). Seeing the fact that price spikes are highly stochastic, the achieved forecast accuracy level is sufficiently good. With more training historical data, the accuracy can be improved accordingly. It is worth mentioning that with the NN wavelet time series forecast model [7], the normal price without spikes can be forecasted with very high accuracy (typically less than 2 5% forecast error). Table 5 Comparison between the actual and forecasted price spikes of the Queensland electricity market during July September 2003 Date, time SD1 RDI Forecast interval Confidence (%) Forecast RRP RRP Forecast error (%) 5 July 2003, 18: [75 100] July 2003, 18: [ ] July 2003, 19: [ ] July 2003, 18: [ ] July 2003, 19: [l00 150] July 2003, 18: [ ] July 2003, 19: [ ] July 2003, 19: [ ] August 2003, 18: [75 100] September 2003, 18: [75 100] October 2003, 18: [75 100]

10 28 X. Lu et al. / Electric Power Systems Research 73 (2005) Combining the normal price range forecast and the price spike forecast results this model is able to provide comprehensive results with very useful and reliable information for an electricity market. 7. Conclusions This paper analyzes the price spikes in electricity market price forecast and proposes a composite forecast model which is able to handle normal prices as well as price using different methods. Our approach ensures the accuracy of normal price forecast and is able to provide price spike forecast with a high level of confidence. Among many different factors in an electricity market, the main factor affecting the market price is the demand supply relationship. This relationship is the essential information used in price forecast in the proposed model. In order to describe their impact on prices precisely and evidently, two new indices are introduced in this paper, the SDI and RDI. Their relationships with RRP are also defined. The normal price forecast module is based on wavelet neural network based time series forecast methods and the price spike forecast is based on Bayesian methods of classification. Based on the SDI and RDI values, the range of price spikes and the level of confidence of the price spikes can be forecasted using Bayesian method, K-closest neighbor method is used for further estimating the value of price spikes in the predicted range of prices. The data set of Queensland electricity market of the Australian NEM is used to test the proposed model with promising results. The main contribution of this paper is the ability to predict the price spikes on top of the normal prices with data mining techniques. This contribution greatly enhanced the applicability of price forecasts by electricity market participants. Acknowledgement The research project is supported by Overseas Scholarship Program, UESTC (University of Electronic Science and Technology of China). The first author would like to thank the School of ITEE of University of Queensland, Australia, for their good working environment. This work is supported in part by the research reported in this paper is supported by Oversea Scholarship Program of UESTC (University of Electronic Science and Technology of China). [2] T. Niimura, H.-S. Ko, A day-ahead electricity price prediction based on a fuzzy-nearo autoregressive model in a deregulated electricity market, in: Proceedings of the 2002 International Joint Conference on Neural Networks (UCNN 02), vol. 2, no , 2002, pp [3] F. Gao, X.H. Guan, X.-R. Cao, A. Papalexopoulos, Forecasting power market clearing price and quantity using a neural network method, in: Proceedings of the IEEE Power Engineering Society Summer Meeting, 2000, pp [4] J.-J. Guo, P.B. Luh, Selecting input factors for clusters of gaussian radial basis function networks to improve market clearing price prediction, IEEE Trans. Power Syst. 8 (2) (2003) [5] F.J. Nogales, J. Contreras, A.J. Conejo, R. Espinola, Forecasting next-day electricity prices by time series models, IEEE Trans. Power Syst. 17 (2) (2002) [6] S. Singh, P. McAtackey, Dynamic time-series forecasting using local approximation, in: Proceedings of the10th IEEE International Conference on Tools with Artificial Intelligence, 1998, pp [7] B.L. Zhang, Z.Y. Dong, An adaptive neural wavelet model for short term load forecasting, Int. J. Electr. Power Syst. Res. 59 (2001) [8] T. Mount, Are price spikes predictable, reproducible and avoidable? PSERC Seminar Presentation, Cornel University, October [9] H.M. Yang, X.Z. Duan, Chaotic characteristic of electricity price and its forecasting model, in: Proceedings of the IEEE Canadian Conference on Electrical and Computer Engineering (CCECE), 2003, pp [10] L. Zhang, P.B. Luh, K. Kasiviswanathan, Energy clearing price prediction and confidence interval estimation with cascaded neural networks, IEEE Trans. Power Syst. 18 (1) (2003) [11] Q.J. Shen, M.X. Huang, X.H. Tao, Electricity price behavior models and numerical solutions with trees, in: Proceedings of the International Conference on Power System Technology, 2002, pp [12] M. Benini, M. Marracci, P. Pelacchi, Day-ahead market price volatility analysis in deregulated electricity markets, in: Proceedings of the IEEE Power Engineering Society Summer Meeting, 2002, pp [13] H.J. Liu, X.F. Wang, W.C. Zhang, O.H. Xu, Market clearing price forecasting based on dynamic fuzzy system, in: Proceedings of the International Conference on Power System Technology, 2002, pp [14] E. Ni, P.B. Luh, Forecasting power market clearing price and its discrete pdf using a Bayesian-based classification method, in: Proceedings of the IEEE PES Winter Meeting, 2001, pp [15] National Electricity Code, available from the National Electricity Code Administrator (NECA). [16] National Electricity Market Statistical Digest, July September 2003, NECA, Australia, [17] M. Kantardzic, Data mining concepts, models, in: Methods and Algorithms, IEEE Press, [18] B. Liu, W. Hsu, Y. Ma, Integrating classification and association rule mining, in: R. Agrawal (Ed.), Proceedings of the Fourth International Conference on Knowledge Discovery and Data Mining, AAAI Press, New York, USA, 1998, pp [19] Characterizing Pool Price Volatility in the Australian Electricity Market, NECA, Australia, [20] J.W. Han, M. Kamber, Data Mining: Concepts and Techniques, Morgan Kaufmann, References [1] J. Contreras, R. Espinola, F.J. Nogales, A.J. Conejo, ARIMA models to predict next-day electricity prices, IEEE Trans. Power Syst. 18 (3) (2003) Biographies Xin Lu got his master s degree from the University of Electronic Science and Technology of China (UESTC) in He is now an Associate

11 X. Lu et al. / Electric Power Systems Research 73 (2005) Professor at the School of Computer in UESTC. His major research fields are data mining, real time system and software engineering. Zhao Yang Dong (M 99) received his Ph.D. in Electrical and Information Engineering from The University of Sydney, Australia in He is now a Senior Lecturer at the School of Information Technology and Electrical Engineering, The University of Queensland, Australia. His research interest includes power system security assessment and enhancement, electricity market, artificial intelligence and its application in electric power engineering, power system planning and management. Xue Li received his Ph.D. in Information Systems from Queensland University of Technology in He is now a Senior Lecturer at the School of Information Technology and Electrical Engineering, The University of Queensland, Australia. His research interest includes data mining, web mining, advanced database applications, and intelligent business systems.