Valuing Reliability in the National Electricity Market. Final Report

Transcription

1 Valuing Reliability in the National Electricity Market

2 DISCLAIMER This report has been prepared for the Australian Energy Market Operator (AEMO) as an input to a consultation process with stakeholders aimed at developing a consistent approach and timeframe to be used by AEMO in calculating and subsequently updating the value of customer reliability (VCR) for each of the NEM regions. The analysis and information provided in this report is derived in whole or in part from information prepared by a range of parties other than Oakley Greenwood (OGW), and OGW explicitly disclaims liability for any errors or omissions in that information, or any other aspect of the validity of that information. We also disclaim liability for the use of any information in this report by any party for any purpose other than the intended purpose. DOCUMENT INFORMATION Project Client Status Report prepared by Valuing Reliability in the National Electricity Market Australian Energy Market Operator Lance Hoch ([email protected]) Stuart James ([email protected]) Date by

3 Table of CONTENTS 1. Background, purpose and limitations Overview of options available Model-based approaches Survey based approaches Direct cost Economic principle of substitution Contingent Valuation (WTP and WTA) Choice modelling Conjoint analysis Detailed review of selected options Victorian Value of Customer Reliability (VCR) Objectives and how the method was used Overview of approach How results have been used Observed advantages and disadvantages for application to transmission reliability investment assessment Refinements to the 2007 methodology Willingness to Pay (via discrete choice) South Australia and New South Wales Objectives and how the method was used Overview of approach How results have been used Observed advantages and disadvantages for application to transmission reliability investment assessment Stated choice (New Zealand) Objectives and how the method was used Overview of approach How results have been used Observed advantages and disadvantages for application to transmission reliability investment assessment Extension of Victorian VCR to other NEM regions (demonstration) Overview of approach taken Survey information Outage probabilities Customer data sectoral weightings Corrections for missing data i

4 4.2. Results Sector-by-sector Year-by-year escalation of VCR Caveats on available data Potential for improved customer and outage data Observations, outstanding considerations and recommendations Summary of the applicability of the available approaches for assessing VCR to AEMO's needs Other considerations Desirable aspects of a VCR measure for transmission investment assessment regardless of approach ii

5 Table of FIGURES Figure 1: Overview of inputs to and calculation of the Victorian VCR Table of TABLES Table 1: Overview of approaches used for calculating VCR and related measures... 3 Table 2: Summary of CRA 2007 survey responses and reported accuracy Table 3: Sample size proposed and achieved in the South Australia WTP study Table 4: Original recommended sample size for NZ value of USE study Table 5: Region-by-region and outage probabilities Table 6: Mapping of New South Wales consumption data to VCR survey sector information Table 7: Region-by-region and sector-by-sector energy weightings Table 8: Region-by-region and sector-by-sector VCR ($/MWh) Table 9: Region-by-region and sector-by-sector VCR NERA methodology ($/MWh) Table 10: Region-by-region and sector-by-sector VCR OGW methodology ($/MWh) iii

6 1. Background, purpose and limitations Because consumers do not have explicit choice regarding the level of reliability they receive in their electricity supply, there is no market price for reliability. However, as improved reliability is a benefit provided in varying degrees by different types of electricity infrastructure investment, it behoves planners to have a value for reliability improvement to use in assessing the overall benefit/cost profile of alternative investment options. Different approaches have been used in various Australian and overseas jurisdictions to quantify the value customers place on the reliability of their electricity supply, but Victoria is the only Australian jurisdiction to date to have used an explicit value of customer reliability in its transmission planning investment decision process, though several other jurisdictions have considered doing so. The basis of the approach used in Victoria is documented in a study 1 undertaken by Monash University s Centre for Electrical Power Engineering in 1997 for the Victoria Power Exchange (VPX). VENCorp, which served as the transmission planning body for Victoria until those functions were transferred to the Australian Energy Market Operator (AEMO), commissioned Charles River Associates to implement the Monash methodology in 2002 and again in 2007 to calculate the value of customer reliability (VCR) 2 for six customer sectors, and for the state as a whole. Annual values between surveys have been derived using an indexation method developed for VENCorp by NERA Economic Consulting. In the 2009 National Transmission Network Development Plan (NTNDP) consultation, AEMO foreshadowed its intention to continue to update the Victorian VCR and to develop similar measures for the other NEM regions. This project was undertaken in light of that intention. Its objective is to (a) review the available approaches for calculating a value of customer reliability, and (b) prepare a set of NEM regional VCRs based on the Victorian approach. These materials will then serve as inputs to a consultation process with stakeholders aimed at developing a consistent approach and timeframe to be used by AEMO in calculating and subsequently updating the VCR for each of the NEM regions. 1 Dr M.E. Khan and Dr M. F. Conlon, Value of Lost Load, 1997, Centre for Electrical Power Engineering (CEPE), Department of Electrical and Computer Systems Engineering, Monash University, for the Victoria Power Exchange (VPX). 2 The corresponding metric derived from other approaches has been referred to variously as the value/cost of unserved energy, value of lost load, outage cost, customer cost of service interruption and value of supply security. Despite differences in how the different approaches conceptualise the value of reliability from the customer's perspective and the computational approach they use to quantify it, all of these measures are trying to quantify the cost of outages to customers, and/or the wider society/economy. 1

7 The timeframe and budget available for this project precluded undertaking any customer surveys. In any case, such an expenditure would be inappropriate given the fact that the calculation of regional VCRs undertaken in this report is only meant to be a demonstration of the way in which that metric would be developed, and therefore should not be taken as an attempt to calculate values that represent the true VCR within those jurisdictions. Consistent with this demonstration objective, the sectoral VCRs determined in the 2007 Victorian VCR study were combined with region-specific data from the other NEM regions on the number and duration of outages experienced over the past several years, and the volume of consumption within each of the six sectors used in the VCR 3. This allowed calculation of a 2007 value at the sector and state level for each NEM region. The 2007 VCRs for each sector and each region were then updated to 2010 values using the VENCorp indexation method. The regional data on outages and sector consumption were provided by the Distribution Network Service Providers (DNSPs) within each of the regions on a very tight timeframe literally less than two weeks from data request to data provision deadline. As a result, although some guidelines were provided, it was not possible to ensure that the data received had been prepared entirely consistently across the DNSPs. As stated in AEMO's Request for Proposal for this project, "The information that AEMO is primarily seeking is therefore the best possible estimates of the VCR for each NEM region that can be assembled in a matter of weeks using any existing data". This approach was deemed to be sufficient for providing a demonstration of how the approach used in Victoria could be extended to the other NEM regions. As stated above, it is not intended that the values calculated here be taken to represent the true VCRs of those other regions. The remainder of this report is organised as follows: Section 2 provides a brief overview of the various approaches that can be used to quantify the value customers place on reliability. Section 3 provides a more detailed review of the three approaches that have been used in Australia and New Zealand to do so. Section 4 undertakes the development of demonstration VCR values for each of the NEM regions using the Victorian approach. The spreadsheet that was used to do this by applying the Victorian VCR values to the data provided by the regional DNSPs has been published simultaneously with this report and should be referred to in conjunction with this section. Section 5 presents our observations regarding the various methods, the demonstration VCRs and the feasibility of using the Victorian approach in all of the NEM regions, along with any outstanding issues requiring consideration, and recommendations regarding the development of a consistent NEM method for valuing reliability from the customers' perspective. 3 The details of the regional data used, how it was developed and how the Victorian VCR values were applied to it to produce sectoral and regional VCR values for each region is presented in section 4. 2

8 2. Overview of options available Two general approaches have been used for calculating the value of customer reliability (VCR) or similar measures such as the value of unserved energy, or customers' willingness to pay for increased reliability or willingness to accept compensation for outages. These are model-based approaches and survey-based approaches. Several variants exist within each of the two basic types, as shown in Table 1 below 4. Table 1: Overview of approaches used for calculating VCR and related measures Customer sectors for which Approach type Specific approach specific approach is generally deemed to be applicable Model-based Income proxy Residential Case studies of blackouts Backup generator proxy GDP per kwh proxy (production function ) All Commercial and industrial Commercial and industrial Survey-based Direct cost Commercial and industrial Contingent valuation (WTP/WTA) Choice modelling Conjoint analysis Economic principle of substitution Residential, small & medium commercial and industrial Residential, small & medium commercial and industrial Residential small & medium commercial and industrial Residential Source: CRA, Value of Customer Reliability Discussion Paper, 2004, for ESIPC. Brief descriptions of these approaches are provided in the following two sub-sections of the report Model-based approaches Model-based approaches rely on data that is generally readily available either from secondary sources or direct observations of expenditures made by customers to ensure that they have a reliable source of electricity supply. As a result, model-based approaches do not require the time and expense of conducting surveys with customers. Examples of model-based approaches include: 4 Most of the information in this section of the report is taken from Value of Customer Reliability Discussion Paper, which was prepared by CRA for ESIPC in October 2004, and which includes significantly more detail on the range and applicability of available methods. 3

9 Income proxy, in which outage cost estimates are developed based on the assumption that the lost time experienced by residential consumers during an outage is equivalent to income earned. While this allows relatively easy calculation of the value, it can both overand under-estimate the cost incurred. For example, outages that occur when people are asleep do not impose any lost income. Although this over-estimation is sometimes said to 'make up' for the fact that indirect and inconvenience costs are not captured, there is little reason to believe that it does so at all accurately. Case studies of blackouts, including coverage of their durations, number of customers affected, maintenance and fix-up costs, etc. In this approach, case studies that have measured the economic costs of actual blackout incidents are reviewed and evaluated. Limitations of this method include the fact that detailed case study information is relatively rare, and even where it does exist, its transferability to other service territories that serve different customers, under different economic conditions may be quite limited. Standby generator proxies in which the cost incurred to purchase standby generators by commercial and industrial customers within the population is assumed to represent the value they place on ensuring reliability. In addition to being heavily dependent on the accuracy of the information available regarding generator costs, operating hours, fuel costs, etc., this approach overlooks the fact that some customers may be willing to pay more for increased reliability, but not as much more as would be required to purchase a stand-by generator, and that some customers may have been willing to pay even more than the cost of a standby generator. GDP or GSP per kwh proxy approaches for the VCR of commercial and industrial customers relate Gross Domestic Product or Gross State Product to electricity consumption and make the assumption that any unserved electricity involves a loss equal to this value. The primary limitation here is that it fails to account for a number of factors that could increase or decrease this value. For example, lost production may be able to be shifted to another location or made up at another time. On the other hand, the production proxy value cannot capture some of the additional costs that outages can impose, such as equipment or product damage, additional costs in re-start procedures, and damage to commercial arrangements if the commercial or industrial enterprise is unable to meet commitments to its customers. The combination of the limitations described above is the reason that most recent attempts to calculate the value of customer reliability have employed survey-based approaches instead Survey based approaches There are two general categories of survey-based approaches: direct cost and indirect cost. The direct cost approach seeks to ascertain from the customer the type and magnitude of costs that would be incurred as a result of outages of different durations and that occur at different times. This type of approach is generally more applicable to larger businesses where a larger proportion of the impacts of outages are experienced in predominantly economic/financial terms, and where there are likely to be staff who can quantify the impact of the outage on the various operating components of the business. 4

10 By contrast, where intangible factors are likely to comprise a significant proportion of the 'costs' imposed by supply interruptions or where it is unlikely that the customer will be able to quantify the impacts of an outage indirect approaches are more appropriate. The prime example is residential customers: while the impact of an outage may have some financial impacts (such as the expense of eating out if the outage prevents food preparation at home, or food spoilage if the outage persists for an extended period), the more significant impacts are likely to be experienced as inconvenience and the disruption of the normal schedule or special plans. Examples include (a) discomfort if heating or air conditioning equipment cannot operate, or if hot water is not available, and (b) the loss of amenity from not being able to use household entertainment equipment or computers for recreation or study. There are a number of different indirect approaches, including the Economic Principle of Substitution, contingent valuation (which includes both willingness to pay and willingness to accept approaches), choice modelling and conjoint analysis. The Economic Principle of Substitution is generally only used with residential customers. All of the other indirect approaches can and have been used with both residential and non-residential customers, with the exception of very large industrial customers for whom the direct cost approach is generally used. The following paragraphs provide more detail on each of the survey-based approaches Direct cost The direct cost method seeks to determine either or both: The economic damage the outage would impose on the customer s business; and/or The cost of the steps the customer would take to mitigate the risk of outage, or its impacts. Under the direct costs methods the survey will generally describe a number of hypothetical outage scenarios that include variations in the timing, duration and frequency of outages. Variations in geographic spread are also sometimes addressed. For each scenario, the respondent is prompted with a series of questions that ask for an estimate of the costs that would be incurred in different operational areas of the business. The categories probed attempt to be comprehensive and would typically include: Lost sales (that could not be made up after the outage); Product or input spoilage; Damage to plant equipment; Increased equipment operations or maintenance costs; Wages paid to staff unable to work; The cost of operating back-up generation equipment; Additional costs in resuming operations. The primary limitations of the direct cost approach are as follows: It is only applicable to respondents that can readily relate their business costs to an interruption of supply, and for whom intangible costs are not significant. In practice, this approach is generally only used with very large consumers of electricity. It can require a significant amount of effort on the part of the respondent, which can lead to poor response rates, or the need to use in-person interviews which are costly. 5

11 To assist in accuracy, it is helpful if the survey is either tailored to the processes of the types of businesses responding or that the interviewers conducting the survey have enough knowledge of the industries to ask follow-up questions that ensure that all relevant costs are considered. These factors again add to cost Economic principle of substitution The Economic Principle of Substitution (EPS) is a survey technique that can be used to evaluate the cost of a good or service that does not have a known value such as the value of electricity supply reliability. It does so by asking customers to choose one or more options with known costs as a substitute for the good or service being investigated. Essentially, survey respondents are provided with a list of possible actions they might take to mitigate the effects of an outage. These actions can be either preparatory (i.e., things the customer would do in anticipation of outages occurring) or reactive (i.e., things the customer would in reaction to a specific outage). Examples of the types of choices that might be provided in the survey include: Purchase candles Go out to eat Purchase a torch and batteries Purchase a portable gas cooker Replace my electric stove with gas Buy a UPS Lease a generator. The cost of each item is also given. The cost of the measure or action that the customer selects essentially provides an estimate of the value of the good that the choice is meant to be a substitute for in this case, the value of reliability of electricity supply. For the technique to work properly, the survey must be constructed with care. The following points are of critical importance: The list of options must provide both low- and higher-cost items that also provide different levels of amenity and reliability as substitutes for mains power supply in order to capture the likely high and low ends of the customer's value for reliability. In addition, however, the list of options must provide a set of costs that are relatively continuous and that do not have significant gaps. If the differences in cost between adjacent items on the list are large, respondents may not have a choice that they are adequately comfortable with, and the value of customer reliability derived from the survey will suffer from over- or under-estimation. The costs of the options should be representative of the costs that respondents would confront in the real world. This can be a problem for certain types of actions. For example, if the option is to go out to dinner, the cost will vary with both the number of people and the type and price level of the restaurant. This requires careful construction of the options in order to ensure that they are expressed in a way their cost is clear to both the respondent and to the analysis. However, this can make certain options very difficult to use in the survey. 6

12 The survey must include a clear statement about the expected frequency, duration and timing of the supply interruption the substitute is meant to address. This is because the respondent's choice of option will be affected by his or her view (or experience) of the likely frequency and duration of outages. As a result, an EPS survey will ask respondents to select options from the same list in response to outage events with different combinations of frequency, duration and time of occurrence in order to see whether and to what degree the respondents' choices change as the context changes. Because the possible combinations can be quite large, EPS surveys are often simplified to address only one or a few of the outage parameters. The advantage of the EPS approach is that it uses real-life options that a customer would be able to choose from. However, as the discussion above indicates, the construction of the questionnaire (and particularly the list of choices) can pose non-trivial problems. Another caution is the fact that the survey presents a context of reliability. While customers have an experience of the frequency and duration of past outages which informs their expectations about future outage frequency and duration, this is not quite as firm as being told that outages can be expected to occur with a given frequency and duration. Finally, the use of the same choices in the option set can pose a risk of respondent confusion or fatigue, thereby limiting the number of scenarios that can be addressed without the risk of diminishing quality of the responses Contingent Valuation (WTP and WTA) Contingent valuation approaches seek to quantify the amount of money customers would be either (a) willing to pay to receive an improved level of reliability (WTP), or (b) willing to accept as compensation for a reduced level of reliability (WTA). The question can be asked in a number of ways: An open-ended format, in which the respondent nominates: the amount they would be willing to pay to avoid being subjected to an outage of a specific duration, or the amount they would be willing to accept in compensation for that outage. A close-ended format, in which the respondent is asked whether they would: pay a specified amount of money to avoid being subjected to an outage of a specific duration, or accept a specified amount in compensation for such an outage. In either case, the results can be used to calculate: A mean WTP or WTA across all respondents; or WTP or WTA values for specific customer segments by the use of regression techniques that link the WTA/WTP responses to demographic (or firmographic) parameters. 7

13 It might be expected that the WTP and WTA approaches would provide similar results that is, that customers' willingness to pay for a reliability improvement would be exactly the same their willingness to be compensated for an equivalent reliability degradation. However, instances where both approaches have been applied to the same service issue indicate that this is not the case. Rather, the results indicate that customers tend to give higher values for how much they would need to be compensated for a service degradation than they say they would be willing to pay for an equal improvement in service. This provides an insight into how the results of each type of approach should be interpreted, and highlights the importance of selecting the more appropriate approach for any particular survey undertaking. For example, a survey seeking to design an interruptibility program to defer network augmentation would probably provide a better estimate of customers participation requirements by using a WTA approach. Valuation of a network augmentation, by contrast, might be more closely approximated by a WTP approach. The primary disadvantage of the open-ended approach is that it is susceptible to cognitive burden; that is, it asks respondents to provide a specific value for their willingness to pay for a good (in this case, electricity reliability) that they have no market experience in purchasing or valuing, which is likely to introduce significant inaccuracy in their responses. Although the close-ended approach avoids the cognitive burden problem of the open-ended approach, it does have a related problem and some other drawbacks: The related problem is that, despite the fact that it provides discrete value choices, it is still asking the respondents to explicitly value a good (reliability of electricity supply) for which they are not accustomed to paying. Only one aspect of service quality (such as frequency of interruption) can be modelled per survey. If trade-offs between possible improvements in different aspects of the good or service need to be considered, the survey will either need to be expanded or a larger sample used (with each respondent only answering a portion of the combined set of questions) 5. If each respondent is asked about his/her willingness to accept a series of values, there is a risk of strategic behaviour in respondents' answers. That is, they may perceive that they are going to be offered a 'better deal' and wait for the choice they think will make them best off (rather than the answer that best represents their willingness to pay or willingness to accept). One way to overcome this is to offer only a limited set of choices to any particular respondent, but this increases the needed sample size Choice modelling Choice modelling 6 like the closed-ended contingent valuation approach asks customers to choose between two or more service/price scenarios. Unlike the contingent valuation approach, however, the choice modelling approach can test customer preferences for services comprised of a number of attributes. By testing customers' preferences across a number of combinations of the values for each attribute of the service in combination with different prices, and processing results through sophisticated econometric techniques, choice modelling can determine the trade-offs customers are willing to make between each of the attributes and price. 5 Note that this limitation is overcome in both the choice modelling and conjoint analysis approaches described below. 6 There are several different types of choice modelling, including discrete choice and stated choice approaches. 8

14 As in the case of contingent valuation, the approach can be used to determine either (or both) willingness to pay or willingness to accept. It can also provide either (or both): A mean value that customers in general would be willing to pay for improved service, or Willingness to pay values for specific customer segments by linking the choice responses to demographic (or firmographic) parameters. A significant advantage of the choice modelling approach is that it allows the testing of a larger and more complex set of service price options than contingent valuation and implicitly accounts for the trade-offs between attributes. Another advantage is that the pattern whereby attribute values are varied is highly complex, and therefore significantly reduces the likelihood of strategic responses. On the other hand, this approach generally requires a significant level of effort in survey design, piloting and re-design prior to full-scale launch of the survey. It also often requires a larger sample size because the number of service attribute combinations is generally too large to be presented to each respondent. In addition to the increased sample size, the analysis of survey responses, while well within the capabilities of most econometricians and statisticians, still requires more time than the other approaches. In addition, the computation of choice modelling results is less intuitively obvious than other approaches, and is therefore sometimes more difficult to explain to interested stakeholders Conjoint analysis Conjoint analysis is similar to choice modelling in that it seeks to determine how customers make trade-offs between the attributes of a good or service and its price. There are three different conjoint approaches: In conjoint rating respondents are asked to provide a score representing their preference for each of a series of service/price bundles, one at a time. One problem with this approach is that while it implicitly reveals the respondent's order of preference, it does not provide any indication of how much each higher ranked option is preferred to the next less preferred option. Similarly, it does not provide any indication of the absolute level of preference of any of the options. Conjoint ranking improves on this approach by asking respondents to provide a relative measure of the respondents' preference for each of the options presented. In paired comparison surveys respondents are asked to rate their level of preference for one alternative service/price scenario over another. As in choice modelling, each alternative can have a number of attributes, but in the paired comparison approach the respondent is asked to indicate by how much he or she prefers one of the alternatives to the other (not just which one they prefer). This provides interval-level data for use in the regression analysis. Like choice modelling, conjoint analysis requires a significant level of effort in survey design, piloting and re-design prior to full-scale launch of the survey. The level of attention conjoint analysis requires from the respondent is also quite similar to that required in choice modelling. Despite its many similarities to choice modelling, it has been used much less often than either choice modelling or contingent valuation approaches in studies seeking to determine customer value of electricity reliability. 9

15 3. Detailed review of selected options The previous section provided a brief overview of the approaches that have been used for assessing customers' value of electricity supply reliability. This section provides more in-depth treatment of three approaches that have been used with some frequency and that have been used in Australia and New Zealand through case studies of: The Victorian VCR, which combines the economic principle of substitution with the direct cost approach, A willingness to pay approach that was used in South Australia and initiated but not completed in New South Wales, and A choice modelling approach (stated preference) that is currently in progress in New Zealand to assess the value of unserved energy Victorian Value of Customer Reliability (VCR) Objectives and how the method was used In 2002 VENCorp, which was then the transmission planning body for Victoria, commissioned Charles River Associates (Asia Pacific) Pty Ltd to conduct a study into the value of customer reliability (VCR) regarding the state's electricity system 7. The aim of that study was to ensure that appropriate values of customer interruption costs at the sector and state levels were available to VENCorp for assessing the market benefits of augmentations to the electricity transmission system. CRA's effort built on work undertaken by Monash University in The state level VCRs calculated in the 1997 Monash and 2002 CRA studies were $28,900 per MWh and $29,600 per MWh respectively. In 2007 VENCorp commissioned CRA to update the sectoral and state level VCRs 9. The updated value determined in the study was $47,850 per MWh. However, the 2007 study was expanded to include a review of the broader social costs that could occur in the event of a widespread power outage. These costs included disruption to community services such as fire, police and/or ambulance services, but were limited to the disruption to normal services and did not include the level of demand for service that might be experienced during the conditions of a widespread or extended outage. Non-tangible and flow-on social disruption costs were also beyond the scope of the 2007 study. Non-tangible costs include impacts on leisure and study time, and interruptions to schools, public administration and public transportation. Flow-on costs include "impacts such as trauma related to injuries/mortalities, fear, panic, and increased incidents of crime" Charles River Associates (Asia Pacific) Pty Ltd, Assessment of the Value of Customer Reliability (VCR), December 2002, for VENCorp. 8 Dr M.E. Khan and Dr M. F. Conlon, op cit. 9 CRA International Pty Ltd, Assessment of the Value of Customer Reliability (VCR), August 2008, for VENCorp. 10 CRA International Pty Ltd, op cit, p

16 The value determined for social disruption cost in the 2007 study at $1,000 per MWh was therefore considered by the study's authors to be very conservative. CRA also recommended that this additional value be used only in situations where a proposed investment had the potential "to contribute to system security across the entire state or significant load centres" Overview of approach Survey approach The question that was effectively asked of survey respondents in each sector was: For residential customers: What cost would your household incur to mitigate the effects of frequent outages of a specific duration? For business (agricultural, commercial, industrial) customers: What cost would your business incur as a result of an outage of a specific duration? This form of question was asked in reference to a range of outage durations from 20 minutes to 24 hours. The surveys conducted with residential customers used the Economic Principal of Substitution, in which the respondent is asked which of a series of actions they would be most likely to take in the event of an outage of a specified duration that was likely to be a frequent occurrence. The cost to the customer of each item on the list was provided, and the same list of items was used to identify how much the customer would spend to mitigate the effects of outages of 1, 4, 8 and 24 hour durations. The residential surveys also collected information on customers' experience with outages (including the season, day of the week, and time of day that customers felt would be the most inconvenient time for an outage to occur), customer demographic and household characteristics (including household income, number of people living in the household, type of house, and gender and education level of the respondent), and whether they also operate a business out of their home. The surveys conducted with customers in each of the non-residential sectors used a direct measurement approach to estimate the damage costs that the customer would experience in regard to outages of different durations. Customers were asked to calculate or estimate the costs they would experience in specific categories that were tailored to each of the sectors, but that included the general areas of operating back-up equipment, damage to equipment, loss of perishable materials, staff costs incurred but which could not be used productively, lost sales, and costs incurred to get the business back into operation (including any overtime labour costs). Respondents were also asked to provide any other costs they would incur that were not included in the specific categories included in the survey. The surveys conducted with non-residential customers also collected information on customers' experience with outages (including when customers felt would be the most inconvenient time for an outage to occur in terms of season, day of the week and time of day), whether they have back-up electricity supply equipment (and if so, its characteristics), information on the type of business the customer conducts and the size of that business (in terms of the number of employees), and whether the business pays for all of its electricity directly or whether some is paid as part of the rental of its facility. 11 CRA International Pty Ltd, op cit, p 9. 11

17 The reported total cost (in each sector for each duration) for all survey respondents was then normalised by the number of kwhs consumed by those who responded within each sector to derive the average cost of an outage per kwh of consumption ($/kwh). Normalised outage costs per duration bucket were then weighted by the probability of an outage of a given duration to produce a cost that is assumed to be representative of the opportunity cost of each kwh of unserved energy (USE) in that sector. The sectoral VCR was then calculated as the sum of those probability-weighted normalised outage costs. Finally, the state-wide VCR was calculated by weighting the sectoral VCRs by the proportion of total state-wide electricity consumption of each sector and summing across the sectors. Figure 1 on the following page provides an overview of the inputs and calculation process of the Victorian VCR. Figure 1: Overview of inputs to and calculation of the Victorian VCR Survey of Victorian customer experience and costs during time of electricity outage (stratified by sector) Average cost / kwh (VCR) of outages of given durations (Victoria by sector) Probability of outages of given durations (by sector and region) (Source: regional DBs) Outage duration weighted sectoral VCRs (by region) Sectoral energy consumption (by region) (Source: regional DBs) Sectoral energy consumption weighted VCRs (by region) State-wide VCR Source: OGW Sectors The sectors addressed in the Victorian VCR studies were: Residential, Agricultural, Small and medium commercial, 12

18 Large commercial 12, Small and medium industrial, and Large industrial. Sample design, size and intended statistical reliability level For the CRA 2007 VCR report for Victoria, survey samples were drawn as follows: As in 2002, the general approach adopted for sampling was random sampling from: a) sources such as the White Pages for residential customers; b) Yellow Pages for small/medium commercial and industrial customers, and agricultural customers; and c) CRA customer lists for larger commercial and industrial customers developed from previous research assignments 13. The 2007 VCR study did not report an intended statistical reliability level, but did report the targeted sample sizes and number of survey responses received, and the error band associated with the VCR that was estimated for each sector, based on the achieved sample size, as shown in Table 2, below. Table 2: Summary of CRA 2007 survey responses and reported accuracy Sector Target completions Usable responses Estimated sectoral VCR ($/kwh) Weighted error bands* Residential /- $0.40 Agriculture /- $0.51 Large commercial 50 4 Small/medium commercial Large industrial Small/medium industrial /- $ /- $2.41 Total 1, /- $8.59 * 95% confident that the true value lies within +/- $x of the estimated value Source: CRA International Pty Ltd, op cit., Tables 1, 6 and 8. Due to the low response rates achieved for the large commercial sector, the results for this category were aggregated with small/medium commercial into a combined commercial category. Similarly, given the low response rates achieved for the large industrial sector, the results for that category were aggregated with small/medium industrial into a combined industrial category. OGW has not undertaken any analysis to confirm the stated error bands for the purposes of this study, but considers CRA's conclusions as to the accuracy of the estimates are valid assuming: 12 Government and institutional facilities were included in small and medium commercial and large commercial sectors, as appropriate, depending on their annual electricity consumption. 13 CRA International Pty Ltd, op cit, p

19 The methodology for assessing the extent of statistical error was appropriate; The samples used were truly randomly selected; and Analysis was free from error of application. However, we note that: The sampling process, at least with respect to both large commercial and large industrial customers, was not truly random, as it was based on CRA contact lists. However, in the circumstances of limited survey resources and difficulty in getting suitably sized organisations to participate in the survey, relying on contact lists to generate responses may have been the only practicable approach to take. The sampling process for residential customers was not completely random, as it excluded households with silent numbers. Other issues concerning the application of the VCR methodology are discussed in Sections and below How results have been used It is understood that the results of the VCR study have been used by AEMO in its application of the regulatory test for transmission infrastructure investment as representing the market benefit of reliability improvements. Conversely, it also represents the maximum value of avoided unserved energy beyond which an investment in transmission augmentation would not be justified Observed advantages and disadvantages for application to transmission reliability investment assessment It is noted that the weighted results for all sectors consider only the relative weights of distribution connected loads. No weighting is given to transmission connected loads, and no attempt was made in the sampling to include transmission-connected customers. Given that transmission connected loads in some regions represent a significant proportion of total regional load, failing to take account of transmission connected loads may create some bias in the calculation of region-wide VCR. The proportion of total regional load in each jurisdiction represented by metered distribution connections is: 14 Victoria 74% Queensland 61% New South Wales 82% South Australia 82% Tasmania 36%. The remainder of system load is a combination of non-metered distribution connected usage, distribution losses, transmission losses and transmission connected load. 14 The proportions presented are for the FY for all regions except Victoria. Victorian data is for FY sourced from the CRA 2007 VCR report for Victoria. (OGW did not collect more recent information from Victorian distribution businesses for the purposes of this report.) The figures represent total reported distribution connected energy consumed within a region as a share of scheduled plus semi-scheduled sent-out generation for that region (as sourced from AEMO s 2010 ESOO). 14

20 If the bulk of the transmission connected load is represented by industrial customers, then the estimated costs that apply to industrial outages of given durations may be similar to those that transmission-connected customers incur but they may not if the types of industries connected to the transmission network differ significantly from those connected at distribution voltage levels. However, even where the distribution of industry types is similar, the patterns and durations of outages at the transmission level will differ from those at the distribution level. Accordingly, at the very least, information on the probability of transmission outages of each duration would need to be applied before a transmission-connected sector VCR could be determined and included in the state-wide value. An additional concern is the significant increase (65.6%) in the value of the VCR from 2002 to To our knowledge there has not been any attempt to verify the magnitude of the change indicated by the survey results. It would be very useful to investigate this further and to ensure that the methodology used to determine the VCR is not itself a source of volatility in the result Refinements to the 2007 methodology OGW s review of the 2007 VCR study identified two areas for improvement. Importance of ascribing a value to all outage durations In order to avoid respondent fatigue and drop-out in the residential surveys, two outage durations were left out 20 minutes and 2 hours. As a result, these durations were effectively ascribed no cost in the calculation of the unweighted sector values of annual USE in the 2007 VCR study. Because the cost of such outages is very unlikely to be zero as evidenced by the fact that outages of both 1 and 4 hours duration were reported by customers as imposing costs the total residential sector VCR and the state-level VCR are both likely to have been under-estimated. Adjustment of the cost of mitigation measures between the 2002 and 2007 surveys In applying the survey methodology from 2002, the cost of the outage mitigation measures (substitutes for grid-based electricity supply) listed as response options in the residential customer surveys were not updated. As a result, the 2007 residential estimates of VCR per MWh are effectively couched in $2002, yet the VCR estimates per MWh for each of the other sectors are couched in $2007 (given that the respondents would have logically used $2007 when providing their estimate of the value losses incurred by their business as a result of an outage). The measures taken in this report to correct these factors in extending the Victorian VCR to the other regions are discussed in Section Willingness to Pay (via discrete choice) South Australia and New South Wales Objectives and how the method was used In 2002, the Essential Services Commission of South Australia and the New South Wales Treasury undertook separate studies of customers' interest in and willingness to pay for improved electricity services. Both of these studies, it should be noted, addressed service quality factors beyond reliability of electricity supply. 15

21 In the case of New South Wales, the study was undertaken to determine whether customers' willingness to pay for particular service improvements would justify additional capital expenditure in the pricing submissions of the state's distribution companies to the regulator for the price determination 15. The South Australia study was also undertaken to provide inputs to a price determination (in this case ESCOSA's determination). The issue here was slightly different, however. ESCOSA had decided that it was going to establish service standards and a performance incentive scheme to reward ETSA, the state's sole electricity distribution company, for improving service in specified areas, and to levy penalties in those cases where service standards were not met. ESCOSA noted that customers' preferences for service and their willingness to pay for specific service improvements are necessary inputs for the proper design of such service standards and associated incentive scheme. The study was undertaken to determine customers' views on these issues. Study results were expected to (and did) provide input into (a) the selection of services to be included in the service standards, which was to reflect those services that are of the greatest importance to customers, and (b) the level of financial incentive incorporated within the performance incentive, which was to reflect customers' willingness to pay for incremental service improvements 16. Both studies were conducted by KPMG, and included a series of focus groups. In both cases the focus groups were used to identify the specific types of services that customers were most interested in, and to capture the language that they used to describe these services and their attributes, in order to assist in the design of the questionnaires. The design in both cases also called for pilot surveys and then a final large-scale survey. Both these steps were undertaken in South Australia, but the New South Wales study was terminated at the conclusion of the focus groups due to the fact that the services identified in the focus groups as being of most interest to customers, and for which the largest proportion of customers exhibited a willingness to pay for improved service, did not align with the areas that had been targeted for capital investment by the DNSPs Information on the New South Wales study is taken from Willingness to Pay Customer Research for Electricity Distribution Networks, prepared by KPMG for the New South Wales Treasury, November 2002, and discussions with the NSW Treasury project manager, and the KPMG project manager. 16 Information on the South Australia study is taken from Consumer Preferences for Electricity Service Standards, prepared by KPMG for ESCOSA, September 2003, and discussions with a member of the ESCOSA project team, and the KPMG project manager. 17 Personal communication from the New South Wales Treasury project manager. It should be noted that the focus groups did not provide statistically robust information on this point, and that the focus groups did not identify the amount that customers were willing to pay for improved service, but only whether they were prepared to pay anything at all for improved service in a variety of areas. 16

22 Overview of approach Survey approach Both studies used the same approach. The focus in both cases was to determine customers' willingness to pay for different service levels. In both cases consideration was given to contingent valuation and discrete choice approaches, and in both cases, the discrete choice approach was selected. The rationale provided by KPMG for that choice was that while the strength of contingent valuation is that it can provide a flexible and accurate estimate of the underlying distribution of the willingness to pay for one, or a small number of, alternatives relative to the status quo, the strength of the discrete choice approach is its ability to estimate trade-offs between different attributes and therefore different goods. In sum, the discrete choice approach was selected because it allows "willingness to pay estimates to be derived for a number of different options in an efficient manner" 18. The focus groups conducted in South Australia identified 19 specific service attributes that customers were interested in and in which they exhibited some willingness to pay for improved service. The price impact of the service improvements was added as the 20th attribute. However, based on the results of the pilot test, two service attributes were deleted from the list. The 17 non-price service attributes tested in the survey addressed the following areas: Unplanned outages (4 attribute values) Planned outages (2 attribute values) Information customers get regarding unplanned outages (3 attribute values) Information customers get regarding planned outages (2 attribute values) Voltage fluctuations (1 attribute value) Undergrounding (1 attribute value) Future power supply improvements (4 attribute values). The total number of possible combinations of service attributes, service attribute levels and bill impacts was over 3.5 x10 13, which was then reduced via a fractional factorial design to 256 for testing in the survey. In the South Australian interviews respondents were presented with 16 pairs of service levels and attributes. Each service was described in terms of the specific values regarding each of its attributes, including price (in terms of the change in the customer's annual bill). One of the service levels in each of the pairs was described as the current level of service and the other as a new level of service, and respondents were asked in each case whether they would prefer and be willing to pay the stated price for the new service 19. In addition to the choice sets, the surveys collected information on: Customers' current perceptions of the service they received in relevant aspects of their electricity supply and customer service, including: how frequently they experienced momentary outages how frequently they experienced outages lasting more than 30 seconds 18 KPMG, South Australia, p Where the new service level was lower than the current service level, the price was expressed as a lower annual bill. 17

23 the total duration of outages they experience on average in a year the longest outage duration they experience in a year the amount they pay annually on their electricity bill Demographics of residential respondents, including: work status number of people living in the home, and number of dependent children location, type and size of the home (number of bedrooms and bathrooms) owner vs renter whether the respondent operates a business from the home Firmographics 20 of non-residential respondent, including: major business activity total number of employees annual turnover proportion of the company's turnover generated in (a) South Australia, and (b) Australia. It should be noted that because the South Australian study sought to assess customers' interest in and willingness to pay for a number of different service improvements, the service pairs included changes in a number of service levels, and not just those related to reliability of supply. This is an essential difference between the South Australian study and those undertaken in Victoria and New Zealand. However, the discrete choice approach used in the South Australia and New South Wales studies is quite similar to the stated preference approach being used in New Zealand, which is described in section 3.3 below, and which is directed solely at reliability of supply, indicating that the approach is applicable to a study that is focussed solely on the reliability of supply. The implementation of the survey itself was quite sophisticated and relatively complex, and involved the following steps: Initial telephone survey to collect attitudinal data, perceptions of electricity service levels delivered, and demographic information Generation of choice sets customised to each respondent by incorporating their current perceptions of electricity service Mail-out of choice sets Telephone interviews to collect responses to the choice sets, information on the equipment used by the respondent and additional attitudinal questions. The telephone interviews were conducted via CATI (computer-assisted telephone interviewing), and respondents were offered an incentive to participate in the survey. 20 'Firmographics', though not a word used in general parlance, is commonly used in market research to refer to those characteristics by which a business entity can be described, such as type of business conducted, number of employees, annual turnover, etc. 18

24 The data on the customers' perception of outages collected in the first telephone interviews was used to establish each customer's perception of the service they receive with regard to each service attribute. This allowed the generation of individualised current service levels to which each of the new service levels could be compared. This individualised baseline and the combination of service bundles that were accepted at the bill level changes associated with them allowed the analysis to calculate the amount that customers in aggregate and by segment were willing to pay for each increment (service level) of improvement in each service attribute. A comparison was undertaken of customers' perceptions of electricity reliability (from the initial telephone survey) and actual service performance (from ETSA HV feeder performance information). Results indicated that customers' perceptions of the mean number of outages experienced in a year were relatively accurate when compared to actual service records. Customers' perceptions of the total minutes off supply, by contrast, did not align well with ETSA's records, though this is not surprising. The study also compared customers' perceptions of their annual electricity bill to the actual billed amounts. Results indicated a better fit among residential customers than business customers, with the residential mean perception of the bill differing from the mean actual bill by only 1.6%. Sectors, sample design and size, and intended statistical reliability level The South Australian sample was made up of four customer sectors and six segments based on geography and current reliability levels. The proposed and achieved sample sizes for the South Australian study are shown in Table 3 below. Table 3: Sample size proposed and achieved in the South Australia WTP study Customer segment Residential Business (<160 MWh pa) Business (>160 MWh pa) Farmers Total Adelaide CBD 100 / / 94 Adelaide metropolitan "good" power 200 / / / / 404 Adelaide metropolitan "poor" power 200 / / / 209 Regional "good" power 200 / / / 283 Regional "poor" power 200 / / / 211 Rural 200 / / / / 287 Total 1,000 / / / / 86 1,600 1,488 Source: KPMG, South Australia, op cit, p In some cases, the number of responses received is higher than the proposed sample size. Although counter-intuitive, this results due to the fact that surveys will be sent to a larger number of respondents than the amount needed for the sample quota in order to allow for some non-response. If less than the anticipated level of non-response results, more surveys will be returned than initially planned for. Once available, the larger number of survey responses is generally used (along with adjustment to the weighting factor to be used) as it provides a larger sample from which to base population estimates. 19

25 As can be seen, most achieved sample sizes were relatively close to the levels that had been targeted, the notable exceptions being some of the small business segments. The residential and non-residential samples were selected through random sampling procedures 22, with quota objectives. The quota objective for the residential sample was to be within 5% of ABS census data regarding: Males and females (a 50/50 split was sought) Labour force characteristics Household income bands Education levels. The quota objectives for the non-residential sample were to match ABS statistics with regard to: Industry group (agriculture, including forestry and fishing; commercial; retail; and industrial) Number of employees (in bands). Although the achieved samples in both the residential and non-residential sectors matched the quotas relatively well, survey responses were weighted by the following factors to provide the best representation of the state's population of electricity customers for the purpose of the study: Demographics/firmographics and electricity supply performance (based on feeder data where available and customer perceptions otherwise) within the residential and nonresidential samples Customer numbers and annual consumption to combine the residential and small business samples into a single model (large business customers' willingness to pay was analysed separately). The KPMG study did not report an intended statistical reliability level for the study, but did state that while 100 responses per cell in the survey segmentation scheme "was considered to be sufficient to estimate reliable models for each segment for which results were reported. It should be noted that the statistical reliability of the model estimates improves significantly if the sample size in each segment is increased to 200 or larger" 23. The study did, however, report the confidence interval of all willingness to pay results and also specifically indicated any results that were not statistically significant at the 95% confidence interval How results have been used The results of the survey were used to inform the re-design of ESCOSA's service standards and performance incentive scheme. They were not used (and never were intended to be used) to inform a formal investment test for transmission system augmentation or reinforcement. However, the results would have functioned at the margin to inform ETSA as to the financial viability of discretionary service reliability investments (i.e., investments undertaken to achieve reliability levels above the service standard). That is, such an investment (if purely discretionary) would only be economic for ETSA if the performance incentive that would be earned exceeded the investment costs. 22 The sample frames from which the random samples were drawn were not identified in the report, however. 23 KPMG, South Australia, op cit, p

26 In this regard it is interesting to note what the survey results indicated regarding customers' willingness to pay for increased electricity supply reliability. Willingness to pay for a reduction of 1 outage per year was estimated to be less than 0.5% of the annual bill for all customer segments, and more typically around 0.25% of the annual bill. Results varied more by location than by customer type, with CBD respondents having the highest willingness to pay for a reduction in the frequency of outages (despite having the second lowest actual frequency) and rural customers having the lowest willingness to pay for outage frequency reductions (despite having the highest actual outage frequency). Willingness to pay for a reduction in the total number of minutes off supply of 1 hour per year was marginally lower than that for outage frequency reduction. None of the customer segments were willing to pay more than 0.33% of their annual bill for such a reduction. Those segments with the highest willingness to pay for such a reduction were large businesses and customers located in metropolitan areas. Those with the lowest willingness to pay for a 1 hour reduction in minutes off supply were rural and CBD customers. Willingness to pay for the longest interruption of the year to be reduced by 1 hour was similarly marginal. Small business and rural customers exhibited the highest willingness to pay for this service improvement at 0.4% of their annual bill each. After consideration of the results, ESCOSA set the performance target at improving the reliability of the bottom 15% of feeders with the worst reliability performance. It should be noted that this does not necessarily ensure that the resulting increase in system reliability will be delivered either (a) to those customers who have the highest value for improved reliability, or (b) at least cost. Although the ESCOSA study did not calculate a value of customer reliability, the willingness to pay figures it provides would appear to suggest that customers are prepared to pay relatively little (compared to Victorian estimates of VCR) for improved reliability: willingness to pay never even reaches 1% of an annual bill for any customer segment. However, two factors must be considered: The dimensions of the two metrics are very different. While willingness to pay is expressed in percentage of the annual bill, the VCR is expressed in $/kwh. In the results discussed above it was found that residential customers were willing to pay 0.2% of their annual bill to reduce minutes off supply by 1 hour, and about 0.28% of their annual bill to experience one less outage per year. Given information on the approximate average bill of residential customers in South Australia ($1,200), their approximate average annual consumption (6,300 kwh), and the probability-weighted average duration of outages (just over 2 hours), these willingness to pay figures translate to VCR values ranging from $1,500 to about $3,500 per kwh. These values are smaller than in the 2007 Victoria VCR study, but are not absolutely small. The Victorian approach explicitly asked survey respondents to consider the impact of outages that occur at the most inconvenient time for them. By contrast, in the case of the SA study, it does not appear that respondents were asked to frame their answers with regard to the most inconvenient outages, but rather were asked what they would pay to reduce total outage time, or to reduce by one the number of outages they would experience in a year. These are different questions and could be expected to engender different responses. 21

27 Observed advantages and disadvantages for application to transmission reliability investment assessment ESCOSA personnel involved with the study reported that they felt that the methodology was very robust and suited their purpose very well 24. They said that no significant problems were experienced in the implementation of the study but did note that the research required long and very detailed, careful planning, and that it was quite costly 25. They did not see any reason why a similar approach would not be applicable to transmission system reliability, and noted that the survey design would be simpler if system reliability were to be the only issue to be addressed Stated choice (New Zealand) Objectives and how the method was used The primary application of the value of unserved energy (USE) 26 in New Zealand is within the Grid Investment Test (GIT) in which it is used to monetise the reliability benefits of investments in transmission infrastructure 27. The value of USE being employed in New Zealand is currently (pending completion of the investigation described below) NZ$20,000/MWh, with sensitivity analyses also required by the GIT at NZ$10,000 and NZ$30,000/MWh. In September 2008 the Electricity Authority of New Zealand 28 embarked upon an investigation into the value of unserved energy (USE) to "evaluate the fitness-for-purpose of the current value(s) of unserved energy under the [NZ Electricity Governance] Rules and to provide advice on any recommended revisions to these value(s) or its application under the Rules" 29. The investigation is being undertaken in three stages: Stage 1 30, which commenced in November 2008 and was completed in December 2008, concluded that the values of USE then being used in New Zealand should be updated, and recommended the survey approach, sample design and survey method to be used (as is discussed in more detail below); 24 Personal communication with ESCOSA staff. 25 This illustrates the fact that a more costly approach may still represent better value than a less costly one particularly where (a) the more costly approach can be shown to provide a material increase in the accuracy or comprehensiveness of the results, and (b) 'cost' of an inaccurate result is high. The use to which the VCR is put certainly meets the latter criterion: an inaccurate VCR will either lead to over-investment (which will drive up costs to consumers) or underinvestment (which will leave consumers and the economy exposed to risks whose mitigation would have cost less than their consequences). 26 The term 'value of USE' is used in this section because that is the term being used in New Zealand. Although the studies use different terms and different survey approaches, both the New Zealand value of USE study and the Victorian VCR study are trying to quantify the same thing the value of supply reliability from the customer's perspective. 27 This is essentially the same purpose for which the VCR is used in Victoria. 28 Successor organisation (as of 1 November 2010) to the Electricity Commission. 29 New Zealand Electricity Commission (now Electricity Authority), Request for Information: Investigation of the Value of Unserved Energy, 3 September 2008, p.13, available at 30 Concept Economics and the New Zealand Centre for Advanced Engineering (CAENZ), Investigation of the Value of Unserved Energy: Stage 1, for the (then) New Zealand Electricity Commission, December Available at of use. 22

28 Stage 2, which is still in progress 31 includes detailed development, testing, and implementation of the surveys, analysis of the survey responses, and preparation of a draft report; and Stage 3, which will include additional analysis of the data as needed, response to submissions received by the Electricity Authority to the draft report and preparation of a final report. The draft report of Stage 2 is expected to be available in the first quarter of Overview of approach Survey approach Stated choice surveys are being used with all customers except large industrial customers (those with annual electricity consumption of 1 GWh or more), who are being surveyed using a direct measurement approach. The large industrial customers will be surveyed in person; all other customers will be surveyed by mail. In the stated choice approach consumers are asked to state their preference between two outage/cost options. The options are characterised by a number of specific dimensions that can have several values each. The specific choices made by survey respondents in aggregate between the options presented allow determination, via econometric techniques, of the tradeoffs that customers are willing to make between each of the dimensions. In the sample questionnaires included in the Stage 1 report, the dimensions used were: Outage frequency and duration Time of outage occurrence (season, day of week and time) The change that would result in the customer's bill. An example is shown below. Situation 1 Tick your preference Either Option A or Option B Option A Option B Your annual discount on electricity bill: $96 $100 Your experience per year: 3 power failures of 10 minutes 1 power failure of 1 hour Season and time of day: Summer Starting at 5pm Winter Starting at 10am PLEASE TICK Source: Peter Smith, Value of unserved energy project, Presentation to the Investment Advisory Group, 6 May 2010; slide 4. Available at of use. 31 Stage 2 was originally scheduled to have been completed in September 2010, but has required additional time due to the need for additional work to be undertaken in survey questionnaire design, sample frame development and data analysis. 23

29 The number of choices that are needed for the econometric analysis to be undertaken properly is a function of the number of levels that need to be tested with regard to each attribute. The Stage 1 report recommended the following: 2 times of year (winter and summer) 2 times of day (8AM and 6PM) 4 frequency levels (0, 1, 2, 3 and 4 outages per year) 4 outage durations (10 minutes, 1 hour, 4 hours and 8 hours) 8 indicative price outcomes. The specific values to be tested were derived from the characteristics of the New Zealand electricity system with regard to times of system peak, typical outage durations and representative levels of outage frequency. Price outcomes were developed through a structured approach based on price/consumption data for different customer groups that reflected the prices those customer groups actually pay. Judgement was then applied to derive meaningful levels for the amounts that customers might be willing to pay for increased reliability. The Stage 1 report authors suggested that a lower bound for this willingness to pay might be twice the current unit cost of electricity for the relevant customer group, and that consideration should be given to testing several other levels as well. They also recommended that price outcomes for residential customers be presented in terms of changes in the bill (monthly or annual), whereas for non-residential customers price outcomes could be presented in unit-cost terms 32. The authors of the Stage 1 report noted that the number of dimensions and levels discussed above result in 147,456 choice combinations, which they felt could be reduced to somewhere between 36 and 64 pairs for testing in the questionnaires. The reduction process needs to be undertaken in light of the statistical techniques to be used in the econometric analysis, and should include consideration of the following: Elimination of options that are essentially redundant Presentation of choice sets in which only one dimensions changes Elimination of choice sets in which one of the options is likely to be clearly preferable Ensuring that the key dimensions of the outage frequency, duration and time of occurrence are adequately represented in the final choice sets. The authors also recommended that no single survey respondent be presented with more than 8 choice sets 33. This in turn may require increasing the overall sample size to ensure that the target sample size for the full set of choice pairs is attained. Questions on customer demographics (or firmographics in the case of non-residential customers), recent experience with outages and the degree and type on inconvenience caused have also been included in the questionnaire to provide more granular resolution of the value customers place on reliability. The inclusion of these questions allows the value trade-off between outage dimensions and price (impact on bill) to be calculated for different groups of customers as defined by their demographic, firmographic and outage experience characteristics. 32 Concept Economics, op cit, p 54. These parameters may have been changed in the final design of the Stage 2 questionnaires. 33 Concept Economics, op cit, pp

30 The direct cost approach being used with large industrial customers. These surveys asked customers to quantify the costs of outages with regard to the following categories: Costs of operating backup equipment Damage to plant or equipment Pay for staff unable to work Overtime labour costs to make up lost production Loss of sales or custom during a failure Sales foregone from production that cannot be made up Costs to bring business back to normal Costs to recover data lost from computer systems Costs to repair possible damage to the environment Agricultural respondents were also asked to consider costs relating to: Loss of livestock Spoilage of perishable products Loss of dairy, egg, fruit or vegetables produce Loss of take or pay irrigation water The sample design calls for a total of 39 responses to be obtained from the large customer sector. As of May 2010, 33 such interviews had already been completed and no difficulties were anticipated in completing the full set of 39. Initial results of the 33 interviews indicated that: There is a wide divergence in energy risk management practices within the surveyed companies Many had not previously calculated the cost of loss of supply There appears to be a wide range in the cost of unserved energy to this sector (initial range to be confirmed is NZ$1000 to NZ$2m /MWh for a 10 minute outage) Longer outages tend to have lower $/MWh costs as the cost of restoration of production is spread across a larger number of MWh Whilst direct costs could be readily identified, indirect costs proved to be difficult to quantify (e.g. lost orders and damage to reputation) Momentary fluctuations were outside the scope of the interviews but many respondents identified these as being at least as damaging and as occurring more frequently than longer duration losses of supply (e.g. longer than 5 minutes) Significant differences exist between companies that use perishable and non perishable input materials (e.g. food harvest perishes within days if not used but trees can remain in the ground and harvested later) 25

31 Service industries can have relatively low direct costs of lost load but high external impacts arising from the loss of service (e.g., oil refinery loss could lead to wider major economic consequences) 34. Sectors The Stage 1 report recommended the use of the following sectors: Residential Agricultural Commercial Large industrial Other industrial. However, in Stage 2 a decision was made to sample non-residential customers based on their annual electricity consumption. The three consumption sectors were defined as follows: Small non-residential: less than 200,000 kwh per annum Medium non-residential: 200,000 to 999,999 kwh per annum Large non-residential: 1 GWh or more per annum 35. No reason is available as yet for this change. It may have been made because identifying customer type was difficult based on information available to the study, or because it was felt that the sectors themselves were too broad and that sampling on customer size provided a better way to ensure that a representative sample was achieved 36. Intended statistical reliability level A target level of statistical reliability was not put forward in the Stage 1 report, but the report's authors stressed that the accuracy of the estimate of the value of USE to be derived is a critical consideration in the design of the research as over-estimation will result in investment that exceeds the value derived by the community from the transmission investment, and under estimation will result in outages that cause more inconvenience and damage to the community than it would have cost to avoid through infrastructure investment. 34 Peter Smith, Value of unserved energy project, Presentation to the Investment Advisory Group, 6 May 2010, slides Available at of use. 35 Peter Smith, op cit; slide The Stage 1 report made the point that the cost of outages for different customers will vary depending on what the individual customer uses electricity for and how critical the use of electricity is to the customer's health, safety, lifestyle or business. It was for this reason that customer type was seen as a likely segmentation approach. However, it is also the case that the larger the number of segments for which statistically reliable results are desired, the larger the number of completed surveys that will be needed. Research design therefore has to balance the desire for segmented information with cost and time limitations. It may have been felt that the sectors that had been selected originally did not describe sufficiently homogenous groups of customers with regard to their value of USE. Presumably, the reason for this change will be discussed in the draft report. 26

32 The authors of the Stage 1 report also stated that one of the significant advantages of the multinomial logistic regression method they recommended for analysing the survey data is that it will provide "standard errors and confidence intervals for all of the parameters [tested in the research with regard to their impact on the value of USE] to indicate how precise and how reliable the estimates of USE are" 37. Sample design and size The authors of the Stage 1 report recommended that a probability proportional to size (pps) sampling approach be used, where size is defined in terms of electricity consumption. Such an approach gives larger customers more likelihood of being selected in the sample (and will therefore result in VCR values that are more reflective of larger customers within any given sector). This was felt to be appropriate as "any costs of additional investment will (at least broadly speaking) be paid for proportionally by customers" 38 based on their electricity consumption. The sample design recommended in the Stage 1 report and its derivation is shown in Table 4 below. Table 4: Original recommended sample size for NZ value of USE study Customer type Number of Consumption Average Percentage Target customers in (MWh) electricity of total sample population consumption electricity per customer consumption (MWh/yr) Residential Users 1,602,943 12, % 4,066 Commercial Users 155,207 8,382, % 2,787 Industrial Users 113,134 16,779, % Of which Agricultural Users 76,150 1,348, % 579 Of which Large industrial users (MEUG members) 21 10,763, , % 21 Of which 'Other' industrial Users 36,963 4,668, % 1,421 All New Zealand customers 1,871,308 37,393, % 8,874 Source: Concept Economics and the New Zealand Centre for Advanced Engineering (CAENZ) 37 Concept Economics, op cit, p. 60. The precision and reliability of the estimates will also depend on the randomness and size of the sample surveyed. 38 Concept Economics, op cit, p

33 However, as discussed above, the sample design for the study has since been changed. While detail of the revised sampling plan is not yet available, the Electricity Authority has reported that it is now planned to send the mail survey to 14,000 customers (residential and non-residential customers other than 'Large Non-Residential'). If the 20% response rate experienced in the pilot test of the survey also characterises the full survey, the expected final sample size will be 2,800, significantly below that recommended in the Stage 1 report How results have been used The study has not yet been completed but is intended to be used, as mentioned above, to monetise the reliability benefits of transmission infrastructure investment. Other applications were also briefly considered in the Stage 1 report, including: A measure of the impact on reliability of any proposal to remove an interconnection asset from service An input to negotiations between the TNSP and a transmission customer regarding the replacement or enhancement of connection assets, or a grid injection or grid exit point A generation adequacy criterion A value of lost load for use in the wholesale market. The authors of the Stage 1 report felt that the value of USE being developed in the investigation would suit the first two of the applications listed above, but not the latter two, which they felt required different approaches and considerations for their development Observed advantages and disadvantages for application to transmission reliability investment assessment The New Zealand work is still in progress, so it is not possible at this time to be definitive regarding the demonstrated advantages and disadvantages of this approach. The fact that it is seeking to develop a metric for the same purpose as AEMO makes the progress and outcome of this approach particularly interesting and potentially highly relevant, so it is recommended that AEMO seek to be included in the distribution of the results of the study and follow up with the Electricity Authority and its consultants to get further information on the results and lessons learned from the conduct of the study. While it is too early to draw conclusions about the value and practicality of the approach used in New Zealand, it is probably fair to say, given the delays to the initial schedule, that the method has required a material level of effort in survey questionnaire design and pre-testing, and data analysis. 39 Peter Smith, op cit, slide 5. 28

34 4. Extension of Victorian VCR to other NEM regions (demonstration) In undertaking an extension of the Victorian VCR to other NEM regions, OGW has endeavoured to replicate (as closely as is practicable) the methodology applied in the CRA 2007 VCR report for Victoria. The spreadsheet that was used to do this - by applying the sectoral VCR values that were developed from surveys with customers in Victoria to the data provided by the regional DNSPs - has been published simultaneously with this report and should be referred to in conjunction with this section Overview of approach taken Because separate surveys have not been undertaken for each region 40, OGW has had to assume that the estimated unserved energy costs for each sector and duration reported in Victoria are representative of similar costs in other regions. Such an assumption is unlikely to be valid (as is discussed in the following sub-section), but is sufficient for the demonstration of the extension of the Victorian method to the other regions that was to be provided in this study. The region-specific information that has been used in translating the Victorian results to other regions is as follows: Sectoral energy use weightings Probabilities of each interruption duration length Survey information No new surveys of customer outage experiences and costs were undertaken, so OGW has relied on the survey results from the CRA 2007 VCR report for Victoria. That is, the costs ascribed to each interruption duration in each sector for Victoria in 2007 were assumed to apply to the corresponding duration and sector for Queensland, New South Wales, South Australia and Tasmania. This is a necessary compromise and is considered appropriate given that the purpose of the present exercise is simply to demonstrate the application of the Victorian method to the other regions. However, surveys with customers would need to be undertaken if this approach is adopted, as there are a number of reasons why the sectoral VCR values could vary across regions. With regard to residential customers, these could include differences in income distribution and income levels and differences in the cost of electricity as compared to household income, as well as differences in the age distribution, household characteristics, climate and types of enduse equipment, urban/rural population proportions, and whether alternative energy sources are used for particular end uses. With regard to non-residential customers, such factors could include differences across regions in the proportions of different types of commercial and industrial customers, the availability and relative use of energy sources other than electricity, and climate. None of these factors were able to be adjusted for in the present study, but would be accounted for in the event that surveys were undertaken in each region. 40 The terms of reference for this work precluded new surveys. 29

35 Outage probabilities In assigning probabilities to outage durations, the CRA 2007 VCR report for Victoria used outage probabilities for each interruption duration based on state-wide data for the period 2004 to Given the available data, the closest that could be achieved for other regions (Queensland, New South Wales, South Australia and Tasmania) was to base outage probabilities on the average number of interruptions recorded from to The probabilities adopted are as outlined in Table 5. Table 5: Region-by-region and outage probabilities Duration Victoria Queensland New South Wales South Australia Tasmania 0.33 hours 4.7% 3.5% 3.7% 6.5% 5.7% 1 hours 44.0% 13.2% 17.0% 13.6% 24.7% 2 hours 28.6% 30.0% 38.2% 29.4% 35.8% 4 hours 16.0% 25.7% 23.7% 32.0% 22.0% 8 hours 4.8% 12.7% 11.6% 13.6% 8.5% 24 hours 1.9% 14.9% 5.9% 5.0% 3.3% Source: Information on the distribution of outages by outage duration was provided by the DNSPs in Queensland, New South Wales, South Australia and Tasmania Customer data sectoral weightings In assigning sectoral weightings, the CRA 2007 VCR report for Victoria used state-wide energy consumption in 2007 in each of the residential, agricultural, commercial and industrial sectors. Given the available data the closest that could be achieved for other regions was as follows: Queensland: sectoral weights for residential, commercial + agricultural and industrial sectors were drawn from ; 41 New South Wales sectoral weights for residential, commercial + agricultural and industrial sectors were drawn from ; 42 South Australia: sectoral weights for residential, agricultural, commercial and industrial sectors were drawn from ; and Tasmania: sectoral weights for residential, commercial + agricultural and industrial sectors were drawn from Energex provided data matching the preferred breakdown of energy consumption by customer category. However: Ergon Energy was unable to supply consumption figures for a separate agricultural sector, but were able to supply data for a combined agriculture + commercial sector. 42 Integral Energy and ActewAGL provided data matching the preferred breakdown of energy consumption by customer category. However: Country Energy was only able to supply a residential breakdown and non-residential breakdown by customer size (less than 160 MWh, MWh, greater than 750 MWh) and only for the FY; EnergyAustralia was only able to supply a breakdown by residential, large industrial and other. 30

36 In the case of both Queensland and Tasmania, the absence of specific agricultural sector electricity usage data required the aggregation of the agricultural and commercial sector results from the CRA 2007 VCR report for Victoria into a combined commercial + agricultural category 44. In the case of New South Wales, sufficiently comprehensive information on energy usage was not available at the sectoral level, except for residential consumption. The mapping of New South Wales consumption data to VCR survey sector information is shown in Table 6. Table 6: Mapping of New South Wales consumption data to VCR survey sector information New South Wales consumption data Under 160 MWh residential Under 160 Business Over 160 MWh Over 750 MWh Victorian VCR sectoral information Residential Agriculture Commercial Industrial Source: OGW based on information provided by the NSW DNSPs. The sectoral weightings used in the extension of the Victorian method to the other regions based on the sectoral energy consumption information provided by the regional DNSPs are shown in Table 7 below. Table 7: Region-by-region and sector-by-sector energy weightings Sector Victoria Queensland New South Wales South Australia Tasmania Residential 34% 40% 35% 38% 41% Agricultural 1% 2% 39% 23% Commercial 34% 28% 34% Industrial 31% 21% 42% 32% 25% Source: OGW calculation based on information provided by the DNSPs within each region. 43 Aurora Energy was unable to supply consumption figures for a separate agricultural sector. Agricultural sector usage is assumed to be included in figures for the commercial sector. 44 Given the nature of the methodology, the results of CRA s surveys for each of these sectors can be added together at the level of: total interruption costs; number of respondents with non-zero interruption costs; and total annual consumption (kwh) of those who responded. 31

37 Corrections for missing data As noted in Section 3.1.5, the CRA 2007 VCR report for Victoria effectively ascribed a zero cost to residential outages of 20 minutes duration and of 2 hours duration. In applying the methodology in the CRA 2007 VCR report for Victoria we developed annual values of unserved energy for customer outages for each of the 20 minutes and 2 hours durations 45. With a full set of valuations for each of the interruption durations for the residential sector, we took the opportunity to re-estimate the residential and state-wide VCRs for Victoria Results Sector-by-sector The region-by-region and sector-by-sector VCRs that result by applying the methodology described above are outlined in Table 8. Table 8: Region-by-region and sector-by-sector VCR ($/MWh) Sector Victoria Queensland New South Wales South Australia Tasmania Residential 20,395 (13,250)* 15,318 17,190 16,469 18,532 Agricultural 111, ,493 62,887 68,396 Commercial 90,763 18,649 76,716 Industrial 36,074 31,427 32,055 32,905 34,157 State-wide 50,258 (47,850)* 37,198 35,085 38,037 42,022 * CRA estimate prior to correction by OGW. Further detail is provided in the spreadsheet circulated with this report Year-by-year escalation of VCR In creating the year-by-year results, OGW has used two different methods for escalating VCR: The method proposed by NERA and applied by AEMO to the VCR for Victoria An alternative method developed by OGW to correct for the absence of a normalisation factor 46. Results are shown in Table 9 and Table 10 respectively below A linear interpolation of the corresponding 1- and 4-hour interruptions was used to determine a value for a 2-hour interruption. The implied 20 minute increment in value was then extrapolated back in time from the 1-hour value to determine a value for a 20 minute interruption. It should be recognised, therefore, that these are estimates and, while plausible, are not a product of direct customer responses. 46 The escalation factors applied to VCR in OGW's view should have been, but were not, normalised for growth in energy consumption (or population) in each sector. Refer Section

38 Table 9: Region-by-region and sector-by-sector VCR NERA methodology ($/MWh) Sector Victoria Queensland New South Wales South Australia Tasmania ,258 (47,850)* 37,198 35,085 38,037 42, ,813 (50,766)* 41,078 37,930 40,496 46, ,111 (54,930)* 44,064 40,997 44,078 49, ,400 (58,935)* 47,658 43,095 45,767 52,196 * AEMO estimate made using indexing information available for 2010, prior to correction by OGW. Source: OGW calculation using NERA methodology Table 10: Region-by-region and sector-by-sector VCR OGW methodology ($/MWh) Sector Victoria Queensland New South Wales South Australia Tasmania ,258 37,198 35,085 38,037 42, ,936 40,126 37,531 40,062 45, ,181 41,997 40,068 43,155 48, ,292 44,308 41,534 44,303 50,967 Source: OGW calculation using a modified escalation methodology 47 All indexation material is drawn from ABS Cat No, , Australian National Accounts, State Accounts, (reissue). Results here differ marginally from those published by AEMO in 2010 Victorian Annual Planning Report, Section 8.3, pp (available at though the reasons for the differences in the non-residential sectors has not been able to be determined. 33

39 The differences in the outcomes for the VCRs of the different regions in 2007 in each of the two tables directly above is the result of the differences in outage duration probabilities and sectoral consumption shown in Table 5 and Table 7 respectively 48. Differences between states in the subsequent years continue to reflect those differences but are also affected by differences between the states in changes in household income, gross agricultural product and gross state product, as discussed in further detail in Section below Caveats on available data Escalation methodology The VCR escalation factors recommended by NERA were applied by AEMO as follows: RI t = ( TGI t / TGI 2007 ) x 100 AI t = ( GVA t / GVA 2007 ) x 100 CII t = ( GSP t / GSP 2007 ) x 100 Where: RI is the residential index; AI is the agricultural index; CII is the commercial and industrial index; TGI is total gross household income; GVA is gross value added of agricultural production at producer prices; and GSP is gross state product. To calculate the headline VCR adjustment index, the average of each of the sector specific adjustment indices is calculated, with each index weighted by the sector s share of total state-wide electricity consumption 49. A problem with this approach is that the recommended escalation indices, as measures of gross income, rise in accordance with prices, incomes and population. Although the aggregate value of the output produced in each sector is a relevant consideration in the valuation of customer reliability, so too is the rate at which energy use in each sector is changing: The long term trend in energy use per household has been rising (though it has recently started decreasing in some jurisdictions) Electrical energy use in the broad economy has been growing more slowly than gross domestic (or state) product. Given that the index components proposed by NERA are measures of gross income and are applied in a manner that ignores the effect of changes in population numbers and energy use, it is relevant to note that in the four years from FY to FY: The combined nominal GSP of the states comprising the NEM regions has increased by 22.7% Differences in these factors explain the difference in the outcomes of this extension of the Victorian VCR to the other regions. In actual fact, there may also be differences in the sectoral VCRs between states, as discussed above. Any such differences (which will only be known if surveys are undertaken in the other regions) would exacerbate the differences between the state VCRs shown in Table 9 and Table VENCorp, The value of customer reliability for the electricity transmission network: Methodology for extrapolating VCR between surveys 2008/ /12, April

40 The combined nominal GSP per capita of states comprising the NEM regions has increased by 17.4% 51 Aggregate energy use in the NEM has increased by 2.9% 52. Failing to correctly recognise the interplay between gross incomes, change in population and change in energy use will have a non-trivial effect on the calculation of VCR per unit of consumption.. Consider the case of an industrial business that installs a gas-fuelled process to replace an electricity-powered process because, in this particular application, the gas-fuelled process is more efficient. Electricity consumption (kwh) falls and gas consumption rises. No other aspect of the business changes production volumes and values remain unchanged, and if there were to be an electricity outage, the entire production process would cease. In the circumstances of an electricity supply failure to this business, the economic damage done per kwh of unserved energy will be higher than would have been the case before the shift to gas. This will have an upward pressure on the VCR per unit of electricity within the sector in direct proportion to the number of businesses making a similar end-use shift. If escalation of per-unit (kwh or MWh) VCR is to be undertaken correctly, measures of gross income should be normalised for changes in aggregate energy consumption in order to eliminate population effects as follows: RI t = [ ( TGI t / RSE t ) / ( TGI 2007 / RSE 2007 ) ] x 100 AI t = [ ( GVA t / ASE t ) / ( GVA 2007 / ASE 2007 ) ] x 100 CII t = [ ( GSP t / CISE t ) / (GSP 2007 / CISE 2007 ) ] x 100 Where: RSE is gross electrical energy consumed by the residential sector; ASE is gross electrical energy consumed by the agricultural sector; CISE is gross electrical energy consumed by the commercial and industrial sector. Given that a contemporaneous population effect appears in both the numerator and the denominator of each of the above terms e.g. ( TGI t / RSE t ) 53 the population effect will be eliminated. The preferred form of indexation would produce an index that is 2.9% lower for the NEM in aggregate than the index proposed by NERA Source: Australian Bureau of Statistics, Cat No Source: Australian Bureau of Statistics, Cat No Source: AEMO, ESOO TGI, as a measure of gross income, includes a population effect, and RSE as a measure of aggregate energy consumption also includes a population effect. 54 The extent of growth in aggregate energy use in the NEM. 35

41 Although normalisation by changes in energy consumption is the preferred approach to calculating escalation factors, it requires (in the present study) four consecutive years of relevant data for each region in order to apply the normalisation process correctly. Given the information available at the time of writing, the preferred form of normalisation could not be undertaken because sectoral energy usage factors were not available for: Victoria, for 2008 and onwards; New South Wales, prior to ; Queensland, prior to ; South Australia, prior to ; and Tasmania, prior to An alternative form of escalation has, instead, been applied as follows: RI* t = ( TGIpc t / TGIpc 2007 ) x 100 AI t = ( GVA t / GVA 2007 ) x 100 CII* t = ( GSPpc t / GSPpc 2007 ) x 100 Where: TGIpc is total gross household income per capita; GVA is gross value added of agricultural production at producer prices; and GSPpc is gross state product per capita. In broad terms, using per capita measures of income growth instead of aggregate measures of income growth (as proposed by NERA) will produce an index that, on average for the NEM, is 5.3 per cent lower over the period to The preferred form of indexation would produce an index that, on average for the NEM, is 2.9% lower than the index proposed by NERA 56. There are two key deficiencies in this alternative measure, however: It does not correct for changes in energy use per household or energy use per business enterprise The agricultural index has not been (and cannot readily be) corrected for per capita changes there is no per enterprise equivalent of gross value added of agricultural production at producer prices. However, given that the agricultural sector only comprises 1% to 2% of the sectoral weighting, the absence of a per-enterprise correction will not, of itself, create any significant bias in the overall VCR measure. Despite the deficiencies in this alternative measure, we believe it is nevertheless an improvement on the escalation approach that has been applied to date. 55 Nominal GSP growth for NEM regions of 22.7% less nominal per capita GSP growth for NEM regions of 17.3% 56 The extent of growth aggregate energy use in the NEM. 36

42 Potential for improved customer and outage data It should be noted that the information on regional customer sector consumption and outage duration frequencies was requested and graciously delivered by the DNSPs in Queensland, New South Wales, the ACT, South Australia and Tasmania in a very short period of time. Without that cooperation at short notice this demonstration of how the Victorian VCR could be applied to the other NEM regions could not have been completed. However, in interacting with the DNSPs as they collected and provided this data it became clear that much more granular information on the outage experience of customers could be provided if the DNSPs were made aware of the need in advance. For example, the DNSPs mentioned that they could also provide the total minutes off supply characterising each of the outage durations. This would allow sectoral VCRs to be weighted by the likely aggregate duration of outages of different durations (and kwh unserved in different outage durations) as well as the probability of those outage durations occurring. It is recommended therefore that regardless of the approach ultimately selected for quantifying the VCR that AEMO consult with the DNSPs to ensure that the approach has access to the most useful outage and consumption information that the DNSPs can provide. 37

43 5. Observations, outstanding considerations and recommendations 5.1. Summary of the applicability of the available approaches for assessing VCR to AEMO's needs Most previous studies have concluded that survey based approaches for calculating the value of customer reliability (or similar measures such as the value of unserved energy) are preferable to model-based approaches due to their greater ability to address the indirect and intangible costs experienced by end-use customers. Three survey-based models have been used in Australia and New Zealand: Direct costs which has been used with large and small business customers in Australia and large industrial customers in New Zealand; Economic Principle of Substitution which has been used with residential customers in the Victorian VCR methodology; and Choice modelling which has been used with residential and small business customers in Australia (for the investigation of customers' willingness to pay for service improvements, including reliability of supply, at the distribution level) and New Zealand. There is general consensus that the direct cost approach is the most appropriate approach for large industrial customers, though the results of the New Zealand study indicated that these customers also experience significant indirect costs which are difficult to capture in the direct cost approach (particularly in the areas of lost orders and damage to reputation). In addition, great care needs to be taken to ensure that the direct costs counted net out revenue losses that (a) might be made up by the enterprise after the outage ends, or (b) are essentially transfers as the lost production in one area might be compensated for in another area (either within or outside the region). The Economic Principle of Substitution is relatively straightforward to implement, but requires significant care in development to ensure that the range of choices provided to the respondents is relevant to each of the outage durations tested, and provides a smooth and relatively continuous and consistent range of mitigation costs across the various outage durations. The fact that the mitigation choices are repeated may lead to respondent fatigue if the number of outage scenarios tested is too large. This imposes a bound on the number of scenarios that can be tested with any individual respondent, which could lead to the need for larger sample sizes. The fact that the 2007 and 2002 VCR surveys undertaken in Victoria produced very different VCR results is also of some concern, and suggests that it would be useful to investigate this further to ensure that the methodology itself is not a source of volatility in the results. The choice modelling that was used in South Australia and that is currently being used in New Zealand appears to offer a very robust means for presenting choices to customers about the trade-offs they would prefer in the various facets of electricity supply reliability and price. However, experience in South Australia and New Zealand also suggests that this approach requires significantly more effort in questionnaire development and data analysis, and larger sample sizes than the Victorian VCR approach, leading to longer times to complete the study and higher costs. 38

44 Finally, the large difference between the VCR identified in the Victorian study and the very much smaller values identified in the South Australia willingness to pay discrete choice modelling need to be considered. The fact that the South Australian study included aspects of service other than supply reliability makes direct comparison of results difficult, as does the fact that the two surveys were undertaken for different purposes, and therefore were not seeking to get answers to the same question. While the results of the New Zealand study (when available) will provide a comparison in which the purpose of the study is virtually identical with that of the Victorian VCR, the comparability of results will still be subject to considerations regarding differences between the two areas in terms of a number of factors including: Climate The percentage that electricity represents of household and business operating costs The electricity intensity of the customers' end uses in the two areas, and the availability and use of alternative energy sources The nature of the customer mix and particularly the relative proportion of different business types Other considerations It is also important to recognise that the drivers for augmentation of a transmission network include a number of inter-related factors: Thermal/steady state loading The value of avoiding outages to end users in identifiable geographic locations (e.g., in radially connected network elements) requires consideration of geographically defined VCR values, although in practice state-wide values are often substituted. By contrast, the value of avoiding cascading outages in a broad area of the power system following a single thermal failure requires consideration of costs that potentially go far beyond the sum of the direct and indirect costs experienced by individual electricity users. These include the social disruption costs that were addressed in part in the 2007 Victorian VCR, but can also include the costs of so-called high impact, low probability (HILP) events, such as the transmission failure that occurred in Auckland in Incorporation of HILP events within the VCR will therefore require additional analysis. In the first instance, additional effort will be required to assess the actual level of cost experienced in HILP events. In addition, some alteration would need to be made to the VCR calculation method in order to incorporate HILP costs or these costs would need to be considered separately. This is because the VCR calculation method weights the costs of outage events by their probability of occurrence only. As a result, the contribution of HILP events to the VCR would be reduced to almost zero. Security/stability Investment in network augmentation for system security and stability reasons is currently not subject to economic considerations. Rather, these concerns are addressed through deterministic rules concerning the ability of the system to function under (typically) N-1 or (in some cases) N-2 credible contingency events. Competition/market benefit Transmission investments may still be justified where there is little or no impact on reducing or avoiding outages, but there may have a market benefit by reducing congestion or enhancing competition and reducing dispatch prices in the wholesale market. These benefits are taken into account in the regulatory investment test for transmission. 39

45 In sum, there are certain situations in which the VCR may not be material or where it is applied in tandem with a deterministic rule that implies quite a different valuation. Understanding the relevant VCR allows planners to balance the value of load at risk and the cost to mitigate some or all of that risk. This is not allowed under the provisions to maintain security and the VCR is not relevant to these investment decisions. Accordingly, events which can lead to breach of the technical envelope for system security are treated as HILP events where individual end-user VCR values are only a part of the implied value at risk. In HILP events, social disruption costs are high 57, and in the absence of robust methods to establish a workable social disruption value, performance standards based on rules have been and continue to be used. A 'regret theory' framework could be used to infer the VCR in these situations. The challenge then would be to identify when an end-user VCR should not be the sole input to the investment trade-off decision. With the present state of the art this can only be done as an act of policy. Finally, there is also the need to recognise the impact of rationing techniques on how VCR values are applied. Specifically, the assessment of the value of transmission augmentation in parts of the network where any outage that does occur would be limited to only a part of the total customer base at a time because of an ability to ration supply, should be based on data for typical outage durations that occur under a rationing scheme generally 1 to 2 hours Desirable aspects of a VCR measure for transmission investment assessment regardless of approach Regardless of the approach adopted, the following are desirable characteristics for any VCR developed to assess the economic value of improved reliability that would result from a particular transmission system investment: Segmentation by customer and/or network location type such that customer VCR values relevant to the asset under consideration can be constructed The ability to represent indirect and intangible costs Some valuation of HILP events in areas where they could occur Recognition of cases in which rationing would be undertaken, including testing of customers' willingness to accept a reduced level of supply (e.g., demand limitation) for specific combinations of frequency and duration The ability to be correlated to measures used at the distribution level for similar purposes. In sum, it would be desirable for VCR values to be developed at a highly granular level that reflect different customer situations and conditions, and for there to be recognised means for using those values to reflect the outage profile and customer composition of the area served by an asset where consideration is being given to an investment that would improve supply reliability. 57 For example, if a widespread outage occurs, the mitigation actions identified through the Economic Principle of Substitution may not be available. You can go out to dinner if the electricity is out in your neighbourhood, but this may not be possible if the outage has affected all of the restaurants in the city. Also, no value is ascribed to lost time (inconvenience) or the inability to contact family and friends when telecommunications networks are affected by power failure. 40