THE SOFT GRID :

Transcription

1 THE SOFT GRID : Big Data & Utility Analytics for Smart Grid Research Excerpt A Greentech Media Company

2 Research Excerpt RESEARCH EXCERPT This is an excerpt from the December 2012 GTM Research report The Soft Grid : Big Data and Utility Analytics for Smart Grid. Research for this report was conducted over a six-month span and included primary and secondary research as well as extensive interviews with both industry players and utilities. GTM Research, a division of Greentech Media, provides critical and timely market analysis in the form of research reports, data services, advisory services and strategic consulting. GTM Research s analysis also underpins Greentech Media s webinars and live events. GTMRESEARCH APRIL

3 Executive Summary And Key Findings 1. EXECUTIVE SUMMARY AND KEY FINDINGS Analytics: Key Findings 1. While utilities like to claim that they have analytics, they really don t. Utilities tend to have last-gen business intelligence (BI) reporting solutions that they call analytics, but that typically amount to not much more than reporting tools or descriptive analytics (primarily based on older database architectures running SQL), as opposed to the real-time and predictive analytics using complex event processing to which the term analytics is now commonly understood to refer. 2. Utilities are now seeking to become more proactive in decision-making, adjusting their strategies based on reasonable predictive views into the future, thus allowing them to side-step problems and capitalize on the smart grid technologies that are now being deployed at scale. Predictive analytics, capable of managing intermittent loads, renewables, rapidly changing weather patterns and other grid conditions, represent the ultimate goal for smart grid capabilities. 3. In this report, we present a taxonomy that identifies the three major domains in which analytics can aid utilities, all of which are ripe with opportunity. A. Enterprise analytics B. Grid operations analytics C. Consumer analytics Figure 1-1: UTILITIES THREE PRIMARY DOMAINS FOR ANALYTICS ENTERPRISE ANALYTICS GRID OPERATIONS ANALYTICS CONSUMER ANALYTICS Moving from Traditional, Historical Analytics to Real-Time Predictive Analytics Complete Situational Awareness Business Intelligence (BI) Trading with live look at the Grid Simulation/Visualization Grid Optimization and Operational Intelligence Asset Management Analytics Crisis Management Analytics DMS Analytics Outage Management Analytics/Fault Detection and Correction Weather/Location data Mobile Workforce Management Energy Theft Behavioral Analytics Tiered Pricing - Trading, Selling Megawatts (DR) Building Energy Management Power Analytics (Load Flow) Social Media Data Intergration DG/EV/Microgrid Analytics COMMUNICATION LAYER END-TO-END COMMS PLATFORM POWER LAYER INFRASTRUCTURE GENERATION TRANSMISSION SUBSTATION DISTRIBUTION HOME / BUILDING / DATA CENTERS DISTRIBUTED GENERATION AND STORAGE UTILITY INFRASTRUCTURE CONSUMER Source: GTM RESEARCH GTMRESEARCH APRIL

4 Executive Summary And Key Findings 4. It is our prediction that in three years, talking about analytics without mentioning big data will be a bit like talking about without mentioning the internet -- the two will become intrinsically linked, with one being an application (analytics) sitting on top of the other, the foundational layer (big data storage and processing). 5. GTM Research believes that it is high-performance analytics, such as predictive analytics, which will prove to be the most significant value-add in the big data age, as new data management technologies prove reliable and fundamental, and as data storage infrastructure moves to commoditization. Utilities Limited Experience With Analytics: Key Findings 1. Based on discussions with utility CIOs, the utility industry appears to be weary of the process of selecting and commissioning custom products from vendors and of the consultant-heavy experience of deploying them; opensource big data products offer a future with more flexibility and lower costs. 2. The four biggest challenges for utilities in terms of having enterprise IT architectures sufficiently prepared for smart grid and big data are: A. Siloed systems that hinder easy data sharing B. Systems integrations is no small task C. No existing platform in place for unstructured data D. No single platform is going to be able to handle all needs All of these obstacles speak to the central challenge of making disparate, incompatible datasets usable and valuable across the enterprise. GTMRESEARCH APRIL

5 The Emergence Of The Soft Grid 2. THE EMERGENCE OF THE SOFT GRID 2.1 Utilities Existing and Evolving IT Architecture Challenges It is necessary to stress that utility IT architectures are in many ways only the jump-off point when it comes to realizing all of the benefits that can be reaped via big data and analytics. In other words, smart IT architecture has to be viewed as the gateway to a smart grid. An informal survey of utility CIO and CTOs that GTM Research conducted confirms that very few utilities have an official, overarching data strategy in place today. The deployment of smart hardware, including smart meters and distribution devices such as automatic voltage regulators, is not only continuing, but will also accelerate as these devices become even more affordable. However, in order to leverage the capabilities of these new devices and to implement smart grid capabilities like dynamic pricing, grid optimization, self-healing grids and renewables integration, utilities desperately need to turn their attention to upgrading their IT systems and architecture. To the industry s credit, many utilities are currently engaged in this process. As they do so, many are discovering that they have an out-of-date patchwork of legacy systems with little, if any, architectural consistency. In the past, ad hoc point-to-point integration between pairs of applications was sufficient to handle basic needs, such as entering outage reports from customer service applications into an outage management system, or creating an engineering work order for execution by a maintenance crew using a mobile workforce management application. However, the age of big data could have devastating results on utility systems if IT architectures are not sufficiently designed and engineered for the level of performance and sophistication it will require. Over the past three to five years, utility executives have been discovering that the ad hoc and unplanned nature of their systems threatens to block their forward progress in achieving smart grid business goals and frankly this realization came before it was evident that big data was on the way! Every utility now needs a pragmatic roadmap that delivers on the promise of smart grid by leveraging and integrating legacy systems too complex to replace (over the immediate term), while putting a comprehensive plan in place to account for big data, as well as overcome the four biggest data challenges utilities are facing (see below). GTMRESEARCH APRIL

6 The Emergence Of The Soft Grid Figure 2-1: FOUR MAJOR IT ARCHITECTURE CHALLENGES FOR UTILITIES 1. Siloed systems that prohibit easy data sharing. Many smart grid applications are composite applications that draw on data and functions from multiple systems. 2. Systems integration is no small task. Related to the challenge of siloed systems is the challenge of creating the underlying architectures that allow easy data access, sharing, and collaboration between systems. It is particularly difficult to upgrade architectures that serve as the foundation for electric grids on which millions of customers depend. 3. No existing platform in place for unstructured data. An estimated 75% to 90% of all new data being generated is unstructured. Utilities as a group are ill prepared for this shift, and most have not explored or tested big data platforms in a meaningful way. 4. No single platform is going to be able to handle all needs. Companies like Facebook and Twitter have had to constantly rebuild and update their architectures in order to meet their ever-evolving, rapidly expanding needs. This experience likely will be applicable to the utility space, as well. Utilities must look to hybrid architectures to integrate the totality of their smart grid systems, as well as their emerging big data needs. Further, massive data warehouses are difficult to support over the long term; often the best data architecture designs are those that keep master data close to the processing engine/analytics, or vice versa. SOURCE: GTM RESEARCH The previously published GTM Research report The Smart Utility Enterprise concluded that only a hybrid architecture that achieves the following conditions will truly be equipped to implement a smart grid. Separates the data management; application logic and presentation into separate layers (i.e., is multitier) Has helper applications that surround legacy systems with new functionality Embraces service-oriented architecture (SOA) that encapsulates application functions into modular components for reuse Utilizes agent-based architecture for distributing intelligence to nodes like IED Supports big and unruly (i.e., unstructured) data Finally, the following list of suggested best practices for utilities moving into smart grid was generated by systems integrator Accenture. GTMRESEARCH APRIL

7 The Emergence Of The Soft Grid Figure 2-2: KEY BEST PRACTICES FOR DEVELOPING AND IMPLEMENTING SMART GRID SOLUTIONS 1. Recognize smart grid data classes and their characteristics to develop comprehensive smart grid data management and governance capabilities. 2. Consider how data sources can support multiple outcomes via analytics and visualization to realize the maximum value from the sensing infrastructure. 3. Consider distributed data, event processing and analytics architectures to help resolve latency, scale and robustness challenges. 4. Consider the whole smart grid challenge when planning data management, analytics and visualization capabilities not just advanced metering infrastructure to avoid stranded investments or capability impediment. 5. Design data architectures that leverage quality master data to match data classes and analytics/ application characteristics. A giant data warehouse is rarely maintainable. 6. Look to new tools such as complex event processing to handle challenges around processing new data classes. Managing the new smart grid data deluge via historical transaction processing approaches is likely not scalable. 7. Develop business process transformation plans at the same time as and in alignment with smart grid designs. SOURCE: ACCENTURE GTMRESEARCH APRIL

8 Big Data And Analytics 3. BIG DATA AND ANALYTICS 3.1 Top 10 Smart Grid Drivers Of Big Data and Analytics The following list identifies ten drivers that will likely increase the speed at which big-data and analytics technologies will be adopted in the utility industry. Figure 3-1: TEN DRIVERS WHICH WILL MOVE UTILITES TO BIG DATA AND ANALYTICS 1. Utilities seeking ROI for advanced metering investments to justify the billions spent on AMI infrastructure. 2. The new technologies will improve the usefulness and granularity of demand-side management and demand response programs in terms of better customer segmentation and other benefits. 3. The new technologies will improve asset management in an asset-intensive industry. 4. More data and analytics will lead to better grid operations management in extreme weather, including reduced outage times, cost savings from better SAIFA and SAIDI indexes, and fewer dissatisfied customers. 5. The new technologies will lead to reduced energy theft and other non-technical losses. 6. The new technologies will smooth the integration of renewables and EVs. 7. The new technologies will facilitate the use of geospatial intelligence to visualize grid operations. 8. The new technologies will ease the strain being placed on traditional business intelligence (BI) and analytic solutions from the exponential growth of data. 9. The speed of adoption will likely increase when key stakeholders in the utility industry acknowledge that today s utility enterprise IT architectures are not sufficient to meet future needs, specifically in terms of their lack of cross-departmental data sharing capabilities. 10. New vendor technologies are driving shifts in terms of both what is affordable and what is possible. SOURCE: GTM RESEARCH GTMRESEARCH APRIL

9 Vendor Profiles And Comparative Analysis 4. VENDOR PROFILES AND COMPARATIVE ANALYSIS 4.1 Introduction In this section, we delve more deeply into the leading technology vendors offering solutions across the various subsectors and submarkets of the soft grid space, including data storage, data infrastructure, data management, and the growing application layer of smart grid has presented utilities with a growing number of new offerings, including those with features such as geospatial visualization, cloud-based solutions, cluster analysis tools, intelligent alarm filtering capabilities, and others. These advances have generated a great deal of enthusiasm, as well as a considerable amount of confusion. This seems like a fitting analogy for where the market stands today: it has thus far been successful in terms of creating a lot of excitement around big data and analytics, but hasn t yet been equally successful in demonstrating either the capabilities or the business case for these new technologies. To some utility executives, the need for analytics is clearly obvious, but it appears that the majority of industry insiders in this traditionally change-averse industry still need a fair amount of education in order to be apprised of current and emerging technologies. Adding to the confusion is the fact that seemingly every company in the market even those with little applicable experience in the field is suddenly developing or offering analytics products. Ultimately, however, both educational efforts and ROI prove-out will need to take place in order to spur investment and market proliferation. However, after having spoken to dozens of utility executives on this topic, as well as conducting an extensive survey of more than 70 North American utility executives, it is clear that interest in these emerging technologies is now beginning to mount. Soon, it is likely that utilities will begin to adopt analytics technologies that will allow them to become more proactive in decision-making and to adjust their strategy based on the predictive views into the future that the technologies will facilitate. This will allow utilities to capitalize on the smart grid technologies that are now being deployed at scale; side-step potential problems; and better handle the steep challenges facing an industry in transition. GTMRESEARCH APRIL

10 Vendor Profiles And Comparative Analysis 4.2 Vendor Taxomony and Vendor Rankings Figure 4-1: LEADING VENDORS IN SOFT GRID SOURCE: GTM RESEARCH Data Management and Movement Layer The data management layer has been a focal point of this report, and along with the enterprise IT architecture that supports it, it represents both the biggest challenge and the biggest opportunity for today s utilities. It is abundantly clear that we are now in the big data age, but how utilities will manage this paradigm shift remains to be seen. Across the industry, a gauntlet has been thrown down, and upstarts springing out of the distributed processing world of Hadoop see a multi-billion-dollar market up for grabs, as the need for real-time analytics in a world of massive, unstructured and complex data demands performance requirements above and beyond the capabilities of the legacy relational database management systems of yesteryear. Today s data no longer fits neatly into columns and rows, and is likely to be generated on the terabyte- or petabyte-scale. As such, old and antiquated architectures are destined to fall. GTMRESEARCH APRIL

11 Vendor Profiles And Comparative Analysis Predictably, there are many vendors (and utilities) taking an If it ain t broke, don t fix it stance on the issue of data management. Year after year, companies like Oracle continue their incremental gains in the speed and performance of their relational database management systems. However, the emerging technologies are not ready to fully replace their predecessors. Principal Hadoop founder Doug Cutting describes his company s platform as augmenting and not replacing regular databases. As should be expected, Oracle, Microsoft and others are experimenting with big-data products and platforms, but every database expert consulted for this report cautioned that at the moment, those offerings remain immature and experimental. The implications of data management for smart grid are vast. Having said that, however, we don t expect utilities to begin making large bets on technologies like Hadoop in the near term, for several reasons. First, the solution offerings are relatively young and utilities historically aren t big risk-takers on new technologies. Second, and more importantly, utilities haven t yet fully grasped the true value and potential of distributed data processing. This may be due to the fact that utilities first foray into dealing with big data namely, smart meter data -- has relied upon meter data management systems that are based on older relational database management systems. Over the past five years, utilities chief data concern has been ensuring that smart meter data flowed reliably into their CIS/billing systems, so that the utility could ensure payement. There have been some gestures and claims made in the industry about integrating siloed departments and building intelligent IT enterprises, but in truth, maintaining accurate and efficient billing standards has been the leading concern. As 2013 approaches, meter data management systems have now been proven to be reliable, and many other concerns, including the question of the reliability of AMI networks, have largely been worked out. As a result, utility CIOs and data experts have been freed up to focus on extending smart grid into other applications. In undertaking this process, they will begin to re-examine how their data is architected and managed. GTMRESEARCH APRIL

12 Vendor Profiles And Comparative Analysis The immense wave of unstructured data that is coming to the grid in the near future is the real big data challenge. Up until now, the data that utilities have had to manage has been predictable. For example, utilities know when data from an average meter read will be sent, and roughly how big the resulting data will be. On the other hand, it is very difficult to anticipate the deluge of data that an extreme weather event will initiate, including inputs from DMS, OMS, integrated weather systems and other systems and sensors related to other grid assets that may be experiencing unusual performance. As the grid starts to send frequent status updates on all critical and non-critical assets, the only way to capture this data will likely be with advanced big-data tools. As such, GTM Research believes that legacy RDBMS will be unable to meet the comprehensive future needs of the smart grid. Up until now, systems integrators and middleware players have been able to patch new solutions onto legacy systems, but at a certain point, big data will begin to overwhelm spaghetti architecture. Figure 4-2: COMPARATIVE VENDOR RANKINGS FOR THE DATA MANAGEMENT AND MOVEMENT LAYER MARKET BREADTH UTILITY RELATIONSHIPS FUTURE NEEDS EXTENSIBILITY FLEXIBILITY SCALABILITY REPORTING & ENTERPRISE TOOLS COST SECURITY WEIGHTED AVERAGE SAS Teradata IBM EMC/ Greenplum Oracle Cisco SAP Versant Hortonworks OSIsoft Cloudera Hadapt =highest score 1=lowest score GTM RESEARCH GTMRESEARCH APRIL

13 Vendor Profiles And Comparative Analysis Analytics and Applications Layer The analytics and applications layer covers the new and necessary solutions that vendors are bringing to the market. In the utility/smart grid space, there are four domains that will increasingly rely on analytics: the enterprise, grid operations (T&D), consumer-oriented offerings, and energy portfolio management and trading. The following diagram demonstrates where three of these domains sit relative to the physical grid infrastructure. Figure 4-3: UTILITIES THREE PRIMARY DOMAINS FOR ANALYTICS ENTERPRISE ANALYTICS GRID OPERATIONS ANALYTICS CONSUMER ANALYTICS Moving from Traditional, Historical Analytics to Real-Time Predictive Analytics Complete Situational Awareness Business Intelligence (BI) Trading with live look at the Grid Simulation/Visualization Grid Optimization and Operational Intelligence Asset Management Analytics Crisis Management Analytics DMS Analytics Outage Management Analytics/Fault Detection and Correction Weather/Location data Mobile Workforce Management Energy Theft Behavioral Analytics Tiered Pricing - Trading, Selling Megawatts (DR) Building Energy Management Power Analytics (Load Flow) Social Media Data Intergration DG/EV/Microgrid Analytics COMMUNICATION LAYER END-TO-END COMMS PLATFORM POWER LAYER INFRASTRUCTURE GENERATION TRANSMISSION SUBSTATION DISTRIBUTION HOME / BUILDING / DATA CENTERS DISTRIBUTED GENERATION AND STORAGE GTM RESEARCH UTILITY INFRASTRUCTURE CONSUMER Virtually all smart grid vendors are competing in the analytics and applications layer. It is difficult to provide an apples-toapples comparison of these vendors and their product and service offerings, as the solutions that each are offering are often unique. The following list identifies the leading solutions that vendors are targeting in the timeframe. Figure 4-4: LEADING UTILITY SMART GRID ANALYTICS FOR Geospatial and visual analytics that offer a centralized view of multiple technologies Outage restoration analytics Grid optimization and power quality (including voltage control and conservation) Peak load management (via demand-side management analytics) and energy portfolio management analytics Asset protection analytics and predictive asset maintenance Service quality analytics Vegetation management analytics Revenue protection (including theft and nontechnical loss analytics) Analytics to correct legacy system errors (such as CIS and MDM) Consumer behavioral analytics (including comparison to neighbors/peers) Home signature and thermostat control analytics Time-of-use pricing analytics Renewable energy and storage analytics SOURCE: GTM RESEARCH GTMRESEARCH APRIL

14 Vendor Profiles And Comparative Analysis Figure 4-5: COMPARATIVE VENDOR RANKINGS FOR DATA ANALYTICS AND APPLICATION LAYER MARKET BREADTH UTILITY RELATIONSHIPS INDUSTRY-LEADING SOLUTION VALUE FOR SERVICES/ SOLUTIONS FUTURE NEEDS OF CUSTOMERS EXTENSIBILITY OF FEATURES SOPHISTICATION OF ANALYTICS END USER EXPERIENCE STRATEGY WEIGHTED AVERAGE SAS IBM Opower Space-Time Insights EcoFactor GE Siemens emeter (a Siemens co.) Accenture ABB/Ventyx Landis+Gyr Aclara Tendril Ecologic Analytics (a Landis+Gyr company) SOURCE: GTM RESEARCH Silver Spring Networks Echelon DataRaker Telvent (a Schneider Electric co.) EnerNOC Itron Tableau Software Energate Grid Net Power Analytics ECOtality =highest score 1=lowest score GTMRESEARCH APRIL

15 Additional Resources FOR MORE INFORMATION ON BIG DATA & UTILITY ANALYTICS High-Performance Analytics for The Smart Grid This white paper presents results from a survey of more than 70 North American utility executives. The research reveals how utilities are defining, conceptualizing and understanding both big data and analytics. The paper also explores some of the barriers utilities face in both day-to-day use and enterprisewide adoption of analytics. SAS Digital Magazine on Energy Transformation This multi-media asset covers the hottest topics in today s energy industry, including Dodd-Frank regulatory impacts, analytics for distribution/asset optimization, and exploration of unconventional oil and gas resources. To download the interactive magazine type this URL into the browser: Explore SAS Visual Analytics SAS Visual Analytics provides unique insights that allow utilities to understand how customer and market behaviors influence drive profitable growth opportunities. With new online demos, a utility can experience how SAS Visual Analytics provides an in-depth knowledge of customers, assets and operations. Log on for a test drive today: Contact SAS SAS is the leader in business analytics software and services. For over 35 years, our solutions have enabled utilities to find hidden patterns in data and create intelligence from disparate data sources for effective decision-making. Find out more at sas.com/utilities. Tim Fairchild Director, SAS Global Energy Practice Tim.Fairchild@sas.com GTMRESEARCH APRIL