Whitepaper. Storm is coming: are you ready for big data? By Johan Crols. Copyright 2012 Ferranti Computer Systems. All rights reserved

Transcription

2

3 3 storm is coming Are you taking any risks? Massive amounts of smart metering data will soon flood your business. Processing this big data requires a highly efficient system, with flexibility to remain compliant and powerful analytics to make sense of it. Will your current system keep you safe without raising cost to serve? Today s hot topic in IT is Big Data: how to store, process and make sense of enormous amounts of data. Meanwhile, after years of experimentation and planning, utilities have started the first large-scale rollouts of smart meters and smart grids. The intersection of these two trends creates numerous opportunities and catapults utilities to the forefront of technological innovation. Big and smart: the data challenge Utilities have started to roll out smart meters and smart grids. In Europe for example, 10s of millions of units will be installed in the coming years, with a target of over 200 million by In the US, over 50% of newly installed meters are smart already. And finally, India has just approved a smart-metering roadmap. Those smart meters will be linked with complex architectures of systems, in order to make their data available to the different players in the utilities market landscape. Grid operators, suppliers, balancing companies... but also new players such as Energy Service Companies (ESCOs) will make use of this data to improve existing services, or offer new ones. As of now, however, most smart metering projects hardly leverage the new infrastructure: they collect limited snapshots and neglect the large majority of data. Not surprisingly, having smart meters in place is only the first step. About the author Johan Crols is a seasoned utilities executive. As director Product Management, he plays a key role in defining the MECOMS product and its roadmap. Johan previously headed the B2C department at Essent Belgium, a subsidiary of RWE. Before that, he led numerous projects at utility companies, as a consultant for Accenture and CSC. Utilities around the world face challenges in transporting, storing and processing all the available data from their smart meters. This not only requires very intensive use of (bi-directional) communication networks, but also intensifies the need for storing and processing large amounts of data at near real-time. johan.crols@ferranti.be T:

4 4 A back-of-the-envelope example Smart meter opportunities come with many challenges both on a business as well as on a technical level. Imagine a typical utility with a customer portfolio of 5 million residential electricity customers, who are historically read on a yearly basis. The utility decides to replace this infrastructure and move forward with Smart Meters. Each meter will be read on a 15 minute basis, registering both consumption and production interval series. [Typically, smart meters will track more than just two registers (consumption & production), but will also register data such as voltage level, power quality etc. Finally, in addition to electricity, other commodities, such as gas or even water will be registered.] Based on this information, following numbers can be calculated: Number of reads (+350 billion values / year) 365 days * 96 reads/day * 2 registers * 5 Mio customers Data Throughput (+50 TB of data / year) 150 Bytes * number of reads/year System Transactions (+2,000 billion transactions / year) Number of reads / year *(#validation + #estimation + #calculation) A quick calculation points out that MDM systems will need to process over 10,000 values per second, having an I/O throughput of 200 Mbit (per second), whilst executing nearly 60,000 transactions on the system (per second). Data storage is the next important challenge. The utility in our example will need to add more than 50TB of data per year, not only about consumptions but also quality-of-service. Classic database solutions will not be sufficient to reliably store such volumes of data. Alternatives exist, but place limits on updating the data. Cloud-based storage looks promising, but is as-of-now still unproven for such large amounts of data, and has its own challenges, such as security. These numbers are dazzling and should be met with adequate solutions. Smart metering projects should be focused on optimising business processes and gaining operational efficiency, but to be able to do this, a strong technology foundation is required for communication, processing and data access. Why we need to get smart Above all, processing this data avalanche will be the most important challenge. Data must lead to meaningful insights and improve utilities effectiveness. Without this, investing in smart infrastructure makes little sense. True Business Intelligence is more than running statistical analysis on samples historical data. An intelligent utility will need to analyse metering events must in near-real-time, to allow the operation of a truly smart grid. For example, when several meters within the same subnet send last gasp messages, it should be able to immediately trigger an intervention by field workers. Furthermore, most of the other trends impacting utilities in the near future are data-driven. Social media, mobile and location-aware technology will only increase the importance of a solid meter data management system.

5 5 New business opportunities Maximising the use of this data leads to new opportunities for utilities in managing complex processes, at both ends of the meter-to-cash process. Grid operators will benefit from pro-active grid management, while suppliers/retailers will be able to bill based on real-time consumptions and prices. In between, processes such as switching or market settlement can be optimized. Pre-payment functionality or outage detection are just a few examples of business processes that are very complex nowadays and can become more accurate and precise in the (near) future. Looking further, options such as peak shaving or demand side management become reality. This could lead to entirely new business models, such as time-of-use tariffs or even interruptible contracts. Furthermore the interaction with the customer can be dramatically improved by making use of smart home devices and gateways to inform about activities and events related to the energy consumption. Turning off the heating after the customer has left his home is already technically feasible. Another possibility would be to inform about unusual device consumption, for example when the refrigerator is left opened, or is in need of replacement. hurdles is the lack of standards in the market: every meter vendor is developing its own protocols and features, trying to differentiate. Similar confusion is present in other parts of the (smart) meter / grid infrastructure, such as the meter management systems, routers / data concentrators, etc. Even the communication protocols are different from vendor to vendor, in the grid (PLC, GPRS,...) as well as in-home (ZigBee, WiFi,...). Furthermore, many countries and regions are creating their own, incompatible standards. A secondary issue is the existence of legacy infrastructure: classic old school automatic and manually read meters, combined with potential investments in smart meter pilots. In these diverse landscapes, old and new systems often need to run in parallel, in order to support all technologies present. Making the right choices in infrastructure and future-proof systems is crucial. Not only to overcome the above-mentioned challenges, but also to achieve the potential benefits. In addition, technologies, markets and customer requirements are changing so fast, that incorporation of lessons learned from previous (pilot) implementations is necessary but not always sufficient. These are just a few examples of what is possible. Utilities will typically focus on two or three benefits, and expand and improve along the way. Unfortunately, in order to make use of all these nice new features, business processes and data flows need to be changed drastically and that creates some fundamental challenges. Utilities that have started pilots or have already completed a full implementation are very aware of the challenges associated with these types of initiatives. One of the first

6 6 MECOMS 2012: taming the data avalanche MECOMS is deployed in a wide variety of MDM situations, and the latest version, based on Microsoft Dynamics AX 2012, has unprecedented scalability to process large amounts of data and turn those into valuable insights and actions. Ferranti believes that grid analytics and customer analytics will be the main value drivers for smart metering and smart grids. Therefore, utilities investing in these areas should focus their strategy on dealing with the upcoming data avalanche. Moreover, in close cooperation with Microsoft, Ferranti is developing cutting-edge technologies to further improve its big data capabilities. Ferranti technologies, such as IMA (In Memory Analytics) and SVS (Smart Volume Streaming), will guarantee input, processing and throughput for millions of records per second, fully integrated with traditional components such as asset management, service management, CRM, billing, energy forecasting etc. All MECOMS components are built on a strong architecture, designed for agility, volume and performance in a liberalized market. The combination of having smart data available in every MECOMS module, with the low cost of MECOMS, opens up new business cases. MECOMS MDM Architecture Validation / Estimation / Editing (VEE) Meter 1 Meter n AMI Head-end Smart Metering Communication Bus Smart Volume Streaming (SVS) In-Memory Analytics (IMA) RDBMS Big Data RDBMS Data abstraction layer (DAL) Enterprise Asset Management Smart Grid Applications CIS Components Market Communication

7 7 Performance Benchmark During a proof of concept, 15,000 meter readings per second were processed, including all validation and storage tasks, with only 1 IMA/SVS node. Combing the load of multiple IMA/SVS nodes together (using regular PCs, a standard database server, a 100 Mbit network and no specific performance tweaking), a theoretical throughput of 270,000 metering values per second was reached, validated and stored in the database. 270,000 meter values/second corresponds to 23,3 billion meter values per day. Extrapolating these values to a realistic situation in the field, with 96 meter values per day per meter (15 minute values) and 2 channels per meter, we see that this situation corresponds to several 10s of millions of meters. The database in our setup was able to process the meter data streams from all 18 nodes, with a CPU load on this machine which never went above 25%. More information on this test is available on request. Next to MDM, MECOMS offers integrated modules that make fully use of the available smart data, enabling real time billing or smart home application support for retail companies. It also integrates with work management, minimizing downtime by predicting grid failures. Finally, its open architecture leaves room to add future functionality or even third-party applications, simply because the smart world is changing faster than we can ever imagine. Conclusion Smart meters bring about many new opportunities, but also new challenges. And for all involved market parties the impacts will be different. One thing, however, will be in common: new services will be invented, based on intelligent use of big data. And MECOMS is the right tool to do so. Its flexibility and state-of-the-art technology will enable utilities to reap the experiences of many preceding projects, and still successfully enable positive business cases. The MECOMS Value Proposition Knowing that the smart meter data avalanche is coming, utilities should choose a future-proof solution. With MECOMS, Ferranti offers a complete suite of tightly integrated modules that support any meter-to-cash implementation, with or without smart meters. MECOMS offers a fully smart-metering-ready MDM solution, able to deal with very high volumes. It foresees an AMI vendor-independent, performing solution with all common functionalities. In addition, it is an integrated solution which is built on Microsoft technology and fully integrated into Microsoft s Dynamics AX.

8 Global contact Antwerp, Belgium Oberhausen, Germany Gurgaon, India Singapore, Singapore Amsterdam, The Netherlands London, United Kingdom Big Data whitepaper V Ferranti Computer Systems NV, all rights reserved. MECOMS is a trademark of Ferranti Computer Systems NV. Microsoft and Microsoft Dynamics AX are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.