Adaptive Rate Stream Processing for Smart Grid Applications on Clouds

Transcription

1 Adaptive Rate Stream Processing for Smart Grid Applications on Clouds Yogesh Simmhan, Baohua Cao, Michail Giakkoupis and Viktor K. Prasanna Center for Energy Informatics Ming Hsieh Department of Electrical Engineering Department of Computer Science University of Southern California, Los Angeles CA 989 {simmhan, baohuaca, mgiakkoup, ABSTRACT The growing deployment of smart meters to continuously measure power usage by consumers as part of a smarter power grid is resulting in the need for utilities to process and analyze thousands of data streams in near realtime to detect system overload. We describe the use of Cloud platforms to perform scalable, latency sensitive stream processing. One unique aspect of our work is the use of an adaptive rate control mechanism to restrict the stream rate to meet accuracy requirements for smart grid applications while consuming less bandwidth and computational overhead within the Cloud for stream processing. We describe our smart grid stream processing pipeline modeled using IBM InfoSphere Streams that we deploy on a Eucalyptus private Cloud, and present experimental results that show 5% bandwidth savings as a result of our adaptive stream rate control. Categories and Subject Descriptors C.2.4 [Distributed Systems]: Distributed applications; D.2.11 [Software Architectures]: Domain-specific architectures; H.2.4 [Systems]: Query processing; J.2 [PHYSICAL SCIENCES AND ENGINEERING]: Engineering General Terms Design, Algorithms, Experimentation Keywords Smart Grid, Stream Processing, Adaptive rate control, Resource management, Cloud platform 1. INTRODUCTION Growing awareness of environmental sustainability and security is causing a renewed focus on improving the electrical power infrastructure to increase its efficiency, and allow optimal use of available power capacity to meet growing energy needs. Electrical utilities are upgrading to Smart Grids, where Advanced Metering Infrastructure (AMI) meters, also known as smart meters, installed at consumer locations allow bi-directional network communication between the utility and power consumer in realtime 1. One consequence of smart metering is the access to power usage information at finer time resolutions than was previously available (once per minute vs. once per month). This information can be used for realtime demand-response optimization, where utilities forecast impending peak power usage and initiate load curtailment operations through pricing signals [9]. For example, the Los Angeles Smart Grid Demonstration Project [14] will use such realtime data from 1.4 million consumers in the largest US municipal utility to ensure that demand does not spike beyond (fixed) available capacity. Low latency stream processing is a key component of a software architecture for demand response applications. The stream processing system ingests the smart meter data arriving from consumers and acts as a first responder to detect local and global power usage skews to alert the utility operator. At 1 KB per event generated each minute, 2 TB of data will stream each day into the Los Angeles power utility. Processing such large scale streams can be compute and data intensive, and public or private Cloud platforms provide a scalable and flexible infrastructure for building such smart grid applications [15]. However, computational and bandwidth constraints at the consumer and utility means that power usage data streamed at static rates from smart meters to the utility can either be at too high a latency to detect usage skews in time or at too high a rate to computationally overwhelm the system. Smart meters connect to the utility using heterogeneous networks that range from low bandwidth Power Line Carriers (PLC) at 2Kbps to 3G cellular networks at 2Mbps and ZigBee at 25Kbps. Network bandwidth can thus be a scare resource at the consumer end. At the utility, bandwidth available into the public Cloud may be less of a concern due to the multiple concurrent streams. However, the bandwidth cost for public Clouds encourages conservation. Too, the smart meter traffic can be bursty since they send data independently, causing instantaneous bandwidth needs to peak. For Los Angeles utility, the average bandwidth works out to 2Mbps for 1KB/min from 1.4M AMIs. If using a private Cloud, sufficient bandwidth for peak network 1 FERC Assessment of Demand Response and Advanced metering, Staff Report, December 28.

2 demand has to be provisioned. These trade-offs between the quality of service required by smart grid applications and the resource constraints in the Cloud motivate the need for an adaptive stream rate control mechanism for generating smart meter tuples. Unlike prior work, we use the application s operational needs needs as our throttle metric. Specifically, for Smart Grid applications, we use the difference between the available power capacity at the utility and the current cumulative power usage by the consumers as the throttle function. Customer Data Sources AMI Industrial/Commercial AMI Residential Building TCP/IP Control Feedback InfoSphere Streams Input Streams Streams Processing Response Throttle Controller In this paper, we make two specific and unique contributions: (1) We implement a smart grid stream processing pipeline for early warning of peak power load on top of a private Cloud platform, and (2) we design and implement an application-driven, adaptive stream rate control that results in a 5% reduction in bandwidth usage over a constant rate pipeline. We demonstrate the pipeline and adaptive stream rate using real power usage data from the USC campus micro-grid testbed for smart meter data from 5 buildings, as a precursor to further scalability tests for city-scale stream processing. The rest of the paper is organized as follows. We present related work in Section 2. We describe our Smart Grid streaming pipeline in Section 3. We introduce the adaptive rate stream processing logic and implementation in Section 4, and experimentally evaluate its performance in Section 5. We present our conclusions and discuss future work in Section RELATED WORK Stream processing systems execute continuous queries on a moving window of data tuples, and have their roots in data processing for sensor networks. Streaming systems such as TelegraphCQ [8] and Aurora [3]/Borealis [2] have made seminal contributions to this field, and a Continuous Query Language (CQL) [4] inspired by SQL has been proposed. Stream processing has also been studied as part of the OGSA-DAI project for Grid computing [13] and for environmental data streams on Cloud platforms [17]. Streaming systems have used resources constraints to modulate both stream rates and their processing. [5] uses tuple dropping ( load shedding ) as a means to ensure maximal output processing rate when computational resources are limited. Another approach [16] addresses short-comings of frequency-based random input sampling and load shedding by proposing an age-based model for sliding window joins of multiple streams in a memory constrained environment. The GrubJoin algorithm in [1] performs optimal window harvesting for m-way stream joins using CPU aware load shedding. All of these approaches use computational, memory or network resource constraints for load shedding rather than an application s quality of service needs. The adaptive stream rate technique we propose uses application sensitive policies to perform stream throttling that does not drop tuples (causing data loss), but instead aggregates tuple values at the stream source and limits the rate at which the tuples are generated. Work on TCP congestion control uses a linear increase and Eucalyptus Cloud Figure 1: Stream processing architecture for Smart Meter data on Clouds. exponential backoff algorithm [11, 12] to maintain packet flow equilibrium at the transport layer. Again, these algorithms use transport level constraints rather than application level knowledge (such as deep packet inspection) to control flow rates. While stream processing is a natural fit for emerging smart grid applications 2, there has been no work to our knowledge that leverages the scalability offered by Cloud platforms for deploying such smart grid analysis and detection pipelines. Our deployment of a streaming smart grid application on the Eucalyptus private Cloud provides early guidance for streaming smart grid applications to leverage Clouds. 3. SMART GRID STREAM PROCESSING ON CLOUDS 3.1 InfoSphere Streams IBM InfoSphere Streams is a stream processing system to continuously analyze massive volumes of streaming data for business activity monitoring and active diagnostics [6, 7]. InfoSphere Streams consists of a runtime environment that contains Stream Instances running on one or more hosts, and the Stream Processing Application Declarative Engine or SPADE stream programming model that is executed by the runtime environment. The SPADE programming model supports stream data sources that continuously generate tuples that contain typed attributes. Operators take one or more streams as input, process the tuples and attributes, and produce one or more streams as output. Some supported operators are stream source to read tuples from, stream sink to write tuples to, functors to perform filtering and projection of tuples, Aggregate for grouping tuples, and so on. Streams can be consumed as sliding or tumbling windows of tuples. A Stream Application is a collection of operators connected by streams. Several stream sources are supported by InfoSphere [1], including TCP, UDP, files and HTTP URLs, with different tuple serialization formats. We use TCP source streams exposed through remote socket servers at named hosts that 2 Oracle Solutions for Utilities, Oracle Corp., 21.

3 Superscript = meter id Subscript = time AMIs (a 1, t 1, u 1 1) N / W if(u i j>u max) Condition (a m, t 1, u m 1) Condition Send Alert if(u i j>1.25*u i avg) Condition Send Alert Store DB/File Update u i sum Running Daily Sum Update u i avg AMI s 15min Avg Utility s 15min Avg Update u * avg R *++ if(c * - u* avg<c safe) Condition Figure 2: Stream processing pipeline used for continuous monitoring of energy usage. Processing elements in dotted lines show the addition of throttle logic. the SPADE program connects to. 3.2 Eucalyptus Cloud Platform Eucalyptus 3 provides open source tools and services to configure and deploy a private cloud platform that provides Infrastructure as a Service (IaaS). Eucalyptus supports deploying and instantiating virtual machine (VM) images that conform to the Amazon Elastic Compute Cloud (EC2) specification 4. Eucalyptus also supports shared storage services that are the syntactic and semantic equivalent of Amazon Simple Storage Service (S3) and Elastic Block Storage (EBS). 3.3 Running InfoSphere Stream on the Eucalyptus Private Cloud Platform R *-- Increase AMI Rate Decrease AMI Rate InfoSphere Streams can run on a cluster but does not support out-of-the-box deployment to a Cloud platform. In order to instantiate a stream processing environment on the Eucalyptus private Cloud, we build customized VM images with InfoSphere Streams deployed on them. When the VM instances come online, the stream instances present in each can communicate with each other. This communication, however, is initiated externally by a SPADE application started on a stream instance and configured with a list of named stream instances on specific hosts. We only use the compute node virtualization of Eucalyptus for our stream instances and not the storage services. In the absence of shared storage, the VM image needs to have a copy of the compiled SPADE application installed in it. All VM instances can access the local SPADE application for execution. Actual exchange of tuples between operators running on the stream instances happens over the network. Figure 1 shows this architecture. Smart meters are present on the public internet and expose the power usage data as streams accessible over TCP sockets. SPADE applications running on Stream Instances in the private Cloud execute the Smart Grid application logic to process the smart meter streams. 3.4 Stream Processing Pipeline for Smart Meter Data Analysis The smart meter stream processing pipeline we implement is used to detect power consumers who individually or collectively exceed certain usage limits set by the utility. This acts as early warning indicator of a peak load event. Figure shows the logical pipeline. Each smart meter is a stream source whose tuples have the identity of the smart meter, power used within a time duration, and the timestamps of the measurement interval. Additional metadata about the smart meter and consumer is part of the payload but will be ignored for the purposes of this discussion. Each tuple is about 1KB in size. The pipeline first checks if each individual power usage tuple reports usage that exceeds a certain constant threshold, U max m defined by the utility. Crossing this will trigger a critical notification to a utility manager. Next, a relative condition checks if the user s consumption increases by more than 25% since their previous consumption, and this triggers a less critical notification. The pipeline then archives the tuple into a sink file and proceeds to compute a running sum of the daily usage by the consumer. Subsequently, the running average over a tumbling window is updated. These operations are performed for each smart meter stream (shaded in brown in Figure 2). Next, the pipeline aggregates smart meter tuples across all streams using a tumbling window to calculate the cumulative consumer energy usage within a 15 min time window. This stream operator (shaded green in Figure 2) calculates the total load on the utility. It can be used to alert the utility manager in case, say, the total consumption reaches 8%, 9% and >1% of available power capacity at the utility. Operators shown in dotted lines are not part of the application logic and form the adaptive throttling introduced next. 4. STREAM PROCESSING WITH ADAPTIVE STREAM RATES The smart meter data analysis pipeline (Figure 2) operates on tuples from the smart meter stream as they are generated. The smart meter controls the rate at which these tuples are sent to the utility, and this is typically set statically when configuring the meter initially. The disadvantages of this static rate are two fold. One, setting too high a rate at which power usage data is published will cause excess resources (network bandwidth, compute VMs for stream processing) to be utilized by the stream processing system. Data usage tuples published every minute is unecessary during, say, the night when the utility has excess power capacity and the need to detect a threshold breach by a specific consumer or across multiple consumers is less critical due to the buffer power capacity. The information about a breach may be useful for future diagnostics but is not latency sensitive. Second, setting too low a static rate at which power usage tuples are published by the smart meters can cause a threshold breach to be missed during peak load periods. During the middle of the day, the total power used by consumers edges closer to the available capacity of the utility. A freak heatwave can, for example, cause consumers to increase power consumption within a short duration and lead to power capacity shortage at the utility leading to brownouts and blackouts. Early warning is necessary for the utility to react by bringing additional power capacity online (e.g. by spinning up gnerators), or initiate load curtailment operations through dynamic power pricing. These factors necessitate a smart meter stream rate that adapts dynamically to the current application policy needs.

4 4.1 Rate Adaptation Logic The adaptation logic that we use for the smart meter data analysis pipeline is intuitive and based on the premise that as the total power usage within the utility approaches total available capacity, usage information is required more frequently to detect and forecast a peak load event with low latency to allow a timely response. Our rate control logic subtracts the power usage across all consumers in the utility during the most recent time window from the total available capacity, and determines a stream rate that is inversely proportional to this value. i.e. StreamRate 1 (AvailableCapacity CurrentUsage) In our implementation, we use stream rates that change nonlinearly, but proportionally, to the above equation by using different rate slabs for different proportions. We also define a static upper bound and lower threshold for the tuple rates to ensure that the minimum and maximum number of power usage tuples sent within a time period is bounded. 4.2 Adapative Rate Control Implementation The adaptive rate control (or throttle) logic is implemented as a trailing operator in the stream processing pipeline after the cumulative power usage across all consumers has been calculated. Unlike other operators that are implemented using SPADE, the throttle operator is implemented as an external Java user defined operator for two reasons. One, it allows finer control over the adaptation logic such as the rate slabs that we use, and allows more complex policies to be used in the future. Second, it allows the stream processing system to connect to the smart meters through an independent control socket for sending the rate control signals to better manage open network sockets. SPADE provides a Java base-class, AbstractOperator, which can be extended to create a user defined operator that accepts streams as inputs and outputs. Our throttle operator accepts the current total power usage value as tuples from the SPADE aggregate operator and evaluates the target rate. If the current stream rate matches the target stream rate using the adaptation logic (or if the target rates is beyond the bounds) then no rate control signal is sent and the current stream rate is maintained. Otherwise, the Java operator connects to the control port of each smart meter and sends a signal which indicates the new stream rate for the smart meter. The smart meters update their internal timers to reflect the new rate and send subsequent power usage tuples at the new rate over the data streams. These throttle operators are shown using dotted lines in Figure 2, and the orange throttle controller block in Figure EXPERIMENTAL EVALUATION Our experiments evaluate the effectiveness of the adaptive stream rate control mechanism to show its conservation of network bandwidth. For our this evaluation, we simulate virtual smart meters that replicate the behavior of smart meters installed at the USC campus micro-grid that is a testbed for the Los Angeles Smart Grid Demonstration Project. Each virtual smart meter is implemented as a light-weight Java thread that has access to historical smart meter power usage data for a single building in the campus micro-grid recorded at 1-minute intervals for a 48 hour period. We use data for 5 buildings on campus and correspondingly instantiate 5 virtual smart meters that each represent one (building) consumer each. Each virtual smart meter has two TCP server sockets: one is the data stream for transmitting power usage tuples to the stream pipeline, and the other is a control port to receive adaptive rate control signals. For our experimental setup, we run the 5 Java threads in a single Java process on a server with 24 AMD Opteron CPU Cores rated at 2.1GHz each and a memory capacity of 32GB. The server is connected to the Eucalyptus private Cloud over 1Mbps Ethernet. The Eucalyptus platform v2. runs on a 128-core Cluster, with each VM instance running CentOS 5 Linux distribution on 1 CPU core with 2GB of available memory. However, for our initial evaluation presented here, the SPADE application pipeline (Figure 2) runs on one InfoSphere Stream Instance v1.2.1 and use just one Eucalyptus VM instance. This SPADE pipeline translates to 11 operators (or processing elements) and three file sinks for each smart meter stream being processed. With 5 virtual smart meters, the stream instance runs 55 SPADE operators and has 15 file handles open. The SPADE application is compiled on Eclipse using the IBM Streams Studio plugin and deployed on the virtual machine image before the Eucalyptus VM is instantiated. 5.1 Constant Rate Stream Processing We run the stream processing pipeline with a static stream rate for the smart meters to provide a baseline for comparison with our adaptive stream rate. To make the experiments more feasible, we perform runs at a 3x time compression where by the 48 hour (288 minute) smart meter data is scaled to a 96 minute wall clock period for the experiment, with an equivalent compression in the event latencies (and expansion of event rates). The power usage data values, however, are retained. Figure 3 plots the cumulative static event rate across all 5 smart meters on the primary Y axis (blue line) as the experiment time varies along the X axis. The secondary Y axis shows the cumulative power usage for all 5 consumers (red line) that is calculated by the SPADE program over the real data timeline of 48 hours shown on the secondary X axis. We make two observations from this plot. One, that the network bandwidth consumed by the events is proportional to the area under the event rate curve (blue line). Two, the observed energy usages (red line) shows high temporal fidelity since we use a relatively high event rate of 25 events per minute (cumulative) across 5 smart meters. So while the area under the blue line is large when using a high static stream rate, this comes with an increased accuracy and lower latency for calculating cumulative energy usage. 5.2 Throttled Rate Stream Processing In comparison, we run the stream processing pipeline with adaptive rate control according to the logic and implementation proposed in Section 4. Similar to the static stream

5 Cumulative Stream Rate(events/min) Real Time(hrs) Cumulative Stream Rate(events/min) Cumulative Power Consumed(KWh) Experiment Time(mins) Cumulative Power Consumed(KWh) Cumulative Stream Rate(events/min) Real Time(hrs) Power Consumed(KWh) Stream Rate(events/min) Experiment Time(mins) Cumulative Power Consumed(KWh) Figure 3: Cumulative static stream rate and observed energy use by the stream processing system for 5 virtual smart meters over a 48 hour real time (96 min compressed experimental time). Figure 4: Cumulative adaptive stream rates and observed energy use by the stream processing system for 5 virtual smart meters over a 48 hour real time (96 min compressed experimental time). rate experiment above, we use a compressed time scale for performing the runs. Figure 4 shows the cumulative adaptive stream rates across all 5 smart meters on the primary Y axis (blue line) as the experiment time varies along the primary X axis. The secondary Y axis shows the cumulative power usage for all 5 consumers (red line) that is calculated by the final aggregate operator in the SPADE pipeline over the real data duration of 48 hours shown on the secondary X axis. We observe from the plot that the adaptive stream rate varies between 2 events/min to 5 events/min, reflecting the upper and lower boundaries confiured for the rate control. The adaptive stream rate plot (blue line) closely trails the calculated power usage plot (red line) as the observed power usage increases getting closer to available power capacity, the stream rate increases, and vice versa. We see that there is a lag in the rate adaptation, seen as a shift in the relative peaks of the power usage and adaptive stream rate curves. This is a consequence using a tumbling window for power usage aggregation rather than a sliding window. As a result, the rate adaptation is propagated only at the end of the tumbling window duration with a corresponding lag. When we compare the area under the event rate curves (blue lines) for Figures?? and 4, we see that the area under the unthrottled curve is almost double that of the throttled curve (32 vs. 163 gridline blocks ). This shows that we are able to achieve a 5% reduction in network bandwidth usage relative to this particular static stream rate. The corresponding loss in power usage calculation accuracy for the throttled case is seen in the three spikes at 26min, 35min and 71min where the adaptive pipeline overestimates the cumulative power used, and at time points 7min, 11min, 49min and 58min where it marginally underestimates the power usage. 6. CONCLUSION AND FUTURE WORK We have described our implementation of a stream processing pipeline for latency sensitive smart grid applications using a private Cloud platform. Our initial evaluation of adaptive rate control reveals its benefits for conserving Cloud resource usage (and hence costs) while satisfying the smart grid application policy requirements. As future work, we plan to evaluate the scalability of the stream processing system with increasing number of VM instances, with the eventual goal of scaling up to 1.4 million smart meters that we expect in the Los Angeles smart grid. These experiments will measure the throughput of the stream processing system and its latency in processing each tuple as the stream rates adapt. This will help construct more advanced rate control algorithms for such cyberphysical systems that incorporate both cyber-infrastucture and physical-infrastructure metrics, and be increasingly relevant for applications such as traffic and weather analysis. For example, the adaptive algorithms will include factors such as the compute resource availability (# of VM instances), the througput of the stream processing system in meeting minimal latency requirements, and cost metrics that trade-off the cost for additional Cloud computation and bandwidth for accurate detection against the cost for bringing additional power capacity online. 7. ACKNOWLEDGMENTS This project is supported by the Department of Energy and the Los Angeles Department of Water and Power. The authors would like to thank Anand Ranganathan from IBM T. J. Watson Resesarch Lab for disussions and assistance with IBM InfoSphere Streams. 8. REFERENCES [1] Ibm infosphere streams: Programming model and language reference, version Technical report, International Business Machines Corporation, 21. [2] D. J. Abadi, Y. Ahmad, M. Balazinska, U. Çetintemel, M. Cherniack, J.-H. Hwang, W. Lindner, A. Maskey, A. Rasin, E. Ryvkina, N. Tatbul, Y. Xing, and S. B. Zdonik. The design of the borealis stream processing engine. In CIDR, pages , 25. [3] D. J. Abadi, D. Carney, U. Çetintemel, M. Cherniack, C. Convey, S. Lee, M. Stonebraker, N. Tatbul, and S. Zdonik. Aurora: a new model and architecture for data stream management. The VLDB Journal, 12:12 139, August 23. [4] A. Arasu, S. Babu, and J. Widom. The CQL continuous query language: semantic foundations and query execution. The VLDB Journal, 15: , June 26.

6 [5] A. M. Ayad and J. F. Naughton. Static optimization of conjunctive queries with sliding windows over infinite streams. In International conference on Management of data (SIGMOD), 24. [6] C. Ballard, D. M. Farrell, M. Lee, P. D. Stone, S. Thibault, and S. Tucker. IBM InfoSphere Streams Harnessing Data in Motion. Technical report, International Business Machines Corporation, September 21. [7] A. Biem, E. Bouillet, H. Feng, A. Ranganathan, A. Riabov, O. Verscheure, H. Koutsopoulos, and C. Moran. IBM InfoSphere Streams for scalable, real-time intelligent transportation services. In Proceedings of the 21 international conference on Management of data, SIGMOD 1, pages , New York, NY, USA, 21. ACM. [8] S. Chandrasekaran, O. Cooper, A. Deshpande, M. J. Franklin, J. M. Hellerstein, W. Hong, S. Krishnamurthy, S. Madden, V. Raman, F. Reiss, and M. A. Shah. TelegraphCQ: Continuous dataflow processing for an uncertain world. In CIDR, 23. [9] E. A. Feinberg and D. Genethliou. Applied Mathematics for Restructured Electric Power Systems, chapter Chapter 12: Load Forecasting, pages Springer US, 25. [1] B. Gedik, K.-L. Wu, P. S. Yu, and L. Liu. A load shedding framework and optimizations for m-way windowed stream joins. In ICDE, pages , 27. [11] V. Jacobson. Congestion avoidance and control. SIGCOMM Comput. Commun. Rev., 18: , August [12] S. Karandikar, S. Kalyanaraman, P. Bagal, and B. Packer. Tcp rate control. SIGCOMM Comput. Commun. Rev., 3:45 58, January 2. [13] C. S. Liew, M. P. Atkinson, J. I. van Hemert, and L. Han. Towards optimising distributed data streaming graphs using parallel streams. In Workshop on Data Intensive Distributed Computing (DIDC), pages , 21. [14] Y. Simmhan, S. Aman, B. Cao, M. Giakkoupis, A. Kumbhare, Q. Zhou, D. Paul, C. Fern, A. Sharma, and V. Prasanna. An informatics approach to demand response optimization in smart grids. Technical report, Computer Science Department, University of Southern California, 211. [15] Y. Simmhan, M. Giakkoupis, B. Cao, and V. K. Prasanna. On using cloud platforms in a software architecture for smart energy grids. In International Conference on Cloud Computing. IEEE, 21. [16] U. Srivastava and J. Widom. Memory-limited execution of windowed stream joins. In Proceedings of the Thirtieth international conference on Very large data bases - Volume 3, VLDB 4, pages VLDB Endowment, 24. [17] D. Zinn, Y. Simmhan, M. Giakkoupis, Q. Hart, T. McPhillips, B. Ludäscher, and V. K. Prasanna. Towards reliable, performant workflows for streaming-applications on cloud platforms. In International Symposium on Cluster, Cloud and Grid Computing (CCGrid), 211.