Smart Grid Reference Architecture

Transcription

1 Smart Grid Reference Architecture Volume 1 Using Information and Communication Services to Support a Smarter Grid Smart Grid realization is a utility s journey through a series of architectures. It starts with the predominant business silo model with point to point interfaces, to one best described as a systems of systems integrated by shared services and infrastructure. SCE Cisco IBM SGRA Team July 14, 2011

2 NOTES ii

3 Contents 1. Executive Summary Introduction...2 The Smart Grid Architectural Challenge Smart Grid Architecture...3 Smart Grid Architectural Goals and Principles...3 From a Siloed Architecture to a Layered Services Architecture Smart Grid Domains and Cross Domain Foundational Services Smart Grid Reference Architecture Views Application Services Analytics Services Data Services Control Services Security Services Communications Services Management Summary Appendix A. System of Systems Design Patterns... Appendix 1 Appendix B. Services Classes Concepts... Appendix 11 Applications Services... Appendix 11 Analytic Services... Appendix 19 Data Services... Appendix 29 Control Services... Appendix 33 Security Services... Appendix 39 Communication Services... Appendix 45 Management Services... Appendix 60 Structural Model Framework Template... Appendix 65 Appendix C. Smart Grid Conceptual Architecture Project (SCAP)... Appendix 67 Business Requirements... Appendix 67 Smart Grid Business Services... Appendix 76 iii

4 Smart Grid Cross Domain Foundational Services... Appendix 86 Appendix D. Roadmap & Maturity Model... Appendix 103 Policy Timeline... Appendix 103 Pursuing the Smart Grid Vision... Appendix 103 Maturity Model... Appendix 107 Appendix E. Glossary of Terms... Appendix 109 Appendix F. Bibliography... Appendix 115 Tables Table 1 Typical Application Specifications Table 2 Applicable Application Standards Table 3 Application Technology Recommendations Table 4 Typical Analytics Specifications Table 5 Recommended Analytics Standards Table 6 Recommended Analytics Technology Table 7 Typical Data Services Specifications Table 8 Data Standards and Technology Recommendations Table 9 Typical Control Specifications Table 10 Recommended Control Standards and Technology Table 11 Advanced Control Elements and Technologies Table 12 Security Standards and Technology Recommendations Table 13 Typical Communications Specifications Table 14 Recommended Communications Standards and Technology Table 15 Recommended Management Standards and Technology Table 16 Analytics Capability Maturity Model... Appendix 29 Table 17 Market Domain Business Services... Appendix 77 Table 18 Operations Domain Business Services... Appendix 78 Table 19 Service Provider Domain Business Services... Appendix 80 Table 20 Generation Domain Business Services... Appendix 81 Table 21 Transmission Domain Business Services... Appendix 82 Table 22 Distribution Domain Business Services... Appendix 84 Table 23 Customer Domain Business Services... Appendix 86 Table 24 Security Services... Appendix 87 Table 25 Communications Services... Appendix 89 Table 26 Data Management Services... Appendix 95 Table 27 Glossary... Appendix 109 iv

5 Figures Figure 1 GWAC Interoperability Context Setting Framework Diagram (GWAC Stack)... viii Figure 2 NIST Smart Grid Framework ix Figure 3 SCAP Requirements by NIST Domain... ix Figure 4 Silo Architecture for EMS/SCADA and Metering/Billing Functions...7 Figure 5 Enterprise Service Bus Architecture...8 Figure 6 Converged Communication Architecture...9 Figure 7 System of Systems Architecture Based on Open Standard Services Figure 8 Smart Grid Conceptual Model Figure 9 Layered Services Tier View Figure 10 Application Services Logical Model Figure 11 Application Services Structural Model Figure 12 Analytics Logical Model Figure 13 Analytics Structural Model Figure 14 Data Services Logical Model Figure 15 Data Services Structural Model Figure 16 Control Services Logical Model Figure 17 Control Services Structural Model Figure 18 Security Logical Model Figure 19 Security Structural Model Figure 20 Communications Services Logical Model Figure 21 Communications Services Structural Model Figure 22 Management Framework Figure 23 Management Logical Model Figure 24 Management Services Structural Model Figure 25 Silo Architecture... Appendix 4 Figure 26 ESB Architecture... Appendix 5 Figure 27 Adapter Architecture... Appendix 6 Figure 28 Service Centric Architecture... Appendix 9 Figure 29 Dis aggregation of a Monolithic System... Appendix 12 Figure 30 Functional Services Organization... Appendix 13 Figure 31 Network Operations planning and optimization... Appendix 14 Figure 32 Smart Grid Analytics Taxonomy... Appendix 20 Figure 33 Analytics Latency Hierarchy... Appendix 26 Figure 34 EIM Framework... Appendix 32 Figure 35 Multi Controller / Multi Objective Design Patterns (van Breeman, 2001)... Appendix 36 Figure 36 Control Center Functional Architecture (IEC, 2005)... Appendix 37 v

6 Figure 37 Hierarchical Control Design Pattern... Appendix 39 Figure 38 Security Threats Classification... Appendix 41 Figure 39 Conceptual and Services Layers... Appendix 45 Figure 40 Communication Services Layer Functions... Appendix 50 Figure 41 Transport Services Consideration Process... Appendix 51 Figure 42 Conceptual Reference Diagram for Smart Grid Information Networks... Appendix 52 Figure 43 Management Layer Organization... Appendix 62 Figure 44 Smart Grid Management Layers... Appendix 64 Figure 45 Structural Model Template... Appendix 66 Figure 46 Communication Services Layer Functions... Appendix 89 Figure 47 California Smart Grid Policy Timeline... Appendix 103 Figure 48 Smart Grid Project Portfolios as a Function of Maturity... Appendix 104 Figure 49 Grid 1.0 Evolution to Grid Appendix 106 vi

7 Smart Grid Reference Architecture Using Information and Communication Services to Support a Smarter Grid About this document This Smart Grid Reference Architecture document is designed to address the challenges, concerns and questions facing smart grid architects implementing smart grid solutions for their utility. As with any reference architecture, it aims to provide a foundation for utilities in the development of their particular smart grid architectures and to serve as a guide for implementing specific features designed to make their electric grid smarter. The architects using this document most likely work within utility organizations in the areas of operations, generation, transmission and distribution, customer services, information technology, and research and development. Reference Architecture Wikipedia definition: A reference architecture provides a proven template solution for an architecture for a particular domain. A proven architecture is problematic due to the scale and immaturity of the smart grid domain. The reader should judge the viability of this document based upon the considerable real world experience of the SCE Cisco IBM SGRA Team. Additional audiences may include: Utility executives needing clarification on developing a coherent investment roadmap to request and secure funding to achieve their enterprise s vision for a smarter grid. Utilities trying to transition from the specialized, targeted architectures developed for individual business units toward a more seamless, transparent architecture to be used across all business units. This architecture is to meet the requirements for optimal smart grid benefits while managing the complexities inevitably introduced by the requirements for a smarter grid. Standards organizations (SDOs, SSOs) and policy makers needing to clearly convey a smart grid vision with enough detail to be actionable by utilities, regardless of size. As utilities transition to the system of systems architectural model a number of challenges and issues will arise requiring assistance from SDOs, SSOs and state/federal policy making bodies. Vendors providing equipment and services to electric utilities, especially those companies whose products become more widely used. vii

8 Foundation Frameworks A number of informative smart grid documents, white papers, and frameworks are available on the Internet. The following are especially relevant to this reference architecture and worthy of study: A Systems View of the Modern Grid by the National Engineering Technology Laboratory (NETL, 2007) this white paper inspired many of the concepts expanded upon in subsequent publications. It identifies five primary interdependent elements desirable for a modern grid and defines those concepts including the seven smart grid principal characteristics listed in section 2. The GridWise Interoperability Context Setting Framework by the GridWise Architecture Council (GWAC, 2008) a work that focuses on the interface between two or more interacting parties and provides a framework for discussing the integration of collaborative processes and independent automation components. It identifies eight interoperability categories defined by the framework and 10 crosscutting issues applicable in every category. The categories are tagged as technical, informational or organizational, and are stacked according to their increasing level of abstraction [Figure 1]. Figure 1 GWAC Interoperability Context Setting Framework Diagram (GWAC Stack) While this structure is not used to organize the reference architecture described in this document, it provides the basis for the structural models presented in the reference architectural views in section 5. The NIST Framework and Roadmap for Smart Grid Interoperability Standards by the National Institute for Standards and Technology (NIST, 2010) a conceptual model created to support the planning and organization of the interconnected networks expected in the Smart Grid. The NIST approach divided the Smart Grid into the seven domains shown in Figure 2. viii

9 Figure 2 NIST Smart Grid Framework 1.0 In 2010 NIST commissioned the Smart Grid Interoperability Panel s (SGIP) Smart Grid Architecture Committee (SGAC) to lead its Smart Grid Conceptual Architecture Project (SCAP). The SCAP working group took top down and bottom up approaches to establish a foundation for the development of smart grid business requirements. The SGAC s top down approach involved a review of all major federal energy legislation signed into law since 2000, more than 9,500 pages. Review of these documents resulted in the identification of 400 goals. The bottom up efforts reviewed more than 600 use cases written by a variety of industry organizations, including more than 20 systems requirement documents produced by industry working groups. The bottom up review produced more than 8000 business requirements for the Smart Grid. Figure 3 illustrates the distribution of the smart grid business requirements by domain. (SGIP SGAC) Figure 3 SCAP Requirements by NIST Domain ix

10 This preliminary work created 400 families of requirements covering the seven NIST domains. These high level business requirements (available on the NIST website) encompass the entire Smart Grid from market to customer from bulk generation to distribution. They form the basis of what the Smart Grid requires from a business standpoint and to meet federal government s energy goals. 0 provides a list of the SCAP Business Requirements and Services. Smart Grid Reference Architecture by the P2030 Working Group (IEEE) this document, expected in the third quarter of 2011 (draft published in March 2011), will provide guidelines for smart grid interoperability, including a discussion of best practices. The P2030 guide will also provide a knowledge base addressing terminology, characteristics, functional performance and evaluation criteria, and the application of engineering principles for grid interoperability with end use applications and loads. At first the SCE Cisco IBM reference architecture team expressed concern on the potential overlap between the P2030 guide and this document, but after a thorough comparison of the two drafts, the consensus is that this reference architecture merely complements the P2030 guide. Whereas the P2030 Guide documents a catalog of interfaces, this document addresses the architecture and foundational services necessary to develop Smart Grid systems, interfaces, and processes. The alert reader will notice this document is entitled Volume 1. It includes a set of views on smart grid architecture largely written from the point of view of system integration. It is expected to be a useful companion to IEEE P2030. As IEEE P2030 catalogues smart grid system interfaces, the SGRA Volume 1 catalogues smart grid system services. Similarly, the NIST SGCA lists elemental smart grid functions (unit operations); the NIST Framework and Roadmap for Smart Grid Interoperability Standards identifies relevant smart grid interface standards; and the GWAC Interoperability Context Setting Framework, as a companion to the NIST Framework, provides rigor around the concept of interoperability. Each contains more than described above, as their authors will attest, but these characterizations are helpful in framing each document in the context of the entire set. Each one of these documents is a valuable contribution to the field of smart grid architecture, and smart grid architects are strongly urged to study each of them. However, after this document was completed, the team sensed a set of architectural views was needed to tie these contributions into a unified whole that would make the SGRA actionable in both the near term and throughout the general smart grid utility transformation journey. That is why this document was labeled Volume 1, to be followed shortly by Volume 2. The Volume 2 document will synthesize the various expositions on interfaces, standards, and services, as well as measurement, data management, control, communications, and security from the above mentioned Smart Grid documents into a practical consolidated architectural guide representing how best to implement smart grids. Volume 2 will also tie together energy delivery elements of particular importance as the smart grid scales up, resulting in interactions of increasing complexity with simultaneous impact to once independent utility tiers. x

11 1. Executive Summary The Smart Grid Reference Architecture is intended as a template for Smart Grid architects to follow as they build Smart Grid Information and Communications Technologies (ICT) architecture for their utility, regardless of the architect s specialty (transmission, distribution, metering, IT, communications). The goal of this document is to accelerate the construction of a utility s Smart Grid architecture and implementation strategy by leveraging the reference architecture constructs contained therein. This reference architecture recognizes the need to logically transition from the existing, largely ad hoc nature of the typical utility grid architecture to one based on open interoperability standards, and designed to manage complex solutions while facilitating the integration of emergent smart grid capabilities. It explores three migrations across four transition states and takes into consideration the many years required to develop sound recommendations for smart grid architecture. Security is an integral element of the grid ICT architecture and a multi-layered approach is advocated, including both physical and ICT-based security mechanisms. Layering smart grid services using the proposed system-of-systems architecture should minimize the stranded costs utilities invest in one-off solutions. A layered service-centric architecture also minimizes the expense, configuration headaches, and management complexity a utility faces pursuing a point-to-point interoperability architecture. Another aspect emphasized in this reference architecture that could lead to considerable cost and time savings for utilities are the implementation of data services and data management. Greater access to and use of data is critical to the realization of a grid s ability to accommodate new capabilities while improving security, reliability and quality. A significant portion of this reference architecture is dedicated to discussing common foundational services and the corresponding architectural views, important for maintaining the high levels of performance and efficiency required by a modern grid ICT. Recognizing that some services are best centralized while others must reside primarily within grid-state aware edge components, this discussion also explores the best central-versus-edge mix for deploying various domain components. This Smart Grid Reference Architecture was produced by a team of architects from Southern California Edison (SCE), Cisco Systems, and IBM. Its development spanned a period of nine months (July 2010 through March 2011) and involved a number of face-to-face team workshops and web-based meetings. An external review by EnerNex added additional insight and content. Executive Summary 1

12 2. Introduction The Smart Grid Architectural Challenge The January 2007 white paper, A Systems View of the Modern Grid, by the U. S. Department of Energy s (DOE) National Energy Technology Laboratory (NETL) identifies seven smart grid characteristics: self-healing able to motivate and engage the customer resistant to attacks provides power quality suitable for 21st century needs accommodates all generation and storage options enables markets optimizes assets and operates efficiently There are a number of Smart Grid definitions, but no matter which is used, there is little dispute over the vastness of program scopes faced by smart grid architects. Incorporating information and communications technologies into what the National Academy of Engineering calls "the greatest engineering achievement of the last century" (NAE, 2011) will be the one of the most complex human endeavors ever undertaken. This robust engineering achievement must be extended to support enhanced situational awareness (via synchrophasors), industrial-scale energy storage, distributed (dispersed) energy resources, improved field worker effectiveness (via wireless communications and automated asset management), remedial action scheme expansion, substation automation, volt/var optimization, fine-grained demand response, distribution automation, improved power quality, power disturbance self-healing, micro-grids, personal and fleet electric vehicles, automated metering, premises area networks, enhanced customer energy management, power grid congestion-management, advanced integrated command and control, transmission/distribution smart sensor deployment, very low-latency protection communications, and not yet identified technologies that are certain to emerge as the Smart Grid matures. All this must be accomplished as the world's largest machine, the electric grid, continues to operate unabated, while maintaining present or improving reliability. In addition, embedded security measures must be built concurrently to marginalize the possibility of successful cyber and physical attacks from an ever-growing number of threats, and prevent unauthorized use of customer personal data and energy usage information. Finally, there are demands from regulators, political leaders, and social groups to make the grid smarter quickly, while addressing environmental objectives and keeping electric rates low. Introduction 2

13 3. Smart Grid Architecture The Smart Grid architectural challenge is a daunting one. This is especially true for those within the electric utility industry known for their conservative approach toward incorporation of ICT-based systems to help run and manage the electric grid. Most automated grid systems in use today were built to address narrowly targeted requirement sets. As a result, a typical utility has a plethora of purchased and homegrown systems stitched together over the last three decades with point-to-point interfaces. This approach is unsustainable for a utility to efficiently and effectively implement smart grid capabilities over the next two decades. This document presents a different approach to utility system design and integration, using newer paradigms to deal with complex and legacy system integration. Smart Grid Architectural Goals and Principles The next two decades will see the Old Grid evolve into a Smart Grid as legacy grid infrastructure is merged with the latest ICT. This will put extraordinary demands on the ICT architecture; therefore, high-level goals and principles are needed to guide the smart grid architects tasked with developing any aspect of an organization s grid architecture. Additionally, a highly flexible, adaptive Enterprise Smart Grid architecture is critical for this transition to be successful. The architecture must support existing ICT infrastructure operations and be able to keep infrastructure complexity manageable as new smart grid capabilities are added. How a utility defines its smart grid architecture will vary according to their organizations particular needs. Some possible goals are: Facilitate bridging new and emerging information and communications technology to legacy architecture over extended time periods (technology roadmap). Manage the increasing complexity of ICT needed to support smart grid implementation. Align technology usage with the utility s smart grid strategic objectives. Provide guidance on how packaged solutions can support the smart grid architectural vision. Facilitate the communication of the utility s smart grid strategy and plans across the enterprise Help sell the utility s smart grid vision to business unit leadership, IT management, suppliers, regulatory agencies, contractors, etc. Help stakeholders (application developers, IT managers, and end users) plan, budget, implement and use smart grid information and communication technologies. Make the utility smart grid architecture easily accessible and transparent. Support the interactions of processes, tools, technology and people to achieve business ICT goals. Once a utility has defined its smart grid architecture goals it should develop a set of written principles to provide high-level direction during architecture development. Some principles a utility may consider important are: Smart Grid Architecture 3

14 1. Design for simplification by exploiting strategic assets Motivations: Reduce complexity and cost Let the enterprise s employees focus on customer, not on internal processes Implications: Establish single architecture control point for requests to expand the portfolio Finish what is started to enable legacy sunsets Reduce complexity and low value work steering investment & effort to high value activities 2. Reuse process, data, and ICT assets whenever appropriate Motivations: Accelerate business capability delivery Reduce cost Increase enterprise consistent use of best practice designs Increase responsiveness to regulatory requirements Implications: Must identify, adopt and reuse process, data and ICT assets for enterprise wide use Must promote business modularity from strategy through deployment Must direct funding to develop and adopt best practice assets 3. Use off-the-shelf rather than build solutions Motivations: Reduce cost and time to market Improve time to market Implications: Understand what off-the-shelf solutions exist and what processes they support Re-engineer the business process or model to use off-the-shelf products and services 4. Use Business Process Driven development to move toward a process-centric organization Motivations: ICT development will be driven by Enterprise Business Process Process will drive continual improvement across the enterprise Use business priorities to drive technology adoption Continually improve ICT effectiveness and efficiency Facilitate cross enterprise integration Implications: Align enterprise processes with enterprise strategy Close business-it partnership with IT engaged early in the solution development process Ongoing maintenance/management of enterprise business processes 5. Base architecture on total cost of ownership (TCO) Motivations: Provide a common approach for dealing with applications Optimize operational manageability while making key business and technology decisions Minimize cost of complexity while making these decisions Free up resources from low value to higher business value application through optimization Smart Grid Architecture 4

15 Implications: Elimination of low value applications at every possible opportunity Avoid introduction of new low-value applications for short lived business requirements TCO based approach to be used for major technology decisions 6. Ensure business decisions are based on information from appropriate trusted data sources Motivations: Achieve highest degree of integrity and validity for business decisions Minimize IT costs for managing and maintaining data Enhance ease of doing business by eliminating manual data integration, normalization, etc Implications: Practitioners more likely to know what trusted sources exist, and which ones to use Solution teams realize reduced cost benefits using appropriate trusted sources 7. Develop data models and a data dictionary for the entire portfolio Motivations: Improve operational excellence Reduce unnecessary transformations of data and related re-work Enable meta data sharing for exchange and integration purposes Improve future system design and programming projects Improved documentation and control mechanisms Implications: Project teams bound by data model/dictionary governance processes Close collaboration between business and IT stakeholders Easy access to data model/dictionary given to designers and programmers 8. Master data element created from one trusted source Motivations: Increased data integrity and reliability Cost reduction for managing information and data quality Implications: Consistently invest in, and comply with, the trusted sources architecture Processes engineered to maintain consistent master data management and consumption 9. Only store copies of data within approved trusted sources Motivations: Achieve highest degree of integrity and validity for business decisions Minimize ICT costs for managing and maintaining data Enhance ease of doing business by eliminating manual data integration, normalization, etc Implications: Practitioners more likely to know what trusted sources exist, and which ones to use Solution teams realize reduced cost benefits using appropriate trusted sources Data currency is in line with business expectations. 10. Implement data quality plans for all business solutions Motivations: Maximize data integrity and validity for business operations and decision making Avoid operational disruption due to data errors Reduce cost and complexity Smart Grid Architecture 5

16 Implications: Real cost implications associated with avoidance of data quality plans Data quality easier to implement and sustain 11. Design solutions that provide measurable business performance and value Motivations: Maximize ICT ROI Focus design and development teams on business success goals Validate the ICT governance model Achieve measurable performance gains Implications: Link strategy to business case and customer requirements to metrics Enable collection, analysis and reporting of metrics, actual to plan Design generation of metric data into solutions Establish process-level Key Performance Indicators 12. Enable applications for reuse and portability as services Motivations: Ability to quickly add, modify, remove or replace service functions Reduction of integration expense and partner boarding costs Facilitate the reuse of strategic applications and business functions Enable application and function relocation for process or cost effectiveness Improve support of acquisitions and divestitures Reduce point-to-point integration solutions Implications: High return investment hotspots emphasized Centralized support for design enablement Enterprise group assigned to enhance competency on standards and references Portability design based upon being agnostic on platform, location and virtualization Adoption of service modeling methodology 13. Design and test solutions to satisfy non-functional requirements Motivations: Stabile platforms supporting the enterprise business Ensured Return On Investment Implications: Need to understand the target audience and expected workload of new solutions Infrastructure impact of new solutions known early in the design cycle Infrastructure constraints considered in analysis of growth markets 14. Design solutions to make use of infrastructure common services Motivations: Reduction of infrastructure duplication and cost Less complexity for the end user Improvement of system interoperability Implications: Re-use of existing common services, reduction of new service construction Need for common services to be robust and user-friendly Smart Grid Architecture 6

17 From a Siloed Architecture to a Layered Services Architecture Architectural evolution can be defined as how a typical utility implements its smart grid transformation in stages. A white paper discussing this evolutionary process in depth can be found in Appendix A. For the purposes of this paper the process has been divided into four stages. Stage One: The predominate architecture of grid systems is a collection of silos. Figure 4 is an example of silo architecture for EMS/SCADA and metering/billing functions. Different functions (SCADA, EMS, DMS, OMS, Billing, and Metering) use disparate information with minimal interaction. Figure 4 - Silo Architecture for EMS/SCADA and Metering/Billing Functions Smart Grid Architecture 7

18 The silo architecture worked well for utilities for decades - each silo served the needs of a business unit, each having very different needs. Utilities operated efficiently with little integration across silos. The silo solution, however, is not sustainable to support the Smart Grid. The number of stakeholders needing real-time data from every silo cannot be support long-term point-to-point interfaces. Stage Two: Some utilities have taken the next step in smart grid evolution the integration of the back office and applications via a common service bus, such as the enterprise service bus architecture shown in Figure 5. This is often a difficult and costly process. In addition, service-bus integration requires enforcement of standards on data models and ICT services. Without the enforcement of standards, the service bus is simply a shared communication device with little resolution of silo weaknesses. Figure 5 - Enterprise Service Bus Architecture Smart Grid Architecture 8

19 Stage Three: The next step towards a unified, shared infrastructure is realized by moving away from a series of single-purpose networks to a converged communication infrastructure [Figure 6]. This shared infrastructure enables as-needed data transfer from end points to consuming applications in accordance with stated requirements (quality of service, criticality, bandwidth, latency, etc.). Figure 6 - Converged Communication Architecture Smart Grid Architecture 9

20 Stage Four: The ultimate Smart Grid ICT architecture is converging on layered, open standard services architecture. It provides capabilities across functional and organizational boundaries; from a data/control center to edge devices and data consumers (applications and end users). Figure 7 System-of-Systems Architecture Based on Open Standard Services Smart Grid Architecture 10

21 The system-of-system architecture shown in Figure 7 supports the following capabilities: Components can be added, replaced or modified without affecting the remainder of the system Components are distributable (can run on arbitrary servers) Components communicate with each other by messages or service invocations Component interfaces are defined using standard metadata Component interfaces are discoverable by application developers One component can replace another with the same interface Services can be used multiple times by disparate applications or the same application. Achieving such smart grid capabilities requires a great amount of interaction among systems. For instance, most utilities rely on customers to report an outage; in the future, the advanced metering infrastructure (AMI) will interact with the outage management system (OMS) to predict and confirm outages. Once OMS confirms an outage, the distribution management system (DMS) calculates the necessary switching steps needed to isolate the fault area and restore service in a timely manner. The field workforce will directly interact with the OMS and DMS, responding to automatically issued work orders and providing a detailed estimate of restoration time. Meanwhile, the customer can be notified of the outage status in real-time via user-defined means (cell phone, web, etc.). As the capabilities of the communication infrastructure advances, additional intelligence will be deployed closer to the customers premises, allowing pro-active decisions to be made locally to avoid or minimize outages, while informing the utility systems and operators of the locally implemented actions for potential adjustment and optimization of energy resources. The Smart Grid s capabilities will need to be facilitated by an architecture that enables the connected devices and systems to securely interact and exchange information and control. Field devices and electrical equipment should not only publish data to help improve real-time monitoring of the electrical grid, they will need to subscribe to other devices' information as well, allowing the devices to respond to control signals and data requests issued by applications and systems responsible for grid monitoring and control. This dispersion of data across the grid poses a significant challenge to utility data management. The quality of the data is also a concern, requiring intelligent devices to be properly configured and maintained. Device configuration control is best performed by a common management tool tasked with component provisioning and de-provisioning. Even though the future communication infrastructure is expected to be more robust and feature high performance communication channels, the large amount of data, data sources and data consumers requires grid intelligence to have decentralized and centralized aspects: Decentralized embedded systems and applications will be responsible for analysis, filtering and taking particular actions based on the data provided by local field devices. Centralized systems will be responsible for coordinating the decentralized systems, ensuring the overall reliability and stability of network. Smart Grid Architecture 11

22 With applications and systems physically dispersed into the network to lower the cost of deployment and maintenance, each component of the Smart Grid system-of-systems will be required to satisfy four key principles of layered services architecture: 1. Individual components can be added, replaced or modified without impact to other systems. 2. Components are distributable, communicating by messages or service invocations. 3. Interfaces between components are discoverable and leverage standard metadata 4. Component services can be easily reused by different applications. While reusable and shared services reduce costs and minimize complexity, they also enable faster deployment of new applications. This will be crucial for the utility to efficiently adapt to evolving regulatory mandates. To transition from a Stage 1 silo architecture, or a Stage 2 partially integrated architecture to a Stage 3 or Stage 4 layered services architecture, the utility must define and plan for strategic investments in modernizing its infrastructure and systems. To be successful, each planned investment must be reviewed and assessed from an enterprise standpoint to ensure investments help the organization transition toward the future Smart Grid Architecture. Adoption of shared services, standards and a unified infrastructure need to be understood as intrinsic requirements for each planned investment. A transition plan some companies have adopted is to first move from a siloed architecture to middleware integration architecture. This is followed by a gradual migration to an open-standards based architecture, incrementally developing and adopting standards and common services. Each increment helps move the enterprise toward the vision of this Smart Grid Reference Architecture. The timing of the transitions is often documented by a smart grid roadmap. Appendix D presents one technique for constructing a roadmap. It also provides a brief discussion on the Smart Grid Maturity Model (SGMM) sponsored by Carnegie Mellon University. The smart grid architect should apply the system-of-systems architecture patterns described to first develop use cases and a comprehensive set of requirements. These can then be used to develop the shared services and target physical architectures needed to support the desired capabilities across the utility s smart grid. Smart Grid Architecture 12

23 4. Smart Grid Domains and Cross Domain Foundational Services Seven domains comprise the conceptual model described in NIST Framework and Roadmap for Smart Grid Interoperability Standards, Release 1.0 (NIST, 2010). A smart grid has inherent power flow and grid state complexity embedded primarily within this model s Operations domain. This SGRA recommends two additional domains should an architect wish to explicitly address these complexities. In addition, the SGRA supports the concept of foundation-level services used by multiple domains. The NIST smart grid conceptual model domains are: Customer: the functional needs within the customers premises, including the ability to generate, store, monitor and control the electricity usage of customers (both residential and commercial). Market: the functional and operational need for operators and participants in the electricity market. This is where efficient matching of energy production and consumption is performed. Service Provider: the functional needs of organizations offering or leveraging utility services. These include power producers, distributors, regulatory agencies, banks, credit bureaus, etc. Operations: managers of electricity movement, responsible for the smooth operation of the grid Bulk Generation: the needs of power generation entities producing more than 300 megawatts. Transmission: the applications and tools to deliver bulk electricity over long distances, such as a Regional Transmission Operator or Independent System Operator (RTO/ISO). Distribution: often considered the primary focus of smart grid changes, offers all the required functional services to electricity distributors to and from customers, as well as the services to manage distributed energy resources, including energy storage and plug-in electric vehicles. Supplementing the NIST-defined domains, the following are potential expansions to the architect s smart grid conceptual model: Balance required for the dispatch of distributed energy resources and demand response due to their roles in balancing supply vs. demand on the grid; increasingly data interaction between the utility, the consumer and the balance authority will be needed in the Smart Grid environment. Interchange the visibility onto grid state required to address increased power flow complexity, placing requirements on smart grid systems for data communications and management. Cross domain foundational services are those that two or more domains rely upon and therefore need to be interoperable across domains. For example, a system having one form of encryption in one domain and a different one in another will lead to problems when the two exchange information, thus causing additional work to correct the problem in the final implementation. This could be avoided by a single encryption method used by all systems as a cross domain foundational service. Cross domain services are broken down into six groups: Analytics, Data, Control, Security, Communication, and Management. Discussions on each service group can be found in Appendix B. Architects and others interested are encouraged to study this appendix for more detail. Smart Grid Domains and Cross Domain Foundational Services 13

24 5. Smart Grid Reference Architecture Views The comprehensive, end-to-end, high-level architecture needed to support the utility business domains and common enabling services discussed in Section 4, uses a model that assumes a service-oriented governance process exists supporting the intrinsic characteristics of layered services architecture. The model s seven business domains are each logical groupings of business capabilities providing related business functions while requiring similar skills and expertise. These match the domains identified by NIST in Special Publication 1108, NIST Framework and Roadmap for Smart Grid Interoperability Standards, Release 1.0 (NIST, 2010). These functional areas form the basis for defining the boundaries of ICT capabilities and systems, and provide a means to classify components and services. Common enabling services provide a variety of services for systems and subsystems to accomplish business functions. Governance provides a framework to define the relationships and processes used to direct and control grid activities, as well as the actions, authority and metrics used to realize business benefits while balancing risk versus reward. The left side of Figure 8 shows the seven NIST utility domains as distinct logical groupings of common capabilities. A utility may elect to combine some domains (i.e. distribution and transmission) and plan a single logical infrastructure to support the Smart Grid. In addition, utilities opting for interactions of Smart Grid elements with the Balance and Interchange domains will need to extend this model. Figure 8 - Smart Grid Conceptual Model The top of Figure 9 names five smart grid end-state architecture tiers (user/device, channel, presentation/edge, integration, services). In this tiered architecture, data flows from left to right (and Smart Grid Reference Architecture Views 14

25 back) through the various tiers, each with layered service groups or capabilities. The large box at the bottom of the figure (spanning the three right tiers) represents the cross domain foundational services discussed in section 4 above. Figure 9 - Layered Services Tier View Each services group discussed in the following subsections provides a: Logical architecture diagram Structural architecture diagram Table of typical architectural specifications for the service group Table of relevant standards for the service group Table of relevant technologies for the service group Smart Grid Reference Architecture Views 15

26 Application Services Smart grid applications services provide functionalities for the presentation tier, the services tier, and a mechanism to integrate various applications through the integration tier as depicted in Figure 9. Applications consume services to present secure, timely, relevant, and understandable information in response to a validated stakeholder request. Applications Services Logical Model The Application Services Logical Model [Figure 10] is made up of three views, (1) application component, (2) application services stack and (3) application synergy. Figure 10 - Application Services Logical Model The application component view depicts in block format the primary functional components used to build the architecture being described. This model follows the IEC and specifications, most Smart Grid Reference Architecture Views 16

27 notably the Interface Reference Model (IRM) categorization of functional components necessary for smart grid transformation within a utility. The application services stack view defines the services required by smart grid applications. Due to the number and broad range of services consumed by applications, the list is best understood by grouping the services into broad service types. The stack services are therefore divided into four categories: Foundation services many key technical foundational services are required by the application architecture tier. Used throughout the architecture, these services are leveraged to varying degrees by other technical and functional services. For example, the enterprise service bus is one of the foundation services used throughout the architecture. Core services the primary service stack for the application architecture are service governance through a services registry and repository, a message gateway, web Integration services, a state machine, and a business process execution language (BPEL) engine and workflow. These core services provide baseline capability in the application tier. Integration services services such as enterprise services bus, rule engine, the Common Information Model (CIM) binding, service orchestration, security agents, and policy cache provide the integration needed between the functional services and other architectural tiers. Support services the application architecture utilizes common support services for security, communication, data management, smart grid management, analytics, and control. Support services underpin application architecture and are re-used extensively in each services group. The application synergy view [Figure 10, upper left] depicts some of the key issues faced by an enterprise architect while architecting the Smart Grid application architecture. The data generated by a device or end point may serve multiple consumers in a format specific to each consumer s needs. The application architect can use the model to identify re-use opportunities for data and integration services as smart grid application functionalities are built. Potential synergy prospects are those with similar paths through the various tiers. The tiers and how they relate to the Smart Grid are: Source Tier: Different data generation sources (sensors, feeder devices, etc.) in the Smart Grid produce the four different data classes (telemetry, waveforms, events, user data) shown in the application synergy model portion of Figure 10. This tier is comprised of the various parameters used to differentiate sources. There are important architectural considerations when selecting the (1) components to process the data, (2) communication infrastructure to transport the data, and (3) security services needed to protect the data. The frequency at which a source generates data is dependent on the purpose of the data. For example, if data is expected to be used for calculating grid state, the data generation frequency will be high. If the data represents consumer usage data, the frequency is much less perhaps once every several minutes. Thus the purpose of each datum and source needs to be analyzed, and only then should relevant architectural components be selected to implement the requirements. Data Classes: There are four high-level data class types supported in Smart Grid architecture. Waveforms data represent values of electrical measurements such as current, voltage, phase, etc. Smart Grid Reference Architecture Views 17

28 Telemetry data represent readings generated by field sensors. Event messages can be generated by field crew, distributed energy resources (DER), automated demand response (ADR), etc. Usage data is generated by consumer smart meters. These data classes are presented to ensure the application architecture supports all data classes, and to help the architect choose the best integration pattern. Application Services: This tier represents how different components must interact to provide desired business functionality. Different data classes connect to the Smart Grid infrastructure through the enterprise services bus (ESB). This provides multiple capabilities, such as handling request/response, publish/subscribe (event) processing, event interception, gateway communication, business rules, CIM binding, policy caching, etc. The core ESB features provided (protocol transformation, routing, etc.) allows the service bus to act as a mediator between the data source and consumer. Web integration services can consume usage data and provide data to customer service hosted by a third party or utility partner. Similarly, third party consumers may connect to smart grid applications through gateway services. Applications often require workflow, CIM binding, access to an external rules engine, and service discovery for building functional capabilities. Refer to Appendix B for more detail on application services. Consumers/Processes: These are the high level business functions identified in the IEC IRM. These business functions are developed using application services to collect and process source data for end-user consumption. Edge presentation services may be used for some consumers while others are supported by application servers via the ESB. Functional components developed on application servers make use of rules engines, workflow, process services, BPEL engines, state machines, complex event processing, gateway services, etc. Web integration services are used to develop support services and user-generated content, including derivative applications from pre-existing sources (mashups). Application Services Structural Model The application services structural model [Figure 11] is provided to help the smart grid architect consider how to best deploy the various application services using a stylized representation of a typical utility network infrastructure. At the top of the model are elements needed to support external agencies (regulators, interchange and balancing authorities, etc.). At the bottom are the application services needed by premise devices (smart appliances, PEV chargers, building energy managers, etc.). In between are a dozen other tiers where application services may be located to support residing devices and functionality. A self-describing template used for this model is on the last page of Appendix B, Services Classes Concepts. Included are descriptions of all fourteen tiers used in the model. Smart Grid Reference Architecture Views 18

29 Figure 11 - Application Services Structural Model Applications Architecture Specifications Each smart grid application will have a list of specifications for the application architecture to support. This topic is, however, does not address every specific smart grid application. It is instead intended to be a discussion of common specifications for an abstracted collection of smart grid applications. Table 1 is therefore a list of general specifications with potential relevance to a smart grid application. The justification for each is provided to enhance the reader s understanding of each specification. Smart Grid Reference Architecture Views 19

30 The table does not include detailed business or technical requirements a specific application would fulfill. Each utility will build a detailed application requirements list as part of any smart grid project. Table 1 - Typical Application Specifications Specification Application access shall be tightly controlled. The application architecture shall use common services instead of building redundant services. The application infrastructure shall leverage virtualization at the data center. The application shall be deployable in a distributed fashion. Places in the grid for application deployment are illustrated in an hierarchical model. Application infrastructure must be rugged for applications that are deployed on substation, feeder, and premises area. The application shall use common auditing/logging capabilities to support comprehensive management architecture. Applications architecture shall promote open standards and avoid vendor lock-in. Whenever possible, management of systems shall be centrally and uniformly managed. Business rules (i.e., implementations of the policies and practices of a business organization) externalization shall be supported. Any commercial-off-the-shelf (COTS) application package used shall be minimally customized. A comprehensive design, build and run platform shall be used to support implementation of the Smart Grid application architecture. All information shall have defined authoritative sources (information stewards). A comprehensive architecture governance mechanism shall be used during application architecture development. As needed, common services shall be developed for communication, security, and data architecture as a precursor to the development of the application services architecture. (similar to second spec listed) Enterprise business service definitions shall be managed and enabled for automatic discovery. Justification Applications must be protected from unauthorized access. Common services such as authentication and authorization, communication, and management services, if reused properly can drastically cut costs. Reducing the consumption of key resources such as energy and personnel while improving the utilization of IT hardware and software can yield environmental as well as financial benefits. Each location where data is generated or information is consumed is a location candidate for the appropriate application. Communication network elements hosting additional software are also good candidates, since communications elements generally exist where data sources and consumers reside. By making use of such elements to host applications, it is possible to provide the benefits of distributed architecture while minimizing the number of devices to be managed and maintained. Zero-touch deployment is crucial. Event logging is necessary to support post mortem analyses of various kinds. With the volume of events generated by a Smart Grid, management of the event logs is a large issue; therefore, care must be taken for selecting consistent mechanisms across the enterprise. Proprietary applications are difficult to integrate and generally defeats system-of-systems primary design principles. Integration with the management services architecture is necessary to make distributed application architecture operations feasible. Decoupling and externalizing business rules can provide a number of advantages: variable logic can easily be changed, duplication of logic at multiple places can be avoided, and discovery of and fixing miscreant rules can be simplified. Minimum customization of COTS allows version upgrades and enhancements. Application development lifecycle management through integrated development platform enables disciplined application development. Data needs to be made available in a timely and accurate form, and must be captured and validated. Architecture governance enables fundamental aspects of service oriented architecture, which provides inputs vital to the development of the application architecture. This is vital to the development of the application architecture, since common services are significant providers of support functions. Sometimes a chicken or egg conundrum, however, for enterprises with an immature SOA infrastructure. Basic service repository mechanisms can reduce integration cost and help in business flexibility. Smart Grid Reference Architecture Views 20

31 Specification Enterprise class service: development of services used across the enterprise shall trump the development of similar or duplicative services provided solely to a particular organization. For low latency applications, implementation shall be located at or very near the data source(s) and the primary outputs destination(s), Software and hardware shall conform to defined standards that promote interoperability for data, applications, and technology. Unique version namespaces shall be used for service versioning. Modular design shall be used wherever possible (moving away from monolithic systems). Multiple channels supporting client application delivery shall be available, with an ability to tailor the delivery mechanism to the client. ICT shall plan, design, and construct for growth and expansion of services across the Smart Grid. The federated architecture shall promote reduced complexity and enable integration to the maximum extent possible through adoption of standards. The architecture shall be based on a design of services mirroring real-world business activities managed by the enterprise business processes. Justification Duplicate capability is expensive to build and maintain. It also poses challenges in maintaining data integrity. This is a direct consequence of the distributed nature of power grids and their control systems. The requirement avoids hops to and from remote processing centers. Embedded processing is used, but the actual application is downloadable. Standards help ensure consistency, thus improving the ability to manage systems, improve user satisfaction, protect existing IT investments, maximize return on investment and reduce costs. Standards for interoperability help ensure support from multiple vendors for an asset, and facilitate supply chain integration. Application programming interface versioning is a common problem in the design of any distributed system, and services are no exception, they must be versioned for correctness and manageability. Modular design provides flexibility in design and helps in scaling a solution. A solution's overall functionality can be decomposed into smaller functional building blocks and, ideally, perform only one conceptual function. Consumers, partners and field force want channel of service delivery to fit their individual preferences and circumstances. Having multichannel access protects against single point of access failure. Application architecture and design must plan for growth to provide business flexibility. Complex application systems with many data and transactional functions are difficult to manage and risky to change. Applications with tightly coupled modules risk creating a dependency failure. Service orientation delivers enterprise agility and boundary-less Information Flow. Architecture Standards and Technology Recommendations The following tables of architecture standards and technology recommendations are provided to the smart grid architect as references. Over time, however, innovations will inevitably diminish their completeness. The architect should always research the latest information on these topics, but these should provide a good starting point. Table 2 lists the most important 2011 standards, while Table 3 lists technology recommendations, with brief explanations of their purpose or relevance. Table 2 - Applicable Application Standards Standards Distributed Network Protocol (DNP3) IEC Purpose/Relevance/Comments Defines a set of communications protocols used between components in process automation systems. This protocol facilitates communications between various types of data acquisition and control equipment and plays a crucial role in SCADA systems. Describe the Inter-Control Center Protocol (ICCP) for data exchange between control centers. Smart Grid Reference Architecture Views 21

32 Standards COMTRADE IEC IEC 61968, IEC CIM GID Interface Services (GDA, HSDA, TSDA, GSE) IEC Common Information Model IEEE NRECA Multispeak Web Services Standards (SOAP, WSDL, WS-* specification) XML ZigBee Smart Energy Profile Purpose/Relevance/Comments Common Format for Transient Data Exchange. It is intended for use by digital-computer-based devices which generate or collect transient data from electric power systems. This standard facilitates exchange of the transient data for the purpose of simulation, testing, validation, or archival storage. For designing electrical substation automation. Describe the business functions, business sub-functions and abstract components for distribution management and set of applications for energy management system. Four services associated with IEC CIM for data interchange in four classes: generic data, high speed data, time series, and events Primary schema for data representation; should also be applied to analytics outputs, which in themselves are treated as data for purposes of transport, persistence, and interface to various consuming systems. Define Phasor Measurement data Defines what data need to be exchanged between software applications in order to support the business processes commonly applied at utilities. This leverages definitions of common data semantics, definitions of message structure and definition of which messages are required to support specific business process steps. For defining interfaces and standards for interoperable system integration and service oriented architecture. For message oriented integration. It enables wireless communication and control between utility companies and common household devices such as smart thermostats and appliances. Table 3 - Application Technology Recommendations Application Technology Application/transaction processing server Business Process Engine Business process integrator Business rules engine Connector framework Enterprise Service Bus (ESB) Event processing Integration middleware Load balancers Message gateway Message queue SOA governance tool Web portal Web servers Purpose/Relevance/Comments A middleware software component dedicated to efficient execution of programs supporting application construction. Implements Business Process Execution Language (WS-BPEL) for process choreography. Automates and optimizes business processes, within the organization and across the customers, partners, and supply chains. Software executing one or more business rules in a runtime production environment. Allows data access from diverse sources. Software providing complex architectures with fundamental services via an event-driven and standards-based messaging engine. Foundational capabilities include invocation, routing, mediation, protocol transformation, process choreography, service orchestration, complex event processing, and QOS measures (reliable delivery, transaction management, & security) Handles messaging resulting from pre-defined circumstances occurring across the enterprise. Allows data contained in one database to be accessed through another. Provides techniques for distributing workload evenly across two or more servers. An integrated suite of components for an easy-to-use entry point to all enterprise business info Allows independent and potentially non-concurrent applications on a distributed system to communicate with each other. SOA control and management tools (service interface management & discovery, monitoring) Integrates and presents information from diverse sources in a unified way. Delivers content (web pages) using the Hypertext Transfer Protocol (HTTP), over the web Smart Grid Reference Architecture Views 22

33 Analytics Services Smart grid analytics generally reside within the cross domain foundational services box in the lower right area of Figure 9 - Layered Services Tier View. Analytics provide the services needed to make sense out of the data being created by smart grid sensors (smart meters, grid components, energy management devices, etc.). Analytic tools are designed to turn large amounts of structured and unstructured data into information useful to humans (consumers, utility operators, regulators) and smart grid control mechanisms (demand response, outage management, advanced load controls). Analytics Logical Model The Analytics Logical Model [Figure 12] consists of three views: (1) analytics synergy, (2) analytics services stack, and (3) analytics component. Figure 12 - Analytics Logical Model Smart Grid Reference Architecture Views 23

34 The synergy model illustrates a key concept of Smart Grid architecture: the manner in which data sources from the grid produce data which, when processed in multiple ways through multiple analytics, can support multiple business outcomes. Therefore, data sources (sensors) and analytics processing tools and components can and should be shared over multiple capabilities. By arranging the architecture to take advantage of the synergy, the Smart Grid architect can minimize infrastructure cost while maximizing realized business value of the sensing and analytics elements of the Smart Grid. While the synergy model is complex and can be difficult to understand, it has the potential for high payback. In this synergy model diagram, data sources are arrayed at the bottom. Data from these sources falls into one or more of four data classes as shown in the second tier. In the third tier, six classes of analytics (process data from the data sources according to data class. In the top tier, analytics support one or more utility business processes. Line coloring does not have a special meaning except to make line groupings somewhat easier to follow on a crowded diagram. Finally, because visualization is pervasive, it is shown as a blue box in the synergy model third tier, essentially underlying all of the analytics classes, signifying their use in GUI-based decision support. Analytics Structural Model The analytics services structural model [Figure 13] is provided to help the smart grid architect consider how to best deploy the various analytics services using a stylized representation of a typical utility network infrastructure. At the top of the model are elements needed to support external agencies (regulators, interchange and balancing authorities, etc.). At the bottom are the analytics services needed by premise devices (smart appliances, PEV chargers, building energy managers, etc.). In between are a dozen other tiers where analytics services may be located to support residing devices and functionality. A self-describing template used for this model is on the last page of Appendix B, Services Classes Concepts. Included are descriptions of all fourteen tiers used in the model. Smart Grid Reference Architecture Views 24

35 Figure 13 - Analytics Structural Model The Analytics Structural Model illustrates how analytics functionality can be distributed across a Smart Grid. This is important to the smart grid architect because a distributed architecture can address issues of latency, scale, and robustness. There are many places on the grid where analytics elements may reside, and a key aspect of smart grid architecture is to develop a proper combination of distributed and centralized analytics elements. There are significant tradeoffs between the degree to which analytics are distributed and the resulting requirements for communications services, as well as data services. Furthermore, the distribution of analytics must be coordinated with the control architecture, since some analytics are consumed by controls systems which are also distributed and require the analytics as inputs on a low-latency basis. Smart Grid Reference Architecture Views 25

36 Analytics Architecture Specifications Table 4 is a list of general analytics specifications. The justification for each is provided to enhance the reader s understanding of each specification. The table does not include detailed business or technical requirements a specific analytics tool would fulfill. Each utility must include requirements for analytics services in every relevant smart grid project. Table 4 - Typical Analytics Specifications Specification Analytics shall be dynamically re-distributable a service to manage the re-distribution must be included in the analytics architecture. Analytics services shall be centrally and uniformly managed. The management mechanism should be integrated with the general network and grid device management services. Analytics services shall be deployable in a distributed fashion. Places on the grid for analytics deployment include: Data centers Control centers Substations Mobile devices Edge devices, including meters, gateways Communications devices Cloud services (private and third party) Distribution feeder power system devices Analytics shall be distributed according to a latency hierarchy. Some analytics will be implemented close to data sources and consumers, while others can be implemented at control centers or data centers. Analytics associated with protection and control should be distributed; analytics for asset health and stress accumulation can be centralized. Analytics for grid state should be distributed. Analytics for consumer behavior can be centralized. Smart grid analytics services architecture shall include analytics tool management. Such tools include those that enable the development and deployment of rules for event processing engines, configuration tools for computational analytics, and workbenches for tools highly interactive in nature. Data quality monitoring for low latency analytics shall be incorporated into the analytics service at the point of deployment for immediate detection of data channel failures. The control systems architecture shall be developed as a Justification Analytics change as the Smart Grid evolves and it is impractical to physically visit distributed analytics elements to make changes. It is impractical to use a distributed architecture that requires field frequent visits to devices, in fact, Zero Touch deployment is necessary for Smart Grids at scale. Integration with the management services architecture for distributed architecture operations to be feasible. Wherever data is generated or information consumed is a location candidate for appropriate analytics. Communication network elements hosting additional software are also good candidates, since generally these elements are where data sources and consumers reside. Use of these to host analytics makes it possible to provide the benefits of distributed architecture while minimizing the number of devices to be managed and maintained. Zero-touch deployment is crucial. The tradeoff between degree of distributed intelligence and communications requirements is one of the most significant decisions smart grid architects must make. This tradeoff must be made early in the design process and revisited periodically as the grid transformation proceeds. As the utility transforms its grid, forms and uses of analytics will change. In addition, new analytics are being developed as grid data becomes available. The utility must not expect analytics deployment to remain static. Thus, having access to the appropriate tools to manage the analytics transformation is crucial. Many data acquisition devices suffer data channel failures with the device not generating a fault alarm. Since the data may be used in a real-time mode, it is necessary to have ways to detect faulty data in realtime. This must be done at or near the source since much of this data will never reside in a database. In addition, real-time control may depend on the correctness of this data and the consequent analytics. Since controls consume analytics widely, the control Smart Grid Reference Architecture Views 26

37 precursor to the analytics services architecture development. Comprehensive data access strategy, grid observability strategy and sensor architecture shall be developed to support the development of the analytics services architecture. Distributed analytics in hierarchical analytics services architecture shall be designed to reduce data volumes for data moving from edge devices up through successive layers of data management and analytics to the points of data persistence. This is one of the points of intersection between analytics services architecture and communications services and data services architectures. Note that this type of volume reduction does not apply to usage (meter) data. Aggregation of non-usage analytics should generally be homologous with the disaggregation of control signals (see Control section). Distributed analytics shall provide logging capabilities for evaluating analytics operation. This includes event processing, and the management of analytics logs is a significant data services architecture issue. Multiple deployment models to implement analytics shall be used for distributed analytics including standalone, hierarchical, and peer models. Event analytics may be broken into event stream processing at endpoints, feeding higher level complex event processing at substations, control centers, and data centers. For low latency analytics, implementations shall be located at or very near the source(s) of data and the primary outputs destination(s), without sending data to and from remote processing centers. Embedded processing shall be used, but the actual analytics must be downloadable. Information-sharing among distributed analytics is often necessary; Smart Grid analytics services architecture shall include a real-time distributed data sharing mechanism. This is essential for access to a global grid state, which is needed not only by analytics, but also by control and other applications. Interface of distributed analytics services at the system level requires that analytics be treated in two ways: as data bound for data persistence (data stores) and as streams (synchronous or asynchronous) bound for consumption by applications in realtime. In some cases, the analytics outputs will be treated both ways simultaneously, leading to implications for data and communications services architecture. As-operated grid connectivity shall be made available to analytics requiring grid context (grid state), each analytic sensitive to grid topology must have the correct grid state time-aligned with the formation of the data or event upon which the analytic depends. Security shall be part of analytics services design. This includes: Permissions for source and target data sources Permissions to execute on centralized and distributed locations systems architecture is a vital analytics design input. Provides inputs vital to the development of the analytics architecture, due to the infrastructure optimization problem discussed in the section for the analytics structural model. The point is that analytics extract key information, generally resulting in a data volume reduction while maintaining essential information. In some cases, raw data may be retained and persisted for later analysis based on application-specific trigger conditions, but generally lower latency analytics will consume raw data and pass along information to either consuming applications, or higher level analytics. This may not be feasible in non-hierarchical analytics services architectures (tier-based data volume management). Event logging is necessary to support post mortem analyses of various kinds. With the volume of events that can be generated in a Smart Grid, management of the event logs is a large issue. It is important to recognize that distributed analytics elements do not have to be functionally complete in themselves; it is often more useful for elements to compute partial results and the either forward them upward into a hierarchy or exchange partial results with other distributed elements. This is a direct consequence of the distributed nature of power grids and their control systems. Electrical effects propagate across the grid at very high speeds and with wide reach. Effects at the distribution level can affect transmission and vice versa. Grid state and other data may be needed almost anywhere to support analytics and control. Some analytics operate on telemetry, but are not used for real-time operations. They may be monitored by agents or simply archived for batch processing to support various business processes. Consequently, there are both communications and data services implications that must be coordinated by the smart grid architect. This problem is fundamental due to the frequent topology changes to distribution grids. It is also an issue for wide area measurement systems. For many processes it is possible for the grid topology to change before a prior event or datum is processed, so past values of grid topology must remain available. This set of security and management features must be integrated with the overall security and management architectures. For implementations of distributed Smart Grid Reference Architecture Views 27

38 Permissions for analytics to communication to other analytics Permissions for users to access the dashboards or KPI Permissions on who can modify/start/stop/deploy analytics Tools for tracking changes in the actual analytical algorithms Visualizations are analytics for the eye/brain; therefore, visualization shall be a part of analytics services architecture. analytics such as multi-agent systems and others providing downloadable, zero-touch capability, it is crucial that the downloadable elements be verifiably legitimate via certificates, etc. The smart grid architect should treat visualization as an integral part of the analytics architecture. Visualization is needed at many grid levels and locations; treating it as a separate centralized function can lead to difficulty in uniformly providing the service. Architecture Standards and Technology Recommendations Technologies will evolve as a utility s Smart Grid transformation takes place over a period of many years. Changes in analytics services technologies are inevitable. As of this writing, a number of technologies and standards are appropriate for implementing analytics services. Table 5 lists these standards with their purpose or relevance; Table 6 similarly lists the in-place or emerging technologies. Table 5 - Recommended Analytics Standards Standards IEC GOOSE messaging IEC CIM GID Interface Services (GDA, HSDA, TSDA, GSE) IEC Common Information Model IEEE Computer Society FIPA (Foundation for Intelligent Physical Agents) Purpose/Relevance/Comments For use in low latency protection and control messaging, where analytics outputs are being used in applications such as adaptive protection and real time grid stabilization Four services associated with IEC CIM for data interchange in four classes: generic data, high speed data, time series, and events Primary schema for data representation; should also be applied to analytics outputs, which in themselves are treated as data for purposes of transport, persistence, and interface to various consuming systems. For analytics implementations using multi-agent systems Table 6 - Recommended Analytics Technology Analytics Technology Data mining Digital signal processing Event stream/complex event processing OLAP/Cubing/Dashboard s Phasor/power system calculation processing Statistical analyses Visualization Purpose/Relevance/Comments Needed to analyze vast volumes of grid data to detect and extract underlying patterns and models Needed to extract information from low level grid data such as waveforms and sensor telemetry Needed to filter, throttle, correlate, and extract situational understanding from multiple asynchronous event streams from up to millions of grid devices; management of event stream bursts. These are visualization tools for data trending and status indication. Such tools should be connected via appropriate security services to data and analytics output feeds from the control center, as well as data residing in enterprise databases. Necessary to support a wide range of real time analytics involving grid state, device utilization, power flow and load control, and fault analysis a variety of statistical analysis tools may be applied to problems such as demand and variable energy resources modeling, as well as customer behavior modeling. Crucial to translate both data and numerical analytics into forms readily comprehended by humans for decision support; typically coupled with geospatial and grid topology information for context; also includes graphics for trending and forecasting on sensor telemetry Smart Grid Reference Architecture Views 28

39 Data Services Smart grid data services generally reside within the cross domain foundational services box in the lower right area of Figure 9 - Layered Services Tier View (labeled Data Management Services). Data Services Logical Model The Data Services Logical Model [Figure 14] consists of three views: (1) data services synergy, (2) data services stack, and (3) data management component. Consumers/ Processes Data Management Services Data Classes Sources Analytics SCADA Data Discovery System Mgmt Data Data Indexing and Reference Data Collection MDM Data Services Synergy Model Protection Equipment Unstructured Content Data Registration and Life Cycle Management Customer Interface Enterprise Application Third Parties Data Access & Update External Data Feeds B2B Field crews Consumer Service Content Management Master Data Management Data Transformation Data Flow Orchestration Data Quality and Integrity Data Storage IVR/Call Center Data Manipulation Data Security Operational Data Time Series Data Event Data Records Meta Data Enterprise Application Data Services Stack Model Foundation Core Integration Visualization Analysis Reporting Network state estimation Electrical Network model Measurement model CIM/61850 interface Customer information Premises information messaging transformation encryption Support Services governance transport storage Network measurements Geospatial information Manually entered information (inspections, readings, etc.) Market operations signals Market operations model Asset catalogue Compatible units Asset data Edge device model data access data synchronization master data management access control services semantic model management message & service definition Data Management Component Model Meta Data Management Semantic Model Management ETL ESB EII Persistent Data Storage Archive Figure 14 Data Services Logical Model The synergy model (upper left area of Figure 14) shows how Smart Grid ICT must simultaneously manage data from many sources to support multiple business outcomes. Data must therefore be decoupled from data sources to allow sharing of both the data and the components used for moving, manipulating and storing it. By arranging the architecture to leverage this synergy, the Smart Grid architect can minimize infrastructure cost while maximizing the realized business value of enterprise Smart Grid Reference Architecture Views 29

40 Smart Grid information management. The Synergy Model has the primary data sources arrayed at the bottom. Data from these sources fall into one or more of the five data classes depicted in the second tier. In the third tier, key data management services of the Utility Services Layer and the Core Business Services Layer process data from the data sources according to data class. In the top tier, various business processes and consumers leverage the data management services to accomplish higher level business functions. Line coloring does not have a special meaning except to make line groupings easier to follow on the crowded diagram. Pervasive security is represented as a blue box in the model s third tier, underlying all data management services. In the data service stack model (upper right area of Figure 14), the core grouping is an abstraction of capability services types (refer to the discussion of services in Appendix B). These services focus on business capabilities and are intended to be independent of any particular business process. The stack model also contains services from the process service and solution service layers used to accomplish higher level services within the foundation grouping. These foundational services, as well as lower service layers, are used by the consumers/processes tier of the synergy model to accomplish needed business functionality. Data Services Structural Model The data services structural model [Figure 15] illustrates how data services can be distributed across a smart grid. This is important to the smart grid architect because distributed data architecture can address issues of data access, storage, transformation, latency, encryption, and synchronization within and across layers in the physical grid. While traditionally these services have been concentrated in data and control centers, in a smart grid these distributed data services will reside in substations, field devices, and in various network locations from the end use premises to the control center. A selfdescribing template used for this model is on the last page of Appendix B, Services Classes Concepts. Included are descriptions of all fourteen tiers used in the model. Smart Grid Reference Architecture Views 30

41 Figure 15 - Data Services Structural Model Data Architecture Specifications Table 7 is a list of general data specifications. The justification for each is provided to enhance the reader s understanding of each specification. Table 7 - Typical Data Services Specifications Specification A semantic model shall be built to create and maintain common business definitions, business rules (requirements), and metadata. Data classification shall be performed. Justification The creation of common business definitions will facilitate communications in all directions within the organization. This should be accomplished as an integral part of incrementally building the enterprise semantic model. Classify data for security and management purposes. Smart Grid Reference Architecture Views 31

42 Specification Data Enrichment shall be provided where necessary. Data integration shall be provided. Data movement shall be provided. Data quality and integrity shall be source guaranteed. Appropriate security shall be applied to data. Data shall not be tied to applications Appropriate access shall be provided to data sources Data synchronization shall be supported where necessary. Data transformation shall be provided where necessary. Data shall be validated as appropriate. Data Object Indexing and referencing shall be provided. Data growth shall be managed. Master data management shall be provided. Methods shall be provided for understanding and removing data duplication Justification Enrich the data by supplementing it with other sources as needed and per the understood the business process that is implemented Integration data for process automation and business transactions. Enable data to move from one source to another. Ensure that the data is guaranteed by the source to be of primary source quality and not derived or distorted by secondary views. Provide security for data in accordance with its classifications. Security activities include authentication and authorization, logging, and control. Data should be made available as master data. A single view of data should exist across the enterprise shared, complete, and accurate. Provide access to data sources, either through application or directly, while maintaining appropriate security. Synchronize data instance and content for multiple systems of same data. Transform data from one semantic and syntax to another. Validate data in accordance with defined business rules. Centralize a robust means of translating data object identifications between disparate systems. Adhering to EIM practices will avoid having the same data expressed in multiple ways, each with the same or similar meaning. Data growth is also managed by understanding what data must be retained for what periods of time and within which physical data stores and systems. Ensure consistent master information across transactional and analytical systems. Have repeatable rules definition for guaranteeing data quality. Data Architecture Standards and Technology Recommendations The following table of data standards and technology recommendations is provided to the smart grid architect for reference. Over time, however, innovations will inevitably diminish its completeness. The architect should always research the latest information on these topics, but this table is a good starting point. Table 8 lists important 2011 standards and technology recommendations applicable to data services, each with a brief explanation of their purpose or relevance. Table 8 - Data Standards and Technology Recommendations Standards Applicable functional standards of the IEC and series of standards IEC IEC Purpose/Relevance/Comments Industry standard message payloads are defined using XSD and RDF. Canonical Data Models (CDMs) of a utility may be compliant with, or based of (if extensions are required) these industry standard message payloads. Object oriented protocol for retrieving data and controlling devices at substations and along feeder networks. System Interfaces For Distribution Management Part 1 Interface Architecture and General Requirements Smart Grid Reference Architecture Views 32

43 Standards Purpose/Relevance/Comments IEC and IEC IEC Common Information Model. Information model for data representation which is used as a basis for defining industry standard application interfaces. MultiSpeak The MultiSpeak project is funded by National Rural Electric Cooperative Association (NRECA). It defines standardized interfaces among software applications commonly used by electric utilities. It defines details of data that need to be exchanged between software applications in order to support different processes commonly applied at utilities. OASIS UDDI Universal Description, Discovery and Integration (UDDI) is a platform-independent, Extensible Specification Markup Language (XML)-based registry is a mechanism to register and locate web service applications. Simple Object Access SOAP, originally defined as Simple Object Access Protocol, is a protocol specification for Protocol (SOAP) exchanging structured information in the implementation of Web Services in computer networks. It relies on Extensible Markup Language (XML) for its message format, and usually relies on other Application Layer protocols, most notably Remote Procedure Call (RPC) and Hypertext Transfer Protocol (HTTP), for message negotiation and transmission. SOAP can form the foundation layer of a web services protocol stack, providing a basic messaging framework upon which web services can be built. This XML based protocol consists of three parts: an envelope, which defines what is in the message and how to process it, a set of encoding rules for expressing instances of application-defined data types, and a convention for representing procedure calls and responses. W3C XML Specification Extensible Markup Language (XML) is a set of rules for encoding documents in... in the XML 1.0 Specification produced by the W3C. WSDL WSDL is an XML-based language for describing Web services and how to access them. WS-I Basic Profile The WS-I Basic Profile (official abbreviation is BP), a specification from the Web Services Interoperability industry consortium (WS-I), provides interoperability guidance for core Web Services specifications such as SOAP, WSDL, and UDDI. The profile uses WSDL to enable the description of services as sets of endpoints operating on messages. Smart Grid Reference Architecture Views 33

44 Control Services Smart grid control services generally reside within the cross domain foundational services box in the lower right area of Figure 9 - Layered Services Tier View. Control Services Logical Model The Control Services Logical Model [Figure 16] has three views: control synergy, control services stack, and control component. Figure 16 - Control Services Logical Model The synergy model diagram (upper left area of Figure 16) is the most complex, containing four tiers. The bottom two tiers match the analytics synergy model described previously. The third tier consists of key smart grid control services. The top tier shows control-oriented business processes. At the bottom of the second tier, connectors illustrate how various grid devices create data for use in control. Line color is only for the purpose of visual clarification the colors do not signify any special property or Smart Grid Reference Architecture Views 34

45 characteristic. The synergy model diagram highlights several crucial issues for the smart grid architect to analyze: Need for grid state aggregation and publication. Use of multi-controller, multi-objective control inherent in smart grid design using grid data sources to support multiple business processes via processing of multiple analytics. Federation of control systems as a consequence of multi-controller, multi-objective control. Disaggregation of control command at the device level. This must take into account the actual grid topology at the time of control actuation, as well as various grid state variables. These significantly impact the design of controls, communications, sensor, and analytics architectures. Control Services Structural Model The control services structural model [Figure 17] is provided to help the smart grid architect consider how to best deploy the various control services using a stylized representation of a typical utility network infrastructure. At the top of the model are elements needed to support external agencies (regulators, interchange and balancing authorities, etc.). At the bottom are the control services needed by premise devices (smart appliances, PEV chargers, building energy managers, etc.). In between are a dozen other tiers where control services may be located to support residing devices and functionality. A self-describing template used for this model is on the last page of Appendix B, Services Classes Concepts. Included are descriptions of all fourteen tiers used in the model. Smart Grid Reference Architecture Views 35

46 Figure 17 - Control Services Structural Model Control Architecture Specifications Table 9 is a list of general control specifications. The justification for each is provided to enhance the reader s understanding of each specification. Smart Grid Reference Architecture Views 36

47 Table 9 - Typical Control Specifications Specification Control signals must be disaggregated in a hierarchical manner in parallel to both the general structure of the power grid and the aggregation of analytics and grid state (see Analytics section). Control system architecture must provide means to control systems not part of the grid itself. Control system must integrate adaptive protection at both transmission and distribution levels. Control systems for demand response and DER control must provide for dispatch from the balancing authority; control signals must be disaggregated to the distribution substation level, and from there to the feeder device and premises level. IVVC must be coordinated with DR at the distribution feeder level, so the control architecture must provide for control disaggregation and recalculation at multiple levels in the grid. Control systems must have access to grid state determined by both wide area measurement as well as deep measurement; deep measurement means grid state at the distribution and premises levels must be part of extended grid state; the control system architecture must provide for wide distribution of grid state across control elements. Control systems must support both event-driven functions and closed loop servo functions. Close loop servo control architecture must accommodate multiple time scale operation, so control architectures must provide means for complex feedback and feed forward. For grids where stabilization or compensation is required at the distribution level, the control system shall be capable of operating with a complete closed loop path execution in less than two power cycles. Grid state determination for wide area measurement and wide area control shall depend to the maximum degree possible on actual measurement, with the minimum estimation necessary to complete the grid state view. Control system architecture must distribute grid state across control systems elements in real time in a secure manner. Justification All grid systems are interconnected through the analog power grid with many interactions. To provide the means to reconcile control commands and eliminate undesirable interactions, the grid structure must guide control signal disaggregation. Control system architecture not providing for disaggregation at the various levels of the grid hierarchy will limit the ability of control system designers to solve interaction problems. Secondary load control for Demand Response and integration of DER, microgrids, and PEV charging stations/networks requires utility interface due to interactions resulting in grid management issues. The increasing penetration of secondary load control systems causes interactions, including non-outage cold load pickups requiring dynamic modification of protection settings based on both grid state and control actions. Without the proper disaggregation, interaction between DR and IVVC can cause grid violations, and can cause DR to be significantly less effective than expected. Other interactions of grid controls and secondary load controls can occur, up to and including wide area blackout. Extended grid state is the essence of deep situational awareness and is the primary set of observations driving control systems. As multiple control sub-systems share common elements, ubiquitous access to grid state becomes crucial. Some control functions can be event based, but other highly dynamic processes such as stabilization require continual closed loop control update. With multi-time scale systems, control loops must be more complex than simple feedback, so control architecture must support multi-feedback, multi-time scale operation. Dynamics of close loop stabilization require much faster response and much lower latencies than traditional distribution automation. New control devices are capable of acting at these speeds (see Control Technology table below). Advanced grid controls need access to actual grid state;, especially at the distribution level where estimation is effectively impossible. In addition, extended grid state contains elements not susceptible to estimation, such as quality state. Smart Grid Reference Architecture Views 37

48 Specification Multiple control systems must be federated before control signals are sent to individual devices. The control service environment shall be a multi-controller, multi-objective environment. Power grid control must be a hybrid of centralized and distributed control elements, with distributed control extending into substations and to distribution grid elements; control elements shall be able to use peer-to-peer communication to cooperate on control tasks and shall be capable of autonomous operation when communication with higher level controls is not possible. Smart grid control systems can require cross tier integration; this may be accomplished in a hierarchical fashion normally or via tier-jumping where very low latency is required. Control system architecture must allow for cross-tier control solutions. System models (connectivity models) for transmission, substations, and distribution are context for control - control systems must have a means to receive the relevant portions of connectivity models from the master sources in a secure and timely manner. Timeliness is dependent on the rate at which connectivity changes can occur relative to the essential time constants of the relevant control subsystems. Teleprotection must be done via packet switching (Ethernetbased) networks rather than circuit switching networks Justification As smart grid functionality increases in sophistication, more control processes have reasons to access the same control elements in the grid. Such sharing is important for economic reasons but the controls must not collide at the control elements; hence the need for federation in the architecture to provide resolution mechanisms. Centralized control is not adequate for fully built out smart grids; using a combination of centralize and distributed control elements provides scalability and robustness while maintaining central supervision and management of the grid. Smart grid controls are increasingly required to deal with effects propagating through the grid, including from distribution to transmission. In addition, some business models allow for secondary control systems external to the utility control system and operating across tiers. System models are context for both analytics and control; without this context proper control action cannot be determined. Tele-protection in a smart grid environment needs flexibility not available via circuit switching Control Architecture Standards and Technology Recommendations The following table of control standards and technology recommendations is provided to the smart grid architect for reference. Over time, however, innovations will inevitably diminish its completeness. The architect should always research the latest information on these topics, but this table is a good starting point. Table 10 lists important 2011 standards and technology recommendations applicable to control services, each with a brief explanation of their purpose or relevance. Table 10 - Recommended Control Standards and Technology DNP3 Standards IEC ,104 IEC GOOSE messaging IEC IEC CIM GID Interface Services (GDA, HSDA, TSDA, GSE) Purpose/Relevance/Comments Commonly used for communication with grid control devices. May be operated in native serial fashion, or over IP. Serial communication for control devices. For use in low latency protection and control messaging Draft standard for PMU communications support for multi-cast Four services associated with IEC CIM for data interchange in four classes: generic data, high speed data, time series, and events Smart Grid Reference Architecture Views 38

49 Standards IEC Common Information Model IEEE C IEEE Computer Society FIPA (Foundation for Intelligent Physical Agents) Purpose/Relevance/Comments Primary schema for data representation; should also be applied to analytics outputs, which in themselves are treated as data for purposes of transport, persistence, and interface to various consuming systems. PMU communications For control implementations using multi-agent systems Most grid control devices and control systems are well known. The section below discusses some advanced control elements and technologies. Table 11- Advanced Control Elements and Technologies Control Technology Distribution level PMU networks DSTATCOM Multi-agent systems STATCOM / SVC / FACTS / fast ancillary storage VAR-controllable AC- DC inverters WAMS/PMU networks Purpose/Relevance/Comments Can provide distribution grid state, fault intelligence, asset utilization measurement and outage intelligence inputs for the relevant control sub-systems. Distribution STATCOM provides electronic stabilization and dynamic power quality compensation at the distribution level. Software technology for implementation of distributed intelligence; can be used to implement distributed control agents. Increasingly important for electronic stabilization as grid rotational inertia is decreased through the displacement of traditional generation with VER. Includes such applications as model power oscillation damping and frequency compensation. When used as grid interfaces from DG and microgrids, and when controlled by the utility, these can provide fast VAR compensation (much faster than capacitors) and can provide much finer granularity in VAR control. Provide necessary state measurement for transmission control. Smart Grid Reference Architecture Views 39

50 Security Services Smart grid security services generally reside within the cross domain foundational services box in the lower right area of Figure 9 - Layered Services Tier View. Security Logical Model The Security Logical Model [Figure 18] has three views: the security component, the security services stack, and security synergy. Figure 18 - Security Logical Model The Security Component Model depicts in block format the primary security components used. Smart Grid Reference Architecture Views 40

51 The Security Services Stack Model has several parts: Support services services that may come from another portion or tier of the architecture but are necessary to support the services in the tier being described Foundation services key security services, generally needed throughout the architecture which other services depend or expand upon Core services the primary security architecture service stack Integration services services specifically providing integration between security and other architectural tiers or business systems The Security Synergy Model illustrates how security services can be shared to meet various Smart Grid security requirements. If the architecture can take advantage of this synergy, the utility can build cost effective security infrastructure services, reused across multiple Smart Grid systems. Security Structural Model The security structural model [Figure 19] is provided to help the smart grid architect consider how to best deploy the various security services using a stylized representation of a typical utility network infrastructure. At the top of the model are elements needed to support external agencies (regulators, interchange and balancing authorities, etc.). At the bottom are the security services needed by premise devices (smart appliances, PEV chargers, building energy managers, etc.). In between are a dozen other tiers where security services may be located to support residing devices and functionality. A selfdescribing template used for this model is on the last page of Appendix B, Services Classes Concepts. Included are descriptions of all fourteen tiers used in the model. Smart Grid Reference Architecture Views 41

52 Figure 19 - Security Structural Model The structural model illustrates how security functionality may be distributed across a Smart Grid. The Smart Grid architect needs to work closely with security experts to ensure appropriate security principles are applied at every level of the Smart Grid structure. As the grid evolves, millions of new components will be introduced; each is expected to ensure power systems operations maintain cyber confidentiality, integrity, and availability. The NIST Interagency Report 7628 (NISTIR 7628), Guidelines for Smart Grid Cyber Security: Vol. 1 (NIST-ITL, 2010) provides a Smart Grid Logical Reference Model with dozens of logical interfaces identified. Chapter 2 and Appendix G of NISTIR 7628 should be studied to ensure appropriate security attributes are addressed for each logical interface category. NISTIR 7628 is freely available on the NIST Smart Grid website: Smart Grid Reference Architecture Views 42

53 Security Architecture Specifications NISTIR 7628, Chapter 3, has an extensive listing of high level Smart Grid security requirements. Security Standards and Technology Recommendations The following table of security standards and technology recommendations is provided to the smart grid architect for reference. Over time, however, innovations will inevitably diminish its completeness. The architect should always research the latest information on these topics, but this table is a good starting point. Table 12 lists important 2011 standards and technology recommendations applicable to security, each with a brief explanation of their purpose or relevance. Table 12 - Security Standards and Technology Recommendations Standards Purpose, Relevance or Comments Digital Signature (DSS, DSA, RSA, etc.) Cryptography and network security DNP3 Secure Authentication User & device authentication, data integrity protection Encryption (AES, RSA, etc.) Facilitates secure (secret) communication Hash (SHA-1, SHA-256, HMAC, etc.) Cryptographic hash functions employed in several widely used security applications IEC Utility Communication Security Developed to handle the security of IEC Technical Committee 57 protocols (IEC 61850, 61968, etc.) IEEE 1686 IED Security Minimum requirement set for intelligence electronic device security compliance with NERC CIP requirements IEEE 1711 Trial Use Standard for a SCADA Serial Cyber Security of substation serial links Link Cryptographic Modules and Protocol IPSec Internet Protocol Security secures IP communications by authenticating and encrypting each session s IP packets NERC Critical Infrastructure Protection (CIP) The NERC Critical Infrastructure Protection program aims to improve Standards physical and cyber security for the bulk power system of North America as it relates to reliability. Public Key Infrastructure (X.509) A cryptographic ITU-T standard for a public key infrastructure (PKI) for single sign-on (SSO) and Privilege Management Infrastructure (PMI) PubsFIPS Federal Information Processing Standards (FIPS) Publications - a series of documents for the U.S. Federal government issued by NIST TLS Transport Layer Security, a network protocol and successor to Secure Sockets Layer (SSL) Wireless Security (WEP, WPA, etc.) IEEE 802.1X encryption standards designed to prevent unauthorized access or damage to computers using wireless networks Smart Grid Reference Architecture Views 43

54 Communications Services Smart grid communications services generally reside within the cross domain foundational services box in the lower right area of Figure 9 - Layered Services Tier View. Communications Services Logical Model The Communications Services Logical Model [Figure 20] has three views: the communications services component, communications services stack, and communications services synergy. Figure 20 Communications Services Logical Model The Communications Services Component Model depicts in block format the primary communications components used in the architecture. Smart Grid Reference Architecture Views 44

55 The Communications Services Stack Model has several parts: Support services services that may come from another portion or tier of the architecture but are necessary to support the services in the tier being described Foundation services key security services, generally needed throughout the architecture which other services depend or expand upon Core services the primary communications architecture service stack Integration services services specifically providing integration between communications and other architectural tiers or business systems The Communications Synergy Model illustrates how communications services can be shared to meet various Smart Grid communications requirements. If the architecture can take advantage of this synergy, the utility can build a cost effective communications infrastructure services, reused across multiple Smart Grid systems. Communications Services Structural Model The communications services structural model [Figure 21] is provided to help the smart grid architect consider how to best deploy the various communications services using a stylized representation of a typical utility network infrastructure. At the top of the model are elements needed to support external agencies (regulators, interchange and balancing authorities, etc.). At the bottom are the communications services needed by premise devices (smart appliances, PEV chargers, building energy managers, etc.). In between are a dozen other tiers where communications services may be located to support residing devices and functionality. A self-describing template used for this model is on the last page of Appendix B, Services Classes Concepts. Included are descriptions of all fourteen tiers used in the model. Smart Grid Reference Architecture Views 45

56 Figure 21 - Communications Services Structural Model Communications Architecture Specifications Table 13 is a list of general control specifications. The justification for each is provided to enhance the reader s understanding of each specification. Smart Grid Reference Architecture Views 46

57 Table 13 - Typical Communications Specifications Specification Architecture for end-to-end communications shall be divided into functional subnetworks including the following: Extranet Data/Control Center Networks Two Tier Wide Area Network Core Network Backhaul Network Substation Network Two Tier Field Area Network Tier 1 Multipurpose FAN Tier 2 Purpose-Built FAN Premises Network Backhaul Wide-Area Networks Deployment Design specs shall: Accommodate tightly controlled access from remote networks Provide a high degree of security Provide high network performance primarily using fiber, microwave and other infrastructures Provide flexible network segmentation/virtualization capabilities, in support of a variety of application, security, and geographical domains Support data, voice, video, and control services Be based on IPv6 technologies Support convergence such that, if desired, traffic from one does not harm or adversely impact traffic from other environments Operational specs should: Support key remote locations where grid logic is employed Be under autonomous control by utility (if private) or accompanied by strict service-level agreements (if public) Provide a high degree of network monitoring and system management Communication architecture shall anticipate and support highly distributed data collection, control, and analytics environments. Communication networks shall be based on standardized packet transport services and utilize IPv6 technology. Justification Each of the sub-network domains has unique functional characteristics regardless of communications technology considered. Provides high performance and resilient connectivity between core wide-area networks and key remote facilities (such as outlying substations) or to points of network concentration (such as remote communications huts). High performance attributes: from hundreds of Megabits to multi-gigabit, extremely low latency, deterministic traffic support, very high reliability. Data collection, control and analytics are evolving from centralized to distributed environments and the communication architecture must support this evolution. IPv6 offers superior networking functionality and can scale to meet the needs of Smart Grid. Smart Grid Reference Architecture Views 47

58 Specification Communication services must be centrally and uniformly managed. The management mechanism should be integrated with grid device management. Completion of a comprehensive set of anticipated use cases is a critical foundation for the development of the communication services architecture Core Wide-Area Networks Deployment Design specs should: Accommodate tightly controlled access from remote networks Provide the highest degree of security Provide high network performance (multi-gigabit, extremely low latency, deterministic traffic support, very high reliability, rapid failover, extensive redundancy) primarily using fiber and other infrastructures Provide flexible network segmentation/virtualization capabilities, in support of a variety of application, security and geographical domains Support data, voice, video, and control services Be based on IPv6 technologies Utilize diversely connected, redundant connectivity architectures (such as ring or mesh topologies) Support convergence such that, if desired, traffic from one environment does not harm or adversely impact traffic from other environments Operational specs should Directly support key remote locations where grid logic has been employed Be under autonomous control by utility (if private) or be accompanied by strict Service Level Agreements (if public) Provide a high degree of network monitoring and system management Development of holistic end-to-end communication architecture is critical to support entire application, control and analytics environments. Justification Communication systems entail many diverse, interconnected and interdependent communication links. These systems are also interdependent with grid devices and each management system should inform the other for service impacts. Provides comprehensive inputs and requirements vital to the development of the communications architecture. Prevents silo view that can lead to duplicate infrastructure and expenses. Provides high performance and resilient connectivity within the same organization s data/control centers, from data/control centers to key remote facilities (such as substations), and from data/control centers to points of backhaul network interconnection Application, control and analytics endpoints span multiple domains of the communications architecture. Smart Grid Reference Architecture Views 48

59 Extranet Network Deployment Design specs should: Specification Provide network security suitable for a plethora of access scenarios Support access for thousands to millions of users (based on the number of customers served by the utility) Interconnect to several public infrastructures (carriers) and offer various networking services (Internet, data, voice, video, control) Provide performance and capacity commensurate to normal business needs as well as sufficient additional capacity to accommodate emergency situations Include link and service failover mechanisms to alternative external networks and/or service providers Operational specs should: Assume that extranet connectivity may not allow utility autonomous control over service Assume that extranet connectivity offers lesser degree of monitoring and management to remote devices than private networking Consider extranet connectivity is dependent on others for reliability and performance Consider extranet connectivity cost structure is largely capacity driven Local Area Networks within Operations/Data Centers Deployment Design specs should: Provide extremely tight access controls Provide the highest degree of security Provide highest degree of network performance (multi-gigabit, extremely low latency, highest reliability, rapid failover, extensive redundancy) using fiber and other infrastructures Provide complete virtualization capabilities, including server, network, storage, security, and application services Support data, voice, video, and control services Minimize power consumption and provide redundant/backup power Operational specs should: Serve as a key location for grid intelligence Serve as termination facility for redundant extranet communications Serve as termination facility for redundant internal core networks Be under full autonomous control by utility or their designee Provide the highest degree of monitoring and system management Justification Provides connectivity outside the utility (to customers, suppliers, business partners, emergency services, the public at large, and to off-site utility employees) with appropriate access and isolation For high-performance hosting of centralized data collection, control and analytics applications; hosting storage systems and enabling operator/worker access, Smart Grid Reference Architecture Views 49

60 Premises Networks Deployment Design specs: Specification Provide controlled access to premises-located devices Provide the highest degree of security Provide network performance commensurate with applications (from tens to several hundred kilobits, application specific latency, sufficient reliability, and redundancy where appropriate) Support data and control services Be based on IPv6 technologies Operational Specs should: Aggregate and deliver traffic from premises devices to energy controllers, displays, or gateways Use unlicensed spectrum under control of premises owner/operator Support self-discovery and auto-configuration of endpoint devices Substation Local Area Network Deployment Design Specs should: Accommodate very tightly controlled access from intra-station devices or from field area networks Provide the highest degree of security Provide high network performance primarily using fiber and other high performance infrastructures Support virtualization capabilities internal to the station as well as to the Backhaul WAN Require segregated overlay solutions for separating process bus traffic from other traffic Support data, voice, video, and control services Be based on IPv6 technologies Be capable (if desired) of supporting diversely connected WAN uplinks Support convergence such that, if desired, traffic from one environment does not harm or adversely impact traffic from other environments Provide autonomous support for low-level networking functions (such as DHCP, distributed DNS, and gateway services) Operational Specs should: Directly support devices and systems where grid logic is collected, executed, and stored Be under autonomous control by utility Provide the highest degree of network monitoring and system management Support interconnection of Field Area Networks Justification To provide connectivity to premises devices (such as inhome displays, programmable communicating thermostats, etc) and premises energy controllers or a gateway, To provide coverage to smart devices within and nearby the substation. High performance attributes: several hundred Megabit or even multi-gigabit, extremely low latency, deterministic traffic support, very high reliability rapid failover, redundancy Smart Grid Reference Architecture Views 50

61 Tier 1 Field Area Networks Deployment Design Specs should: Specification Provide controlled access from field-located devices or from subtending Tier 2 Field Area Networks Provide the highest degree of security Provide high network performance commensurate with applications (from one to tens of Megabits, low latency, sufficient reliability, and redundancy where appropriate) Support virtualization capabilities to maintain traffic segmentation Support data, voice, video, and control services Be based on IPv6 technologies Operational Specs should: Aggregate and deliver traffic to places where grid intelligence has been distributed Require the spectrum to be under autonomous control by the utility when wireless networking technologies are deployed Provide the highest degree of network monitoring and system management Support self-discovery and auto-configuration of endpoint devices Support interconnection of Tier 2 Field Area Networks Tier 2 Field Area Networks Deployment Design Specs should: Provide controlled access from field-located devices Provide the highest degree of security Provide network performance commensurate with applications (from tens to several hundred kilobits, application specific latency, sufficient reliability, and redundancy where appropriate) Support data and control services Be based on IPv6 technologies Operational Specs should: Aggregate and deliver traffic to places where grid intelligence has been distributed Require the spectrum to be under autonomous control by the utility when wireless networking technologies are deployed Provide the highest degree of network monitoring and system management Support self-discovery and auto-configuration of endpoint devices Justification To provide general multi-purpose connectivity and access services to field devices so that they can connect to data/control centers, distributed applications (perhaps at substations), and to other fieldlocated endpoints (for peer-topeer communications) To provide purpose-built coverage and access services to field devices designed to connect to data/control centers, distributed applications (perhaps at substations), and to other fieldlocated endpoints (for peer-topeer communications) Communications Architecture Standards and Technology Recommendations Table 14 lists key standards for the Smart Grid communications architecture view. Standards continually change as technologies evolve; therefore this list should not be considered a comprehensive or exhaustive list of standards and technology recommendations. Smart Grid Reference Architecture Views 51

62 Table 14 - Recommended Communications Standards and Technology Standards Purpose/Relevance/Comments IEC family of standards IEEE 1613 IPv6 Multi-Protocol Label Switching (MPLS) OSI Reference Model PHY - MAC standards typical to the various communication sub-networks: Extranet PSTN technologies (ITU-T), SONET/SDH technologies, Metro-Ethernet Data/Control Center Ethernet (IEEE family), Fiber Channel Protocol (INCITS T11) WAN SONET/SDH technologies, Metro-Ethernet Field Area Networks LTE (carrier-provided), IEEE (WiMAX), IEEE (WiFi), IEEE (Smart Utility Networks), IEEE (Ethernet) Premises Area Networks IEEE (WiFi), IEEE (Zigbee), IEEE P1901 (Power Line Communications) Describes communication principles, models, and capabilities used to support applications in electric power systems. Deals with environmental requirements for communication networks supporting electric power substations. The latest version of the Internet Protocol which has significantly more address space and enhanced networking features A hybrid data-link/network-layer packet-based transport service used in Wide Area Networks. Can support circuit-oriented or packetoriented communications. Capable of supporting deterministic traffic. Offers high degree of network virtualization. Reference model for describing communication systems using seven layers of functionality (physical, data link, network, transport, session, presentation, application). Each layer provides a common set of services to other adjacent layers. Each communication sub-network has functional requirements commonly addressed by using certain types of communication technologies. While standardization at the network layer and above helps support end-to-end interoperability, choices must be made at the physical layer (PHY) and media access control layer (MAC) largely based on geographic and environmental issues. Smart Grid Reference Architecture Views 52

63 Management Smart grid management services generally reside within the cross domain foundational services box in the lower right area of Figure 9 - Layered Services Tier View. Management Framework The Management Framework [Figure 22] presents smart grid management services as a stack of layers reaching from electrical devices at the bottom to systems management at the top. This should help the reader visualize the pervasive nature of management services and the resulting architectural scope.. Figure 22 - Management Framework Smart Grid Reference Architecture Views 53

64 The architecture is a model with four management layers (MLs). Element ML is at the bottom, the System ML and Services ML in the middle and Smart Grid ML at the top. Lifecycle and Technology Management Services are on the Element ML. Event, Asset and Security Management comprise the System ML. Performance and Policy Management is on Services ML. The Manager of Managers, performing all services orchestration, is on the Smart Grid ML. The System ML manages Smart Grid Systems. The Element ML interacts with Smart Grid Communications Elements. Management Logical Model The Management Logical Model [Figure 23] contains three views: management component, management services stack, and management synergy. Figure 23 - Management Logical Model In the Management Synergy model (upper left portion of Figure 23), the data sources are at the bottom and data consumers at the top. The Management Services (middle layer) depict the shared Smart Grid Reference Architecture Views 54

65 management services across systems and is divided into three sub-layers. The Lifecycle and Technology Management Services deal with end devices. The Configuration, Event, Security and Performance Management Services take data from the layer below, and process the data for consumption by the Policy Management Layer at the top. Various utility operations are end-consumers of management services. Operational functions request Management Services by SLAs and policies. Policy Management orchestrates all management services per the policies and the SLA. Two broad types are depicted. Data exchanged between end devices are based on raw standards. The higher data classes are exchanged between Consumers-Processes and Policy Management to ensure correct SLA and policy rules are being followed. Management Structural Model The management structural model [Figure 21] is provided to help the smart grid architect consider how to best deploy the various management services using a stylized representation of a typical utility network infrastructure. At the top of the model are elements needed to support external agencies (regulators, interchange and balancing authorities, etc.). At the bottom are the management services needed by premise devices (smart appliances, PEV chargers, building energy managers, etc.). In between are a dozen other tiers where management services may be located to support residing devices and functionality. A self-describing template used for this model is on the last page of Appendix B, Services Classes Concepts. Included are descriptions of all fourteen tiers used in the model. Smart Grid Reference Architecture Views 55

66 Figure 24 - Management Services Structural Model Management Architecture Principles The following are architecture principles recommended for developing management architecture: Management across Systems: The management architecture should support management across systems. For example, if an urgent (security) need is being addressed to push new firmware to all meters, the performance management will address contextual needs and inputs from other systems (such as Volt/VAR, fault management etc.) before the push is performed. Ease of deployment independent of scale: management architecture deployment should be ambivalent to the size of the existing systems and the new systems to be deployed. Support for legacy and emerging systems: management infrastructure must support legacy (protocols, technologies, devices) and emerging systems Smart Grid Reference Architecture Views 56

67 Exhaustive support for management services: management infrastructure shall support all Smart Grid management services (discussed in Appendix B). Management operations interface at policy level: The default operations interface of the management architecture should be at the policy level (giving policies as inputs to the management architecture), as opposed to configuration level. For example, security configuration at the element level might be ACL configuration and at a policy level. Multiple such ACL configurations along with privacy and authentication configurations can be part of a remote access policy. The security operator must deploy the management architecture to interface with the architecture giving it the remote access policy as the input as opposed to dealing with multiple configurations at multiple devices. Support for input from internal and external entities: The management architecture should support taking inputs from utility actors (such as actors from generation, transmission, distribution and consumer functions) and actors external to an utility (such as NERC, weather systems, etc) Support for disaster recovery scenarios: The management architecture should support management services for disaster recovery scenarios such as local management (with out access to central management resources) and out-of-band management Architecture Choices The following are the architecture choices made in support of the above principles: Hierarchical architecture for ease of operations: a hierarchical architecture is optimal for ease of operations (policy level management at the top and the element level management at the bottom). Distributed architecture for support of deployment: data, software (such as firmware and configuration settings) and intelligence required for Smart Grid elements must be close to end devices to avoid network bottlenecks during firmware update distributions. Therefore, a distributed architecture is optimal for Management Architecture. Modular architecture for support of scale: Management architecture (policy management at the top and element management at the bottom) requires policy management to remain stable to changes, and to element additions (Element Management Layer) to the Smart Grid infrastructure. Therefore, the management architecture should be a modular architecture with support for multiple types of management at the element management layer would fit into the policy management layer Architecture Standard and Technology Recommendations The following table of management standards and technology recommendations is provided to the smart grid architect for reference. Over time, however, innovations will inevitably diminish its completeness. The architect should always research the latest information on these topics, but this table is a good starting point. Table 15 lists important 2011 standards and technology recommendations applicable to management, each with a brief explanation of their purpose or relevance. Smart Grid Reference Architecture Views 57

68 Table 15 - Recommended Management Standards and Technology Technology SNMP V3 for get/set configurations of smart grid elements Syslog for logging capabilities of smart grid elements IEC suite for common configuration across electric and communication devices Netflow for event and behavior analysis XML based standards for complex device management NETCONF SSL and SSH for device-level configuration Justification SNMP V3 has embedded security and provisions for encryption SNMP is widely deployed across network devices of various technologies. Syslog standard can be exported securely. Many parsing functions are available for syslog. IEC is well-defined and becoming the standard for electrical and the communication devices that interface with those electric devices. Netflow is a simple protocol. Netflow is suitable for behavior analysis. XML is becoming the standard for: Substation Configuration Language: CIM: CIM is based on XML SOAP for configuration NETCONF XML is slow because of the parsing involved, NETCONF is the IETF protocol for manipulating and installing configuration files base CLI-based configuration should be done through secure versions of management protocols such as SSL and SSH. Smart Grid Reference Architecture Views 58

69 6. Summary Introduced less than 50 years ago, Information and Communications Technology (ICT) Architecture remains a relatively new discipline. Unlike architectural disciplines dating back to ancient times, those that produced the many human artifacts we use and tend to take for granted today, ICT has relatively limited experience with the materials it employs. While many classes of information and communications materials are now stable and well understood, some ICT aspects continue to evolve quickly. Thus, it is not surprising that Smart Grid Architecture, being even more recent than ICT, must proceed while its building blocks remain untested in large-scale utility deployments. This document is one of the first end-to-end Smart Grid Reference Architecture descriptions developed in North America. Unlike many existing technology-dependent smart grid frameworks, the intent of this work was to develop technology-neutral smart grid architecture, and to document methods and recommendations for utilities continued reference as smart grid technologies and applications mature. Several conclusions became evident while this document was being debated and written: 1. The architecture and resulting infrastructure must be based on layered services architecture principles. As new ideas and applications emerge, it is critical for the architecture to not only evolve with them, but to support them with little or no re-working. 2. Interoperability is key. Many vendors will develop pieces of equipment, software and other items to be integrated with the grid. True interoperability is critical to reducing the cost of adding each item to the mix. It also lowers the threshold for testing innovative ideas if the interoperability testing setup cost is too high, new ideas die on the drawing board. 3. Data flows from many sources to many applications. This data interweaving requires an ability to move massive volumes of data quickly. Data exchange between applications must also be segregated based on operational needs. An infrastructure with robust data networks, adaptors, and service busses is critical for a utility to have a fully functional smart grid. 4. A utility based on silo-optimized business functions is incompatible with grid renewal. As smart grid systems must interoperate seamlessly, so must the underlying business functions. This architecture document can help the Old Grid evolve into a Smart Grid as legacy grid infrastructure is merged with the latest ICT. It recommends high-level goals and principles to guide those tasked with developing smart grid architecture. Additionally, it describes a highly flexible, adaptive architecture critical for grid transformation. If properly planned, existing ICT infrastructure operations will continue while infrastructure complexity remains manageable as capabilities are added. Demands on each utility s smart grid will change as results from experiments and demonstration projects become available. Smart grid architectural demands will also change as new applications evolve. Finally, many unknown factors will impact the Smart Grid ICT architecture over time. For all these reasons, this document must be revised periodically to remain useful. Summary 59

70 NOTES Summary 60

71 Appendix A. System of Systems Design Patterns Smart Grid System-of-Systems Architectures System Evolution to Guide Strategic Investments in Modernizing the Electric Grid K. Mani Chandy, California Institute of Technology Jeff Gooding, Southern California Edison Jeremy McDonald, Saker Systems Introduction The electrical power system which has served humanity efficiently for a century must now evolve to meet changing requirements: increasing renewable energy sources, decreasing fossil fuel usage, managing greater total demand, using electricity to fuel transportation, enabling more customer control of both demand and supply, dealing with security threats, and adapting to disruptive technologies. An established industry must adapt to rapid, unaccustomed rates of change. System of Systems Architectures Smart Grid designers face the challenge of planning the evolution of the grid s architecture from its current instantiation to meet the needs of a changing uncertain future. This challenge can be met by managing the evolution of the grid as a System of Systems (SoS). A plan for the systematic evolution of Smart Grid architecture, as a System of Systems, should be the basis for: developing requirements and standards, making design decisions, procuring solutions, and managing the Smart Grid over the coming decades. A SoS is defined as a collaborative set of systems in which its components: 1) Fulfill valid purposes in their own right, and continue to operate to fulfill those purposes if disassembled from the overall system and 2) are managed (at least in part) for their own purposes rather than the purposes of the whole. The components systems are separately acquired and integrated to form a single system but the components maintain a continuing operational existence independent of the collaborative system. Consequently properties, which do not belong to any of the constituent parts, will emerge from the combined system of systems. Moreover, the system of systems evolves as constituent systems are replaced. System of Systems Design Patterns Appendix 1

72 Central Principles of System of Systems Architecture Architects must enforce the following two central principles of SoS to ensure that the Smart Grid is not overwhelmed by change: The complexity of the SoS framework does not grow as constituent systems are modified, and SoS concepts for integrating constituents remain unchanged even as components are added and removed. The constituent systems do not need to be re-engineered as other constituent systems are added, removed, or replaced. These ideas are not new and form the basis of engineering design in many domains including software. We show how to evolve the architecture of the Smart Grid in a systematic, evolutionary manner, by adhering to these well-tested principles. This paper describes four Smart Grid SoS architecture patterns and their benefits and risks. In this paper we propose a specific evolutionary trajectory for the architecture of the grid as it takes on characteristics of these four patterns. Tradeoffs between different evolutionary trajectories are discussed, and the risks in the specific plan we propose are described in detail. Trends that Impact Evolution of Grid Architecture We next list trends in technology that impact the Smart Grid and show how these trends impact the systematic evolution of grid s architecture. A System of Everything: The Smart Grid is evolving to include control of many devices that were not considered in designs of the current grid; these devices include distributed energy sources and storage, electric vehicles, and appliances. Smart Grid architecture must consider integration at some level of widely different kinds of systems that deal with all aspects of life from transportation to healthcare. We use the hyperbole, system of everything, to make the point that the grid is one of the focal points by which individuals and organizations monitor and control their lives. The Penetration of the Internet and the Web: Large segments of the public use the Internet as a core technology for their work, social networking and recreation. Widespread access to broadband and increasing use of smart phones and tablets has resulted in large segments of the public, especially the young, viewing the Internet as a system of everything. This view is likely to strengthen as today s young enter the workforce. Worldwide penetration of Internet and cellular data technologies will continue to make these technologies more powerful and affordable. The evolution of the Smart Grid architecture will reflect evolution of Internet architecture because the public will not want two competing systems of everything. Moreover, consumers and organizations System of Systems Design Patterns Appendix 2

73 will want tighter control of their electrical devices as energy tariffs based on time of day become common and they will expect to manage their devices using the same Internet protocols that they use for other activities. Continuous Evolution of a Heterogeneous Smart Grid: The information infrastructure supporting the grid will remain highly heterogeneous for several reasons. Different grid functions have widely different systems characteristics such as requirements for security, timeliness, and bandwidth; for example these requirements are very different for fault protection and metering, and for demand response in homes and data centers. The Smart Grid will evolve continuously over decades; there will not be a single massive replacement of the current grid. Information technologies for sensing, metering, actuation, communication, and computation are developing continuously and rapidly. So, there is no ideal single point in time for a wholesale replacement of all devices. The architecture must be designed for continuous adaptation. Next we discuss the costs and benefits of using different SoS architectures at different points in the evolution of the Smart Grid Smart Grid SoS Architecture Patterns The Silo Architecture The current SoS architecture of the electric grid can be characterized as a collection of silos. Different functions of the grid such as billing and energy distribution use different information silos that are integrated by a thin IT layer. The silo architecture worked well for utilities for decades because each silo served the needs of a business unit, and different business units in a utility had very different needs. Utilities operated efficiently with little integration across silos. System of Systems Design Patterns Appendix 3

74 Figure 25 - Silo Architecture The silo architecture, though adequate in the past, will be inefficient for the future for several reasons. One reason is the increasing demand by a number of stakeholders for a greater control of energy usage. From the 1960s to 2010, utilities and ISOs controlled generation and distribution, and they specified well-defined interfaces to consumers: Customers turned switches on and off, paid bills, and called utilities when power failed. In the future utilities will coordinate activities by multiple stakeholders across multiple interfaces. For example, integrating distributed generation requires new interfaces. If the silo architecture is retained then separate interfaces will have to be developed between each type of stakeholder and each type of silo. Moreover, these separate interfaces will have to evolve separately as requirements evolve. Our challenge is to design an evolutionary path from today s silo architecture to an evolving SoS architecture. System of Systems Design Patterns Appendix 4

75 Integration using Enterprise Services Buses Figure 26 - ESB Architecture A step in the evolution is to integrate the back office using an ESB. Some utilities are taking this step. Though the step appears simple, it requires considerable cost, time and effort from IT staff and business units. Most importantly, integration using an ESB requires discipline that enforces enterprisewide standards on data models and IT services. If this discipline is not enforced, the ESB merely serves as an integrated physical communication opportunity, but all the problems of the silo architecture remain. Integration of the back office results in considerable efficiencies in utility operation. Back office integration does not, however, solve problems of multiple interfaces between different types of agents System of Systems Design Patterns Appendix 5

76 who will participate actively in consuming and producing electrical energy. Moreover, the ESB architecture doesn t move towards the utility customer s need for an integrated system of everything. Thus, the ESB architecture is a good step, but not the final step, in the evolution of Smart Grid architecture. Adapter Architecture Figure 27 - Adapter Architecture A possible next step in the evolution of the Smart Grid SoS architecture is that proposed for networkcentric warfare by the Department of Defense. A central feature of this architecture is that it provides each participant a user-defined operational picture (UDOP) for situational awareness. As a battlefield situation unfolds, possibly in unpredicted ways, UDOP provides each warfighter with the information that he or she needs, while ensuring that warfighters are not overloaded with data irrelevant to their System of Systems Design Patterns Appendix 6

77 operations. DoD is using network-centric architectures to integrate army, navy, marine and air force operations to help provide overall situation awareness. Major benefits of this architecture are the enforcement of protocols that guarantee: Security throughout the system and Rapid delivery of high-priority information. The DoD enforces standards on its vendors: New IT components are designed and tested to guarantee system-wide security and performance. The challenge for utilities is to derive the benefits of DoD s network-centric architecture while dealing with a marketplace that is very different from DoD s operational environment. The Smart Grid consists of a collaboration of multiple utilities and ISOs. As the electricity grid gets increasingly integrated across states and even countries, the number of collaborating utilities and ISOs will increase. New types of stakeholders, such as manufacturers of electric cars, distributed energy generators, and energy storage devices participate in designing new interfaces to the grid. Control by a single agency of multistate, multi-national, multi-vendor integration, while possibly desirable, is more difficult to implement in the Smart Grid than in the DoD. The adapter architecture has critically important features; notably it supports the continuous evolution of a heterogeneous system of systems. The architecture does not, however, adequately meet the public s needs for a system that integrates all devices. Nor does the adapter architecture sufficiently exploit the huge opportunities provided by open Internet technologies. Protocols layered on top of the Internet are not ideal for all applications; however, these protocols will evolve over time to meet needs for security, performance and other features required by many applications including the Smart Grid. Though the adapter architecture goes a long way towards meeting Smart Grid requirements, the architecture doesn t go far enough to meet the technological or societal needs of a utility s customers. Architecture Based on Open Standard Service Mechanisms A great deal has been written about Service-Oriented Architecture (SOA). A definition of SOA, offered by Roy Schulte of Gartner who coined the term, is that an SOA system satisfies the following five principles. 1. Components can be added, replaced or modified individually without affecting the remainder of the system. 2. Components must be distributable, i.e., run on arbitrary servers and communicate with each other by messages or service invocations. 3. Component interfaces must be defined using standard metadata and the interfaces must be discoverable by application developers. 4. A component can replace another with the same interface. System of Systems Design Patterns Appendix 7

78 5. Services can be used multiple times by disparate applications or the same application. These characteristics also satisfy the two main principles of System of Systems architecture. Services may be organized in business domains so that some services are only available to others within the same domain; however, a central point of standard services is common metadata for service interfaces regardless of the business domain in which the service lies. The Adapter Architecture is a service-oriented architecture. The central differences between adapter architectures and one based on standard open services are the protocols by which services are invoked. Since the IT infrastructure for the Smart Grid will need to take steps towards satisfying customers needs for a System of Everything, we are faced with two options. Ensure that the mechanisms for specifying, invoking and maintaining services for the Smart Grid are the same as, or consistent with, mechanisms for invoking other services. Have two mechanisms for managing services, one for the Smart Grid and the other for everything else, and build bridges between Smart Grid and everything else. There are advantages and disadvantages for both approaches. An advantage for the two-mechanism approach is that a Smart Grid IT infrastructure is designed specifically for the particular needs of the Smart Grid. A disadvantage of this approach is that all the thousands of utilities and other agents participating in the Smart Grid will have to adopt either a single standard (or a very small number of standards) designed specifically for the Smart Grid, and build integration layers between the Smart Grid standard and protocols to manage both business and personal needs in other domains. System of Systems Design Patterns Appendix 8

79 Figure 28 - Service-Centric Architecture The development of a Smart Grid system of systems based on Internet (e.g. REST) or Web-services protocols requires an understanding of the key points of interoperability across the entire Smart Grid. The silos in the current infrastructure can be architected in layers, with layers across different silos having the same interface. The new architecture must specify standard interfaces at each layer so that different business functions are architected in the same way and reuse components. The architecture must also specify common services such as network management, security, and diagnostics, and these common services should be accessible to all systems and devices used to execute different business functions. System of Systems Design Patterns Appendix 9

80 Summary Strategic investments in modernizing the grid must be based on a plan for transitioning today s silo architecture to a system of systems architecture based on widely adopted standards, common services, and loosely coupled systems. Strategic investments will pay off only if they designed for a carefully planned trajectory of grid architecture. Successful execution of a transition plan requires early and continuing investments in Smart Grid standards, unified infrastructure and the co-evolution of processes to support migration from centralized to distributed operation. Massive changes in architecture, operations, and training will be required over the transition period. So, a utility should adopt an evolutionary approach that doesn t exceed the utility s capacity in these domains. A transition plan that some companies have adopted is to first move from a silo architecture to an enterprise service bus (ESB) architecture and then move incrementally to an open-standards based architecture, developing and adopting standards and common services in steps. The utility should, however, have an overall architecture transition plan, including designs for system-wide security and system-wide timeliness at every step of the plan, before making incremental changes. Each of the four architectural patterns discussed in this paper has benefits and risks. Each transition from one architectural pattern to another has substantial costs, takes time, and requires expertise in both the grid and IT architecture. Utilities must, however, develop architecture plans before making strategic investments in the grid. System of Systems Design Patterns Appendix 10

81 Appendix B. Services Classes Concepts Applications Services As discussed previously, the utility industry is being transformed by Smart Grid initiatives.. This transformation affects functional business processes, as well as associated technology and applications. These changes will lead to a re-engineering of how these applications are used and interact with other domains. Applications will be transformed to leverage shared services to better allow the utility business domains to interchange available information. Interoperability and loose coupling between application functionalities is best assured if the application services interact through well-defined interfaces based upon standards, e.g. IEC and IEC Smart Grid architects, over time, will disaggregate monolithic systems into smaller atomic functional components. These will be used to aggregate, compose, choreograph and orchestrate business processes leveraging smaller atomic functional components. The optimizes assets and operates efficiently baseline Smart Grid goal requires applications to be designed for maximum flexibility and reusability. If applications are adequately service enabled, the design of flexible, smart grid processes can leverage industry standards such as BPEL (Business Process Execution Language). For example, suppose a giant distribution management system is decomposed into many smaller functional components, each with small functions implemented as self-contained services. A utility functional section can, through BPEL, rapidly devise business processes by orchestrating a set of services to perform a business process. Since the services are designed for re-use, another business group could devise a similar process using the same services orchestrated in a way to best meet its particular business needs. Business analysts can design a multitude of different business processes by assembling the same functional components into different combinations. This approach provides analytics needed to measure process efficiency since properly designed services include introspection and self-testing. Analytics to tune process efficiency can be implemented with each process piece, and re-used by every business process using the services. P1 Service A Service E P1 P2 P3 P4 Monolithic Application P2 P3 Service B Service C Service F Service G Service I Service J Service K P4 Service D Service H Figure 29 - Dis-aggregation of a Monolithic System Services Classes Concepts Appendix 11

82 Functional Aspects of Application Services The Operations functional domain manages the movement of electricity through the efficient operation of the power system. It therefore impacts many other business domains: a. Network Operations: includes supervision of substation topology and control equipment status, network connectivity, loading conditions, fault management, outage management and location of field crews. b. Network Extension Planning: develops long-term plans for the adequacy and reliability of the electrical networks needed to fulfill evolving energy usage needs. c. Operations Planning and Optimization: includes the definition, preparation and optimization of workflows, plus the operational sequences required for the maintenance, monitoring and control of networks. d. Metering: performs reading of the information recorded at customers service points; sends alerts and notifications; controls customer equipment interfaces. e. Record and Asset Management: includes electrical substation and network assets either owned by the utility or having legal responsibility for. f. Maintenance and Construction: addresses functional needs for equipment to perform better and extend its service life. g. Customer Support: manages operational aspects of customer interfaces such as trouble calls, point of sale, account management, etc. h. Enterprise Systems: supports the overall utility businesses, including, but not limited to, human resources, corporate finances, and supply chain management. i. Bulk Energy Management: serves Generation and Transmission Control, Topology Processing, Account Settlement, Market Operations, Load Management, Energy and Transmission Scheduling, Dispatcher Training Simulation, SCADA (Supervisory Control And Data Acquisition), Alarm Processing, etc. Services Classes Concepts Appendix 12

83 Figure 30 - Functional Services Organization As discussed above, the Operations domain boundary includes an edge to most utility domains and thereby can provide a large set of application services. As an illustrative example, consider the application services offered by the Network Operations Planning and Optimization. The related services can be classified into three sub-categories, which can be further decomposed into various functional, modular and reusable components or services: Services Classes Concepts Appendix 13

84 Figure 31 - Network Operations planning and optimization The Network Operations and Optimization Application Services are: Load Forecast Service: The Load Forecasting function predicts short-term, mid-term and long-term system loads and maintains a real-time forecast and a study forecast. The real-time forecast is based on actual historical load and weather data and generates a load forecast for the current minutes, hour and day. The study forecast uses a completely independent set of historical and predicted data the operator may configure to evaluate hypothetical situations in the future. Power Flows Service: This service provides the calculation engine for external systems to estimate the network conditions based on provided network status and measurements. This service is used in real-time to provide dispatchers with an accurate and precise estimation of current network state. This service is also leveraged by advanced network features requiring real-time base power flow results for further analysis, (e.g. Contingency Analysis, Volt/VAR Control, or FLISR Fault Location, Isolation and Services Restoration). This service can also be leveraged by Network Planning engineers to determine the effects of control actions (breaker switching, tap changing, and interchange adjustments). Contingency Analysis Service: This service enables the real-time study of the effects from a system component failure. Contingency analysis relies on the Incident Simulation Service to model the effects of identified contingencies. Once all relevant contingencies have been simulated, the Services Classes Concepts Appendix 14

85 Contingency Analysis Service ranks the results based on the current network conditions to inform the grid operators of areas of possible risks or congestions. The Contingency Analysis Service can also be in used in an engineering exploration or training environment. Short-Circuit Analysis service: This service is responsible for analyzing short circuits in transmission and distribution networks. These include single line to ground, line to line, double line to ground, three lines to ground, single line open, double line open, etc. Optimal Power Flow service: This function allows dispatchers or network planning engineers to optimize power system control actions based on given criteria (flows of real or reactive power, switching of compensation devices, optimal settings for voltage control, etc.). In optimal power flow, the control actions are automatically predetermined within the limitations of the power system. Supply Restoration Assessment Service: This service analyzes the switching options after a network fault is identified in order to restore the power for as many customers as possible in the shortest time possible. In distribution systems, this service is generally invoked by the Fault Location, Isolation and Services Restoration service to provide alternate switching plans for the operators to manually or automatically select for restoring power to customers when the normal feeder configuration does not allow the supplying power to affected customers. Switching Simulation Service: This service enables the simulation of switching operations. This service is leveraged by many other services, in both real-time or study environments. For example, the real-time Supply Restoration Assessment Service calls it to simulate the switching steps necessary to isolate faulty devices or restore power to customer. Another example, in the training environment settings, when the trainee operators issues a supervisory control, the Operator Training Simulator calls the Switching Simulation Service to operate the selected devices. Incident Simulation Service: This service provides the ability to simulate and recreate an incident on the network for analysis or training purposes. This service leverages the Switching Simulation Service to perform switching steps on the network and on the Power Flow Service to compute the resulting network conditions. The Incident Simulation Service is leveraged in both real-time and study mode environments. For example, the Operator Training Simulator heavily relies on this service to drive the simulated network state and conditions. Weather forecast analysis service: Weather is monitored to predict impacts on the electrical networks, especially where outages are likely to occur due to heavy storm activity. Temperature and wind speed are sometimes used to calculate dynamic load limits on electrical network assets. Fire risk analysis service: It analyzes the risk of fire in the network. For example, thermal rating of network equipment. There is a certain temperature for which transformers may be at risks of malfunctioning, failing or worst, exploding. Define operational limits service: It defines operating limits for monitoring operations of the transmission and distribution facilities. Thermal ratings of network equipment and lines: Changes in electrical asset performance limits based on temperature and wind speed and current season. This is distinguished from Asset Investment Planning (AIP) counterpart, which includes new construction and re-conductoring. For all these services to properly interact and fulfill the utility objectives, an integration layer is required to ensure the orchestration and communication of these services. The next section describes these integration services. Services Classes Concepts Appendix 15

86 Integration Services Integration services and patterns are very important in reducing ICT complexity while allowing it to quickly and easily deliver on business needs while enabling continued innovation. Integration services leveraging the enterprise service bus pattern (1) offer ways to cut down point-to-point connectivity; (2) maintain a higher level of reliability; and (3) improve flexibility in the architecture. The presence of an ESB is fundamental to simplifying the task of invoking services services can be used wherever needed from wherever they reside within the enterprise; used without specific knowledge of where the services reside or how to transport requests across the network to invoke those services. Some of common capabilities needed in integration services are routing and brokerage, mediation, protocol conversion, namespace translation, service virtualization, and aggregation. Business Process Orchestration and Choreography Service Business process choreography and orchestration capabilities, associated with the composition and coordinated execution of services, provide flexibility in building new processes and managing process change. Business process execution language (BPEL) is one of the standards used for building orchestrated ICT-enabled business processes. Once developed, the process is deployed on servers running a BPEL engine and utilized by business systems shepherding the processes for the enterprise. Choreography is used when more complex behavior is needed, with each atomic-process reacting to the actions of other processes using pre-established heuristics, without central coordination. Business Rules Services As ICT Systems began supporting ever increasing, ever changing business processes, events and decisions, the need to identify and manage business rules became obvious. The rules needed to reflect business policy in a form comprehensible to business users and executable by a business rules engine. Separating the business rules from application code simplifies system changes, allows re-use of the rules across applications/processes, and supports rapid rules selection and execution. Registry and Repository Services The registry functions provide publication of metadata about the function, requirements, and semantics of services allowing service consumers to find services or to analyze relationships between services. The repository function provides the ability to store, manage, and version service metadata. Business application services Business application services provide ability to deploy and manage robust, agile and reusable SOA business applications and services of all types. These are service components designed specifically as services within a business model and represent the basic building blocks of the utility s business design services not decomposable within the business model, but composable to form higher level services. This allows creation of new services and service enablement of legacy packages and applications. These Services Classes Concepts Appendix 16

87 services allow the user to develop innovative applications through their support of open standards and programming models, including: OSGi, CEA, JPA, SAML, SCA, SDO, SIP, Web 2.0, and XML. Gateway Services Gateway services provide filtering and data stream processing, receive all events from the device and send control commands to device. They also act as interaction services for machine to machine interaction, control access to applications, services and data based on customizable roles and rights thereby enable consistent enforcement of security and governance policies. These services also enables web integration services through the support of Web 2.0 technologies with JSON filtering and validation Complex Event Processing (CEP) Services Complex Event Processing services provide the ability to simultaneously access and analyze thousands of data streams from both inside and outside the corporate firewall and delivering the result directly to business application services. It supports events conformant to the Common Base Event specification and other messaging formats. It extends the one message at a time paradigm by detecting complex situations involving composition of messages/events sensitive to context (i.e. semantic, temporal, spatial, spatio-temporal, or scope). Workflow services A workflow service provides automation of a series of tasks assigned to application users based on their roles. When the application containing the workflow is instantiated, a user is assigned a task based on his or her role. This service is used for developing simple service composition. When complex, parallel service composition is required, workflow is best served by choreography services. CIM Binding Services The Common Information Model (IEC, 2003) is a key standard in the utility business domain. To expose a business service, this binding service maps the CIM model with backend data attributes. This service provides a CIM data model to handle the large portfolio of existing and extensible CIM classes. Visualization Services Visualization services include presentations of grid topology, alarms, switching state, and business capabilities. Presentation often is tailored to role-sensitive contexts system behavior and visual cues adjust based on user job role and, perhaps, user location. To address the risk of data overload, presentation services can help transform the flood of smart grid data into nuggets of information presented in an optimal format for grid operators to make appropriate decisions. Thus, visualization services are essential to Smart Grid decision support systems. Services Classes Concepts Appendix 17

88 Web Integration Services There are many characteristics of Web Integration (i.e. Web 2.0) services such as web as a platform, harnessing collective intelligence, lightweight programming model, software above the single device, light weight user interface apply to smart applications. There are multiple applications of concepts of these services within utility domain (i.e., mashups) to build visualization and situational services, aggregation of usage, including PEV charging usage and customer information etc. Cloud Services The cloud computing architecture is a flexible computing platform implemented with a highly virtualized, automated and service-oriented design. Utility companies can leverage cloud computing for rapid, real-time access to considerable computing power, immense storage and numerous applications. By using cloud computing, utilities can improve new application development and deployment and quality of service. It also enables prompt response to evolving consumer priorities and market challenges on a utility s core products. As agility is key to future business success, and cloud computing provides ICT agility, cloud services may fit well in utility System of Systems architectures. Cloud computing s inherent reliance on standardization will increase operational efficiency. Public clouds drive process standardization by identifying scalable workloads able to be managed in mass. Private clouds leverage the scale inherent in existing utility hardware by dramatically improving asset utilization. Rather than deploying and maintaining multiple application instances, cloud computing enables standardization on a single instance. Standardization on this scale significantly reduces labor and other ICT operating expenses while increasing availability. Similarly, highly virtualized infrastructure reduces ICT capital expenses, consolidates data center resources and reduces investments in hardware and facilities. Non Functional Aspects of Application Services The issues of High Availability, Performance and Scalability are key architectural concerns for all enterprise architects, particularly for Smart Grid architects. As enterprise ICT architectural approaches are increasingly applied to the grid, solution resiliency increases in importance. These issues are highly interrelated and often impacted by technological, temporal, spatial and budget constraints. Additionally, there is limited precedence for the Smart Grid architect to rely upon, and the domain is rapidly evolving. This section addresses these aspects by defining terms and their inevitable trade-offs. Performance and Scalability Performance has two facets. The first is the speed the system supports a single request. The second, system throughput, is the number of simultaneous requests supported with acceptable response times. Scalability is a measure of how well the system accepts higher volumes with acceptable performance and availability. A scalable architecture allows a system to grow with reasonable hardware and Services Classes Concepts Appendix 18

89 software accommodations. In contrast, growth of a non-scalable architecture requires major changes to subsystems, hardware configurations, or new processor types. Performance and Scalability approaches are highly interdependent. High availability and performance goals can be achieved by paralleling system components, resulting in load reduction and response improvement for any element. Generally, additional units can be easily added if the architecture has the units paralleled. At the same time, improved availability is enabled by the presence of redundant hardware in this architecture. An architecture using parallel elements can also apply dynamic Quality of Service rules to ensure critical transaction completion when components are compromised. Analytic Services Analytics are data transformations used to extract information actionable by systems and people. As such, some analytics are cyber (machine-to-machine), while others are used for decision support after appropriate presentation (machine-to-human). The scope of analytics includes: Basic database queries for routine operational and business process issues Real time streaming data transformations used in closed-loop control systems Large scale data visualizations used for operator decision support KPI s and dashboards used for executive review Reports for submission to regulators Data mining for system planning and asset management Many Smart Grid analytics are used to transform raw grid-derived data into a high-level depiction of grid state, supporting total situational awareness. Grid state includes knowledge of operational electrical parameters, device/system status, power quality, stress factors, and loading conditions. This grid state information is embedded in low-level data too voluminous and detailed to be comprehensible or actionable. Analytics extract the essential meaning from the data at the numerous grid intelligence tiers, from Protection & Control to System Planning & Optimization. As information and system complexity increase, presentation sophistication must necessarily increase. For some target audiences, reports or dashboards are inadequate. Advanced visualization capabilities are often needed to present analytics derived from sets of large, complex and dynamic data. Types of Analytics A wide variety of Smart Grid analytics exist, and more are constantly being developed. These analytics can be categorized in multiple ways, but one in particular is helpful. Figure 32 below depicts a taxonomy of Smart Grid analytics. Services Classes Concepts Appendix 19

90 Figure 32 - Smart Grid Analytics Taxonomy The taxonomy has six major classes of grid analytics, all driven by data from grid devices and systems. The classes support three high-level intelligence categories: Technical and Engineering, Operational, and Business. Signal Analytics Signals are the lowest level data coming from grid sensors. Therefore, Signal Analytics are often the "earliest" analytics (the ones first applied to raw data) and generally take the form of digital signal processing functions. The Signal Analytics profile follows: Example Usage: transform voltage and current waveform samples into Root Mean Square (RMS) values, along with real and reactive power, Total Harmonic Distortion (THD), and other relevant characterizations Data Frequency: real-time, may be hundreds of waveform samples per cycle over multiple channels for three phase voltages and currents. Data Types: integer digitized values, later converted to float engineering units Analytic Processing Frequency: ranges from per sample up to typical SCADA data rates (e.g. four second cycle time) Event Analytics Events are discrete, real-time messages from grid components caused by temporal or value limits being reached (logs, alarms, alerts). The Event Analytics profile follows: Services Classes Concepts Appendix 20

91 Example Usage: processing power outage notification message bursts from smart meters during an outage or voltage sag Inbound Data Frequency: real-time, can be thousands of events within a few seconds Data Types: numerous event message formats Analytic Processing Frequency: real-time to manage bursts with low (millisecond to second) latency without message loss Result Set Publish Frequency: per event, asynchronous State Analytics State data from the grid supports real-time utility network analysis, monitoring, and control. The State Analytics profile follows: Example Usage: monitoring and control of grid power flows Inbound Data Frequency: real-time, from per cycle up to typical SCADA data rates (e.g. four second cycle time) Data Types: various SCADA and sensor formats Analytic Processing Frequency: real-time as per control system requirements; can be SCADA cycle time or as fast as GOOSE message times (4 msec max latency) Result Set Publish Frequency: SCADA rates to minutes for most processes; milliseconds for protection and control functions Engineering Analytics Engineers need analytics to perform long-term planning and operational analysis. The Engineering Analytics profile follows: Example Usage: circuit and substation planning Inbound Data Frequency: real-time - mostly SCADA data rates Data Types: usually floating point values from circuits, customer usage data records, maintenance event logs, waveform files Analytic Processing Frequency: quarterly, yearly, multi-year Result Set Publish Frequency: on demand as planning and analysis processes are performed Operations Analytics Operations depends on analytics for short-term optimization of asset effectiveness. The Operations Analytics profile follows: Example Usage: all aspects of asset management for utilities, including asset health assessment via remote monitoring, asset stress accumulation monitoring for support of condition-based and predictive maintenance, and asset utilization, both in real-time and for system planning. Inbound Data Frequency: real-time at SCADA data rates or slower; directed mostly to historian databases, except for data used in load balancing and optimization. Data Types: floating point sensor readings and integer operations counts Analytic Processing Frequency: SCADA rates for real-time control functions; as needed for analysis reports Result Set Publish Frequency: same as Analytic Processing Frequency Services Classes Concepts Appendix 21

92 Consumer Analytics Utilities better serve their customers by analyzing data on electric producers and consumers (prosumers). The Consumer Analytics profile follows: Example Usage: all aspects of prosumer interaction with the utility, from billing and complaint management to interface for distributed energy resources, demand response (DR), and pluggable electric vehicle charge management. Inbound Data Frequency: rates for meter reading, DR feedback and prosumer interactions are slow compared to grid operations; may be minutes to months Data Types: meter usage data, DR data (Smart Energy Profile), event log messages Analytic Processing Frequency: on demand, mostly monthly or longer; some analytics for DR may be from minutes to hours Result Set Publish Frequency: on demand Smart Grid Data Classes Given the many potential sources of power grid data, for analytics and data management purposes it is helpful to classify them into seven categories. These categories aid the proper implementation of analytics, the design and placement of data persistence elements, and the sizing of communication network elements. Operational data Operational data is mostly real-time state measurements of circuits and devices. As this data changes from moment to moment, most analytics are concerned only with present data values. This data is persisted in various ways, primarily by overwriting old values. A log may be kept by placing snapshots of operational data in a time series (historian) database. Non-operational data This data category has telemetry used to monitor remote assets: equipment condition, environmental monitoring, stress indicators, etc. Most of this data is collected on a regular basis and persisted in a time series database. Meter usage data Besides its obvious use for meter-to-cash, meter data is often input to various technical and business analytics. While some analytics could use the default meter data repository, this approach may not scale well for more advanced, real-time analysis. Alternate data services could therefore be necessary for some metering analytics. Event streams Many Smart Grid devices produce asynchronous event messages. Analytics for such data streams may need special processing tools (Complex Event Processing engines) to quickly respond to periodic data bursts from grid devices. Event messages may also be logged for post-mortem analytics. Services Classes Concepts Appendix 22

93 Waveform data Waveforms are often produced by grid devices based on some event or condition. This data are generally collected into files (COMTRADE or PQDIF formats) for post-event analysis. The files are typically batch transferred into a waveform repository for later use by analytics tools. Analytics as data Many analytics are real-time data streams, especially those used for line sensing and remote asset monitoring. These low-level output streams can be inputs to other analytics. They can also be stored in a time series data store for later analysis. Meta-data A wide variety of grid environmental data give context to actionable information. The architecture must provide, persist and distribute pertinent meta-data for use by analytics. Analytics Properties and Issues Architects should understand fundamental analytics properties and usage issues to properly align analytics into Smart Grid architecture. This understanding will reveal analytics as more than simply a set of services provided to the Smart Grid System of Systems. Analytics as filters Analytics may be viewed as both filters and data reduction mechanisms. Raw data from grid devices can be too voluminous for centralized processing. Low-level analytics services effect a data reduction by extracting useful information to characterize larger data sets while preserving essential content. Event processing as analytics Many grid devices generate asynchronous event messages based on a variety of physical and virtual events. Such event streams contain valuable information but typically arrive in large bursts needing low latency handling. As most utility systems cannot deal directly with such data, a complex event processing (CEP) analytics tool is used to simultaneously extract core information while throttling or filtering event message bursts. An example is the processing of power outage notification message bursts from smart meters; CEP is used to reduce the burst of raw messages to a single informative event message representing the underlying physical event. In addition, grid sensor event data streams can also be marshaled by CEP, such as Phasor Measurement Unit (PMU) data frames. When processed through CEP, PMU information can be used to detect, characterize and locate fault and pre-fault transient phenomena in real-time. Real time analytics for protection/control Many grid analytics support real-time protection and control operations. Since these analytics are often characterized by very low latency (as fast as sub-cycle), the data must be processed faster than any Services Classes Concepts Appendix 23

94 database persistence can be applied. The data source is therefore connected directly to the analytic, and the analytics output routes directly to a machine consumer. Telemetry monitoring Much of Smart Grid data consists of remote asset and grid state monitoring telemetry. These produce high data volume with a steady and relentless delivery. Under normal operating conditions, the data is unremarkable. As monitored parameters trend away from nominal, actions should be taken, but the large data volume makes it impractical for people to constantly watch the data. A solution is to have analytics agents continually scan the data for specific conditions or trends, then issue appropriate alerts and notifications. Defining all possible alert analytics rules is impossible, therefore the analytics agents should be configurable, with subscriptions made or dropped as needed. End users of the analytics should also be allowed to perform some configuration. The Smart Grid Architect will need to secure this flexibility by providing appropriate access control via role-based identity management. Visualization Many Smart Grid analytics involve extensive numerical data used in making operational decisions. The need to present information in an operations-optimized format makes visualization a crucial element of the analytics architecture. While its overall usefulness to analytics makes visualization is a core analytical tool, it is also useful to other grid services (data synchronization, workforce enablement, customer empowerment). This therefore places visualization as a cross-cutting asset in the Smart Grid architecture. Concatenated analytics Many analytics are best structured in concatenated form, where output from one analytic becomes the input to another. For Smart Grid, this can occur in one of two ways. In a distributed architecture, individual analytic agents may cooperate in a calculation, with each agent contributing a portion of the result. Alternately, analytics may be structured in tiers. An example would be the low-level analytics converting power quality (PQ) information into measures of grid asset stress. The stress analytic output are inputs to a second tier of analytics converting stress to abstract measures such as Loss of Life, Expected Time to Failure, and Asset Failure System Risk. These abstract measures are then made available to an asset management application incapable of processing the original low-level PQ data. Sourcing of data for analytics A key issue for analytics architecture is how to best minimize the number of new sensors. Properly chosen and located sensors can produce grid data capable of supporting many separate business needs when processed through multiple analytics. This multi-faceted aspect makes necessary a sensing strategy to accommodate business requirements while minimizing investment in new sensing infrastructure. Services Classes Concepts Appendix 24

95 Delivery of analytics results In good grid architecture, analytics results will be distributed in various ways. Many analytics can be used by more than one business process (1:N delivery model). Some business processes require the combined results of many analytics (N:1 delivery model). Finally, should both situations fit, the N:N delivery model could be employed. These delivery models need to influence the modeling of any business process reliant on analytics. Persistence of analytics Analytics output, if treated as data, may need some type of persistence. One simple approach is to store analytics outputs in a time series database (historian). Not all analytical results need to be kept for long periods of time. For example, analytics used for converting raw grid data to grid state information will persist until over-written with fresh grid state; the persistence may be in a distributed or shared database, with individual values lasting only seconds before being refreshed. Analytics Architecture Considerations The Smart Grid Architect must consider many factors while developing the analytics services architecture. This section outlines several key issues for analytics architecture. Analytics Latency Hierarchy How analytics outputs are consumed is an important consideration for a Smart Grid system. Some applications must perform with very low latency in response to events or conditions, whereas others may respond over days or months. The vast majority fall in the middle, with latency ranging from seconds to hours. Thus, analytics can be categorized based on a latency hierarchy [Figure 33]. Services Classes Concepts Appendix 25

96 Figure 33 - Analytics Latency Hierarchy Classification of an analytic by its latency requirement is an important architectural consideration, as it may dictate where the analytic must be executed. It also strongly impacts two other Smart Grid architectural elements: data persistence and communications. The distribution of intelligence in a Smart Grid system is influenced by latency and scalability. Analytics can also be used to normalize data bandwidth requirements at various tiers in the network architecture by deploying their filtering capability in a distributed hierarchal form. The tradeoff between distributed intelligence (and distributed analytics in particular) and network requirements is a crucial Smart Grid design issue. Data persistence architecture must align with this tradeoff decision, which can lead to distributed hierarchal data persistence design issues. Scalability As Smart Grids evolve, data volumes will increase and analytics consumers will demand increasingly more stringent delivery requirements. This evolution will require periodic review of analytic engine scalability and the sizing of associated architectural elements (persistence and communications). Some centralized analytics approaches don't scale well, caused either by the engine becoming a bottleneck, or because the network design prevents the embedded analytics engines to transition from handling dozens of inputs to thousands, or millions. As described above, the data reduction aspect of analytics, when coupled with appropriate network architecture, can address scalability in a general way. The same feature provides an incremental build-out capability compatible with typical utility roll-outs. Services Classes Concepts Appendix 26

97 Availability and Failover Some Smart Grid analytics are crucial to real time operations. Such analytics may require a high availability (HA) implementation consistent with the critical operational processes supported. The Smart Grid architect must first classify analytics appropriately and then specify the necessary HA architectural elements. Analytics and Data Representation Analytics architecture is necessarily closely aligned with data architecture, with a focus on two areas: Persistence models and data stores Data representation The first is influenced by Smart Grid data classes and latency requirements. The second is part of a larger Smart Grid issue. At the lowest levels, Smart Grid devices will produce data in many different formats and protocols, especially legacy devices. However, even newer devices may use IEC 61850, IEEE C37.118, DNP 3, SEP 2.0, etc. As data and analytics move up the hierarchy from grid devices toward databases and application systems, these disparate formats should be converted to the IEC Common Information Model (CIM) representation. The CIM usage benefits of enhanced interoperability and architectural "future-proofing" are significant in any Smart Grid design, especially one based on an evolving System of Systems approach toward Smart Grid transformation. The location and timing for converting non-cim representations to CIM are key architectural decisions. Management of Analytics Environments Like other complex systems, analytics require management functions for proper deployment, version control, etc. Analytics configuration issues are essentially the same as those for devices or applications. For complex event processing, CEP engines are generally rule-based, with the rules taking many forms (state transition models, extended SQL query sets, etc.). Managing CEP rule sets, including the proper distribution and synchronization in a distributed analytics environment, is a critical aspect of the overall analytics architecture. The architect may merge this with the cross-cutting issue of managing services, systems, and devices to evolve a unified solution for communication networks, grid devices, and analytics elements, including CEP rule sets. Centralized, Distributed, and Hybrid Analytics Architectures Enterprise analytics architectures have traditionally been centralized, with business data repositories and warehouses, including analytics assets such as ordinary database queries, data cube engine (OLAP) tools, Key Performance Indicators (KPI's), executive dashboards, and data mining tools. For Smart Grid, all of the foregoing still applies, but the issues of real time control and operator decision support must also be addressed. Due to the data volumes and latency requirements supporting utility Services Classes Concepts Appendix 27

98 operations, it is impossible for all analytics to reside at the enterprise back office. At a minimum, analytics are needed in the control centers, and likely beyond. Control center analytics may be viewed as centralized and therefore logical deployment sites for analytics engines. Since grid data are often streams, analytics engines must accept real-time data feed, with access to the meta-data providing context for grid data interpretation. This may require a combination of calculation engine, CEP engine and visualization package operating in real time. If any analytical results are used in an automatic fashion, the analytics engines must have network connectivity capable of delivering these outputs to the consuming systems. This connectivity will also be needed to allow periodic migration of some grid data and real-time analytics results to back office data repositories supporting various business analyses, reporting, and planning tools.. In advanced Smart Grid architectures, intelligence and any supporting analytics may be moved outside of the control center to substations and beyond. One approach is to view key substations as grid data centers in their own right, with analytics treated in a manner analogous to the control center. Another approach is to make use of truly distributed models, where analytics elements at the substation interact and cooperate with analytics at the control center. Finally, a highly distributed edge computing architecture would push intelligence and analytics beyond the substations to locations near, or on, edge devices. In addition to edge grid devices, network devices are also logical homes for elements of grid intelligence and analytics. The distributed analytics model has compelling advantages in terms of latency minimization, robustness, incremental rollout, and flexibility. However, the model also raises several issues, including: Interaction models distributed analytics may be implemented in embedded dedicated form or as agents in a multi-agent system. The nature of interactions among distributed analytics elements is a major architectural issue. Share models if distributed analytics elements are to interact with each other and with utility devices and systems, then data sharing models become crucial. These models range from various kinds of memory sharing (shared RAM, distributed databases, shared-nothing, etc.) to hierarchal and lattice data flow models. Peer-to-peer messaging an implication of using distributed approaches for analytics architecture is the necessity of peer-to-peer messaging for most models. This adds to the communications network and security design requirements. Regardless of how analytics are distributed, most models require a centralized mechanism to manage the distributed analytics. This must be included in the analytics architecture design. Services Classes Concepts Appendix 28

99 Analytics Capability Maturity As with many other aspects of information system architecture, implementation, and operation, analytics architecture at any utility will progress through levels of capability maturity. A simplified model of capability maturity for analytics in Smart Grid systems follows: Table 16 Analytics Capability Maturity Model Self-learning Optimizing Predictive Automated Operational Level Description Comments Business/ Operational Intelligence Measure/report Conclusion Analytics automatically learn from data and experience to modify themselves so as to improve accuracy and efficiency in information extraction and information discovery Analytics are used as the basis for optimization of operations, asset management, and prosumer interaction Analytics are advanced enough to make significant predictions and are routinely used in a forward-looking manner Analytics processes are fully integrated and run automatically; most analytics describe present and recent past Analytics are routinely used in operations and business processes Data acquisition is integrated to some business and operational processes Basic ability to make measurements, collect data, and generate batch reports Analytics tools include autonomous goal seeking capabilities aligned with the utility s business goals and continually operate to improve their performance against the large scale goals of the business Analytics are deeply integrated with the higher levels of control and decision-making in the utility Utility processes make routine use of the predictive capabilities to plan alternatives and perform tradeoffs; predictive analytics are commonly used in decision support Integration of analytics with processes and systems is extensive; very little manual intervention is required for analytics to receive data or deliver results to consumers Some amount of data and analytics integration is present; some manual integration still being done Much of the integration is manual; processes are still fragile and siloed Elementary capability; most utilities are well past this stage Ultimately, the Smart Grid Architect must provide an analytics architecture consonant with the rest of the Smart Grid design, and consistent with the System of Systems approach as a set of services able to evolve with the Smart Grid. The analytics architecture must also be closely coordinated with data management services, communication services, and security architectures. Data Services Data Governance Data Governance is the discipline of treating data as an enterprise asset. Its goal is to optimize, secure and leverage data as an enterprise asset. Data Governance uses an organization s people, process, technology and policy to derive optimal value from enterprise data. It also helps align disparate and often conflicting policies that are a contributing cause for many data anomalies. Services Classes Concepts Appendix 29

100 Treating data as a strategic enterprise asset implies the need for organizations to complete data initiatives analogous to ones for physical assets. For example, utilities need to build an inventory of their existing data. The typical utility has an excessive amount of data on grid components, plants, customers, vendors and products, but has not formally documented where this data resides. A second data initiative for utilities is cyber security - business-critical data (Operational, Financial, Enterprise Resource Planning, and Human Resource) must be protected from unauthorized change or transmittal. Data is both an organization s greatest source of value and its greatest source of risk. Poor data management often leads to bad business results and to greater exposure to compliance violations and data theft. To mitigate this risk, many companies attempt to strike a balance between easy data access and appropriate data use by using mandates (rules, policies and regulations). However, the business imperative to provide better customer service by quickly using trusted data can cause some to pay short shrift to these mandates. Similarly, effective use of disparate information can enhance innovation, but overly strict data security could make such innovative data correlations impossible. An appropriate balance among data access, data quality and cyber security must be struck by each utility. Organizations must also consider the business value of their unstructured data. Often referred to as content, this data needs governance similar to structured data. An example is the creation of a company-wide records management program. Many firms must retain electronic and paper records for specific time periods. A rigorous records management policy allows records to be produced during a legal discovery process in a timely and cost-effective manner, in compliance with established retention schedules for specific document types. The following are some benefits derived through data governance: Improve the level of trust the users have in reports Ensure data consistency across multiple reports from different parts of the organization Ensure appropriate safeguards over corporate information for auditors and regulators Improve the level of customer insight to drive innovation and marketing initiatives Directly impact the three prime factors for any organization: increasing revenue, lowering costs and reducing risk Enterprise Information Management According to Gartner (White, Comport, & Schlier, 2005), Enterprise Information Management (EIM): 1. represents an organizational commitment to structure, 2. secures and improves the accuracy and integrity of information assets to solve semantic inconsistencies across all boundaries, and 3. support the technical, operational and business objectives within the organization's enterprise architecture strategy Services Classes Concepts Appendix 30

101 EIM encompasses not only traditional ICT systems, but all systems and devices for which the enterprise is responsible. Applied to a utility, grid management information is therefore considered within scope. Accordingly, a utility EIM encompasses information flowing to/from grid edge devices (distributed energy resources, plug-in electric vehicles, premises area networks, etc.). This ambitious scope is facilitated by the utility industry use of consistent models (IEC 61850, 61968, and series). During the early stages of Smart Grid system of system implementations, traditional utility data acquisition and control systems (for substations, feeders, and meter head-end) will manage edge device information. These systems use a mix of proprietary and industry standard protocols (DNP3) to bring telemetry to a server tasked to make the data available to other enterprise systems. In these cases, EIM will interact with edge device information representations, not the edge devices themselves. Over time, as more intelligence is pushed toward the grid edges, EIM will need to simultaneously extend further into the network. While physical implementations will necessarily vary considerably (due to hardware, media and computational capabilities functioning within security constraints), the data management service categories described in this section are expected to extend to all service enabled applications and devices in any utility Smart Grid system of systems. A commitment to EIM recognizes enterprise information is as important as process (application development) and infrastructure (technology). EIM enables raw data to be turned into information, intelligence, knowledge, and wisdom. As information systems become increasingly vital to business success, information management must be addressed holistically. In summary, EIM: Enables utilities to take ownership, responsibility and accountability for the improvement of data quality, information accuracy and consistency. Enables utilities to establish single version of truth for data over time. Improves utilities process and operational efficiency and effectiveness. Provides a strategy and technique to mitigate the risks as well as maximize the value of implementing commercial packaged applications. Reduces the number and size of data integration efforts over time. Enables the control of wasteful data duplication and proliferation. Enables process integrations to be more flexible and scalable. Improves data quality, integrity, consistency, availability, and accessibility. Maximizes the return on SOA technology investments. Establishes a critical component of the utility Enterprise Architecture. Provides guidance and services to information management efforts. Enables consistent implementations of SOA and information management for major programs. Services Classes Concepts Appendix 31

102 Enterprise Information Management Strategy EIM frameworks [Figure 34] and strategies provide a roadmap to help utilities establish necessary information governance and technology solutions. EIM can also drive and enable the convergence of operational, communications and information technologies, key to a utility Smart Grid realization. Enterprise Vision & Strategy Enterprise Architecture Enterprise Business & IT Core Processes Enterprise Business & IT Organizations Enterprise Infrastructure EIM Vision & Strategy EIM Governance EIM Core Processes EIM Organization EIM Infrastructure Vision Mission Strategy Goals & Objectives Value Propositions Sponsorship Stewardship Policies, Principles & Tenets Alignment Structure Data Quality Data Integrity Data Security & Protection Data Lifecycle Management Data Movement Semantics Management Database Management Master Data Management Information Services Services & Support CSFs & KPIs Structure (Virtual, Hybrid ) Roles & Responsibilities Functional Services Business Value and Relationship Management Information Architecture Blueprint Management Technologies (DBMS, Content Mgmt, ETL, EAI, EII, Data Modeling, BI/DW, Collaboration..) Knowledgebase and Repositories Standards & Best Practices Figure 34 - EIM Framework Since much public domain content describes aspects of EIM, the remainder of this section focuses on how semantic consistency can be improved by leveraging an Enterprise Semantic Model (ESM). The ESM provides the logical representation of enterprise information assets used to manage or facilitate business processes. Establishing EIM without an ESM life-cycle design provides little return on investment since it greatly lessens opportunities for the effective reuse of project artifacts. Systematic use of ESM supports the following attributes, collectively difficult to achieve otherwise: Enterprise Information Driven The ESM should utilize internal models, metadata, and terminology already in use in the utility enterprise. Existing models and common vernacular, whether or not they are documented, are important sources for ESM development. Services Classes Concepts Appendix 32

103 Enterprise Owned the utility owns the ESM, including its terminology, semantics, and implementation. The utility decides if and when externally controlled semantics are introduced. Stable The ESM must be stable, keeping established semantics clear and unambiguous. Non-Static ESMs are non-static in nature, allowing semantics to evolve toward greater clarity as existing business information evolves or new information is introduced. Openly Accessible The ESM must provide open access to business-critical information about semantics, data restrictions, entity refinement, and specific business constraints. Semantic Traceability Semantic traceability and lineage are important correlations for internal information (non-esm semantics). Industry Standards Aware A robust ESM provides mechanisms to systematically allow input from applicable industry standard models (CIM, data types, and code lists) to incorporate standard and broadly-adopted semantics. Multiple Standards Capable An ESM must be capable of incorporating multiple external reference models, even allowing for referencing more than one standard from a single business entity. Concise Enterprise Semantics By benefiting from both internal sources and available industry standards, an ESM provides concise enterprise semantics appropriate for business information across the enterprise. Business Context Capable - The ESM must support data exchange and information sharing within a particular business context. Leverage Available Methodologies - When appropriate, any existing modeling methodologies should be used in order to avoid reinvention or introduction of proprietary concepts. Proprietary methods limit choices of service providers, which indirectly will drive up maintenance and enhancement costs. Data Management Services All Smart Grid domains require Data Management (DM) services, and most require more capability than they have today. The increase in data volume, combined with tightening timing requirements, will cause most Smart Grid enterprises to evaluate their legacy services for future suitability. Data services can be broken down into data acquisition, transformation, persistence and archiving. A list of data services and related functions and definitions may be found at <insert url>. Control Services Control for electric utilities covers a considerable span of activities. All relate to the operation of control devices for power flow manipulation, but the scope ranges from consumer basic voltage regulation to grid interchange scheduling. This section s focus is on transmission and distribution control. Types of Control Grid control is decomposed into three major categories, each with unique requirements. Services Classes Concepts Appendix 33

104 Transmission and Substation (System) Control Transmission and substation level control involves control of transmission systems and substations. The primary elements are: Power flow control of switchgear defining the power flow paths Balance dispatch of generation to meet forecasted load, usually by continually driving a calculated error value toward zero. In the past this was viewed as generation control, but with the rise of Distributed Energy Resources (DER) and Demand Response (DR), balance authorities will also dispatch at the distribution and retail levels. Stabilization control of ancillary services and devices such as generation and Flexible AC Transmission Systems (FACTS) to maintain rotor angle stability, voltage stability, and frequency stability; increasingly this includes control of new devices (e.g. flywheel storage) and dispatch of hydro power to compensate for the effects of some DERs (i.e. solar, wind). This also includes using FACTS and Phasor Measurement Units to dampen modal power oscillations. Volt/VAR regulation at the transmission level. This can involve Static VAR Compensators and other FACTS devices, as well as shunt capacitors at the substation Distribution Level Distribution level control has expanded from simple SCADA-based control to advanced distribution automation, increasingly becoming more complex as the distribution grid becomes the focus for DER integration. Distribution level controls include: Volt-VAR regulation regulation of voltage levels and reactive power flow control (and power factor) at the distribution feeder level. This involves load tap changes, voltage regulators, and control of capacitor banks. Control of DER-based DC-AC inverters to provide VAR regulation will also grow in importance as the grid evolves. Power flow control operation of switchgear and sectionalizing devices to control power flow at the distribution feeder level Stabilization control of devices at the distribution level, including use of distribution STATCOM in addition to more traditional distribution automation Secondary Load Control Some forms of secondary load control (load shed for demand side management) have existed for decades. However, recent requirements have emerged to support DR through the distribution-level control of ancillary devices. Managing secondary loads with distribution-level controls presents potentially thorny issues for utility control services and processes. Generally, this means non-utility control devices residing on customer premises are to be commanded by the utility or a third party (e.g. aggregator). The requirements on utility control processes are largely dependent on how well the DR/DER controls are integrated with the utility control systems. Full integration will necessitate close collaboration between the utility and its customers, probably governed by a formal agreement between the parties. Little or no integration will reduce the potential of secondary load DR for reducing the level of expensive daily peak power reserves the utility must provide. The extent of secondary load control Services Classes Concepts Appendix 34

105 will likely be driven by economic incentives, which likely means the level of secondary load integration will gradually grow over time from a small starting point. Control Data Classes Several classes of data are important to control systems. The Smart Grid Architect must consider how to deliver each class to the appropriate destinations within the specified latency. The data classes are: Telemetry most grid state measurement data (voltages, currents, real and reactive power) are in the form of telemetry, instrument data produced and collected on a regular basis. In some designs, sensor data is sent to the control system only upon significant change (i.e. RBE - report by exception ), thus reducing average bandwidth requirements. Events asynchronous event messages are generated by a myriad of grid devices, many of which require control actions to be taken. Therefore the control services architecture must support receipt and processing of such messages. RBE data can be viewed as event messaging. Forecasts utility control processes make extensive use of various kinds of forecasts - load forecasts, weather forecasts, and solar/wind generation forecasts being prime examples. System models utility control processes use system models extensively and the dependence on such models is increasing in the Smart Grid environment. Two major types of system models used in grid control processes are: o Power state also known as grid state, the power state represents the instantaneous values of voltages, currents, and real and reactive power flows. o Topology power grids have connectivity and the models that represent the circuit and substation connectivity are crucial context for grid control processes. Grid state is a powerful concept used extensively at the transmission level, where grid state estimation is a standard process used in system control. However, it has not been used as much at the distribution level. The main reasons for this are: 1. Distribution level connectivity models are complex and relatively inaccurate, and 2. Distribution circuits are treated as unbalanced, making grid state estimation costly As distribution grid observability increases, it becomes feasible to replace grid state estimation with grid state determination. This combines direct grid state measurement with a minimal amount of estimation. The availability of distribution grid state is a key factor in enabling sophisticated distribution level control processes. Control Properties and Issues Multi-Objective/Multi-Controller systems as control processes become more complex in a Smart Grid environment, it becomes desirable or necessary to operate specific grid devices for multiple business purposes. This need raises architectural issues as to how potentially conflicting control commands are resolved. Several design patterns for multi-controller structures are shown in Figure 35. Services Classes Concepts Appendix 35

106 Figure 35 - Multi-Controller / Multi-Objective Design Patterns (van Breeman, 2001) Federation multiple control sub-systems must co-exist in a Smart Grid environment, especially at the distribution level. Some control systems, as mentioned above, may not reside inside of the utility, but their grid effect can impact the utility. The federation of multiple, interacting control sub-systems or processes presents architectural issues unknown in siloed systems. Disaggregation as more controllable devices are installed at the distribution level, high-level control commands will increasingly be decomposed to better fit control actions to specific Services Classes Concepts Appendix 36

107 conditions on individual circuits or circuit sections. This disaggregation process is a key element the Smart Grid control services architecture must support. Context and representation as mentioned above, the context for control actions (and analytics) is tied directly to the physical connectivity of the relevant grid segment. Consequently, the representation of grid structure and related information (grid state) are key issues for grid control services. The Common Information Model (IEC, 2003) schema is a core standard for such representation and is therefore crucial to the control services architecture. Control architecture considerations A number of factors affect control services architecture. Key is the utility s long-term transition strategy on how to accommodate legacy control systems while integrating new Smart Grid control capabilities. Present utility control systems tend to exist in the form of siloed, tightly-bundled systems. Some examples are Energy Management Systems, SCADA, Distribution Automation, Outage Management Systems, and the presently evolving Distribution Automation Systems. A traditional control center functional architecture is shown in Figure 36 below. Figure 36 - Control Center Functional Architecture (IEC, 2005) Services Classes Concepts Appendix 37

108 A transition to a layered, services-oriented system of systems architecture will cause these systems to evolve away from their siloed and monolithic current state. At present, interoperability efforts are helping to better integrate these systems, but modifying these products to a full services-oriented model will be a longer transition. It will probably pace the evolution of the entire Smart Grid. In the meantime, the system-of-systems philosophy can help the Smart Grid Architect develop large-scale control services architecture by integrating available control system elements (listed above). Utility control systems must deal with a range of latency requirements, and as the Smart Grid becomes more capable, the latencies are expected to decrease. SCADA cycles previously measured in seconds are being displaced by closed loop stabilization controls operating on the time scale of two cycles (1/30 th second). Some processes will continue to have relatively long latencies, but application requirements driving control services architecture hardest will be the ones with the tightest time limitations. The Smart Grid concept includes many more control points than the traditional grid, especially at the edges (distribution level). As controllable points are added, control architecture scalability becomes an increasingly important issue. Not only are more points being controlled, but the number of data points used to compute the control will grow dramatically. While control systems are vital to smart grid success, the power reliability and availability provided by the current control services remain of highest importance to power delivery. The smart grid migration plan must never degrade existing core grid attributes. Grid control services architecture must therefore have enough flexibility to ensure grid reliability is unaffected as new control features are implemented. The smart grid environment will involve increasing numbers of control devices as well as processes and applications. Management of the devices and applications will be complex. The control services architecture must closely integrate with the management services architecture to provide a unified approach toward managing settings, protection curves, code versions, exception events, etc. Control Architectures Where legacy utility control system architectures were centralized (e.g. EMS, SCADA, OMS), many future smart grid control systems will be fully distributed or have distributed components. A few utility control systems are already distributed an example is a fault isolation system using peer-topeer radio and pre-programmed logic on the sectionalizers to isolate faults in distribution circuits. The smart grid architect should expect control services architecture to transition from mostly centralized, to partially-centralized, to a final hybrid centralized/distributed state. There are various hierarchical alternatives for how to arrange the hybrid controls service. An example design pattern for hierarchical control architecture is shown in Figure 37 below. Such structures will raise implementation issues Services Classes Concepts Appendix 38

109 while the utility begins to move away from the mostly centralized legacy control. Managing the hybrid control services architecture is expected to be complex and time-intensive. Figure 37 - Hierarchical Control Design Pattern Conclusion The control services architecture presents many challenges to the smart grid architect, especially the coordination of control services with other smart grid services architectures. The challenge is similar to that posed by the switchover from manual meter reading to AMI any portion of the system is crucial to operations, cash flow, and customer satisfaction, leaving little or no margin for experimentation or error. Security Services Grid systems security has largely relied upon obscure proprietary protocols and lack of remote communications access (security by obscurity) to protect grid control devices. The smart grid is expected to provide a more secure and reliable grid infrastructure. Security Threats Smart Grid Cyber Security concerns dictate a multi-faceted approach to security. The system must provide security controls to address concerns unique to the implementation environment, and to interactions and interdependencies with other enterprise and business applications. The smart grid security architecture must address these concerns with a rich set of controls designed to facilitate tailored and highly granular supervision. Each control must be engineered for its environment, Services Classes Concepts Appendix 39

110 mission, and user. The resulting solution must manage the full lifecycle security for every aspect of information and technology, from edge to back office devices and from engineering to implementation. Physical Security Threats Utilities need to respond to increased physical threats on their electrical assets with a greater number of physical security systems (e.g. video surveillance, biometric-based access controls, alarms, motion detection sensors, etc.), as well as other measures. Physical threat counter-measures demand coordination, management, and communication infrastructure to relay information to appropriate Security Control Centers and Emergency Centers (i.e. the Police, Fire Department and local Hospitals). Training must also be provided to utility personnel on a regular basis to ensure a high-level of security awareness in each employee. Each person needs to know how to identify potential physical threats and how to contact the appropriate resource to mitigate the potential risk. Cyber Security Threats The increased use of Information and Communications Technologies (ICT) in the smart grid also brings additional security threats potentially impacting electric power system reliability. Sophisticated cyberattacks could devastate this infrastructure, and the dependent economy. Cyber security threats have a broad sweep, from relatively minor (an attempt to bypass a meter), serious (an organized crime attempt to extort utility funds by threatening service disruptions), to existential (national adversaries seeking to destabilize the country s critical infrastructure). Other potential threats include intentional attacks by disgruntled employees or unintentional errors made by well-meaning employees. Considering the millions of electronic devices (e.g. smart meters) being deployed with expected lifetimes exceeding years means applying ex post facto security can no longer be tolerated. Smart grid systems must be secure by design. Using Internet Protocol (IP) for smart grid communications and open standard protocols for grid control necessitates securing the grid at multiple points. The Smart Grid Security Services provides security enforcement at these points (e.g. SCADA Network systems, the Utility Communication Link, Advanced Metering Infrastructure, Substation Remote Monitoring Equipment, and Meters to Meter- Data Collection Points). In addition, utilities may need to enhance the protection applied to their public-facing portals (i.e. web pages). These will inevitably be attacked as Demand Response prosumer energy controls are activated by smart grid projects. Security architects will need to defend against these cyber-attacks while protecting sensitive customer data without disrupting any prosumer using utility internet tools provided to help manage electricity. Services Classes Concepts Appendix 40

111 This security perimeter expansion has both topological and identity-based elements. This allows Security architects to classify and prioritize security vulnerabilities for each communication infrastructure perimeter, as illustrated below by Figure 38: Figure 38 - Security Threats Classification With all the potential security threats to the smart grid, how can a Security architect ensure grid security and reliability? Simply complying with regulatory requirements is usually not sufficient to protect critical systems against determined adversaries. Security architecture must be pervasive. It requires considerable planning, analysis and design concurrent with other grid architecture designs. Security Assessment In order to address the diverse grid threats, the security architecture must perform an assessment to identify ICT security vulnerabilities and risks. To be successful, all utility business units and support organizations must participate, allowing access to their ICT infrastructure and providing the transparency needed to uncover all known and potential security attack vectors. Each security assessment must also consider evolving legal and regulatory security requirements. This helps the utility remain compliant with external mandates (i.e. NERC CIP, NISTIR, NIST 800-xx, various state and local directives). The Security Architect must clearly communicate the objectives of each assessment: Review existing security policies and identify potential gaps to reduce organizational risk Ensure necessary security controls are integrated into each project s design Provide documentation outlining any security vulnerabilities and suggest solutions The following methodology outline is an effective means to conduct a security assessment: Services Classes Concepts Appendix 41

112 Requirement Study and Situation Analysis Document Review Risk Identification Vulnerability Scan Data Analysis Report & Briefing As noted above, the Security Assessment identifies security vulnerabilities. It also helps define the security strategy for the smart grid system-of-systems. The huge number of interconnected points in smart grid makes its security very challenging. The Security Assessment helps the utility determine the assets to security-enhance and those with an acceptable risk level. The Levels of Criticality and Levels of Trust, described later in this section, are relevant tools for this assessment. The Smart Grid Cyber Security concerns dictate a multi-faceted approach to security. The system must provide security controls to address concerns unique to the implementation environment and interdependencies among enterprise and business applications. The smart grid security architecture must address these concerns with a rich set of controls designed to facilitate tailored supervision with a high degree of granularity. Each control must be engineered for its environment, mission, and user. The resulting solution will manage security from cradle to grave for every aspect of information and technology, from edge devices to Back Office; from engineer to implementation. The imperative to address emergent cyber security concerns as the smart grid matures requires a Risk Adaptive Architecture. This architecture combines risk management and principled systems engineering methods to produce a scalable, modular solution. This equips security architects with tools needed to respond as threats, technology, regulation, and usage independently change. The Security Architect must also study technology providing direct security capabilities, as well as functionality to simply enable secure policy, architecture, and configuration. The security architecture should use the best cryptography along with filtering, firewalls, segmentation, and virtualization. Physical security must be interwoven with cyber security to ensure controls are appropriate, effective, economical, and enduring. Finally, recognizing even the best of systems fail and no armor is perfect, the security architecture must develop a holistic approach to deter, prevent, detect, react, and recover. Security Context Central to the notion of security services for risk adaptive security architecture are the related concepts of security domains and levels of trust. The two broad smart grid service domains are described below. The trust level is derived from data quality and source trustworthiness. Since smart grid spans a complex trust environment, it must be capable of distinguishing, labeling, and segmenting data based on its origin and the system s level of confidence in the data. Smart grid Services Classes Concepts Appendix 42

113 functionality must have enough resilience to ensure essential functions remain available should data sources be suddenly deemed untrustworthy and dropped from analytics, displays, etc. Security Service Domains A security service domain is an area with a relatively consistent set of constraints and expectations. There are two security service domains: Levels of Criticality The Centralized Security Services domain drives the security scheme, providing automated security services to all elements of the system as well as management capability for each of the automated services. The Decentralized or Edge Security Services domain provides the field component counterparts for relevant services in the Central Security Services domain. Both domains leverage physical access controls to compliment electronic controls in creating a complete protection picture. NERC currently defines three levels of criticality when it comes to CIP: High, Medium, and Low: Levels of Trust Bulk Electric System (BES) Subsystems have High Impact if, when destroyed, degraded or otherwise rendered unavailable, they could directly cause, contribute or create an unacceptable risk of BES instability, separation or a cascading sequence of failures. BES has Medium Impact, if, when destroyed, degraded or otherwise rendered unavailable, they could directly affect the electrical state or the capability of the BES, directly affect the ability to effectively monitor and control the BES. BES has Low Impact if, when destroyed, degraded or otherwise rendered unavailable, they could not directly cause, contribute to, or create an unacceptable risk of BES instability, separation, or a cascading sequence of failures. The level of trust for a particular datum starts with an assessment provided by the data source and retained in the data header s quality field. The system combines this value with a measure of the trust assigned to its source to define the data level of trust as one of the following: Trusted: The data meets all predetermined criteria and is displayed and/or incorporated normally. The operator retains the ability to manually override the decision. Questionable: The data meets some but not all criteria or falls within a designated window whereupon the system warns the operator of the condition and prompts for data inclusion. Untrusted: The data fails to meet predetermined criteria and is not displayed or incorporated in calculations. The operator retains the ability to manually override the decision, however the system will indicate it is in an override condition. Services Classes Concepts Appendix 43

114 NISTIR 7826 Security Guidelines (NIST-ITL, 2010) In August 2010, the NIST Information Technology Laboratory (ITL) issued a three-volume NIST Interagency Report (NISTIR 7628) entitled Guidelines for Smart Grid Cyber Security (NIST-ITL, 2010) to discuss ITL s research, guidance, and outreach efforts in computer security and its collaborative activities with industry, government, and academic organizations. A month later, the Smart Grid Interoperability Panel (SGIP) Cyber Security Working Group published the Introduction to NISTIR The following text is from a sidebar on page 3 of this introduction, At a Glance: Report Objectives: The transformation of today s electricity system into a Smart Grid is both revolutionary and evolutionary. Persistence, diligence, and, most important, sustained public and private partnerships will be required to progress from today s one-way, electromechanical power grid to a far more efficient digitized system of systems that is flexible in operations, responsive to consumers, and capable of integrating diverse energy resources and emerging technologies. This progression will unfold over the span of many years, during which several generations of technologies will compose the evolving Smart Grid. As a consequence, the cyber security strategy for the Smart Grid must also be a continuing work in progress so that new or changing requirements are anticipated and addressed. Guidelines for Smart Grid Cyber Security is both a starting point and a foundation. As Smart Grid technology progresses, the Cyber Security Working Group (CSWG) will continue to provide additional guidance as needed. This first installment of the guidelines is: An overview of the cyber security strategy used by the CSWG to develop the high-level cyber security Smart Grid requirements; A tool for organizations that are researching, designing, developing, implementing, and integrating Smart Grid technologies established and emerging; An evaluative framework for assessing risks to Smart Grid components and systems during design, implementation, operation, and maintenance; and A guide to assist organizations as they craft a Smart Grid cyber security strategy that includes requirements to mitigate risks and privacy issues pertaining to Smart Grid customers and uses of their data. The guidelines are not prescriptive, nor mandatory. Rather, they are advisory, intended to facilitate each organization s efforts to develop a cyber security strategy effectively focused on prevention, detection, response, and recovery. Services Classes Concepts Appendix 44

115 NISTIR 7628 should be studied closely by all smart grid security architects. The three volume report contains a myriad of methods and related information designed to help organizations assess risk and then to apply appropriate security requirements to mitigate the risks identified. Communication Services Communication services (comms) are fundamental to electric grid operations. They enable devices to be intelligent, information to flow, controls to execute, and people to manage utility functions and services. Large numbers of new smart devices will be deployed in a smarter grid, each needing some form of communications. Planning for this deployment is difficult since most devices don t yet exist and initial functionalities will change as grid operators figure out innovative ways to leverage their intelligence. Thus, it is critical for the future grid communication architecture to be extraordinarily resilient and flexible. It is strategic to develop a clear understanding at the conceptual layer of the interplay between smart grid objects (endpoints), the information they communicate, and the channels across which that information flows. This will provide context for rationalizing the communication functions at the services layer. Figure 39 - Conceptual and Services Layers Every endpoint, all information types, and each communications channel are different. Their combinations are staggering and their capabilities continually change. The comms requirements to support smart grid use cases are proof of this diversity (see Table 25 Communications Services). This Services Classes Concepts Appendix 45

116 will remain true as new smart grid use cases emerge. Characterizing these requirements can be extremely challenging. For example, performance requirements can range from exceptionally stringent to relatively modest. Information flows could be streamed, polled, infrequent, message mode, file transfer mode, or asynchronous event-driven. Application architectures can vary from highly centralized, to distributed, and some could even be peer-to-peer. The quantity of actors in these use cases could range from dozens to millions. Most are stationary, but others require mobility. Some are critical to grid operation while others are of marginal operational importance. This diversity challenges the enterprise architect to devise flexible, scalable smart grid communications architecture. The Nature of Smart Grid Endpoints, Information and Channels A framework around which communication services might be considered should examine the nature of: (1) the actors (endpoints and their functional requirements), (2) smart grid information, and (3) the channels through which information will flow. Thoroughly understanding these drivers will help the smart grid architect put the supporting communication services in a proper context. Smart Grid Endpoints Grid endpoints include a range of devices. In general, they include field devices such as meters, reclosers, capacitor banks, and so on. There are a variety of devices in substations including IEDs, SCADA RTUs, teleprotection devices, measurement devices, security appliances, and so on. At data and operations centers there are head-end processors, data collection units, control systems, customer interface portals, etc.. In general, these endpoints provide the smart grid application functions. For purposes of this reference architecture, the communication nodes are not considered endpoints since they don t provide application functions. Each of these endpoints has unique operating capabilities. The smart grid architect should consider a variety of aspects of these endpoints when planning comms solutions including: Diversity of purpose Will the endpoint support a single purpose or is it a multipurpose device? For example, meters are typically thought of as devices providing usage data. But in smart grid, meter functions expand to include usage, remote connect/disconnect, outage alerting, and power quality data. Metering groups need the usage data but grid operations teams would be interested in outage and power quality data. How does this impact comms architecture? As an example, if a meter is placed in a virtual network designed for metering, then additional challenges need to be considered in the overall architecture in regards to security, performance, and operations in order to direct outage and performance data to the appropriate recipients. Diversity in scope Will the endpoint communicate to just one (or redundant) other endpoint or will it communicate with a diverse set of other endpoints? Extending the example of metering, it may be necessary to regularly collect usage data from a centrally located meter collection engine. Outage information could be captured by distributed OMS pre-processors located at substations so event messages don't congest the backhaul networks. Power quality Services Classes Concepts Appendix 46

117 data may need to be periodically collected by analytics systems. In this example, meter endpoints may communicate with three or more host system, each existing in different domains and at different locations. However, teleprotection devices may not have the same multi-domain requirement. For operational reasons it may also be necessary to restrict the devices able to send traffic to the teleprotection equipment. Criticality Is the purpose of the endpoint to support: a critical real-time function, for billing or analytics purposes, or for informational but non-critical functions? Some devices are seen as more critical than others for a variety of reasons. For example, if a teleprotection device fails to operate in a timely manner it could result in the loss of electrical service to thousands of customers which could extend for weeks. Associated damage to grid equipment could cost many hundreds of thousands of dollars. Therefore communication solutions for teleprotection devices warrant significantly more investment for reliability and resiliency than communication solutions for a feeder-capacitor bank with a potential to affect far fewer customers if information cannot flow in a timely manner. Mobility Is the device mobile, fixed, or transitional? Most smart grid devices do not need mobility protocol support since they will be mounted on homes, poles or in building facilities. However, there is increasing need to support mobile workers and mobile machine-to-machine solutions. Mobility introduces many architectural challenges (coverage, access methodology, roaming support, location awareness, etc.). Quantity How many devices will exist in a mature implementation? How will their numbers be spread across the various communication domains? How will management systems support potentially millions of devices such as meters? What are the impacts to traffic capacities and congestion control? Geographic density Where will devices be deployed? For the various kinds of devices, will they be widely dispersed, densely placed, adjacent or in close proximity to other grid devices? How does this impact communication media choice? Can proximity be leveraged or does it present challenges (interference, signal power constraints, etc.)? Degree of intelligence How intelligent is the endpoint? Is it highly dependent on other systems for resources and functionality? If so, where do they exist and what is their criticality to other endpoints? Where does the application intelligence for this endpoint exist? Most legacy grid systems use centralized intelligence; therefore centralized communication architectures can support them (very common for SCADA and AMI systems). However, if intelligence becomes distributed, then the communications architecture must also be distributed. For example, if grid intelligence is distributed to the substation, could field-mounted devices effectively communicate to the substation? Smart Grid Information The information transmitted between smart grid endpoints varies widely. It comes in many forms, sizes, criticality, and regularity. For a variety of reasons, not all communication systems are conducive for handling the breadth of requirements from required smart grid information sources: Regularity of transferring information Will information be a continuous stream, scheduled transactional, or event-driven? This issue concerns how to think about congestion management. Streaming information (like PMU data) and scheduled transactions (like AMI reads) are Services Classes Concepts Appendix 47

118 predictable and easier to plan for. But event-driven information can be very challenging. Furthermore, how does information regularity impact communication session management and security mechanisms (for example key management, access control, and authentication)? Size What is the size of the information? This issue concerns how to think about bandwidth requirements. Is it streaming and as such is continuous? Is it a predictable file size? Is it messageoriented and contained in predictable bytes or even bits? Form Is the information structured (XML file), bit-oriented (on/off), or byte-oriented (DNP3, IEC) defined messages? Is the information unstructured (such as a continual tone)? Duration/longevity Is the information archivable (meter and other forms of measurement), transactional with a short time-to-live (outage or state data), or potentially a repeatable transaction (control data with message acknowledgement)? How does this impact communication system choice and failover schemas? Criticality Is the information critical, not critical, or somewhere in between? Does it warrant high availability or redundancy? If so, to what degree of reliability should the system be designed (is % reliability necessary or would marginally lower reliability suffice)? How does information criticality impact the failover schema at the application layer versus at the communication layer? Simulcast Will information be broadcast for any interested device to receive (publishsubscribe systems), multicast to a defined set of multiple devices, or unicast to just one device? Broadcast and multicast information can challenge the communications solution. Ownership Who owns the information? What policies govern access to the information? What are the implications at the application layer versus the communications layer? Diversity of usefulness Is the information useful across multiple applications? Example: some AMI transactions combine usage and power quality information. Smart Grid Channels There are a wide variety of channels (media) over which smart grid information might travel. Most of the time, information will travel over a network of networks each optimized for some collective need. Understanding why different networks are necessary and how they function is critical to achieve optimal communications architecture. Access mechanisms How is the channel accessed? This issue is especially of concern for wireless systems where media sharing can be challenging. Some solutions use network-based synchronized time slots (polling) schemes to limit collisions but this can affect performance. Other solutions might use application layer polling that, if combined with network-based polling, could be unnecessarily redundant. Shared use If the channel is shared, how will traffic utilization be handled? For example, expensive wide area networks will likely need to support high priority critical traffic and lower priority informational traffic. How will traffic priorities be administered? What happens if devices that share the same channel (meters for example) cannot effectively hear one another? Do data link layer capabilities exist to prevent genuine information from appearing as noise? Distance between devices Does the distance between devices warrant powerful transmission signal levels (many miles)? If so, how would it affect system cost structure? Is the distance short enough to allow communications capabilities at one device to support other nearby devices? Services Classes Concepts Appendix 48

119 Diversity of function Will channel use support a variety of applications with varying performance and security needs, or will the channel be purpose-built for a particular solution? Performance What are the latency capabilities of the channel? Some media (fiber optics) can support extraordinary performance characteristics. However, due to contention and framing mechanisms, wireless systems generally support more moderate characteristics and their effective performance range can vary widely. Bandwidth What is the bandwidth of the channel? Different media types support varying amounts. At the low end, some long-reach wireless systems might only support 10 s of kilobytes per second. At the upper end, some fiber solutions can support many gigabytes per second. Reliability What reliability schemes are used on the channel? There is a wide spectrum of reliability mechanisms a communications system could deploy. Cost considerations must be applied to obtain the necessary degree of reliability and performance these schemes can deliver. For example, do applications on field area networks warrant % reliability? What is the cost for the desired reliability? Is a 50 millisecond failover scheme on a Synchronous Optical Network (SONET) based teleprotection circuit acceptable to application users? What would happen if the failover detection was only a few milliseconds but the reconvergence time for information flow measured in the hundreds of milliseconds? Media cost structure What is the cost structure of the media? When it comes to communications, the balance between cost and performance must always be considered. Some wireless solutions may not require much channel costs (i.e. free space propagation, licenseexempt systems, and very limited path engineering). However other wireless solutions might require licensing and very diligent system engineering. Furthermore, fiber placement might require public right-of-way and construction costs. Costs must be reasonable for the set of applications supported by the media. For example, under what circumstances would fiber deployment be warranted for meter communications? Equipment cost structure What is the cost structure of the equipment? Communication systems have a wide spectrum of features and capabilities (power consumption, interface redundancy, management, etc.). The architect must determine what is reasonable for the environment and the applications being addressed. While many capabilities are generally desirable, if unchecked they may encumber the solution cost structure. For example, is it rational to deploy communication devices with no power supply or interface redundancy at transmission substations? Environment What is the nature of the channel environment? Does it require special hardening? Is the equipment location under the direct and exclusive control of the utility? Administration How will the channel be administered? Will the channel be under the management and administration of the utility or will the utility obtain channel services from a provider? How will channel administration affect service assurance and restoration? How will periodic changes on communication system(s) be coordinated with utility business operations? Communication Services Understanding the conceptual layer of endpoints, information and channels helps frame the communication services necessary to support smart grid operation. Once identified, the Services Classes Concepts Appendix 49

120 communication services can be mapped to the various network of networks comprising the overall communications architecture. The Communication Services layer [Figure 40] requires the following functions: Transport services to move information around the network, Virtualization services to help segregate and optimize transport services, Control services to help manage information flow and support endpoints needing network access, Gateway services to facilitate legacy applications to utilize next generation networks, and Mobility services to support roaming or mobile endpoints. Figure 40 - Communication Services Layer Functions Transport Services Transport services are the means to move information around the network. Transport services are derived when communication links are combined to form networks. Since smart grids can cover a wide expanse of territory, a network of networks must be developed and interconnected so endpoints can send and receive information across the networks transport services. When considering transport services for smart grid, the smart grid architect will make a series of choices each time providing more and more detail. The highest level choice defines the type of transport service to deploy. Next, the architect must determine logical divisions for each network within the network of networks (subnetworks). Only then can the architect establish each subnetwork s operating characteristics. Figure 41 illustrates this process below: Services Classes Concepts Appendix 50

121 Figure 41 - Transport Services Consideration Process 1. Type First, the architect must make the decision of what type of transport service to deploy. There are two fundamental types of transport networks: circuit switched and packet switched. In the general world of telecommunications, it has been long established that packet-based solutions provide the most ideal form of transport services. The primary value of packet-based solutions comes from their ability to support an extraordinary array of applications over a common infrastructure. This provides exceptional economic and operating benefits. Another key value is that packet solutions were a product from a methodology supporting a layered approach whereby applications could evolve independent of the network. Abstracting the applications from the network provides both versatility and stability for each environment. In the case of smart grid where innovation is expected and requirements will be so diverse, packet-based transport solutions are the only logical choice. Although virtually all new investments are packet-based, there are still large amounts of legacy applications and infrastructure that will co-exist for some time. Adapting these systems to packet transport services is necessary and will be covered below in the Gateway Services section. 2. Logical Division The next decision that must be made is the logical division for each subnetwork making up the end-toend solution. Implied is that there isn t a single ideal transport network for accommodating smart grid. Therefore some form of subnetworking must take place to achieve various transport service capabilities (described below). For example, some networks are optimized for ultra-high speed but its cost structure can be burdensome. Since not all applications warrant such costs, networks of lower capabilities may be needed. The same tradeoff can be made regarding other characteristics such as performance and security requirements. Network technologies have limits and thus the Services Classes Concepts Appendix 51

122 implementation of subnetworks is necessary. For making subnetworking decisions, it is best to establish functional categorization. While it is tempting to categorize transport services according to media type (fiber, wireless, powerline carrier, etc), this leads to ambiguity. For example, the kind of fiber networks used in data centers is different from fiber systems used to reach extraordinarily long distances across a utility s operating territory. And the variety of wireless systems includes solutions such as long-haul microwave, metropolitan-based WiMAX, local WiFi, and even Zigbee personal area networks. It is better to categorize transport services into functional subnetworks. It appears this is what NIST has done in their Smart Grid Conceptual Reference Diagram [Figure 42] as they identify: Internet / e-business Enterprise Bus Wide Area Networks Substation LANs Field Area Networks Premises Networks Figure 42 - Conceptual Reference Diagram for Smart Grid Information Networks (NIST, 2010) Services Classes Concepts Appendix 52

123 The NIST Conceptual Reference Diagram is a functional categorization, except for the use of the Operations domain term Enterprise Bus (usually a software-based mechanism for abstracting applications from transport services) instead of Operations Center LANs. The Internet / e-business clouds in Figure 42 suggest communications to external entities such as trading partners, customers, and other market operators. The Wide Area Networks cloud reflects the communications required across expansive territories from information processing centers to moderately remote assets (substations, data concentrators). The Field Area Networks cloud represents connectivity from various points of concentration out to numerous, scattered, and small endpoints. Functional categorization of transport systems provides the architect a framework to evaluate subnetworks and to make business, economic and technical choices. For example, the architect may initially opt for a carrier-provided public wide area or field area network. As the system evolves certain critical or highly used links could be replaced with utility-owned infrastructure (fiber or some form of wireless media). There may be instances where the utility initially operates a wireless solution based on cost structure but the evolving smart grid eventually causes fiber links to supplant the wireless links. The functional transport service architecture also helps the architect make the decision when to converge or segregate subnetworks. For example, the wide area network domain provides connectivity between Operations or Control Centers and remote substations. From an electric grid standpoint, the wide area network is generally associated with covering the transmission grid and substations whereas the field area network covers the distribution grid beyond the substation (Figure 42). Applications and endpoints associated with the transmission grid require high performance capabilities and warrant the diversity and higher reliability associated with more costly network solutions. As a media choice, fiber offers the highest degree of performance and is therefore often preferred in the wide area network. Since it can be costly to operate multiple wide area networks over the same geographic footprint, converging traffic onto a single (virtualized) fiber-based wide area network can be advantageous. In any case, the unique properties for each network domain are important architectural considerations. 3. Characteristics After choosing the type of transport service and the logical divisions for subnetworking, the architect must then make key business, operational, and technical decisions regarding each subnetwork s operating characteristics. Each issue impacts architectural choices just as they influence design choice. Common domain issues to consider include: Bandwidth - How much is required and available? How is it consumed and/or allocated? How does bandwidth utilization impact architecture decision? Latency and jitter How does the transport service architecture support and manage latency and jitter? How do protocols used across the transport service impact latency and jitter? How do latency and jitter influence network element architecture choice? Services Classes Concepts Appendix 53

124 Domains How are transport services virtualized into segregated domains to support bandwidth, latency, privacy, and management? (Also covered in the Virtualization Services section below.) Resiliency How do the failover mechanisms support the transport services within each of the subnetworks and from an end-to-end perspective? What are the performance capabilities of the resiliency mechanisms? Quality of Service What quality of service capabilities exist for the transport services in each of the different subnetworks? Reliability What is the reliability of the transport service at each of the subnetworks? Access How will devices gain access to the transport service? Will there be many devices attempting to access the services simultaneously? Standards What standards apply? Are there functional gaps not covered by standards? Management How are transport services granted, restricted and controlled? Costs What is the cost structure for various types of transport services? Differences among the domains begin to drive further architectural considerations. For each of the transport service domains, key issues have been identified: Internet / e-business Purpose provide connectivity to customers, suppliers, business partners, emergency services, and the public at large Design Issues generally open access to public environments, security, accessible by millions of users, public infrastructure (carriers), Internet, data/voice/video/control services, performance, capacity seldom measured beyond a few Megabits, failover to alternative external networks/providers, infrastructure dependent on carrier preference, global reach Operational Issues lack of autonomous control, lesser degree of monitoring/management, dependence on others for reliability and performance, recurring expenditure, service contracts, capacity driven Operations Center LANs Purpose provide connectivity to control systems, operators, high-end applications, servers, and storage systems Design Issues extremely tight controlled access, highest degree of security, highest degree of performance, multi-gigabit support, high availability designs, thorough virtualization, data/voice/video/control services, primarily fiber infrastructure, building area reach Operational Issues key location for grid intelligence, full autonomous control, highest degree of monitoring/management, highest reliability, highest performance, infrastructure can be capital intensive, project driven Wide Area Networks Purpose provide high performance connectivity (a) between data/control centers, (b) from data/control centers to key remote facilities such as substations, (c) from data/control centers to points of data concentration (such as meter data concentrators) Services Classes Concepts Appendix 54

125 Design Issues tightly controlled access, high security, high performance, range from tens of Megabits to multiple Gigabits, situational high reliability, failover mechanisms very responsive, virtualization and converged resources, data/voice/video/control services, primarily fiber and microwave, SONET, MPLS, IP, public or private infrastructure, regional geographic reach Operational Issues -- reach to places where grid logic has been distributed, situational control (public or private), high degree of monitoring/management, desired interaction between grid and telecom management systems, high reliability, high performance, capital or expense funding, project driven Substation Local Area Network Purpose provide coverage to smart objects within and near by the substation Design Issues controlled access, security, high performance commensurate with applications, range from tens of Megabits to Gigabits, high reliability, failover mechanisms including high availability, may require segregated overlay solutions for separating process bus traffic from other traffic, data/voice/control services, primarily fiber and wireless technologies, reach in the immediate vicinity of the substation Operational Issues reach to dozens of endpoints which can support other devices through distributed grid intelligence, total control by utility (private), moderate monitoring/management, desired interaction between grid and telecom management systems, high reliability, high performance, capital funding, project driven Field Area Networks Purpose provide broad coverage to remote field endpoints so that they can connect to (a) data/control centers, (b) distributed applications (perhaps at substations), and (c) to other remote endpoints Design Issues controlled access, security, moderate performance commensurate with applications, range from one to tens of Megabits, moderate reliability, failover mechanisms often unavailable, may require segregated overlay solutions optimized for the applications, data/voice/control services, primarily wireless and powerline carrier technologies, reach may be similar to that of the serving substation Operational Issues reach to thousands or millions of endpoints which have lesser degree of grid intelligence, situational control (public or private), moderate monitoring/management (desire selfdiscovery and auto-configuration), desired interaction between grid and telecom management systems, moderate reliability, moderate performance, capital or expense funding, project driven Premises Networks Purpose provide coverage to smart objects within the home, business, and commercial enterprise Design Issues controlled access, moderate security, moderate performance, range from tens of kilobits to a few Megabits, moderate reliability, seldom use of failover mechanisms, likely segregated overlay network, data/control services, reach within the customer premises, predominantly responsibility of customer rather than utility Operational Issues reach several to many endpoints with emphasis on premises intelligence, virtually no control since under authority of premises owner, moderate to no Services Classes Concepts Appendix 55

126 monitoring/management (desire self-discovery and auto-configuration), largely isolated from grid management systems, moderate reliability, moderate performance, capital expense by premises owner, service subscription driven Network Control-Plane Services There are many network control-plane services to be accommodated in smart grid too numerous to go into detail in this work. A few control-plane services with smart grid interdependency warrant discussion. To adequately address future smart grid requirements, the architect must understand network addressing, name-address resolution, and subnetwork access management. 1. Network Addressing The mature smart grid will see a dramatic increase in smart objects needing IP addresses. It is more strategic to deploy new smart objects using IPv6 (and its addressing) rather than currently dominant IPv4. Among the many benefits IPv6 offers, the avoidance of Network Address Translation (NAT) functionality is one of the more critical reasons to adopt IPv6. The use of NAT can be burdensome to control-based applications. When network anomalies require failover, the NAT function must rebuild the address translation mapping. This process can burden (or break) time-critical grid applications. IPv6 addressing can avoid this problem at least for communication within the utility domain (extranet access could still have this problem if private IPv6 addresses are used). In the IPv4 era, private IP addressing and NAT were implemented to slow down consumption of the rapidly depleting pool of IPv4 addresses. In this time of transition, it is recommended that utilities rapidly move to adopt IPv6. There are several types of IPv6 addresses including: unicast, anycast and multicast. In addition, IPv6 addressing supports both public and private address spaces. From an addressing architecture perspective, it is recommended that private IPv6 addresses (called local unique addresses) be used behind company firewalls as an added security measure against unintended access to grid devices from outside parties. Careful attention should be given to prevent reuse of private addresses as this will result in problems tracking SYSLOG issues. Another addressing architecture consideration involves the use of static or dynamic IPv6 addresses. Many control functions will rely on static (unchanging) addresses. However, many grid applications (premises metering) could benefit from dynamic addressing. Part of the architecture development is to decide where to use dynamic addressing. For non-mobile devices, dynamic DHCPv6 addressing may be best enabled from the substation or where Field Area Network interfaces backhaul communications. And finally, address space allocation must be planned with the end-state in mind. As the smart grid evolves, more devices will be placed into the field. Address space allocation should accommodate the eventual conversion of remote grid devices (including substation devices, feeder devices, meters, and perhaps even premises devices) to IPv6 addressable devices. Services Classes Concepts Appendix 56

127 2. Unique local Addresses Domain Name Services (DNS) will be necessary in the smart grid. DNS permits the abstraction of a device s IPv6 address from its name making application systems less vulnerable to periodic changes. While a common DNS solution could serve both the enterprise and operations environments, there are advantages for segregating these services. Due to the magnitude of most smart grids, the Operations DNS easily become more heavily populated and used than the enterprise DNS. One important architectural concern is how to sustain name services at remote locations when host/remote links are severed. It is preferable to operate DNS in an HA configuration with primary zone servers in redundant control centers. In the case where backhaul links are lost, name resolution for regional traffic can be sustained only as long as the Time To Live setting permits. Therefore, for critical applications, it might become necessary for secondary zone servers to be distributed regionally in key substations. 3. Authentication, Authorization and Accounting function AAA servers are required in the grid to aid with the authentication, authorization and accounting function of devices connected to the network. From a communications architecture perspective, it is important to segregate AAA functionality serving the smart grid from enterprise AAA functionality. It will also be preferable to distribute AAA functionality operated with primary and secondary systems located at diverse control centers. In the event of the loss of backhaul connectivity, critical substations could benefit from having local AAA services. Other substations may lose accessibility except via local console access. Mobility Services Support for mobility is required for both smart grid devices (especially inside the FAN) and users (such as the linemen of the future). The mobility service needs to encompass: Life cycle management of mobile devices, including registration and provisioning of mobile devices on to the network, such as configuration through the wireless LAN controller, and Secure deployment and configuration services, such as forcing the mobile user/device into a DMZ, enforcing a profile examination etc before given access to the network Gateway Services The term gateway can be ambiguous for telecommunications. For this document, it refers to the set of services used to adapt legacy or otherwise non-conforming endpoints to packet-based transport services. In doing so, gateways translate traffic from one protocol to another. This presents a challenge for Smart Grid development for two reasons: complexity increases with each required adaptation and scalability is constrained due to the limited usefulness of proprietary devices. Although it is preferable all applications standardize on Ethernet- and IP-based transport services, it is also recognized it will Services Classes Concepts Appendix 57

128 take some time for existing/embedded systems using proprietary protocols to be replaced. Therefore gateway services must be implemented in various parts of the network to adapt those devices. In the architectural context, gateways bridge the application and network domains. Therefore, when dealing with gateways, network architects and application architects should address gateway service concerns. Two key areas of interest include the appropriate location for gateways to reside and the feature sets they offer. Some common legacy gateways include the electric meter (bridging traffic into the home), AMI gateways (supporting ANSI C12.22 traffic across proprietary networks), DNP gateways (often used in recloser operations and for line sensors), and station managers (adapting legacy traffic in substations to use packet infrastructures). Very often, legacy systems integrate communication control and transport functions into the application rather than cleanly abstracting the two. Even though many such solutions may be standards based (ANSI, DNP, Zigbee, etc), they don t inherently use the standard transport protocols common to packet (IP) solutions. Therefore the traffic must be adapted. Although it can be confusing, there are some gateways deployed for smart grid for reasons other than transport service adaptation. For example, XML gateways are deployed to offload some security and application layer functions from application systems. These are not considered network layer gateways. In addition, research and development occurring in the fields of distributed intelligence and complex event processing will necessitate systems deployment further out in the network. The first decision the architect must make is where gateways should reside in the smart grid ecosystem. The time-proven, primary end-to-end principle in IP networking requires application layer issues to reside at the endpoints and not within the network. Applying this principle to the necessity of smart grid gateways makes it preferable for gateways to be pushed as far as possible toward the network edge. Therefore meters would not have to adapt traffic and protocols between home devices and the utility s network (in fact Zigbee supporters are adopting the use of IP transport systems eliminating the need for protocol translation at the meter). AMI gateways would be pushed to the edge of the Field Area Network. DNP gateways would be collocated with legacy devices, and substation managers would only serve legacy, non-ip-enabled devices within the substation. The second area of interest to smart grid architects is the breadth of functionality performed inside gateways. Some gateway suppliers are embedding more functionality into legacy devices. The result could be an increased dependence on gateways (rather than a transition to standards-compliant devices). In addition, it is also tempting to deploy low-level standards-based networking functions inside gateways rather than derive these functions from true network-layer equipment. Such practices only deepen the dependence on legacy gateways/protocols and restrict the adoption of open and standardized architectures. Services Classes Concepts Appendix 58

129 Virtualization Services The smart grid will cover an expansive territory. The cost to create information networks covering such an area can be prohibitive especially considering the need to support the diverse array of smart grid systems and system users. Although carriers already offer multi-purpose networks, there will be a growing need to adopt network virtualization services to meet smart grid demands. Virtualization services abstract physical and functional elements for a variety of purposes. Without network-level virtualization services, transport services would become overly complex and expensive. However, virtualization is certainly not limited to the network domain; in fact virtualization has its roots in application layer systems. This section deals with network-layer virtualization, which offers more transport service flexibility, to the point of supporting dynamic virtualization. There are numerous reasons to converge multiple networks onto a common platform. First, networks are becoming increasingly powerful offering the ability to reduce their associated capital and operating costs. Second, networks needing to be converged and virtualized are increasingly relying on each other for smart grid capabilities; it is easier to manage the interactions between those networks at multiple layers when they are converged and virtualized on a single platform. The network architect must be concerned with how virtualization supports the needs of smart grid applications and how it impacts the network. First, virtualization should serve the business and technical requirements of the use cases. The virtualization architecture should address performance concerns including address bandwidth allocation, latency, collision avoidance, virtualized Layer 2 domains, failover and so on. Second, virtualization must consider how to establish service boundaries, whether and what layers of the OSI stack can be virtualized sensibly. Finally, virtualization must support appropriate security on the virtual domains. Reasons to converge multiple networks onto a common platform are numerous. Networks are becoming increasingly powerful offering the ability to reduce their associated capital and operating costs. But while it is technically achievable to utilize a common physical platform, there is justification for making it function as if it were many virtual platforms. For example, suppose multiple operational owners need to allocate only their specific costs to their business unit therefore it would be important to have a mechanism to support ownership virtualization to allocate traffic capacity and insure against cross-subsidies. Some devices on the common platform may require access by a small subset of users, thus requiring a logical separation of the network for security reasons. Another scenario could have subsets of devices in the network sharing a community of interest with other adjacent devices, making virtual network zones appropriate. Therefore, the architect has many options available to establish the logical virtualization architecture best aligned with the lines of business, security, and performance. Services Classes Concepts Appendix 59

130 The next step is to determine how network virtualization supports application performance within the various network domains. It is likely multiple technologies will be used in each domain. Therefore a plan to accommodate virtualization at the physical layer and the requisite mapping between network technologies is needed. Furthermore, most virtualization must be then extended across several domains to accommodate end-to-end support. The virtualization architecture should address performance concerns including address bandwidth allocation, latency, collision avoidance, virtualized Layer 2 domains, and failover. The architect must then consider how the virtualization architecture addresses security and traffic isolation. Virtualization is by no means the final answer to security but it can be an important contributor. Virtualization helps control who can access devices, what network they reside on at the logical layer, and the degree of exposure devices may have to undesirable conditions (whether purposeful or accidental). For example, a misbehaving device flooding the network in one virtual domain shouldn t adversely affect other virtual domains on the same infrastructure. Management Services The smart grid requires increased interactions between the power systems, information systems and network systems; therefore a common management infrastructure across all the systems is warranted. Accordingly, this Smart Grid Management Services section will address the management aspects of the power and communications network connecting all power grid components, functions and services. The integration of the ICT architecture with energy management systems and domains provides a centralized way to manage the entire smart grid, including the communications network and power grid devices. Business Drivers The management vision for the utility smart grid architect should be to devise a path towards a common management platform for the various smart grid systems, functions and projects. The common management platform will ideally serve the management needs for all smart grid use cases. The following are the business drivers for the management services architecture: The number of use case actors warrants zero or one-touch management schemes across smart grid. The use cases complexity implies managing policies at a business level (not at the actor level). The variety of utility and third party entities involved in use cases requires good measurable metrics for each of the management schemes As new smart grid projects and corresponding systems emerge, the new functionality needs to be easily incorporated into the existing management infrastructure The scope for this chapter is any device which communicates and can be controlled through configuration, including communication networks, applications, servers, configurable IEDs, etc. Services Classes Concepts Appendix 60

131 This section first introduces existing management architecture literature for the smart grid architect to consider. Next, the layout and categorization of different management function types is covered. Finally, an example use case is presented covering how management functions can be categorized and organized. Existing Management Architecture Literature Existing industry standards include Telecommunications model (TMN), ITIL (Information Technology Infrastructure Library) and Open Systems Interconnect (OSI) TMN: The Telecommunications Management Network is a protocol model defined by ITU-T for managing open systems in a communications network. The TMN model consists of five layers, where Business Management is at the top, Service Management is the second layer, Network Management is the third layer, and Element Management the fourth layer with the physical network elements is represented in the bottom layer. We are proposing placing the Smart Grid Management Systems at the Business Management layer in order to enable future integration with these systems. The Open Systems Interconnect (OSI) group has defined management functionality in network operations as shown in the five categories listed in table below. This same model has also been adopted and supported by the standards body, ITU-T. This categorization is a functional view and does not attempt to describe business related roles within a telecom or data network. The FCAPS sub-functions within one of the five major groups are typically performed at differing levels within the TMN model described in the TMN section. For example, fault management at the element management level in TMN is detailed logging of all discrete alarms or other events. The element management level then filters and forwards alarms/events to the network management level where alarm correlation, corrective action and other actions take place. The FCAPS table below represents common examples of the sub-functions performed under the model definitions. Different applications of the model will typically contain sub-functions specific to the network operations management center involved. TMF etom: TM Forum's Business Process Framework (etom - Enhanced Telecom Operations Map) is a key element of the TM Forum Framework Integrated Business Architecture. It is the industry's common process architecture for both business and functional processes and has been implemented by hundreds of service providers around the world. During this phase we will be addressing all the Operations aspects of the data communications network enabling the smart grid. Within Operations, we will focus on Assurance and Resource Management (application, computing and network). ITIL: It is the most widely accepted approach to ICT service management in the world. ITIL provides a cohesive set of best practice, drawn from the public and private sectors internationally. It is supported by a comprehensive qualifications scheme, accredited training organizations, and implementation and assessment tools. While the smart grid architect needs to assess the present ICT management architecture and assess the integration of the smart grid management in the present architecture, the rest of this section will focus on the management functions in general and the application of the management functions to smart grid. Services Classes Concepts Appendix 61

132 Smart Grid Management Functions Management services for smart grid can be viewed in several different ways. First, smart grid consists of elements across multiple utility domains (consumer, distribution, transmission, generation, markets, etc) and technology domains (communications, applications, servers, wireless, video etc) that need to be managed. Second, from a System-of-Systems perspective, smart grid can be organized into systems, which need to be managed. Third, each of the systems interacts with each other for specific use cases, which also need to be orchestrated and managed. The following section will organize smart grid management services as layers accommodating the above-described functions. Organization of Management Functions The following is the organization of management functions for smart grid into the three areas described above - the management functions applicable for actors (which we call Element Management Layer), management functions applicable for systems (System Management Layer) and management functions applicable at the use case level (Services Management Layer). Each management function in the Services Management Layer will make use of various functions in System Management Layer and Element Management Layers. These different management schemes can be organized in the following way Each layer in Figure 43 is discussed below. Figure 43 - Management Layer Organization Element Management Layer The following are the different management functions and corresponding processes involved in element management: Services Classes Concepts Appendix 62

133 Lifecycle Management: Life cycle management is the service dealing with automatic device discovery, automatic update of firmware, and automatic provisioning into the system. The service also deals with secure decommissioning of devices. Life cycle management and its automation are critical to smart grid management due to the large number of relevant devices. System turn-up Auto-discovery Provisioning Change management Decommissioning Technology/control plane management: smart grid is comprised of various technologies, whose control plane needs to be managed. The technologies include the following: Routing and switching, including control plane and data plane services Wireless Mobile Systems Management Layer The smart grid architect must support the following common management functions across systems: Event Management: The following are the event management functions Collection of power/cyber events from all grid and communication assets Analyzing and correlating the events for actionable incidents Taking the corresponding action Security Management: The following are the security management functions that need to be managed across systems (example: key management when a DA system needs to talk to an AMI or a Transmission system) Access control management Security monitoring System audit Configuration Management: Configuration management includes any configuration changes on the end assets (network, data, information, application, electric) including settings, firmware upload and down load. The smart grid architect needs to plan for a common configuration management platform for different smart grid projects and different types of electrical and network assets involved Services Classes Concepts Appendix 63

134 Services Management Layer The above management functions apply for actors and systems The smart grid architect needs to analyze each use case in scale and analyze the management requirements. The following are the management functions that would apply at the use case level (Services/Business Management Level): Performance Management: This includes tuning the assets for the quality of experience that is required for the use case. Bandwidth and latency provisioning, quality of service provisioning will be part of the performance management. Policy management: this category is the management of business policies across the enterprise. This could be security policies such as NERC CIP or any business policies such as disaster recovery and management, change management, etc. The services management layer would use the systems management layer and element management layer functions for end device configuration. Figure 44 depicts the management layers, each management function inside the layers, and their applicability to the actors, systems and use cases. Figure 44 - Smart Grid Management Layers Services Classes Concepts Appendix 64

135 Conclusion The smart grid management architecture has to be constructed in a way supportive of a common management platform for the various smart grid systems. The architecture also has to strive towards the services management layer which can orchestrate the systems and corresponding elements involved to achieve the SLAs demanded by use cases non-functional requirements. The smart grid architect should also tactically choose standard protocols and local management policies whenever possible as explained below: Standard protocols: The EA needs to strive to adopt and encourage standard protocols for management such as SNMP, Syslog etc. IEC has features that could be used for configuration control and management. Standards based management functions will be easier to integrate for cross project management capabilities. Local and centralized management: The EA needs to strive for local management schemes in disaster recovery scenario. Structural Model Framework Template The SGRA Section 5, Smart Grid Reference Architecture Views, contains a structural model for each view (Application Services, Analytics Services, Data Services, Control Services, Security Services, Communications Services, and Management Services). Each is provided to help the smart grid architect consider how to best deploy the various services using a stylized representation of a typical utility network infrastructure. The template for these models is provided below [Figure 45] to help explain why services were distributed among the fourteen tiers in the template. The template graphical representation on the left side is identical across all seven structural models in Section 5. The right-hand table describes the devices and capabilities inherent in each tier. Services Classes Concepts Appendix 65

136 Figure 45 - Structural Model Template The smart grid architect, once all seven structural models are created for the utility, should consolidate the seven models into a single structural model for the entire smart grid. This consolidation exercise can strengthen the overall architecture. For example, since most analytics require communication and data support, any consolidated tier containing analytics, but missing these supporting services, should be studied more closely. All architectural weaknesses need to be directly addressed with impacted architectural views subsequently updated. Services Classes Concepts Appendix 66

137 Appendix C. Smart Grid Conceptual Architecture Project (SCAP) Background information on the Smart Grid Conceptual Architecture Project is provided on page ix. Business Requirements These top level business requirements are derived from the SCAP work done by the Smart Grid Interoperability Panel (SGIP) Smart Grid Architecture Committee (SGAC). These 380 requirement families abstract over 8000 more detailed business requirements identified by the SCAP (SGIP SGAC). Customer Domain Service Level data shall be collected to ensure service level agreements are met Communications shall meet defined Quality of Services (QoS) requirements. Shall be configurable. Shall be able to operate at a sufficient granularity (e.g., individual customer). Shall support continuous evolution of smart grids. Shall support aggregated management of Customer domain assets. Shall support recovery from an electric grid cold-start. Shall support role-based authorization. Shall assure access by authorized entities to data. Shall employ non-intrusive load monitoring where appropriate. Power Quality monitoring shall be supported. Shall respond to dynamic incentive signals (e.g., price signals). Shall leverage common infrastructure with other services (e.g., water management). Shall provide outage restoration management. Shall manage reactive power in customer loads (e.g., building energy management systems). Shall support charging of electric vehicles. Shall process demand response pre-event notifications. Shall provide energy management. Quality of Service (QoS) requirements shall be defined. Shall support display of information. Shall support manual override of automated actions. Shall require permission to access Customer domain services from other domains. Shall collect localized weather data. Shall manage Service Level Agreements (SLAs). Shall minimize load sags and swells. Shall collect load characteristics. Shall provide information in support of operational needs. Shall collect energy production environmental impact data. Shall support energy trading transactions. Smart Grid Conceptual Architecture Project (SCAP) Appendix 67

138 Generation Domain Transmission Shall control a plant's compliance to meet contractual obligations Require security Require a high availability of information flows Require a high accuracy of data Data shall be managed across organizational boundaries Require validation of the cost-benefit of the function Shall facilitate islanding of the power system Shall provide the ability to forecast Shall facilitate control of generator power Shall provide real time intelligence regarding equipment state Shall provide real time management of equipment Shall minimize impacts Shall facilitate scheduling Shall perform congestion management analysis on proposed energy schedules Shall support commitment activities Shall dispatch regulation units to provide voltage support based on emergency requirements Shall maintain the facilities in operating condition Shall facilitate the procedures particular to bringing a cold unit on line Shall provide advanced notice of scheduled outages Shall support minimum cost real time scheduling of generation units Shall ensure that the overall system remains intact despite severe challenges Shall facilitate monitoring of generator power Shall facilitate control of generator emissions Shall facilitate monitoring of generator emissions Shall monitor equipment contingencies Shall perform security analysis on proposed energy schedules Shall provide high quality power Shall provide ancillary services Shall provide rolling reserves Shall provide contingency Shall respond to changes in customer load Shall maintain the topology of the system including the ends of all segments Shall forecast alternatives for generation sources is to anticipate long term generation needs Shall maintain transmission system in operating condition Shall provide credible contingency information Shall determine long term future transmission system needs Shall have contingency plans for catastrophic events Shall optimize planned maintenance down-time Shall automatically collect information relating to system disturbances Shall automatically adjust voltage to optimize network efficiency Smart Grid Conceptual Architecture Project (SCAP) Appendix 68

139 Shall adjust power flow to optimize network efficiency Shall maintain the system within its limits of stability Shall monitor the status of measurement equipment Shall maintain load forecasts for all system equipment Shall recommended changes to correct limit violations Shall notify stakeholders of emergencies Shall detect equipment fault conditions Shall develop emergency response in case of catastrophic external events Shall provide operating guidelines for interfacing with distribution companies Shall provide short term forecasting for time frames of interest Shall provide short term forecasting for locations of interest Shall include DER in the balancing of demand Shall use activation of interruptible loads to balance the system Shall use rotating blackouts as a last resort to balance the system Shall manage substation voltages to manage the overall system Shall minimize transformer clipping conditions Shall maintain characteristic (nameplate) information on all system components Shall monitor equipment for equipment failures Shall track which actors had access to what system at what time and the changes they made Shall monitor the electrical flow characteristics at all major pieces of equipment in the system Shall simulate future configurations of the network Shall perform maintenance on the system to maintain the system in operational condition Shall manage the protection schemes to optimize the system protection Shall maintain the system within the frequency limits Shall shed load as required to maintain system stability Shall use wide area monitoring to prevent issues arising from incipient system instability Shall implement optimum control actions to maintain system stability Shall perform long term forecasting to support system reinforcement planning Shall do long term load forecasting to plan back-up generation Shall plan manual switching to prevent fault behavior Shall estimate the remaining capacity in the system Shall calculate system utilization to prevent overloads Shall predict failure of equipment Shall optimize usage of equipment Shall schedule equipment replacement is to prevent failure of equipment Shall dispatch field crews in an efficient manner Shall do 3-phase monitoring of the system Shall be able to do 3-phase unbalanced operation Shall optimize VAR control Shall support the integration of Storage Shall be able to handle sudden shifts in power flow Shall shed generation as required to maintain system stability Shall perform long term forecasting to support system extension planning Shall manage access to critical facilities Shall dynamically rate equipment capacity based on environmental factors Smart Grid Conceptual Architecture Project (SCAP) Appendix 69

140 Shall perform phase to phase balancing actions Service Provider Shall provide electrical service to its customers Shall provide electrical service to its customers that is of high power quality Shall be capable of detecting outages Shall employ techniques to minimize outage occurrences and duration Shall employ techniques to prevent damage resulting from malfunction of provider services Shall develop techniques to avoid permanent damage to the environment in the course of providing electrical service to its customers Shall provide information enabling the customers to make decisions on their DSM participation Shall provide a Demand Side Management function that is available Shall be capable of directly controlling customer loads. Shall provide a means of classifying information that allows for some data to be treated in a Confidential manner. Shall provide communication capability that is available. Shall provide a method of communications that provide 'rapid response' with a goal of maintaining load/generation equilibrium. Shall provide a communication capability that is accurate. Shall provide accurate individual meter readings to authorized parties. Shall provide a means of obtaining information from customers Shall establish two-way communication with customers. Shall be capable of obtaining readings from customer meters in a manner that minimizes reading errors. Shall provide a means of efficiently obtaining information from meters. Shall provide a means of obtaining accurate information from meters Shall read meters in a manner that is convenient to the customer. Shall implement accurate meter reading Shall provide means of communicating with customers Shall provide price of electricity information to their customers Shall provide individual meter readings to its customers at intervals that are aligned with customer needs Shall enable selling energy to customer Shall enable buying energy from customer Shall provide forecasts of demand for planning purposes Shall be capable of aggregating customers Shall bundle services Shall provide energy information services Shall provide power quality services Shall provide sub-metering service Shall provide technical support services Shall provide billing services Smart Grid Conceptual Architecture Project (SCAP) Appendix 70

141 Distribution Shall maintain a topological model of distribution circuits in near real-time Shall manage Intelligent alarm processing Optimizes Voltage in relation to Reactive Power Shall manage Voltage Shall support management of reactive power Shall manage work crews for authorized field work Provides distribution operation personnel with comprehensive training Shall monitor power quality on the distribution system Shall perform service restoration Shall support improvement of power quality Shall provide an audit trail Shall manage power quality information available from within customer sites Shall prepare for storm conditions Shall prepare for storm conditions Shall manage unbalanced current flows Shall manage unbalanced three-phase voltages Shall adjust feeder loads for optimal load flow Shall support Distribution Power Flow analyses Shall support Contingency Analysis Shall rank contingencies Shall perform Fault Level Analysis Shall support Short Circuit Analysis Shall support Contingency Load Transfer Shall support technical Loss Minimization Shall support state estimation Shall support monitoring of state Shall support planned outage management Shall support distribution planning Shall support field crew root cause analysis of distribution system problems Shall support transformer management Shall locate Faults Shall isolate Faults Shall manage system protection Shall manage system protection Shall support diagnostic analysis Shall support the management of Distributed Energy Resources (DER) Shall support real-time emergency operations Shall support outage management Shall support distribution asset management Shall support transmission operations Shall support updating field equipment Shall support system protection Shall support short term load forecasts Smart Grid Conceptual Architecture Project (SCAP) Appendix 71

142 Markets Shall support equipment level configuration management Shall support the development of the future distribution model Shall locate non-technical losses Shall support condition based maintenance Shall maintain an adequate operations platform Demand Response (DR) maintenance staff test Demand Response (DR) equipment is to validate the interconnecting infrastructure to allow the DR's to participate in the market operations Identical generation devices is to provide electrical power to residence while remaining in parallel with Energy Service Provider (ESP) Shall provide clear location based price signals that serve as a value benchmark for consumers to use as input in deciding when they use electricity Will enable customers aggregators to directly access the market Will enable demand-response aggregators to directly access the market Will provide location based price signals that reflect locational differences. Will correctly reflect the price of energy Will correctly reflect the price of renewable energy Will correctly reflect the price of emissions Will correctly reflect the price of transmission Will correctly reflect the price of reactive power Will correctly reflect the price of harmonics Will operate in a nodal fashion Will operate in a regional fashion Will provide forward pricing signals Will provide hedging contracts Will provide long term contracts Will provide historical pricing history Will provide historical demand Will provide forward demand curves Will coordinate cross market trading Will provide clearing of bi-lateral contracts Will provide registration of market participants Will provide direct access to the wholesale market for large enough end users Will provide access to trading analytics Will provide a safety valve for runaway trading Will verify the ability of a party to ability to cover their physical commitments. Will clear the market Will match counter parties to affect trades Will permit net open trades to the limits of the system Will net out trades Will encourage the emergence of a market maker in each area Will develop new trading vehicles Will encourage the development of a transparent market Smart Grid Conceptual Architecture Project (SCAP) Appendix 72

143 Operations Will encourage price discovery Will encourage counter party transparency Will register counter parties Will provide settlement information for the market Will encourage rapid settlement of the market Will provide a dispute registration mechanism Will provide a dispute resolution mechanism Will maintain a counter party history Will provide credit check capability Will provide background check capability Will create a framework of market rules Will be able to determine the impact of market changes from simulations Will provide location based grid state condition information Will communicate information to all interested parties Will retain a market history Will communicate abnormal conditions to all interested parties Will communicate all planned outages to interested parties Will run product auctions for all applicable products Will create long term demand forecasts by customer class Will provide demand forecast for multiple near term time periods Will provide demand response generation forecasts for multiple near term time blocks Will create market products that are capability based i.e. technology agnostic Will eliminate unnecessary barriers to entry Will develop new market products that account for advanced capabilities Will limit market exposure to the participant's credit limit Will make all markets and products available to DR and storage if capabilities can be met Will create products that encourage adequate capacity in the market Will create products that can be used to balance intermittency of renewable resources Will enable energy efficiency to participate in the market Will support participation by distributed energy resources Will create dynamic prices for all products and services Will provide intermittent generation forecasts for multiple near term time blocks Will develop new market products to price emissions Will develop new market products for voltage support Will develop new market products for VAR Will perform power flow simulations of the grid Will support exchange of network models with other stakeholders Provide the ability to manage direct load control programs Manage electrical frequency Report the health of the primary equipment to the control center Provide configuration management Provide performance management Smart Grid Conceptual Architecture Project (SCAP) Appendix 73

144 Shall be able to authenticate end-use customers Shall be able to communicate with end-use equipment Shall be able to communicate with measurement equipment Shall be able to communicate with mobile transportation loads Shall be able to manage end customers in load programs Shall communicate with end customers Shall control the flow of power to individual end customers Shall forecast resources Shall implement black start Manage restoration protocol Shall manage ancillary services Monitor DR/grid interaction Shall manage power quality Shall manage reserves Shall optimize asset utilization Shall provide intermittent generation forecast for multiple near-term time blocks Shall manage power interchange Shall manage the operation of the electrical grid Provide voltage regulation Shall be able to reconfigure (power) system Schedule generation Schedule transmission Shall actively manage demand response programs Shall be able to monitor micro-grids Shall be able to provide historical information for any specific time-frame Shall respond to abnormal conditions in a coordinated fashion Shall validate demand response operations Shall maintain emissions records for all operations Shall maintain the state of the systems managed Shall provide environmental monitoring Shall provide demand side management Assess the vulnerabilities of management systems Shall Maintain a secure environment for operations Shall maintain operations quality of service Shall manage communications between physical locations for all information Shall maintain load profiles of end customers Shall measure energy consumption of customers Shall provide information for Crew dispatching Shall provide control information to distributed resources Shall be able to access data from all distributed resources in a timely fashion Shall be able to forecast weather impacts for any reasonable time frame in the future Shall be able to manage the electrical protection of the system Shall be able to simulate future operations of the system Shall be able to minimize disruption of energy flow to end customers Shall maintain topology information for the power grid Smart Grid Conceptual Architecture Project (SCAP) Appendix 74

145 Shall be able to schedule maintenance at times of least impact Shall be able to support energy storage devices Shall be able to support distributed energy resources Shall manage voltage stability Shall be able to request procurement of additional market products Shall maximize available transmission capacity Shall review Remedial Action Schemes Shall perform contingency analysis Shall manage scheduled maintenance requests. Shall be able to analyze past grid events Shall be able to directly dispatch resources in exceptional circumstances Shall provide location based grid condition indicator Shall manage time synchronization of the grid Shall manage the variability of intermittent resources Shall provide demand forecast for multiple near term time periods Shall directly control end use devices Shall communicate with neighboring control areas Cross Cutting Domain Shall Manage Records for Auditing Shall Manage Records for Reporting Shall manage authentication Shall manage access control Shall manage discovery services Shall manage communities Shall manage firewall policies Shall Manage Privacy Policies Shall manage end devices Shall manage applications Shall manage security policies Shall manage distributed data Shall manage calibration of field equipment Shall manage non-repudiation Shall manage communications continuity Shall manage communications reliability Shall manage communications availability Shall manage personnel security credentials Shall manage risk assessments Shall manage services acquisition security policies Shall manage security engineering practices Shall manage Security Policy Exchange Service Shall provide security services against Denial-of-Service attacks Shall provide security assurance service Shall manage enforcement of security policies for field equipment Smart Grid Conceptual Architecture Project (SCAP) Appendix 75

146 Shall provide a Trust Establishment Security Service Shall provide secure mobility for field equipment communications Shall provide support for multi-cast communications Shall manage time synchronization of data Shall test equipment prior to use Shall manage underlying information support platforms Shall manage vulnerability assessments Shall manage network addressing Shall manage network multi-homing Shall manage transaction processing Shall manage physical security of smart grid operations Shall manage security of critical data Shall manage communication faults Shall manage communications performance Shall manage accounting for communications use Shall manage communications configuration Shall manage residual risk Distribution Automation (DA) critical data requires very high security Smart Grid Business Services Working from the requirements families listed above, the Smart Grid Architecture Committee (SGAC) developed a set of business services distributed across the seven Smart Grid domains. In addition, a set of cross-domain foundational business services was developed. In this section, the business services are presented by domain. The actual company performing the service may vary by area of the country. For instance, in California many of the market services belong to the California ISO; in Florida, there is no ISO, so these services belong to the state utilities. The SGAC team assigned to SCAP is developing additional use cases to define system requirements for two domains in need the Customer and the Service Provider. For the Customer domain, the business cases are from a customer point of view. For the Service Provider, the cases are from a non-electric service provider point of view. Once this work is complete, it is expected the following Business Services listings will change to accommodate the additional requirements. Market Domain Table 17 lists the SCAP business services required to support the overall business in the Market domain. The actual operator of the business services will vary by geographic region. Table 17 Market Domain Business Services Name Description Check Background Perform a background check on the participant Check Credit Confirm that the participant has enough credit to meet the requested transaction Smart Grid Conceptual Architecture Project (SCAP) Appendix 76

147 Name Clear Market Clear Trades Correlate Events Cross Market Trade Develop Balancing Products Develop Capacity Products Develop Distributed Energy Products Develop Energy Efficiency Products Develop GHG Emission Products Develop Market Products DR Forecast Exchange Network Models Facilitate Market Maker Facilitate Trades Forecast Intermittent Generation Grid Condition Indicator Long Term Demand Forecast Maintain History Maintain Market History Manage Aggregators Manage Direct Access Manage Dispute Manage Energy Contracts Manage Market Rules Manage Notification Manage Participation Market Transparency Monitor Markets Near Term Demand Forecast Net Trades Outage Notification Price Energy Products Protect Markets Description Identify the most economically optimal schedule of resources while maintaining the security of the grid. Provide the ability to financially clear bi-lateral trades Predict future events by analyzing current patterns Provide the ability to trade energy products between markets Develop market products that will provide the capabilities necessary to balance intermittent supply resources. Develop market products for capacity that encourages participation of various parties Develop market products for distributed energy resources that encourages participation of various resources Develop market products for energy efficiency that encourages participation of various parties Develop market products for GHG emissions that encourages participation of various parties Develop new market products Provide forecast of demand response capability over multiple near term time periods Provide a mechanism to exchange network models between entities Actively encourage the emergence of a market maker in new markets Facilitate bi-lateral trades between parties Provide intermittent generation forecasts for various timescales as needed for grid management Provide a location-based dynamic metric that illustrates the desired change in energy consumption. Maintain a long term demand forecast for various customer classes within the control area Maintain a history of market transactions Provide management of historical market information Determine mechanism for aggregators of energy resources to engagge in markets Provides the ability for end-customes to purchase power in the wholesale market Provide a mechanism to coordinate disputes on the market processes provide a mechanism to manage the operation of energy contracts between market participants Develop mechanisms to manage the evolution of market rules Provide notification of market events to interested parties develop guidelines for market participation Develop mechanisms to publish key market attributes to the public Analyze the effectiveness of market rules in maintaining an effective electric power market Provide forecast of demand over multiple near term time periods Will net trades across grid constraints up to the acceptable limit on that constraint Provide notification of planned outages to interested parties Provide time based prices for energy products for individual customers Provides a mechanism to protect the market from potentially damaging trading behavior Smart Grid Conceptual Architecture Project (SCAP) Appendix 77

148 Name Provide Communication Platform Provide Demand Forecast Provide Demand History Provide Operational Platform Reduce Settlement Timeline Register Dispute Register Participant Resolve Dispute Settle Market Simulate Market Simulate Power Flow Validate Commitment Validate DR Resource Interconnection Description Provide infrastructure for secure, accurate and timely communication Provide a forward looking schedule for energy requirements Provide history of demand at all tracked locations on the grid Provide infrastructure necessary to operate power market Develop mechanisms to settle market at the earliest possible time Provide a mechanism to register a dispute on the market processes Provides a customer the ability to engage in market activity Manages the progress of a registered dispute through resolution manage the settlement of market charges Provide mechanisms to simulate the power market Provide simulation mechanisms to analyze power flow of the electric grid Confirms the ability of a participant to provide their proposed commitments Validates that DR resources connection comply with relevant standards Operations Domain Table 18 lists SCAP business services required to support the overall business in the Operations domain. The actual operator of the business services will vary by geographic region. Table 18 Operations Domain Business Services Name Description Analyze Event Reconstruct the dynamic values of grid attributes around the time of a past grid event Authenticate User Confirm the identity of the user Contingency Analysis Calculate the impact of failures on the electrical power system Control Energy Resources Provide direct control of energy resources Control Power Flow Dynamically configure electrical power system to manage power flow to end-customer Define Transmission Capacity Define the dynamic safe operating limits of the transmission system Demand Forecast Provide forecast of demand over multiple near term time periods Energy Measurement Provide measurement of end-customer consumption Forecast Resources Provide forecasts of system resources Forecast Supply Resources Provide forecasts of supply resources Forecast Weather Impacts Forecast the potential impact of weather events on the grid assets Grid Condition Indicator Provide a location-based dynamic metric that illustrates the desired change in energy consumption. Maintain Network Model Maintain a time based model that represents the topology of the electrical power network Manage Configuration Maintain configuration parameters of equipment Manage Contingency Event Provide pre-determined protocol instructions for responding to a contingency event Manage Customers Facilitate communication with end customers Manage Demand Provide a mechanism to reduce end-customer energy costs Smart Grid Conceptual Architecture Project (SCAP) Appendix 78

149 Name Manage Direct Load Control Program Manage Distributed Energy Resources Manage Dr Program Manage Emissions Audit Manage Energy Resources Manage Energy Storage Manage Frequency Manage Grid Operations Manage Load Control Participants Manage Performance Manage Power Interchange Manage Power Quality Manage Quality Of Service (QoS) Manage Reserves Manage System Protection Manage System Re-Start Manage Variability Manage Voltage Minimize Disruption Monitor Emissions Monitor Energy Resources Monitor Equipment Monitor Resources Optimize Asset Utilization Outage Analysis Physical Security Provide Black-Start Provide Connectivity Provide Demand Profile Provide Historical Data Request Market Products Review Protection Schemes Schedule Generation Schedule Outages Schedule Transmission Schedule Work Crew Simulate System Description Offer managed load reduction schemes using direct operation of devices Develop mechanisms to include distributed energy resources (DER) in the operation of the electrical grid Offer managed load reduction schemes Maintain emissions history for grid assets Facilitate management communications with energy sources attached to the grid Develop mechanisms to include energy storage in the operation of the electrical grid Maintain electrical frequency within applicable limits Balance supply with demand of the electrical grid Facilitate participant involvement in load control Maintain performance of equipment within applicable limits Ensure power flow with neighboring control areas is coordinated Maintain attributes of the power flow on the system Provide mechanism to ensure business services remain within predetermined metrics Manage an inventory of reserves to cover contingencies Provide protection schemes that ensure the security of the system under fault conditions Re-initiate stable operation of a power grid after a failure Provide reserves that can balance the variability of energy resources Maintain electrical voltage within applicable limits Develop mechanisms to minimize interruption of electrical service to customers Measure the environmental impact of grid assets Report the status of energy sources attached to the grid Report the health of equipment Measure the real-time status of grid resources Identify dynamic limits for grid assets Calculate the impact of maintenance requests on the electrical power system Provide secure locations Provide the ability to produce power without first requiring power from the grid Maintain communication to equipment Provide a time versus demand curve that represents the typical demand of a customer Provides information for requested time frame Provides the capability to request the procurement of energy market products Reviews proposed remedial action schemes (RAS) Identify economically optimal schedules for supply resources Schedules outages when they will have the least impact to the reliable operation of the grid Provide a mechanism to match supply schedules with transmission capacity Schedule the appropriate resources for outage resolution Provide simulation of the electrical power system Smart Grid Conceptual Architecture Project (SCAP) Appendix 79

150 Name Synchronize Time Validate Demand Response Vulnerability Assessment Description Manage frequency so that a clock connected to the grid is kept within tolerance of a reference clock Evaluate response of resource to a demand response dispatch Assess the resilience of grid management systems Service Provider Domain Table 19 lists SCAP business services required to support the overall business in the Service Provider domain. The actual operator of the business services will vary by geographic region. Table 19 Service Provider Domain Business Services Name Aggregate Customers Bundle Services Buy Energy Classify Data Collect Meter Data Communicate Price Deliver Electrical Service Detect Outages Direct Load Control Forecast Demand Manage Accuracy Of Communication Manage Accuracy Of Information Manage Availability Of Communication Manage Communication Manage Customer Communication Manage Energy Balance Manage Meter Data Collection Manage On-Going Two-Way Communication Manage Outages Obtain Customer Device Information Prevent Demand Asset Damage Prevent Environmental Damage Provide Billing Services Description Provide mechanisms to aggregate customers for various electricity buying and selling options Provide mechanisms to bundle various services Maintain protocols that enable buying energy Provide predetermined protocols for classifying data Collect accurate meter data from end customer in a manner that is convenient to the customer Communicate customer specific price of electricity information to end customer Deliver high quality electrical service to customers with in applicable limits Ensure mechanisms to detect outages Control loads directly to respond to the reliability needs of the electric grid Determine protocols for forecasting demand Ensure communication of system parameters are accurate Ensure communication of system parameters are accurate Ensure communication of system parameters available to the customer within applicable limits Maintain communication of system parameters as they relate to customer decision making Maintain communication of system parameters as they relate to customer decision making Maintain communication of system parameters as they relate to customer decision making Collect accurate meter data with various levels of granularity from end customer using protocols to minimize cost Maintain communication channel with the end customer that allows two-way communication Maintain the distribution system to minimize outage occurrences Maintain communication channel with the end customer that allows two-way communication Ensure mechanisms to prevent damage to demand assets Ensure mechanisms to prevent environmental damage Provide accurate billing data to customers in a timely manner Smart Grid Conceptual Architecture Project (SCAP) Appendix 80

151 Name Provide Energy Information Services Provide Power Quality Services Provide Sub Metering Service Provide Technical Support Services Sell Energy Share Meter Data Description Maintain energy information services that allow the customer to manage their loads Manage the quality of electrical service by ensuring that it is within applicable limits Make accurate sub metering data of various loads available to various authorized parties Provide technical support services that allow the customer to manage their loads Maintain protocols that enable selling energy Ensure authorized parties have access to accurate meter data Bulk Generation Domain Table 20 lists SCAP business services required to support the overall business in the Generation domain. The actual operator of the business services will vary by geographic region. Table 20 Generation Domain Business Services Name Analyze Schedules Balance System Discover Equipment Status Least Cost Dispatch Maintain Facilities Maintain System Integrity Manage Adverse Conditions Manage Ancillary Services Manage Cold Start Manage Contractual Compliance Manage Control Of Generation Manage Data Sharing Across Organizations Manage Emissions From Units Manage Equipment Manage Forecasting Manage Islanding Manage Past Incipient Failures Manage Power Quality Manage Scheduled Outages Description Provide analytics to determine issues with planned operations Provide a mechanism to respond to changes in the supply-load equation that will maintain the system within compliance limits Provide a mechanism to discover the current characteristics of equipment Provide a mechanism to allow for the lowest cost units to be called on first for providing energy Provide maintenance sufficient to maintain generation units in operational condition Provide the controls that will facilitate several scenarios for keeping the energy flowing through the system when issues occur Provide analytics to determine adverse impacts that can be managed before they become operational problems Provide a set of reserve resources that can be used to maintain the grid within compliance limits Bring an out of service unit on line as an independent source Units have contractual commitments on when to run at what rate Provide a mechanism to directly control the inputs and outputs of generation units Provide stakeholders in different organizations with required data Provide a mechanism that allows control of the emissions of a generation unit Provide a mechanism to control equipment operation in due time Provide a mechanism to create scenarios of future demand for supply capability Provide the ability to disconnect sections of the grid from each other to operate independently Provide a mechanism switch away from incipient failures to maintain the system within compliance limits Provide a mechanism to maintain energy quality within compliance limits Plan out of service time for maintenance on generation facilities Smart Grid Conceptual Architecture Project (SCAP) Appendix 81

152 Name Manage Scheduling Of Facilities Manage Unit Commitment Monitor Generator Emissions Monitor Generator Performance Provide Communications Provide Cost-Benefit Analysis Provide Fail Over Options Provide Security Description Provide analytics that will develop an operations plan for generation units based on known characteristics Provide a mechanism to confirm schedule operational compliance in generation units Provide a mechanism to measure emissions of a generation unit Provide a mechanism to measure generation unit operational characteristics Provide a way to coordinate different facilities operations regardless of outside interference Maintain business cases that provide traceability of project costs and benefits Provide a mechanism to maintain the system within compliance limits when a resource is lost The expectation is that the generation units will provide enough reserve to allow for the loss of part of the system Transmission Domain Table 21 lists SCAP business services required to support the overall business in the Transmission domain. The actual operator of the business services will vary by geographic region. Table 21 Transmission Domain Business Services Name Balance Phases Communicate Emergencies Conduct Grid Planning Conduct Planning Conduct Short Term Forecasts Create Long Term Forecasts Detect Faults Dispatch Maintenance Teams Forecast Equipment Life Forecast Equipment Loading Maintain Contingency Plans Maintain Equipment Characteristics Maintain Physical Security Maintain Stability Maintain State Information Maintain System Maintain Topology Description Maintain the energy balance from one phase to another phase Maintain channel to stakeholders to for emergency information Create scenarios for future system sizing to support possible load changes Create scenarios for future system supply to support possible load changes Manage a set of near term projections for system use by location Maintain a set of simulations that can provide future loading projections on the system Monitor system for system faults (outages) Manage the available workforce Using the information gathered in monitor equipment determine the remaining life of electrical components installed in the grid Maintain equipment specific future load profiles Develop plans, including fail over, that allow the system to continue to operate when problems arise Maintain all characteristics of electrical components installed in the system Manage the physical facilities of the system to maintain control of physical access to the system Manage energy flow in the grid to ensure stable grid operation Using the information gathered in monitor equipment determine the remaining capacity of the system to handle more energy Manage the maintenance of the system equipment to maintain operational capability Manage the mapping of the logical connectivity possible in the overall system Smart Grid Conceptual Architecture Project (SCAP) Appendix 82

153 Name Maintain Voltage Manage Access Manage Blackouts Manage Demand Response Manage DER Manage Emergency Response Manage Equipment Rating Manage Interconnect Guidelines Manage Load Limits Manage Load Shed Manage Power Flow Manage Protection Scheme Manage Stability Manage Storage Manage Supply Manage Switching Manage Transformer Load Manage Var Manage Voltage Monitor All Phases Monitor Equipment Monitor Equipment Accuracy Monitor System Status Operate Unbalanced Optimize Equipment Optimize Planned Outage Optimize Power Flow Regulate Frequency Simulate System Description Manage embedded equipment in the grid to adjust voltage to stay within compliance limits Manage a mechanism to monitor who accessed what when Operate a mechanism to manage rolling blackouts as part of an overall balancing scheme Operate a mechanism to manage interruptible loads as part of an overall balancing scheme Manage DER as a part of the resource mix for balancing energy demands Maintain a plan for recovering from major system disruptions Using the information gathered in monitor equipment determine the current energy capacity of each electrical component of the system Maintain a mechanism for managing the exchange of energy with distribution Operate equipment in a manner that avoids overloads Manage the system balance by shedding load React to rapid changes in the energy flow to maintain system parameters within compliance limits Maintain a set of schemes for protecting the system from damage by electrical overload Manage energy flow in the grid to ensure stable grid operation Maintain the integrated operation of storage at any point in the system Manage the level of supply in the system to balance the system while maintaining the compliance limits Manage a set of protocols to switch load on the system Maintain the load on transformers to avoid distortion of the wave form Manage VAR for all customers within compliance limits Operate a mechanism to regulate voltage within compliance limits for all customers Monitor all three phases of the system for energy flow characteristics Operate a mechanism to maintain temporal characteristics on electrical components installed in the system Maintain status of accuracy for instrumentation in the field Manage instrumentation in the grid to maintain a picture of the grid situation Manage the system will different amounts of power flowing on each of the three phases Manage energy flow so that equipment usage is optimized Manage the optimization of equipment availability for maintenance Manage the flow of energy in the grid to minimize losses while supplying all demand Maintain system frequency within compliance limits Shall manage a mechanism to create on demand digital simulations of the system Distribution Domain Table 22 lists SCAP business services required to support the overall business in the Distribution domain. In most cases the distribution utility will operate all services. This is the only domain likely to be similar from region to region. Smart Grid Conceptual Architecture Project (SCAP) Appendix 83

154 Table 22 Distribution Domain Business Services Name Audit trail management Automated Service Restoration Condition Based Maintenance Contingency Load Transfer Contingency Ranking Customer Power Quality Information Management Development of Future Distribution Model Diagnostic Analysis of Distribution Systems Distributed Energy Resource Management Distribution Energy Loss Minimization Distribution operations training Distribution Planning Distribution Systems Asset Management Distribution Systems Field Equipment Configuration Management Distribution Systems Support of Transmission Operations Fault Isolation Fault Level Analysis Fault Location Field Equipment Upgrade Management Field work management Intelligent Alarm Processing Loss location management Manage System Protection Execution Manage unbalanced current Optimal Load Flow Description This service provides the documentation necessary to provide an audit trail for system event reporting. This service manages automated service restoration functions. Measurement of this service would follow accepted indices on system restoration performance. This service manages condition based maintenance for distribution field equipment. This service manages contingency load transfers. This service ranks contingencies for distribution system operators. This service manages power quality information available from within customer sites throughout the distribution system. This service manages the development of future distribution systems This service manages the diagnostic analysis of distribution system field equipment. This service provides management of Distributed Energy Resources (DER) across the distribution system. This service manages the technical loss of energy to minimize losses on the distribution system. This service provides a comprehensive program of training for distribution operations personnel. This service manages the distribution planning processes. This service manages distribution systems asset management processes. This service manages the configuration of field equipment. Metrics for this service are determined by emerging policies for maintaining field equipment documentation. This service manages the integration of distribution systems in support of transmission operations. This service manages the isolation of faults to minimize damage to utility and customer equipment. This service manages Fault Level Analysis on distribution systems This service manages the location of faults on the distribution system. Metrics for this service would follow industry indices of system performance This service manages the updating of field equipment This service manages the dispatch of work crews authorized for servicing field equipment This service manages all levels of alarms to maintain correct operator action priorities. The metric for this service is related to correctly handling the variety of alarms that may be faced by operators. This service manages the location of non-technical energy losses on the distribution system. This service manages system protection post event analysis; This service manages the analysis of protection actions following an event episode. This service manages unbalanced current flows to keep distribution systems within acceptable engineering tolerances to optimize efficiency. This service optimizes load flows on the distribution system Smart Grid Conceptual Architecture Project (SCAP) Appendix 84

155 Name Optimize Voltage to Reactive Power Outage Management Planned Outage Management Power Flow Analyses Power Quality Improvement Power Quality Monitoring Prepare Static Systems for Storm Conditions Reactive Power Management Real-Time Emergency Operations Root Cause Analysis Short Circuit Analysis Short Term Load Forecasts State Estimation Support System Protection System State Monitoring Topological model management Transformer Management Voltage Management Description This service manages voltage in relation to reactive power within the distribution system. This service manages outages on distribution systems. The metric for this service is related to system reliability indices. This service supports planned outage management for distribution systems. This service manages the Power Flow Analyses processes. This service manages the improvement of power quality within the distribution system. This service monitors power quality on the distribution system. This service includes the management of static preparations in anticipation of system disturbances due to impending storm conditions This service manages reactive power on the distribution system to maintain reactive power This service manages real-time emergency operations within the distribution system. This service manages the support of field crew investigations into root cause analysis of distribution system failures This service manages the analysis of short circuits within distribution systems This service manages short term load forecasting This service manages the state estimation process using available information available from field equipment. This service manages system protection functions This service manages the migration to state monitoring making use of increasing numbers of field devices that have monitoring capability. Metrics for this service would be the reduction in the band of error surrounding system state. This service maintains the topological model to maintain the system model in near real-time. This service manages the life-cycle of transformers in field operations This service manages customer service delivery voltages to stay within industry accepted tolerances Customer Domain Table 23 lists SCAP business services required to support the overall business in the Customer domain. The actual operator of the business services will vary by geographic region. Table 23 Customer Domain Business Services Name Description Access Control Management Provide a mechanism to support role-based access control to services. Configuration Management Manage configuration of the system. Deployment Management Manage ongoing revision of the system. Energy Asset Aggregation Manages a group of devices in the customer domain Energy Management Provide a mechanism to influence operation of devices to meet business energy-usage goals. Energy Monitoring Provide a mechanism to monitor aspects of energy as required to meet business goals. Smart Grid Conceptual Architecture Project (SCAP) Appendix 85

156 Name Environmental Monitoring Information Presentation Infrastructure Planning Manual Override Non-Intrusive Monitoring Outage Management Description Provide a mechanism to collect environmental data about the impact of energy supply sources in the system. Provide a mechanism to display information from the system. Building the Smart Grid System to support additional devices (e.g. Water, Gas) Provide a mechanism to override automated actions while assuring safety. Derive device load levels using non-intrusive measurements combined with analytic methods. Provide a mechanism to mitigate impact power failures through use of alternate energy supply sources. Smart Grid Cross-Domain Foundational Services Cross Domain Foundation services are relied upon by all business domains and are therefore required to be interoperable across domains to enable flexible and resilient grid architecture. For example, two domains utilizing disparate encryption mechanisms will require additional work to support their interaction and be indicative of architectural fragility. Cross domain services break down into three basic groups: Security Communication Data Each group is discussed below. Security Services Central Security Services are comprised of Automated Services as well as the Managed Services responsible for configuring the Automated Services. Automated Services require no human intervention once configured and are built for speed and efficiency, whereas Managed Services expect user input to set configurations, preferences, etc. for the particular characteristic being managed. The NISTIR 7826 Security Requirements have been translated into the security services in Table 24. Table 24 Security Services Name Access Control Account Management Alarming Archive Audit Audit Qualification Authorization Backup Description Manages the secure admission. (see Central Security). Maintains actor security credentials Provides a mechanism to pass alerts to actors Providing an accurate long-term copy of information Provides a rigorous review of actor actions Provides a trusted method to determine that audit actors are competent Provides permission for an actor to take an action A process that provides an accurate alternate copy of information Smart Grid Conceptual Architecture Project (SCAP) Appendix 86

157 Name Camouflage Central Security Challenge Management Classification Management Compliance Management Configuration Continuity Control Management Corrective Action Management Disposal Distraction Encryption Entry Management Identification Identity Management Incident Reporting Information Release Management IPR management Key Management Life Cycle Management Log Protection Logging Non-Repudiation Parameter Monitoring Password Management Penetration Physical Security Privilege Management Reliability Response Management Retention Risk Assessment Risk Management Role Management Screening Hiding items from unauthorized actors Description The Central Security Service shall maintain a database of transaction permissions at the component level. Specifically, the Central Security Service explicitly defines which users and components have permission to perform what transactions in what context. All Smart Grid end-users and components must verify sufficient privileges prior to execution of any transaction instruction. Maintains a secure challenge process for actors Manages the labeling of sensitive information Maintains a process to ensure that actors have not violated security policy Arranges the pieces to provides an authorized whole Provides for actors to continue to securely function at an acceptable level Maintains a set of limits on an actor's access A process to ensure that any security issues are closed Provides a trusted process to remove information permanently A process of enticing unauthorized actors to interact with decoys Provides a mechanism to encode information in a secure fashion Provides a mechanism for allowing an authorized actor access A trusted process of determining the identity of an actor With the large number of end-users of Smart Grid, including the customers, utility employees, grid control operators, etc., the Smart Grid components must provide unique and traceable identification of each user as well as data with each message and transaction. provides a method to report security breaches to a trusted authority Allows control of information to be passed to other actors A process to manage rights to use intellectual property Provides a trusted process to maintain the security of a secret A process to manage the cradle to grave security Provides a mechanism that keeps logs from being tampered with Provides an auditable trail of actors access Provides a mechanism to prove that a specific actor took a specific action Watching the edges of a physical location for unauthorized actions Manages the process of dealing with all actor passwords Provides a mechanism to test the security environment for vulnerabilities Provides a process to ensure that physical items are secure from unauthorized access Maintains the privileges for actors Provides a process to monitor actors for unexpected changes in behavior Provides a mechanism to manage actor actions for alerts Provides a process to determine parameters for saving information Determines the risk associated with the security environment Maintains an assessment of the vulnerabilities in the security environment Manages actor roles Maintains a trusted process for reviewing actors for credentials Smart Grid Conceptual Architecture Project (SCAP) Appendix 87

158 Name Security Assessment Security Authority Security Management Security Monitoring Security Policy Management Security Reporting Security Test Development Security Training Management Separation Management Threshold Management Time Stamp Update Management Vulnerability Assessment Vulnerability Management Description Provides a mechanism to determine security needs Provides trusted resource that can authorize changes Maintains the overall security environment The analysis of collected information on the status of the security environment Creates an overall policy for security Provides a mechanism to inform trusted actors Provides a trusted process to develop test processes for security Provides actors with training required to maintain their credentials Provides a mechanism to ensure that an actor does not have multiple security roles Provides a mechanism to set parameters for generating an alert Provides an auditable mechanism for attaching accurate time to an action A process that updates to the security system navigate prior to being placed in service Determines what potential security holes exist Provides a mechanism to monitor the status of security holes Communication Services Communication services are fundamental to electric grid operations. They underpin the means by which devices become intelligent, for information to flow, controls to be executed, and for people to manage utility functions and services. Movement toward a smarter grid will necessitate vast quantities of new smart devices to be implemented all needing some form of communications. The fact that many of these devices don t yet exist adds to the challenge. As the devices are implemented, their functionality and value will expand as grid operators figure out innovative ways to leverage smart device intelligence. Thus, it is imperative for the Smart Grid communication architecture to be extraordinarily forward looking and accommodating. At the communication services layer, the following functions are required: 1. Transport services to move information around the network, 2. Virtualization services to help segregate and optimize transport services, 3. Control services to help manage information flow and support endpoints wanting network access, 4. Gateway services to facilitate legacy applications to utilize next generation networks, and 5. Mobility services to support roaming or mobile endpoints. Smart Grid Conceptual Architecture Project (SCAP) Appendix 88

159 Figure 46 - Communication Services Layer Functions Table 25 lists the network and communication services required to support the smart grid. Table 25 Communications Services Service Access Of Media Address Management Administration Service Alerts Application Caching Assured Delivery Asynchronous Messaging Atomic Multicasting Automated Messaging Automated Process Automatic Call Routing Backbone Backup Broker Support Description The capability for a user to view specific Library Mgmt System functions depending on the "Set-up and Control" for that user e.g. searching more than one repository. Assignment and maintenance of dynamic device addresses to ensure that there are no duplicate addresses on the network The maintenance and management of local policies, activities or procedures. Notification using some mechanism (e.g. SMTP message) that a message was delivered/notdelivered/dispatched/rejected/accepted. The capability of the system to provide the capability to cache objects, pages etc to optimize access speeds. This includes caching for static and dynamic pages. The ability to know that when a message is sent thru or between systems that the message will either be received or that notification will happen to the originating system or a designated operator Fire and Forget messaging system that exists between source and target applications or/and databases. The capability to send a message to more than one target. The message can be atomic (fire and forget). Machine generated messages that are sent to specified addresses in pre-designed formats Event triggered machine operations with a pre-designed set of routines Using the header information of a telephone message to send the call to a specific hand set or one of a bank of handsets The high speed interconnect on the network that provides the location where multiple systems and users can exchange data. (NOTE: The backbone may be wholly contained in a single device such as an enterprise router) The processing of saving documents for storage to another storage medium e.g. disk for retrieval/restoration as needed. Backups are normally removed to another location for safe storage The ability to manage exchange of information between systems and processes through a central controller Smart Grid Conceptual Architecture Project (SCAP) Appendix 89

160 Call Groups Service Capacity Management Capacity Monitoring Chat Management Collaboration Computer Telephony Integration (CTI) Configuration Management Connection Pooling Connectors Content Delivery Data Acquisition Data Cleansing Data Distribution Datastream Conversion Delivery Assurance Delivery Mgmt Service Dial-In Management Services Directory Framework Management Directory Framework Parameter and Control Set-ups Dispatch Description Maintenance of an automated list of radio handsets that make up the radios attached to a specific transmitter on a specific frequency and the unique identifier of each handset, so that the radio call goes to the right transmitter and the right handset The ability to create and manage the rules required to determine how resources will be assigned to the available services The ability to monitor the loads and performance of the systems to determine when the system either needs additional resources assigned or should shed tasks or users The ability to set rules and define forums or private areas for on-line chat for collaboration or discussion Allows near real-time cooperation between two or more people in two or more locations to communicate, operate on the same data and see each other's operations and results in an interactive fashion as though they were in a meeting The ability to use a computer to answer the phone and/or to provide information automatically to the human on the answering end of the phone to support the processing of the call in a faster and more orderly fashion The ability to manage the system software, application, system settings, and user information on similar devices automatically and from a remote location Connection pooling allows the efficient use of data access services. The creation and ripping down of a connection to a database takes up valuable processing cycles pooling allows a connection that is already existing to be used by many separate processes (by queuing requests). By managing a fixed number of connections (which can automatically be increased in multiples), the connection time overheads are largely removed, resulting in potentially significant performance benefits. Exposure of functionality (application or databases) that enables an application or database to conduct transactions (CRUD) with the target. Involves the delivery of content to a particular channel Collection of a stream of data from instrumentation and sensors that either constantly send data or are trigger driven, the acquisition has to collect the data and store it for each required reading, no unanticipated down time is acceptable An automated process of reviewing data for impurities and correcting these defects automatically if possible, based on a set of defined rules, logging all errors for manual review The ability to transport from a central master data set information to remote locations Translate incoming and outgoing data (or sets of transactions) into the required formats based on business rules supporting one or more clients/systems on the fly The ability to assure delivery of a message packet to another system or notify the sending system or a designated operator Management of delivery of services either by internal or external resources, ie collection of waste materials. Monitoring of inventory of items, ie bins. The ability to manage the occasionally connected data lines from various partners and customers, with regard to quality of connection, connection time and security The Management activates of the System Directory, includes distributed management. Configuration, Set-up of seed data, business rules, business defined criteria that govern the Directory. May actually call other services such as Users, Groups, Authorizations, etc. Firing of the message to the Messaging Service or Network Interface Service. Smart Grid Conceptual Architecture Project (SCAP) Appendix 90

161 Service Domain Name Services Electronic Messaging Enterprise Storage Event Monitoring Event Notification Events Flow Exception Management External File Transfer Fault Monitoring Gateway Control Gateway Management Gateway Services Information Exchange Infrastructure Management Infrastructure Monitoring Infrastructure Prioritization Integrated Voice Response (IVR) Inter-Network Connections Interactive White Board Interconnection Services - Network Description Creation and maintenance of a cross-reference list of physical addresses, routes and aliases for devices on the network (internal and possibly external), so that humans can enter the alias to automatically communicate with the desired device using industry standard The use of digital distribution of information over a network that may or may not extend beyond the enterprise with a central distribution and addressing center The ability to share storage between services and applications within the enterprise to provide flexibility and management for the storage media and the content of the media Creation, monitoring and extraction an auditable log of events and using pattern matching to a set of pre-defined rules to alert an operator of an abnormal condition. May also feed alerts to a device for further processing or process control Notification to a user or device that an event has occurred, implemented thru a series of rules and devices, so that the system will keep escalating utilizing different methods or addresses until it is successful in delivering the alert the ability to orchestrate a cascading set of events in an automated fashion thru a broker or other central controller Provide reports of incidents in any service in the enterprise. The ability to send a digital package of information via a standard transfer service between locations (e.g. FTP) The ability to monitor the services for unexpected operations and determine the cause of the issue - then report the issues to an appropriate location based on rules The ability to control the flow of information thru either an internal or an external gateway based on the rules that have been installed in the gateway The ability to create and maintain the rules that will be used to determine how a gateway will react to specific messages, either from the addressing or the content of the message The ability to use a central director to determine how to route specific digital documents and information around the enterprise or to and from partners Automated transfer of data between two or more applications or two or more databases based on pre-defined keys, formats and rules The ability to remotely maintain the infrastructure in the organization including creating and maintaining rules about resource prioritization, allocation, administration authority, etc. The ability to monitor what is happening in the enterprise infrastructure, log any events that are not expected, and notify an administrator based on the notification rules The ability to determine which service or information should go first, when there is contention for the resources that comprise the infrastructure The ability to use technology to create and maintain call message hierarchies that are used to interact with inbound callers to provide information for standard queries without the intervention of a human operator and record the results of the query. The IVR normally has the ability to route the call and any information gathered to a human operator if the query cannot be successfully answered The ability to peer with or connect to foreign network for communication, both networks that are controlled by partners and those that are public networks A process and supporting software to allow two or more people at two or more physical terminals to view an electronic chalkboard and to add information to the chalkboard that is visible to everyone else in the session. Change ability can be controlled via security services The ability to connect (or peer) one or more networks together using standard industry protocols Smart Grid Conceptual Architecture Project (SCAP) Appendix 91

162 Service Interconnection Services - Wired Interconnection Services - Wireless Intermittent Services Internet Access Link Management Linking Services Log/Audit Logging Message Completeness Management Message Composition/ Decomposition Message Filtering Message Logging Message Routing Message Translation Mobile Mobile Device Management Network & Connectivity Management Network Caching Network Device Maintenance Description The ability to physically connect devices so that they can use standard protocols to communicate both with other devices and with the enterprise The ability to connect devices so that they can use standard protocols to communicate both with other devices and with the enterprise over an Radio Frequency carrier Services which can be started and stopped, either by demand, or by rule or by administration which can be used to change the behavior of the rest of the environment (e.g. an intermittent service to re-route traffic because of a denial of services attack for a commercial web site) Provision to allow authorized users and devices productive access to the Internet using IP protocols Maintenance of specific legs in the network, providing health information on the link to the operator, so that adjustments can be made The Capability to access other modules functionality in the application e.g. accessing property module to associate people, rates or licenses. Record the master data management system and master data management service events, normal and abnormal. The full Audit service provides all the logging necessary to support a rich history of master data management processing events. In practice the Audit service may rarely be used to its full capacity, partly because the Audit service may have an impact on the performance of the master data management system. Audit trail of jobs processed, manual interventions and incidents The ability to detect when a message does not contain a complete payload and notify the sending systems Manages the copying of a message to different formats - utilizes the Transformation service, can be utilized by multicasting. Also can involve breaking the message up into components. The determination of where a message is allowed to be sent/received based on security and data protection requirements. Creation of a archival quality list of the header or header and body information of all the information being passed between a specified set of addresses or with specific header information The ability to route messages from one system to another automatically The Transformations from a source message to the requirements of the target. The ability to disconnect from the network and work on standalone applications, supports bidirectional synchronization when reconnected to the network The ability to create rules about how mobile devices connect to the enterprise and what requirements have to be met prior to allowing access (e.g. authentication, authorization, patch level, etc) The ability to create and maintain the rules by which a network can connect to another network either with a partner or a public network Caching is often used to improve performance of systems, by moving the data nearer the consumer. This can be by caching data in memory for database access, caching networked data on a local storage device for network access, caching pre-constructed Web pages and other information for use either externally (often referred to as reverse caching) or internally (reducing network usage). The ability to remotely manage the health of a network device and update any firmware or software installed in the device Smart Grid Conceptual Architecture Project (SCAP) Appendix 92

163 Service Network Device Security Management Network Interfaces Network Management Network Monitor Network Performance Tuning Network Traffic Simulation Node Management Non-Delivery Notification Services Monitoring Object Inspection Path Tracking Performance Modeling Portal Private Wireless Configuration Management (Talkgroups) Queue Management Radio Radio Frequency Management Real-time Reliable Multicasting Remote Access Remote access management Description The ability to remotely manage the rules, settings and permissions for the secure operation of the netowork Provide network access outside the Local Area network (other LANs in the Wide Area or external networks) Management of the network devices including: routers, hubs and switch monitoring and configuration control Logging, triggering (simple test) and analyzing (complex time delayed testing) the movement of traffic on the network to determine the health of the network and alerting devices or humans of problems with the health, based on a pre-defined set of rules. The ability to monitor and automatically adjust the performance of a network or a portion of a network the ability to create and maintain a model of the actual network and to create traffic scenarios for the network to determine when the network would run into contention and other operational issues The ability to allocate and manage the way services access nodes on a processing system The ability to recognize that a message that should have been delivered has not been delivered and make the appropriate re-tries or notifications The ability to monitor and report the health of the notification process The ability to review object for integrity and replace or sequester objects that do not pass the testing The ability to determine the route that traffic took during the trip between locations including all the devices that it crossed during the trip and time statistics The ability to model a service and its supporting infrastructure and understand how the service will perform based on scenarios of usage Providing an individual human readable front end to the information and systems that they are authorized to access, in a common format to hide the complexity of the underlying systems in a commercially standard format The ability to segment the user population by various parameters to efficiently use the limited radio bandwidth that is available from a transponder Handling of the incoming and outgoing queues of events, and messages -including the ability to recognize priority items in the queue and move them forward, ensuring that all items are released from the queue to the process that they are assigned to Supports use of specific frequencies to send information thru the air between handsets and fixed antennas using a known addressing scheme Maintaining control of the specific radio frequencies that are licensed to the facility to reduce non-productive messaging and optimize productive use The ability to interact with data in a short enough period of time to effect the next operation on the item reporting the data (e.g. acting on data that shows a broken drill bit in a tool, before it can eject the current part and start on the next one) so that the required actions are taken within the cycle of the device Multicasting that results in a response. Allowing users and devices to connect to the network from public or non-dedicated services such as the internet or dial in services The ability to limit who can use remote access and what they can do from a remote location Smart Grid Conceptual Architecture Project (SCAP) Appendix 93

164 Service Remote access security management Remote configuration management Remote multicast printing Remote print Remote systems administration Routing Routing modeling Sensor SMS SMS Channel Synchronization Synchronous Network Interfaces System Monitoring Telephony Text Chat Traffic Prioritization Transmission Trigger (Initiate) Tunneled Transmission User Directory Services User Monitoring Video Communication Video management Voice Channel Voice Management Description The ability to provide authentication and other security mechanisms for remote access The ability to maintain the configuration of clients that are remotely accessing services or assets. The ability to send a single print file to a number of remote locations for remote printing The ability to create hard copy documents in a geographic location that is not where the print file is generated The ability to securely manage servers and other assets when not sitting on the console that is directly attached to the device Sending traffic over only required network segments to a known address on the other end of the network links and preventing traffic without valid need from affecting the operation of and/or being visible to non-required network segments. Limiting the visibility of broadcast messages to specifically required network segments, based on the header of the broadcast Routing rules for transporting master data management occurences from master data management system to local application. Interrogates and reports real-time readings on a specific physical characteristic (ie temperature, or a chemical in the air) The ability to communicate with devices that use the industry standard for short messaging service in a bi-directional fashion SMS Channel Service (Short Message Service used for message event notification and simple interaction services. This is a specific type of Mobile Service). The capability to manage the synchronization of user authentication and other details to ensure that information across applications is updated. This is the "Join" Service that allows relationship management through synchronization and replication. The ability to do bi-directional handshaking between devices on the network where each step is acknowledged by the other device on the network System monitoring based on certain criteria e.g. KPIs. Support for the standard addressing scheme for telephone handsets to route voice conversations and other voice encoded traffic over the telephone network Alpha-numeric information for collaboration and discussion between devices in near real time Using the header information on IP traffic to determine whether to allow it unlimited, limited or delayed assess to slower network links Providing information for routing over an IP network with machine readable headers Signalize an event to an automated process in the course of the master data management process. Provides information for routing over an IP network with external headers, to a specific gateway address, while encrypting the real headers and information as data The facility to maintain constituent's name and address register, property registers, supplier information, etc. The ability to watch the behavior of any user on the system and determine any behavior that is outside of the expected or accepted operation of the systems. Use of moving pictures, normally in near real-time to communicate between two locations Video management (CRUD) Voice Channel Services (Delivery and receipt of information through a voice channel including touchtone, speech recognition, automatic voice response and text to speech services) Management of voice lines, PBX's, assignments, billing usage, CIT, VRU, etc. Smart Grid Conceptual Architecture Project (SCAP) Appendix 94

165 Service Voice Recording Voic Web Services Description Recording all voice traffic between specific handsets or telephone instruments Storage of voice messages that can be played back by users who have access to specific telephone handset addresses Provide access to pre-formatted sets of messages and routines that use industry standards for the exchange of data between parties Data Management Services Data Management Services are required in all domains of the smart grid. All 7 domains require some level of data management services, and most required more capability than they have today. The increase in the volume of data and the tightening of the timing requirements means that most enterprises that will operate the smart grid will need to evaluate their legacy services for suitability the future. Data services can be broken down into data acquisition, transformation, persistence and archiving. Table 26 lists conceptual automation services that support data: Table 26 Data Management Services Service Ad-Hoc Management Reports Adapter Adaptor Matching Adaptors Analyse Information Analysis And Strategy Application Certification - Ecosystem Application Hosting Archive Management Archive Rules Archiving Backup And Restoration Description This service includes the generation of ad-hoc reports in either electronic or hard copy (e.g. job log reporting or site statistical reporting.) Connecting the scheduler to processing nodes Defines a standard application for connecting to heterogeneous enterprise information systems, such as ERP, mainframe transaction processing and database systems. The architecture defines a set of scalable, secure, and transactional mechanisms that describe the integration of enterprise information systems to an application server and enterprise applications. Provides interfaces to Application for the exchange of data between applications. Analysis tools available for forecasting and analyzing historic data to find patterns etc. Includes OLAP; modeling. Analysis tools available for forecasting and analyzing historic data to find patterns etc. Includes OLAP; modeling. The ability to successfully complete the testing standards for information devices and computer systems with in a community of companies that have decided to exchange data and/or system access and have defined community standards (e.g. Wal-Mart Vendor Standards) The process required to run a commercial package for to support business requirements Manages the archiving of content to external storage. Use of schedule, filters, etc.. In order to target content for archiving. Rules for archiving of information, applications, settings, configurations, and system software for permanent storage. Permanent or very long-term back up of data on media that is designed to withstand the long-term storage Backup and Recovery are key aspects to any information system and consist of both ICT services and supporting processes. Backup is the activity of copying information from a system so that it is preserved in case of equipment failure or other catastrophe. Restore is the activity of restoring a system using the backups of the information and/or systems applications. Smart Grid Conceptual Architecture Project (SCAP) Appendix 95

166 Service Backup Management Business Continuity Pre- Disaster Services Calendar Service Calendaring - Job Management Capacity Planning Cleansing Cloning Cold-Starting Configuration Connectors Content Aggregation Content Guideline Management Content Import Content Load Management Content Management Content Publication Description The ability to create rules sets that drive the process of backup and restoration of data, configurations, applications and systems in an enterprise The operation of the services required to allow for business continuity to be available when a disaster happens A system that provides the ability to do timekeeping that defines the beginning and length and divisions of the year. Also to provide the basis for a scheduling service for activities The ability to manage a set of activities between a set of systems, such that the desired end result is achieved in an automated fashion on a regular schedule The ability to forecast the level of resources that needs to be assigned to a service to support the anticipated load of the service based on the rules for capacity management Data cleansing, also called data scrubbing, is the process of amending or removing data in a database that is incorrect, incomplete, improperly formatted, or duplicated. The process of saving transactions to a mirror system, as the transactions are processed to provide a "hot" recovery capability Whenever a system is being brought on-line for the first time, it must be initialized and synchronized with the integration infrastructure. The system may be designed to be event driven, but in order for it to be able to properly process a change to a data item, it must first be placed in a known initial state. This is true for its internal functions as well as for it to be able to properly perform its designated services within the context enterprise level business processes. In case for some reason the integration platform is unavailable, provisions need to be made for a subset of mission critical functions to be able to function in a minimally acceptable fashion. For example, it is possible to start up a process that is completely independent of the integration platform and have a temporary point-to-point interface. Then, when integration platform is available again, the point-to-point interface can be disabled. This should be considered only for certain mission critical and mission important services that rely on each other. From the integration infrastructure point of view (vs. the application point of view), to the maximum extent reasonable, a common approach is desired for configuring applications and data stores into the enterprise environment. When an application interface is initiated (adaptor or services), it needs to understand the context, environment, and basic configurations. Exposure of functionality (application or databases) that enables an application or database to conduct transactions (CRUD) with the target. The process of pulling content from a variety of sources and presents it to the user in a unified format The ability to create and maintain the rules for loading content and annotation of the content to maintain consistancy for delivery across channels The ability to bring content, both structured and unstructured into a document template Provides the services for loading content and descriptors into the content management system and documenting the content with additional information to support structured queries. The ability to create and maintain the information that is displayed in an interface or into a hard copy publication for use by customers, employees and other consumers of the information, and the ability to provide a consistent set of information across multiple communications channels The ability to create a packaged view on the content that is available to view by consumers in any channel (e.g. a printed catalog, a portal, or a sales flyer) Smart Grid Conceptual Architecture Project (SCAP) Appendix 96

167 Service Content Relationship Management Data Access Data Aggregation Data Analysis Data Consolidation Data Distribution Management Data Entry Data Exploration Data Export Data Filtering Data Flow Orchestration Data Import Data Landscape Modeling Data Life-Cycle Management Data Management (Structured) Data Management (Unstructured) Data Mapping Rules Data Mining Data Movement Data Notification Data Replication Data Replication Management Data Service Discovery Data Service Registration Data Store Integration Description Manage the rules that link rich content (text, video, image, ) to a product template The ability to reach into organized data repositories and retrieve the information that is useful to answer the query that is being asked The ability to move (or link) data from many systems into a meaningful single repository (physical or virtual) to allow data mining The ability to look at each piece of data in context and determine the quality of the data and the value of the data to the enterprise Bringing data from several databases instances together into a single instance using a key structure between the databases The ability to create and maintain rules about what data will be sent to which locations and what the restrictions are on the sending (or receiving) of the data The ability to enter data via an interface (e.g. keyboard, microphone, scanner, etc) into a digital format for storage in an information system The ability to create parameters for data mining. Data mining is the collection of large amounts of data from one or more repositories. Once a useful connection is developed, the resulting rules are moved to production data mining The ability to move data out of a structured data store Filter out certain data from the local applications during selection. The service will use role definitions to identify the receiver user of the data and to filter from the local application on this basis. To define and manage and monitor complex data flows across the system. The ability to bring data into a structured data store Modeling of local and master data management application and system landscape (logical and technical). Maintain and delete services with a private or public registry during the full lifecycle of the service (creation, maturation, aging, retirement). Storage of structured data is normally within a database management system (DBMS), often a Relational Database Management System. The ability to create and maintain document and image stores which are organized in a fashion to allow a user with some familiarity to retrieve the right documents Mapping rules between master data management object model and the local application obejcts models. Input for Inbound and Outbound Mapping. Data Mining is the specific investigation of data in a warehouse looking for new and useful connections that will provide insight into marketing and other activities. The ability to move data from one repository to another repository in an automated and scheduled fashion Signalize an event to a user in the course of the master data management process. The ability to replicate changes in data to various identical data stores either local and/or distributed Creation and maintenance of the rules that allow for orderly replication of data from both un-tethered systems and enterprise systems To discover existing data management services. To register in a private or public registry during the full lifecycle of the service. The ability to transform data at the transaction level for movement of the transactions or records between two or more structured data stores Smart Grid Conceptual Architecture Project (SCAP) Appendix 97

168 Service Data Transformation Data Update Data Validation Data Warehousing Database Management Decision Support Defined Management Reports Delay Notification Digital Media Production EDI EIS (Executive Information System) Electronic Publishing Encrypted Data For Communication Encrypted Data For Storage Description The ability to change the format of the data elements from one system to support the operation on those data elements by another system or for delivery to another system or a user (e.g. changing Euros to dollars, or English to Chinese, or taking a short integer and making it an integer) Services for creating, updating and deleting data, whether the data is retained by an application, persistent data store or some other means. These services are responsible for ensuring updates to this data are performed in accordance master data management services, including data replication performed for performance reasons. The quality of the data and confidence level of the accuracy of the data is indicated. The golden record will have a quality of data measure while an unsanctioned copy would be expected to have a much lower data quality rating. Regarding data currency, there is typically a metadata field that gives an indication of either when the data was updated (time stamp) or how long since the data has been updated (age indicator). Data Validation Services are both those services tht allow data validation within a database management system or with external services for example reference data or external services such as Postal Address Files (PAF) The ability to combine disparate data from disparate systems into a single data store with valid relationships for the purposes of data mining and reporting. Normally done overnight or once a week to give a historical view Secure, structure, maintain, log and provide access to electronic data in support of distributed and or localized processing Maintenance and summarization of data using pre-determined rules to provide information supporting rapid decision making. This service includes the generation of defined reports in either electronic or hard copy (e.g. job log reporting or site statistical reporting.) This service is responsible to notify the consumer through a channel (might be a prefered channel) of delay in regards of one or multiple services he is "registered" to (like a flight delay for instance). The ability to create digital content for display to an audience (e.g. employee training, sales promotion, professional production) the content could be video, audio, textual, etc The ability to use exchanges standards for moving data between systems or enterprises in electronic form. The standards include but are not limited to OFX, UNEDIFACT, etc. Executive Information System (EIS) software component provides you with a powerful, yet simple tool that allows you to view and analyze key factors and performance trends in the areas of sales, purchasing, production, and finance. The executive information system allows you to see both summary and detail level data. You can display summarized data that reflects the overall status of your enterprise. EIS spotlights areas that need attention before situations become critical and costly by highlighting key performance indicators. You can use drill-down capabilities to trace problem areas to any level. The ability to create complex compound documents for distribution either in electronic form or in hard copy - document templates may be setup to be automatically completed with component data on a scheduled basis Encryption of data within the boundaries of a communication method. This would cover all of the transport methods including those that use a persistent data store such as the file storage mechanism found in many store and forward paradigms while still clearly delineating the boundaries between a persistent data store and the transport mechanism. Encryption of data within data repositories. This covers the encryption of data governed by applications using file system and database formats as well as other persistent data stores not associated with data transport mechanisms. Smart Grid Conceptual Architecture Project (SCAP) Appendix 98

169 Service Enterprise Rollback Event Scheduling Service Exception Handling Extract/Transform/Load Extraction Fee Management File Services File Sharing Global Setting Management Governance Services Grid Grid Management Implementation And Controls Set-Up Inbound Data Mapping Indexing & Referencing Information Integration Information Management Description The ability to rollback an update or change to information contained in multiple systems and databases in a coordinated and automated fashions The ability to create and maintain a schedule of events that should be automatically intiated and terminated in the enterprise From an integration infrastructure point of view, it is helpful if exceptions are presented in a consistent manner. Exceptions can be divided into at least two categories: system and functional. Systems exceptions are those generated as the result of program and/or hardware system failure. Functional exceptions are those generated as the result of data, business process, or logic errors. Each category can be further classified as fatal, warning, and information only. This service is also often linked to the enterprise system management and monitoring service for process. Services to extract data from existing data sources, transform the data (into the required input format) and then load the data, in a highly efficient manner, into the destination database. Mechanisms for transferring relevant data from local systems to the master data management environment. Master data instances are retrieved from the local systems. Selection is based either on predefined rules or based on the authorization level of the user. Assessment of rates and fees required for an application Maintain and provide access to files in a file system The ability to allow and restrict users and systems access to specific unstructured documents with in the enterprise based on rules, permissions, and authorization The ability to create and maintain the overall setting for all users of a service or for how the service will behave To manage the life cycle of a data items; Define and track chain of custody, versioning, etc. This capability provides the necessary checks before a message is published, during system integration and also on a continuing basis for B2B situations. Validation rules on message syntax, data integrity, and business logic conformance can be developed and implemented as a service to be executed by the adaptor layer. Any exceptions raised by this service will determine whether the data can be published or not. The ability to create a virtualized server farm, where services run based on rules that determine how many resources are devoted to each service. The ability to create and maintain the rules for the prioritization of use of the resources by the rules Configuration, Set-up of seed data, business rules, business defined criteria that govern a service or application Association of meta data definitions and conversion rules between local application data models and the master data management data model. The Indexing & Referencing (I&R) service maps application specific identifiers to and from global identifiers at each interface. It generates unique identifiers for messages and/or objects for canonical message data and is responsible for storing the transformation mappings persistently. These identifiers establish a unique reference between an application message and its corresponding canonical message data, thus enabling the decoupling of sources of data from consumers of data. The ability to bring information into a single view for data mining or business analysis The ability to manage the information in the enterprise across all platforms and services to maintain the health of the data stores Smart Grid Conceptual Architecture Project (SCAP) Appendix 99

170 Service Job And Process Queue Management Job Restart Knowledge Indexing Knowledge Repository Knowledge Searching Management Reports Management & Monitoring Master Data Management Services Message Creation Message Execution Message Explosion Message Queue Messaging Persistence Metadata Management Modeling Description Job queue management functions - management of the machine assets that are assigned to a specific job or process, including launch and termination time, priority slices on hardware, prioritization of access to data stores, spawning additional copies of processes and trimming the resources that are allocated to the process as higher priority processes enter the queue Restart of an automated processing job after a failure has occurred. Either restart from the beginning of the job, or from an identified restart point. The ability to index the content of unstructured knowledge objects, both within the enterprise and in locations in the enterprise and the extended partner network for future query The ability to consolidate key unstructured documents from many sources, including individual user devices into an enterprise data store, to provide a higher level of availability and consistency to the queries of the authorized users The ability to query one or more knowledge indexes for information that is relevant to the query being performed, normally the service includes the ability to rank the results based on the match to the query This service includes the generation of defined and ad-hoc Management reports. A middleware platform is often linked to an enterprise system management tool, but now done using consistent semantics. Systems and/or functional exceptions that require the attention of appropriate people can be sent electronically. Related information is logged for auditing, diagnosing, processing, and manual interventions. A system of record is an information storage system which is the data source for particular data elements and information sharing sets (e.g., message payload). A service should only be allowed to modify items for which it is the system of record. To achieve the desired benefits of COTS, data replication will be unavoidable (vs. customizing off-the-shelf packages to obtain data from a central database). Therefore, the integration infrastructure must ensure that there is only one system of record for each piece of information at any given time. It is possible to dynamically change the system of record via chain of custody management, but this ability may not be worth the tradeoff of increased integration infrastructure design and implementation complexity. Creation of a machine readable header to wrap information in, so that it can be transmitted between devices on the network Identifies the message types and formats the incoming message. Common Service that passes managing the steps needed to get a message from source to destination. Uses Messaging, Transaction, Network and/or Process Management services. The ability to automatically route a message to more than one internal service or addressee A facility that ensures that a message is delivered to its target, includes message ordering and prioritization. This capability provides for archiving the messages for auditing, workflow, and business intelligence purposes. Typical integration solutions support the internal needs for message persistence for auditing and workflow, but may not have a way to provide business intelligence on top of messaging. Message persistence, in conjunction with the Indexing and Referencing service, are fundamental to establishing event histories and reporting capabilities that transcend individual systems. Mechanisms for the configuration and maintenance of predefined settings for the master data objects, application landscape, and master data object mappings/routing. It has services to model and define procedures and rules. Setup and utilization of complex analytics that will automatically review records, based on a set of rules to determine trends in the records that can be displayed to users Smart Grid Conceptual Architecture Project (SCAP) Appendix 100

171 Service Multi-Media Multi-Media Chat Multi-Tasking Multiprocessing OLAP (On-Line Analytical Processing) OLTP (On-Line Transaction Processing) Operational Data Store ORB (Object Request Broker) Platform Management Portal Management Portal Page Creation Portal Pages Framework Purging Query Management Recovery Replay Resource Adapter Restart Management RFID Roll Back Schedule Searching Directory Description Integrated graphics, audio, video, and text The ability to use text, video, audio and other media formats for on-line discussions and collaboration in near real time The ability to have a single device handle multiple services simultaneously thru a single physical pipeline The ability to break up a stream of work and have it handled across a number of physical devices Decision support software allowing the user to quickly scrutinize information summarized into multidimensional views and hierarchies. OLAP included the use of multidimensional models to represent complex data relationships, using slice and dice to look at information from different viewpoints. The ability to take records from users and services one record at a time, provide validation, route error condition messages back to the creator and store the valid transactions or invalid transactions with error messages in a structured data store The ability to create and maintain information across traditional applications and data stores in a single data store to support use of the data by multiple systems. An ODS is an operational version of a data warehouse and is normally up to date with the movement of data in the enterprise The ability to orchestrate the services request for information from other services or data stores and manage the process of the exchange The ability to monitor and manage the hardware platform the infrastructure is built upon The Management activities of the Portal Application, includes selection of standard alerts, indexing and monitoring of criteria (based on set-up). Also includes capabilities such as caching and pooling. The capability to build Portal Pages based on either Framework Templates or algorithm. These pages will be served to the various channels and may utilize common services. The templates provided within Portal Products to assist in the formulation of consistent look and feel for Portal Content Pages The process of verification of data on archival media and the subsequent removal of data from the primary storage media Handling all requests for information or data from a specific data store - requests can be handled on a priority basis and they can be interleaved with other processes based on assigned priorities Transfer of information from a backup media to a specific system with a specific configuration for operation then loading it with a specific set of data based on an agreed to time and date The reinstating of an event upon failure Prepackaged modules that provide connectivity and or transformation between applications and databases. The ability to determine the state of an ICT process and if it has stopped, return the process to operation from the point it stopped at, or at the beginning of the current step or record Locating and identifying packages by utilization of location device(s) Retraction database changes to a specific previous date and time or to the beginning of a logical transaction Time-based execution of a formalized chain of master data management events. The querying of Portal databases to find data (e.g. users, groups, authorizations) based on the search criteria, from the Directory Smart Grid Conceptual Architecture Project (SCAP) Appendix 101

172 Service Searching Other Databases Semantic Model Management Services Staging State Management System Directory Services System Management Tracking Transaction Controls Transaction Management Transaction Monitoring Transaction Rules Trigger Workflow Version Management Voice Input Processing Workflow Services Description The querying of other databases to find data (e.g. media, information) based on the search criteria, from the Portal Product To ensure semantic consistency across all interfaces independent of technologies employed and vendor products used, these services enable use of artifacts (e.g., schemas for messages, schemas for persistent data stores, etc.), derived from the semantic model across all nodes and all layers. Store & forward master data instances across its entire life-cycle. Determine the status of an master data instance being processed and initiate and determine the workflow based on this status. The ability to provide system to system addressing in the network to allow users and services to connect to authorized services and authenticate Management of the computing platforms including: load monitoring, capacity, software distribution, and configuration control of the desktop systems and servers Service to analyze the audit trail in order to reconstruct past events. Configuration, Set-up of seed data, business rules, business defined criteria that govern the ERP General Ledger Module. Processing of data across multiple databases. Monitoring of the transaction to determine if it is successful or not (passes message back to Message Execution Service). Includes the use of two phased commits, multiple targets and distributed transactions. Rules specific to the target databases e.g. cascade update, two phase commit. A call to a workflow component when user intervention is required for an event (within an event flow). Manages the various version of rich content, check-in, check-out, etc. Conversion of audio information to digital information that can either be recorded as textual data, records, or machine commands The set-up and running of processes to control document flows including escalation, notification, task management, prompts and reminders Smart Grid Conceptual Architecture Project (SCAP) Appendix 102

173 Appendix D. Roadmap & Maturity Model Policy Timeline Figure 47 depicts a typical policy timeline for a California utility. Each utility should employ a similar diagram to identify deadlines set by federal, state and local regulatory and legislative bodies. This timeline provides high level input to the utility Smart Grid development roadmap. Figure 47 - California Smart Grid Policy Timeline (Southern California Edison, 2010) Pursuing the Smart Grid Vision A high-level development roadmap for the smart grid identifies the technologies a utility should plan to pursue over the course of the next two decades in order to make its smart grid vision a reality. The smart grid will evolve in complexity and scale over time as the richness of systems functionality increases and the reach extends to greater numbers of intelligent devices. Most utilities will create and follow development roadmaps, which tend to have several major stages. Within each development roadmap stage, there are two activity portfolios to be managed simultaneously. The smart grid deployment project portfolio consists of smart grid technologies commercially ready for deployment. The technology evaluation portfolio includes initiatives to identify, evaluate, and test emerging technologies expected to be deployed at a later stage. Figure 48 illustrates the distinction between these two portfolios in terms of technology maturity over time. Roadmap & Maturity Model Appendix 103

174 Figure 48 - Smart Grid Project Portfolios as a Function of Maturity (Southern California Edison, 2010) Technology evaluation portfolio projects fall into the Leading Edge areas of the maturity curve. These projects require further evaluation of emerging technologies to better understand the capabilities such technologies could contribute to the smart grid vision, their progress towards technical maturity, and the corresponding value that they might unlock. Smart grid deployment portfolio projects, on the other hand, involve the planning and execution of deployment plans for commercially available smart grid technologies. Although these technologies have crossed the chasm of the maturity curve, given the urgency of state and national policy goals they increasingly fall within the Early Majority or later areas of curve. Historically, most utilities have preferred to adopt technology later in the maturity lifecycle, allowing for greater confidence in implementation and operation. Earlier adoption and adaption indicates that significant project risks could be substantially mitigated through an effective technology evaluation process A four stage roadmap structure is recommended for smart grid transformations. In this model, the four stages of the smart grid development roadmap and high level plans for deployment and technology evaluation projects to be included within each stage are: Stage 1: Foundation (past work) Comprised of already completed foundational work for the deployment of advanced measurement and control systems, this is a preliminary period where many utilities get early experience with new Roadmap & Maturity Model Appendix 104

175 technologies such as wide-area measurement and control systems - smart meter deployments were initiated by a number of utilities during this stage. Stage 2: Inform and Automate (near-term) In this stage most utilities focus on the following areas: Evaluation of energy storage Integration of renewable and distributed energy resources Development and interoperability testing of home area network devices and vehicle charging equipment Ongoing development of interoperability and cyber-security standards Electric system studies and engineering analysis regarding operational impacts from dynamic resources, bi-directional distribution flows and new operating paradigms Workforce safety and productivity technologies A number of North American Smart Grid pilot projects, many funded by the American Recovery and Reinvestment Act (ARRA), will commence during this period. Efforts to educate the general public on Smart Grid topics should also gain traction during this time. Stage 3: Interactive (mid-term) This stage focuses on technologies that leverage prior investments and involves retrofit work. The aim is to begin the process of building Grid 2.0. The anticipated evolution from Grid 1.0 to Grid 2.0 is depicted in Figure 49 for a variety of grid characteristics. Grid 2.0 fully automates the energy delivery system across the entire value chain with increased interactions among smart grid devices, the utility, and customers. It has both hard grid assets (advanced energy storage, super-conducting equipment), and soft grid assets (next-generation computing and analytics systems) to glean the full value of the new grid for utility customers. Several documents, including Grid 2.0: The Next Generation (Willis, 2006), suggest Grid 2.0 will be fully decentralized. By 2030, a highly interactive and hybrid grid will include large central resources and substantial decentralized supply and demand resources. This is analogous to 2011 hybrid information networks linking large centralized data centers, cloud computing, highly distributed personal computing, and smart phones. Roadmap & Maturity Model Appendix 105

176 Figure 49 - Grid 1.0 Evolution to Grid 2.0 This renewed electric system will enable seamless integration of large renewable and distributed generation resources. It will also facilitate the integration of energy storage technologies to support state and federal legislation and policy goals such as greenhouse gas reductions, Renewable Portfolio Standard (RPS) and electric transportation initiatives. Grid 2.0 incorporates the next generation of broadband wireless and field area telecommunications technologies to support high speed, low latency information exchange among highly distributed devices. Smart grid efforts will merge advanced data analytics and intelligent systems into existing grid control systems, resulting in a complex system-ofsystems to provide integrated grid control and ubiquitous, real-time grid-state information. As a result, opportunities will emerge to reliably link customer demand response and other smaller distributed resources into wholesale market operations with the requisite ability to coordinate operational dispatch between wholesale market objectives and distribution grid objectives. Stage 4: The Intuitive and Transactive Grid (long-range) The 2030 decade will see continued deployment of Grid 2.0 capabilities across most utility systems and the introduction of highly distributed intelligent controls involving machine-to-machine transactions. This stage of the smart grid development roadmap assumes full convergence of information and energy systems, as well as continued breakthroughs in computing architectures, cyber-security, internet technologies, autonomous multi-agent control systems, artificial intelligence applied to electric system operations, wireless telecommunications, energy storage, power electronics, energy-smart consumer devices, consumer information technology and sensing technologies. Results should include wider deployments of distributed computing technologies for faster system response times, the integration of many more sensory and control nodes at the distribution and customer levels, and the ability to manage and precisely react to supply and demand imbalances at the micro level or through aggregation at any level or nodal point across the transmission and distribution grids. Roadmap & Maturity Model Appendix 106

177 Maturity Model To move forward with a smart grid transformation, a utility should assess its current state and then define its own roadmap and strategy for achieving the desired functionalities. This section presents an industry-standard methodology to help utilities plan smart grid implementation, prioritize options, and measure progress. The Smart Grid Maturity Model (SGMM) was developed by electric power utilities for use by electric power utilities. It is under the stewardship of the Software Engineering Institute at Carnegie Mellon University. The SGMM is a framework for understanding the current state of smart grid deployment capabilities within an electric utility. It also serves as a context for establishing future strategies and work plans pertinent to smart grid implementations. The model is comprised of eight domains, each containing six levels of maturity. Implementation of a Smart Grid involves major technological transformations to enable seamless communications among grid components and systems to fulfill the required capabilities. Electric utilities executives and senior leaders must understand the potential impacts these technological transformations will have on their existing organizational structure. This is critical because: 1. A typical utility ICT infrastructure is currently somewhere between The Silo Architecture and the Integration using Enterprise Services Buses in the range of System-of-Systems Design patterns (Appendix A), 2. To satisfy long-term Smart Grid capabilities, a utility must move to an architecture allowing any system to interact with any other system, and 3. Fundamental changes to how a utility operates are necessarily disruptive. Utilities can use their SGMM level assessment to start discussions among their functional domains on the potential organizational transformations needed to ensure Smart Grid success. It also provides a context for the new technologies in terms of prosumer energy control, and utility grid efficiency improvements for cost containment. For more information about the Smart Grid Maturity Model, the reader is encouraged to visit the Software Engineering Institute website: Roadmap & Maturity Model Appendix 107

178 NOTES Roadmap & Maturity Model Appendix 108

179 Appendix E. Glossary of Terms The Smart Grid Today glossary (SmartGridToday) is an additional source for the quizzical reader, should any desired acronym or term not be in Table 27. Table 27 - Glossary Term Meaning 1 GigE Gigabit Ethernet see BEC AAA Authentication, Authorization, Accounting - acronym used in computer/information security to describe protocols supporting these attributes (RADIUS, Diameter, TACACS, etc.) AC Alternating Current electric power in which the movement of electric charge periodically reverses direction. See DC. ACL Access Control List - typically a list of permissions attached to an object in a computer file system ADR Automated Demand Response - the ability to manage customer consumption of electricity in response to supply conditions AMI Advanced Metering Infrastructure - electric utility equipment needed to install, use, and manage Smart Meter with real-time sensors, power outage notification, and power quality monitoring BAAM Behavioral Awareness and Monitoring - heuristics used to detect abnormal or threatening human behavior patterns BEC Gigabit Ethernet Channel a communications channel for Gigabit Ethernet (GbE or 1 GigE) technologies for transmitting Ethernet frames at a rate of a gigabit per second BP Basic Profile - see WS-I BP BPEL Business Process Execution Language - an OASIS standard for specifying business process actions via web services. BPEL processes export and import information exclusively through web service interfaces (also known as WS-BPEL) BRE Business Rules Engine - a software system executing one or more business rules in an ICT production environment. CDM Canonical Data Model - a design pattern used to communicate between different data formats CEA Customer Experience Analytics - software used to identify and analyze customer behavior patterns within and across multiple access points. CEP Complex Event Processing - a computing technique used to process action messages from all organizational layers by identifying those most meaningful, analyzing their impact, and taking subsequent action in real time Choreography See Process Choreography CIM Common Information Model - an electric power industry standard officially adopted by the IEC to allow application software information exchange CIP Critical Infrastructure Protection - the preparedness and serious incident response involving the critical infrastructure of a region or nation CLI Command-line interface - provides interaction with software by typing commands to perform specific tasks (instead of WIMP - see GUI) Comms Communications this substitution is widespread in casual conversation and technical writing COMTRADE COMmon format for TRAnsient Data Exchange for power systems - an IEEE file format for oscilloscope data. COTS Commercial Off-The-Shelf - computer systems developed and marketed by commercial software vendors Glossary of Terms Appendix 109

180 Term Meaning DA Distribution Automation a broad set of capabilities in the utility distribution system, including control center-based SCADA, distribution management system, substation automation, primary and secondary network automation, associated smart controls, etc. DC Direct Current electric power in which the electric charge flows in one direction. See AC. DER Distributed Energy Resource a small energy source, typically producing power near where the power is used (very little power transmission or distribution between generator and power consumer) DMS Distribution Management System - a system used by electric utility grid operators to manage distribution system performance DNP3 Distributed Network Protocol - communications protocol used between process automation components (used by SCADA systems) DoE U.S. Department of Energy - a Cabinet-level U.S. government department with executive authority on energy and nuclear material handling policies DR Distributed Resource see DER DSM Demand Side Management - the modification of consumer demand for energy through financial incentives, education, electronic devices, and other means (also known as Energy Demand Management) EDI Electronic Data Interchange - the structured transmission of data between organizations by electronic means. EDM Energy Demand Management - see DSM EIM Enterprise Information Management - optimizes use of information within organizations, for instance to support decision-making or day-to-day operational processes requiring availability of knowledge by managing information on an enterprise level. EIS Executive Information System - a type of information system used to facilitate and support the decisionmaking needs of senior executives EMS Energy Management System - a system of computer-aided tools used by electric utility grid operators to manage generation/transmission system performance ESB Enterprise Service Bus - a software architecture construct with fundamental services for complex architectures via an event-driven and standards-based messaging engine ESM Enterprise Semantic Model - the logical representation of enterprise information assets used to manage or facilitate business processes. FACTS Flexible AC Transmission Systems application of solid-state switches, coupled with capacitors, used to simultaneously regulate transmission voltages, fine-tune reactive power and remove noise from the AC signals. FACTS are important enablers of variable energy generators (wind, solar). FAN Field Area Network a communications network typically with wide geographical coverage and low to medium bandwidth, depending on the needs of specific use cases FIPA Foundation for Intelligent Physical Agents - an IEEE standards organization developing and setting computer software standards for heterogeneous, interacting agents and agent-based systems FIPS Federal Information Processing Standards (FIPS) Publications - a series of documents for the U.S. Federal Publications government issued by NIST Firewall Device or set of devices controlling propagation of network transmissions based upon a rule set. FISMA Federal Information Security Management Act of 2002 FLISR Fault Location, Isolation and Services Restoration GbE Gigabit Ethernet see BEC GDA Generic Data Access - an IEC X GID interface used for model management, access, and update distribution GES Generic Events and Subscriptions - an IEC X GID interface used for pub/sub of generic XML messages Glossary of Terms Appendix 110

181 Term GID GOOSE Grid 2.0 GSE GSSE GUI HAN HEC HSDA HVDC ICT IEC IED IEEE Internet IP IPS IRM IRM IT ITU-T IVR JPA JSON MAC Meaning Generic Interface Definition - a set of common services used for enterprise integration by utilities; part of IEC standard Generic Object Oriented Substation Events - control model mechanism to package any data format into a data set and transmit within 4 millisecond - one of two subdivisions of GSE See Smart Grid Generic Substation Events - a control model defined by IEC providing a fast and reliable mechanism to transfer event data over entire substation networks Generic Substation State Events - an extension of event transfer mechanism exchanging only status data using a status list (string of bits) rather than a GOOSE dataset Graphical User Interface - allows interaction with an electronic device through the use of images (utilizes WIMP instead of CLI) Home Area Network - communications network for customer's home - see PAN HDMI Ethernet Channel technology consolidating video, audio and data streams into a single HDMI cable High Speed Data Access - an IEC X GID interface used for access to real-time measurement data High Voltage Direct Current an electric transmission system using direct current for the bulk transmission of electrical power. Over long distances, HVDC is expected to be less expensive, with lower electrical losses compared to a similar alternating current system. Information and Communications Technology - an extended synonym for ICT stressing the role of unified communications in modern information technology International Electrotechnical Commission - a non-profit, non-governmental international standards organization which publishes International Standards for electrical and electronic technologies Intelligent Electronic Device - a microprocessor-based controller of power system equipment, such as circuit breakers, transformers, and capacitor banks. Institute of Electrical and Electronics Engineers - a non-profit professional association dedicated to advancing technological innovation related to electricity A global system of interconnected computer networks using IPS Internet Protocol - the principal communications protocol used for relaying packets across a network using the Internet Protocol Suite Internet Protocol Suite - the set of communications protocols used for the Internet and similar networks. Also known as TCP/IP, from two of the most important IPS protocols: the Transmission Control Protocol (TCP) and the Internet Protocol (IP) Interface Reference Model (IEC ) provides the framework for identifying information exchange requirements among utility business functions Institute of Risk Management - a leading international professional education and training body for risk management Information Technology - any microelectronics-based collection of computing and telecommunications capabilities designed to acquire, process and disseminate textual, numerical, and non-structured information. See ICT. Telecommunication Standardization Sector, part of the International Telecommunication Union (ITU) based in Geneva, Switzerland. Interactive Voice Response technology allowing computer-human interaction through the use of voice and keypad inputs. Java Persistence API - a Java programming language framework managing relational data in applications JavaScript Object Notation a lightweight data-interchange format Media Access Control Layer Glossary of Terms Appendix 111

182 Term Mashup ML MoM MOM MultiSpeak NERC NERC-CIP NETL NIST NISTIR 7628 NRECA OASIS OFX OLAP OLTP OMG OMS OPC Orchestration OSGi OSI model P&C PAN PCC PHY PKI PLC PMI Meaning A web application that combines data and/or functionality from more than one source, sometimes called a web application hybrid Management Layer one of four smart grid management types (Element, System, Services, Smart Grid) Manager of Managers a smart grid construct to support distributed grid management services coordinated by a single component, called the Manager of Managers Message-Oriented Middleware infrastructure used to send and receive messages between distributed systems. A specification defining standardized interfaces between software applications commonly used by electric utilities North American Electric Reliability Corporation - a nonprofit corporation promoting the reliability and adequacy of North American bulk power transmission See CIP National Energy Technology Laboratory - a science, technology, and energy laboratory operated by DoE National Institute of Standards and Technology - a measurement standards laboratory operated as a nonregulatory agency of the United States Department of Commerce Guidelines for Smart Grid Cyber Security issued in August 2010 as an Interagency report (NISTIR) by the United States Department of Commerce, National Institute of Standards and Technology (NIST) National Rural Electric Cooperative Association - an organization representing over 900 electric cooperatives in the United States Organization for the Advancement of Structured Information Standards - a global consortium supporting the adoption of e-business and web service standards Open Financial Exchange - a data-stream format for exchanging financial information On-Line Analytical Processing - a business intelligence approach used to swiftly answer multi-dimensional analytical queries On-Line Transaction Processing - a class of systems used to facilitate and manage transaction-oriented applications Object Management Group - a consortium focused on modeling programs, systems and business processes Outage Management System - a computer system used to restore power by operators of electric distribution systems OLE for Process Control - an open standard used by GID See Process Orchestration Open Services Gateway initiative - a Java service platform implementing a complete and dynamic component model, characterized by "bundles" of functionality Open Systems Interconnection model - a sub-division of network communications into seven smaller parts - each layer is a collection of similar functions providing services to the layer above it and receives services from the layer below it Production and Control Premises Area Network - communications network for grid customer, more generic than HAN Point of Common Coupling see PoCC Physical Layer - the first and lowest layer in the seven-layer OSI model of computer networking. Public Key Infrastructure - the set of assets needed to create, use and manage digital certificates Programmable Logic Controller a small industrial computer used to replace relay logic. A PLC is similar to RTU, but is typically easier to program. RTU and PLC technology are slowly merging. See RTU for detail. Privilege Management Infrastructure - supports a strong authorization system via the management and use of privileges Glossary of Terms Appendix 112

183 Term PMU PoCC PQ PQDIF Process Choreography Process Orchestration PubsFIPS QoS RBE RDF RMS RPC RPS RTU SAML SCA SCADA SCAP SDO SDO SED SEP-2 Service Choreography Service Orchestration SGAC Meaning Phasor Measurement Unit - a device designed to measure the electrical waves on an electricity grid to determine the health of the system Point of Common Coupling the place to which a collection of DER (DR) connects to the Power Quality - the set of electrical property limits allowing electrical systems to function in their intended manner without significant loss of performance or life. Power Quality Data Interchange Format - a binary file format (specified in IEEE Std ) used to exchange voltage, current, power, and energy measurements between software applications A form of service composition in which interactions between partner services are defined globally. At runtime each participant executes according to other participant behaviors with no central control The process of coordinating an exchange of information through web service interaction. Orchestration typically is guided at runtime by a central control mechanism. See FIPS Publications Quality of Service - a computing mechanism providing different execution priority rankings to different computing objects (e.g. users, destinations, applications, data types, data flows). It also can be used to guarantee a certain level of performance to a data flow Report By Exception information is transmitted only when a pre-defined threshold or condition is reached. RBE reduces communication traffic and subsequent data handling. Resource Description Framework - a general-purpose language for representing information in the Web. Root Mean Square - a measure of the magnitude of a varying signal Remote Procedure Call - an inter-process communication allowing software to cause the execution of a procedure in another address space without coding the remote interaction Renewable Portfolio Standard a regulatory mandate to increase energy production from renewable energy sources (wind, solar, biomass, geothermal). Remote Terminal Unit or Remote Telemetry Unit an electronic device interfacing field devices to a distributed control system (i.e. SCADA). Similar to PLC, but typically preferred where communications are difficult. RTU and PLC technology are slowly merging. See PCL for more detail. Security Assertion Markup Language - an XML-based open standard for exchanging authentication and authorization data between security domains Service Component Architecture - a programming model for building applications and solutions based on a Service Oriented Architecture Supervisory Control And Data Acquisition - industrial control systems, computerized to monitor and control commercial and infrastructure processes Smart Grid Conceptual Architecture Project - the Smart Grid conceptual reference model for which the SGAC is responsible Service Data Objects - a technology allowing heterogeneous data to be accessed in a uniform way. Standard Developing Organization - an organization with the scope of establishing national, regional or international engineering standards Smart Electronic Device used interchangeably with IED (Intelligent Electronic Device) Smart Energy Profile an open standard for implementing secure, easy-to-use wireless home area networks to manage energy consumption See Process Choreography See Process Orchestration Smart Grid Architecture Committee - responsible to the SGIP for creating and refining a Smart Grid conceptual reference model, including lists of the standards and profiles necessary to implement the vision of the Smart Grid Glossary of Terms Appendix 113

184 Term SGIP SGMM SIP SLA Smart Grid SOA SOAP SoS SQL SSL SSO SSO STATCOM THD TLS TSDA UDDI UN/EDIFACT VAR VOIP Volt/VAR VPN VPP W3C WAN Web Web 2.0 Web Service Meaning Smart Grid Interoperability Panel - a stakeholder organization administered under a NIST contract to identify, prioritize and address new and emerging requirements for Smart Grid standards Smart Grid Maturity Model a management tool used to help plan a smart grid implementation, prioritize options, and measure progress (maintained by the Software Engineering Institute at Carnegie Mellon University) Session Initiation Protocol - an IETF-defined signaling protocol used to control multimedia communication sessions (VOIP, etc.) Service Level Agreement a service contract formally defining expectations, usually based on response times, between a customer and a service provider. A utility power distribution grid enabled with computer technology and two-way digital communications networking. Service-oriented architecture - a set of design principles aimed toward packaging functionality as a suite of interoperable services, residing in multiple systems, and usable by many business domains Simple Object Access Protocol - a protocol specification for exchanging structured information via web services System of Systems - a collection of task-oriented or dedicated systems with pooled resources and capabilities to create a more complex 'meta-system' offering more functionality and performance than the sum of the constituent systems Structured Query Language - a database computer language designed for managing data in relational database management systems (RDBMS), and originally based upon relational algebra and calculus Secure Sockets Layer - a network protocol succeeded by Transport Layer Security (TLS) Single Sign On - a property of access control of multiple related, but independent software systems allowing a user to log in once and gain access to all systems without additional log in prompts Standards Setting Organization - an organization promulgating or maintaining standards Static Synchronous Compensator a regulating device on AC electricity transmission networks, acting as a source or sink of reactive AC power on the electrical network. See FACTS. Total Harmonic Distortion - an inverse measurement of the fidelity of a signal to its source Transport Layer Security - a network protocol and successor to Secure Sockets Layer (SSL) Time Series Data Access - an IEC X GID interface used for access to historical measurement data Universal Description, Discovery and Integration - a platform-independent, XML-based registry for businesses to be listed on the Internet, including a mechanism to register and locate web service applications United Nations/Electronic Data Interchange For Administration, Commerce and Transport - an international EDI standard Volt-Ampere Reactive - a unit used to measure reactive power in an AC electric power system Voice Over Internet Protocol - technology used for the delivery of voice communications and multimedia sessions over an Internet Protocol network Used to express the need for careful management of the interaction between voltage and VAR control. Virtual Private Network Virtual Power Plant a collection of distributed generation managed by a central entity (aggregator) World Wide Web Consortium - an international standards organization for the World Wide Web Wide Area Network See WWW A loosely defined set of web applications characterized by collaboration, user-defined design, participatory information sharing, and standards-based interoperability Software designed to support interoperable machine-to-machine interaction over a network Glossary of Terms Appendix 114

185 Term WIMP WS-BPEL WSDL WS-I BP WWW XML XSD Meaning Window, Icon, Menu, Pointing device - see GUI Web Services Business Process Execution Language - see BPEL Web Services Description Language - an XML-based construct used to characterize web services T Web Services Interoperability Basic Profile - a specification providing interoperability guidance for core Web Services specifications such as SOAP, WSDL, and UDDI. World Wide Web - a system of interlinked hypertext documents accessed via the Internet extensible Markup Language - a set of rules for encoding documents in machine-readable form XML Schema Definition - a set of rules to which a valid XML document must conform Glossary of Terms Appendix 115

186 NOTES Glossary of Terms Appendix 116

187 Appendix F. Bibliography ARRA. (n.d.). American Recovery and Reinvestment Act of Retrieved March 2011, from Recovery.gov - Track the Money: GWAC. (2008, March). Publications. Retrieved March 2011, from GridWise Architecture Council: IEC. (2003). IEC Application integration at electric utilities - System Interfaces for distribution management - Part 1: Interface architecture and general requirements (Preview). Retrieved 2011, from International Electrotechnical Commission Webstore: IEC. (2005) Energy management system application program interface (EMS-API) - Part 1: Guidelines and general requirements (Preview). Retrieved March 2011, from International Electrotechnical Commission Webstore: IEEE. (n.d.) Smart Grid Interoperability Series of Standards. Retrieved March 2011, from IEEE Standards Association: NAE. (2011). Electrification. Retrieved March 2011, from Greatest Engineering Achievements of the 20th Century: NETL. (2007, January). Smart Grid Implementation Strategy (SGIS) - Reference Shelf. Retrieved March 2011, from NETL - the Energy lab - Where energy challenges converge and energy solutions emerge: NIST. (2010, January). NIST Framework and Roadmap for Smart Grid Interoperability Standards, Release 1.0. Retrieved March 2011, from National Institute of Standards and Technology: NIST. (2010, January). Smart Grid. Retrieved March 2011, from National Institute of Standards and Technology: NIST-ITL. (2010, September). NIST - Computer Security Division - Computer Security Resource Center. Retrieved March 2011, from National Institute of Standards and Technology: SGIP SGAC. (n.d.). SGAC Conceptual Architecture Working Party. Retrieved March 2011, from SGiP - NIST Smart Grid Collaboration Site: SGMM. (n.d.). Smart Grid Maturity Model. Retrieved March 2011, from Software Engineering Institute Carnegie Mellon: Bibliography Appendix 117

188 SmartGridToday. (n.d.). Glossary of Terms and Abbreviations. Retrieved March 2011, from Smart Grid Today The Worldwide Daily Journal of the Modern Utility Industry: Southern California Edison. (2010). Smart Grid Strategy and Roadmap. Retrieved March 2011, from Southern California Edison: %20Smart%20Grid/100712_SCE_SmartGridStrategyandRoadmap.pdf van Breeman, A. J. (2001). Agent-Based Multi-Controller Systems. Retrieved March 2011, from University of Twente: White, A., Comport, J., & Schlier, F. W. (2005, January 17). Enterprise Information Management Is a Core Element of Your IT Architecture. Willis, R. (2006). Grid 2.0: The next generation. Retrieved March 2011, from Rebecca Willis: Bibliography Appendix 118