IP/MPLS Marios Parperis - Alcatel-Lucent Energy Systems Integration Division October 2012
Agenda Today s Landscape in Electric Utilities Why Consider IP/MPLS as a Network solution? MPLS Concerns TDM Applications Timing and Synchronization Teleprotection Bandwidth Management Security Network Management Migration path Moving from SONET to IP/MPLS in a phase approach Cost analysis MPLS Vs NG Optical Packet June 2010
Intelligent Layers of the Electric Utility Network Asset Management Energy Management Integrated Volt/VAR Meter Data Management Demand Side Control Outage Management Management Monitoring (Transformer, Workforce Management Circuit Breaker, Distributed Generation SCADA Load Management Intermittent generation) Smart charging (for EVs) Application Plane Communication Plane Power Delivery Plane
Electric Utility Applications
Today s Communication Network Requirements More bandwidth Minimal delay and jitter Legacy technologies + new applications Flexible capacity and speed High resiliency Security
IP-centric communications From: Separated service networks To: Converged service network Optimization Simplification All services in one network Each service has its own network A mix of networking technologies Network transformation to provide the required communications foundation for emerging smart grids June 2010
Why consider IP/MPLS for Network Solution Current TDM Network difficult to scale? Current Network or segments are out of capacity? Troubleshooting is time consuming? Multiple Networks Multiple Teams? Expensive leased circuits? Consider IP/MPLS as the SOLUTION
Reliable Communications with MPLS Standards-based IETF, ITU-T, IP/MPLS Forum A Multi-vendor interoperability Deterministic behavior Connection Oriented approach Similar to ATM/TDM/SONET Resiliency 50ms Fast Re-Route (SONET Like) Active/Standby path provisioning for high availability Protocol Independent Any Layer 2 or Layer 3 protocol runs over it Traffic Engineering More efficient use of network resources Provides service guarantees and resource reservation Supports multiple VPN schemes Emulation of private Layer 2 and Layer 3 networks Inherent Operation & Maintenance capabilities (OAM) Easier to troubleshoot LSP TDM Ethernet IP ATM etc Z IP/MPLS network Pseudowire MPLS Tunnel IP/MPLS network LSP B MPLS Router 8 IP/MPLS Utility Networking
MPLS-based VPN services Virtual Leased Line (VLL) Point-to-point pseudowire connections PE B Virtual Private LAN Service (VPLS) ATM service PE C Layer 2 bridged multipoint Ethernet service Ethernet service PE A IP/MPLS network TDM service VPLS service PE B B B VPLS service PE D PE C Layer 3 IP-VPN RFC 4364 routed multipoint R R PE B VPRN PE C R Service 1 R B B PE A B PE D IP/MPLS network B B Virtual bridge PE A VPRN R Virtual router Service 2 IP/MPLS network PE D R 9 IP/MPLS Utility Networking
What are some of the concerns on IP/MPLS? TDM applications Can MPLS handle legacy applications? How about voice (traditional and VoIP)? How about timing and synchronization? Teleprotection Can MPLS handle the strict latency requirements of Teleprotection? Can it deliver sub-8ms circuits? Security Is MPLS secure? How can it improve security? Management Is MPLS too complex to manage? Migration How can I safely migrate from SONET to MPLS? Cost Is MPLS too expensive? What is the alternative? June 2010
Bell Labs: Delay and Priority Requirements for Application Traffic Application Delay Allowance (minimum) ms Priority 0-max 100-min Delay <= 10 ms (High speed) Protection Information* 8, 10 2 Load shedding for underfrequency 10 20 10 ms < Delay <= 20 ms Breaker reclosures* 16 15 Lockout functions* 16 12 Many Transformer Protection and control 16 12 applications* System Protection (PMU)* 20 12 20 ms < Delay <= 100 ms SynchroPhasor Measurements (Class A) 60 10 SCADA data poll response 100 25 PTT signaling (critical) 100 30 PMU clock synchronization 100 20 100 ms < Delay <= 250 ms VoIP bearer (inc. PTT) 175 50 VoIP signaling (inc. PTT normal) 200 60 Dynamic Line Rating (DLR) 200 40 Real-time video (mobile WF) 200 55 On demand CCTV video 200 55 Other SCADA operation 200 45 Enterprise data - preferred 250 70 Most distribution and SCADA apps 250 65 AMI - critical 250 60 Application 250 ms < Delay < 1 s Delay Allowance (minimum) ms Priority 0-max 100-min AMI - priority 300 70 CCTV stream - normal 400 75 PMU (class C) 500 80 Some Transformer Protection and Control 500 80 Applications Enterprise data - other 500 80 1 s <= Delay Image files 1000 90 Fault recorders 1000 90 (Medium speed) monitoring and control 1000 90 information (Low speed) O and M information 1000 90 Fault isolation and Service restoration 1000 90 Distribution applications 1000 90 AMI periodic measurements 1000 85 Text strings 1000 90 Audio and video data streams 1000 78 Fault Recorders 1000 90 Best effort, Default 2000 100 * Traffic for these applications is only between two substations connected with transmission line. This traffic must be designed to be only single hop. Thus the corresponding delay requirements must be considered only single hop. All other delay requirements may have to be satisfied over multiple network hops.
TDM over MPLS The two principal services are used for structured and unstructured connections CESoP Circuit Emulation Service over Packet Provides fractional services (nxds0) SAToP Structure Agnostic TDM over Packet Provides unstructured T1/E1 services Two services are collectively referred to as Circuit Emulation Services (CES) Services are transported over an MPLS Network using Pseudowire point-to-point tunnels TDM MPLS Tunnel TDM CES IWF CES IWF The CES Interworking Function (IWF) applies the proper encapsulation to the nxds0 or T1/E1 traffic Pseudowires (PWE3) identify the specific CES connection MPLS Tunnels transport traffic from point A to B Flexible configuration of buffers within the CES IWF allows control of packetization, latency and jitter which meets the requirements for TDM services. 12 IP/MPLS Utility Networking
Network timing and synchronization Synchronization is as basic to network operation as power and grounding An all-packet infrastructure requires special consideration and capabilities External Synchronization Line Synchronization L2 or L3 PSN SDH Synchronization Options Adaptive Clock Recovery External Synchronization Line Synchronization Synchronous Ethernet IEEE 1588v2 Timing over Packet (Adaptive Clock Recovery, IEEE 1588v2) Client L2 or L3 PSN Primary Reference Clock Synchronous Ethernet Synchronous Ethernet MPLS service router PSN: Packet Switching Network Different options are available depending on network requirements 13 IP/MPLS Utility Networking
Teleprotection Enabling Teleprotection Relay Direct Attachment G.703, IEEE C37.94 Avoiding Converters Better Management Less Complexity Also support for legacy (X.21/E&M) TPR Connectivity Options IP/MPLS network NMS TPR a E&M Substation TPR b X.21 Microwave TPR c G.703 TPR d IEEE C37.94 RTU TPR IED Analog phone June 2010
Lab tests characterized successful IP/MPLS network Teleprotection service support with predetermined delays Customers today are running Teleprotection over IP/MPLS Successful proof of concept testing Deterministic delays on an IP/MPLS network Complete isolation even with severe congestion Relay X.21 Areva DIP 5000 MPLS Router Ethernet traffic generator Testbed Ethernet traffic generator MPLS MPLS ATM 100BaseT Router 100BaseT Router Mux Relay 4xE1 10BaseT 100BaseT E1 X.21 A 7705SAR B C D 7705SAR E F 7710 SRc12 7710 SRc12 MainStreet 3600 Δt = 2ms Δt = IP/MPLS network delay Δt = One way Delay readings on DIP-5000 includes 4.5ms processing delay Network Transit Delays Areva DIP 5000 Test summary available at: <Teleprotection Test Report> Scenario Description Measured Delay in ms 1 X.21 to X.21 on same MainStreet 3600 node 2.00 2 X.21 to X.21 on same 7705 SAR node 1.75 3 Two 7705 SAR hops with 4 T1 to X.21 on 2.50 MainStreet 3600 4 Two 7705 SAR hops with 8 T1 to X.21 on 2.75 MainStreet 3600 5 Two 7705 SAR hops with 100BaseT to X.21 on 2.50 MainStreet 3600 6 Four MPLS hops T1 and Eth (7705 SAR and 5.00 7750 SR-c12) to X.21 on MainStreet 7 Four MPLS hops T1 and Eth (7705 SAR and 7750 SR-c12) to X.21 on MainStreet 3.00 15 Communications Transformation October 2010
Bandwidth Management: Traditional SONET network architecture OC-3 SONET/SDH STM-1 ADM OC-3 n x T1/E1 n x T1/E1 n x T1/E1 STM-1 TDM TDM TDM multiplexer multiplexer multiplexer RTUs V.24 RTUs RS-232 TPR Relays V.24 SCADA Application-server Internet Telephones V.35 Telephones V.35 Telephones V.35 farms Computers Computers Computers Operations PBX NOC
Better Managing Existing Bandwidth Traditional QoS Voice Reserve 2 Mb/s Video SCADA Normal Data Reserve 4 Mb/s Reserve 4 Mb/s Reserve 8 Mb/s What happens when traffic is not consuming all the reserved bandwidth?
How Does MPLS Enable Security? Traffic Isolation: MPLS Virtual Private Networks Availability: MPLS Fast Re-Route and Traffic Engineering
How Does MPLS Enable Security? Physical Security Perimeter: Enables efficient Video Surveillance Electronic Security Perimeter: Enables encryption and Firewalls
Managing the Network: Service Aware Management Integrated Solution for Managing Elements, Networks and Services in a Single Platform Service Management + Network Management + Element Management Service Management Service Configuration Service Profiles and Policies Service Associated Alarms Service Topology Service Performance Monitoring Network Management Physical Topology Representation Network Representation (Tunnels & LSP) Network Configuration & Validation Routing Protocol Configuration Element Management Configuration & Inventory Management Alarms Collection Performance Counter Collection The operator manages the services with the network infrastructure information June 2010
Migration Framework OVER SONET PHASE ONE ADD IP/MPLS ROUTERS FOR IP AND ETHERNET This allows the introduction of new IP services and Ethernet connectivity while continuing to support TDM services on the SONET infrastructure, for cost savings and reduced disruption TRANSFORMATION FRAMEWORK 2 FROM SONET CONNECTIVITY PHASE TWO SWITCH TDM SERVICES ONTO IP/MPLS IP/MPLS routers support traditional TDM services including Synchronous Transport Mode 4/Optical Carrier 12 (STM-4/OC-12), STM-1/OC-3 (STM-1/OC-3), T1/E1, RS-232, V.35, X.21, and Ear and Mouth (E&M) circuits allowing the migration of these services away from the SONET infrastructure onto the new IP/MPLS service platform. 3 CONSOLIDATE ACCESS ONTO IP/MPLS PHASE THREE REMOVE SDH/SONET OR REPLACE WITH WDM Transform to one IP network that is converged, scalable, intelligent, multiservice, eco-sustainable, and cost effective 21 COPYRIGHT 2011 Alcatel-Lucent ALCATEL-LUCENT. 2010 ALL All RIGHTS Rights RESERVED. Reserved
MIGRATION PHASE 1 ADDING IP/MPLS ROUTERS FOR IP AND ETHERNET IP Services Ethernet Services POS TDM Multiplexer Low speed interfaces (RS-232, E&M,.) E1/T1 IP Services Ethernet Services Low speed interfaces (RS-232, E&M,.) TDM Multiplexer POS E1/T1 SDH/SONET SDH/SONET SDH/SONET SDH/SONET POS E1/T1 TDM Multiplexer IP Services Ethernet Services Low speed interfaces (RS-232, E&M,.) POS E1/T1 TDM Multiplexer IP Services Ethernet Services Low speed interfaces (RS-232, E&M,.)
MIGRATION PHASE 2 CONSOLIDATING TDM SERVICES INTO IP/MPLS ROUTERS IP Services Ethernet Services POS E1/T1 TDM Multiplexer Low speed interfaces (RS-232, E&M,.) IP Services Ethernet Services Low speed interfaces (RS-232, E&M,.) POS E1/T1 TDM Multiplexer SDH/SONET SDH/SONET SDH/SONET SDH/SONET POS E1/T1 TDM Multiplexer IP Services Ethernet Services Low speed interfaces (RS-232, E&M,.) POS E1/T1 TDM Multiplexer IP Services Ethernet Services Low speed interfaces (RS-232, E&M,.)
MIGRATION PHASE 3 CONSOLIDATING TDM ACCESS, REMOVE SDH/SONET IP Services Ethernet Services Low speed interfaces (RS-232, E&M,.) IP Services Ethernet Services Ethernet Ethernet Ethernet Ethernet IP Services Ethernet Services Low speed interfaces (RS-232, E&M,.) Low speed interfaces Low speed interfaces (RS-232, E&M,.) IP Services Ethernet Services (RS-232, E&M,.)
MIGRATION PHASE 3 ALTERNATIVE CONSOLIDATE TDM ACCESS, REPLACE SONET WITH WDM IP Services Ethernet Services Ethernet Low speed interfaces (RS-232, E&M,.) IP Services Ethernet Services Low speed interfaces (RS-232, E&M,.) Ethernet WDM WDM WDM WDM Ethernet IP Services Ethernet Services Low speed interfaces (RS-232, E&M,.) Ethernet IP Services Ethernet Services Low speed interfaces (RS-232, E&M,.)
Financial Comparison $$ SONET to Packet: IP/MPLS Vs NG-Packet Optical $$ Bell Laboratories business modeling study: Financial comparison model from SONET to MPLS Vs NG-Packet Optical solution. Focused on N. American utility operations and represents a 10 year period. Both the core and access utility communications networks. Requirements AMI: 64Kbps DA/SCADA: 64Kbps -128kbps Substation monitoring requires 384 Kbps 512 k of bandwidth per substation Teleprotection (highest priority, low latency): 8 kb/s Legacy traffic requires 192 Kbps of bandwidth per substation Security surveillance video: 512 k 2 Mb/s (4-5 cameras per substation) Asset monitoring: 128 kb/s (e.g., key swipes, biometrics) Land Mobile Radio (LMR can be multicast, jitter sensitive): 128 kb/s Manned substations: 384 kb/s 1 Gb/s Phone, WMS (workforce management and dispatch systems), AVL, GIS June 2010
Present mode of operation 1. SONET -Present Method of Operation (PMO) Core rings are OC-12, Aggregation Rings are OC3. Access is DS3 and T1 muxed spurs with Enterprise class routers Multiple network management systems for transport and router nodes June 2010
Future mode of operation NG Packet Optical Future Method of Operation (NG-FMO) Core and Aggregation rings are Packet Optical transport with additional IP Routers. Access Rings are Packet Optical Aggregation plus a router for IP-layer services Two Network management systems modeled IP/MPLS Future Method of Operation (M-FMO) Core and Aggregation Rings are s at each transport location NOC. Access Rings are IP/MPLS routers at each substation. Network Management is integrated June 2010
June 2010
June 2010
Conclusions IP/MPLS is a mature technology for mission critical applications in the Electric Utility Industry IP/MPLS is a Standard Technology offering a deterministic behavior and high resilience Ready for critical, legacy, synchronous applications with very low latency and jitter tolerance Allows deployment of new Applications such as video surveillance, VoIP, Smart Metering and e-scada Migration paths can be defined to the comfort level of the Electric Utility IP/MPLS is less expensive than the alternatives in the long term Cairo june 2010 Copyright 2010 Alcatel-Lucent. All rights reserved
Thank You June 32 Presentation 2010 Title January 2008