Design and Operational Considerations

Transcription

1 Design and Operational Considerations for Electrical Grid Protection Systems Built on Packet-Based Communication Networks Ken Rabedeau, CTO Energy Systems Integration Division Sept 8, 2011 UTC Region 9, Newport, OR

2 AGENDA 1. Current TDM Environment 2. IP/MPLS Environment 3. Design Considerations a) Components of Latency b) Fine Tuning Latency in MPLS Routers c) Latency Optimization Link/Path Design d) Timing e) MPLS Service Catalog for Teleprotection 4. Operational Considerations: Next Generation Network Management

3 Current TDM Environment A Collection of Hard Mapped Circuits TPR SONET OC3 SONET Cross- Connect Protected SONET Cross- Connect SONET OC12 Hard external patch EMS System 1 SONET OC48 SONET OC3 TPR EMS System 2

4 Current TDM Environment A Collection of Manually Hard Mapped Circuits TPR SONET OC3 EMS System 1 SONET Cross- Connect Protected SONET Cross- Connect Temporary re-route SONET OC48 SONET OC12 X SONET OC3 Hard external patch TPR Node or Fibre Maintenance Device Config to Select Path EMS System 2

5 Current TDM Environment Recap Design-wise: Heavy Engineering effort up front, circuit by circuit Point-to-point, fixed path, 1 failover path maximum Dedicated Network Resources Stranded Capacity Consistent performance Operationally: Blind: no end-to-end, pro-active monitoring capabilities Binary: works or doesn t Difficult and time consuming to troubleshoot Complex and onerous design documentation Multiple Independent systems to monitor

6 AGENDA 1. Current TDM Environment 2. IP/MPLS Environment 3. Design Considerations a) Components of Latency b) Fine Tuning Latency in MPLS Routers c) Latency Optimization Link/Path Design d) Timing e) MPLS Service Catalog for Teleprotection 4. Operational Considerations: Next Generation Network Management

7 Why MPLS? No Compromises! 1. No single point of failure 2. Applications Security VPN s 3. QoS Engineered Prioritization of data streams 4. End-to-End Management 5. Interface breadth (Ethernet, RS-232, x.21, T1, fractional T1, DS0, E&M, G.703, C37.94 and more) 6. Flexible network topologies (Ring, Mesh, Hub and Spoke & Hybrids)

8 MPLS Enables The Future IP-Centric Communications Separated service networks Converged service network Optimization Simplification All services in one network Each service has its own network A mix of networking technologies Network transformation to provide the required communications foundation for the emerging smart grid

9 Traditional Drivers & Challenges Transporting TDM over Packet Networks Network Operator Drivers Achieve lower cost base transport per T1 Avoid proportional scaling of costs with number of T1s needed Convergence over single packet network for all services Future-proof Lower OPEX with fewer networks to manage Network Operator Challenges Operational transition from a Layer 1 network to a IP/MPLS network New packet-orientated equipment/design concepts Network synchronization Statistical nature/qos OAM differences between Layer 1 (SONET/PDH) network to IP/MPLS networks No synchronous interface to transport timing Use of new evolving timing over packet technologies Network Engineering to ensure services meet delay and jitter requirements

10 IP/MPLS Environment Substation kv Substation E&M RS-232 Ethernet G.703 C37.94 TPR 7705 SAR 7710 SR Δt IP/MPLS 7750 SR TPR 7705 SAR E&M RS-232 Ethernet G.703 C37.94 Design-wise: Design once methodology (Service Catalogs) Multiple failover backup paths (FRR) Priority Access to Shared Resources and Engineered performance (H-QoS / QoS / RSVP-TE) Efficient Capacity Utilization Operationally: Capable of end-to-end, pro-active monitoring (SLA) Historical and real-time path information is a necessity for management and troubleshooting Path asymmetry is a risk Ability to bridge multiple systems into homogeneous network

11 AGENDA 1. Current TDM Environment 2. IP/MPLS Environment 3. Design Considerations a) Components of Latency b) Fine Tuning Latency in MPLS Routers c) Latency Optimization Link/Path Design d) Timing e) MPLS Service Catalog for Teleprotection 4. Operational Considerations: Next Generation Network Management

12 Components of Latency Where is the Biggest Culprit? TPR SONET OC3 SONET OC12 SONET OC48 SONET OC3 TPR??ms <10ms? Network Latency 16ms??ms

13 AGENDA 1. Current TDM Environment 2. IP/MPLS Environment 3. Design Considerations a) Components of Latency b) Fine Tuning Latency in MPLS Routers c) Latency Optimization Link/Path Design d) Timing e) MPLS Service Catalog for Teleprotection 4. Operational Considerations: Next Generation Network Management

14 Fine Tuning Latency in IP/MPLS Routers TDM Packets moving in this direction DS1 / E1 Access Circuit DS1/E1 LIU Data Si g Packetization GigE Packet Switched Network (PSN) GigE Jitter Buffer Data Si g DS1/E1 LIU DS1 / E1 Access Circuit Packetization As TDM traffic from the Access Circuit (AC) is received, it is packetized and transmitted into the PSN Two modes of operation: CESoPSN (RFC5086) for structured nxds0/64k channels SAToP (RFC4553) for unstructured T1/E1 Network Fixed delay (physical limits) Packet transfer delay based on link speeds and distances from end to end Variable delay (design) the number of and type of switches queuing point in the switches QoS is key to ensure effective service delivery Playout TDM PW packets are received from the PSN and stored into its associated configurable jitter buffer Play-out of the TDM data back into the AC when it s at least 50% full Synchronization and timing is reconstituted

15 Example End-to-End Latency Calculation for a TDM PW TDM Packets moving in this direction DS1 Access Circuit DS1 LIU Data Si g Packet Switched Packetization GigE GigE Jitter Data Network Buffer (PSN) Si g DS1 LIU DS1 Access Circuit Packetization PD Network ND Playout JBD The total end-to-end latency is calculated by summing the packetization delay (PD), network delay (ND) and jitter buffer delay (JBD) as shown here: Total Latency = PD + ND + JBD e.g. PD of 2 ms (16 T1 frames/packet), ND of 3 ms, JBD of 4 ms (JB size 8 ms) Total Latency = = 9 ms ** THIS IS ONLY AN EXAMPLE varies by hardware vendor 15 MPLS Network for Teleprotection Sep 2009

16 TDM Packetization over IP/MPLS Latency Characteristics The two principal services are used for structured and unstructured connections CESoP Circuit Emulation Service over Packet Provides fractional services (nxds0) SAToP Structure Agnostic TDM over Packet Provides unstructured T1/E1 services Two services are collectively referred to as Circuit Emulation Services (CES) Services are transported over an MPLS Network using Pseudowire point-to-point tunnels TDM MPLS Tunnel TDM CES IWF CES IWF The CES Interworking Function (IWF) applies the proper encapsulation to the nxds0 or T1/E1 traffic Pseudowires (PWE3) identify the specific CES connection MPLS Tunnels transport traffic from point A to B Flexible configuration of buffers within the CES IWF allows control of packetization, latency and jitter which meets the requirements for TDM services. 16 MPLS Network for Teleprotection Sep 2009

17 AGENDA 1. Current TDM Environment 2. IP/MPLS Environment 3. Design Considerations a) Components of Latency b) Fine Tuning Latency in MPLS Routers c) Latency Optimization Link/Path Design & Latency Recap d) Timing e) MPLS Service Catalog for Teleprotection 4. Operational Considerations: Next Generation Network Management

18 Latency Optimization Link/Path Design Addressing Variable Delay Unnecessary packet processing by IP/MPLS routers will add latency. MPLS traffic engineering capability enables deterministic and predictable performance. IP Domain Service Aggregation Routers IP Domain Core Service Routers IP Domain Service Aggregation Routers Transport Domain Fiber / SONET / Microwave / DWDM

19 Fine Tuning Latency in MPLS Routers Recap 2.5ms latency is feasible DS1 / E1 Access Circuit DS1/E1 LIU Data Si g Packetization GigE Packet Switched Network (PSN) GigE Jitter Buffer Data Si g DS1/E1 LIU DS1 / E1 Access Circuit Packetization Increase in bandwidth Decrease latency Fixed delay Network Packet transfer delay based on link speeds and distances from end to end Variable delay the number of and type of switches queuing point in the switches QoS is key to ensure effective service delivery Playout Decrease in jitter buffer Decrease latency Decrease jitter tolerance

20 AGENDA 1. Current TDM Environment 2. IP/MPLS Environment 3. Design Considerations a) Components of Latency b) Fine Tuning Latency in MPLS Routers c) Latency Optimization Link/Path Design d) Timing e) MPLS Service Catalog for Teleprotection 4. Operational Considerations: Next Generation Network Management

21 Flexibility in Timing is a Necessity External Synchronization L2 or L3 PSN PRC Line Synchronization PDH, SDH, NTR, I-frame Timing over Packet (Adaptive Clock Recovery, IEEE 1588v2 PTP, NTP) Client L2 or L3 PSN Synchronous Ethernet Synchronous Ethernet Reconstituting the TDM signal demands highly accurate clocking capabilities from the hardware. Flexibility to work with a variety of clocking sources and modes is a significant factor to implementation.

22 AGENDA 1. Current TDM Environment 2. IP/MPLS Environment 3. Design Considerations a) Components of Latency b) Fine Tuning Latency in MPLS Routers c) Latency Optimization Link/Path Design d) Timing e) MPLS Service Catalog for Teleprotection 4. Operational Considerations: Next Generation Network Management

23 MPLS Service Catalog for Teleprotection Pre-define Services and Utilize Templates RSVP-TE Jitter Buffer Size c-pipe Synchronization MPLS Payload Size H-QoS Playout Buffer Size CESoPSN (RFC5086) QoS FRR Network Delay VLL SAToP (RFC4553) Teleprotection Key Requirements End-to-end latency less than 16ms (typical 10ms) Low jitter Teleprotection Service over MPLS (Service Catalog) Design Once and Replicate

24 AGENDA 1. Current TDM Environment 2. IP/MPLS Environment 3. Design Considerations a) Components of Latency b) Fine Tuning Latency in MPLS Routers c) Latency Optimization Link/Path Design d) Timing e) MPLS Service Catalog for Teleprotection 4. Operational Considerations: Next Generation Network Management

25 Operational Considerations Next Generation Management Platform - Requirements Migration to IP/MPLS networking for Teleprotection is enabled by next generation, advanced network management platforms. Key functionalities for consideration should include: 1. Ease of Troubleshooting IP/MPLS is extremely dynamic, does the network manager provide real-time and historical control plane information for service paths? Is this information presented in an interactive graphical display? 2. Latency Monitoring and Alarming Is the network manager capable of pro-actively testing and alarming on conditions where the Teleprotection parameters are not met? 3. Path Symmetry and Alarming Certain Teleprotection schemes are bi-directional in nature and are sensitive to variations between transmit and receive circuit performance. Can the network management platform monitor and alarm if there is an asymmetrical circuit condition in the network?

26 Operational Considerations Next Generation Management Platform - Example Real-time and historical information presented in a graphical format combined with the capability to pro-actively test and alarm on SLA violations are needed to facilitate ease of Operations. Reverse Path 7705 SAR 7750 SR 7750 SR 7705 SAR 7705 SAR 7750 SR 7750 SR 7705 SAR TPR TPR Forward Path 7705 SAR 7705 SAR Next Gen Network Management Performs OA&M tests and reports on results Raises alarm if pre-set SLA threshold crossed Alarms if asymmetrical condition exists on teleprotection circuit Detects network topology and records path changes Alarm

27 Operational Considerations Consistent End-to-End GUI Wizard based service provisioning for Services/Tunnels. Deployment of a multiple-site service can be created and applied in one operation. Mapping services to both physical & logical entities to ensure the correct QoS Simplifies service creation for new and existing customers Real-time Config Database network-wide, end-to-end

28 Operational Considerations Control Plane History and Auditing From June 18, :00 AM To June 18, :35 AM Select Time Interval to Investigate Overlay time this event happened Length of history is dependent on the number of objects kept in the database and the rate of change in the network OSPF adjacency added in this interval (the only control plane event in that interval) Can drill down to see how many times and last occurence Major change in checkpoints infrastructure. CPAM now tracks *all* changes, not just snapshots. Green: new link Red: deleted link Yellow: modified link (filterable) Purple: flapping link (flap count)

29 Take Aways Current TDM Network difficult to scale? Current Network or segments are out of capacity? Troubleshooting is time consuming? Multiple Networks Multiple Teams? Expensive leased circuits? Consider IP/MPLS as the SOLUTION 29

30 Bell Labs Whitepaper: NERC CIP Compliance with Secure MPLS Networks

31 Ken Rabedeau

32

33 Backup Material

34 Network Architecture - TDM OC-3 NxT1 NxT1 NxT1 NxT1 OC-3 Base Station Base Station Base Station Base Station RTU RTU RTU RTU Operations Omni PCX LMR Management Public Internet SCADA collection Billing System Collaboration tools

35 Network Architecture IP/MPLS Broadband IP Traffic Broadband IP Traffic Broadband IP Traffic Broadband IP Traffic Radio + Data Radio + Data Radio + Data Radio + Data GigE Base Station Base Station Base Station Base Station RTU RTU RTU RTU Omni PCX LMR Management Public Internet Operations SCADA collection Billing System Collaboration tools

36 Circuit Emulation Services Over MPLS/GRE for T1/E1 Private Line Transport Structured & Unstructured T1/E1 transport over IP/MPLS or GRE Tunnels 5620 SAM PBX Comprehensive Synchronization Solutions plus embedded OAM and management STM-1,OC-3 ch. T1/E SR 7705 SAR T1/E SAR T1/E1 T1/E1 T1/E SAR 7705 SAR T1/E1 Highly Scalable T1/E1 fan-in Leverage a Transformed PSN Infrastructure for Legacy Services