for Network IP Migration Network Access Timing and Synchronization Test & Measurement Agenda: Architectures and Topologies Product life cycle Media and Protocol Conversion Application Cases Conclusion http://www.cxr.com Mike Ritz MRitz@cxrlarus.com +1-408-573-2700 Page 1
Introduction Do SCADA and Transmission Network evolve at the same rate of change? What benefits from communication technology progress? Performance, added value services, lower cost Is there a universal communication technology that fits all the industry requirements? What are the specific requirements in my organization that have impacts on my network design Topology and geography, resources, network history Page 2
SCADA System Architecture A typical SCADA system is made of 3 parts: Remote systems or remote sites: generator, substations, RTU, sensors, relays, valves, etc. A Master / Control Center or NOC (Network Operation Center) A communication / transmission network Remote site Transmission Network Control Center Page 3
Remote Monitoring Remote Control and Monitoring relay, generators, lights, doors sensor, temperature, humidity, voltage, current, pressure, smoke detection, water levels, etc. air conditioning, alarms, sirens, doors Page 4
Remote Monitoring Additional associated operation services voice communications Network access Images and video Impacts on bandwidth service availability latency for real time applications Quality, packet loss Page 5
Network requirements Network Topologies Bandwidth Topology and number of remote sites, geography, distance Network ownership : public, private Technologies T1 / DS3 SONET Ethernet / IP MPLS Cellular 3G / LTE Multi-layer encapsulations: Ethernet over T1, T1 over IP over MPLS, etc. Media : copper / fiber / Microwave / cellular Anticipate next requirements Page 6
Network Topologies Private and Public network advantages cost of ownership / OPEX, control of operation, availability and performance, security, coverage, etc. Can I Speed up existing communication infrastructure? A copper pair can deliver tens of Mbps for Ethernet traffic Fiber upgrade and CWDM multiplexing MW upgrades IP SCADA, video, Ethernet, voice SONET SCADA, video, Ethernet, voice SCADA, video, Ethernet, voice Page 7
Life Cycles SCADA systems and network technologies : different rate of change Network technology evolution since 70 s : Multi-drop base band modems, dial-up/ll modems, T1, DS-3, SONET, narrow-band MW, xdsl, Frame Relay, Ethernet / IP, ATM, 2/3G, MPLS, 4G/LTE, MPLS-TP 300 bps to +40 Gbps SCADA systems : RS-232 and E&M have been used since the 70 s Communication conversion equipment delivers a path between SCADA systems and the transmission network Throughput RS-232 and E&M still here! CWDM MPLS 40/100 GE SONET FR T1 Modems Page 8 t
Communication Interface Conversion RS-232 to DS-0 in a T1 DACS, with option for T1 over IP emulation Conversion : over-sampling, V.14, X.50, etc. RS-232 to IP conversion Best effort, application aware adaptation layer Security : authentication, non falsification, encryption Packetization latency Rawip Application Data COM port SSL TCP - UDP IP Ethernet Data from the serial interface https Telnet Adaptation interface Authentication & Encryption Page 9
Communication Interface Conversion Circuit Emulation Services over packet networks SAT over packet : RFC 4553 CES over packet : RFC 5086 AAL1 (ATM) over packet : RFC 5087 Significant overhead on lower rate TDM applications Negligible on multiple T1 s, DS3, SONET TDM frames CES SAT TDMoIP header (4x B) RTP option (12x B) UDP (8x B) IP (20x B) Ethernet (18x B+) 1x T1 frame : 24x Bytes Header : up to 62x Bytes Page 10
Communication Interface Conversion Circuit Emulation Services over Packet Network Adaptive Clock Recovery G.823 compliant Accuracy : 10-7 to 10-9 according to the oscillator quality. TDM Network Circuit / connection mode Guaranteed throughput Negligible overhead Negligible network lattency Constant bit rate Carries synchronization Negligible bit error rate and data loss Packet Network Connected or non-connetced mode Shared bandwidth Large overhead Significant and variable network lattency Unpredictable throughput if best effort No synchronization, requires Sync-e & PTP Packet loss, QOS, congestion Page 11
Communication Interface Conversion Circuit Emulation applications Clock Recovery T1, DS3, SONET T1, voice BTS FXO/FXS CES Over IP Packet Network CES Over IP TDM Network E&M, leased line RS-530 / X.21 RTU Pseudo-Wire Domain Page 12
Communication Interface Conversion Case for Legacy interfaces : E&M, RS-232 DS-0 interfaces in T1 multiplexers / channel banks Ethernet / IP integrated access devices with embedded Circuit Emulation Services MW Ethernet T1 CES - IAD Packet Network E&M RS232 Eth Voice FXS SCADA - RTU Page 13
Customer Application Case #1 A Utility network, based on legacy T1 multiplexers over Microwave High availability and resiliency : multiple loop design Redundant synchronization network Add T1 s when need to grow and add new services Substation : RS-232, E&M, Ethernet, FXS Fiber T1 MW Page 14
Customer Application Case #2 A Utility network based on SONET High availability and resiliency inherent to SONET and T1 Scalability : OC-3 / OC-12 / OC-48 / etc. High Ethernet bandwidth : Ethernet over SONET RTU communication interfaces from T1 channel bank SONET + T1 DACS combined equipment RTU RS-232 E&M Video Ethernet Voice Eth FXS SONET OC-n SCADA, video, Ethernet, voice Page 15 SCADA, video, Ethernet, voice
Customer Application Case #3 A Utility telemetry network based on public 3G mobile network Requirements : no network access, autonomous operation Digital input to metering devices RS-232 Control RS-232 over IP SMS text messaging Dial-up / modems Remote site Dial-up 3G IP Page 16
Customer Application Case #4 Modern transportation communication network Ethernet optical network CES over packet and RS-232 to IP conversion for legacy SCADA CES over packet for PBX voice services Ethernet / IP services : VLAN & QOS Customer requirements : Communications cost savings, Internet access Station Ethernet RS-232 E&M T1 Ethernet Optical Network Control Public Network Page 17
Conclusions Remote Monitoring is not yet full IP! Overall architecture is made of hybrid technologies, and will be for quite a while Significant cost already to build existing infrastructure Nobody wants a forklift upgrade! Conversion solutions required at the remote sites Each application has different constraints for QOS / SLA, resilience and availability, latency, security, cost, scalability Page 18
Thank You! Any questions? MRitz@cxrlarus.com Booth # 63 www.cxr.com CARRIER - UTILITIES - DEFENSE TRANSPORT Page 19