Design Considerations for Robust EtherNet/IP Networks

Transcription

1 Design Considerations for Robust EtherNet/IP Networks PUBLIC Copyright 2015 Rockwell Automation, Inc. All ri

2 Is This Important? tion Requirements What is real-time? Application dependent.. only you can define what this means for your application. Function Communication Technology Source: ARC Advisory Group Information Integration, Slower Process Automation.Net, DCOM, TCP/IP Time-critical Discrete Automation Industrial Protocols - CIP Motion Control Hardware and Software solutions, e.g. CIP Motion, PTP Period 10 ms to 1000 ms 1 ms to 100 ms 100 µs to 10 ms Industries Applications Oil & gas, chemicals, energy, water Pumps, compressors, mixers, instrumentation Auto, food & beverage, semiconductor, metals, pharmaceutical Material handling, filling, labeling, palletizing, packaging Subset of discrete automation Printing presses, wire drawing, web making, pick & place Copyright 2015 Rockwell Automation, Inc. All ri

3 Is This Important? & Information Convergence Scalable, robust, secure and fu ready infrastructure: Application rnet of Things, Internet of Everything Software Network Copyright 2015 Rockwell Automation, Inc. All ri

4 is this important? ial Automation & Control System Convergence Flat and Open Network Infrastructure Flat and Open Industrial Automation and Control Structured System and Hardened Network Infrastructure IACS Network Infrastructure Copyright 2015 Rockwell Automation, Inc. All ri

5 trial Network Design Methodology et/ip Network Design Considerations rstand application and functional requirements ices to be connected industrial and non-industrial a requirements for availability, integrity and confidentiality munication patterns, topology and resiliency requirements Enabling Avoiding OEM es of traffic information, control, safety, time synchronization, drive control, voice, video lop a logical framework (roadmap) rate from flat networks Because Convergence-Ready to structured and hardened Network networks ine zones and segmentation, Network place applications and devices Sprawl!! e logical framework based on requirements Solutions Infrastructure Matters lop a physical framework to align with and support the logical framework y a Holistic Defense-in-Depth Security Model ce risk, simplify design, and speed deployment: information technology (IT) standards low industrial automation technology (IAT) standards ize reference models and reference architectures AUDIT MANAGE / MONITOR ASSES DESIGN IMPLEMENT Copyright 2015 Rockwell Automation, Inc. All ri

6 orking Design Considerations Reference Architectures ion, design considerations and guidance to help reduce network Latency and Jitter crease the Availability, Integrity and Confidentiality of data, and to help design an a Scalable, Robust, Secure and Future-Ready EtherNet/IP network infrastructure gle Industrial Network Technology bust Physical Layer mentation / Structure (modular & scalable building blocks) ritization - Quality of Service (QoS) dundant Path Topologies with Resiliency Protocols e Synchronization PTP, CIP Sync, Integrated tion on the EtherNet/IP network lticast Management nvergence-ready Solutions urity Holistic Defense-in-Depth lable Secure Remote Access eless Copyright 2015 Rockwell Automation, Inc. All ri

7 orking Design Considerations Reference Architectures ion, design considerations and guidance to help reduce network Latency and Jitter crease the Availability, Integrity and Confidentiality of data, and to help design an a Scalable, Robust, Secure and Future-Ready EtherNet/IP network infrastructure gle Industrial Network Technology bust Physical Layer mentation / Structure (modular & scalable building blocks) ritization - Quality of Service (QoS) dundant Path Topologies with Resiliency Protocols e Synchronization PTP, CIP Sync, Integrated tion on the EtherNet/IP network lticast Management nvergence-ready Solutions urity Holistic Defense-in-Depth lable Secure Remote Access eless Copyright 2015 Rockwell Automation, Inc. All ri

8 le Industrial Network Technology ayer Reference Model Open Systems Interconnection Application Layer 7 Network Services to User App What makes EtherNet/IP industrial? Layer Name Layer No. Function Examples CIP IEC Presentation Layer 6 Encryption/Other processing Session Layer 5 Manage Multiple Applications Transport Network Routers Layer 4 Layer 3 Reliable End-to-End Delivery Error Correction Packet Delivery, Routing IETF TCP/UDP IETF IP Data Link Switches Layer 2 Framing of Data, Error Checking IEEE 802.3/802.1 Physical Cabling Layer 1 Signal type to transmit bits, pin-outs, cable type TIA Physical Layer Hardening Infrastructure Device Hardening Common Application Layer Protocol 5-Layer TCP/IP Model Copyright 2015 Rockwell Automation, Inc. All ri

9 orking Design Considerations Reference Architectures ion, design considerations and guidance to help reduce network Latency and Jitter crease the Availability, Integrity and Confidentiality of data, and to help design an a Scalable, Robust, Secure and Future-Ready EtherNet/IP network infrastructure gle Industrial Network Technology bust Physical Layer mentation / Structure (modular & scalable building blocks) ritization - Quality of Service (QoS) dundant Path Topologies with Resiliency Protocols e Synchronization PTP, CIP Sync, Integrated tion on the EtherNet/IP network lticast Management nvergence-ready Solutions urity Holistic Defense-in-Depth lable Secure Remote Access eless Copyright 2015 Rockwell Automation, Inc. All ri

10 st Physical Layer ration of Partners n and implement a robust physical layer onment Classification - MICE than cable nectors h panels le management e mitigation Grounding, Bonding and Shielding ard Physical Media d vs. Wireless per vs. Fiber vs. STP lemode vs. Multimode LC vs. SC ard Topology Choices ch-level & Device-Level Industrial Ethernet Physical Infrastructure Reference Architecture Design Guide ODVA Guide Fiber Guide ENET-TD003 Cable Sele ENET-WP Copyright 2015 Rockwell Automation, 18 Inc. All ri

11 st Physical Layer mental Focus M.I.C.E. creased Environmental Severity TIA 1005 Office Industrial M.I.C.E. provides a method of catego the environmental classes for each pl Cell/Area zone. This provides for determination of the of hardening required for the networ media, connectors, pathways, devices enclosures. The MICE environmental classificatio measure of product robustness: Specified in ISO/IEC Part of TIA-1005 and ANSI/TIA-568-C.0 sta Examples of rating: 1585 Media : M 3 I 3 C 3 E 3 M12: M 3 I 3 C 3 E 3 RJ-45: M 1 I 1 C 2 E 2 Copyright 2015 Rockwell Automation, Inc. All ri

12 orking Design Considerations Reference Architectures ion, design considerations and guidance to help reduce network Latency and Jitter crease the Availability, Integrity and Confidentiality of data, and to help design an a Scalable, Robust, Secure and Future-Ready EtherNet/IP network infrastructure gle Industrial Network Technology bust Physical Layer mentation / Structure (modular & scalable building blocks) ritization - Quality of Service (QoS) dundant Path Topologies with Resiliency Protocols e Synchronization PTP, CIP Sync, Integrated tion on the EtherNet/IP network lticast Management nvergence-ready Solutions urity Holistic Defense-in-Depth lable Secure Remote Access eless Copyright 2015 Rockwell Automation, Inc. All ri

13 entation red and Hardened Network Infrastructure ler modular building blocks to help 1) minimize network sprawl ) build scalable, robust and future-ready network infrastructure aller fault domains (e.g. Layer 2 loops) aller broadcast domains aller domains of trust (security) le techniques to create smaller network building blocks (Layer 2 domains) ucture and hierarchy Logical model geographical and functional organization of IACS devices Campus network model - multi-tier switch model Layer 2 and Layer 3 Logical framework mentation Multiple network interface cards (NICs) e.g. CIP bridge Network Address Translation (NAT) appliance Virtual Local Area Networks (VLANs) VLANs with NAT Integrated Services Router Copyright 2015 Rockwell Automation, Inc. All ri

14 ple NIC Segmentation ide / Site-wide Network Enterprise-wide Business Systems Levels 4 & 5 Data Center Enterprise Zone Plant LAN VLAN17 - Layer 2 Domain Plant IP - Subnet /24 Plant-wide Site-wide Operation Systems Line/Area Controller Level IDMZ Level 3 - Site Operations Industrial Zone Levels 0-3 Physical or Virtualized Servers FactoryTalk Application Servers & Services Platform Network Services e.g. DNS, AD, DHCP, AAA Remote Access Server (RAS) Call Manager Storage Array Cell/Area Zones Levels 0-2 Cell/Area Zones Levels 0-2 Cell/Area Zones Levels 0-2 Cell/Area Zone #1 Subnet /24 Cell/Area Zone #2 Subnet /24 Cell/Area Zone #3 Subnet /24 Copyright 2015 Rockwell Automation, 30 Inc. All ri

15 Appliance Segmentation ide / Site-wide Network Enterprise-wide Business Systems Levels 4 & 5 Data Center Enterprise Zone Plant LAN VLAN17 - Layer 2 Domain Plant IP - Subnet /24 Plant-wide Site-wide Operation Systems Level IDMZ Level 3 - Site Operations Industrial Zone Levels 0-3 Physical or Virtualized Servers FactoryTalk Application Servers & Services Platform Network Services e.g. DNS, AD, DHCP, AAA Remote Access Server (RAS) Call Manager Storage Array Cell/Area Zones Levels 0-2 Cell/Area Zones Levels 0-2 Cell/Area Zones Levels 0-2 Cell/Area Zone #1 Subnet /24 Cell/Area Zone #2 Subnet /24 Cell/Area Zone #3 Subnet /24 Copyright 2015 Rockwell Automation, 31 Inc. All ri

16 rated Services Router Segmentation ide / Site-wide Network Enterprise-wide Business Systems Levels 4 & 5 Data Center Enterprise Zone Plant LAN VLAN17 - Layer 2 Domain Plant IP - Subnet /24 Plant-wide Site-wide Operation Systems Level IDMZ Level 3 - Site Operations Industrial Zone Levels 0-3 Physical or Virtualized Servers FactoryTalk Application Servers & Services Platform Network Services e.g. DNS, AD, DHCP, AAA Remote Access Server (RAS) Call Manager Storage Array Cell/Area Zones Levels 0-2 Cell/Area Zones Levels 0-2 Cell/Area Zones Levels 0-2 Cell/Area Zone #1 Subnet /24 Cell/Area Zone #2 Subnet /24 Cell/Area Zone #3 Subnet /24 Copyright 2015 Rockwell Automation, 32 Inc. All ri

17 entation without NAT ide Network Enterprise-wide Business Systems Levels 4 & 5 Data Center Enterprise Zone LAN VLAN17 - Layer 2 Domain IP - Subnet /24, every requires a unique IP address Plant-wide Site-wide Operation Systems Level IDMZ Level 3 - Site Operations Industrial Zone Levels 0-3 Physical or Virtualized Servers FactoryTalk Application Servers & Services Platform Network Services e.g. DNS, AD, DHCP, AAA Remote Access Server (RAS) Call Manager Storage Array Cell/Area Zones Levels 0-2 Cell/Area Zones Levels 0-2 Cell/Area Zones Levels 0-2 Cell/Area Zone #1 VLAN10 ubnet /24 Cell/Area Zone #2 VLAN20 Subnet /24 Cell/Area Zone #3 VLAN30

18 entation with NAT ide Network Enterprise-wide Business Systems Levels 4 & 5 Data Center Enterprise Zone LAN VLAN17 - Layer 2 Domain IP - Subnet /24 Plant-wide Site-wide Operation Systems Level IDMZ Level 3 - Site Operations Industrial Zone Levels 0-3 Physical or Virtualized Servers FactoryTalk Application Servers & Services Platform Network Services e.g. DNS, AD, DHCP, AAA Remote Access Server (RAS) Call Manager Storage Array Cell/Area Zones Levels 0-2 Cell/Area Zones Levels 0-2 Cell/Area Zones Levels 0-2 Cell/Area Zone #1 VLAN10 ubnet /24 Cell/Area Zone #2 VLAN20 Subnet /24 Cell/Area Zone #3 VLAN30

19 mentation entation Considerations odular building blocks to help 1) minimize network sprawl ble, robust and future-ready network infrastructure ains (e.g. Layer 2 loops) t domains of trust (security) es to create smaller network building blocks (Layer 2 domains) rarchy

20 esign Considerations rchitectures considerations and guidance to help reduce network Latency and Jitter, to Availability, Integrity and Confidentiality of data, and to help design and, Robust, Secure and Future-Ready EtherNet/IP network infrastructure: l Network Technology l Layer / Structure (modular & scalable building blocks) uality of Service (QoS) h Topologies with Resiliency Protocols ization PTP, CIP Sync, Integrated thernet/ip network gement eady Solutions stic Defense-in-Depth e Remote Access.11

21 ath Topologies with Resiliency Protocols ndant s Cisco Catalyst 3750 StackWise Switch Stack Switch-level Topologies Ring Resilient Ethernet Protocol (REP) Cisco Catalyst 3750 StackWise Switch Stack Star/Bus Linear Cisco Catalyst 3750 StackWise Switch Stack isco atalyst 2955 HMI HMI Controller HMI Controllers Controllers HMI Controllers, Drives, and Distributed I/O Cell/Area Zone Cell/Area Zone Controllers, Drives, and Distributed I/O Cell/Area Zone Controllers, Drives, and Distributed I/O Cell/Area Zone Device-level Topologies Stratix 8300 Switch-level and Device-level Topologies VFD Drive I/O I/O I/O HMI VFD Drive Controllers, Controller Servo Drive HMI

22 ath Topologies with Resiliency Protocols ockwell Automation Stratix 5700/8000 Managed Industrial yer 2 Access Switch Rockwell Automation ControlLogix Programmable Automation Controller create a switching (bridging) loop nfiguration, a loop will lead to a broadcast storm, flooding the network, which will consume available ke down a Layer 2 switched (bridged) network t frames do not have a time-to-live (TTL) can loop forever

23 ath Topologies with Resiliency Protocols: idance Forwarding Blocking cy protocol maintains redundant paths while avoiding switching (bridging) loop

24 ath Topologies with Resiliency Protocols: idance Link Failure Blocking ence (healing, recovery, etc.) must occur before the Industrial Automation and IACS) application is impacted

25 ath Topologies with Resiliency Protocols: idance Link Failure Forwarding ence must occur quickly enough to avoid a Logix Controller connection timeout: instruction - Explicit, CIP Class 3 ut - 30 second default /Consumer - Implicit, CIP Class 1 ut - 4 x RPI, with a minimum of 100 ms it, CIP Class 1 ut - 4 x RPI by default

26 ath Topologies with Resiliency Protocols: idance Blocking Don t forget about potential loops on the switch itself

27 ath Topologies with Resiliency Protocols trial versus COTS - Panel & DIN Rail Mounting vs. Table & Rack (e.g. 1RU) ged versus Unmanaged Advantages Disadvantages aged ches anaged ches A Embedded ch Technology Loop prevention Security services Diagnostic information Segmentation services (VLANs) Prioritization services (QoS) Network resiliency Multicast management services Inexpensive Simple to set up Cable simplification with reduced cost Ring loop prevention & Network resiliency Prioritization services (QoS) Time Sync Services (IEEE 1588 PTP Transparent Clock) More expensive Requires some level of support and configuration to start up No loop prevention No security services No diagnostic information No segmentation or prioritization services Difficult to troubleshoot No network resiliency support Limited management capabilities May require minimal configuration

28 iliency Protocols tion Driven iency col Mixed Vendor Ring Redundant Star Network Convergence > 250 ms Network Convergence ms Network Convergence 1-3 ms Layer 3 Layer D) X X X X (802.1w) X X X X X (802.1s) X X X X X + X X X X X X X Channel 802.3ad) X X X X inks X X X ODVA) X X X X Wise X X X X X X X X X X X X X

29 ath Topologies with Resiliency Protocols ix 5700 witch Forwarding Beacon CompactLogix Controller Active Ring Supervisor Beacon ETAP Announce Blocking Announce POINT I/O ArmorPoint I/O POINT I/O PowerFlex Ring Topology with Device Level Ring Protocol locks traffic on one port n frames on both ports to detect break in the ring restored, supervisor hears beacon on both ports, and transitions to normal ring mode,

30 ath Topologies with Resiliency Protocols ndard enabling suppliers to develop cts nd linear topologies, fiber and copper managed to ensure timely delivery of lity of Service, IEEE-1588 Precision Time st Management) ault tolerant network ms convergence for simple e networks

31 ath Topologies with Resiliency Protocols a chain of switch ports connected to each other and e same REP segment ID ring switch-level topology can be built with REP a single fault tolerant network or IACS applications that can tolerate up to a 100 ms ence recovery time (on fiber interfaces). level resiliency protocol Industrial Automation and IT applications, included with Cisco switch stack, Stratix 5700, 8300 Segment 1 VLAN 10 Catalyst 3750X StackWise Switch Stack Segment 1 VLAN 10 Segment 2 VLAN 20

32 ernet Protocol ple Industrial Zone HMI I/O CIP Class 3 IES VFD Drive dundant ntrollers IES IES CIP Class 1 Class 3 REP IES IES Controller I/O DLR CIP Class 1 IES HMI Controller DLR CIP Sync CIP Motion VFD Drive I/O Safety Controller CIP Safety Safety I/O I/O Servo Drive

33 ernet Protocol entation Considerations ter convergence (recovery from a failure) than RPVST+ or MSTP for a switch ring or IACS applications that can tolerate up to a 100 ms network convergence recovery rfaces). Consume connections with slow RPIs fast RPIs tions that require a faster network convergence recovery time, Cisco and Rockwell mend either a redundant star switch topology with the Flex Links resiliency protocol, ring topology utilizing the ODVA DLR resiliency protocol. nd SFPs for all inter-switch links ring and redundant star switch-level topologies.

34 ath Topologies with Resiliency Protocols entation Considerations dant Path Topology and Resiliency Protocol is application dependent evice-level topologies nt Star Topology dor environment - Legacy Migration rsion of EtherNet/IP IACS devices e hierarchal architecture - Layer 2 vs. Layer 3 etwork Convergence time, Packet loss, Latency & Jitter

35 esign Considerations rchitectures considerations and guidance to help reduce network Latency and Jitter, to Availability, Integrity and Confidentiality of data, and to help design and, Robust, Secure and Future-Ready EtherNet/IP network infrastructure: l Network Technology l Layer / Structure (modular & scalable building blocks) uality of Service (QoS) h Topologies with Resiliency Protocols ization PTP, CIP Sync, Integrated thernet/ip network gement eady Solutions stic Defense-in-Depth e Remote Access.11

36 -Ready Network Solutions rtner Solution(s) e.g. Machine Plant-wide Industrial Automation Systems rtner Solution(s).g. Process Skid Plant-wide / Site-wide Industrial Automation Systems n and deployment considerations that a trusted partner (e.g. OEM, SI, actor) has to take into account to achieve seamless integration of their solution achine, skid) into their customers plant-wide/site-wide network infrastructure. Early, open and two-way

37 -Ready Network Solutions entation Considerations trial network technology that fully utilizes standard Ethernet and IP networking technology as the multietwork infrastructure. nfrastructure devices asset utilization ainability a d User (OT/IT) or OEM? ss), subnet, default gateway (routability) ventions static/dynamic, hardware/software configurable, NAT/DNS ices, switch-level and device-level topologies prioritization revention, redundant path topologies with resiliency protocols on Services sion Time Protocol (PTP w/e2e); CIP Sync applications first fault, SOE, CIP Motion nment with end user: business practices, corporate/local standards, industrial security policies, rent status of network infrastructure security, access control lists, application security (FactoryTalk Security), remote access

38 aterial itectures ntwide Ethernet (CPwE) esilient Ethernet Protocol (REP) in a ntwide Ethernet Architecture 11 Wireless LAN Technology within a ntwide Ethernet Architecture s structure Application Guide mendations for Plant-wide EtherNet/IP Deployments facturing Computer and Controller Assets re Remote Access to plant-floor Applications and Data

39 aterial and Certification al Networking Specialist ertification dules (pre-learning courses) ems Fundamentals for Industrial ICINS) undamentals for Industrial Control ICS) ining dustrial Networks with Cisco echnologies (IMINS) NS CCNA for Industrial Applications - Training and Certification Training - TBD Exam - TBD Industrial IP Advantage E-learning modules CPwE Design Considerations and Best Practices

40 dustrial Networks with Cisco Networking s Training Class tion now offers an exclusive opportunity to become a certified Cisco king Specialist. stall, maintain and troubleshoot industrial network d drawings to recognize industrial topologies and aterials rk availability, reliability and cyber security to pass the Industrial Networking Specialist Exam ( )? Take the Managing with Cisco Networking Technologies (IMINS) being held April 27-May 1 st in the Burr ention you were at the Process Summit and receive a 10% discount on the Course!

41 aterial for educational, technical and ip information about industrial ication t Protocol (IP) for Industrial inded companies e-newsletters with and video sters s available

42 siderations for Robust EtherNet/IP Networks