REFERENCE ARCHITECTURES FOR MANUFACTURING

Transcription

1

2 Synopsis Industry adoption of EtherNet/IP TM for control and information resulted in the wide deployment of standard Ethernet in manufacturing. This deployment acts as the technology enabler for the convergence of manufacturing and enterprise networks. By gaining timely access to production key performance indicators (KPIs) at the right levels, manufacturers benefit from network convergence. Information convergence between manufacturing and business systems also enables greater business agility and opportunities for innovation. This technology and network convergence creates an unclear demarcation line for network ownership. Groups that traditionally had limited interaction within manufacturers now collaborate. To support this network convergence, controls engineers and Information Technology (IT) professionals experience both organizational and cultural convergence as well as share best practices. The emergence of manufacturing IT, distinct from enterprise IT, takes this collaboration to a new level. To support and accelerate this network convergence, Rockwell Automation and Cisco collaborated to develop Reference Architectures for Manufacturing. These resources provide users with the foundation for success to deploy the latest technology by addressing topics relevant to both engineering and IT professionals. Reference Architectures for Manufacturing provides education, design guidance, recommendations and best practices to help establish a robust and secure network infrastructure that facilitates manufacturing and enterprise network convergence. This whitepaper outlines the recommendations and best practices described with the Reference Architectures for Manufacturing. At the end of this whitepaper is a listing of additional reference material. This listing includes resources not specifically described within this whitepaper. For additional information on Reference Architectures for Manufacturing, see notes 1 and 2 within the listing on the last page of this whitepaper. Control and Information Convergence Convergence is not a new concept. For example, companies often undergo convergence through expansion, mergers and acquisitions. Enterprise-wide systems unite disparate business systems into a common enterprise resource planning (ERP) system. Finally, users converge voice, video, and data into a common information network. In the manufacturing industry, islands of automation for production and control systems have increasingly converged into an integrated plant-wide control and information platform. Users also unite disparate batch, continuous process, discrete, safety, motion, and drive control industrial network technologies into a multidisciplined industrial network by utilizing EtherNet/IP, a standard Ethernet technology. 2 Wide deployment of EtherNet/IP in manufacturing triggered migration from the traditional 3-tier network model to a converged Ethernet model, as shown in Figure 1. Convergence has not flattened the network model. Segmentation of functions, geographic areas, and security for domains of trust requires a multi-tier model.

3 The traditional 3-tier network model evolved during the early days of Ethernet. Characteristics such as collision domains, half-duplex and 10Mbps limited Ethernet usage in production control applications. Proprietary, vendor-specific industrial networks proliferated early on, until organizations like ODVA began promoting a Common Industrial Protocol (CIP TM ). By dividing a network by function and geographic area into smaller local area networks (LAN), the 3-tier network model provides natural segmentation. This lessens the impact of traffic management and security. By connecting devices such as drives and robots with a controller, a device-level network controls, configures, and collects data from these intelligent devices. A device-level network in one area does not typically interact with other device-level networks. By acting as a backbone for device-level networks, control networks interlock controllers and provide connectivity to supervisory computers. A gateway maps information from the manufacturing systems to the enterprise systems. The manual, store-and-forward mapping mechanism required significant implementation and support efforts. Traditional 3-Tier Manufacturing Network Model Figure 1 Converged Ethernet Manufacturing Network Model The naturally information-enabled, converged Ethernet model eliminates the need for dedicated gateways. Although the technology has converged, the model has not flattened. Data access from anywhere at anytime presents a new challenge. Manufacturers must protect their assets from both internal and external threats (people with good intentions that make mistakes and those wishing to inflict harm) because users typically know how to plug into Ethernet. No longer isolated in the manufacturing realm, industrial networks make manufacturing computing and controller assets susceptible to the same security vulnerabilities as their enterprise counterparts. Plant-wide networking with Ethernet technology requires planning and structure. Establishing smaller LANs, to shape and manage network traffic as well as creating domains of trust that limit access to authorized personnel requires a multi-tier, segmented methodology. 3

4 Built on Industry Standards and Methodology Designing and deploying a robust and secure network infrastructure requires a wellplanned roadmap. The manufacturing process dictates usage of equipment such as sensors and actuators as well as their geographic deployment. By consulting operations, users can determine information flow requirements. Users should also identify what production information the business system needs. For example, a business system may require KPIs or regulatory compliance data. Finally, the roadmap should address standards implementation for common terminology, methodology, and best practices. Reference Architectures for Manufacturing are built on technology and manufacturing standards common between IT and manufacturing. These include technology standards such as IEEE s standard, unmodified Ethernet, Internet Engineer Task Force (IETF) Internet Protocol (IP), and ODVA s CIP. Additionally, Reference Architectures for Manufacturing uses manufacturing standards to establish a Manufacturing Framework as shown in Figure 2. This framework establishes a foundation for network segmentation for traffic management and policy enforcement, such as security, remote access, and Quality of Service (QoS). The framework uses standards such as the ISA-95 Enterprise-Control System Integration, ISA-99 s Manufacturing and Control Systems Security, and the Purdue Reference Model for Control Hierarchy. Figure 2 Manufacturing Framework Rockwell Automation and Cisco share a common technology view by supporting the facilitation and acceleration of network convergence as well as the promotion of standard, unmodified Ethernet. In addition to jointly serving as principle members of ODVA, the companies individually participate in standard organizations like ISA. For additional information about ODVA, see note 4. 4 Throughout the Reference Architectures for Manufacturing, terminology refers to layers, levels, and zones. The Open Systems Interconnection (OSI) seven-layer

5 reference model defines layers e.g. layer 1 for Physical, layer 2 for Data Link, layer 3 for Network. Layer 2 devices forward data and provide network services based on Data Link layer characteristics such as Media Access Control (MAC). Layer 3 devices forward data and provide network services based on IP. For additional information on the OSI network model, see note 5. Figure 2 depicts levels and zones of the Manufacturing Framework. Both ISA-95 and the Purdue Reference Model for Control Hierarchy segment industrial control devices into hierarchical levels of operations within a manufacturing facility. Using levels as common terminology breaks down and determines plant-wide information flow. For enhanced security and traffic management, ISA-99 segments levels into zones. Zones establish domains of trust for security access and smaller LANs to shape and manage network traffic. For additional information about ISA, see note 7. The Manufacturing Framework groups levels into the following zones for specific functions: Enterprise Zone: Levels 4 and 5 handle IT networks, business applications/servers (e.g. , enterprise resource planning ERP) as well as intranet. Demilitarized Zone (DMZ) This buffer zone provides a barrier between the Manufacturing and Enterprise Zones, but allows for data and services to be shared securely. All network traffic from either side of the DMZ terminates in the DMZ. No traffic traverses the DMZ. That is, no traffic directly travels between the Enterprise and Manufacturing Zones. Manufacturing Zone: Level 3 addresses plant-wide applications (e.g. historian, asset management, manufacturing execution systems - MES), consisting of multiple Cell/Area Zones. Cell/Area Zone: Levels 0, 1 and 2 manage industrial control devices (e.g. controllers, drives, I/O and HMI) and multidisciplined control applications (e.g. drive, batch, continuous process and discrete). Shaping and Managing Network Traffic Developing a robust and secure network infrastructure requires protecting the integrity, availability and confidentiality of control and information data. Users should address the following when developing a network: Is the network infrastructure resilient enough to ensure data availability? How consistent is the data? Is it reliable? How is data used? Is it secure from manipulation? Reference Architectures for Manufacturing provides recommendations, design guidance, best practices, methodology (Figure 3) and documented configuration settings. This helps establish a robust and secure network infrastructure for control and information data availability, integrity, and confidentiality. Built on industry standards and a future-ready network foundation, Reference Architectures for Manufacturing addresses today s applications like safety through CIP Safety TM, and tomorrow s applications like motion through CIP Motion TM, time synchronization 5

6 through IEEE 1588 precision time protocol (PTP) with CIP Sync TM, and incorporation of voice over IP (VoIP) and video on demand (VOD). IT professionals frequently use reference architectures as a common concept and tool within the enterprise. From retail companies to data centers, Cisco develops reference architectures for a variety of industries and applications. Reference Architectures for Manufacturing, as shown in Figure 3, incorporates the Rockwell Automation Integrated Architecture TM and Cisco Ethernet-to-the-Factory, a Cisco Validated Design. For additional information on the Integrated Architecture, see notes 1 and 8. Figure 3 Reference Architectures for Manufacturing 6 To align with the Manufacturing Framework shown in Figure 2, Reference Architectures for Manufacturing utilizes the Campus Network Reference Model. Common with enterprise networks, this multi-tier model naturally segments traffic into three main tiers: core, distribution and access. Layer 2 access switches aggregate control devices within the Cell/Area Zones. Additionally, layer 2 provides network services such as switching, resiliency via spanning tree protocol (STP), Quality of Service (QoS), virtual local area network (VLAN) and security. Multilayer (layers 2 and 3) distribution switches reside in the Manufacturing Zone (level 3), brings together access switches from the Cell/Area Zones and provide network services. Services include layer 2 and 3 switching, routing, load balancing, resiliency via Hot Standby Routing Protocol (HSRP), QoS and security. Finally, the core switch aggregates distribution switches and provides high speed switching. Like Reference Architectures for Manufacturing, IT professionals frequently use core/distribution/access as a common concept and tool within the enterprise.

7 Designing a resilient network infrastructure with low latency and jitter increases the availability and integrity of control and information data. Latency, or delay, represents the time elapsed from when one device transmits data until another device receives it. Jitter represents the variation of delay. Converging multidiscipline control and information traffic into a common industrial network requires reducing latency and jitter. To reduce network latency and jitter, Reference Architectures for Manufacturing recommends segmenting and prioritizing network traffic. Segmentation reduces the impact of broadcast and multicast traffic. Reducing network latency and jitter starts with the Cell/Area Zone. When designing the Cell/Area Zone, users should create smaller layer 2 Cell/Area Zone network segments organized by function or geographic area. Restrict data flow out of the Cell/Area Zone unless plant-wide operations explicitly require it. Each Cell/Area Zone should be implemented with a dedicated VLAN and IP subnet. VLANs segment network traffic and help restrict broadcast and multicast traffic as well as simplify security policy management. As a best practice, use the layer 3 distribution switches to route information between Cell/Area Zone VLANs and plant-wide operations in the Manufacturing Zone. Avoiding large layer 2 networks helps simplify network management. For additional information on VLANs, see notes 1, 2, and 5. Network topology choice impacts the availability and integrity of control and information data. Figure 3 depicts the bus/star, ring and redundant star topologies described in Reference Architectures for Manufacturing. Since applications drive topology choice, users should address key considerations. These include application performance requirements, network latency and jitter tolerance, downtime and meantime-to-repair (MTTR) tolerance as well as future upgrade and expansion requirements. From right to left, Figure 3 depicts increases to network resiliency, modularity, flexibility and implementation complexity. As a best practice, implement a resilient topology such as the recommended redundant star topology. Redundant star provides natural segmentation, shapes traffic to help reduce latency and jitter by improving data integrity as well as offers the resiliency required for greater data availability, which helps reduce downtime. Modularity of the redundant star also increases scalability and flexibility for network expansion and upgrades. Not all network traffic is created equal, nor should users treat it equally. To minimize application latency and jitter, control data should have priority within the Cell/Area Zone. Quality of Service (QoS) gives preferential treatment to some network traffic at the expense of others. Control data is more sensitive to latency and jitter than information data. To minimize latency and jitter, users should apply QoS to control data within the Cell/Area Zone. Before implementing QoS within the Manufacturing Zone, use a multidiscipline team of operations, engineering, IT and safety professionals to establish a QoS policy for the Manufacturing Zone. This policy should support the needs of operations, including what to apply QoS to and when. Additionally, the multidiscipline team should understand that this policy may differ from the enterprise QoS policy. Enterprise QoS policies commonly give priority to VoIP. Although not specifically addressed within Reference Architectures for Manufacturing, developing a robust network infrastructure requires proper design and implementation of an industrial Physical layer. Physical media, layer 1, within the Cell/Area Zone is subjected to environmental and noise conditions not found in the enterprise. These conditions can impact availability and reliability of data, introducing latency and jitter. For additional information on physical media planning and installation, see note 6. 7

8 Recommendations and best practices for the Cell/Area Zone include: Shape and manage traffic by implementing smaller Cell/Area Zones with a separate VLAN and IP subnet per Cell/Area Zone. Use managed layer 2 access switches to segment traffic with VLANs, prioritize traffic with QoS, implement security policies with port security and access control lists (ACL), and provide diagnostics. Utilize a redundant star topology for greater network resiliency and modularity, along with rapid spanning tree protocol (RSTP) to manage loops. Implement multiple spanning tree (MST 802.1s) version of RSTP (802.1w) to support usage of multiple VLANs. For additional details on MST and RSTP, see note 2. Lower network latency and jitter by using Gigabit Ethernet ports for trunks and uplinks, VLANs to reduce broadcast traffic, Internet Group Management Protocol (IGMP) to reduce multicast traffic, QoS to prioritize traffic and redundant star topology for natural segmentation. For additional information on these best practices, see note 2. The Manufacturing Zone contains all systems, devices and controllers critical to controlling and monitoring plant-wide operations. This zone includes Site Manufacturing Operations and Control functions (level 3) as well as multiple Cell/Area Zones. To preserve smooth plant-wide operations and functioning of the systems and network, this zone requires clear isolation and protection from the Enterprise Zone via the Demilitarized Zone (DMZ). All manufacturing assets required for the operation of the Manufacturing Zone should remain there. Assets include Rockwell Automation FactoryTalk Integrated Performance and Production Suite as well as other applications and services, such as Active Directory, DNS, and DHCP. Level 3, Site Manufacturing Operation and Control, has a dedicated network segment within the Manufacturing Zone and contains the FactoryTalk servers. Users should assign this network segment with its own IP subnet and VLAN. The FactoryTalk servers connect to a dedicated multilayer access switch, which aggregates into the layer 3 distribution switches. The distribution switches act as the network segment s default gateway. To provide redundant default gateways to the Cell/Area Zones, distribution switches should use Hot Standby Routing Protocol (HSRP) or Gateway Load Balancing Protocol (GLBP). Distribution switches will route all traffic to and from the level 3 network segment. Recommendations and best practices for the Manufacturing Zone include: Keep FactoryTalk within the Manufacturing Zone. For additional information, see note 10. Keep replicated services such as DNS, Active Directory and DHCP within the Manufacturing Zone. Implement a level 3 (Site Manufacturing Operations and Control) network segment with its own IP subnet and VLAN. 8 Use layer 3 distribution switches to route between Cell/Area Zone VLANs and the level 3 network segment VLAN.

9 Use HSRP or GLBP on the distribution switches to provide redundant default gateways to the Cell/Area Zones. For additional information on these best practices, see note 2. Securing Manufacturing Assets The recommended defense-in-depth approach, depicted in Figure 4, helps to address internal and external security threats as well as helps provide confidentiality for control and information data. By utilizing multiple layers of defense (physical and electronic) at different levels within manufacturing, this approach addresses disparate types of threats. No single technology or methodology fully secures industrial networks. A comprehensive security model should be designed and implemented as a natural extension to the manufacturing process. Security should not be implemented as an afterthought or bolt-on component. For the purpose of this whitepaper, defense-in-depth layers for securing manufacturing assets include: Physical Security: This limits physical access to authorized personnel for areas, control panels, devices, cabling, the control rooms and other locations as well as escorts and tracks visitors. Network Security: This contains the infrastructure framework, such as firewalls with intrusion detection and intrusion prevention systems (IDS/IPS). Computer Hardening: This includes patch management and antivirus software as well as removal of unused applications, protocols and services. Application Security: This contains authentication, authorization and audit software. Device Hardening: This handles change management and restrictive access. For additional information on defense-in depth, see notes 11 and 13. Figure 4 Defense-in-Depth - Multiple Layers 9

10 The recommended Manufacturing Network Security Framework, utilizing defensein-depth is depicted in Figure 5 and includes: Manufacturing Security Policy: This security policy roadmap identifies vulnerability mitigation. A multidiscipline team of operations, engineering, IT and safety should develop this manufacturing security policy. Demilitarized Zone (DMZ): This buffer zone provides a barrier between the Manufacturing and Enterprise Zones, while allowing users to securely share data and services. All network traffic from either side of the DMZ terminates in the DMZ. No traffic traverses the DMZ, which means that traffic does not directly travel between the enterprise and manufacturing zones. Defending the manufacturing edge: Users should deploy stateful packet inspection (SPI) firewalls (barriers) with intrusion detection/prevention systems (IDS/IPS) around and within the industrial network. Protecting the Interior: Users should implement access control lists (ACLs) and port security on network infrastructure devices such as switches and routers. Endpoint Hardening: This restricts access, prevents walk up, plug in access and uses change management to track access and changes. Domains of Trust: Users should segment the network into smaller areas based on function or access requirements. Physical Security: This restricts physical access to manufacturing assets and network infrastructure devices. Security, Management, Analysis and Response System: This monitors, identifies, isolates and counters network security threats. Remote Access Policy For employee and partner remote access, implement policies, procedures and infrastructure. For additional information on remote access, see note 12. Recommendations and best practices for securing manufacturing assets include: Deploy holistic security based on defense-in-depth. Conduct a security risk assessment, see note 15 for additional information. Develop a manufacturing security policy that support manufacturing operation requirements based on enterprise security policy best practices. Implement a manufacturing network security framework to establish domains of trust and appropriately apply security policies. Establish a DMZ between the Enterprise and Manufacturing Zones. Prevent traffic from traversing the DMZ. 10 Use application mirroring within the DMZ to converge Manufacturing and Enterprise Zone information, noted in next section.

11 Harden computers and controllers, see note 13. Utilize industry standards such as ISA-99. Leverage Rockwell Automation Network and Security Services, see note 15. For additional information, see note 13. Figure 5 Manufacturing Network Security Framework Information Convergence via the DMZ Information convergence has helped provide manufacturers with greater business agility and opportunities for innovation. With these opportunities, come challenges. Manufacturing computing and controller assets have become susceptible to the same security vulnerabilities as their enterprise counterparts. Protecting manufacturing assets requires a defense-in-depth security approach. For additional details, see notes 11 and 13. The best practices described within Reference Architectures for Manufacturing, utilizing defense-in-depth, help to provide a robust and secure network infrastructure facilitating information convergence between manufacturing and business systems. The first best practice calls for establishing a DMZ between the Enterprise Zone and the Manufacturing Zone. As noted earlier, the DMZ is a buffer zone providing a barrier between the Manufacturing and Enterprise Zones, but allows for data and services to be shared securely. All network traffic from either side of the DMZ terminates in the DMZ. No traffic traverses the DMZ. That is, no traffic directly 11

12 travels between the Enterprise and Manufacturing Zones. Finally, users should contain all manufacturing assets, such as FactoryTalk, required for manufacturing operations within the Manufacturing Zone. To maintain these best practices while allowing information convergence between the Enterprise and Manufacturing Zones, Manufacturing Zone applications should replicate data to an application mirror within the DMZ. Users should then replicate the data from this application mirror to an application within the Enterprise Zone. This can be either unidirectional or bidirectional. Figure 6 FactoryTalk Transaction Manager and MSSQL Server An example of data mirroring is shown in Figure 6. FactoryTalk applications that utilize Microsoft SQL (MSSQL) server, for example can maintain the best practices and methodology noted above. For additional information on FactoryTalk, see notes 9, 10, and 11. Figure 6 also demonstrates that FactoryTalk Transaction Manager provides two-way data exchange between tags, such as Logix Controller or FactoryTalk View, and applications like an MSSQL server. These tags may contain KPIs or other important data that needs to integrated into an enterprise application. Since traffic cannot traverse the DMZ, an MSSQL server in the Manufacturing Zone cannot directly transfer data to and from an MSSQL server in the Enterprise Zone. This means that all traffic between the two zones must be initiated or terminated in the DMZ. 12

13 Users should implement the methodology shown in Figure 6 to enable information convergence while maintaining DMZ best practices. The FactoryTalk Transaction Manger with MSSQL server solution involves: The FactoryTalk Transaction Manager server (level 3) is configured to read/write its SQL data to and from an MSSQL server (data mirror) located in the DMZ. The MSSQL server data mirror in the DMZ then replicates the data to and from the Enterprise Zone MSSQL server. Business systems within the Enterprise Zone only access the enterprise MSSQL server. Summary The convergence of manufacturing and enterprise networks increases access to manufacturing data, which assists manufacturers in making better business decisions. This business agility provides a competitive edge for manufacturers that embrace convergence. With these opportunities, come challenges. Network convergence exposes manufacturing assets to security threats traditionally found in the enterprise. Users also face an unclear demarcation of network ownership and cultural difference between deploying enterprise and manufacturing assets. Implementing best practices from both engineering and IT along with the recommendations described in Reference Architectures for Manufacturing will help users establish the secure and robust network infrastructure needed to facilitate manufacturing and enterprise network convergence 13

14 Additional Reference Material Notes: 1) Reference Architectures for Manufacturing Website 2) Design and Implementation Guide (DIG) ) Ethernet Design Considerations for Control System Networks ENET-SO ) ODVA 5) Network Infrastructure for EtherNet/IP: Introduction and Considerations 6) EtherNet/IP Media Planning and Installation Manual _Planning_and_Installation_Manual.pdf 7) ISA-99, Industrial Automation and Control System Security 8) Rockwell Automation Integrated Architecture 9) FactoryTalk Website 10) FactoryTalk Positioning within Reference Architectures for Manufacturing Whitepaper 11) FactoryTalk Security Quick Start Guide 12) Remote Access Whitepaper 13) Securing Manufacturing Computing and Controller Assets Whitepaper 14) Rockwell Automation Knowledgebase ) Rockwell Automation Network and Security Services EtherNet/IP, CIP, CIP Safety, CIP Motion and CIP Sync are trademarks of ODVA. FactoryTalk is a registered trademark of Rockwell Automation, Inc. Integrated Architecture is a trademark of Rockwell Automation, Inc. Publication ENET-WP004A-EN-E-November 2008 Copyright 2008 Rockwell Automation, Inc. Printed in USA