PowerNet Network Communications Guidelines Application Notes August 2002
This page intentionally left blank.
Table of Contents Introduction...1 Network Architecture...1 Functional Network Requirements...2 Acceptable Network Architectures...2 Enterprise Networks...2 Stand-Alone Network...5 New Network Installations...5 NetLink vs. EMINT...6 Network Routing...6 Network IP Addressing...7 Network Hardware Configuration Requirements...7 Auto-Negotiation...7 Repeater Based Ethernet Network...8 Switched Ethernet Network...8 Quality of Service (QoS)...8 Approved Ethernet Hub and Switch List...8 Network Cabling Requirements...9 Backbone Cabling...9 Horizontal Cabling...10 Horizontal Cable Test Requirements...11 Grounding...12 Network Security...12 Safety And Compliance Standards...13 Network Standards...13 Technical Support...13
This page intentionally left blank.
PowerNet Network Communications Guidelines Introduction The Eaton Corp. Cutler-Hammer PowerNet system uses the Internet Protocol (IP) and Ethernet technology to communicate with distributed system components in a common Local Area Network (LAN) environment. The PowerNet system collects data from various energy management system components and compiles this data into a database for periodic and prompted reports. Communications between the host and remote data collection nodes is synchronous in nature (poll-response) and includes data collection commands to individual nodes and broadcast messages. This document addresses the connectivity requirements for the PowerNet Ethernet interface devices, EPONI and EMINT. The EPONI and EMINT are referred to as edge interface devices, as they are used as slave devices at the edge of the PowerNet network. Special considerations outside the scope of this document are required when using alternative backbone infrastructures such as FDDI and ATM. Whatever backbone infrastructure is used, end-to-end, Ethernet (IEEE 802.3) connectivity, and the propagation of UDP broadcast packets are required in order to implement functions involving time-synchronous commands, such as energy snapshot broadcasts. Traffic through Ethernet networks can fail for a variety of reasons: components such as switches or routers may lose power, bursts of network traffic may prevent packet delivery, network wiring errors may cause intermittent problems, and so on. During periods of network outages, the PowerNet system will be unable to communicate to the edge interface devices (EPONIs and EMINTs). Loss of polled data will result during these periods, as PowerNet will time out when requesting information from edge interface devices. System requirements vary from application to application. For systems that can tolerate temporary network outages (such as if periodic loss of data is not important), edge interface devices can readily be employed without much concern about the underlying Ethernet network. This document is intended to provide guidance for those systems that cannot tolerate loss of data due to periodic network outages. In particular, if an application requires energy snapshot broadcasts (such as for energy trending and energy billing) to Ethernet edge interface devices, then due consideration must be given to ensuring that network outages do not occur. If that is not possible, then on the problematic Ethernet segments, NetLinks should be used, and edge interface devices (EPONIs and EMINTs) should not be used. The rest of this document provides guidance in ways to implement PowerNet edge interface devices in a network environment such that loss of data due to network outages is minimal. Network Architecture This section addresses Ethernet and IP network architecture issues that need to be considered when planning the implementation of a PowerNet system in a network environment. PowerNet Network Communications Guidelines 1
Functional Network Requirements PowerNet system products can be implemented in any standard Ethernet network where the following functional network requirements can be met. Static IP Each PowerNet networked device must be assigned a static IP address. UDP Broadcast The network (all switches and routers) must be able to support UDP broadcasts. Network Response Time The network must ensure the response time of PowerNet, UDP packet data traffic. IEEE 802.3 Compliant The network must be compliant with all Ethernet standards. Acceptable Network Architectures PowerNet system products can operate in a wide range of Ethernet network environments that meet the above requirements. The recommended characteristics for the various Ethernet network environments are summarized below in their preferred order of implementation. These recommendations are designed to support the robust network response time requirements of the time-sensitive data collection requests. Refer to the Network Hardware Configuration Requirements on Page 7 for specific network configuration requirements. Enterprise Networks Enterprise networks are defined as networks that already support large volumes of IP-based data communications across the Enterprise. As the window of available Network bandwidth narrows, the performance of PowerNet polling is adversely affected in an exponential fashion. (Refer to Figure 1 below.) PowerNet UDP Poll-Response Errors %Ethernet Utilization Figure 1: PowerNet UDP Poll-Response Errors Vs. % Ethernet Utilization 2 PowerNet Network Communications Guidelines
Competing for network bandwidth introduces the need to examine additional network characteristics, which are prioritized as follows. Supporting any one of these recommended configurations will greatly improve network response time for PowerNet products. Quality of Service Many large sophisticated networks today have the ability to prioritize different types of data traffic within the Internetworking Operating System (IOS). The preferred choice is to implement Quality of Service (QoS) or Traffic Shaping rules within the network IOS that would place a high priority on PowerNet s UDP traffic. A worst case analysis has shown that a DeviceServer configured with the maximum number of devices, polling all attributes simultaneously, will have a worst case network utilization of approximately 20% of the network bandwidth (2.5% for a NetLink). Note that a typical PowerNet system s network utilization ranges from about 0.5% to 2%. Virtual LANs Many Ethernet networks today have the ability to support Virtual LANs or VLANs. VLANs provide the ability to segment a network at the Data Link layer (OSI Layer 2). Where this technology is available, it is highly recommended that PowerNet system products be placed on their own VLAN. Though all VLANs compete equally for backbone network bandwidth, VLANs confine common traffic such as network broadcasts to their own VLANs, and provide an additional layer of security between PowerNet devices and the rest of the network. Figure 2 on Page 4 illustrates and Enterprise LAN using switched VLAN technology. Switched Ethernet PowerNet products collect time sensitive data and therefore must be able to communicate within predictable time intervals. As such, networks where PowerNet products are being planned for deployment must utilize Ethernet Switches for workgroup connectivity. Ethernet Repeaters Ethernet repeaters (also referred to as hubs) do not isolate traffic between devices, which can result in a high number of Ethernet frame collisions. Since PowerNet products collect time-sensitive data, devices must be able to communicate within predictable time intervals. Therefore, use of PowerNet products should not be considered on networks supporting multiple applications/users that connect through the same collision domain via Ethernet Repeaters. PowerNet Network Communications Guidelines 3
SD PowerNet Database Server PowerNet Client Workstation NetLink VLAN A: Dedicated VLAN for User Network VLAN B: Dedicated VLAN for PowerNet System E M I N T EPONI VLAN A VLAN B Ethernet Switces With Multiple VLAN Capability VLAN A VLAN B IEEE 802.1Q VLAN Trunk or Seperate Physical Connection Figure 2: Enterprise Switched Network with VLANs 4 PowerNet Network Communications Guidelines
SD CASCADE ID CASCADE ON CASCADE MANAGEMENT MANAGEMENT IN OUT 2345 1 2 3 4 5 6 7 8 9 10 11 12 LINK / ACTIVITY 1 2 3 4 5 6 7 8 9 1011 12 PARTITION MODULE PWR SNMP COL SD CASCADE ID CASCADE CASCADE MANAGEMENT ON MANAGEMENT IN OUT 2 345 1 2 3 4 5 6 7 8 9 10 11 12 LINK / ACTIVITY 1 2 3 4 5 6 7 8 9 1011 12 PARTITION MODULE PWR SNMP COL SD Stand-Alone Network In the event that the recommendations outlined above for Enterprise Networks are not achievable, placing the PowerNet system devices on a stand-alone LAN is recommended. The simplest form of a stand-alone network would be a repeated or shared Ethernet as shown as follows in Figure 3. Under this configuration, all of the PowerNet system devices would be isolated to a common collision domain using Ethernet repeaters. As previously stated, only PowerNet devices should be connected to this type of network. PowerNet Database Server PowerNet Client Workstation NetLink E M I N T EPONI Ethernet Repeater Hubs LANCAST SuperHub LANCAST SuperHub Ethernet Link Between Repeater Hubs (Maximum 4 Repeater Cascade) New Network Installations Figure 3: Stand Alone Network with Hubs For new network installations, the following is a list of acceptable network architectures. They are listed in order from most preferred to least preferred. 1. Stand-Alone Network utilizing hubs and/or switches 2. Enterprise Network with QoS Implemented for PowerNet system devices 3. Enterprise Network with dedicated VLAN for PowerNet system devices 4. Enterprise Network employing a switched architecture PowerNet Network Communications Guidelines 5
NetLink vs. EMINT Regardless of what network architecture is in place, the decision of whether to use a NetLink or EMINT should be based on the applications that are to be supported. In networks where critical applications such as Energy Billing, Energy Logging, Energy Trending, Analog Alarming and Device Alarming must be supported, Eaton Cutler-Hammer recommends using NetLink devices. When system requirements include placing PowerNet devices in remote facilities connected through Wide Area Network (WAN) links, placing a NetLink on the same switch segment as the PowerNet Edge Interface Devices (i.e. EMINT and EPONI) is considered mandatory to eliminate lost data due to possible WAN link outages. (Refer to Figure 4 below.) Figure 4: Acquisition of Remote Device Data Across a WAN using a NetLink Network Routing Integrating a PowerNet system on to a routed LAN is acceptable; however, there are PowerNet system components that are designed to use broadcast messages to communicate with network interface devices. Routers are typically configured to block broadcasts. In the event that a PowerNet system must be integrated onto a routed network, the routers must be configured to allow forwarding of UDP broadcast packets on Port 5150. In addition, the default gateway address (normally the router interface IP address) will need to be specified in the EMINT and EPONI setup parameters. Integration of a PowerNet system into a routed network requires extensive participation of the Owner s IT group. 6 PowerNet Network Communications Guidelines
Network IP Addressing PowerNet system devices require static IP addresses to communicate across the network. The planning of an IP addressing scheme on an existing network must be coordinated with the Owner s IT group. Private addresses, as set aside by the Internet Assigned Numbers Authority (IANA), should be used as described in RFC 1918, and as summarized below in Table 1. Public addresses (provided and maintained by the InterNIC) should not be used for PowerNet devices. The IP addressing scheme will be designed to use a single subnet with room for future expansion. All subnet addresses must be consistent throughout the network to reduce network latency. Network No. of Private Networks Address Range Total Number of Usable Addresses Available Class A 1 10.0.0.0 10.255.255.255 16,777,214 Class B 16 172.16.0.0 172.31.255.255 65,534 Class C 255 192.168.0.0 192.168.255.255 254 Table 1: IANA Private Network Address Ranges Note When using an entire Class of addresses, keep in mind that there are two addresses not available for use on the network. These are the first address in the range (e.g., 92.168.2.0, all zeros for the host portion), and the last address in the range (i.e. 192.168.2.255, all ones for the host portion). The first address in a range is reserved for the network address, and the last address is reserved as the broadcast address for the subnet. Classless Inter-Domain Routing (refer to RFCs 1517, 1518, 1519 and 1520) requires unique identification of network and broadcast addresses. Network Hardware Configuration Requirements The following sections address the recommended network hardware configuration requirements to ensure the consistent real-time data gathering features of the PowerNet system. In order to provide accurate data gathering, the PowerNet system and components require a network infrastructure that is stable, thereby providing a reliable link between the Powernet system s DeviceServer and the edge interface devices (EMINT and EPONI). Auto-Negotiation Most Ethernet switches produced today (and some Ethernet repeaters/hubs) provide a mechanism for auto-negotiation of the Ethernet link (e.g. 10/100 Mbps and half-duplex/fullduplex). If the link partners (Ethernet Switch and end device) fail to negotiate to the same link speed and configuration settings, a link may be established, but performance problems can arise. PowerNet Network Communications Guidelines 7
The Ethernet interfaces within the EPONI and EMINT only support 10 Mbps and half-duplex communications. To prevent a compatibility mismatch, it is highly recommended that autonegotiation is disabled at the corresponding Ethernet switch ports, and that these ports be configured for static 10 Mbps, half-duplex operation. Repeater Based Ethernet Network At a minimum, an Ethernet network consisting of a single collision domain (repeater based) can be used to provide end-to-end connectivity. However, this type of network should be standalone with no other services attached or supported. In addition, all components must be fully compatible with the IEEE 802.3 specifications, and the repeater (hub) ports connected to EMINTs and EPONIs must be configured for 10 Mbps and half-duplex operation. In a single-collision domain environment, it is important that the PowerNet system network is physically independent from the Owner s user network. In addition, compliance to IEEE 802.3 standards must be maintained including cable distance constraints and the four-repeater rule, dictating a maximum of four repeaters in cascade within any network segment. Switched Ethernet Network Performance enhancements will be noticeable if the devices are connected to a switched Ethernet network. If connecting to an existing switched network, it is recommended that the use of Virtual LAN (VLAN) technology be used to isolate PowerNet system traffic from the user network. The VLAN should be established end-to-end between the Device Server and edge devices. Standard VLAN trunking using IEEE 802.1Q encapsulation should be supported by the switched Ethernet backbone. In this VLAN configuration, static ARP caching should be used to avoid convergence delays during system reconfiguration. Note that if alternative backbone technologies (e.g., ATM, FDDI, etc.) are in use at the site, the backbone must be configured to allow the propagation of UDP broadcasts from the PowerNet system Device Server(s) to all edge equipment. Quality of Service (QoS) In a switched LAN environment, it may be desirable to provide guaranteed bandwidth, and minimize network backbone delays by instituting traffic shaping, particularly on switched network trunks. Fundamentally, Quality of Service (QoS) enables congestion management by prioritizing specific traffic on a network. To ensure appropriate response time when using PowerNet to implement logging of energy data ("Energy Logging"), the network administrator should consider using QoS to prioritize the broadcast traffic (UDP, Port 5150) that is sent from the Device Server to the edge devices. The responses from the edge interface devices are less time sensitive and would not have to be prioritized. Approved Ethernet Hub and Switch List The following Ethernet hub and switch devices have been tested and approved for use with the PowerNet system: 1. NETGEAR Fast Ethernet switch (Model # FS105) 2. NETGEAR 4 port Hub (Model # EN104) 3. SMC EtherEZ Hub (Model # 3608TAC) 8 PowerNet Network Communications Guidelines
4. LINKSYS 4 port 10/100 EtherFast (Model # EWFAH05W) 5. Black Box 9 port personnel MiniHub (Model # Le2690A) Network Cabling Requirements All backbone and horizontal cabling must meet or exceed the ANSI/TIA/EIA ANSI/EIA/TIA- 568-B.1, B.2 and B.3 standards. In addition, all backbone and horizontal cabling must be installed in a professional and workmanlike manner, and in accordance with the respective manufacturers specifications industry standards and practices (e.g., ANSI/EIA/TIA, BICSI, NEC, etc.), and all Federal, State and Local codes and ordinances. Backbone Cabling The backbone cabling extends from the Main Cross-Connect or network core to the Horizontal Cross-Connect, and it is typically constructed of multimode and/or singlemode fiber-optic cables. The backbone architecture and cabling should meet or exceed the ANSI/TIA/EIA-568- B.1 and B.2 specifications for backbone cabling. Table 2 (below) and Figure 5 (Page 10) summarize the key specifications of theses standards. Parameter Multimode Cable Singlemode Cable Wavelength (Meters): 850/1300nm 1310/1550nm Min. Bandwidth (MHz): 200/500 > 500 Max. Attenuation (db/km) Indoor: 3.5/1.5 Outdoor: 3.5/1.5 Indoor: 1.0/1.0 Outdoor: 0.5/0.5 Fiber Size 62.5/125µm 8.3/125µm Construction Type: Indoor = Tight Buffered Outdoor = Loose Tube Indoor = Tight Buffered Outdoor = Loose Tube Cable Min. Bending Radius: 16 x OD During Installation 10 x OD After Installation 15 x OD During Installation 10 x OD After Installation Strand Min. Bending Radius: 0.75-inches 0.75-inches Connector Loss: 0.75dB per connector 0.75dB per connector Splice Loss: 0.3dB per splice 0.3dB per splice Table 2: Minimum Backbone Optical Fiber Cabling Requirements PowerNet Network Communications Guidelines 9
Main Cross-connect B Intermediate Cross-connect C Horizontal Cross-connect A Horizontal Cross-connect Media Types A B C Multimode 62.5/125µm Optical Fiber 2000 m Maximum 2000 m Distance C Maximum 2000 m Distance B Maximum Multimode 50/125µm Optical Fiber 2000 m Maximum 2000 m Distance C Maximum 2000 m Distance B Maximum Singlemode Optical Fiber 3000 m Maximum 3000 m Distance C Maximum 3000 m Distance B Maximum Figure 5: Backbone Distances Horizontal Cabling The horizontal cabling extends from the Horizontal Cross-Connect located in the Telecommunications Room to the Telecommunications Outlet located in the Work Area, and is normally constructed of Unshielded Twisted-Pair (UTP) or Screened Twisted-Pair (ScTP) copper cables. All horizontal cabling should meet the ANSI/TIA/EIA-568-B.1 and B.2 specifications for Category 5e cables and connecting hardware. The following table and diagram summarize the minimum UTP and ScTP cabling requirements. Note In high-noise (EMI) environments, the horizontal cabling system must be constructed of multimode fiber-optic cable. Shielded Twisted-Pair (STP) cabling is also acceptable, however, there are no industry standards currently governing STP cabling installations. For STP installations, the telecommunications outlet jack, the patch panel jack assembly, and cable must all be shielded, and installed in strict compliance with the respective manufacturer s installation specifications. In addition, the STP cabling system must meet all of the performance requirements for UTP cabling as described in this document. 10 PowerNet Network Communications Guidelines
Gauge: Pair Count: Impedance: Jack/Patch Panel Configuration: 24 AWG, solid copper 4-pair 100Ω + 15Ω T568-B or T568-A Minimum Bend Radius: 4 times the cable diameter Table 3: Minimum Horizontal Cabling Requirements Hub or Switch Horizontal Cross-connect Telecommunications Outlet NetLink, EMINT or EPONI Cutler-Hammer Patch Cord < 5m UTP or ScTp Cable < 90m Equipment Cord < 5m Figure 6: Maximum UTP & ScTP Cable Lengths Horizontal Cable Test Requirements All cabling must be tested in accordance with the ANSI/TIA/EIA-568-B.1 specifications for either Permanent Link or Channel testing. Figures 7 and 8 (below and on Page 12) illustrate the setup/demarcation points for each type of test, whenever possible, full Channel Testing is recommended. Field Test Instrument Horizontal Cross-connect Telecommunications Outlet Field Test Instrument Patch Cord Equipment Cord UTP or SCTP Cable Begin Permanent Link End Permanent Link Figure 7: Permanent Link Test Configuration PowerNet Network Communications Guidelines 11
Field Test Instrument Horizontal Cross-connect Telecommunications Outlet Field Test Instrument Patch Cord Equipment Cord UTP or SCTP Cable Begin Channel End Channel Figure 8: Channel Link Test Configuration Each cable must meet or exceed the ANSI/TIA/EIA-568-B.1 Performance Specifications for the following parameters. The parameters shown are based channel testing. Frequency (MHz) P-to-P NEXT (db) PS NEXT (db) P-to-P ELFEXT (db) PS ELFEXT (db) Attenuation (db) 1.0 > 60.0 > 57.0 > 57.4 > 54.4 < 2.2 4.0 > 53.5 > 50.5 > 45.4 > 42.4 < 4.5 8.0 > 48.6 > 45.6 > 39.3 > 36.3 < 6.3 10.0 > 47.0 > 44.0 > 37.4 > 34.4 < 7.1 16.0 > 43.6 > 40.6 > 33.3 > 30.3 < 9.1 20.0 > 42.0 > 39.0 > 31.4 > 28.4 < 10.2 25.0 > 40.3 > 37.3 > 29.4 > 26.4 < 11.4 31.25 > 38.7 >35.7 > 27.5 > 24.5 < 12.9 62.5 > 33.6 > 30.6 > 21.5 > 18.5 < 18.6 100 > 30.1 > 27.1 > 17.4 > 14.5 < 24.0 Grounding Table 4: Horizontal Cable Performance Specifications Grounding of the network at Building Entrance Points, Telecommunications Rooms and Equipment Rooms shall comply with all applicable laws, regulations, standards, codes, industry practices, and the National Electric Code (NEC), Article 250 and Article 800, and the ANSI/TIA/EIA 607 - Commercial Building Grounding and Bonding for Telecommunications. Network Security Appropriate security measures need to be considered in order to minimize attacks to the PowerNet system and the network in general. Security breaches can include harmless types of access, such as information gathering or unauthorized viewing of data, to the more insidious types of access such as disclosure of confidential information, denial of service, and/or the destruction of data. If the network is connected to the Internet, then perimeter protection in the form of a Firewall and/or Proxy Server should be implemented. 12 PowerNet Network Communications Guidelines
Internal threats can be handled through Intrusion Detection technology, which monitors inappropriate activities on the network. At a minimum, PowerNet servers should be scanned for vulnerabilities using commercially available security scanning tools at the time PowerNet servers are placed in a production environment. Any significant vulnerability that is detected during the scans should be eliminated prior to user accessibility to the system. Additionally, a disaster recovery plan is strongly recommended in case of hardware or software failure, as well as virus attacks. Safety And Compliance Standards Network Standards Institute of Electrical and Electronics Engineers (IEEE) 802.3 ANSI/EIA/TIA-568-B.1, Commercial Building Telecommunications Cabling Standard, Part 1: General Requirements ANSI/EIA/TIA-568-B.2, Commercial Building Telecommunications Cabling Standard, Part 2: Balanced Twisted-Pair cabling Components ANSI/EIA/TIA-568-B.3, Commercial Building Telecommunications Cabling Standard, Optical Fiber Cabling Components Standard ANSI/EIA/TIA-569-A, Commercial Building Standard for Telecommunications Pathways and Spaces, TSBs and Addenda ANSI/TIA/EIA 607 - Commercial Building Grounding and Bonding for Telecommunications Building Industry Consulting Services International (BICSI), Telecommunications Cabling Installation Manual, 2 nd Edition Building Industry Consulting Services International (BICSI), Telecommunications Distribution Methods Manual, 9 th Edition National Fire Protection Association (NFPA 70) Publication: National Electrical Code (NEC) Technical Support For technical support on this topic or PowerNet in general, contact Power Management Application Support (PMAS) at 800-809-2772 Option 1 or outside the U.S. at 412-490-6714. PMAS can also be reached via e-mail at pmpapps@eaton.com. PowerNet Network Communications Guidelines 13
This page intentionally left blank. 14 PowerNet Network Communications Guidelines