Evaluation of clock synchronization accuracy of coexistent Real-Time Ethernet protocols

ISPCS 28 International IEEE Symposium on Precision Clock Synchronization for Measurement, Control and Communication Ann Arbor, Michigan, September 22 26, 28 Evaluation of clock synchronization accuracy of coexistent Real-Time Ethernet protocols P. Ferrari 1, A. Flammini 1, S. Rinaldi 1, and G. Gaderer 2 1 University of Brescia, DEA Via Branze 38 25123 Brescia Italy 2 Austrian Accademy of Sciences Viktor Kaplan Strasse 2 A-27 Wiener Neustadt Austria Email: paolo.ferrari@ing.unibs.it, Tel.+3933715445 Fax. +3933814 Abstract This paper investigates Real-Time Ethernet (RTE) protocols for industrial applications and their clock synchronization performance. In fact, if different RTE protocols coexist on the same network sharing the same infrastructure, clock synchronization capabilities can be affected. This paper introduces a simulation environment to evaluate coexistence of RTE protocols. The proposed tool is described and validated by comparison with measurement data taken from real networks. Finally, simulation results are presented: the behavior of Ethernet/IP with IEEE1588-PTP, when it is transported over an isochronous PROFINET IO network, is shown and some improvements to the PTP tracking algorithms to ease coexistence are proposed. Keywords Real-Time Ethernet, Industrial Automation, Simulation, Coexistence, Clock synchronization I. INTRODUCTION The IEC 61748-2 Standard [1] defines eleven Real-Time Ethernet (RTE) communication protocols, designed to replace traditional fieldbus technologies. This situation is due to the extreme market fragmentation, typical for the industrial automation world [2]. A common aspect of all RTE protocols is the ability to synchronize the clocks of network devices, distributing a single reference time to them. Accurate clock synchronization is required mainly for two reasons: to implement complex real-time applications at higher levels (e.g., motion control). Usually, top performance systems guarantee a stable cycle (e.g. period of 1 ms or less) with variations in the order of few microseconds or below. to maintain the communication at the communication level (e.g. time division multiple access, TDMA). Unfortunately, the method to achieve clock synchronization along the network is done in many different flavors. For instance, some of commercial RTE systems use IEEE1588, a.k.a Precision Time Protocol (PTP) [3], that can achieve synchronization accuracies down to the nanosecond range. Other RTE systems use proprietary clock synchronization protocols [4], but with rigid medium access policy. In practical applications such as plant automation engineering, a very important problem is the coexistence of two or more different RTE protocols in the same infrastructure. This is not only due to the fact, that such infrastructures are usually incrementally built rather than completely new engineered but also due to the economic reason that sharing the same network reduces installation costs. Today RTE protocols are in general designed under the condition to be able to ease the parallel use of ordinary TCP/IP traffic [5]. However, the coexistence between RTEs is yet not investigated. A deeper analysis shows, that successful coexistence among different RTEs, each one implementing its own strategy to achieve a real-time behavior, depends on several factors: real-time requirements at application level: the more relaxed these requirements are, the higher the chance of coexistence in the same network infrastructure: several RTE protocols require specialized hardware support thus, lack of an adequate infrastructure impedes the coexistence. order of installation: often coexistence is not reciprocal. For instance, RTE protocol 1 can be hosted in a system based on RTE protocol 2 but not vice-versa. Since recent experimental results [6] highlight synchronization problems in a particular test environment, a more general instrument is needed. The aim of this paper is to provide a novel simulation environment in order to evaluate coexistence of different RTE protocols. The main advantage is the possibility to test coexistence problems before realizing real networks. The proposed simulation models have been created for two of the major RTE protocols, PROFINET and Ethernet/IP. The validation of the simulation models is done using real data obtained from the distributed instrument described in [6,7]. In order to show the usability of the proposed environment, the study of the coexistence of a system where Ethernet/IP is transported in a PROFINET IO network has been done. Finally, the simulation results have been used to extract important indications to increase the coexistence level. II. SIMULATION MODELS AND VALIDATION The simulation environment consists of a collection of models of RTE devices that can be interconnected to implement the desired network. The simulation platform is OMNeT++ [8] which has the advantage of an open software approach and is freely available under the Academic Public License. OMNeT++ allows models to have arbitrary complexity in order to simulate things that each RTE protocol depends on and things that each protocol may distort. 978-1-4244-2275-3/8/$25. 28 IEEE 87

However, in this paper only protocols behavior has been modelized while physical layer of Ethernet (e.g. symbols transmissions and Rx-Tx PLL locking) has not been taken into account. In fact, the influence of the physical layer is very small if compared to effect in terms of disturbance caused by coexisting RTE protocols. Up to now, the developed models include Ethernet/IP and PROFINET IO devices with clock synchronization capabilities. These two protocols have been chosen for their popularity in the European and US market. A. Simulation Environment The first protocol under consideration for the simulation environment, Ethernet/IP, is based on the CIP (Common Industrial Protocol) which can be inside TCP and UDP packets. For high-performance real-time applications, the CIPsync profile must be used; such a profile uses the IEEE1588 PTP to achieve clock synchronization. Clock synchronization in IEEE1588 is done for the Ethernet Profile via UDP multicasts. Therefore, best performance can be obtained only if the network infrastructure supports: fullduplex, IGMP snooping for multicast handling, QoS management and PTP. In particular, the real-time behavior of an Ethernet/IP system with CIPsync directly depends on clock synchronization accuracy, since sampling and actuation is carried out in a time-triggered fashion, and not on the basis of the arrival of communication packet (which suffers from an variable delay in switched Ethernet). The proposed model of an Ethernet/IP device with CIPsync is derived from the PTP model in [1] to which a new application level, called CIPApp, has been added in parallel with PTP. CIP real-time application data is generated by the CIPApp in a node and consumed by the CIPApp in another node. The generation rate, and therefore the network traffic, can be configured and generation instants are based on the PTP clock. The second part of this study is the PROFINET IO (PNIO) protocol. It has three classes of performance: Conformance Class C uses a TDMA scheme to achieve the best real-time performance. In fact, in PROFINET IO the real-time data exchange is cyclic and the network infrastructure reserve a certain time of the cycle to transfer only PROFINET IO real time messages (called Class 2 and Class 3 messages). As a consequence the PNIO Class 2/3 traffic always have a free channel to be transmitted on, while other frames (e.g. TCP packets) wait inside the switches for their turn. Clearly, the infrastructure components are synchronized in order to change modality (PNIO Class 2/3 rest of the traffic) in a coordinated way [9]. Usually, Class C devices embed a switch that is able discriminating between PNIO messages classes, fulfilling all the previous requirements. The proposed model of a PROFINET IO Class C device has been implemented modifying the model of a cut-through switch, as described in the next section. A PROFINET layer has been connected (internally) to the switch port and handles all the PROFINET data exchange. Moreover, the cut-through switch architecture has been integrated with the code for handling the PTCP (Precision Transparent Clock Protocol) which is used by PROFINET IO for synchronization. The basic functions to obtain simultaneous change of behavior in the infrastructure (the so called IRTflex modality) have been also implemented. B. Details of the PROFINET IO Switch Model This section describes the implementation of the PNIO switch model, since it is a critical components and the main responsible for coexistence issues with other RTE protocols. For the analysis of the jitter of the propagation delay in a network with coexisting RTE protocols the implementation of an high accuracy model (i.e. at hardware level, like a model of the ASIC - ERTEC4 - used to implement PROFINET IO Class C) is not necessary. In fact, these errors are in the order of some μs [6], and a behavioral simulation model is enough. The PNIO switch model developed in this work is shown in Fig. 1. The switch is composed by a set of sub-models, each of them in charge of a different task. The PNIOClock model is mainly involved in the time management. It gives the time reference to the other blocks, (like PNIOApp or the MACRelayUnit) in order to synchronize the application level or the transmission level (the different phase defined by PNIO protocol) of different nodes. The local time of each node is synchronized to the time of the PNIO Master clock by means of the PTCP messages exchanged on the network during the phase reserved to Class 2/3 traffic. The EthMAC block provides to system the Ethernet Medium Access Control (MAC) level and supports also some functionality required by the PTCP synchronization, like the low level timestamping and the estimation of the bridge propagation delay. The PNIOApp level supports the management of Class 2/3 traffic; it produces and consumes Class 2/3 data, following the simulator configuration parameters, in order to simulate a PNIO device. Moreover this layer implements a simplified version of the synchronization stack. The most important block of the switch is the MACRelayUnit. In fact, this model manages the receiving of PNIOClock MACRelayUnit EthMAC_ EthMAC_1 EthMAC_n Fig. 1. Model of PNIO switch. PNIOApp 88

the frames from the port and the routing to right direction. Its structure differs from standard switch logic because its behavior depends by the received frame (real-time or not) and by the PNIO phase in which the data has been received. In particular, there are three phases instead of two: Open phase (also called Green phase ): the PNIO switch works as a cut-through switch when it receives any frames. So, it reads the first 64 byte of the incoming frame and then sends it to the right port. From real measurements, T CFD, that is frame retransmission delay in cut-through working mode, is about 5.8 μs. Transition phase: during this extra phase, which lies between Green and Red phase, the switch stores all the frames it receives in a internal queue and then retransmits them on the right port (store and forward working mode). The retransmission is started only if the end of the outgoing frame does not overlap the Red phase. In the transition phase the retransmission delay introduced by the switch depends on the length of the received frame. Hence, the T SFD, that is frame retransmission delay in store and forward working mode, is defined as the frame bit length multiplied by the bit time. Class 2/3 reserved phase (also known as Red phase ): the PNIO Class 2/3 frames, like synchronization messages and real-time data, are produced and routed only in this phase. The switch in this phase works in cut-through working for Class 2/3 messages, while for other kind of frames (including TCP) the switch operates differently - if the destination of the frame is a port where a Class C device is attached to, the frames is stored in a queue and forwarded only at the end of the Red phase. In this case, the frames have to wait in the queue at least for T Red (i.e. the length of Red phase); - if the destination of the frame is a port, where no Class C device are attached to, the frame is forwarded in the cut-through modality. C. Validation against Real Measurement Data Both of the proposed models have been validated independently, since the RTE protocols they are based on are independent. In particular, simulation results are similar to the measure reported in [5] and [6]. However, this paper is focused on testing coexistence of two RTE protocols on the same network, hence a further validation is needed to ensure that a mixed network can be successfully simulated. The network architecture used for the validation of the models is depicted in Fig 2. Two Ethernet/IP nodes (E_IP and E_IP1) are connected passing through two PROFINET IO devices (PNIO_ and PNIO_1). The Frame Propagation Delay, T FPD, between node E_IP and E_IP1 is logged by the simulator, whereas the traffic load between the same nodes can be varied from to 25 Mbit/s. The PNIO Class C components have a communication cycle time of 1ms and a Red phase duration of 3.5 μs. Values obtained from the simulation models (traffic load = Mbit/s) are compared in Fig. 3 with data from a real network with commercial components [6]. The shapes of the T FPD distribution are close each other, fact that confirms the applicability of the implemented model. Since no traffic is present on the network, the majority of the samples are concentrated around to most probable values (corresponding to the PNIO Green and transition phase, i.e. the flat zones of the time graphs). Other values of T FPD have a low frequency and they correspond to frames that try to travel along the network during the PROFINET reserved phase; frames are randomly delayed by PROFINET infrastructure. The maximum (peak) T FPD for the simulated system is 57 μs while in the real case is 66 μs. This difference can be explained considering that in the real network an idle line condition is impossible to reach; PROFINET switches (like any other managed switch) always generate management packets. III. RESULTS As already highlighted in Fig. 3, the transport of Ethernet/IP traffic over a PROFINET IO Class C increases the variance of T FPD, making the realization of a CIPsync profile based application very difficult. TFPD (μs) TFPD (μs) Fig. 2. Network architecture used for validation of the proposed models. 1 5 45 55 65 75 85 Time (s) 1 5 45 55 65 75 85 Time (s) Frequency Frequency 1% 5% % 1% 5% 14 25 36 46 57 68 T FPD (μs) % 14 25 36 46 57 68 T FPD (μs) Fig. 3. Frame Propagation Delay, T FPD, between nodes E_IP1 and E_IP2 for a traffic load of Mbit/s. Comparison between real measurement (top graphs) and simulated data (bottom). 89

In order to show this phenomenon, the network depicted in Fig. 4 has been simulated. As shown, four Ethernet/IP nodes (E_IP-1-2-3) connected to two PNIO nodes (PNIO_ and PNIO_1). Node E_IP is also a PTP master clock which synchronizes the nodes E_IP1-2-3 with a synchronization interval of 2 s. The PROFINET Class C components again work with a communication cycle time of 1ms and a Red phase duration of 3.5 μs. The simulation results about the clock offset of the Ethernet/IP nodes with no traffic load is shown in Fig. 5. It should be noted that the high jitter introduced by the PNIO infrastructure on T FPD decreases the synchronization accuracy. Fig. 6 shows the average clock offset (and the standard deviation) of node E_IP3 when the network traffic load between Ethernet/IP nodes varies between and 15 Mbit/s. Generally, the simulation results shown can be used to predict a system behavior; however, the subsequent challenge is to improve coexistence. Hence, the goal is to employ the previous simulation results to enhance the coexistence grade between Ethernet/IP and PROFINET, or in other words, reduce the offset variability experienced by Ethernet/IP nodes. One of the possible solutions is to use the knowledge about the distribution of the T FPD, given by the simulations, to implement an efficient filtering algorithm which rejects PTP synchronization messages which are obviously delayed. In particular the algorithm decides to accept or discard synchronization messages on the basis of the jitter distribution and reasonability of the message timestamps. For each incoming synchronization message, the master to slave delay, T MSD, and the one-way delay, T OWD, are tested to be inside an acceptance window centered in the most probable T FPD value. In case of positive match the message is used to correct the PTP clock tracking, otherwise it is discarded and a counter is increased. If the counter value exceeds an imposed threshold before a Sync is accepted, then the filtering algorithm is disabled for a short time. However, as this algorithm does not converge necessarily, for instance at power-up, and special handling of these cases must be introduced. Values for these thresholds can be experimental determined with an simulation setup. A sample curve of the standard deviation of the clock offset as a function of the acceptance window size in the compensation algorithm, is reported in Fig. 7. It is clear that, with the help of the simulation tool, the network engineer can easily choose the window size that minimizes the standard deviation of the clock offset. The simulation results regarding the case where the filtering algorithm is enabled are presented in Fig. 8. The acceptance windows size has been set to 1.5μs. Both standard deviation and average of the clock offset of node E_IP3 are 4 Offset (μs) 2-2 -4 5 1 15 Bandwidth (Mbit/s) Fig. 4. Network architecture used for the study of the synchronization accuracy of two coexistent RTE protocols (Ethernet/IP and PROFINET IO). Fig. 6. Average value and standard deviation of the clock offset of E_IP3 with variable traffic load among Ethernet/IP nodes (2 samples). Offset [us] 7 4 1 E_IP1 E_IP2 E_IP3 Std. dev. (μs) 4 3 2 1-2 53 58 63 68 73 Time [s] 2 4 6 8 1 12 14 16 Acceptance window size (μs) Fig. 5. Clock offset of nodes E_IP1-2-3 with respect to the grandmaster clock E_IP when the considered network with no traffic load ( Mbit/s). The high variation depends on PROFINET IO infrastructure only. Fig. 7. Standard deviation of the clock offset of nodes E_IP1 with respect to the size of the acceptance window of the compensation algorithm (network with no traffic load - Mbit/s). (2 samples). 9

Offset (ns) 18 12 6-6 -12 5 1 15 Bandwidth (Mbit/s) Fig. 8. Average value and standard deviation of the clock offset of E_IP3 with variable traffic load among Ethernet/IP nodes after the compensation algorithm (with acceptance window size of 1.5 μs ) has been activated (2 samples). [7] Ferrari P., Flammini A., Marioli. D., Taroni A., A Distributed Instrument for Performance Analysis of Real-Time Ethernet Networks, IEEE Trans. on Industrial Informatics, Vol.4, No.1, Feb. 28, pp. 16-25. [8] OMNeT. OMNet++ Discrete Event Simulation System. www.omnetpp.org. [9] Jasperneite J, Shehab K., Weber K., Enhancements to the Time Synchronization Standard IEEE1588 for a system of Cascaded Bridges, in Proc. of IEEE WFCS24, Vienna, 24, pp.239-244. [1] Praus F., Granzer W., Gaderer G., and Sauter T., A Simulation Framework for Fault-Tolerant Clock Synchronization in Industrial Automation Networks, in ETFA'7 The 12th International Conference on Emerging Technologies and Factory Automation, pp. 1465-1473, Patras, Greece, September 27 reducing, if compared with the ones of Fig. 6; the synchronization accuracy increases from tens of microseconds to tens of nanoseconds. IV. CONCLUSION The study of the clock synchronization accuracy of different RTE systems that share the same network is important to evaluate real-time properties. In this paper a new simulation tool is proposed. Such a tool is based on models which properties are derived from measurements on real network. Simulation of scenarios with coexisting RTE protocols have been done and results show a general decrease of the clock synchronization performance. In conclusion, it is worth mentioning that the coexistence between different RTE protocols can be improved by means of compensation algorithms or fine tuning of network parameters but, excluding rare cases, a seamless coexistence (i.e. with no side effects) cannot be obtained. The key point is to chose the right tradeoff between performance reduction and benefit of sharing the same network infrastructure; the proposed simulation environment helps to make this choice. REFERENCES [1] IEC 61784-2 Ed. 1, Industrial communication networks - Profiles - Part 2: Additional fieldbus profiles for real-time networks based on ISO/IEC 882-3, IEC, 27 [2] Richard Zurawski, The Industrial Communication Technology Handbook, Taylor & Francis, 25. [3] IEEE. Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems. IEEE, New York, 22. ANSI/IEEE Std 1588-22. [4] Kopetz H., Krüger A., Millinger D. et. Al. A, Synchronization Strategy for a Time-Triggered Multicluster Real-Time System, in Proc. of the 14th IEEE Symposium on Reliable Distributed Systems, Bad Neuenahr, Germany, 13--15 September 1995. [5] Pereira N., Tovar E., and Pinho L. M., INDEPTH: Timeliness assessment of Ethernet/IP-based system, in Modeling, Analysis, and Simulation of Computer and Telecommunications Systems (MASCOTS24), pp 192-21, October 24. [6] Ferrari P., Flammini A., Marioli D., Rinaldi S., Taroni A., Testing coexistence of different RTE protocols in the same network, in Proc. of IEEE WFCS28, Dresden, 28, 91