Improving the Accuracy of Software-based Clock Synchronization and Encountering Interrupt Coalescence Danube University Krems. The University for Continuing Education. by Aneeq Mahmood Authors: Aneeq Mahmood, Thilo Sauter 1
Outline Motivation and introduction to problem Setup of the proposed solution Experimental validation Impact of interrupt coalescence Concluding remarks 2
Motivation Improving the timing accuracy and precision - Stable oscillators and timers - Timestamping techniques - Control techniques Software timestamping based networks - Accuracy limited to microsecond range - Transition to hardware timestamping - No proper guidelines to implement accurate timestamping Goal: Interrupt-driven sofwtare timestamping for synchronization - A setup including an external timestamper - Setup can also be used for calibrating system delays - Improved precision by providing pseudo-hardware timestamping 3
Proposed Setup 4
Modeling of error sources-software timestamping Measurement-based approach to model error from software timestamps www.donau-uni.ac.at PCIe card Timestamping the hardware interrupt itself CPU Timestamping error=t b -t a North / South bridge PCI bus WLAN card PCIe link XO t b Counter t a PCI interrupt line PCIe card 5
Modeling of error sources-software timestamping (2) Measurement of timestamping error - 2.4 GHz Axiomtek industrial PC - Atheros WLM54G chipsets - Syn1588 PCIe card from Oregano Systems as clock source 450 700 400 600 350 300 500 Frequency 250 200 Frequency 400 300 150 200 100 50 100 0 2 2.5 3 3.5 4 4.5 5 5.5 6 6.5 0 4.5 5 5.5 6 6.5 7 7.5 8 8.5 9 SW delays for the transmission side (us) SW delay for reception (us) 6
Measurement setup for PTP synchronization A wireless setup is created to avoid impact of line delays - Atheros chipset - Ath5k driver for Linux Trial run carried out using a sync1588 PCIe NIC from Oregano Built-in HW timestamper inside with timestamp jitter = 12 ns 1 PPS signal to determine synchronization performance 7
HARDWARE KERNEL SPACE Sync Packet Sync Packet USER SPACE Measurement setup for PTP synchronization(2) www.donau-uni.ac.at AP PTP St ack Timestamp Collector PTP Stack Clock Servo Node Timestamps UDP/ IP UDP/ IP Timestamps Clock Update PCIe NIC Driver WLAN Driver WLAN Driver PCIe NIC Driver Hardware Clock MAC (Firmware) MAC (Firmware) Hardware Clock 1 PPS Interrupt PHY PHY Interrupt 1 PPS Oscilloscope 8
1 PPS comparison 100 90 80 70 60 Mean 7 ns Frequency 50 40 σ 78 ns 30 20 10 0-300 -200-100 0 100 200 300 Clock o, set (ns) 9
1 PPS comparison (2) Clock o, set (7s) 2.5 2 1.5 1 0.5 0 Software Only Current Approach Mean 400 ns 7 ns σ 910 ns 78 ns -0.5-1 -1.5 Software only Cur rent approach 10
1 PPS comparison (3) Clock o, set (ns) 200 150 100 50 0-50 -100 Current Approach HW Solution Mean 7 ns 8.3 ns σ 78 ns 15 ns -150-200 Current approach With Hardware support 11
Limitations of the solution Role of the timestamping platform - Low cost - Low energy consumption Matching timestamp to the interrupt signal - maximum number of interrupts (which can be handled by the CPU) per second) > maximum packet arrival or departure rate - For 2 Ghz industrial computers, interrupt handling delay are 4µs and 7µs - System capable of dealing with 100BASE-TX Ethernet or IEEE802.11n Dead end for high speed networks - Not feasible for 1000BASE-X Ethernet or optical network - Interrupt coalescence (IC) limits the number of interrupts after packet arrivals or departures to reduce the CPU load 12
Dealing with IC IC helps CPUs but affects one-way delay measurements Information about IC from vendors - Maximum interrupt rate - Maximum buffered packets before raising interrupt - Time between receiving a packet and subsequent raise of the interrupt (rx-usecs) Differences at master and slave side - Using ethtool at the slave side, rx-usecs = 0 - Master side unknown Adding information in the Announce messages 13
One-way delay estimate 150 rx-usecs = 0 rx-usecs = 50 rx-usecs = 100 100 Frequency 50 0 80 90 100 110 120 130 140 150 160 170 180 190 One way delay estimate (7s) 14
Concluding remarks A solution proposed to improve the performance for systems using software timestamping Design can also be used to calibrate interrupt handling delays of the systems Accuracy similar to hardware timestamping based solutions Precision depends upon delay between the I/O interface and CPU Biggest limitation is high speed networks and IC Adding information in the PTP meesages can help dealing with IC related uncertainities 15
Danube University Krems. The University for Continuing Education. Thank You Zentrum für Integrierte Sensorsysteme Donau-Universität Krems www.donau-uni.ac.at/ziss Aneeq.mahmood@donau-uni.ac.at