Globally Synchronized Time via Datacenter Networks

Transcription

1 Globally Synchronized Time via Datacenter Networks Vishal Shrivastav, Cornell University Joint with Ki Suh Lee, Han Wang, and Hakim Weatherspoon COSENERS 2016

2 How can we scalable synchronize clocks with high precision? Scalable Entire datacenter High precision bounded precision; e.g. no clock differs by more than hundreds of nanoseconds Capability essential for network and distributed applications Networks One-way delay, consistent updates, etc Distributed systems consensus, snapshots, etc 2

3 Problem: Clock synchronization is non-trivial Precision: difference between any two clocks Typical clock offset synchronization Offset GPS Atomic clock t 0, t 1, t 2, t 3 t 0, t 1, t 2 Time master t 0 t 0, t 1 3

4 Problem: Clock synchronization is non-trivial Precision: difference between any two clocks Typical clock offset synchronization Offset = ((t 1 t 0 ) (t 3 t 2 ))/2 [d+offset] [d-offset] GPS Atomic clock t 3 t 2 d = delay t 0 t 1 Time master 4

5 Problem: Clock synchronization is non-trivial Precision: difference between any two clocks Problems affecting precision Oscillator skew (i.e. frequency of clocks differ) Reading remote clocks: timestamps, network stack, network jitter Resynchronization frequency GPS Atomic clock t 3 t 2 Time master t 0 t 1 5

6 Problem: Clock synchronization is non-trivial Precision: difference between any two clocks Synchronization Protocols Precision Scalability Overhead Extra Hardware NTP us Good Moderate None PTP sub-us Good Moderate PTP-enabled devices GPS ns Bad None Timing signal receivers, cables GPS Atomic clock t 3 t 2 Time master t 0 t 1 6

7 Outline Introduction Design Evaluation Conclusion 7

8 Use the PHY to synchronize clocks Application Transport Network Data Link Physical Protocol in the PHY Each physically link is already synchronized! No protocol stack overhead No network overhead Scalable: peer-to-peer and decentralized GPS Atomic clock t 3 t 2 Time master t 0 t 1 8

9 DTP 10 Gigabit Ethernet Idle Characters (/I/) and Control blocks (/E/) Packet i Packet i+1 Packet i+2 Application Transport Network Data Link Physical Standard requires at least 12 idle characters /I/ between pkts i.e. At least one 64-bit Control Block /E/ between pkts Idle characters / control blocks sent even if no packets to send DTP overwrites idle characters (control block) to send protocol messages DTP does not effect standard at all 9

10 DTP 10 Gigabit Ethernet Idle Characters (/I/) and Control blocks (/E/) Packet i Packet i+1 Packet i+2 Media Access Con trol (MAC) Reconciliation Sublayer (RS) Application Transport Network Data Link Physical TX 32bit XGMII MHz RX 32bit Physical Coding Sublayer (PCS) TX 16bit XSBI MHz RX 16bit Physical Medium Attachment (PMA) Physical Medium Depende nt (PMD) 10

11 Datacenter Time Protocol (DTP) Precise and bounded synchronization 4 oscillator ticks (25ns) bounded peer-wise synchronization 150ns precision synchronization for an entire datacenter No clock differs by more than 150ns Free No network traffic: Use the PHY! GPS Atomic clock t 3 t 2 Time master t 0 t 1 11

12 Outline Introduction Design Evaluation Conclusion 12

13 Evaluation Compare measured precision of DTP and PTP Measurement and observation period was two days PTP: Compare precision between Timeserver and Servers MellanoxNIC (hardware), IBM G8264 Switch, Timekeeper server DTP: Compare precision between leaf servers and switches Terasic DE5 FPGA-based development Net board S0 S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 IBM Switch Timeserver DTP-NIC Mellanox 13

14 PTP Idle Network (No Network Traffic) Clocks differ by a few hundred nanoseconds 14

15 PTP Medium Loaded Network (4Gbps Traffic) Clocks differ by tens of microseconds 15

16 PTP Heavily Loaded Network (9Gbps Traffic) Clocks differ by hundreds of microseconds 16

17 DTP Heavily Loaded Network (9Gbps Traffic) Clocks never differed by more than 4 clock ticks, 25ns Bounded Precision 17

18 Outline Introduction Design Evaluation Conclusion 18

19 Conclusion DTP provides Bounded precision: 4 oscillator ticks (25ns) Scalability: 150ns for entire datacenter Free No network traffic: Use the PHY! Needs hardware modifications (just like PTP) 19

20 Thank you 20