e-vlbi Development at Haystack Observatory Alan R. Whitney MIT Haystack Observatory

e-vlbi Development at Haystack Observatory Alan R. Whitney MIT Haystack Observatory

e-vlbi program overview Mark 5 VLBI data system 1 Gbps demonstration e-vlbi experiment National and international e-vlbi experiments Development of special e-vlbi protocols New UltraLight project (proposed) New DRAGON project (proposed) VSI-E development

Mark 5A VLBI Disk-Based Data System 1 Gbps continuous recording/playback to/from set of 8 inexpensive (ATA) disks Based primarily on COTS components Two removable 8-pack disk modules in single 5U chassis With currently available 200GB disks capacity of single 8-pack 1.6TB; expected to increase to 2.5TB by late 2003 with 320GB disks Network connection for real-time and quasi-real-time e-vlbi operations Inexpensive: ~$16K fully assembled and tested ~30 Mark 5 systems now installed at stations and correlators; expect ~50 by end 2003 VSI-compatible version (Mark 5B) to ready early 2004

Mark 5 e-vlbi Connectivity Mark 5 supports a triangle of connectivity for e-vlbi requirements Disk array Data Port PCI bus/network (e-vlbi) (64bit/66MHz) Mark 5 can support several possible e-vlbi modes: e-vlbi data buffer (first to Disc Array, then to Network); vice versa Direct e-vlbi (Data Port directly to Network); vice versa Dual Gigabit Ethernet connections will be necessary to support for 1024 Mbps; new motherboards are being examined More information at www.haystack.edu/mark5

Bossnet 1 Gbps e-vlbi demonstration experiment Westford Haystack (correlator) Future Initial experiment USNO (correlator) NASA/GSFC

Details of path from Haystack to GGAO: Part 1 WS/ Mark 5 A1 WS/ Mark 5 B1 WS/ Mark 5 B3 Westford Antenna A2 Summit 1i Haystack Correlator 28-1A B2 28-1B B4 1 8 A3 14 13 Summit 5i B5 Dedicated - no scheduling Dedicated - no scheduling 16 15 A4 B6 Congress St Boston Millstone RPE Bldg (Lorraine) C3 C1 Summit 5i C2 WDM Schedule to be coordinated with LL LL Plan install by 1 Jan 02 (Shared with LL on scheduled basis) Cisco 1500 To be upgraded from 2 to 3λ ϑαν 2002 WDM Schedule to be coordinated with LL Cisco 1500 E1 E2 To be provided by Haystack E5 GlowNet D3 Lincoln Lab (Lorraine/Steve) E3 Summit 5i E4 To be replaced by Juniper M20 Jan 02 with OC-48 to Bossnet Test WS E6 Summit 5i E7 MIT Campus 1310 E8 Other users (no control) To be updated to OC48 Jan02 WDM F1 1510 Amp D2 GlowNet D1 Bossnet(Steve/Jim) Amp Figure 1: e-vlbi Path - Haystack to ISI-E Scheduling on Bossnet calendar F2 Amp ISI-E evlbi011.drw 20 Dec 01

Details of path from Haystack to GGAO: Part 2 ISI-E (Tom) Max @ ISI-E (Jerry) Bossnet Scheduling on Bossnet calendar Tuner WDM G1 Plan to update to OC-48 ~Jan 02 Plan mid Jan 02 G2 Transponder G3 G4 OC-48 Test WS? G6 Summit 5i G5 G7 G9 Need details G8 Juniper M40 G10 G12 G11 Switch H1 H2 Backup plan OC-48 Borrowed OC48 interface - may not be able to keep Test WS? Juniper M160 H3 H4 H5 OC-48 Need details No scheduling; sufficient capacity for e-vlbi Max @ ISI-E (Jerry) H7 Juniper M160 Need details UMCP (Jerry or Dan) Test WS? J3 J2 J1 Cisco H8 3508 J4 Switch H9 No jumbo frames Preferred e-vlbi path, but may not be in place. Pat working with Jerry and Dan. Scheduling necessary if this path used Other users ( no control) J5 LuxN WDM J6 H6 WDM Will try to schedule either K2 or K3 exclusively for e-vlbi experiment K1 GSFC/Bldg 28 (Pat) GGAO/Bldg 202 (Pat) GGAO Antenna Trailer (Bill) K2 K3 Mac with Yellowdog Linux; need details K4 Summit 5i Test WS K5 Other users (no control) 1310 K6 H7 K7 Dedicated - no scheduling necessary L1 1600 ft of fiber to be installed Jan 02 Summit 5i L2 On order L3 Dell Linux currently at Haystack will also be available here WS/ Mark 5 L4 MAC on order due Jan-Feb 02 Eventually replaced with Mark 5 unit Figure 2: e-vlbi Path - ISI-E to GSFC/GGAO evlbi011.drw 20 Dec 01

Performance test results Haystack/GGAO Average sustained rate >900 Mbps over 10 hours

Westford-GGAO e-vlbi results First near-real-time e-vlbi experiment conducted on 6 Oct 02 Recorded data at 1152 Mbps on Westford-GGAO baseline GGAO disk-to-disk transfer at average 788 Mbps transfer rate Immediate correlation on Haystack Mark 4 correlator Nominal fringes observed Direct data transfer experiment conducted on 24 Oct 02 Direct transfer of GGAO data to disk at Haystack at 256 Mbps Immediate correlation with Westford data Nominal fringes observed Next step full real-time e-vlbi Mark 5 system is capable of transmitting in real-time But, still need additional work on correlator software to synchronize correlator operation to real-time Hope to conduct first experiment in early 2003 Conclusion e-vlbi at near Gbps speeds over ordinary shared networks is possible but still difficult Full report at www.haystack.edu/e-vlbi

International e-vlbi experiments Westford, MA to Kashima, Japan - experiments in Oct 02 and Mar 03 Files exchanged over Abilene/GEMnet networks Nominal speed expected to be ~20 Mbps; best achieved so far ~11 Mbps Correlation on Mark 4 correlator at Haystack and PC Software correlator at Kashima; nominal fringes obtained Further experiments are scheduled; network tuning is in progress Kauai, Hawaii to Wettzell, Germany (in progress) Daily experiments of ~100GB are ideal candidate for early e-vlbi Data will be transferred to Haystack Observatory for processing (OC-3 speeds are possible) Network links are now being brought up

Antenna Kokee Park NASA Comm ATM Site A Terminal Equipment (TE) Requirements?? ~8 km FO cable 12 pairs? TM ATM Makaha Ridge FO Comm ATM FO Gig E TE? MM Link STM-1 155 Mb/s UH Router ATM Seattle Westin Bldg DREN ATM Pacific NW Giga pop Abilene Router Pacific Wave Oahu 155 Mb/s OC48 2.5 Gb/s HIC OC3 155 Mb/s DREN Abilene HIC Router ATM UofH Manoa Keller Hall SPAWAR Bldg 992 OC3 GWU OC3 155 Mb/s DCGW Microwave Link DREN OC3 155 Mb/s UMd CLPK MAX Fiber Ring OC48 2.5 Gb/s DCNE Qwest DARPA ARLG ~2.4 km ISI-E PROPOSED DARK FIBER LEASE OC48 2.5 Gb/s DC Area PMRF Barking Sands OC48 3xDS3 135 Mb/s BOSSNET/ GLOWNET USNO Comm Mux, ATM SW HIC Router ATM 3xDS3 135 Mb/s Experimental Link TE? Haystack Obs VLBI Correlators Gb/s @ 80 GHz MM Link

Further extensions to national and global community Possibilities for international connections Surfnet U.S. to Europe at 2.5 Gbps TransPAC U.S. to Japan at 655 Mbps (upgrade to 1.2 Gbps planned) GEMnet currently ~20 Mbps, planning to upgrade to 2.2 Gbps APAN U.S. to Japan at 655 Mbps AMPATH possible connections to telescopes in Chile and Brazil A sample of others under construction TEIN Paris to Seoul EUMEDCONNECT Europe to Mediterranean NeDAP Europe to Russia ALIS Europe to Latin and South America One of the most interesting! IEEAF Europe/U.S. link at 10 Gbps! Donated by Tyco, Inc. Dedicated-lambda networks are being investigated as possibility

New IP Protocols for e-vlbi Based on observed usage statistics of networks such as Abilene, it is clear there is much unused capacity New protocols are being developed to utilize networks in background mode for applications such as e-vlbi Take advantage of special characteristics of e-vlbi data Will scavenge and use secondary bandwidth Will give priority to normal users Requires a new end-point adaptive strategy Work being carried out by MIT Haystack Observatory in collaboration with MIT Laboratory for Computer Science and MIT Lincoln Laboratory Funded 3-year program; will demonstrate e-vlbi connections both nationally and internationally Dr. David Lapsley has joined Haystack staff to lead this effort

Typical bit-rate statistics on Abilene network 1.0 0.1 Usage >20Mbps less than 1% of the time 0.01 0.001 100 500 Mbps Conclusion: Average network usage is only a few % of capacity

Typical distribution of heavy traffic on Abilene 1.0 0.9 0.8 0.7 <10% of bulk transfers exceed ~100 secs 200 400 1000 secs Conclusion: Heavy usage of network tends to occur in bursts of <2 minutes

UltraLight (An Ultra-scale Optical Network Laboratory for Next Generation Science) Packet-switched and circuit-switched hybrid experiment research network with transcontinental 10Gbps wavelengths on National Light Rail Collaboration of Caltech, MIT Haystack, U. of Florida, Florida IU, U. Mich, SLAC, Fermilab, CERN and others; commercial partner Cisco Systems Partner projects TransLight, Netherlight, ULlight, AMPATH, CA*Net4 Flagship applications High-energy physics e-vlbi High-resolution near-real-time medical imaging New techniques to be explored End-to-end monitoring agents to determine how to best manage network data flows Dynamic traffic routing Dynamic scheduling of additional wavelengths Tunneling protocols to set up sub-paths with guaranteed BW Key part of proposal is to develop plan for connecting U.S. antennas Proposed 5-year project, ~$10M, with several M$ contribution from industry

DRAGON (Dynamic Resource Allocation via GMPLS Optical Networks) Collaboration with Univ. of Maryland, Mid-Atlantic Crossroads (MAX), Information Sciences Institute (USC) and NASA/GSFC Develop a Generalized MultiProtocol Label Switching (GMPLS) network to provide deterministic network resources at the packet, wavelength, and fiber cross-connect levels Will develop a set of API s for application-level use of GMPLS Industry partner Movaz Networks will provide pre-production GMPLSenabled wavelength-selective switches (MEM s based switching fabric) Proposed 4-year project

VSI-E VSI-H defines input and output interfaces of a VLBI data system, but specifically excludes details of media or data format However, e-vlbi intercompatibility among heterogeneous VLBI data systems is highly desirable Consequently, the VSI technical development committee is now turning its attention to this problem Goal is define a common e-vlbi data protocol and format, called VSI-E First rough draft has been created and distributed to form a basis for further discussion It now appears that some variation of RTP protocol may be most suitable for VSI-E A rough draft RTP protocol for e-vlbi will be presented by David Lapsley; we hope to have further discussion at this meeting Goal: Complete VSI-E specification by end 2003!

622 Mbps +10 Gbps λ Transoceanic donations to IEEAF (in red)

Northern Europe donations to IEEAF (in red)

AMPATH: Research and Education Network and International Exchange Point for the Americas Launched in March 2000 as a project led by Florida International University (FIU), with industry support from Global Crossing (GX), Cisco Systems, Lucent Technologies, Juniper Networks and Terremark Worldwide Enables wide-bandwidth digital communications between the Abilene network and 10 National Research and Education Networks (NRNs) in South and Central America, the Caribbean and Mexico Provides connectivity to US research programs in the region AMPATH is a project of FIU and the National Science Foundation s Advanced Networking Infrastructure & Research (ANIR) Division Note: VLBI telescopes currently in Chile and Brazil