ISTITUTO DI RADIOASTRONOMIA DiFX at IRA: cluster, network and dbbc testing
vlbi-mgr DiFX software server infiniband infiniband infiniband tank-mc tank-nt tank-srt tank-?? Cluster architecture Dedicated storage node for each antenna
OS CPU RAM Motherboard 3WARE PCI- Express GT/s RAID Card Scientific Linux 6.4 2x XEON E5260 @ 2.4 GHz 12 GB X8DTH-IF 7x16 6.4 1x 9750DR Storage nodes architecture
Cluster view Infiniband connection 10 Gbit network 24 disks bay Now 72 TB available
2200 2000 1800 1600 correlation time (sec) 1400 1200 1000 800 600 400 200 0 2 antennas 3 antennas 4 antennas Correlation benchmark 300 GB per antenna / 5 hrs rec 30 cores 24 min to correlate 3 antennas
Antenna Now 2014 2015 Mc - 32 mt 10 Gbit/s 10 Gbit/s 10 Gbit/s Nt - 32 mt 1 Gbit/s 10 Gbit/s 10 Gbit/s SRT - 64 mt - 10 Gbit/s 10 Gbit/s IRA - HQ 10 Gbit/s 10 Gbit/s 3 x 10 Gbit/s
Juniper POP and Huawei Lambda DWDM devices
IPERF transfer tests Bologna -> Metsahövi time data speed latency packets lost lost % 30 sec 22.0 GB 6.28 Gbit/s 0.014 ms 2399/1540294 (0.16%) Metsahövi -> Bologna time data speed latency packets lost lost % 30 sec 13.8 GB 3.96 Gbit/s 0.027 ms 38/1666666 (0.0023%)
dbbc VSI VSI XFP Fila10 XFP XFP Glapper CX4 SFP+ MARK-5C SFP+ SFP+ SFP+ SFP+ SWITCH Correlator WAN Storage Antenna station network layout as set in Medicina
dbbc & Fila10G board upon installation
Fila 10G board test Medicina -> Bologna recording : successful @ 4 Gbit/s
Issues fixed and not fixed
Issues fxmanager.cpp & dedisperse_difx.cpp had something missing... #include<string.h> has helped a lot. Thanks Chris and Adam!
SIGNS!!! Apparently no antenna follows a directive about delay signs. gps-fmout (or fmout-gps) in antenna logs can differ from what is written in the EVN db As far as we know, Geodesy likes delay sign - whereas Astronomy likes delay sign + Only way to make things work... Try both signs. MC and NT have positive signs on delays (on the.v2d file)
En route to fringes First test got wrong - snowing in Medicina Second test cables inverted due to previous observation setup Finally got fringes on April 15th 2013
Mk4/DiFX fourfit 3.8 rev 756 Amp. and Phase vs. time for each freq., 60 segs, 2 APs / seg (4.00 sec / seg.), time ticks 10 sec BLLAC.wwdeyt, No0010, MS MEDICINA - NOTO, fgroup C, pol LL Fringe quality 5 Error code G SNR 2222.8 Int time 238.000 Amp 159.867 Phase 63.2 PFD 0.0e+00 Delays (us) SBD 0.077572 MBD -0.018253 Fringe rate (Hz) 0.001721 Ion TEC 0.00 Ref freq (MHz) 4966.4900 AP (sec) 2.000 Exp. Exper # Yr:day Start Stop FRT apr15 16383 2013:105 124833.00 125231.00 125027.00 Corr/FF/build 2013:150:102748 2013:151:102953 2013:074:164457 RA & Dec (J2000) 22h02m43.2914s +42 16 39.980" U Validity L M S 4966.49 4982.49 4998.49 5014.49 Freq (MHz) 68.3 54.5 66.9 80.7 Phase 202.1 224.1 210.0 7.4 Ampl. 131.6 131.3 131.4 132.0 Sbd box U/L 119/119 119/119 119/119 119/119 APs used M 1510 1510 1510 1510 PC freqs S 1510 1510 1510 1510 PC freqs M:S -174:115 171:-48 174:-87 153:152 PC phase M:S 0:0 0:0 0:0 0:0 Manl PC M 87 90 94 82 PC amp S 28 36 36 14 M C03U,C01L 26,28,30,32,10,12,14,16 C07U,C05L 27,29,31,33,11,13,15,17 C11U,C09L 26,28,30,32,10,12,14,16 C15U,C13L 27,29,31,33,11,13,15,17 Chan ids Tracks S C03U,C01L C07U,C05L C11U,C09L C15U,C13L Chan ids 26,28,30,32,10,12,14,16 27,29,31,33,11,13,15,17 26,28,30,32,10,12,14,16 27,29,31,33,11,13,15,17 Tracks Group delay (usec) 1.87618141400E+03 Apriori delay (usec) 1.87619966718E+03 Resid mbdelay (usec) -1.82532E-02 Sband delay (usec) 1.87627723895E+03 Apriori clock (usec) 1.0620000E+01 Resid sbdelay (usec) 7.75718E-02 Phase delay (usec) 1.87619970253E+03 Apriori clockrate (us/s) 2.7470000E-12 Resid phdelay (usec) 3.53505E-05 Delay rate (us/s) 1.07286064600E-01 Apriori rate (us/s) 1.07285645984E-01 Resid rate (us/s) 4.18616E-07 Total phase (deg) 21.8 Apriori accel (us/s/s) -2.23758137823E-06 Resid phase (deg) 63.2 +/- +/- +/- +/- +/- All 63.2 160.9 131.5 4.0E-06 1.6E-05 1.4E-08 2.1E-10 0.0 RMS Theor. Amplitude 159.867 +/- 0.072 Pcal mode: NORMAL, NORMAL Pcal period (AP s) 9999, 9999 ph/seg (deg) 25.2 0.2 Search (256X16) 147.499 Pcal rate: 1.301E-08, 8.509E-08 (us/s) sb window (us) -1.000 1.000 amp/seg (%) 13.4 0.3 Interp. 147.655 Bits/sample: 2 SampCntNorm: disabled mb window (us) -0.031 0.031 ph/frq (deg) 10.2 0.0 Inc. seg. avg. 179.583 Sample rate(msamp/s): 16 dr window (ns/s) -0.010 0.010 amp/frq (%) 55.6 0.1 Inc. frq. avg. 160.908 Data rate(mb/s): 256 nlags: 128 t_cohere infinite ion window (TEC) 0.00 0.00 M: az 296.6 el 36.4 pa 59.6 S: az 302.4 el 30.4 pa 65.8 u,v (fr/asec) -48.042 29.330 iterative interpolator Control file: default Input file: /space2/fullvlbit/1234/no0010/ms..wwdeyt Output file: Suppressed by test mode
dbbc bitstream The vex track assignment differs completely from the MARK-5B format which supposedly takes after. bit-stream (2-33) Astronomy mode 18 1LS 19 1LM 26 5LS 27 5LM 2 1US 3 1UM 10 5US 11 5UM 20 2LS 21 2LM 28 6LS 29 6LM 4 2US 5 2UM 12 6US 13 6UM 22 3LS 23 3LM 30 7LS 31 7LM 6 3US 7 3UM 14 7US 15 7UM 24 4LS 25 4LM 32 8LS 33 8LM 8 4US 9 4UM 16 8US 17 8UM Table 2: dbbc bitstream assignment
VDIF tests on dbbc The VDIF header recorded from the Medicina dbbc differs significantly from the one generated with DiFX tool vlbi_fake Tools used to record : NeXPRES s vlbistreamer and DiFX s captureudpvdif Endianess problem! VDIF vlbi_fake 0000000 \0 \0 \0 \0 * 0000010 \0 325 f \0 0000014 \0 \0 \0 033 0000020 $ 003 \0 002 0000024 t T \0 004 0000030 \0 \0 \0 \0 * 0000050 A A A A VDIF Md 0000000 \0 \r 257 020 0000004 [ \0 323 322 0000010 005 \0 003 377 0000014 004 \0 M d 0000020 253 315 357 001 0000024 020 376 334 272 * 0000040 377 374 300 \0 0000044 377 377 \0 \0 0000050
VSM Vlbi Storage Manager
Recording Which tool is best? What is going to be supported?