Cyber Performance for Network Performing Arts Production Workshop London, May 2015

Transcription

1 Cyber Performance for Network Performing Arts Production Workshop London, May 2015

2 Locations Prague: Music and Dance Faculty of the Academy of Performing Arts London: Royal College of Music Distance: ~1000 km / 650 miles Performance: real-time playing music together over distance using 4K Gateway hardware HAMU RCM

3 Music 1) J. Brahms: Violin concerto in D major Op. 77, 1st movement 2) M. Ravel: La Cigane Matouš Pěruška violin Alison Rhind - piano Prague London

4 Network Amsterdam London Prague Frankfurt

5 Low-latency transmissions for playing together and cyber performances Sven Ubik CESNET, Prague, Czech Republic Royal College of Music London, 6 May 2015

6 CESNET Czech Academic Network, 100 Gbps backbone, connected to Géant Now operating e-infrastructure (network, cloud computing & storage, remote collaboration services) for universities and research institutions National and EU research projects

7 Motivation Low-latency audio and video transmissions can enable real-time collaboration in multiple fields, including performing arts - Driven by practical uses cases as well as curiosity We cannot beat the propagation delay, but we can minimize additional processing delay, by building a real-time system Movie industry offered 4K experience several years ago, why not to do 4K distributed concerts

8 Hardware development 4K Gateway device, up to 2x 4K video, up to 96 audio channels FPGA-based, HW processing connector-to-connector approx. 3 ms latency in uncompressed mode for audio and 4K video in JPEG 2000 mode 5 frames / subframe latency multi-point connections are possible (8 locations) spinoff company created to sell a commercial version in 2015 lightweight version (fewer video channels, low-latency compression)

9 Prague / London side equipment 4K Blackmagic 6G-SDI to Design camera 4xHD-SDI 6G-SDI audio embedder 4K Gateway audio input 10GE to Géant studio mixer 4xHD-SDI to HDMI HDMI 4xHD-SDI 4K TV video from remote side audio dis-embedder 2x XLR balanced audio output

10 Applications Playing together over distance for teaching, rehearsals, etc. - minimum latency required Distributed performances for cultural exchange - high-quality video required Medical, industrial, post-processing,

11 emusic project 18-months GN3plus OpenCall project Partners: - 3 academic networks (CESNET, GARR, JANET) - 3 musical schools (HAMU in Prague, Conservatorio Tartini in Trieste, University of Edinburgh) Goal: to verify if real-time collaboration in music over Géant can be improved by the Bandwidth on Demand (BoD) service Experiments in playing together, remote sound mixing, medical voice inspection

12 Prague, SAGElab laboratory Playing together for the Geant Symposium, February 2015 Approx km away Prague Vienna Bratislava Athens, Megaron Conference Center Athens

13 Marry keys quartet in Prague (piano, drums, saxophone, vocal) Sebastiano Frattini in Athens (violin) Several jazz-style pieces Music

14 SAGElab Network Visualization Lab of CESNET + Czech Tech. Univ. LCD wall >8K resolution (9600 x 4320 pixels), bezels <5 mm between tiles 10 Gbps connection to Géant

15 Dance with delay Experimental evaluation of delay effects on dance performance Dance was especially selected to be sensitive to delay, by exchaging a virtual ball Performed in the SAGElab Visualization Laboratory in Prague

16 Dance with delay

17 3D live music between continents Live music harp and guitar in the Academy of Performing Arts in Prague Transmission using 4K Gateway, January 2013 University of Hawaii

18 Cyber Performances Dancing in space, for APAN 38 meeting in Taiwan, August 2014 Live performance between Prague, Barcelona, Miami and Taiwan APAN38 NanTou 20G Chicago To be filled 10G Miami 20G Amsterdam 10G 10G Prague Barcelona Taipei 10G 20G NanTou

19 Artists Musical trio in Prague: Stage 2 Dancing in Space Artists on Prague site Ivan Vokac V. Cello Ivan Vokac is a graduate of Prague Conservatory of Music. For the time being Ivan is studying with Prof. Miroslav Petras at the Academy of Performing Arts in Prague. Jan Mracek Violin Choreography and dance in Barcelona: Lukas Klansky Piano Music in Miami: Andrew Larson New World Symphony Dance in Taiwan: Alain Baumann Rosa Sánchez Victoria Macarte Live video interaction Choreography Dance Yi Tzu Lee Winnie

20 Video capture and projection CZ Trio Prague US Cellist Miami Normal Background Barcelona BC Dancer Black Background Taiwan TW Dancer Composed Picture

21 Main stage in Nantou, Taiwan Music in Miami: Musical trio in Prague Dance in Barcelona Dance in Taiwan

22 Main stage in Nantou, Taiwan

23 Cyber performance for APAN meeting in Daejeon, Korea Live music at the Academy of Performing Arts in Prague Dancers in Barcelona, Salvador, Daejeon Real-time performance between 4 countries, 3 continents, approx. 100 people involved, main audience in Daejeon at APAN 36 meeting 4K video from Prague transmitted by 4K Gateway

24 Cyber performance for APAN meeting in Daejeon, Korea

25 Delay components Network propagation delay Transmission equipment (codec + processing) Camera, TV / projector Video convertors

26 Video converters Between HDMI, SDI, analog, etc. Converters without frame buffer (cannot change frame rate) good for playing together, several µs latency Converters with frame buffer (can also change frame rate) bad for playing together, tens of millisecond latency

27 Audio delay measurement Signal generator, 1 khz sin wave in bursts audio chain Two channel oscilloscope

28 Audio delay measurement embedder + disembedder (2x AJA HD10AMA) audio delay 2.2 ms embedding + disembeding as a function in video converters (BMD HDMI to SDI + AJA Hi5-3G) audio delay 800 microseconds

29 Video delay measurement video chain video generator, SDI output 2x digital to analog Two channel oscilloscope

30 Video delay measurement Adding video interface convertors HDMI to SDI + SDI to HDMI (BMD + AJA Hi5-3G) video delay 8 microseconds

31 Audio + video delay measurement Loop Prague Athens Prague (a pair of L3 tunnels) 2x 4K Gateway, embedder, disembedder total audio and video delay 48 ms (network RTT 36 ms, equipment 12 ms) => one way delay will be 30 ms

32 Camera delay measurements Flash light Photodetector SDI to analog Two channel oscilloscope

33 Camera delay measurement Canon Legria M56 (consumer HDMI camera): video delay 100 ms GoPro Hero4 Black Edition: video delay 60 ms Sony XC-HR70 (analog): video delay 25 ms Blackmagic Design 4K: video delay 60 ms

34 TV / projector delay measurements SDI to HDMI / DVI video generator, SDI output SDI to analog Photodetector Two channel oscilloscope

35 Placement of photodetector Be aware, that monitor changes a frame from top to bottom, Placement of photodetector matters, here captured by GoPro in 720p240 mode

36 TV / projector delay measurements ACER Full HD LCD monitor HDMI input DVI input video delay 16 / 36 ms Projector Optoma LH501 (Full HD) HDMI / DVI / analog input video delay 60 ms

37 Technology summary Technology that combines low latency transmissions for playing together with high-quality video is available Network requirements are high, but it was shown that it can be done even over long distances In addition to network propagation delay, additional delay is now mostly in camera and TV / projector (transmission equipment is negligible)

38 Observations It seems that the order of importance for good feeling in playing together is: 1) good sound, 2) feeling other musicians and 3) good vision of other musicians Good sound means not only small delay, but also the quality of sound capture and reproduction, volume, dynamics, space and echo For what it was considered useful? Mostly for rehearsals, also for occasional live performances, but probably not for recording

39 Thank you for your attention! Sven Ubik