> CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands >

Transcription

1 DLR.de Folie 1 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > A Dataset to Support and Benchmark Computer Vision Development for Close Range On-Orbit Servicing Martin Lingenauber, Simon Kriegel, Michael Kaßecker and Giorgio Panin , 13th ASTRA Symposium ESA/ESTEC, Noordwijk, The Netherlands

2 DLR.de Folie 2 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > Close Range OOS Close Range On-Orbit Servicing LEO From 2-3m to contact Tasks Approach & Grasp Inspection Repair Refuel Avoid new space debris Computer Vision for high accuracy CROOS-CV challenges Specular reflections Sensor saturation Abrupt brightness changes DEOS mission Orbital Express Mission (video credit DARPA [1])

3 DLR.de Folie 3 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > Challenges for Computer Vision in Orbit CROOS-CV challenges Specular reflections Sensor saturation Abrupt brightness changes Images from the Orbital Express mission (image credit DARPA [1])

4 DLR.de Folie 4 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > Computer Vision Datasets Freely available datasets Input data and ground truth data Comparison based on same data Forster research with fair competition Learn from each other Dataset philosophy Training set for parameter tuning Test set for blind tests Test set > Training set Real data if possible YACVID Index [4] 276 computer vision datasets for download 0 space computer vision dataset Middlebury stereo vision benchmark [2] KITTI Vision Benchmark Suite [3]

5 DLR.de Folie 5 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > Agenda Recording Setup Recording Method Dataset Experiment & Results Summary & Conclusion

6 DLR.de Folie 6 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > Recording Setup

7 DLR.de Folie 7 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > Recording Setup Overview (Video) Check the CROOS-CV webpage for the video

8 DLR.de Folie 8 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > Recording Setup Details Satellite mockup (real scale) approx. 1.8m diameter 6 LIFs (Launcher Interface) All Attachments & MLI wrapping Stereo cameras each 780 x 582 px Focus 20cm sharp at target FOV 56 x 44 (6mm focal length) 600mm base line Industrial robot KUKA KR mm worst case positioning error 220 to 20cm distance to target

9 DLR.de Folie 9 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > Recording Method

10 DLR.de Folie 10 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > Mockup & Sun Pose Estimation Pose of fully wrapped mockup? Pose of light source optical axis? Camera calibration with DLR CalDe & CalLab [5] Pose estimation with AprilTag Library [6]

11 DLR.de Folie 11 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > Alignment of 3D Model and Mockup Initial Fine (ICP) Iterative Closest Point (ICP) algorithm for 3D registration [7] ± 2mm accuracy Registration of laser scans of attachments Compliance of 3D model and mockup

12 DLR.de Folie 12 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > Robot Path 6 LIFs 5 different paths 4 are parallel to center path (10cm) 30 linear approach paths in total LIF always in center of camera 0 Camera turning during outer paths 2m range (220-20cm to LIF) LIF-3: 90cm (110 20cm, workspace limitations) Automatic path planning Robot reconfiguration during approach Trajectory = {path, sun position, shutter time}

13 DLR.de Folie 13 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > Recording Procedure Preparation Camera calibration Pose estimation of mockup Laser scans of attachments Align mockup and 3D model At each sun position Pose estimation of light source Training set 2 shutter times per sun position Separated recording sessions for shutter times Test set 9 shutter times per sun position All shutter times in one recording session

14 DLR.de Folie 14 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > Dataset

15 DLR.de Folie 15 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > Stereo Camera Pointing Camera 0 Camera 1 Video sequences from the training set, sun position 2, shutter time 0.01 s, LIF3_0 please check the CROOS-CV website to watch the video

16 DLR.de Folie 16 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > Camera Turning Video sequences from the training set, sun position 2, shutter time 0.01 s, LIF3 please check the CROOS-CV website to watch the video

17 DLR.de Folie 17 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > Illumination Conditions Sun 0 ( 90 ) Sun 1 ( 31 ) Sun 2 ( -31 ) Sun incidence angle wrt. front face normal

18 DLR.de Folie 18 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > Illumination Differences between LIFs

19 DLR.de Folie 19 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > Shutter Times in Training Set (2 per sun position) Sun 0 Sun 1 Sun s s s 0.07 s 0.01 s 0.01 s In total 180 trajectories

20 DLR.de Folie 20 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > Shutter Times in Test Set (9 per sun position) s 0.01 s 0.02 s 0.03 s 0.04 s 0.05 s 0.06 s 0.07 s s In total 810 trajectories

21 DLR.de Folie 21 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > Effects in Dataset Brightness changes (only in test set) Flickering light (random) Shutter flicker (random) s Robot reconfiguration Jumps between images (within robot accuracy) s Video sequences from the test set, sun position 0, shutter time and s, LIF0_0 please check the CROOS-CV website to watch the video

22 DLR.de Folie 22 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > Effects in Dataset Occlusions (beginning of LIF4 & 5) Visible background masking feasible Temporary shadows on target Specular reflections (wandering) Saturation Hard shadows Video sequences from the test set, sun position 1, shutter time 0.02 and 0.04 s, LIF5_0 and LIF1_0 please check the CROOS-CV website to watch the video

23 DLR.de Folie 23 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > Experiment & Results

24 DLR.de Folie 24 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > Visual Tracking Experiment

25 DLR.de Folie 25 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > Summary & Conclusion

26 DLR.de Folie 27 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > CROOS-CV Dataset Facts First publically available dataset for OOS Real scale satellite mockup 6 targets (Launcher Interface) Stereo cameras, each 780 x 582 px 3 different sun incidence angles Controlled illumination conditions 30 approach paths 2m (90cm) covered range 180 training trajectories Parameter tuning 2 shutter settings 810 test trajectories Blind testing 9 shutter settings Random effects >400k stereo images & ground truth poses 3D mesh of mockup and of LIFs

27 DLR.de Folie 28 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > CROOS-CV Dataset Facts First publically available dataset for OOS Real scale satellite mockup 6 targets (Launcher Interface) Stereo cameras, each 780 x 582 px 3 different sun incidence angles Controlled illumination conditions 30 approach paths 2m (90cm) covered range 180 training trajectories Parameter tuning 2 shutter settings 810 test trajectories Blind testing 9 shutter settings Random effects >400k stereo images & ground truth poses 3D mesh of mockup and of LIFs

28 DLR.de Folie 29 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > Future work Improve setup Increase accuracy Metrics for benchmarking Increase variety of tasks, targets and paths Download the CROOS-CV dataset from Remarks, requests or ideas? Please contact us:

29 DLR.de Folie 30 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > Enjoy using the dataset! Download the CROOS-CV dataset from Remarks, requests or ideas? Please contact us:

30 DLR.de Folie 31 > CROOS-CV Dataset > Martin Lingenauber 13th ASTRA Symposium, ESA/ESTEC Noordwijk, Netherlands > References [1] DARPA Orbital Express: [2] The Middlebury Computer Vision Pages: D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis., 47(1-3):7 42, [3] KITTI Vision Benchmark Suite: A. Geiger, P. Lenz, and R. Urtasun. Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite. In Conf. Comp. Vis. Pattern Rec. (CVPR), [4] Yet Another Computer Vision Index To Datasets (YACVID): [5] DLR CalDe and CalLab: K. H. Strobl and G. Hirzinger. More accurate camera and hand-eye calibrations with unknown grid pattern dimensions. In Proc IEEE Int. Conf. Robot. Aut. (ICRA), pages , Pasadena, California, USA, May IEEE. [6] AprilTag Library: E. Olson. AprilTag: A robust and flexible visual fiducial system. In Proc IEEE Int. Conf. Robot. Aut. (ICRA), pages IEEE, May [7] P.J. Besl and N.D. McKay. A Method for Registration of 3-D Shapes. IEEE Trans. Pattern Anal. Mach. Intell., 14(2): , 1992.