ARC 3D Webservice How to transform your images into 3D models. Maarten Vergauwen

Transcription

1 ARC 3D Webservice How to transform your images into 3D models Maarten Vergauwen

2 Overview What is it? How does it work? How do you use it? How to record images? Problems, tips and tricks

3 Overview What is it? How does it work? How do you use it? How to record images? Problems, tips & tricks

4 What does the 3D Webservice do? The ARC 3D Webservice allows you to transform your images into 3D models automatically. You only need a digital camera, a PC and access to the Internet. images Maurizio Forte, CNR Rome

5 What are the different steps? Step 1: Record a sequence of images of a scene or object Step 2: Upload the images to the ARC server Step 3: The server computes the 3D reconstruction Step 4: Download the results from the ARC website Step 5: Visualize the results

6 Overview What is it? How does it work? How do you use it? How to record images? Problems, tips & tricks

7 How does it work? The 3D Webservice only uses images as input. Computer vision techniques are used to compute the results. Automatic matching of images Automatic computation of camera calibration Automatic dense matching for depth map generation

8 Image Matching The entire process is based on finding matches between images. Record your pictures not too far apart, so the computer can match them easily!

9 Camera Calibration No prior knowledge on camera calibration is available, so all information must be recovered from the images It is therefore important that enough information is present in the images! General motion of the camera General structure in the scene Enough overlap (only points that are visible in at least 3 images are useful) See tips and tricks for more info

10 Dense Matching After recovery of the camera calibration, dense depth maps are computed These contain the depth of every pixel and a quality measure (how confident we are of each particular pixel)

11 Overview What is it? How does it work? How do you use it? How to record images? Problems, tips & tricks

12 How do you use it? Obtain an account from the website Download and install the programs Use the upload program Load your images Upload them to the server Have a coffee Download the results from the server Visualize the results

13 Obtain an account Surf to and navigate to Request Login

14 Obtain an account (2) Fill in name, address and the captcha form

15 Obtain an account (3) Your login information will be sent to your address

16 Download and install Install as Administrator in the default place or as normal user, specifying a writeable folder

17 Upload Client

18 Upload Client (2) * If ARC decides to use your images in a publication, it will add this reference. See conditions on the ARC homepage. Images selected by user Subsample to smaller size for faster upload and computation Authentication with ARC Organization into scenes and sequences Select or create a new scene Supply a sequence label Add academic reference if desired*

19 Have a coffee

20 Download the results The ARC service reports its result to you by If the reconstruction has succeeded, a URL is specified from where you can download a zip file containing the results. If the reconstruction failed, some more information is given, including tips on how to improve the result.

21 Visualize the results Two tools are available for inspecting the results The ARC team recommends MeshLab, a tool developed by CNR-ISTI in Pisa, Italy A simple model viewer comes with the ARC installer

22 Visualize with MeshLab Download MeshLab from meshlab.sf.net and read the documentation

23 Visualize with Model Viewer Download results from server Unzip file with images, cameras, depth and count maps Select one image on the left. Reconstruct 3D model for this specific depth map Parameters to tune: Minimum quality (count) Subsample for smaller models Minimum angle w.r.t. viewing direction. This ensures that shearing angles are removed Median filter size: larger filter for smoother (less noisy) models

24 Visualize with Model Viewer (2) Select the model tab and view the result in a 3D widget Texture comes from the original image Export to different formats (X3D, PLY, OBJ, VRML) is possible

25 Some Examples

26 Temple of the Masks, Edzna Maurizio Forte, CNR Rome

27 Temple of the Masks, Edzna Movie available at

28 Arc de Triomphe Movie available at

29 Overview What is it? How does it work? How do you use it? How to record images? Problems, tips & tricks

30 How To Record images? The ARC 3D Webservice has been developed with robustness in mind. However to minimize frustration it is best to take heed of some simple guidelines! We will explain some dos and don ts

31 A Good Sequence Shoot a picture of the same location for every step made in the shooting sequence. This results in multiple pictures of the same scene, but viewed from slightly different sides. Walk with the camera in an arc around the scene, while keeping the scene in frame at all times.

32 Bad Sequences not pan from the same location, as if you were recording a panorama. It is not possible to determine enough 3D information from such a sequence. Do

33 Bad Sequences (2) Also don t shoot multiple panorama s from different viewpoints.

34 Bad Sequences (3) Do not walk in a straight line towards or inside the scene you want to reconstruct.

35 Bad Sequences (4) It is better to shoot too many pictures than not enough. A minimum of five or six images are required for a good reconstruction. Less than four and the reconstruction will fail. The viewing angle between images should not be too small, i.e. adjacent images should not be too far apart

36 Bad Sequences (5) Do not use a turntable with a static camera and a scene that rotates on a perfect circle in front of the camera. Because of this very specific motion, there is not enough information in the images to allow camera calibration.

37 Bad Sequences (6) Do not take pictures of a planar (or almost planar) scene These scenes do not contain enough information for camera calibration

38 Bad Sequences (7) Do not zoom in or out during recording!

39 Recording Equipment What kind of camera should I use? More pixels = more 3D points = longer upload and processing time 5MP (2500x2000) is typically good enough Good lens = less distortion = better result Good lens = more light = better result

40 Overview What is it? How does it work? How do you use it? How to record images? Problems, tips & tricks

41 Problems, Tips and Tricks Two major steps in the pipeline Camera calibration Dense matching Both vulnerable to problems Most important rule to realize All information is retrieved from the images, so take good care when you shoot them! Better to spend 2 more minutes than have to go back and re-shoot!

42 Image Content Information is found by matching images Make it possible to do so automatically! The texture (color, intensity) of the scene object is critical! Enough texture must be available on the object Appearance of object must stay the same!

43 Not Enough Texture

44 No Constant Appearance

45 No Constant Appearance

46 No Constant Appearance

47 No Constant Appearance The first thing people try, is to upload their baby pictures This will not work! Human skin is notorious for changing appearance when viewed from a different angle

48 Occlusions and Self-occlusions Objects blocking the scene (e.g. trees) make the reconstruction of hidden parts impossible. Sometimes the object is self-occluding: parts sticking out hide parts behind it

49 Quality: Beware Blurry Pictures When the camera moves during recording, the image becomes blurred This causes problems for the reconstruction or degrades the result Hence avoid it!

50 Dynamic Scenes = No No The scene must be static: it should not change during the recording!

51 Lighting Conditions Perfect Overcast sky Constant illumination Worse Direct sunlight Bad Changing conditions

52 More Info? More information about the ARC webservice can be found here Read the MVA publication: M. Vergauwen and L. Van Gool, Web-based 3D Reconstruction Service, in Machine Vision Applications (MVA), 17, pp , Send an to