Pallas Ludens. We inject human intelligence precisely where automation fails. Jonas Andrulis, Daniel Kondermann

Transcription

1 Pallas Ludens We inject human intelligence precisely where automation fails Jonas Andrulis, Daniel Kondermann

2 Chapter One: How it all started The Challenge of Reference and Training Data Generation

3 Scientific Data Which algorithm works best in which situation?

4 Biological Data How does an organism form from cells?

5 Medical Data How to perform computer aided surgery?

6 Astronomy Data How does the geology of Mars look like?

7 Weather Data Where does all the sand come from?

8 Entertainment Data How to insert special effects into this movie?

9 Sports Data How to track progress of professional players?

10 Augmented Reality Data How to insert computer graphics into a scene?

11 Industry Data How to judge the quality of products on the belt?

12 Transportation Data How to train autonomous vehicles?

13 Lightfield Data Which materials occur in the scene?

14 Reference/Training Data Without Ground Truth With weak Ground Truth With Ground Truth Experts Annotations Measurements More Data Graphics Bernd Jähne Environmental Sciences Ground Truth Christoph Schnörr Optimization Theory Fred Hamprecht Machine Learning Biology Björn Ommer Object Detection Art History Christoph Garbe Modelling Remote Sensing

15 Chapter Two: Science!

16 Reference Data With Weak Ground Truth Optical Flow Based on Human Annotations? Cost per frame has to be very small Cannot ask users to annotate pixelwise flows Application Scenario: No accurate measurement devices exist Human brain performs better than algorithms Very large amounts of data needed Bias and other systematic errors less important Idea: Use Crowdsourcing! Axel Donath, ICVS 2013

17 What is Crowdsourcing? Merriam Webster: Crowdsourcing is the process of obtaining needed services, ideas, or content by soliciting contributions from a large group of people, and especially from an online community, rather than from traditional employees or suppliers. Wikipedia: It combines the efforts of numerous self-identified volunteers or part-time workers, where each contributor of their own initiative adds a small portion to the greater result. The term "crowdsourcing" is a portmanteau of "crowd" and "outsourcing"; it is distinguished from outsourcing in that the work comes from an undefined public rather than being commissioned from a specific, named group.

18 A Very Quick Related Work Check

19 The Crowdsourcing Landscape

20 Axel Donath, ICVS 2013

21 Crowdsourcing for Optical Flow? How Good is Crowdsourcing for Optical Flow Ground Truth Generation? Axel Donath, Daniel Kondermann (ICVS 2013)

22 Optical Flow Step One: Contours Axel Donath, ICVS 2013

23 Optical Flow Step Two: Features Axel Donath, ICVS 2013

24 Resulting Weak GT: Input based on Ce Liu s work Annotations by Crowd Endpoint Error (0..2px) Axel Donath, ICVS 2013

25 MICCAI 2014: Crowd versus Experts Can Masses of Non-Experts Train Highly Accurate Image Classifiers? Lena Maier-Hein, Sven Mersmann, Daniel Kondermann, Sebastian Bodenstedt, Alexandro Sanchez, Christian Stock, Hannes Gotz Kenngott, Mathias Eisenmann, Stefanie Speidel Crowdsourcing for Reference Correspondence Generation in Endoscopic Images Lena Maier-Hein, Sven Mersmann, Daniel Kondermann, Christian Stock, Hannes Gotz Kenngott, Alexandro Sanchez, Martin Wagner, Anas Preukschas, Anna-Laura Wekerle, Stefanie Helfert, Sebastian Bodenstedt, Stefanie Speidel

26 Crowd vs. Experts: Contours Lena Maier-Hein, et al. MICCAI 2014

27 Crowd vs. Experts: Contours Lena Maier-Hein, et al. MICCAI 2014

28 Crowd vs. Experts: Correspondences Lena Maier-Hein, et al. MICCAI 2014

29 Crowd vs. Experts: Correspondences Lena Maier-Hein, et al. MICCAI 2014

30 Crowd vs. Experts: Results Contours: Correspondences: Slightly better Extremely similar (KW = Knowledge Worker, raw = without postprocessing) Lena Maier-Hein, et al. MICCAI 2014

31 Chapter Three: Business!

32 Use-Cases Use-Case Descriptions Example for a medical use-case Speed-up of time-critical processes Sometimes there is just not enough time to look through thousands of images. Within minutes we provide a dataset that has the areas of interest already preselected - so the experts can focus on these parts first A patient comes in during the night with suspicion of internal bleeding. Time is short and only few doctors are available. We send the MRT pictures directly into the crowd and provide the expert with an already processed dataset within minutes. The doctor immediately identifies the problem without having to go through all the images and can react in time Additional aides for experts The task of an expert working with medical imaging data can be substantially supported by out data saving time and eliminating tedious work A huge histological dataset has to be analysed by experts. The experts sends the dataset to us and shortly has all relevant tissues already preselected. She flips through the images and applies a few corrections here and there based on expert knowledge overall saving huge amounts of time by focusing on tasks requiring specialist knowledge Fast and affordable analysis of very big datasets Current conditions make the annotation of datasets containing of millions of frames unfeasible. With the speed and cost efficiency we offer that is about to change In order to conduct a big study on surgical interventions with cancer patients video data from different angles has to be processed. Here all the tools have to be annotated along with correspondences for 3D tracking of tools and tissue. This resulted in millions of individual annotations tasks that we were able to finish within few days

33 More Use-Cases(?) Choice of Parameter Settings Application Testing/Referencing Remote Supervision Henry Ford 2.0?

34 Typical Project Workflow DATA COLLECTION PREPROCESSING ANNOTATION POSTPROCESSING DATA ANALYSIS & UTILIZATION Collect image sequences Associate metadata Removal of sensitive data Adjustment of user interfaces Automatic initial guess Annotation of boxes, labels, contours, features, correspondences First quality assurance in crowd Manual quality assurance Sanity checks Computation of final results based on existing algorithms Data check Publication

35 Most Computer Vision Can Be Solved With Four Task Types Crowd Task Boxes Contours Correspondences Image Processing Solution - Detection (Objects, Features, Properties) - Object Tracking - Segmentation, (Alpha Matting) - Texture Detection and Classification - Denoising - Optical Flow, Frame Interpolation (Slow Motion) - Stereo Depth Maps, View Interpolation - Camera Tracking (Matchmoving) - 3D Point Cloud Reconstruction Labels - Classification (Objects, Parts, Attributes, Materials) - Scene Understanding/Description - Action Recognition -

36 Issues With MTurk Data Throughput Batch Processing Cost Student Supervision Interface Design Mturk Spam Control Pallas Ludens GmbH

37 Cost Parameters Influencing factors and options Advanced aspects Size of the image sequence Number of frames/ images Expected number of objects/ required detections Automatic interface for new images Near-time Integration Required Pre-Processing/ Prior Knowledge Legal pre-processing Modification of UI and examples Definition of special cases Integration of existing data Pre-processing with filters Integration of existing tracking and detection methods (e.g. with depth information) Type of annotation Annotation/ detection (e.g. with boxes) Addition of object-contours, features, correspondences, semantics Validation of existing results Applied post-processing methods No Post-Processing Calculation of optical flow or depth map Feature extraction and relevance analysis Accuracy/ quality Allowed pixel deviation Quality verification with the crowd Consistency analysis Plausibility check with cluster analysis Mark-up of conflicts and uncertainties

38 General Issues With Crowdsourcing Privacy, Copyright Protection Solution: Image obfuscation, private crowds Compliance (e.g. worker age) Solution: Crowd filters, legal consultation Crowd Retention Solution: Motivation via gamification Quality Assurance Solution: Serverside computer vision and machine learning Crowd Size Solution: Games Part of our Service

39 Our Crowds Size: Monthly Active Users (MAU) Speed: Daily Active Users (DAU) m concurrent ??? 3-7m concurrent

40 How Much Work Can They Do? 50k Bigpoint 625 PT per day (ca. 2.8 years) 27000k Riot PT per day (ca. 255 years)

41 Gamification of Crowdsourcing

42 Summary Crowdsourcing scales very well is very cost effective Quality is the key issue For selected applications we could show excellent quality compared to experts User Experience is of utmost importance Pre-/Postprocessing of tasks can reduce cost Solve many issues with privacy, compliance, etc. We offer a full service on all aspects You give us the data, we take care about everything else

43 Our Website We are at VISION 2014! (D10 3.3)

44 We are looking forward to working with you We are looking forward to working with you. Please contact us with any questions and ideas. Thank you