Intelligent Systems: Three Practical Questions. Carsten Rother

Transcription

1 Intelligent Systems: Three Practical Questions Carsten Rother 04/02/2015

2 Prüfungsfragen Nur vom zweiten Teil der Vorlesung (Dimitri Schlesinger, Carsten Rother) Drei Typen von Aufgaben: 1) Algorithmen 2) Definitionen und Wissensfragen 3) Theoretische Herleitungen Fragen werden auf Deutsch gestellt 04/02/2015 2

3 1) Algorithmen Was würde ein parallelisierter ICM Algorithmus in den nächsten zwei Schritten machen? Bitte zeichnen sie ein. Gegeben die Energy: x 1 x 2 Initializer Zustand: x 1 =0 x 2 =1 Hinweis: dunkle Konten werden im ersten Schritt nicht verändert x 4 x 3 x 4 =0 x 3 = 1 x i {0,1} θ 1 0 = 0, θ 1 1 = 1 θ 2 0 = 1, θ 2 1 = 1 θ 3 0 = 2, θ 3 1 = 1 θ 4 0 = 1, θ 4 1 = 2 θ ij x i, x j = x i x j For all i, j Schritt 1: x 1 =? x 2 =? x 4 =? x 3 =? Schritt 2: x 1 =? x 2 =? x 4 =? x 3 =? 04/02/2015 3

4 2) Definitionen und Wissensfragen Frage: Was ist ein strukturiertes Model ( structured model )? Antwort: Ein Modell in dem die Ausgaben nicht unabhängig sind. Der Zusammenhang der Ausgaben wird modelliert. 04/02/2015 4

5 3) Theoretische Herleitungen Man berechne die Wahrscheinlichkeit dafür, dass die Summe der Augenzahlen zweier voneinander unabhängig gewürfelten Spielwürfel durch 5 teilbar ist. (ähnliche Aufgaben wurden in der Vorlesung und Übung betrachtet) 04/02/2015 5

6 Roadmap this lecture The question of the combinatorial explosion The question of generalization The questions with Big Data 04/02/2015 6

7 Roadmap this lecture The question of the combinatorial explosion The question of generalization The questions with Big Data 04/02/2015 7

8 Going back to 1973 Sir James Lighthill report to the British Parliament The general purpose robot is a mirage Ein Roboter der alles kann ist eine Illusion Full report on Youtube: 04/02/2015 8

9 Going back to 1973 Sir James Lighthill report to the British Parliament He specifically mentioned the problem of "combinatorial explosion" or "intractability", which implied that many of AI's most successful algorithms would grind to a halt on real world problems and were only suitable for solving "toy" tasks. 04/02/2015 9

10 (My) View: 2015 Better computers (faster, more memory, clusters, GPUs) More powerful methods which can deal with combinatorial exploration (e.g. graph cut, search) Better sensors (e.g. time-of-flight) More (labeled) data can give better, more adaptive methods (e.g. deep learning) Humans can do it, and they have a finite-sized brain. 04/02/

11 Roadmap this lecture The question of the combinatorial explosion The question of generalization The questions with Big Data 04/02/

12 Mushroom example Measure attributes: - Size in centimeters - Color: Average whitness of the whole mushroom, e.g. diagonal in RGB cube 15cm Size 2cm 0 white 255 Task: Build a decision tree such that you can distgunish form mushrooms. 04/02/

13 Decision Tree Size>8cm 15cm Size 2cm 0 white /02/

14 Decision Tree Size>8cm white>180 15cm Size 2cm 0 white /02/

15 Decision Tree Split Criteria Before split Split 1 Split 2 Information gain Shannon s entropy Think of minimizing Entropy 04/02/

16 Decision Tree Split Criteria We have S = 12 with S L = 6 and S R = 6 In S we have 6 red and 6 blue points (2 classes) H(S) = 0.5 log log 0.5 = 1 We look at two possible splits: 1) 50%-50% class-split (each side (S L and S R ) gets 3 red and 3 blue) H S L = 0.5 log log 0.5 = 1 H S R = 0.5 log log 0.5 = 1 I(S) = H(S) ( ) = H(S) 1 = 0 (Lower information gain) 2) 16%-84% class-split (left side has 5 red and 1 blue, right side has 5 blue and 1 red) H S L = 1 log log 5 = H S R = 1 log log 5 = I S = H S = H S 0.64 = 0.36 (Higher information gain) 04/02/

17 Generalization Training Data: Test Data: Size>8cm white>180 System is optimal! 04/02/

18 Generalization Training Data: Test Data: Size>8cm white>180 System may be optimal! 04/02/

19 Generalization 15cm Size 2cm white BUT its BUT its!! 04/02/

20 Definition Over-fitting: Is the effect that the model perfectly memorizes the training data, but does perform well on test data Generalization: One of the most important aspect of a model is its ability to generalize. That means that new (unseen) test data is correctly classified. A model which overfitts does generalize well. How to avoid over-fitting? Idea 1: Max-Margin Idea 2: Do make the trees too deep 04/02/

21 Max-Margin Size>9cm white>190 15cm Size 2cm 0 white 255 Place decision boundary such that the distance to all (some) examples is maximized! 04/02/

22 Do make the trees too deep Size>9cm 15cm Output: p() = 2/3 p( ) = 1/3 Size 2cm 0 white /02/

23 Do make the trees too deep - Comparison Test data: 15cm Not Size System with 2 nodes makes: 2 mistakes on test data System with 1 nodes makes: 2 mistakes on test data 2cm Not 0 white 255 This means: system with one node makes a mistake during training, but performs equally well on testing phase. 04/02/

24 Example: People tracking body joint hypotheses input depth image body parts Bodypart Labelling Clustering Body is divided into 31 body parts (simple centroid computation) front view side view top view 04/02/

25 Small Model capacity Model Is the part round? Is it darker below me? Color red Test time: Training data 04/02/

26 Large Model capacity Model Is the part round? Is it darker below me? Tree is very deep (e.g. depth 30) Is the colour of my pixel 130? Test time: Training data The Model perfectly memorizes the training data! 04/02/

27 Tree Depth More training data pushes this boundary Number of Parameters (tree depth) Decision trees 04/02/

28 Infinite Training Data Model Is the colour of my pixel 130? Is the colour of my pixel 131? Tree is very deep (e.g. depth 30) The Model perfectly memorizes the training data! Test time: If we were to have all possible training images that can occur (inclusive noise) 04/02/

29 Real System that runs on Xbox! accuracy Amount of Training Data is one of the main factors 04/02/

30 Roadmap this lecture The question of the combinatorial explosion The question of generalization The questions with Big Data 04/02/

31 The important factors Model Power Performance (e.g. tree depth) Training Data (unlabeled and labeled!) More Training Data = More powerful models = Better Performance 04/02/

32 The Increase in Data 2014 ~10M images ~60M Parameters accuracy rate: ~85% ImageNet Challenge 2005 ~100 training images ~100 Parameters accuracy rate: ~20% 04/02/

33 Deep Models have many parameters The return of Neural Networks from the ~70s made them work Major leap forward in ImageNet Challenge [ImageNet Classification with Deep Convolutional Neural Networks, Hinton et al. 2013] 04/02/

34 Where does the training Data come from? 1. Anation by researchers: ~1K training images 2. Search Engines: type in cow 3. Crowd sourcing: ~1M training images 4. Simulation: ~10M training images 04/02/

35 Where does the training Data come from? 1. Get images of tigers 2. Search Engines: type in tiger 3. Crowd sourcing: run e.g. Amazon Mechanical Turk to filter out bad images. (Start-up companies) 04/02/

36 Where does the training Data come from? ImageNet challenge 1.2 Million anated images; classes 04/02/

37 Where does the training Data come from Simulation: ~10M training images Record mocap 100,000s of poses Retarget to varied body shapes [Vicon] Render (depth, body parts) pairs + add noise we start to look into this with biologists 04/02/

38 Roadmap this lecture The question of the combinatorial explosion The question of generalization The questions with Big Data 04/02/

39 Thanks! 04/02/