Solving Big Data Problems in Computer Vision with MATLAB Loren Shure

Size: px

Start display at page:

Download "Solving Big Data Problems in Computer Vision with MATLAB Loren Shure"

Bathsheba Carter
8 years ago
Views:

1 Solving Big Data Problems in Computer Vision with MATLAB Loren Shure 2015 The MathWorks, Inc. 1

2 Why Are We Talking About Big Data? 100 hours of video uploaded to YouTube per minute 1 Explosive increase in number of imaging devices Webcams Smartphone Cameras IP Cameras Industrial Cameras 1- KPCB 2013 Internet Trends 2

increase in number of imaging devices Webcams Smartphone Cameras

3 2. Can I find patterns or models to represent my image data? 5. What if my data is too big to fit into memory? Goal: Show you new functionality in MATLAB to augment existing workflows to solve these challenges 3

Problem: Image Category Classification Given a large set of images of cars, planes and motorbikes How to model the data to recognize which category an image belongs to?

4 Problem: Image Category Classification Given a large set of images of cars, planes and motorbikes How to model the data to recognize which category an image belongs to? Image Processing Algorithm motorbike Data from Caltech 101: L. Fei-Fei, R. Fergus and P. Perona. Learning generative visual models from few training examples: an incremental Bayesian approach tested on 101 object categories. IEEE. CVPR 2004, Workshop on Generative-Model Based Vision

5 imageset 2. Can I find patterns or models to represent my image data? 5. What if my data is too big to fit into memory? 5

6 imageset 2. Can I find patterns or models to represent my image data? 5. What if my data is too big to fit into memory? 6

trainimagecategoryclassifier Classifier / Model hat imageset

7 Machine Learning Workflow Using Images Training Data Feature Extraction Learning or Modelling imageset bagoffeatures trainimagecategoryclassifier Classifier / Model hat imageset Input Image bagoffeatures Feature Extraction evaluate Classification 8

8 imageset to manage large collections of images 2. Can I find patterns or models to represent my image data? Computer Vision System Toolbox, Statistics and Machine Learning Toolbox 5. What if my data is too big to fit into memory? 9

Computer Vision System Toolbox, Statistics and Machine

9 imageset to manage large collections of images 2. Can I find patterns or models to represent my image data? Computer Vision System Toolbox, Statistics and Machine Learning Toolbox 5. What if my data is too big to fit into memory? 10

10 imageset to manage large collections of images 2. Can I find patterns or models to represent my image data? Computer Vision System Toolbox, Statistics and Machine Learning Toolbox Image Batch Processor App 5. Can I acquire large sets of images using MATLAB? 11

Computer Vision System Toolbox, Statistics and Machine Learning

11 imageset to manage large collections of images 2. Can I find patterns or models to represent my image data? Computer Vision System Toolbox, Statistics and Machine Learning Toolbox Image Batch Processor App 5. What if my data is too big to fit into memory? 12

12 Parallel Computing with MATLAB MATLAB Distributed Computing Server MATLAB Parallel Computing Toolbox 13

13 14

14 imageset to manage large collections of images 2. Can I find patterns or models to represent my image data? Computer Vision System Toolbox, Statistics and Machine Learning Toolbox Image Batch Processor App Parallel Computing Toolbox, MATLAB Distributed Computing Server 5. What if my data is too big to fit into memory? 15

Computer Vision System Toolbox, Statistics and Machine Learning Toolbox Image

15 imageset to manage large collections of images 2. Can I find patterns or models to represent my image data? Computer Vision System Toolbox, Statistics and Machine Learning Toolbox Image Batch Processor App Parallel Computing Toolbox, MATLAB Distributed Computing Server 5. What if my data is too big to fit into memory? 16

Big Data Stored on Hadoop Easily migrate programs from workstation to Hadoop On Your Workstation Access, explore, and analyze out-of-memory data Prototype MapReduce based algorithms Access portions

16 Big Data Stored on Hadoop Easily migrate programs from workstation to Hadoop On Your Workstation Access, explore, and analyze out-of-memory data Prototype MapReduce based algorithms Access portions of data sets on HDFS Connected to Hadoop Run algorithms on Hadoop against data in HDFS MATLAB Distributed Computing Server Production Hadoop Create applications or components that execute on Hadoop MATLAB Compiler 17

data sets on HDFS Connected to Hadoop Run algorithms on Hadoop against data in HDFS MATLAB Distributed

17 Scaled Analysis Prototype Access Explore Share/Deploy Scale Load, Analyze, Discard datastore, parfor MapReduce Distributed Memory SPMD out-of-memory in-memory Embarrassingly Parallel Complexity Non-Partitionable 18

MapReduce Distributed Memory SPMD out-of-memory

18 imageset to manage large collections of images 2. Can I find patterns or models to represent my image data? Computer Vision System Toolbox, Statistics and Machine Learning Toolbox Image Batch Processor App Parallel Computing Toolbox, MATLAB Distributed Computing Server 5. What if my data is too big to fit into memory? MapReduce, distributed arrays, SPMD, Hadoop 19

Computer Vision System Toolbox, Statistics and Machine Learning Toolbox Image Batch

19 Thank You! 25

2015 The MathWorks, Inc. 1

2015 The MathWorks, Inc. 1 25 The MathWorks, Inc. 빅 데이터 및 다양한 데이터 처리 위한 MATLAB의 인터페이스 환경 및 새로운 기능 엄준상 대리 Application Engineer MathWorks 25 The MathWorks, Inc. 2 Challenges of Data Any collection of data sets so large and complex