Leveraging Big Data. Processing ISR Data. JP Morgenthal Cloud Ranger. Cloud and Virtual Data Center Services EMC Consulting

Transcription

1 Leveraging Big Data and Cloud for Processing ISR Data JP Morgenthal Cloud Ranger Cloud and Virtual Data Center Services EMC Consulting 1

2 Video Is A Key Component to Many ISR Programs Comment, Collaborate, Distribute, Actionable Content Enhance, Edit, Tag, Annotate, Report, Notify/Alert Capture, Convert, Store, Disseminate Students, Peers, COI, Decision Makers Analysts, Researchers, Teachers Infrastructure Providers, Tactical Responders 2

3 Making Video Actionable Well, That s Another Story Function Action Blog/Discuss/Annotate Share/Collaborate Enhance Analyze Real-time/Video-on- Demand Retrieve/View Metadata Index/Search Video Cloud Store/Distribute 3

4 Video Requires Specialized Architecture Media & Application Servers Video On-demand Asset & Metadata Management Storage Pedabytes storage management Optimized for high-performance video read/write rite and streaming Network Appliances Multicast IP routers IGMP enabled Encoders/Decoders Multi-format Transcoding Traffic shaping Std Def/Hi-def Cluster management 4

5 Analytics Also Requires Specialized Architecture Extends the boundaries for data storage in an elastic, scalable and cost-effective manner Removes the need for summarization of data, which is lossy Expands the time frame for data retention allowing for the identification of patterns that were previously missed due to summarization and smaller population sizes Provides cost-effective ti compute power that t scales to demand of the processing requirements 5

6 The New ITaaS Architecture Transforming the Application Layer New End-user Computing TR RUST Private New Application Platform Automated Hybrid Cloud Infrastructure Public O RCHEST TRATION 6

7 Analytics Also Requires Specialized Infrastructure Extends the boundaries for data storage in an elastic, scalable and cost-effective manner Hence, analyzing HD video Removes the need for summarization of data, which is requires lossy specialized Expands the time frame for data retention architecture! allowing for the identification of patterns that were previously missed due to summarization and smaller population sizes Provides cost-effective ti compute power that t scales to demand of the processing requirements 7

8 Breaking Down The ISR Solution Real time vs. Archived Video Analysis Real time looks for movement and object analysis Archived analysis looks for patterns over time Types of questions consistent with transactional / social data analysis Did we see this pattern in the past and miss it? How does time impact our ability to monitor? Was a particular object in a particular location at multiple times? 8

9 What This Presentation Will Show 1. Using Hadoop in non-traditional manner Going beyond word counting 2. Focus on architecture of Map/Reduce Data Scientists will analyze patterns of interest Architects and developers will put pattern into production 3. Focus on existing area of interest Collaborating around video surveillance 9

10 Cloud Video Solution Architecture Customer Site Service Provider IP cameras Local Storage Application server Cloud Storage Router / Switch 10

11 What is Big Data? Majority of people believe that Big Data is all about volume Luke Lonergan, CTO, Greenplum it s about volume and complexity Commodity hardware available on-demand for metered usage makes previously cost-prohibitive processing possible 11

12 Continuous Surveillance May Change Infrequently 12

13 Detect Changes While Avoiding Noise 13

14 Fourier Transforms 14

15 The Map Task Take 3 consecutive frames and look for significant ifi change Avoid noise Fourier Transforms provide one the best tools for rapidly analyzing change in images over time If found, return the timestamp of 2 nd frame 15

16 The Reduce Task Aggregate all the timestamps 16

17 Hadoop Workflow Map Task putforma atter Map Task or Collect Indicies In 17

18 Enhancements Leverage GPU cluster to perform fourier transforms Reduce the number of nodes needed to complete creating the index Return actual frames and construct new video of just changes Compress the resulting index by detecting start and end of change Person entering and exiting frame 18

19 Additional Information Resources Infocus Blogs 19

20