Big Data to Decision. Thomas E. Potok, PhD Group Leader Computational Data Analytics Group Oak Ridge National Laboratory

Transcription

1 Big Data to Decision Thomas E. Potok, PhD Group Leader Computational Data Analytics Group Oak Ridge National Laboratory

2 Computational Data Analytics Group Research 10 years in data mining and machine learning Last year organized 7 research workshops and published 70 research papers 2 Managed by UT-Battelle Key University Partnerships University of Tennessee Center for Intelligent Systems and Machine Learning North Carolina State University Emory, University of Chicago, Georgetown Key Government and Industrial Partnerships Office of Naval Research Department of Homeland Security Department of Defense Intelligence Community Department of Energy Lockheed Martin Battelle Memorial Institute Technology Transfer People and Awards Piranha license to TextOre Piranha license to Lockheed Martin ORCA license to Lockheed Martin 25 staff members, 15 PhDs in computer science R&D 100 Award (Oscars of invention) in 2007

3 Recent News Stories and Programs Major Projects: Centers for Medicare and Medicaid Services Objectives:! Facilitate easy, consistent analysis for CMS staff and stakeholders! Decrease turn-around time from data to decision! Give analysts insight into opportunities for policy improvement!"#$%&'()*+,-$"+(.&(/0.1*"#0(*$2#0-3%.$+4( Future:! Drive the ability to inform policy and policy makers based on near real-time data! Provide the ability to perform predictive modeling and simulation of policy scenarios, (outcomes and impacts)! Supply tools and techniques for social network, risk, and other advanced analysis 50.6(,#78(( /#0+/#3%1#+(&.0(B#.<0-/?*3(C-0*-%.$( ( D0.1*"#(*$2#0-3%1#(0#/.02+(-$"(1*+E-,*:-%.$+(2.(#F2#0$-,( E+#0+( (!"#$%&'(-G0*>E2#+(-30.++(6-++*1#(-6.E$2+(-$"(2'/#+(.&("-2-( 7 3 Managed by UT-Battelle Managed by UT-Battelle Big Data Intrusion Detection

4 Data is a blessing or curse Warning signs existed The data was never fully understood 4 Managed by UT-Battelle

5 The people who understand their data will succeed Data Channels RSS feeds Blogs Twitter Linkedin Youtube Facebook The cost for not keeping up is high The value of being informed is high Ahhh, I am not, ahhh what was that link again... ahhh, let me get back to you 5 Managed by UT-Battelle

6 Recommendations: Apple itunes Genius Arranges your music Recommends new music based on your music library 6 Managed by UT-Battelle

7 Personalized Music Recommendation My List Stairway to Heaven - Led Zeppelin Johnny B. Goode - Chuck Berry Like A Rolling Stone - Bob Dylan Playlist 2 Johnny B. Goode - Chuck Berry Won't Get Fooled Again Who All Along The Watchtower Jimi Hendrix Playlist 3 Purple Haze - Jimi Hendrix Whole LoFa Love - Led Zeppelin (I Can't Get No) SaGsfacGon Rolling Stones Term List Stairway Heaven Led Zeppelin Johnny Goode Chuck Berry Like Rolling Stone Bob Dylan Wont Fooled Who Along Watchtower Jimi Hendrix Purple Haze Whole LoFa Love SaGsfacGon Vector Space Model Term List 1 List 2 List 3 Stairway Heaven Led Zeppelin Johnny Goode Chuck Berry Like Rolling Stone Bob Dylan Wont Fooled Who Along Watchto wer Jimi Hendrix Purple Haze Whole LoFa Love SaGsfacG on Similarity Matrix!"#$%&!"#$%'!"#$%(!"#$%&!""# $"#!"#!"#$%'!""#!!#!"#$%(!""# Genius Recommendations You bought music by Chuck Berry Won't Get Fooled Again Who You bought music by Chuck Berry All Along The Watchtower Jimi Hendrix You bought music by Led Zeppelin Whole Lotta Love Led Zeppelin 7 Managed by UT-Battelle

8 Piranha Document Analysis - Terms are weighted according to their frequency. - TF-ICF was used for this work as an example of one weighting algorithm that does not require a static corpus. - A document vector is a VSM representation of the document s terms and associated weights. Similarity Matrix Doc 1 Doc 2 Doc 3 Doc 1 100% 17% 21% Doc 2 100% 36% Doc 3 100% Documents to Documents Euclidean distance Cluster Analysis D1 D2 D3 Most similar documents Time Complexity O(n 2 Log n) 8 Managed by UT-Battelle

9 Term Frequency Weights Term Frequency/Inverse Document Frequency Document Frequency Significant Term centrifuge InteresGng Term mechanism Stop word the Theme word Obama Set Frequency Inverse Document Frequency 9 Managed by UT-Battelle Inverse Document Frequency Strengths Finds significant terms in a random set of document Weakness High O Does not work well for single topic set

10 Term Frequency/Inverse Corpus Frequency Document Frequency Significant Term centrifuge or words not in the corpus InteresGng Term mechanism Stop word the Theme word Obama Corpus Frequency Inverse Corpus Frequency Inverse Corpus Frequency Strengths Finds significant terms in common topic sets Parallel Weakness Stop words Managed by UT-Battelle 0 for the U.S. Department of Energy

11 Challenge Highly weighted terms, but not significant terms "computer", "ieee", "architecture", "data", "algorithms", "applications", "submitted", "researchers", "energy", "due:", "device", "infrastructures", "library", "methods", "optimized", "symposium", "version", "systems", "processes", "models", "present", "paper", "performance", "scale", "intern", "technology", "communities", "result", "large", "distributed", "standards", "september", "july", "provide", "annual", "interests", "areas:" Add a query for each term against the repository Terms that generate few returned documents, or low IDF and deemed significant 11 Managed by UT-Battelle

12 Personalized Content Recommendations Computational Data Analytics Group Main Idea: What you see is what you want What you see is what you get 12 Managed by UT-Battelle AFenGon Time User Interest Personalized RecommendaGons Problem Statement: How to detect user interests and automatically recommend interesting contents in a personalized way. Technical Approach: Detecting user interests through attention time, i.e. time spent by a user on reading a certain webpage. Collaboratively mining semantic contents of user reading materials along with one s implicit feedbacks. Advanced data fusion algorithms for user interest inference. Dynamic content recommendations according to inferred user interest profile. Advantage over the State-of-the-Art: Leverage an ontology based approach for noise tolerant user interest inference. Can autonomously recommend interesting contents to end users without explicit user participation. Capable of detecting dynamic user interest shift fully automatically and adjust algorithm behaviors accordingly.

13 Piranha Cluster View Report Date: 1 April, FBI: Abdul Ramazi is the owner of the Select Gourmet Foods shop in Springfield Mall, Springfield, VA. [Phone number ]. First Union National Bank lists Select Gourmet Foods as holding account number Six checks totaling $35,000 have been deposited in this account in the past four months and are recorded as having been drawn on accounts at the Pyramid Bank of Cairo, Egypt and the Central Bank of Dubai, United Arab Emirates. Both of these banks have just been listed as possible conduits in money laundering schemes 13 Managed by UT-Battelle

14 PiranhaG - 1 million documents in 12 minutes on a GPU cluster 30-fold performance improvement in text analysis Researchers in the Applied Software Engineering Group (ASER) in collaboration with North Carolina State University has created a cluster of 1M documents in 12 minutes using a 6 node GPU cluster 14 Managed by UT-Battelle

15 PiranhaX: Petascale text analysis ORNL s Jaguar is the 2nd fastest computer in the world 255,000 cores -10PB (13,400 1TB drives) of Storage -362TB of memory Google has indexed 1 Trillion unique URLs, but has not analyzed the content of the information We are currently developing petascale text analysis techniques to cluster (deep analysis) of 1 trillion documents using Jaguar 15 Managed by UT-Battelle

16 VERDE: NOM: Visualizing Energy Resources Dynamically on Earth National Outage Map Capability: Platform provides wide area spatiotemporal electric grid situational awareness Situational awareness of transmission lines (above 230KV) Situational awareness of distribution outages (status of approximately 40 Million customers served) Wide-Area Power Grid Situational Awareness Streaming Data Impact Models and Data Analysis Real-time weather overlays Predictive and post-event impact modeling and simulation Data analysis Energy infrastructure views Population impacts 16 Managed by UT-Battelle Distribution Outages Analysis Real-time Weather Overlays

17 TRACS: The Resiliency Analysis & Coordination System Computational Data Analytics Group Problem Statement: Critical Humanitarian Assistance/Disaster Recovery data resides in multiple domains Growing HA/DR information in social media Disaster response requires real-time access to common community information Technical Approach: Apply Web 2.0 and social media technologies for sharing heterogeneous data across organizations Allow mapping of data to one or more assessment frameworks to track progress towards stated goals Advantage over state-of-the-art: TRACS contextualizes social media, crisis mapping, and network analysis data within one or more common societal models Provides visualization and intuitive displays to support pre- and post-event analysis 17 Managed by UT-Battelle 17

18 Human Expert Interpretation of Images Images courtesy of Memorial Sloan-Kettering Cancer Center via 18 Managed by UT-Battelle

19 Mammography Data Temporal Aspects Two latest normal reports Two latest suspicious reports Band Coefficients Band 1 identifies recent abnormalities Band 2 identifies early abnormalities Wavelet transform of the sequence of abnormal s-grams counts 19 Managed by UT-Battelle

20 Knowledge Discovery in Linked (Big)Data Computational Data Analytics Group RDF, SPARQL, OWL etc. Data SEEKER: Schema Exploration and Evolving Knowledge Recorder Models and Algorithms Data association, Prediction, Summarization, Data fusion Inference, Network-analytic behavior extraction, etc. Inference Emergent behaviors SNAKE: Social Network Analytic Knowledge Extraction Problem Statement: Data is not rectangular it is freeform. Knowledge is buried in associations (links) of disparate data. How can we find what is interesting from the data? What is the data trying to tell us? Technical Approach: Domain Digitization Machine traversable (and evolving) data model as taxonomies, vocabulary and links. Automated data schema exploration and analysis. Graph-theoretic space-time-topological model Captures relationships, attributes, and temporal/spatial variations. Change detection in time-varying graphs. Statistical inference and visualization Hypothesis tests for anomalous behavior. Advantage over State-of-the-Art: Domain-inclusion makes knowledge discovery sustainable over time. Change-management strategy recommendation. Sensing emergent behaviors. 20 Managed by UT-Battelle

21 Summary Processing large volumes of text is a challenging problem Missing information is expensive Discovering information is profitable Piranha can keep you from missing information, and help you to discover new valuable information 21 Managed by UT-Battelle