1 Big Data R&D Initiative Suzi Iacono CISE Directorate National Science Foundation National Academies of Science Integrating Environmental Health Data to Advance Discovery January 11, 2013 Image Credit: Exploratorium.
2 Advances in information technologies are transforming the fabric of our society and data represents a transformative new currency for science, engineering, education and commerce. Image Credit: CCC and SIGACT CATCS
3 Where do the data come from? Why do we have a national initiative?
4 The Big Data Landscape I: Big Science Science gathers data at an ever-increasing rate across all scales and complexities of natural phenomena Sloan Digital Sky Survey in 2000 collected more data in its 1 st few weeks than had been amassed in the entire history of astronomy Within a decade, over 140 terabytes of information collected Large Hadron Collider generates scores of petabytes a year The proposed Large Synoptic Survey Telescope (3.3 gigapixel digital camera) will generate 40 terabytes of data nightly By 2015, the world will generate the equivalent of approximately 93 million Libraries of Congress
5 The Big Data Landscape II: Smart Sensing, Reasoning and Decisionmaking Environment Sensing Emergency Response Percepts (sensors) Agent (Reasoning) Situation Awareness: Humans as sensors feed multimodal data streams Credit: Photo by US Geological Survey People Centric Sensing Actions (controllers ) Pervasive Computing Social Informatics Smart Health Care Evaluate Sense Personal Sensing Public Sensing Social Sensing Intervene Assess Identify Source: Sajal Das, Keith Marzullo
6 The Big Data Landscape III: New Paradigms for Communications 1988 Remarkable Pace of Innovation Today MOBILE SOCIAL NETWORKS VIDEO VOIP BLOGS
7 Communications Volume & Traffic Diversity VoIP Video Twitter Broadband 663M registered Skype users in Represents 20% of long distance minutes world-wide. If Skype were a carrier, it would be the 3rd largest in the world (behind China Mobile and Vodaphone). Largest provider of cross-border communication. Recent estimates as high as 60% of internet traffic is video and music sharing; 35 hours of new videos are uploaded every minute in 2011; 2 billion views per day. Currently 175 million registered users. 20% of global internet users have residential broadband; d; 68% in US subscribe be to broadband. d Mobile 5.3 billion mobile phone subscribers; 85% of new handsets will be able to access the mobile web; 1 in 5 has access to fast service, 3G or better; IM, MMS, SMS expected to exceed 10 trillion message by 2013.
8 The Big Data Landscape IV: The Long Tail of fscience Hundreds of thousands of scientists and engineers work individually or in small, distributed, disconnected groups all generating data that collectively represent an enormous, largely untapped scientific resource From running simulations, experiments, etc. Making heterogeneous data across many areas of science more homogeneous could give way to breakthroughs across all areas of science and engineering Estimated 40 exabytes of unique new information generated worldwide in 2010 Only 5% of the information created is structured, however, in a standard format of words or numbers; the rest are unstructured text, voice, images, etc.
9 How Big is Big? Big Data : Datasets whose size are beyond the ability of typical database software tools to capture, store, manage, and analyze -McKinsey Global Institute, Big data: the next frontier for innovation, competition, and productivity, May Image Credit: Sigrid Knemeyer
10 Not Just Volumes of Data The science of big data is not just about volumes and velocity of data, but also Heterogeneity and diversity Levels of granularity Media formats Scientific ifi disciplines i Complexity Uncertainty Incompleteness Representation types
11 Why is Big Data Important? Critical to transforming how science is done and to accelerating the pace of discovery in almost every science and engineering discipline Transformative implications for commerce and economy Potential for addressing some of society s most pressing challenges Image Credit: Chi Birmingham
12 Paradigm Shift: from Hypothesis-driven to Data-driven Discovery The Economist, The data deluge and how to handle it: A 14 page special report (Feb 25, 2010). The Fourth Paradigm: Data Intensive Scientific Discovery (2009, Microsoft Corporation).
13 The Age of Data: From Data to Knowledge to Action Data-driven discovery is revolutionizing i i scientific exploration and engineering innovations Automatic extraction of new knowledge about the physical, biological and cyber world continues to accelerate Multi-cores, concurrent and parallel algorithms, virtualization and advanced server architectures will enable data mining and machine learning, and discovery and visualization of Big Data
14 Potential for Transformational Science & Engineering: From Data to Knowledge to Action Integration ti of discipline i (or media format ) specific data, examine for relationships Disaster informatics 3D toxic fume images Simulations of gas spread Maps of census concentrations First responder on-the- ground findings Evacuation routing
15 From Data to Knowledge to Action Researchers seek to fundamentally transform understanding of spinning giants The task involves assembling data from more storm variables-such as updraft, downdraft and vorticity or g p, y regions of spin--than what can be observed from ground tornado chasers or even actually produced in the atmosphere. For example, researchers need to better understand how changes in wind direction with height cause the updrafts in a storm to rotate, preceding the formation of a tornado. To solve the quandary, Amy McGovern, an associate professor in OU's School of Computer Science, and her team create tornado models with super computers that can process vast amounts of data. McGovern and her colleagues use the models to analyze how storm variables interact in order to identify tornadic and nontornadic storms.
16 Examples of Research Challenges More data are being collected than we can store Analyze the data as it becomes available Decide what to archive and what to discard Many data sets are too large to download Analyze the data wherever it resides Many data sets are too poorly organized to be usable Better organize and retrieve data Many data sets are heterogeneous in type, structure, semantics, organization, granularity, accessibility Integrate and customize access to federate data Utility of data is limited by our ability to interpret and use it Extract and visualize actionable knowledge Evaluate results Large and linked datasets may be exploited to identify individuals Design management and analysis with built-in i privacy preserving characteristics
17 A National Imperative PCAST calls on the Federal government to increase R&D investments for collecting, storing, preserving, managing, g, analyzing, and sharing the increasing quantities of data. Furthermore, PCAST observed that the potential to gain new insights to move from data to knowledge to action has tremendous potential to transform all areas of national priority. Source: PCAST (December 2010), Report to the President and Congress: Designing a Digital Future a periodic congressionally-mandated review of the Federal Networking and Information Technology Research and Development (NITRD) Program.
18 Administration s Big Data Research and Development Initiative Big Data Senior Steering Group chartered in spring 2011 under the Networking and Information Technology R&D (NITRD) Program Members from DARPA, DOD OSD, DHS, DOE-Science, HHS, NARA, NASA, NIST, NOAA, NSA, OFR, USGS, etc. Co-chaired by NSF (and NIH) Initial charge was to come up with a plan, a strategy 18 Image Credit: Fuqing Zhang and Yonghui Weng, Pennsylvania State University; Frank Marks, NOAA; Gregory P. Johnson, Romy Schneider, John Cazes, Karl Schulz, Bill Barth, The University of Texas at Austin
19 Big Data Membership Biven Laura DOE Science Blatecki Alan NSFNational Science Foundation Collica Leslie NISTNational Institute of Standards and Technology Deift Abby NSFNational Science Foundation Downing Gregory HHSDepartment of Health and Human Services Espina Pedro OSTPWhite House Office of Science and Technology Policy Gerr Neil DARPADefense Advanced Research Projects Agency Gundersen Linda USGSU.S. Geological Survey NOAANational Oceanic and Atmospheric Hall Alan Administration Iacono Suzanne NSFNational Science Foundation Jakubek David OSDOffice of the Secretary of Defense ATL Kaufman Daniel DARPADefense Advanced Research Projects Agency Larson OSTPWhite House Office of Science and Technology Phillip P. Policy Lee Tsengdar NASANational Aeronautics and Space Administration NIHNational Institutes of Health /NLMNIH s National Lipman David Library of Medicine /NCBI Little Michael NASANational Aeronautics and Space Administration Luker Mark NCONational Coordination Office for NITRD /NITRDNetworking and Information Technology Research and Development Marth Lisa NISTNational Institute of Standards and Technology Muoio Patricia A. DNI Pantula Sastry NSFNational Science Foundation NIHNational Institutes of Health /NLMNIH s National Preuss Don Library of Medicine /NCBI Quade Brittany NSFNational Science Foundation Romine Charles NISTNational Institute of Standards and Technology Smith Darren NOAANational Oceanic and Atmospheric Administration Spengler Sylvia NSFNational Science Foundation Statler Tom NSFNational Science Foundation Strawn George NCONational Coordination Office for NITRD /NITRDNetworking and Information Technology Research and Development Suskin Mark NSFNational Science Foundation Villani Jennifer NIHNational Institutes of Health /NIGMS Wigen Wendy NCONational Coordination Office for NITRD /NITRDNetworking and Information Technology Research and Development Zhao Fen NSFNational Science Foundation Nowell Lucy DOE Science
20 Big Data Membership Bristol Sky USGSU.S. G eological Survey usg s.gov Kielm an Joseph DHSDepartment o f H om eland Security jo Petters Jonathan DOE Science science.doe.gov Carver Doris NSFNational Science Found ation d v Adolfie Laura O SD O ffice of the Secretary of Defense osd.mil Chadduck Robert NSFNational Science Foundation nsf.gov Crow der Grace NSANational Security Agency nsa.gov Dunn M ich e lle NIHNational In st itu t es of Health dunnm mail.nih.gov Florance V a le rie NIHN a tional In stitu t es o f H ealth mail.n ih.go v Frehill Lisa O SD O ffice of the Secretary of Defense lisa.fr eh ill.c d ar p a.m il Helland Barbara DOEDepartment of Energy Science science.doe.gov Hoang Thuc DOEDepartment of Energy NNSA nnsa.doe.gov Kannan Nandini NSFNational Science Foundation W arnow T and y NSFN a tional S c ie nce F ound a tion t nsf.g o v Dean David DOE Science science.doe.gov Ly ster Peter NIHNational In st itu t es of Health m ail.n ih.go v M ille m a ci John O SD O ffice of the Secretary of Defense jo osd.m il Pearce Claudia NSANational Security A gency nsa.gov Allen M arc NASANationalAeronautics Aeronautics and Space Administration marc.a nasa.gov Pearl Jennifer NSFNational Science Foundation jslim ow nsf.gov Blaszkowsky David TreasuryDepartment of the Treasury OFR david.blaszkow treasury.gov Szykm an James EPAEnvironm ental Protection Agency ja Tom pkins Jerry NSANational Security Agency gsto m p rad ium.ncs c.m il Flood M ark TreasuryDepartment of the Treasury OFR ry.g ov Holm Jeanne Data.gov je anne.m.holm.jpl.nasa.gov Misawa Eduardo NSFNational Science Foundation
21 Big Data Launch Federal Big Data R&D Initiative launched by White House OSTP on March 29, 2012 at AAAS Federal Announcements: NSF Subra Suresh NIH Francis Collins USGS Marcia McNutt DoD Zach Lemnios DARPA - Ken Gabriel DOE William Brinkman Panel Discussion: Moderator - Steve Lohr, New York Times Daphne Koller, Stanford University James Manyika, McKinsey & Company Lucila Ohno-Machado, UC San Diego Alex Szalay, Johns Hopkins University Image Credit: National Science Foundation More information available at:
22 Strategy to Address Big Data Foundational research to develop new techniques and technologies to derive knowledge from data New cyberinfrastructure to manage, curate, and serve data to research communities ii Policy New approaches for education and workforce development New types of inter disciplinary collaborations, grand challenges, and competitions
23 Core Techniques and Technologies for Advancing Big Data Science & Engineering (BIG DATA) Program Solicitation: NSF Foundational research to extract knowledge from data Foundational research to advance the core techniques and dtechnologies for managing, analyzing, visualizing, and extracting useful information from large, diverse, distributed and heterogeneous data sets. Image Credit: Jurgen Schulze, Calit2, UC San Diego Cross Directorate t Program: NSF Wide Multi agency Commitment: NSF and NIH
24 BIG DATA Research Thrusts Collection, Storage, and Management of Big Data 3 awards Foundations of big data management Mitigating tradeoffs among speed of data ingestion, quicker answers and the freshness of data through the design of new storage devices with extreme capacities 4 awards Data Analytics Novel machine learning where multi-dimensional vector data points are replaced by distributions Design and test mathematical and statistical techniques for large-scale heterogeneous data in DNA repositories Research in Data Sharing and Collaboration 1 award (+1 shared with data collection) Open source tools for infrastructure for improving discovery through use of social analytic data Databridge linking data, human interactions, and usage practices for the long-tail of science Databridge linking data, human interactions and usage practices for the long-tail of science Data analytics problems in next generation sequencing Theory and algorithms fro couples tensors and associated software toolkits to make analysis possible Credit: Fermilab Photo Eight mid scale (up to $1M a year) awards out of over 136 projects announced on Oct. 3.
25 Award Citations DCM: Dan Suciu University of WA A formal foundation for big data management Michael Bender SUNY at Stony Brook & Martin Farach-Colton Rutgers University Eliminating the data ingestion bottleneck in big data applications Arcot Rajasekar University of North Carolina, Chapel Hill & Gary King Harvard University & Justin Zhan North Carolina Agriculture & Technical State University it Databridge A sociometric system for long-tail science data collections 25
26 Data Analytics Award Citations Eli Upfal Brown University Analytic approaches to massive data computation with applications to genomics Aarti Singh Carnegie-Mellon University Distribution-based machine learning for high dimensional datasets Srinvas Aluru Iowa State University & Wuchun Feng Virginia Polytechnic Institute & State University & Oyekunie Olukotun Stanford University Genomes Galore Core techniques, libraries, and domain specific languages for high throughput DNA sequencing
27 Award Citations Data Analytics (continued) Christos Faloutsos Carnegie Mellon University & Nikolaos Sidiropoulos University of Minnesota Twin Cities Big Tensor Mining: Theory Scalable Algorithms and Applications 27
28 Award Citations E-Science Collaboration Environments Thorsten Joachim Cornell University & Paul Kantor Rutgers University Discovery and social analysis for large-scale scientific literature 28
29 Ideation Contest Launch Opportunity to expand the innovation ecosystem Joint among NASA, NSF and DOE Office of Science A contest focused on How to make heterogeneous data seem more homogeneous? 5 judges 5 criteria Launched on Challenge.gov and the Top Coder platform on Oct. 3 with a two week window topcoder com/coeci/nitrd/
30 Ongoing Big Data Programs at NSF Dear Colleague Letters: Encourage CIF21 IGERTs to educate and support a new generation of researchers able to address fundamental Big Data challenges: Data-Intensive t Education-Related t d Research Funding Opportunities announcing an Ideas Lab, for which cross disciplinary participation will be solicited, to generate transformative ideas for using large datasets to enhance the effectiveness of teaching and learning environments: Data Citation to the Geosciences Community to encourage transparency and increased opportunities for the use and analysis of data sets:
31 Earthcube: GEO Science Infrastructure EAGER awards announced as part of White House Big Data Launch Integrates geosciences data and high-performance computing technologies in an open, adaptable and sustainable framework to enable transformative research and education in Earth System Science Innovative Model: Community designed, community owned, community governed Interdisciplinary research: Building and sustaining new communities Workshops to bring together (GEO, SBE, CISE) communities EAGER awards to seed new research
32 A Complex Policy Setting Researchers want data. Public policy requires access to data. Public policy also requires protection of privacy and intellectual property and other sensitive information. Much more to be done: Policy on data management and data access.
33 Data Privacy Never more important than today However, not all data contain people s identities (as in data landscape III) Not a Big Brother scenario Government (NSF) invests in privacy research Values in design research community: Identity cloaking, anonymization Do-not-track cookie management Obfuscation, blurring Privacy preserving data mining, search, payment Just-in-time crypto Secure data distribution. Privacy in technology; privacy inspired technology
34 Opportunities for the Future Our investments in research and education have already returned exceptional dividends to the Nation. Many of tomorrow s breakthroughs will occur as a result of new techniques and technologies for advancing Big Data science and engineering. In turn, Big Data scientific discovery and technological innovation are at the core of our response to national and societal challenges from environment, energy, transportation, sustainability, and healthcare to cyber security and national defense.
HARNESSING THE POWER of DIGITAL DATA for SCIENCE AND SOCIETY Report of the Interagency Working Group on Digital Data to the Committee on Science of the National Science and Technology Council January 2009
Testimony of Farnam Jahanian, Ph.D. Assistant Director Computer and Information Science and Engineering Directorate Before the Committee on Science, Space, and Technology Subcommittee on Technology and
Big-Data Computing: Creating revolutionary breakthroughs in commerce, science, and society Randal E. Bryant Carnegie Mellon University Randy H. Katz University of California, Berkeley Version 8: December
PREPUBLICATION DRAFT Subject to Further Editorial Correction Frontiers in Massive Data Analysis Committee on the Analysis of Massive Data Committee on Applied and Theoretical Statistics Board on Mathematical
May 2011 Big data: The next frontier for innovation, competition, and productivity The McKinsey Global Institute The McKinsey Global Institute (MGI), established in 1990, is McKinsey & Company s business
Cover Page DEMYSTIFYING BIG DATA A Practical Guide To Transforming The Business of Government Prepared by TechAmerica Foundation s Federal Big Data Commission 1 TechAmerica Foundation: Federal Big Data
Trends in Information Technology Infrastructure in the Ocean Sciences OITI Working Group Members Tom Powell, University of California, Berkeley (chair) Mark Abbott, Oregon State University Scott Doney,
For Big Data Analytics There s No Such Thing as Too Big The Compelling Economics and Technology of Big Data Computing March 2012 By: 4syth.com Emerging big data thought leaders Forsyth Communications 2012.
ta REPORT TO THE PRESIDENT BIG DATA AND PRIVACY: A TECHNOLOGICAL PERSPECTIVE Executive Office of the President President s Council of Advisors on Science and Technology May 2014 REPORT TO THE PRESIDENT
Vol. 19 No. 4 2012 Building a national program for cybersecurity science Globe at a Glance According to the Experts Pointers Spinouts GUEST Editor s column Frederick R. Chang, PhD Considered by most to
New Data for Understanding the Human Condition: International Perspectives OECD Global Science Forum Report on Data and Research Infrastructure for the Social Sciences Data-driven and evidence-based research
AAPOR Report on Big Data AAPOR Big Data Task Force February 12, 2015 Prepared for AAPOR Council by the Task Force, with Task Force members including: Lilli Japec, Co-Chair, Statistics Sweden Frauke Kreuter,
Global Headquarters: 5 Speen Street Framingham, MA 01701 USA P.508.872.8200 F.508.935.4015 www.idc.com W H I T E P A P E R B i g D a t a : W h a t I t I s a n d W h y Y o u S h o u l d C a r e Sponsored
BIG DATA IN ACTION FOR DEVELOPMENT This volume is the result of a collaboration of World Bank staff (Andrea Coppola and Oscar Calvo- Gonzalez) and SecondMuse associates (Elizabeth Sabet, Natalia Arjomand,
B.2 Executive Summary As demonstrated in Section A, Compute Canada (CC) supports a vibrant community of researchers spanning all disciplines and regions in Canada. Providing access to world- class infrastructure
SOFTWARE ENGINEERING Key Enabler for Innovation NESSI White Paper Networked European Software and Services Initiative July 2014 Executive Summary Economy and industry is experiencing a transformation towards
JANUARY 2013 REPORT OF THE DEFENSE SCIENCE BOARD TASK FORCE ON Cyber Security and Reliability in a Digital Cloud JANUARY 2013 Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics
National Spatial Data Infrastructure Strategic Plan 2014 2016 Federal Geographic Data Committee December 2013 Federal Geographic Data Committee Federal Geographic Data Committee, Reston, Virginia: 2013
CHAPTER 1.7 Harnessing the Power of Big Data in Real Time through In-Memory Technology and Analytics SAP AG Companies today have more data on hand than they have ever had before. It is estimated that an
McKinsey Global Institute November 2012 The social economy: Unlocking value and productivity through social technologies Professional services The McKinsey Global Institute The McKinsey Global Institute
The Government Performance and Results Act (GPRA) (Public Law 103-62) requires Federal agencies to develop strategic plans setting forth missions, long-term goals, and means to achieving those goals, and
The Problem of Data Lori Jahnke and Andrew Asher Spencer D. C. Keralis with an Introduction by Charles Henry August 2012 Council on Library and Information Resources Digital Library Federation The Problem