Big Data Analytics in Space Exploration and Entrepreneurship

Transcription

1 Space Society of Silicon Valley Big Data Analytics in Space Exploration and Entrepreneurship Tiffani Crawford, PhD January 14, 2015

2 Big Data Analytics Data Characteristics Large quantities of many data types Structured Semi structured Unstructured Human Machine 2 Big Data Analytics in Space Exploration and Entrepreneurship

3 Big Data Analytics Valuable Insights Getting to Value Quantitative Qualitative Correlation Longitudinal Social Search Operational Inferential Ethnographic Interview based Causal 3 Big Data Analytics in Space Exploration and Entrepreneurship

4 Big Data Analytics Valuable Insights (x + y)/y + <10 sentiment blogs x/2 * 5 Data Points * 3 if over 18 years old Getting to Value Quantitative Qualitative Correlation Longitudinal Social Search Operational Inferential Ethnographic Interview based Causal 3 Big Data Analytics in Space Exploration and Entrepreneurship

5 Big Data Analytics Proven in Industry Satellites and other geospatial systems Image processing and analysis Research & Development High technology and Internet Social media Telecommunications Knowledge bases and content systems Life sciences, health care and genetics Infrastructure systems and monitoring Operations and transportation Financial systems and compliance controls 4 Big Data Analytics in Space Exploration and Entrepreneurship

6 Big Data Analytics Key Benefits High speed data processing 1 big job => many small jobs (reduce complexity) Redundancies prevent data loss Temporary storage is faster Long term storage is easier to access High speed query and results processing (<10 sec) High speed predictive analytics & responsive models Low cost (2 10x cheaper) up to massive scale Easy to start small and practice before large builds Technology agnostic Open source, cloud or large supplier install base Proof of Concept, Proof of Value, Proof of Technology Early analytics modeling, Proof of Model 5 Big Data Analytics in Space Exploration and Entrepreneurship

7 Big Data Analytics Technology-Agnostic Space-Specific Big Data Analytics Execution Components Business Context Information Delivery Analytics & Insights Processing Service Offerings Integration & Management Consulting Data Governance Top 4 Solution Research & Engineering Development Big Data Dimensions Volume Variety Complexity Velocity Space Exploration Space Research & Development Space Entrepreneurship Commercial Market Growth Business Transformation 6 Big Data Analytics in Space Exploration and Entrepreneurship

8 Big Data Analytics Data Science Making sense of data is both Art and Science Generate better insights Gain confidence in decisions Visualize the data Understand the data and communicate it Learn how to learn and adapt with agility Methods Data mining Machine learning Artificial intelligence Information retrieval Statistical analysis Gap analysis 7 Big Data Analytics in Space Exploration and Entrepreneurship

9 Big Data Analytics Activity Visualization 8 Big Data Analytics in Space Exploration and Entrepreneurship

10 Big Data Analytics Capacity Visualization Capacity Management (Interstate Traffic Core) Capacity Management (Intrastate Traffic Transport) = Content Provider Network Sites = Interstate Traffic (CORE) = Interconnections (ENNI, NNI, Colo, etc.) Leased Facilities = Content Provider Network Sites = Intrastate Traffic (Transport) = Interconnections (ENNI, NNI, Colo, etc.) Leased Facilities = Interstate Traffic (CORE) 9 Big Data Analytics in Space Exploration and Entrepreneurship

11 Big Data Analytics Resource Availability 10 Big Data Analytics in Space Exploration and Entrepreneurship

12 Who uses Hadoop and NoSQL? 11 Big Data Analytics in Space Exploration and Entrepreneurship

13 Technology Adoption Maturity General Low Business Insight and Value High Implement Learn Low Optimize Big Data store established Technology evaluation Technical POCs Reports, Analysis from Big Data First set of Business Data integrated Basic infrastructure Knowledge and skills Business POCs, Pilots Pattern analysis and discovery Social and sentiment analysis enabled Innovate Comprehensive data platform for Data Scientists Technology Vendor ecosystem Big Data focus group Big Data Cluster established Business Analytics on Structured data enabled Text parsing and analysis Big Data Technology Adoption Level Big Data Analytics driven Business Innovation Business strategy modeling Unstructured data integrated Deep data mining Predictive modeling and decision insight Machine Learning For pattern discovery Big Data Roadmap, engagement model Big Data Architecture, design standards High 12 Big Data Analytics in Space Exploration and Entrepreneurship

14 Forrester Wave Enterprise Hadoop 13 Big Data Analytics in Space Exploration and Entrepreneurship

15 Gartner Magic Quadrant DW + Hadoop 14 Big Data Analytics in Space Exploration and Entrepreneurship

16 Gartner Magic Quadrant BI + Hadoop 15 Big Data Analytics in Space Exploration and Entrepreneurship

17 Common Misconceptions Big does not mean that it s a monolith project. Start small and grow. Capacity management planning is easier. Waterfall is not required simply because it s Big. If you love waterfall, you can still see regular reporting without your development teams suffering through the methodology, risks and other implications. Few clients are actually Dinosaurs. Really. Hype does not mean that more burn in is needed. Hadoop has been in use in enterprises since 2005, picked up steam in 2007 and expanded exponentially each year since then. Big Data processing or storage alone does not bring the business value. Analytics without data architecture, governance, validation and quality do not bring the business value. 16 Big Data Analytics in Space Exploration and Entrepreneurship

18 Big Data Analytics Space Systems Data Space Interoperability Systems ISS, Supply Rockets, Mission Monitoring Satellites, Orbit Monitoring Tourism, Rescue Space Based Telescopes and Earth Monitoring Systems Hubble PlanetLabs, GPS Systems Space Manufacturing 3D Printing Systems Operational Intelligence Platforms Astrobiology and Space Medicine Human Monitoring, IoT Predictive Diagnostics Highly Curated Information Systems Auto Adapting Software/Networks 17 Big Data Analytics in Space Exploration and Entrepreneurship

19 Future Space Systems Data INCOMING! Transactions per Second (TPS) in Terabytes (TB) Storage per Day or Year in Petabytes (PB) or Exabytes (EB) Hadoop nodes 30+ planned Data source systems 10+ Dedicated nodes Multiple data hubs Volume Multiple datacenters Variety Structured Semi Structured Unstructured Real time data from social networks Mobile data Machine data Imaging data External data Data streaming Velocity Data processing in Real Time (<5 minutes) or Near Real Time (NRT) (<30 minutes) Query and results in <10 seconds Visualization streaming Business reporting Application processes and networks Complexity Predictive statistics Data architecture Data pipeline analysis Computational models Insight granularity Query and results packaged for usability and interoperability for analytics applications 18 Big Data Analytics in Space Exploration and Entrepreneurship

20 Future Space Systems Big Data Analytics Business Value = VVCV + SQVV + VVVV Security Data models, masking, encryption, system architecture Quality Data architecture and governance over streaming data Verification Determine source on streaming data intake Validation Current data values, complete transmissions 19 Big Data Analytics in Space Exploration and Entrepreneurship

21 Future Space Systems Big Data Analytics Business Value = VVCV + SQVV + VVVV Veracity and Data Story Truth Indicative meaning in prediction Variability Data types, values, ranges and formats Virtualization Virtual machines and network management Vexation Inherent resilience, % uptime, ease of support 20 Big Data Analytics in Space Exploration and Entrepreneurship

22 Big Data Analytics Space Systems Data Data Acquisition Data Processing Data Storage Query Intake Query Processing Data Retrieval Auditing and Reporting Central Logs, Device Logs Web NW Data Data Query Data Processing Cores Hadoop, NoSQL In-Memory Data Quality and Curation Query Analysis Resilient Storage Analytics Processing Packaged Query Results 21 Big Data Analytics in Space Exploration and Entrepreneurship

23 Space Entrepreneurship Opportunities Space exploration and environment data management Space systems operational data management Space investment data validation (mining studies, technology) Economic development analytics and decision making Data query results systems with fast processing power Genetics management for selection and health care Space science and business value delivery Research process automation, analysis and discovery Intellectual property development Contingency and emergency modeling before human cases Data ubiquity, security and privacy 22 Big Data Analytics in Space Exploration and Entrepreneurship

24 How to Innovate, Streamline and Grow? New data types and workflows Cluster management and node forecasting Automated deployment, network and application process monitoring Replay, backup, recovery and failover Cloud Social media and mobile integration SDN (software discovers needs and provisions the system) 23 Big Data Analytics in Space Exploration and Entrepreneurship

25 How to Frame Our Thinking? What are we trying to accomplish? How is this valuable? What data questions do we have now? What questions may follow? Can the opportunity or problem benefit from Big Data Analytics? Does the solution have growing room? 24 Big Data Analytics in Space Exploration and Entrepreneurship

26 Use Cases Near-Term Business Value 25 Big Data Analytics in Space Exploration and Entrepreneurship Mission to Mars Data system automation Robotics Life support Space Debris Image analysis Trajectory change alerts Satellite down alerts Material analysis Investment validation Space Manufacturing Process optimization Intelligent automation 3D print share

27 Use Cases Exploration and Settlement Locations and Objects Monitoring orbits Monitoring Lagrange Points Image analysis Kepler analysis Asteroid analysis Transportation Trajectory and path Fuel, power and life support Acquisitions en route High velocity options Communications Health impacts of space 26 Big Data Analytics in Space Exploration and Entrepreneurship

28 Use Cases Operationalizing Space Space data centers Unlimited storage Processing power Adaptive software Automation Space systems Intelligent controls Monitoring Maintenance Resource allocation Planning Research on demand Predictive analysis 27 Big Data Analytics in Space Exploration and Entrepreneurship

29 Use Cases Life Sciences & Health Care Genetics Genome analysis Health prediction Selection of pairs Tailoring for environments Biological Adaptation for Space Investigation of immunities Uses of bacteria Pre terraforming studies Find reality in futurist ideas Health Improvement Capabilities of mind Extended life 28 Big Data Analytics in Space Exploration and Entrepreneurship

30 Use Case Social Media and Sentiment Get a Big Space Program Up! Funding Investment Support Entrepreneurship Prove the Value of Space! Validate social models Design alternative living Explore compatibility for space Collaborate around the Earth Adapt policies before needing them Advance human learning 29 Big Data Analytics in Space Exploration and Entrepreneurship

31 References Search online using the terms and logos in this presentation. ActionEconomy is the only company in the world that builds Big Data Analytics platforms for Space. Visit the startup at Big Data Analytics Project Management book by Tiffani Crawford Data Analytics Project Management/dp/ Hadoop NoSQL database.org/ AWS Ecosystem Data Science Agile Scrum 30 Big Data Analytics in Space Exploration and Entrepreneurship

32 Contact Information Tiffani Crawford, PhD Tiffani Crawford, PhD, builds global Big Data Analytics systems. She has 20 years of high technology experience with Fortune 500 companies, including Cisco Systems, Facebook, Microsoft/Skype, Bank of America, VISA/Inovant, BAE Systems, Shoretel, Ditech Networks, Starview, ActivIdentity, Applied Competitive Technologies, Big 4, defense contractors and startups. She has developed groundbreaking technologies in Big Data, analytics, cloud, networking, telecommunications, software development, mobiles, cloud security, ediscovery, transaction systems, distributed multi tier applications, digital entertainment, geographic information systems/gps, intelligent transport systems and finance. She is a credited software developer in Java for network element management and a published author. She earned her PhD from the University of Southern California in She has also earned her Master's, Bachelor's and various technology certifications. 31 Big Data Analytics in Space Exploration and Entrepreneurship