Big Data for Good or Evil Lessons from the NSA PRISM Scandal Jason Bloomberg Chief Evangelist jason@enterpriseweb.com @theebizwizard Copyright 2014, EnterpriseWeb LLC EnterpriseWeb is a Registered Trademark of EnterpriseWeb LLC. All rights reserved. Chief Evangelist at EnterpriseWeb Formerly with ZapThink 2001 2013 About Jason Bloomberg Global thought leader in the areas of Cloud Computing, EA, & SOA Created the Licensed ZapThink Architect (LZA) SOA course & associated credential Blogger for DevX New book, The Agile Architecture Revolution, is now available! Sign up for my newsletter at http://www.enterpriseweb.com/newsletter Copyright 2014, EnterpriseWeb LLC 2 1
What are Big Data? Datasets whose size is beyond the ability of typical database software tools to capture, store, manage & analyze Copyright 2014, EnterpriseWeb LLC 3 2012 Big Data Technology Landscape Copyright 2014, EnterpriseWeb LLC 4 2
Today s Big Data are Tomorrow s Small Data? Definition intentionally subjective & moving definition of how big a dataset must be No fixed threshold As technology advances, size of datasets that qualify will increase Copyright 2014, EnterpriseWeb LLC 5 Big Data May Include Historical Data What about yesterday s data? If the amount of data doubles every two years, then half your data are always over two years old Copyright 2014, EnterpriseWeb LLC 6 3
Big Data Crisis Point Quantity & complexity of information The Big Data crisis point Ability to deal with quantity & complexity of information Time Copyright 2014, EnterpriseWeb LLC 7 Parkinson s Law (Big Data Corollary) Quantity of data will always expand to exceed available capacity for storing & processing it Copyright 2014, EnterpriseWeb LLC 8 4
Corollary to Parkinson s Law in action If you re not collecting Big Data, then someone else is The easier it is to collect Big Data, the more important it is to govern them If Someone Can Collect Big Data, then Someone Will Copyright 2014, EnterpriseWeb LLC 9 You must govern your metadata Metadata may even contain most of the business value Not just technical value Metadata governance at least as important as data governance Metadata may be Big Data as Well Copyright 2014, EnterpriseWeb LLC 10 5
Big Data analytics focuses on finding the nuggets of gold in the dross The data you don t want must still be governed, secured, & managed As Big Data sets grow, governing the dross is increasingly challenging Govern the Data You Don t Want Copyright 2014, EnterpriseWeb LLC 11 Not just valuable but dangerous Personally identifiable information Risk of false positives Big Data Results May be Dangerous Nuggets of Uranium, not Gold Copyright 2014, EnterpriseWeb LLC 12 6
Big Data Used to Mislead According to the figures published by a major tech provider, the Internet carries 1,826 Petabytes of information per day. In its foreign intelligence mission, NSA touches about 1.6% of that. However, of the 1.6% of the data, only 0.025% is actually selected for review. The net effect is that NSA analysts look at 0.00004% of the world s traffic in conducting their mission that s less than one part in a million. Put another way, if a standard basketball court represented the global communications environment, NSA s total collection would be represented by an area smaller than a dime on that basketball court. NSA Copyright 2014, EnterpriseWeb LLC 13 In Other Words 7.5 terabytes of analytical results to process manually every day Would be equivalent of Call Detail Records for 5 million calls every day per person on the planet! Copyright 2014, EnterpriseWeb LLC 14 7
NSA spies on data in the US, so Keep your data out of the US, right? Assumes: Your country isn t spying on you too! Your country isn t working with the NSA! The NSA can t spy on data outside the US! Wrong Conclusion? What are your Big Data policies? Copyright 2014, EnterpriseWeb LLC 15 Governance the Old Way Information Problem Tools Policies for using the tools Governance Copyright 2014, EnterpriseWeb LLC 16 8
Today s Data Governance (simplified) Our data are unclean! Great! Here are policies & processes for how to manage data quality using our tool. Let s use this data quality tool. Copyright 2014, EnterpriseWeb LLC 17 Governance the New Way Information Problem Tools Policies for using the tools Meta policies for dealing with governance Next generation governance tools Best practice approach to Big Data Crisis Copyright 2014, EnterpriseWeb LLC 18 9
Meta requirement Requirement that applies to other requirements E.g., Business Agility requirement Meta Thinking Meta methodology Methodology for creating or modifying methodologies Following the Agile principle responding to change over following a plan even if the plan is to follow Agile Meta policy Policy for how to perform governance Copyright 2014, EnterpriseWeb LLC 19 Dealing with Change Meta thinking doesn t look at something Meta thinking means looking at how something changes Meta thinking is typically manual Always includes people Copyright 2014, EnterpriseWeb LLC 20 10
Avoiding Hall of Mirrors Problem Meta policy: how to we automate policy enforcement? Meta meta policy: how to we automate meta policy enforcement? Answer: we don t (yet)! Copyright 2014, EnterpriseWeb LLC 21 Big Data Governance (even more simplified), part 1 We have too much information! Let s use this Big Data tool. Great! Here are policies & processes for how to use the Big Data tool. Uhh, our Big Data got too big for the tool. 22 Copyright 2014, EnterpriseWeb LLC 11
Big Data Governance, part 2 Dang. Here s our policy for how to deal with ever increasing quantities of data. Huh? We need a way to manage policies for dealing with ongoing Big Data challenges Copyright 2014, EnterpriseWeb LLC 23 Especially if central challenge is data quality Big Data sets tend to be unclean Structured, semi structured, & unstructured Good and bad mixed together The move from traditional analytics to Big Data analytics is a move to poorer levels of data quality Big Data Analytics Tools May be Governance Tools Copyright 2014, EnterpriseWeb LLC 24 12
Not your Parents Governance! Not just governance of technology Governance with technology Largely automated Proactive Inherently iterative Agile Copyright 2014, EnterpriseWeb LLC 25 Governance Leads to Empowerment The more powerful the tools, the more important it is that people know how to use them properly IT should empower the people in the organization Copyright 2014, EnterpriseWeb LLC 26 13
SOA Governance (Supposedly) Works this Way! SOA Policy Security Policies, Routing Policies, etc. Registry/ Repository ESB Meta policy Policies for handling governance in the reg/rep Copyright 2014, EnterpriseWeb LLC 27 Cloud shifts IT provisioning & management to the user Cloud automates previously manual tasks Greater risk of mucking things up Increased need for governance How Cloud Changes the Equation Copyright 2014, EnterpriseWeb LLC 28 14
Next Generation Governance Copyright 2014, EnterpriseWeb LLC 29 the Key to the Big Data Explosion Copyright 2014, EnterpriseWeb LLC 30 15
Big Data will always be too big Big Data challenge will always be changing Next generation data governance tools must drive business agility Our Tools are Only as Good as our Architecture Tools will always fall short without architecture that supports and drives change Copyright 2014, EnterpriseWeb LLC 31 About EnterpriseWeb EnterpriseWeb is an application creation and deployment platform built from the ground up for the Cloud. Use EnterpriseWeb to build an amazing diversity of enterprise class, Cloud ready applications, from case management tools to Telco operational support systems to workflow management apps and more. EnterpriseWeb tackles complex and multifaceted application and integration requirements the more complex, the better. EnterpriseWeb supports rapid development of policy based business processes, dynamically integrating information and capabilities to deliver personalized user experiences. Instead of rigid processes, businesses benefit from responsive workflows that respond to the business context the right functionality and the right information for the right person, every time. Deploy your own EnterpriseWeb on premise or in the Cloud, or leverage our public PaaS (due third quarter 2014) to build your own Enterprise SaaS applications. For more information, visit http://www.enterpriseweb.com Copyright 2014, EnterpriseWeb LLC 32 16
Jason Bloomberg Chief Evangelist, EnterpriseWeb jason@enterpriseweb.com @theebizwizard Copyright 2014, EnterpriseWeb LLC EnterpriseWeb is a Registered Trademark of EnterpriseWeb LLC. All rights reserved. 17