Applying Semantics to Unstructured Data (Big and Getting Bigger)

Transcription

1 Applying Semantics to Unstructured Data (Big and Getting Bigger) Wednesday, November 30, :00 5:00 Bryan Bell Vice President, Enterprise Solutions, Expert System Lynda Moulton, Analyst & Consultant, LWM Technology Services Peter O'Kelly Principal Analyst, O'Kelly Associates

2 Overall Session Agenda Introduction and context-setting "Big Data" 101 for Business Semantics and the Big Data Opportunity 2

3 Big Data 101 Agenda Big data in context Recap Risks Recommendations 3

4 Big Data in Context What is big data? Unhelpfully, both big data and NoSQL, generally considered a key part of the big data wave, are defined more in terms of what they aren t than what they are A typical big data definition (Wikipedia): [ ] data sets that grow so large that they become awkward to work with using on-hand database management tools Often associated with Gartner s volume, variety (and complexity), and velocity model Also value and veracity considerations 4

5 Big Data in Context Why is big data a big deal now? Commoditized hardware, software, and networking Capability and price/performance curves that continue to defy all economic laws Cloud services with radical new capability/cost equations Maturation and uptake of related open source software, especially Hadoop Powerful and often no- or low-cost 5

6 Big Data in Context Why is big data a big deal now (continued)? Market enthusiasm for NoSQL systems Useful and often open source /public domain data sources and services Mainstreaming of semantic tools and techniques 6

7 A Prime Minicomputer, c1982 7

8 Fast-Forward to

9 Fast-Forward to

10 Fast-Forward to

11 Fast-Forward to

12 Fast-Forward to

13 Google BigQuery 13

14 Hadoop Hadoop is often considered central to big data Originating with Google s MapReduce architecture, Apache Hadoop is an open source architecture for distributed processing on networks of commodity hardware From Wikipedia: Map step: The master node takes the input, divides it into smaller sub-problems, and distributes them to worker nodes Reduce step: The master node then collects the answers to all the sub-problems and combines them in some way to form the output the answer to the problem it was originally trying to solve 14

15 Hadoop Commercial application domains include (from Wikipedia) Log and/or clickstream analysis of various kinds Marketing analytics Machine learning and/or sophisticated data mining Image processing Processing of XML messages Web crawling and/or text processing General archiving, including of relational/tabular data, e.g. for compliance 15

16 Hadoop Hadoop is popular and rapidly evolving Most leading information management vendors have embraced Hadoop There is now a Hadoop ecosystem 16

17 Meanwhile, Back in the Googleplex Dremel, BigQuery, Spanner, and other really big data projects 17

18 Meanwhile, Back in the Googleplex 18

19 Google Now 19

20 A NoSQL Taxonomy From the NoSQL Wikipedia article: 20

21 A View of the NoSQL Landscape 21

22 Another NoSQL Landscape View

23 NoSQL Perspectives The NoSQL meme confusingly conflates Document database requirements Best served by XML DBMS (XDBMS) Physical database model decisions on which only DBAs and systems architects should focus And which are more complementary than competitive with DBMS Object databases, which have floundered for decades But with which some application developers are nonetheless enamored, for minimized impedance mismatch, despite significant information management compromises Semantic (e.g., RDF) models Also more complementary than competitive with RDBMS/XDBMS Also consider: the traditional DBMS players can leverage the same underlying technology power curves 23

24 Data as a Service The (single source of) truth is out there?... High-quality data sources are being commoditized Value is shifting to the ability to discern and leverage conceptual connections, not just to manage big databases Some resources and developments to explore Social networking graphs and activities Data.com (Salesforce.com) Data.gov Google Knowledge Graph Linked Data Microsoft Windows Azure Data Marketplace Wikidata.org Wolfram Alpha 24

25 Mainstreaming Semantics Tools and techniques applied in search of more meaning, e.g., Vocabulary management Disambiguation and auto-categorization Text mining and analysis Context and relationship analysis It s still ideal to help people capture and apply data and metadata in context Semantic tools/techniques are complementary 25

26 Mainstreaming Semantics The Semantic Web is still more vision than reality But Google, Microsoft, and Yahoo, and Yandex, for example, are improving Web searches by capturing and applying more metadata and relationships via schema.org schemas in Web pages And Google s Knowledge Graph is about things, not strings, with, as of mid-2012, 500 million objects, as well as more than 3.5 billion facts about and relationships between these different objects 26

27 Recap Commoditization and cloud Very significant new opportunities Hadoop and related frameworks Complementary to RDBMS and XDBMS NoSQL Likely headed for meme-bust Data services Game-changing potential Semantic tools and techniques Rapidly gaining momentum 27

28 Risks The potential for an ever-expanding set of information silos Focus on minimized redundancy and optimized integration GIGO (garbage in, garbage out) at super-scale New opportunities for unprecedented self-inflicted damage, for organizations that don t model or query effectively Cognitive overreach The potential for information workers to create and act on nonsensical queries based on poorly-designed and/or misunderstood information models Skills gaps can create competitive disadvantages Modeling, query formulation, and data analysis Critical thinking and information literacy 28

29 Recommendations Aim high: big data is in many respects just getting started A lot of technology recycling but also significant and disruptive innovation Work to build consensus among stakeholders on the opportunities and risks Focus on human skills e.g., critical thinking and information literacy For now, an instance of the most creative and powerful type of semantic big data processor we know of is between your ears 29