Enterprise Archietcure and Big Data. Assoc. Prof. Dr. Thanachart Numnonda Executive Director IMC Institute 25 September 2015

Transcription

1 Enterprise Archietcure and Big Data Assoc. Prof. Dr. Thanachart Numnonda Executive Director IMC Institute 25 September 2015

2 Internet of Things Cloud Computing Big Data 2

3 3

4 Data created every minute Source: Trendwise Analytics 4

5 Data! The New York Stock Exchange generate about 1TB of new trade data per day. A commercial aircraft generates 3GB of flight sensor data in 1 hour. Vodafone generates 3TB of Call Detail Record (CRDs) per day. Between 2009 and 2014, the total number of U.S. online banking households will increase from 54 million to 66 million. 5

6 6

7 7

8 Impact of Big Data Analytics Source: Outsourcing Malaysia 8

9 9

10 Sample Data v.s. Big Data Can you judge a persons life expectancy? Given: DNA Medical records Food Lifestyle (smoking, drinking, driving, exercise) Source: PatrickVideos.com 10

11 Food Credit card tied up with grocery store How much meat or oil does you consume? Medical records at your doctors office will show who consume the most. Combined with DNA Your lifestyle: smoking, exercise, cellphone movement We can all be caring a smartphone. Source: PatrickVideos.com 11

12 2009 Flu Virus Compare to 1918 Spanish flu that killed millions. Center of Disease Control (CDC) requested doctors to report flu cases. Two week lag. Google cloud predict them in real time using a combination of 45 search terms. They processed 450 million different mathematical models. Source: PatrickVideos.com 12

13 13

14 Predicting crime: patrol officers 14

15 Data Sources Technology Analytics 15

16 B y 2015, 20% of Global 1000 organizations Will have established a strategic focus on information infrastructure Gartner 16

17 The Need for new technology Size of data is too big RDBMS are the square peg for all round holes Not all data requires ACID Many data are unstructure 17

18 Big Data Technology!! 18

19 Big Data Technology Storage Analytics 19

20 Big Data Future Architecture Spreadsheet Predictive Analytics Data Market Place Familiar End User Tools BI Platform Hundreds of TB of Data (structured) Petabytes of Data (Unstructured) Hadoop On Private Server Hadoop On Cloud Sscial Media Images S S R S s Connectors Data Warehouse / NewSQL Crawlers NoSQL ERP Unstructured and Structured Data CRM LOB APPs 20

21 Enterprise Data Warehouse Architecture Source: HDInsight Essentials - Second Edition 21

22 Hadoop-Based Enterprise Data Architecture Source: HDInsight Essentials - Second Edition 22

23 Hadoop-Based Enterprise Data Architecture Source: HDInsight Essentials - Second Edition 23

24 Enterprise Architecture is... Source : TOGAF9 Framework for Enterprise Architecture, A. Tomsky 24

25 EA Artifacts Business Architecture Defines the business strategy, governance, organization, and key business processes Data Architecture Describes the structure of an organization s logical and physical data assets and data management resources Application Architecture A blue print for the individual application systems to be deployed, their interactions, and their relationships to the core business processes of the organization Technology Architecture Hardware, Software IT infrastructure, middleware, networks, communications, processing, standards, etc 25

26 EA View of Big Data Variety of Big Data database technologies hardware technologies applications reporting tools Variety of sources Open source providers Licensed providers In-house developed Custom built providers 26

27 What architectural disciplines relate to Big Data? Business Architecture Business Continuity Operations Architecture Marketing Architecture Disaster Recovery Architecture Storage Architecture Infrastructure Architecture Network Security Application Architecture Information Architecture Reporting Architecture Integration Architecture Application Portfolio Management Technology Portfolio Management Applicaton Security Data Architecture Data Governance Architecture Master Data Management (MDM & RDM) Metadata Management Data Security Source: Pragmatic Enterprise Architecture; Knowledgent 27

28 Big Data: Technology Architecture Competing open source foundations Apache Software Foundation (ASF) Free Software Foundation (FSF) Many individual companies Competing open source providers of Hadoop Hortonworks, Cloudera MapR Source: Pragmatic Enterprise Architecture; Knowledgent 28

29 Big Data: Application Architecture Applications use assorted Big Data technologies Open source Licensed Traditional databases (OldSQL) Traditional Big Data technologies Non-traditional (NoSQL) OLTP (New SQL) fully ACID Proprietary breed Quantum computing Source: Pragmatic Enterprise Architecture; Knowledgent 29

30 Big Data: Application Architecture Approved Data sources Technologies Time period Approved Data sources Use cases Technologies Time period Source: Pragmatic Enterprise Architecture; Knowledgent Approved Data sources Data target Time period 30

31 Big Data Ecosystem Without EA Uncoordinated Inconsistent Complex Source: Pragmatic Enterprise Architecture; Knowledgent 31

32 EA Driven Big Data Ecosystem Source: Pragmatic Enterprise Architecture; Knowledgent 32

33 Big Data Ecosystem Without EA Uncoordinated Inconsistent Complex 2014 Knowledgent Group Inc. All Rights Reserved

34 EA Driven Big Data Ecosystem 2014 Knowledgent Group Inc. All Rights Reserved

35 Big Data Survey tinyurl.com/qg2k7aw 35

36 36

37 Thank you