Data Integrity & Scalability The Value of Accuracy. Data Quality in Big Data

Transcription

1 Data Integrity & Scalability The Value of Accuracy Data Quality in Big Data

2 Data Quality in the news 2

3 And some more examples... 3

4 High Quality Information as competitive differentiator Business today... Online, Instantaneous and Connected Critical customer, supplier & portfolio information will change in the blink of an eye Global markets Highly competitive Regulatory compliance High quality information is business critical For industries banking, insurance, logistics, retail, healthcare, telco, government etc. Auctioning, marketplaces, mobile apps, global markets, benchmarking, social media. Copyright 2012, Information Builders. Slide 4

5 Effective Use of HQ Data is Key Competitive Differentiation in a Changing Environment Improve the Top Line Manage Risk and Regulatory Compliance Increase Operational Efficiency Customer insight to improve organic growth. Single view of the customer to provide a seamless and relevant cross channel channel experience. Put trusted information in the hands of frontline employees to improve productivity and keep service levels high Accurateinformation for riskmanagement optimization. Use information to accurately prevent, identify and mitigate fraud and operational risk. Leverage the business value of information with information governance programs that improve the management of people, process and information technology. Deliver trusted information to improve productivity, implement new processes and offerings and enhance existing process via automation Create trusted enterprise insight and analytics to drive business optimization and improve revenue, productivity, it operations, and financial i results. Optimize and truncate content centric process to reduce costs and improve customer experience. Copyright 2012, Information Builders. Slide 5

6 High Quality Data Facts about things in a context Copyright 2012, Information Builders. Slide 6

7 What does data growth look at? Copyright 2012, Information Builders. Slide 7 7

8 Enterprise Information Management Master Data Governance UpStream, InStream, Downstream Copyright 2012, Information Builders. Slide 8

9 Copyright 2012, Information Builders. Slide 9

10 What is Big Data?

11 Setting the stage Big Data: A massive volume of both structured and unstructured data that is so large that it s difficult to process with traditional database and software techniques Big data technologies describe a new generation of technologies and architectures, designed to economically extractvalue from very large volumes of a wide variety of data, by enabling high velocity capture, discovery, and/or analysis Data is a new class of economic assets, like currency and gold Source: World Economic Forum 2012

12 The numbers Walmart handles more than 1 million customer transactions every hour. Facebook handles 30 billion pieces of content shared a month Decoding the human genome originally took 10 years to process; now it can be achieved in one week. At CERN the 27 kilometer long Large Hadron Collider (LHC) creates 1 petabyte of data per second. 15 petabytes per year is available for analytics. Since you are probably not investigating the origin of the universe 12

13 Social 2012, the # s Facebook 900+ million April 2012 United States Qzone 536 million December 2011 China Windows Live 330+ million June 2009 United States Tencent Weibo 310 million December 2011 China Sina Weibo 300+ million February 2012 China Habbo 230 million September 2011 Finland Google million April 2012 United States Vkontakte 290+ million March 2012 Russia Badoo 151+ million April 2012 United States Skype 145 million September 2010 United States Twitter 140+ million March 2012 United States Bebo 117 million July 2010 United States LinkedIn 100+ million March 2011 United States 13

14 Where does your data come from? Online (Web) Applications To run applications (content, video, blog, posts, transactions) To give the data context (friends, social media, collaborative working) The keep applications running (logs, metrics) Regulatory compliance Data Governance initiatives Risk Management Electronic Archiving of Information ProductLifecycle Management and Catalogues Mobile Community Expanding Markets 14

15 External Big data Open Data (openbare data collecties) Overheid maar ook KNMI bv. Vraag antwoord combinaties Verkiezingskaart Scholen CBS Statline Social Media (feeds, likes, tweets, profiles) News, Press, Images, Video External Big Data is a new type of source for analytical purposes. p It does not mean it s your Big Data problem. It might be an integration or integrity problem however. 15

16 Why Big Data is important?

17 Key Drivers for Big Data management Social Network and Relationship Analysis Cost Reduction Fraud Detection and Prevention Digital Marketing Optimization Data Exploration and Discovery Single Views (Quality of Service) Responsiveness and User Experience Flexibility and Scalability supporting growth Data Retention Archiving Compliance 17

18 Potential Value of Big Data Source: McKinsey Global May

19 What is the impact of Big Data to your IT organisation?

20 Big Data spans four dimensions Gigabytes, Terrabytes, Petabytes Volume Velocity Realtime Capture and Realtime Analytics Big Data Variety Unstructured data like Images, Documents, Structured data like files, tables, messages Quality Integrity of the data

21 Two types of Big Data solutions For Analytical purposes Columnar Oriented and In Memory database technology Appliances Schema and Data model based architecture Allow huge amounts of data to be joined, aggregated and queried Super fast response times Limited it DBA requirement Information Builders Hyperstage For Storage purposes Hadoop Distributed File System (Open Source) started by Yahoo and Google Hd Hadoop Map Rd Reduce Distributed ib t computation tti Changing schema support Document Oriented Horizontal Scalability Cloud proof Hive SQL layer Information Builders Hadoop Adapter 21

22 Everybody is making decisions BI Com mmunity Developers Analysts & Power Users 20 % Business Users 80 % Partners Customers And Beyond 22

23 Big Data Analytics at our Customers Automotive data for Product Sales purpose Time To Market: New products Total Cost of Ownership: Development Maintenance User Experience: Performance and Mobile Police of Richmond, Virginia Quality of Service : #5 to #95 (out of 354 cities) Innovation: Predict Crime based on historical data Performance Management: Strategic Insight Banking data for Credit Card dscoring Innovation: new Internet Banking Solutions (SaaS) Scalability: 1 Million users and growing Revenue: Boost usage of online channels Marketing data to improve Campaign Management Increased Sales: Upsell and Cross sell Customer Retention: Single View 23

24 Big Data Technology for Analytics

25 IT Manager s try to mitigate these response times.. How Performance Issues are Typically Addressed by Pace of Data Growth Tune or upgrade existing databases 66% 75% Upgrade server hardware/processors 54% 70% Upgrade/expand storage systems 33% 60% Archive older data on other systems Upgrade networking infrastructure 21% 30% 32% 44% High Growth 4% Low Growth Don't Know / Unsure 7% 0% 20% 40% 60% 80% 100% When organizations have long running queries that limit the business, the response is often to spend much more time and money to resolve the problem Source: KEEPING UP WITH EVER-EXPANDING ENTERPRISE DATA ( Joseph McKendrick Unisphere Research October 2010)

26 Data Warehousing Challenges More Data, More Data Sources Real time data Multiple databases External Sources Limited Resources and Budget More Kinds of Output Needed by More Users, More Quickly Traditional Data Warehousing Labor intensive, heavy indexing, aggregations and partitioning Hardware intensive: massive storage; big servers Expensive and complex

27 Pivoting Your Perspective: Columnar Technology.

28 The Limitation of Rows These Solutions Contribute to Operational Limitations 1. Impediments to business agility: Organizations often must wait for DBAs to create indexes or other tuning structures, thereby delaying access to data. In addition, indexes significantly slow data loading operations and increase the size of the database, sometimes by a factor of 2x. 2. Loss of data and time fidelity: IT generally performs ETL operations in batch mode during non business hours. Such transformations delay access to data and often result in mismatches between operational and analytic databases. 3. Limited ad hoc capability: Response times for ad hoc queries increase as the volume of data dt grows. Unanticipated i t queries (where DBAs have not tuned dthe database in advance) can result in unacceptable response times, and may even fail to complete. 4. Unnecessary expenditures: Attempts to improve performance using hardware acceleration and database tuning schemes raise the capital costs of equipment and the operational costs of database administration. Further, the added complexity of managing a large database diverts operational budgets away from more urgent IT projects.

29 The Limitation of Rows The Ubiquity of Rows 30 columns Row-based databases are ubiquitous because so many of our most important business systems are transactional. 50 millions Rows Row-oriented databases are well suited for transactional environments, such as a call center where a customer s entire record is required when their profile is retrieved and/or when fields are frequently updated. But - Disk I/O becomes a substantial limiting factor since a row-oriented design forces the database to retrieve all column data for any query.

30 Pivoting Your Perspective: Columnar Technology Employee Id Name Location Sales 1 Smith New York 50,000 2 Jones New York 65,000 3 Fraser Boston 40,000 4 Fraser Boston 70,000 Row Oriented (1, Smith, New York, 50000; 2, Jones, New York, 65000; 3, Fraser, Boston, 40000; 4, Fraser, Boston, 70000) Column Oriented Works well if all the columns are needed for every query. Efficient for transactional processing if all the data for the row is available (1, 2, 3, 4; Smith, Jones, Fraser, Fraser; New York, New York, Boston, Boston, 50000, 65000, 40000, 70000) Works well with aggregate results (sum, count, avg. ) Only columns that are relevant need to be touched Consistent performance with any database design Allows for very efficient compression

31 Introducing WebFOCUS Hyperstage

32 Introducing WebFOCUS Hyperstage. How is it architected? Hyperstage combines a columnar database with intelligence we call the Knowledge Grid to deliver fast query responses. Bulk Loader Hyperstage Engine Knowledge Grid Compressor Unmatched Administrative Simplicity No Indexes No data partitioning No Manual tuning

33 Introducing WebFOCUS Hyperstage. What does this mean for Customers? Self managing: 90% less administrative effort Low cost: More than 50% less than alternative solutions Scalable, high performance: Up to 50 TB using a single industry standard server, no appliance required Fast queries: Ad hoc queries are as fast as anticipated queries, so users have total flexibility Compression: Data compression of 10:1 to 40:1 that means a lot less storage is needed, it might mean you can get the entire database in memory!

34 WebFOCUS Hyperstage Engine How does it work? Column Orientation Knowledge Grid statistics and metadata describing the super-compressed data Data Packs data stored in manageably sized, highly compressed data packs Smarter Architecture No maintenance No query planning No partition schemes No DBA Data compressed using algorithms tailored to data type

35 Performance Benchmarks WebFOCUS Hyperstage SQL Server Report Query 1 Query 2

36 Compression Ratio Datastore Size SQL Server WebFOCUS Hyperstage

37 WebFOCUS Hyperstage Example: Query and Knowledge Grid SELECT count(*) FROM employees WHERE salary > AND age < 65 AND job = Shipping AND city = Toronto ; salary age job city All values match Completely Irrelevant Suspect

38 WebFOCUS Hyperstage Example: salary > SELECT count(*) FROM employees WHERE salary > AND age < 65 AND job = Shipping AND city = Toronto ; salary age job city 1. Find the Data Packs with salary > All values match Completely Irrelevant

39 WebFOCUS Hyperstage Example: age < 65 SELECT count(*) FROM employees WHERE salary > AND age < 65 AND job = Shipping AND city = Toronto ; salary age job city 1. Find the Data Packs with salary > Find the Data Packs that contain age < 65 All values match Completely Irrelevant Suspect

40 WebFOCUS Hyperstage Example: job = shipping SELECT count(*) FROM employees WHERE salary > AND age < 65 AND job = Shipping AND city = Toronto ; salary age job city 1. Find the Data Packs with salary > Find the Data Packs that contain age < Find the Data Packs that have job = shipping All values match Completely Irrelevant Suspect

41 WebFOCUS Hyperstage Example: city = Toronto SELECT count(*) FROM employees WHERE salary > AND age < 65 AND job = Shipping AND city = Toronto ; salary age job city 1. Find the Data Packs with salary > Find the Data Packs that contain age < Find the Data Packs that have job = shipping 4. Find the Data Packs that have city = Toronto All values match Completely Irrelevant Suspect

42 WebFOCUS Hyperstage Example: Eliminate Pack Rows SELECT count(*) FROM employees WHERE salary > AND age < 65 AND job = Shipping AND city = Toronto ; 1. Find the Data Packs with salary > Find the Data Packs that contain age < Find the Data Packs that have job = shipping 4. Find the Data Packs that have city = Toronto 5. Eliminate All rows that have been flagged as irrelevant salary age job city All packs ignored All packs ignored All packs ignored All values match Completely Irrelevant Suspect

43 WebFOCUS Hyperstage Example: Decompress and scan SELECT count(*) FROM employees WHERE salary > AND age < 65 AND job = Shipping AND city = Toronto ; 1. Find the Data Packs with salary > Find the Data Packs that contain age < Find the Data Packs that have job = shipping 4. Find the Data Packs that have city = Toronto 5. Eliminate All rows that have been flagged as irrelevant 6. Finally we identify the pack that needs to be decompressed salary age job city Only this pack will be de-compressed All packs ignored All packs ignored All packs ignored All values match Completely Irrelevant Suspect

44 Q&A