PLATYPUS SYMPOSIUM BIG DATA. Associate Professor Paul Kennedy University of Technology, Sydney

Transcription

1 PLATYPUS SYMPOSIUM BIG DATA Associate Professor Paul Kennedy University of Technology, Sydney

2 Big Data Associate Professor Paul Kennedy School of Software Faculty of Engineering & IT, UTS UTS Centre for Quantum Computation and Intelligent Systems

3 Big Data What is it? How is it used? Why is it important?

4 Big Data What is it?

5 Collecting Data Humans have always collected, checked and organised data 5500 years ago Sumerians marked tax records onto dried mud tablets Scientists have looked through microscopes and telescopes and drawn what they saw Market researchers ran surveys or had TV diaries Medical laboratories took dozens of measurements per patient Source: Walters Art Museum / Wikimedia Commons / Public Domain

6 Data Analysing Since then, people have sought ways to use the recorded information to improve their lives (financially, health,...) Understanding People can understand these amounts of data and maybe make predictions for the future But nowadays, there is a data explosion

7 Data explosion Most data now goes straight to computers without humans seeing them Tax records submitted electronically Telescopes operated remotely and digital images goes to computer files Market and POS data go to data warehouses High throughput technology make simultaneous measurements of 1000s of genes per patient This deluge of data is useless to unaided people!

8 Is it really an explosion? 2011: 1.8 zetabytes of information created globally and expected to double each year = 200 billion 2-hour HD movies that one person could watch for 47 million years straight! From sensors, satellites, social media, mobile communications, , RFID and enterprise applications Source: TechAmerica Foundation, 2012, Demystifying Big Data

9 Big Data... Huge global interest currently Obama administration in 2011 announced $200m for Big Data R&D in US TechAmerica Foundation released report describing transformational power of Big Data and recommendations for training huge number of data scientists and analysts urgently needed Source: TechAmerica Foundation, 2012, Demystifying Big Data

10 Big Data large volumes of high velocity, complex and variable data that require advanced techniques and technologies to enable the capture, storage, distribution, management and analysis of the information Challenge Opportunity making sense of the data exploiting it to enhance business Source: TechAmerica Foundation, 2012, Demystifying Big Data

11 Helping to catch the backpacker killer Australia s most notorious serial murder case Early 1990s, 7 young backpackers murdered Police had developed a profile Huge dataset generated of vehicle records, gym memberships, gun licensing and police records 18 million suspects! Link analysis software from Sydney company NetMap Analytics, narrowed list to hundreds then 32, which included the murderer - Ivan Milat

12

13 Big Data - 4 Vs Volume - the amount of data has increased more sources, higher resolution sensors Velocity - speed of production and change real time analysis gives improved decisions Variety - different formats and sources e.g. social media, video, chat, genomics Veracity - quality and provenance of data inconsistent, incomplete, ambiguous, latency

14 Data Structured rows & columns, like Excel spreadsheets 15% of data Unstructured generally human generated text, multimedia, audio 85% of data Semi-structured As for unstructured, but metadata or tags

15 From Data to Knowledge Data raw uninterpreted facts e.g. Tom, 20 years old, student Information relates items of Data together e.g. Tom is 20 years old Knowledge relates items of Information together Tom is 20 years old Tom pays > $1500 insurance Modelling the world (= generalising) [18-25] years old P(accident) = high

16 Data Analytics

17 Data Analytics is the analysis of large databases to find novel, commercially valuable and exploitable patterns. Aim discover meaningful insights and pus knowledge from data Discoveries expressed as models Data mining = process of building models

18 Models A model Captures the essence of the discovered knowledge Can assist in understanding the world Can be used to make predictions

19 Data Mining / Data Analytics Extremely large datasets Discovery of the non-obvious No specific hypothesis (compare to statistical hypothesis testing) Useful knowledge to improve processes Impossible to do manually Knowledge from the data in any way possible

20 Fitting to the business Understand the business context, and stronger, framing a business question Translating the business question into a data analytics question Collecting, understanding and processing data from across the business and possibly externally Build models and evaluate them Deploying the results in the business to deliver benefits Iterative process

21 Two main modelling Unsupervised methods approaches Model tries to make sense of the data Clustering, association rule mining Supervised methods Models learns a relationship between inputs and outputs from old data Model can then be used to predict output for new inputs Classification, prediction, regression Decision trees, neural networks, support vector machines, random forest

22 Source: Kenneth Jensen / Wikimedia Commons / Public Domain CRISP-DM Shearer. The CRISP DM model: The new blueprint for data mining. Journal of Data Warehousing, 5(4):13 22, Fall 2000.

23 Attributes Instances Friends

24 Business Problem: Who has better access to other friends? structural component

25 Possible answer

26 Business Problem: Predict whether someone gets sunburned The Class The mining table

27 One possible answer: Characterisation of the type of person with a decision tree Hair Colour blonde ginger brown Lotion Lotion Lotion no yes no yes no sunburned no sunburned no no

28 Source: TechAmerica Foundation, 2012, Demystifying Big Data Information Flow

29 Big Data How is it used?

30 LinkedIn 2006: LinkedIn ~8M accounts Business problem: users not seeking out connections with others on the site enough Jonathan Goldman found a way to predict whose networks a given profile would land in Added a module to present to the user some people that they might possibly know, but were not in their network Achieved a click-through rate 30% higher than the rate obtained by other prompts to visit site pages Credited with increasing growth trajectory of LinkedIn Davenport & Patil, Data Scientist: The Sexiest Job of the 21st Century, Harvard Business Review, October 2012.

31 Predicting the 2012 US election result Nate Silver used predictive analytics & statistics to correctly predict outcomes of 50 out of 50 states from polling and related data Republican pundits were confident in their landslide-win predictions. Democrat pundits predicted razor-thin victory Shows the power of a data-centric approach over gut-feeling

32 The lion, the witch and the wardrobe Movie galaxies

33 Fellowship of the ring

34 The return of the king

35

36 Fraud Detection Large Australian project management website Few accounts laundered money through credit card transactions Historical data to predict accounts likely to be fraudsters Deployed to filter and mark potential fraudsters Accuracy ~98% First week detected a large Ukrainian fraud syndicate with >50 user accounts, two smaller groups from China and Vietnam and several other minor fraudsters Source: WikiCommons: Repro in book: Rosén: En ren historia. Ljungby, 1992.

37 Big Data Why is it important?

38 Makes use of the unstructured 85% of data that is otherwise unusable Potential for evidence-based data-driven decision-making Competitive edge e-business - after the buzz-word has faded, the principles will underpin society Affordable due to convergence of technology

39 Institute of Analytics Professionals of Australia Our mission is to unite, inform, support and promote analytics professionals in Australia. We provide information sources, a virtual community, a networking hub and a professional identity. We promote the benefits of analytics in modern business.

40 Information is the oil of the 21st century, and analytics is the combustion engine. Peter Sondergaard, SVP, Gartner Research. Speech given at Gartner Symposium/ITxpo

41 Thanks & Questions... Associate Professor Paul Kennedy School of Software Faculty of Engineering & IT, UTS