How To Improve Data Quality

Transcription

1 Big Data Promises and Pitfalls David J. Hand Imperial College, London and Winton Capital Management July 2015 Policy making in the Big Data Era 1

2 The world of data is changing Not something which happens and we can then settle into the new world Rather change is the only constant tomorrow s IT will be different from today s (think Facebook, Twitter, YouTube, Instagram, etc) The future you have tomorrow won't be the same future you had yesterday. Chuck Palahniuk In times of change, learners inherit the Earth, while the learned find themselves beautifully equipped to deal with a world that no longer exists. Eric Hoffer Policy making in the Big Data Era 2

3 What s new in today s world of data? source of data: automatic acquisition of data speed of acquisition of data: streaming data size of data sets: big data diversity of data complexity of data Policy making in the Big Data Era 3

4 Source: Modern data capture technologies Automatic data collection: electronic measurements: point of sale credit card terminals, petrol pumps, contactless travel cards, phone records, s, GPS, CCTV cameras,... Properties of automatic data collection: immediate complete??? untouched by human hands??? Policy making in the Big Data Era 4

5 Internet of things Social media data data directly from the web Administrative data Data not primarily collected for research purposes e.g. supermarket purchases, credit card transactions, tax records, education records, health records, transport movements,... Administrative data research is secondary analysis, so may not be ideal for the research purpose issues of consent may arise changes to the collection procedures may change nature of data quality issues different from those of surveys selection distortion who s in the database? Policy making in the Big Data Era 5

6 The Administrative Data Research Network Aim: to facilitate access to and linkage of de identified administrative data routinely collected by government departments and other public sector organisations Four centres: England, NI, Scotland, Wales + ADS Partnerships with Nat Stats Institutes UK wide governance Safe and secure data access Accredited researchers Public engagement Policy making in the Big Data Era 6

7 Speed: realtime data collection and analysis This has several major implications, threatening the position of NSIs? e.g. timeliness e.g. analytic tools and methods Policy making in the Big Data Era 7

8 Timeliness The balance of timeliness against accuracy Example: UK GDP 1 st estimate: 44% of the data available by 25 days, 2 nd estimate: 88% by 55 days, 3 rd estimate: 85 days Policy making in the Big Data Era 8

9 What about inflation rate?: Elaborate procedure to collect sample data Contrast with direct recording from transactions: Google has created a new inflation measure the Google Price Index based on the cost of goods sold online which could prove more accurate and up to date than official statistics. Google's mountain of web shopping data could also be used by the online group for economic forecasting ahead of the publication of official statistics, the Financial Times reported. Google's chief economist Hal Varian said that he is working on predicting the present by using real time search data to forecast official figures which often are published at least a month after the period they cover. Policy making in the Big Data Era 9

10 Analytic tools and methods Streaming data : the data keep on coming, like water from a hose Permanently executing analytic tools, processing the data as it arrives anomalies changes summaries (trends, averages, variability, maxima,...) Realtime automatic analysis Contrast the more familiar: a fixed database Policy making in the Big Data Era 10

11 Note: perhaps we cannot store the data once it has been processed In that case we need to know what questions we will ask We cannot later ask arbitrary questions, but only those that can be answered from our summary statistics Summarising a stream Subsetting a stream: sampling, but requires different approaches from classical survey sampling Filtering a stream: accept only those cases which meet some criterion Policy making in the Big Data Era 11

12 Size: Large data sets Census: the largest set of data collected by NSIs until relatively recently? But now administrative data, register based data some countries (e.g. Scandinavian) ahead of the field transaction data social media, Google searches, twitter messages, transaction logs, phone logs, transport logs,... Policy making in the Big Data Era 12

13 Diversity of data Survey, census, administrative, transaction, experimental,... Numerical tables, image, text, signal, networks,... Different kinds of data have different properties e.g. survey data: answers to the questions you choose but slow and expensive to collect, response bias? e.g. transaction data: fine granularity, both spatial and temporal, immediate, but may not address the question you want an opportunity: Perhaps data of different kinds can be combined synergistically, to overcome the problems of each individual kind Policy making in the Big Data Era 13

14 Stitching different kinds of data together Linking Matching Merging Technical challenges have begun to be addressed in different fields e.g. medical combining information from scans with traditional numeric and text files Great potential benefit from cross disciplinary collaboration and awareness Policy making in the Big Data Era 14

15 survey and census data is what people say: administrative and transaction data is what people do Policy making in the Big Data Era 15

16 survey and census data is what people say: administrative and transaction data is what people do New forms of data are closer to social reality? Policy making in the Big Data Era 16

17 Complexity of data Especially, networks and linked data Social networks Cybersecurity Fraud detection Policy making in the Big Data Era 17

18 Ping Pong Fraud Ring Model (HBOS Data Mining Team, 2007) Within the Mortgage Industry, in addition to individual fraudulent applications, fraud can also occur in groups often referred to as Fraud Rings. These groups can consist of applicants, brokers and/or other professionals that get together to cheat the system. The purpose of the Ping Pong Fraud Ring Model is to assist the Mortgage Fraud team in identifying these Fraud Rings: 1. Ping Pong clustering: aggregating applications that have common names, addresses, telephone numbers, employers, etc 2. Ranking: inconsistency rules are used to rank the clusters in order of severity. 3. Linkage analysis: identifying links between customers, brokers, employers, and/or solicitors. Policy making in the Big Data Era 18

19 OTHER CRITICAL ISSUES Data quality most big data that have received popular attention are not the output of instruments designed to produce valid and reliable data amenable for scientific analysis Lazer et al 2014 Caution: No data set is without potential quality issues Role and reputation of NSIs, as standard bearers of quality Policy making in the Big Data Era 19

20 Example: Mistaken idea that admin data/transaction data has no quality issues Quality Assurance of Administrative Data Administrative Data Quality Assurance Toolkit UK Statistics Authority, January 2015 Policy making in the Big Data Era 20

21 Top Tips UKSA Administrative Data Quality Assurance Toolkit, 2015 These five tips summarise the main pointers for statistical producers to develop a good understanding of the quality issues of administrative data: Don t trust the safeguards: Check if safeguards are functioning effectively Get involved: work and share with suppliers, such as through secondments and webinars, to develop a common understanding Raise a red flag: identify potential data quality concerns using input and output quality indicators and investigate anomalies See the big picture: identify what investigations and audits have been conducted and what they found Corroborate the evidence: confirm the levels and the trends shown by the stats derived from the admin data Policy making in the Big Data Era 21

22 Example: transaction implies a complex sociological selection process e.g. reject inference in credit scoring Policy making in the Big Data Era 22

23 Example: Automatic data capture can go wrong Wind speed in metres per second, against time Policy making in the Big Data Era 23

24 Example: Proportion of homeowner missing values vs time Policy making in the Big Data Era 24

25 The rate of change of technology Implications: for series based on a particular technology, what happens when it changes, or disappears altogether (Windows XP) surveys have been around forever, but Google Trends? Policy making in the Big Data Era 25

26 Statistical complications manipulation vs inference two types of statistical model regression to the mean selection bias individual behaviour vs population behaviour impact of stated targets, feedback, gaming, Policy making in the Big Data Era 26

27 Data manipulation: Sorting, adding, selecting, matching, concatenating, aggregating, etc mathematical/computing challenges Examples Google maps to find directions apps for train and bus status find the shop with the item you seek locate that out of print book identify the name of that piece of music Policy making in the Big Data Era 27

28 Inference A statement about the entire population, some other population, the future, etc. statistical challenges Examples which teaching strategies are most effective? social influences on sport participation and consequent impact on population health and on NHS costs Policy making in the Big Data Era 28

29 Two types of statistical model Theory driven Based on underlying theories of social behaviour, policy impact, economics, etc e.g. microeconomic theories of how agents interact Data driven Based purely on empirical relationships found in the data e.g. empirical logistic regression trees in credit scoring Policy making in the Big Data Era 29

30 Regression to the mean Policy making in the Big Data Era 30

31 Google flu trends In February 2013,... Nature reported that [Google Flu Trends] was predicting more than double the proportion of doctor visits for influenza like illness than the Centers for Disease Control and Prevention, which bases its estimates on surveillance reports from laboratories across the United States... despite the fact that GFT was built to predict CDC reports. Lazer et al 2014 Policy making in the Big Data Era 31

32 Initial version: find best matches from 50 million search terms for 1152 data points Overfitting! part flu detector, part winter detector Updated, 2009: still consistently overestimated flu prevalence (100 out of 108 consecutive weeks); autocorrelated errors, etc Some understanding of classical time series modelling could help Policy making in the Big Data Era 32

33 Correlation = Policy making in the Big Data Era 33

34 Predicting the financial markets Preis T, Moat H.S., and Stanley H.E. (2013) Quantifying trading behavior in financial markets using Google trends. Scientific Reports, 3, By analyzing changes in Google query volumes for search terms related to finance, we find patterns that may be interpreted as early warning signs of stock market moves. Policy making in the Big Data Era 34

35 Chose 98 search terms related to stock markets Implemented a strategy which sold DJIA following an increase in searches and bought following a decrease in searches Concluded: "We find that returns from the Google Trends strategies are significantly higher overall than returns from the random strategies (<R>=0.60; t=8.65, df=97, p<0.001, one sample t test)." But this is an inappropriate test Telling us nothing about the effectiveness of the strategy Policy making in the Big Data Era 35

36 1) the 98 terms were purposively sampled, not a random sample from a population of possible terms: the authors confused fixed and random effects; 2) the standard deviation used in the denominator of their t test is irrelevant to the variability of the mean return: it s the wrong statistic; 3) they ignored the correlation between the random aspect of the different words returns: anyone working in the financial sector will be aware of the critical importance of correlation! Policy making in the Big Data Era 36

37 Google Price Index My earlier quote about the Google Price index was from 2010 Where is the GPI now? [Hal Varian, Google s Chief Economist] responded that the GPI was never intended to be a public project, data source, etc. It was simply a project internal to Google that got hyped up by the press happened to the google price index Policy making in the Big Data Era 37

38 Crime maps For many years, all forces have mapped crimes and incidents to help them focus investigations, analyse hot spots and tackle crime vigorously. The information now on the forces websites has a different more communityfocused perspective and means the public can now look at crime levels in their community simply by putting their postcode into their local police force s website. Policy making in the Big Data Era 38

39 More than 5.2 million people have not reported crimes for fear of deterring home buyers or renters since the online crime map was launched in February 2011 A quarter (24 per cent) of people would not report a crime for fear it would harm their chances of selling or renting their property /news Policy making in the Big Data Era 39

40 Trust, privacy,... people readily give data to supermarkets, travel companies, phone companies, credit card companies,.. government misuse, targeting subgroups. Formal separation of statistical offices from government? Anonymisation De identification Trusted Third Party linkage methods need for sta s cians to learn new skills broader data scientists Policy making in the Big Data Era 40

41 Interaction with the commercial world The credit bureau model: everyone benefits NSI takes fine granularity commercial (e.g. transaction) data, combines it with public data, analyses it, and feeds back the results to the commercial body for a price NSI retains control, protects public Policy making in the Big Data Era 41

42 The skills gap Big data and open data are only useful if you have the skills to extract information New skills are needed What are they? How many NSI statisticians are at ease with real time transaction data? The major sources of inaccuracy in survey samples are (?) selection bias and sampling variability What are the major sources of error in administrative data? Policy making in the Big Data Era 42

43 An ethical framework Context: purpose of collecting data, using data Consent and choice Reasonable breadth and depth of data Substantiated data Ownership of results Fairness to all parties, compensation Considered consequences Access to data by subjects Accountability Policy making in the Big Data Era 43

44 Conclusions huge opportunities technical and social challenges but we need the capability to grasp the opportunities we need people with the skills Policy making in the Big Data Era 44

45 Conclusions huge opportunities technical and social challenges but we need the capability to grasp the opportunities we need people with the skills And above all: Policy making in the Big Data Era 45

46 Data are evidence Policy making in the Big Data Era 46