DATA VISUALIZATION: FINDING PICTURES IN NUMBERS

Transcription

1 DATA VISUALIZATION: FINDING PICTURES IN Pratap Vardhan, Data Scientist, Gramener

2

3

4 You will see 3 questions. You have 30 seconds. Try it! Your timer starts now A DATA VISUALISATION CHALLENGE

5 HOW MANY NUMBERS ARE ABOVE ?

6 HOW MANY NUMBERS ARE BELOW ?

7 WHICH QUADRANT HAS HIGHEST TOTAL?

8 The same questions again. But with a few visual cues. See how long it takes now. Your timer starts now A DATA VISUALISATION CHALLENGE

9 HOW MANY NUMBERS ARE ABOVE ?

10 HOW MANY NUMBERS ARE BELOW ?

11 WHICH QUADRANT HAS HIGHEST TOTAL?

12 YOU WILL BE SHOWN A SET OF NUMBERS ALONG WITH A SUMMARY (AVERAGE, ETC) CAN YOU MAKE SENSE OF THE FIGURES? WHY VISUALISE?

13 DO THESE FOUR CITIES LOOK IDENTICAL TO YOU? Take a look at the sales report alongside. A company has branches in 4 cities, and each branch changes the product price every month. This leads to a corresponding change in the sales. Here is the performance of the 4 branches with their monthly price and sales for each month. Looking at the average, the four branches have an identical performance. DO YOU AGREE? 2010 Month Boston Chicago Detroit New York Price Sales Price Sales Price Sales Price Sales Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Average Variance Average price is the same. Variance in price is the same. Average sales is the same too. So is the variance in sales.

14 ARE THEY REALLY IDENTICAL? CHECK AGAIN But in fact, the four cities are totally different in behaviour. Boston Chicago Detroit New York Boston s sales has generally increased with price. Detroit has a nearly perfect increase in sales with price, except for one aberration. Chicago shows a decline in sales beyond a price of 10. New York s sales fluctuates despite a nearly constant price.

15 A data analytics and visualisation company We handle terabyte-size data Gramener visualises your data via non-traditional analytics and visualise it in real-time. Gramener transforms your data into concise dashboards that make your business problem & solution visually obvious. We help you find insights quickly, based on cognitive research, and our visualisations guide you towards actionable decisions.

16 INDIAN ODI BATTING GRAMENER.COM/CRICKET/

17

18 Jan 100 YEARS OF INDIA S WEATHER Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

19 19 S

20 IN 2014 ELECTIONS, WHICH STATE PRODUCED MOST NUMBER OF CROREPATI CANDIDATES? AND WHICH STATE HAS HIGHEST % OF CROREPATI CANDIDATES?

21 GEOGRAPHY OF CANDIDATE WEALTH Uttar Pradesh, with over 400 crorepati candidates, tops the list. Number of Candidates The Northeastern states have the largest percentage of crorepati candidates. Percentage of Crorepati Candidates

22 AMONG THE MAINSTREAM PARTIES, WHICH PARTY HAS HIGHEST % OF CRIMINAL CANDIDATES?

23 CRIMINAL CASES Size: Number of candidates Color: % of criminal candidates MNS seems like a winner here. Closely followed by RJD, MDMK 23 S

24

25

26 AND, ONE MORE THING.. NAMESAKES OF 2014 ELECTIONS

27 CHANDU LALS OF MAHASAMUND Winner s Margin: 1,217 votes Namesakes' polled:,000+ votes

28 MOST OF WHAT I DO TODAY IS VISUALISING DATA ANOMALIES YOU DON T NEED SOPHISTICATED ANALYSES FOR THIS IT CAN BE EASY TO SPOT THEM

29 PREDICTING MARKS What determines a child s marks? Do girls score better than boys? Does the choice of subject matter? EDUCATION Does the medium of instruction matter? Does community or religion matter? Does their birthday matter? Does the first letter of their name matter?

30 LET S LOOK AT YEARS This is a dataset ( ) that has been around for several years, and has been studied extensively. Yet, a visualization can reveal patterns that are neither obvious nor well known. More births Some special days like April Fool s day are avoided, but Valentine s Day is quite popular Fewer births OF US BIRTH DATA For example, Are birthdays uniformly distributed? Do doctors or parents exercise the C-section option to move dates? Is there any day of the month that has unusually high or low births? Are there any months with relatively high or low births? on average, for each day of the year (from 1975 to 10) Most people prefer not to have children on the 13th of any month, given that it s an unlucky day Relatively few births during the Christmas and Thanksgiving holidays, as well as New Year and Independence Day. Very high births in September. But this is fairly well known. Most conceptions happen during the winter holiday season

31 THE PATTERN IN INDIA IS QUITE DIFFERENT This is a birth date dataset that s obtained from school admission data for over 10 million children. When we compare this with births in the US, we see none of the same patterns. More births Fewer births Such round numbered patterns a typical indication of fraud. Here, birthdates are brought forward to aid early school admission For example, Is there an aversion to the 13th or is there a local cultural nuance? Are holidays avoided for births? Which months have a higher propensity for births, and why? Are there any patterns not found in the US data? on average, for each day of the year (from 2007 to 2013) We see a large number of children born on the 5th, 10th, th, 20th and 25th of each month that is, round numbered dates Very few children are born in the month of August, and thereafter. Most births are concentrated in the first half of the year

32 THIS ADVERSELY IMPACTS It s a well established fact that older children tend to do better at school in most activities. Since many children have had their birth dates brought forward, these younger children suffer. Higher marks Lower marks CHILDREN S MARKS The average marks of children born on the 1st, 5th, 10th, th etc. of the month tend to score lower marks. Are holidays avoided for births? Which months have a higher propensity for births, and why? Are there any patterns not found in the US data? on average, for children born on a given day of the year (from 2007 to 2013) Children born on round numbered days score lower marks on average, due to a higher proportion of younger children 32

33 EXPLORING THE MAHABHARATA How does Mahabharata, one of the largest epics with 1.8 million words lend itself to text analytics? Can this unstructured data be processed to extract analytical insights? What does sentiment analysis of this tome convey? Is there a better way to explore relations between characters? How can closeness of characters be analysed & visualized?

34

35 MMS SPEECHES

36 AAP DONATIONS

37 FLAGS OF THE WORLD

38 CALVIN AND HOBBES

39 DETECTING FRAUD ENERGY UTILITY We know meter readings are incorrect, for various reasons. We don t, however, have the concrete proof we need to start the process of meter reading automation. Part of our problem is the volume of data that needs to be analysed. The other is the inexperience in tools or analyses to identify such patterns.

40 BILLING FRAUD AT AN ENERGY UTILITY An energy utility (with over 50 million subscribers) had 10 years worth of customer billing data available. Below is a simple histogram (or frequency distribution) of usage levels. Each bar represents the number of customers with a customers with a specific bill amount (in units, or KWh). Most fraud detection software failed to load the data, and sampled data revealed little or no insight. Tariffs are based on the usage slab. Someone with 101 units is billed in full at a higher tariff than someone with 100 units. So people have a strong incentive to stay at or within a slab boundary. This plot shows the frequency of all meter readings from Apr-2010 to Mar An unusually large number of readings are aligned with the slab boundaries. This can happen in one of two ways. First, people may be monitoring their usage very carefully, and turn of their lights and fans the instant their usage hits the slab boundary. Or, more realistically, there s probably some level of corruption involved, where customers pay a small sum to the meter reading staff to ensure that it stays exactly at the slab boundary, giving them the advantage of a lower price.

41 Github: Elections: Speechopedia: AAP: Cricket: Flags: LINKS

42 Try it! All you need is some data and some curiosity to VISUALISE DATA