Collision Data Analysis, A Mul0 Dimensional Approach Presented by: Howard Sco> Needham, Sandarbh Singh

Size: px

Start display at page:

Download "Collision Data Analysis, A Mul0 Dimensional Approach Presented by: Howard Sco> Needham, Sandarbh Singh"

Elfreda Anthony
8 years ago
Views:

1 Masters Defense Collision Data Analysis, A Mul0 Dimensional Approach Presented by: Howard Sco> Needham, Sandarbh Singh

2 Introduc0on! We wanted to find a large open source database so we can mine and experiment with! We were interested in experimen:ng with tools that can aid in mining large data stores and what tools an be used to present this data in a forma;ed structure (reports, graphs etc.)! We had a goal of selec:ng an interes:ng dataset that would provide real life (common) interest and of tangible value outside of the computer industry! Found an open database of interest! Na:on Collision Database (NCDB) 5.2M records! 25 columns

data in a forma;ed structure (reports, graphs etc.)!

3 Contribu0ons! We found limited tools that were available for novice users! Datasets are to large to be handled by common tools that novice users use to work with (eg. Excel, Access, Crystal Reports)! Tools were limited in func:onality and limited to two dimensional databases and data structures! Those that were capable of MDD analysis were expensive and required complex programming elements! Oracle Reports Server! EBM InfoSphere Master Data Management! Our solu0on provided a Mul0- Dimensional Database! Allows users to access data on mul0ple dimensions and generate related reports and graphs via Microso; Excel! Use built in data mining algorithms in Microso; Analysis Services to create sta0s0cal reports and predica0ons

Those that were capable of MDD analysis were expensive and required complex programming elements! Oracle Reports Server! EBM InfoSphere Master Data Management!

4 Research Ques0ons! What trends can we find in the collision database?! What a;ributes are significant in effec:ng these trends?! Can we provided predic:ons based on selected a;ributes?! Can we provide a web interface for the collision database?! While trying to answer these research ques:ons we learned how to transform raw data into meaningful rela:onships in which the trends can emerge

! Can we provided predic:ons based on selected a;ributes?

5 ! MicrosoT Analysis Tool was selected as our primary ETL tool! Sequence of data loading and cleansing! Extract: from csv file to MicrosoT SQL database! Transform and Load: converted from original data structures (strings) to appropriate data types (:ny int, varchar etc)! Dimensions Staging: Created a staging database to stage the dimension and fact table. Explana:on selec:ng dimension and facts on next slide! A>ribute Banding: defined range for data elements! Data cleansing: missing or unknown or corrupt we decided to keep it and mark it.! Replace Data for Data Cleansing: looked for data that did not make sense for example! Fact Table Loading: created scripts that populates the fact table rows by joining the transformed table with dimension table

Dimensions Staging: Created a staging database to stage the dimension and fact table. Explana:on selec:ng dimension and facts on next slide!

6 ! Making sense of data to decide dimension table and fact tables! Fact - The data point that you are interested in. For example Sale of flight 0ckets. Each sale of a 0cket is a fact.! Dimensions A>ributes that are related to the facts you want to analyze. For Example A sale of flight 0cket might be related to weather condi0ons, adver0sement made for the promo0on offer for flights etc.

7 ! CSV File raw Data C_YEAR C_MNTH C_WDAY C_HOUR C_SEV C_VEHS C_CONF C_RCFG C_WTHR C_RSUR C_RALN C_TRAF V_ID V_TYPE V_YEAR P_ID P_SEX P_AGE P_PSN P_ISEV P_SAFE P_USER UU M UU UU M UU UU F ! Transformed staging table

2 2 34 UU 1 5 3 3 1 6 1990 1 M 41 11 1 UU 1 1999 1 1 20 2 2 34 UU 1 5 3 3 2 1 1987 1 M 19

8 ! Look up table from Data Dic0onary

9 ! Collision Dimension table! Keys, Each row represents a unique type of collision

10 ! Dimensions- facts rela0on Methodology

11 ! User Interface! We wanted to test interac:ng with the database and Mul:- Dimensional Data Cubes! Developed a preliminary version for front end reports and graphs using MicrosoT Excel! Access the cubes directly by firing the MDX queries! Results are to drive the development of a web based system

12 ! Displays the dimensions defined! Expand on each dimension! Select appropriate a;ributes

13 ! Dynamically Change Chart Types [Collision per Year]

14 ! Filter on secondary dimensions Methodology

15 ! Dynamically change the data elements to query on

16 ! Dynamically change the data elements to query on

17 ! Dynamically format the graph(s) drag elements

18 ! Combine reports and graphs! Dynamically format the graph(s) drag elements

19 Results! Data Warehouse! Development of data cubes! Rela:onships we can look at in MDD Cubes! Temporal logic and trend analysis! Excel Interface! Easy to use and intui:ve! Excel crashed a lot while working with it! Unable to map MDD Cubes to MDD graphs! Must have direct access to the database! Was able to establish some trends with the help of excel! Good prototype

Excel Interface! Easy to use and intui:ve! Excel crashed a lot while working with it!

20 Conclusion! We looked at how to transform raw data into meaningful structure to find trends! Look at a>ributes that contribute trends! Development of a web based applica0on is feasible! TBD Predic0on Analysis

21 Limita0ons! Dataset we accessed is very limited! Was the air bag deployed! Was the driver of the vehicle impaired or under the influence! Excel is limited in its ability to display data in graphs and charts (no MDD capabili0es)! More trends may emerge if finer grained dimensions are implemented

SQL Server 2012 Business Intelligence Boot Camp

SQL Server 2012 Business Intelligence Boot Camp Length: 5 Days Technology: Microsoft SQL Server 2012 Delivery Method: Instructor-led (classroom) About this Course Data warehousing is a solution organizations