Business Problems and Data Science Solutions
|
|
- Buddy Hampton
- 7 years ago
- Views:
Transcription
1 CSCI E-84 A Practical Approach to Data Science Ramon A. Mata-Toledo, Ph.D. Professor of Computer Science Harvard Extension School Unit 1 - Lecture 2 February, Wednesday 3, 2016
2 Business Problems and Data Science Solutions This lecture is based primarily on Chapters 2 and 3 of the book Data Science for Business by Foster Provost and Tom Fawcett, Thanks also to Professor P. Adamopoulos (Stern School of Business of New York University) and Professor Tomer Geva (The Tel Aviv University School of Management) Figures are used with the authors permission.
3 Lecture Objectives At the end of this lecture, the student should be able to identify and define concepts such as: Data Mining Process Data Mining Models Data Mining Tasks Supervised Segmentation (purity, information gain, entropy) In addition, this lecture will introduce you to: Supervised versus Unsupervised Methods Data Warehousing Database Querying On-line Analytical Data Processing Categorical and Quantitative data
4 Business Scenario You just landed a great analytical job with MegaTelCo, one of the largest telecommunication firms in the US. They are having a major problem with customer retention in their wireless business. In the mid-atlantic region, 20% of cell phone customers leave when their contracts expire. Communications companies are now engaged in battles to attract each other s customers while retaining their own. Marketing has already designed a special retention offer. Your task is to devise a precise, step-by-step plan for how the data science team should use MegaTelCo s vast data resources to solve the problem.
5 MegaTelCo: Predicting Customer Churn What data you might use? How would they be used? How should MegaTelCo choose a set of customers to receive their offer in order to best reduce churn for a particular incentive budget?
6
7 Terminology Model: A simplified representation of reality created to serve a purpose Predictive Model: A formula for estimating the unknown value of interest: the target The formula can be mathematical, logical statement (e.g., rule), etc. Prediction: Estimate an unknown value (i.e. the target) Instance / example: Represents a fact or a data point Described by a set of attributes (fields, columns, variables, or features)
8 Terminology (Continuation) Model induction: The creation of models from data. The procedure that creates the model is called the induction algorithm or learner Training data: The input data for the induction algorithm Individual: Any entity about which we have data (customer, business, industry)
9 Terminology (Continuation) Informative variables (attributes): provide information about the characteristics of the entity. It is desirable to choose those variables that correlate well with the target variable.
10 What is a model? A simplified * representation of reality created for a specific purpose * based on some assumptions deemed important for the target Examples: map (abstraction of physical world), engineering prototypes, the Black-Scholes option pricing model, etc. Data Mining example: formula for predicting probability of customer attrition at contract expiration! classification model or class-probability estimation model
11 Data Mining Process Data mining is a process that breaks up the overall task of finding patterns from data into a set of well-defined set of subtasks. The solutions to these subtasks can often be put together to solve the overall problem. science + craft + creativity + common sense Data Mining is a process
12 Cross-Industry Standard Process for Data Mining CRISP-DM Business Understanding Data Understanding Data Preparation Modeling Evaluation Deployment Determine Business Objectives Collect Initial Data Select Data Select Modeling Technique Evaluate Results Plan Deployment Assess Situation Describe Data Clean Data Generate Test Design Review Process Plan Monitoring & Maintenance Determine Data Mining Goals Explore Data Construct Data Build Model Determine Next Steps Produce Final Report Produce Project Plan Verify Data Quality Integrate Data Assess Model Review Project Format Data
13 Data Warehousing / Storage Data Mining versus Data Warehousing/Storage Data warehouses coalesce data from across an enterprise, often from multiple transaction-processing systems
14 Data Mining versus Database Querying and Reporting Database Querying / Reporting (SQL, Excel, QBE, other GUI-based querying) Queries are requests posed to a database for particular subsets of the data Very flexible interface to produce reports summaries of the data No modeling or sophisticated pattern finding Most of the cool visualizations
15 Data Mining versus On-line Analytical Data Processing OLAP On-line Analytical Processing OLAP provides easy-to-use GUI to explore large data collections Exploration is manual; no modeling Dimensions of analysis preprogrammed into OLAP system
16 Data Mining versus Traditional statistical analysis Mainly based on hypothesis testing or estimation / quantification of uncertainty Should be used to follow-up on data mining s hypothesis generation Automated statistical modeling (e.g., advanced regression) This is data mining, one type usually based on linear models Massive databases allow non-linear alternatives
17 Answering business questions with these techniques.. Who are the most profitable customers? Database querying Is there really a difference between profitable customers and the average customer? Statistical hypothesis testing But who really are these customers? Can I characterize them? OLAP (manual search), Data mining (automated pattern finding) Will some particular new customer be profitable? How much revenue should I expect this customer to generate? Data mining (predictive modeling)
18 Common Data Mining Tasks Classification estimation Given a sample population predict for each individual the class to which it belongs. The set of classes are generally small and disjoint (non overlapping). Class probability estimation (scoring) A model produces, for each individual, a value or score which is the probability that the individual belongs to the class. How likely is this consumer to respond to our campaign? Regression Given a sample population, regression, attempts to estimate the relationship between a selected dependent variable and one or more independent variable. How much will she use the service?
19 Common Data Mining Tasks (continuation) Similarity Matching Process that attempts to identify similar individual based on particular data about them. Can we find consumers similar to my best customers? Clustering Process that attempts to group individual in a population based on similarities w/o a specific target. Do my customers form natural groups? Co-Occurrence Grouping (known also as frequent itemset mining, association rule discovery, market-basket analysis) Process attempts to find associations based on their transactions. Customers that bought this item also bought
20 Common Data Mining Tasks (continuation) Profiling Process that attempts to characterize the typical behavior of an individual, group, or population. Often used to establish a baseline. Used also to observe out of the normal behavior for fraud detection. What items do my best customer buy more frequently? Link Prediction Process that attempts to predict connections between nodes of a graph based on the information of the individual nodes. Often used in social networks such as Facebook or LinkedIn. People that you may know Data Reduction Process that transforms a large set of data to replace it with a smaller set that contains most of the relevant information. What items are purchased together?
21 Common Data Mining Tasks (continuation) Casual Modeling Process that attempts to help us understand the actions or events that influence other events. Did the advertising campaign increased the volume of purchases? This type of technique may include randomized controlled experiments with two variants (A/B tests). One of the variants serves as the control; the other is the variation of the experiment. Which of the two promotional codes increased volume of sales? (assuming an advertisement sent two identical populations with variations in the promotional code)
22 Supervised versus Unsupervised Data Mining Unsupervised (no specific target) Do our customers naturally fall into different groups? No guarantee that the results are meaningful or will be useful for any particular purpose Supervised (specific target) Can we find groups of customers who have particularly high likelihoods of canceling their service soon after contracts expire? Results are generally more useful than unsupervised. Supervised data mining require different techniques than unsupervised. Supervised Data Mining requires that the target information be in the data. Example: attempts to retrieve historical behavior of customer for last year when data only has been kept for 6 months.
23 Common Data Mining Task Supervised vs. Unsupervised
24 Supervised Data Mining & Predictive Modeling Is there a specific, quantifiable target that we are interested in or trying to predict? Think about the decision Do we have data on this target? Do we have enough data on this target? Need a min ~500 of each type of classification Do we have relevant data prior to decision? Think timing of decision and action The result of supervised data mining is a model that predicts some quantity A model can either be used to predict or to understand
25 Subclasses of Supervised Data Mining
26 Data Mining Model in Use New data item Probability estimation model prediction of prob. of class
27 Model in use: Data Mining versus Use of the Model
28 Supervised Segmentation How can we divide (segment) the population into groups that differ from each other with respect to some quantity that we would like to predict? Which customers will churn after their expiration date? Which customers are likely to default? Informative attributes Find knowable attributes that correlate with the target of interest Increase accuracy Alleviate computational problems E.g., tree induction
29 Supervised Segmentation (Continuation) How can we judge whether a variable contains important information about the target variable? How much? What single variable give the most information about the target? Based on this attribute, how can we partition the customers into subgroups that are as pure as possible with respect to the target (i.e., such that in each group as many instances as possible belong to the same class). A subgroup is pure if every member of the group (instance) has the same value for the target otherwise is impure.
30 Selecting Informative Attributes Example: A binary (Yes, No) classification problem with the following attributes: Head shape (square, circular) Body shape (rectangular, oval) Body color (red, blue, purple) Target variable: Yes, No No Yes Yes Yes Yes No Yes No
31 Purity Measure Information Gain Entropy Difficulties encountered in segmenting the population due to the impurity of the subgroups can be addressed by creating a formula that evaluates how well each attribute separates the entities into segments. The formula is called a purity measure. The most common criterion for segmenting the population is called information gain (IG). Information gain is based on a purity measure called entropy. Entropy is a measure of the degree of disorder that can be applied to a set. The entropy is defined as follows: entropy = pi lg(pi) P i : the probability (the relative percentage) of property i within the set P i = 0 when no member of the set satisfy the property P i = 1 when all the members of the set satisfy the property lg(p i ) stands for the logarithm of base 2: log 2 p i
32 Calculating the Information Gain IG (parent, children) = entropy(parent) [p(c1) entropy(c1)+p(c2) entropy(c2) + ] Parent Child 1 (c 1 ) Child 2 (c 2 ) Child Note: Higher IG indicates a more informative split by the variable.
33 Selecting Informative Attributes Example: Set of 10 people divided into two classes: will default (write-off) and will not default (non-write-off). 7 people will not default; p(n-w-o)=7/10 = 0.7 and p(w-o)=3/10 entropy (S) = -[0.7 x lg(0.7) x lg(0.3)] = 0.88 lg(x) stands for logarithm of base 2. Log (x) is the logarithm of base 10. The logarithm of number x in base 2 equals the logarithm of x in base 10, divided by the logarithm of 2 in base 10. Remember: logarithms of numbers between 0 and 1 are negative.
34 Information Gain Information gain measures the change in entropy due to any amount of new information being added
35 Information Gain
36 Information Gain (continuation) 13/30 approx /30 approx Relative percentages of the subset with respect to the parent. This percentage is calculated dividing the cardinality of the childset by the cardinality of the parent-set.
37 Attribute Selection Reasons for selecting only a subset of attributes: Better insights and business understanding Better explanations and more tractable models Reduced cost Faster predictions Better predictions!
38 Example: Attribution Selection with Information Gain This dataset includes descriptions of hypothetical samples corresponding to 23 species of gilled mushrooms in the Agaricus and Lepiota Family Each species is identified as definitely edible, definitely poisonous, or of unknown edibility and not recommended This latter class was combined with the poisonous one The Guide clearly states that there is no simple rule for determining the edibility of a mushroom; no rule like leaflets three, let it be for Poisonous Oak and Ivy.
39 Example: Attribution Selection with Information Gain (continuation)
40 Example: Attribution Selection with Information Gain
41 Example: Attribution Selection with Information Gain
42 Example: Attribution Selection with Information Gain
43 Multivariate Supervised Segmentation If we select the single variable that gives the most information gain, we create a very simple segmentation If we select multiple attributes each giving some information gain, how do we put them together?
44 Categorical and Quantitative Variables A categorical variable places an observation in one (and only one) category chosen from two or more possible categories. If there is no ordering that can be done between the categories, the variable is nominal. If there is some intrinsic order that can be assigned to the categories, the variables are ordinal. Examples of categorical data Your gender (male, female) Your class in school (Freshman, Sophomore, Junior, Senior, Graduate) Your performance status (Probation, Regular, Honors) Your political party (Democrat, Republican, Independent) Your hair color (blonde, brown, red, balck, white, other) Your type of pet (cat, dog, ferret, rabbit, other) Your race (Hispanic, Asian, African American, Caucasian) Machine settings (Low, Medium, High) Method of payment (Cash, Credit) The color of some object (red, orange, yellow, green, blue, purple)
45 Variables and the Case format Before analyzing the data, prepare charts or graphs, or do any statistical test, it s important to gain a sense of what data is all about. Why is this necessary? The type of data available dictates what we can do with it. Do you want to prepare a scatterplot or perform a linear regression to generate a predictive model? Fine as long as you two quantitative variables preferable both continuous Do you want to do a one-way Analysis of Variance (ANOVA)? No problem, as long as you have multiple collections of quantitative variables, and you can split them up into groups using one of the categorical variables you have collected. Do you want to construct a bar plot? OK, but you will need categorical data or else you should be preparing a histogram instead.
46 Questions?
Data Mining for Business Analytics
Data Mining for Business Analytics Lecture 2: Introduction to Predictive Modeling Stern School of Business New York University Spring 2014 MegaTelCo: Predicting Customer Churn You just landed a great analytical
More informationData Mining Applications in Higher Education
Executive report Data Mining Applications in Higher Education Jing Luan, PhD Chief Planning and Research Officer, Cabrillo College Founder, Knowledge Discovery Laboratories Table of contents Introduction..............................................................2
More informationData Mining Algorithms Part 1. Dejan Sarka
Data Mining Algorithms Part 1 Dejan Sarka Join the conversation on Twitter: @DevWeek #DW2015 Instructor Bio Dejan Sarka (dsarka@solidq.com) 30 years of experience SQL Server MVP, MCT, 13 books 7+ courses
More informationLearning Example. Machine learning and our focus. Another Example. An example: data (loan application) The data and the goal
Learning Example Chapter 18: Learning from Examples 22c:145 An emergency room in a hospital measures 17 variables (e.g., blood pressure, age, etc) of newly admitted patients. A decision is needed: whether
More informationBig Data: Rethinking Text Visualization
Big Data: Rethinking Text Visualization Dr. Anton Heijs anton.heijs@treparel.com Treparel April 8, 2013 Abstract In this white paper we discuss text visualization approaches and how these are important
More informationData Mining mit der JMSL Numerical Library for Java Applications
Data Mining mit der JMSL Numerical Library for Java Applications Stefan Sineux 8. Java Forum Stuttgart 07.07.2005 Agenda Visual Numerics JMSL TM Numerical Library Neuronale Netze (Hintergrund) Demos Neuronale
More informationIn this presentation, you will be introduced to data mining and the relationship with meaningful use.
In this presentation, you will be introduced to data mining and the relationship with meaningful use. Data mining refers to the art and science of intelligent data analysis. It is the application of machine
More informationMethods for Interaction Detection in Predictive Modeling Using SAS Doug Thompson, PhD, Blue Cross Blue Shield of IL, NM, OK & TX, Chicago, IL
Paper SA01-2012 Methods for Interaction Detection in Predictive Modeling Using SAS Doug Thompson, PhD, Blue Cross Blue Shield of IL, NM, OK & TX, Chicago, IL ABSTRACT Analysts typically consider combinations
More informationData, Measurements, Features
Data, Measurements, Features Middle East Technical University Dep. of Computer Engineering 2009 compiled by V. Atalay What do you think of when someone says Data? We might abstract the idea that data are
More informationDatabase Marketing, Business Intelligence and Knowledge Discovery
Database Marketing, Business Intelligence and Knowledge Discovery Note: Using material from Tan / Steinbach / Kumar (2005) Introduction to Data Mining,, Addison Wesley; and Cios / Pedrycz / Swiniarski
More informationChapter 20: Data Analysis
Chapter 20: Data Analysis Database System Concepts, 6 th Ed. See www.db-book.com for conditions on re-use Chapter 20: Data Analysis Decision Support Systems Data Warehousing Data Mining Classification
More informationSTATISTICA. Clustering Techniques. Case Study: Defining Clusters of Shopping Center Patrons. and
Clustering Techniques and STATISTICA Case Study: Defining Clusters of Shopping Center Patrons STATISTICA Solutions for Business Intelligence, Data Mining, Quality Control, and Web-based Analytics Table
More informationIntroduction to Data Mining
Introduction to Data Mining 1 Why Data Mining? Explosive Growth of Data Data collection and data availability Automated data collection tools, Internet, smartphones, Major sources of abundant data Business:
More informationData Mining: Overview. What is Data Mining?
Data Mining: Overview What is Data Mining? Recently * coined term for confluence of ideas from statistics and computer science (machine learning and database methods) applied to large databases in science,
More informationEnvironmental Remote Sensing GEOG 2021
Environmental Remote Sensing GEOG 2021 Lecture 4 Image classification 2 Purpose categorising data data abstraction / simplification data interpretation mapping for land cover mapping use land cover class
More informationIndex Contents Page No. Introduction . Data Mining & Knowledge Discovery
Index Contents Page No. 1. Introduction 1 1.1 Related Research 2 1.2 Objective of Research Work 3 1.3 Why Data Mining is Important 3 1.4 Research Methodology 4 1.5 Research Hypothesis 4 1.6 Scope 5 2.
More informationNine Common Types of Data Mining Techniques Used in Predictive Analytics
1 Nine Common Types of Data Mining Techniques Used in Predictive Analytics By Laura Patterson, President, VisionEdge Marketing Predictive analytics enable you to develop mathematical models to help better
More informationCOMP3420: Advanced Databases and Data Mining. Classification and prediction: Introduction and Decision Tree Induction
COMP3420: Advanced Databases and Data Mining Classification and prediction: Introduction and Decision Tree Induction Lecture outline Classification versus prediction Classification A two step process Supervised
More informationData Mining Part 5. Prediction
Data Mining Part 5. Prediction 5.1 Spring 2010 Instructor: Dr. Masoud Yaghini Outline Classification vs. Numeric Prediction Prediction Process Data Preparation Comparing Prediction Methods References Classification
More informationAzure Machine Learning, SQL Data Mining and R
Azure Machine Learning, SQL Data Mining and R Day-by-day Agenda Prerequisites No formal prerequisites. Basic knowledge of SQL Server Data Tools, Excel and any analytical experience helps. Best of all:
More informationSocial Media Mining. Data Mining Essentials
Introduction Data production rate has been increased dramatically (Big Data) and we are able store much more data than before E.g., purchase data, social media data, mobile phone data Businesses and customers
More informationAn Overview of Database management System, Data warehousing and Data Mining
An Overview of Database management System, Data warehousing and Data Mining Ramandeep Kaur 1, Amanpreet Kaur 2, Sarabjeet Kaur 3, Amandeep Kaur 4, Ranbir Kaur 5 Assistant Prof., Deptt. Of Computer Science,
More informationIntroduction to Data Mining and Machine Learning Techniques. Iza Moise, Evangelos Pournaras, Dirk Helbing
Introduction to Data Mining and Machine Learning Techniques Iza Moise, Evangelos Pournaras, Dirk Helbing Iza Moise, Evangelos Pournaras, Dirk Helbing 1 Overview Main principles of data mining Definition
More informationINTRODUCTION TO DATA MINING SAS ENTERPRISE MINER
INTRODUCTION TO DATA MINING SAS ENTERPRISE MINER Mary-Elizabeth ( M-E ) Eddlestone Principal Systems Engineer, Analytics SAS Customer Loyalty, SAS Institute, Inc. AGENDA Overview/Introduction to Data Mining
More informationApplied Data Mining Analysis: A Step-by-Step Introduction Using Real-World Data Sets
Applied Data Mining Analysis: A Step-by-Step Introduction Using Real-World Data Sets http://info.salford-systems.com/jsm-2015-ctw August 2015 Salford Systems Course Outline Demonstration of two classification
More informationCONTENTS PREFACE 1 INTRODUCTION 1 2 DATA VISUALIZATION 19
PREFACE xi 1 INTRODUCTION 1 1.1 Overview 1 1.2 Definition 1 1.3 Preparation 2 1.3.1 Overview 2 1.3.2 Accessing Tabular Data 3 1.3.3 Accessing Unstructured Data 3 1.3.4 Understanding the Variables and Observations
More informationIBM SPSS Statistics 20 Part 4: Chi-Square and ANOVA
CALIFORNIA STATE UNIVERSITY, LOS ANGELES INFORMATION TECHNOLOGY SERVICES IBM SPSS Statistics 20 Part 4: Chi-Square and ANOVA Summer 2013, Version 2.0 Table of Contents Introduction...2 Downloading the
More informationASSIGNMENT 4 PREDICTIVE MODELING AND GAINS CHARTS
DATABASE MARKETING Fall 2015, max 24 credits Dead line 15.10. ASSIGNMENT 4 PREDICTIVE MODELING AND GAINS CHARTS PART A Gains chart with excel Prepare a gains chart from the data in \\work\courses\e\27\e20100\ass4b.xls.
More informationA Prototype System for Educational Data Warehousing and Mining 1
A Prototype System for Educational Data Warehousing and Mining 1 Nikolaos Dimokas, Nikolaos Mittas, Alexandros Nanopoulos, Lefteris Angelis Department of Informatics, Aristotle University of Thessaloniki
More informationBIG DATA What it is and how to use?
BIG DATA What it is and how to use? Lauri Ilison, PhD Data Scientist 21.11.2014 Big Data definition? There is no clear definition for BIG DATA BIG DATA is more of a concept than precise term 1 21.11.14
More informationData Mining is sometimes referred to as KDD and DM and KDD tend to be used as synonyms
Data Mining Techniques forcrm Data Mining The non-trivial extraction of novel, implicit, and actionable knowledge from large datasets. Extremely large datasets Discovery of the non-obvious Useful knowledge
More informationHow Organisations Are Using Data Mining Techniques To Gain a Competitive Advantage John Spooner SAS UK
How Organisations Are Using Data Mining Techniques To Gain a Competitive Advantage John Spooner SAS UK Agenda Analytics why now? The process around data and text mining Case Studies The Value of Information
More informationFraming Business Problems as Data Mining Problems
Framing Business Problems as Data Mining Problems Asoka Diggs Data Scientist, Intel IT January 21, 2016 Legal Notices This presentation is for informational purposes only. INTEL MAKES NO WARRANTIES, EXPRESS
More informationEasily Identify Your Best Customers
IBM SPSS Statistics Easily Identify Your Best Customers Use IBM SPSS predictive analytics software to gain insight from your customer database Contents: 1 Introduction 2 Exploring customer data Where do
More informationClass 10. Data Mining and Artificial Intelligence. Data Mining. We are in the 21 st century So where are the robots?
Class 1 Data Mining Data Mining and Artificial Intelligence We are in the 21 st century So where are the robots? Data mining is the one really successful application of artificial intelligence technology.
More informationVisualization methods for patent data
Visualization methods for patent data Treparel 2013 Dr. Anton Heijs (CTO & Founder) Delft, The Netherlands Introduction Treparel can provide advanced visualizations for patent data. This document describes
More informationIntroduction Course in SPSS - Evening 1
ETH Zürich Seminar für Statistik Introduction Course in SPSS - Evening 1 Seminar für Statistik, ETH Zürich All data used during the course can be downloaded from the following ftp server: ftp://stat.ethz.ch/u/sfs/spsskurs/
More informationRole of Customer Response Models in Customer Solicitation Center s Direct Marketing Campaign
Role of Customer Response Models in Customer Solicitation Center s Direct Marketing Campaign Arun K Mandapaka, Amit Singh Kushwah, Dr.Goutam Chakraborty Oklahoma State University, OK, USA ABSTRACT Direct
More informationSpecific Usage of Visual Data Analysis Techniques
Specific Usage of Visual Data Analysis Techniques Snezana Savoska 1 and Suzana Loskovska 2 1 Faculty of Administration and Management of Information systems, Partizanska bb, 7000, Bitola, Republic of Macedonia
More informationData Preprocessing. Week 2
Data Preprocessing Week 2 Topics Data Types Data Repositories Data Preprocessing Present homework assignment #1 Team Homework Assignment #2 Read pp. 227 240, pp. 250 250, and pp. 259 263 the text book.
More informationData Mining. 1 Introduction 2 Data Mining methods. Alfred Holl Data Mining 1
Data Mining 1 Introduction 2 Data Mining methods Alfred Holl Data Mining 1 1 Introduction 1.1 Motivation 1.2 Goals and problems 1.3 Definitions 1.4 Roots 1.5 Data Mining process 1.6 Epistemological constraints
More informationData Mining Fundamentals
Part I Data Mining Fundamentals Data Mining: A First View Chapter 1 1.11 Data Mining: A Definition Data Mining The process of employing one or more computer learning techniques to automatically analyze
More informationInsurance Analytics - analýza dat a prediktivní modelování v pojišťovnictví. Pavel Kříž. Seminář z aktuárských věd MFF 4.
Insurance Analytics - analýza dat a prediktivní modelování v pojišťovnictví Pavel Kříž Seminář z aktuárských věd MFF 4. dubna 2014 Summary 1. Application areas of Insurance Analytics 2. Insurance Analytics
More informationData Mining with SQL Server Data Tools
Data Mining with SQL Server Data Tools Data mining tasks include classification (directed/supervised) models as well as (undirected/unsupervised) models of association analysis and clustering. 1 Data Mining
More informationMicrosoft Azure Machine learning Algorithms
Microsoft Azure Machine learning Algorithms Tomaž KAŠTRUN @tomaz_tsql Tomaz.kastrun@gmail.com http://tomaztsql.wordpress.com Our Sponsors Speaker info https://tomaztsql.wordpress.com Agenda Focus on explanation
More informationIntroduction to Data Mining
Introduction to Data Mining Jay Urbain Credits: Nazli Goharian & David Grossman @ IIT Outline Introduction Data Pre-processing Data Mining Algorithms Naïve Bayes Decision Tree Neural Network Association
More informationKnowledge Discovery from patents using KMX Text Analytics
Knowledge Discovery from patents using KMX Text Analytics Dr. Anton Heijs anton.heijs@treparel.com Treparel Abstract In this white paper we discuss how the KMX technology of Treparel can help searchers
More informationData Mining: An Introduction
Data Mining: An Introduction Michael J. A. Berry and Gordon A. Linoff. Data Mining Techniques for Marketing, Sales and Customer Support, 2nd Edition, 2004 Data mining What promotions should be targeted
More informationnot possible or was possible at a high cost for collecting the data.
Data Mining and Knowledge Discovery Generating knowledge from data Knowledge Discovery Data Mining White Paper Organizations collect a vast amount of data in the process of carrying out their day-to-day
More information2015 Workshops for Professors
SAS Education Grow with us Offered by the SAS Global Academic Program Supporting teaching, learning and research in higher education 2015 Workshops for Professors 1 Workshops for Professors As the market
More informationPractical Data Science with Azure Machine Learning, SQL Data Mining, and R
Practical Data Science with Azure Machine Learning, SQL Data Mining, and R Overview This 4-day class is the first of the two data science courses taught by Rafal Lukawiecki. Some of the topics will be
More informationIntroduction to Regression and Data Analysis
Statlab Workshop Introduction to Regression and Data Analysis with Dan Campbell and Sherlock Campbell October 28, 2008 I. The basics A. Types of variables Your variables may take several forms, and it
More informationIntroduction to Data Mining Techniques
Introduction to Data Mining Techniques Dr. Rajni Jain 1 Introduction The last decade has experienced a revolution in information availability and exchange via the internet. In the same spirit, more and
More informationData Mining for Fun and Profit
Data Mining for Fun and Profit Data mining is the extraction of implicit, previously unknown, and potentially useful information from data. - Ian H. Witten, Data Mining: Practical Machine Learning Tools
More informationExample: Credit card default, we may be more interested in predicting the probabilty of a default than classifying individuals as default or not.
Statistical Learning: Chapter 4 Classification 4.1 Introduction Supervised learning with a categorical (Qualitative) response Notation: - Feature vector X, - qualitative response Y, taking values in C
More informationEXPLORING & MODELING USING INTERACTIVE DECISION TREES IN SAS ENTERPRISE MINER. Copyr i g ht 2013, SAS Ins titut e Inc. All rights res er ve d.
EXPLORING & MODELING USING INTERACTIVE DECISION TREES IN SAS ENTERPRISE MINER ANALYTICS LIFECYCLE Evaluate & Monitor Model Formulate Problem Data Preparation Deploy Model Data Exploration Validate Models
More informationGerry Hobbs, Department of Statistics, West Virginia University
Decision Trees as a Predictive Modeling Method Gerry Hobbs, Department of Statistics, West Virginia University Abstract Predictive modeling has become an important area of interest in tasks such as credit
More informationAn Introduction to Advanced Analytics and Data Mining
An Introduction to Advanced Analytics and Data Mining Dr Barry Leventhal Henry Stewart Briefing on Marketing Analytics 19 th November 2010 Agenda What are Advanced Analytics and Data Mining? The toolkit
More informationClassification and Prediction
Classification and Prediction Slides for Data Mining: Concepts and Techniques Chapter 7 Jiawei Han and Micheline Kamber Intelligent Database Systems Research Lab School of Computing Science Simon Fraser
More informationData Mining Techniques
15.564 Information Technology I Business Intelligence Outline Operational vs. Decision Support Systems What is Data Mining? Overview of Data Mining Techniques Overview of Data Mining Process Data Warehouses
More informationCoolaData Predictive Analytics
CoolaData Predictive Analytics 9 3 6 About CoolaData CoolaData empowers online companies to become proactive and predictive without having to develop, store, manage or monitor data themselves. It is an
More informationThe Data Mining Process
Sequence for Determining Necessary Data. Wrong: Catalog everything you have, and decide what data is important. Right: Work backward from the solution, define the problem explicitly, and map out the data
More informationDigging for Gold: Business Usage for Data Mining Kim Foster, CoreTech Consulting Group, Inc., King of Prussia, PA
Digging for Gold: Business Usage for Data Mining Kim Foster, CoreTech Consulting Group, Inc., King of Prussia, PA ABSTRACT Current trends in data mining allow the business community to take advantage of
More informationAn Introduction to Data Mining. Big Data World. Related Fields and Disciplines. What is Data Mining? 2/12/2015
An Introduction to Data Mining for Wind Power Management Spring 2015 Big Data World Every minute: Google receives over 4 million search queries Facebook users share almost 2.5 million pieces of content
More informationDATA MINING METHODS WITH TREES
DATA MINING METHODS WITH TREES Marta Žambochová 1. Introduction The contemporary world is characterized by the explosion of an enormous volume of data deposited into databases. Sharp competition contributes
More informationData Mining for Knowledge Management. Classification
1 Data Mining for Knowledge Management Classification Themis Palpanas University of Trento http://disi.unitn.eu/~themis Data Mining for Knowledge Management 1 Thanks for slides to: Jiawei Han Eamonn Keogh
More informationPrinciples of Data Mining by Hand&Mannila&Smyth
Principles of Data Mining by Hand&Mannila&Smyth Slides for Textbook Ari Visa,, Institute of Signal Processing Tampere University of Technology October 4, 2010 Data Mining: Concepts and Techniques 1 Differences
More informationMBA 8473 - Data Mining & Knowledge Discovery
MBA 8473 - Data Mining & Knowledge Discovery MBA 8473 1 Learning Objectives 55. Explain what is data mining? 56. Explain two basic types of applications of data mining. 55.1. Compare and contrast various
More informationJanuary 26, 2009 The Faculty Center for Teaching and Learning
THE BASICS OF DATA MANAGEMENT AND ANALYSIS A USER GUIDE January 26, 2009 The Faculty Center for Teaching and Learning THE BASICS OF DATA MANAGEMENT AND ANALYSIS Table of Contents Table of Contents... i
More informationSTA-201-TE. 5. Measures of relationship: correlation (5%) Correlation coefficient; Pearson r; correlation and causation; proportion of common variance
Principles of Statistics STA-201-TE This TECEP is an introduction to descriptive and inferential statistics. Topics include: measures of central tendency, variability, correlation, regression, hypothesis
More informationTHE PREDICTIVE MODELLING PROCESS
THE PREDICTIVE MODELLING PROCESS Models are used extensively in business and have an important role to play in sound decision making. This paper is intended for people who need to understand the process
More informationData Mining with SAS. Mathias Lanner mathias.lanner@swe.sas.com. Copyright 2010 SAS Institute Inc. All rights reserved.
Data Mining with SAS Mathias Lanner mathias.lanner@swe.sas.com Copyright 2010 SAS Institute Inc. All rights reserved. Agenda Data mining Introduction Data mining applications Data mining techniques SEMMA
More informationGrow Revenues and Reduce Risk with Powerful Analytics Software
Grow Revenues and Reduce Risk with Powerful Analytics Software Overview Gaining knowledge through data selection, data exploration, model creation and predictive action is the key to increasing revenues,
More informationWebFOCUS RStat. RStat. Predict the Future and Make Effective Decisions Today. WebFOCUS RStat
Information Builders enables agile information solutions with business intelligence (BI) and integration technologies. WebFOCUS the most widely utilized business intelligence platform connects to any enterprise
More informationInternational Journal of Computer Science Trends and Technology (IJCST) Volume 2 Issue 3, May-Jun 2014
RESEARCH ARTICLE OPEN ACCESS A Survey of Data Mining: Concepts with Applications and its Future Scope Dr. Zubair Khan 1, Ashish Kumar 2, Sunny Kumar 3 M.Tech Research Scholar 2. Department of Computer
More informationMachine Learning. Term 2012/2013 LSI - FIB. Javier Béjar cbea (LSI - FIB) Machine Learning Term 2012/2013 1 / 34
Machine Learning Javier Béjar cbea LSI - FIB Term 2012/2013 Javier Béjar cbea (LSI - FIB) Machine Learning Term 2012/2013 1 / 34 Outline 1 Introduction to Inductive learning 2 Search and inductive learning
More informationMachine Learning and Data Analysis overview. Department of Cybernetics, Czech Technical University in Prague. http://ida.felk.cvut.
Machine Learning and Data Analysis overview Jiří Kléma Department of Cybernetics, Czech Technical University in Prague http://ida.felk.cvut.cz psyllabus Lecture Lecturer Content 1. J. Kléma Introduction,
More informationBanking Analytics Training Program
Training (BAT) is a set of courses and workshops developed by Cognitro Analytics team designed to assist banks in making smarter lending, marketing and credit decisions. Analyze Data, Discover Information,
More informationDMDSS: Data Mining Based Decision Support System to Integrate Data Mining and Decision Support
DMDSS: Data Mining Based Decision Support System to Integrate Data Mining and Decision Support Rok Rupnik, Matjaž Kukar, Marko Bajec, Marjan Krisper University of Ljubljana, Faculty of Computer and Information
More informationNCSS Statistical Software Principal Components Regression. In ordinary least squares, the regression coefficients are estimated using the formula ( )
Chapter 340 Principal Components Regression Introduction is a technique for analyzing multiple regression data that suffer from multicollinearity. When multicollinearity occurs, least squares estimates
More informationHYPOTHESIS TESTING: CONFIDENCE INTERVALS, T-TESTS, ANOVAS, AND REGRESSION
HYPOTHESIS TESTING: CONFIDENCE INTERVALS, T-TESTS, ANOVAS, AND REGRESSION HOD 2990 10 November 2010 Lecture Background This is a lightning speed summary of introductory statistical methods for senior undergraduate
More informationData Mining: An Overview. David Madigan http://www.stat.columbia.edu/~madigan
Data Mining: An Overview David Madigan http://www.stat.columbia.edu/~madigan Overview Brief Introduction to Data Mining Data Mining Algorithms Specific Eamples Algorithms: Disease Clusters Algorithms:
More informationExtend Table Lens for High-Dimensional Data Visualization and Classification Mining
Extend Table Lens for High-Dimensional Data Visualization and Classification Mining CPSC 533c, Information Visualization Course Project, Term 2 2003 Fengdong Du fdu@cs.ubc.ca University of British Columbia
More informationData Mining. SPSS Clementine 12.0. 1. Clementine Overview. Spring 2010 Instructor: Dr. Masoud Yaghini. Clementine
Data Mining SPSS 12.0 1. Overview Spring 2010 Instructor: Dr. Masoud Yaghini Introduction Types of Models Interface Projects References Outline Introduction Introduction Three of the common data mining
More informationSurvival Analysis of the Patients Diagnosed with Non-Small Cell Lung Cancer Using SAS Enterprise Miner 13.1
Paper 11682-2016 Survival Analysis of the Patients Diagnosed with Non-Small Cell Lung Cancer Using SAS Enterprise Miner 13.1 Raja Rajeswari Veggalam, Akansha Gupta; SAS and OSU Data Mining Certificate
More informationDATA MINING TECHNOLOGY. Keywords: data mining, data warehouse, knowledge discovery, OLAP, OLAM.
DATA MINING TECHNOLOGY Georgiana Marin 1 Abstract In terms of data processing, classical statistical models are restrictive; it requires hypotheses, the knowledge and experience of specialists, equations,
More informationUsing Predictions to Power the Business. Wayne Eckerson Director of Research and Services, TDWI February 18, 2009
Using Predictions to Power the Business Wayne Eckerson Director of Research and Services, TDWI February 18, 2009 Sponsor 2 Speakers Wayne Eckerson Director, TDWI Research Caryn A. Bloom Data Mining Specialist,
More information131-1. Adding New Level in KDD to Make the Web Usage Mining More Efficient. Abstract. 1. Introduction [1]. 1/10
1/10 131-1 Adding New Level in KDD to Make the Web Usage Mining More Efficient Mohammad Ala a AL_Hamami PHD Student, Lecturer m_ah_1@yahoocom Soukaena Hassan Hashem PHD Student, Lecturer soukaena_hassan@yahoocom
More informationHow to use Big Data in Industry 4.0 implementations. LAURI ILISON, PhD Head of Big Data and Machine Learning
How to use Big Data in Industry 4.0 implementations LAURI ILISON, PhD Head of Big Data and Machine Learning Big Data definition? Big Data is about structured vs unstructured data Big Data is about Volume
More informationTechnology and Trends for Smarter Business Analytics
Don Campbell Chief Technology Officer, Business Analytics, IBM Technology and Trends for Smarter Business Analytics Business Analytics software Where organizations are focusing Business Analytics Enhance
More informationIBM SPSS Statistics 20 Part 1: Descriptive Statistics
CALIFORNIA STATE UNIVERSITY, LOS ANGELES INFORMATION TECHNOLOGY SERVICES IBM SPSS Statistics 20 Part 1: Descriptive Statistics Summer 2013, Version 2.0 Table of Contents Introduction...2 Downloading the
More informationPredictive Analytics Certificate Program
Information Technologies Programs Predictive Analytics Certificate Program Accelerate Your Career Offered in partnership with: University of California, Irvine Extension s professional certificate and
More informationData Mining with R. Decision Trees and Random Forests. Hugh Murrell
Data Mining with R Decision Trees and Random Forests Hugh Murrell reference books These slides are based on a book by Graham Williams: Data Mining with Rattle and R, The Art of Excavating Data for Knowledge
More informationFoundations of Artificial Intelligence. Introduction to Data Mining
Foundations of Artificial Intelligence Introduction to Data Mining Objectives Data Mining Introduce a range of data mining techniques used in AI systems including : Neural networks Decision trees Present
More informationII. DISTRIBUTIONS distribution normal distribution. standard scores
Appendix D Basic Measurement And Statistics The following information was developed by Steven Rothke, PhD, Department of Psychology, Rehabilitation Institute of Chicago (RIC) and expanded by Mary F. Schmidt,
More informationProduct recommendations and promotions (couponing and discounts) Cross-sell and Upsell strategies
WHITEPAPER Today, leading companies are looking to improve business performance via faster, better decision making by applying advanced predictive modeling to their vast and growing volumes of data. Business
More informationData Warehousing and Data Mining in Business Applications
133 Data Warehousing and Data Mining in Business Applications Eesha Goel CSE Deptt. GZS-PTU Campus, Bathinda. Abstract Information technology is now required in all aspect of our lives that helps in business
More informationLecture 6 - Data Mining Processes
Lecture 6 - Data Mining Processes Dr. Songsri Tangsripairoj Dr.Benjarath Pupacdi Faculty of ICT, Mahidol University 1 Cross-Industry Standard Process for Data Mining (CRISP-DM) Example Application: Telephone
More informationOLAP and Data Mining. Data Warehousing and End-User Access Tools. Introducing OLAP. Introducing OLAP
Data Warehousing and End-User Access Tools OLAP and Data Mining Accompanying growth in data warehouses is increasing demands for more powerful access tools providing advanced analytical capabilities. Key
More informationAdvanced In-Database Analytics
Advanced In-Database Analytics Tallinn, Sept. 25th, 2012 Mikko-Pekka Bertling, BDM Greenplum EMEA 1 That sounds complicated? 2 Who can tell me how best to solve this 3 What are the main mathematical functions??
More information