Unit 6. Business intelligence and Data Mining

Transcription

1 Unit 6 Business intelligence and Data Mining

2 Objectives Upon completion of this unit you will be able to: define business intelligence discuss the importance of data mining apply drivers the for business intelligence initiatives in modern organizations recognize the structure, components, and process of BI

3 Business Intelligence (BI) Business intelligence (BI) is a broad category of applications and technologies for gathering, storing, analyzing, and providing access to data to help enterprise users make better business decisions. Generally according to Al-Azmi (2013) business intelligence refers to the skills, processes, technologies, applications, and practices used to leverage an institution s internal and external information assets to support and improve decision making. Business Intelligence solutions can be divided into two groups of analysis types. Query-Reporting-Analysis: This type of analysis is often query based and is normally used for determining What happened? in a business over a given period of time. Because queries are used the user already knows what kind of information to search for. Additionally, Business Intelligence solutions of this kind are generally operated manually and are therefore time consuming. Intelligent Analysis (Data Mining): While the Query-Reporting-Analysis is able to provide answers for questions of the What happened? kind, Data Mining utilizes clever algorithms for a much deeper and intelligent analysis of data. BI solutions using Data Mining techniques are then capable of handling What will happen? and How/why did this happen? matters. All this is done in a semi- or full-automatic process saving both time and resources.

4 Business Intelligence (BI) Cont... BI applications include the activities of decision support systems, query and reporting, online analytical processing (OLAP), statistical analysis, forecasting and data mining. Business intelligence applications can be: Mission-critical and integral to an enterprise's operations or occasional to meet a special requirement Enterprise-wide or local to one division, department Centrally initiated or driven by user demand

5 Business Intelligence (BI) Cont... The term BI was used as early as September, 1996, when a Gartner Group report said: By 2000, Information Democracy will emerge in forward-thinking enterprises, with Business Intelligence information and applications available broadly to employees, consultants, customers, suppliers, and the public. The key to thriving in a competitive marketplace is staying ahead of the competition. Making sound business decisions based on accurate and current information takes more than intuition. Data analysis, reporting, and query tools can help business users wade through a sea of data to synthesize valuable information from it - today these tools collectively fall into a category called Business Intelligence. The terms business intelligence and business analytics are often used interchangeably. However, there are some key differences

6 Business Intelligence (BI) Cont... Business Intelligence answers questions like: Business Analytics answers questions like: What happened? When? Who? How many? Why did it happen? Will it happen again? What will happen if we change x? What else does the data tell us that never thought to ask?

7 Business Intelligence (BI) Cont... BI includes: Reporting (KPIs, metrics) Automated Monitoring/Alerting (thresholds) Dashboards Scorecards OLAP (Cubes, Slice & Dice, Drilling) Ad hoc query BA includes: Statistical/Quantitative Analysis Data Mining Predictive Modelling Multivariate Testing

8 Business Intelligence (BI) Cont... Why institutions concern about Business Intelligence? The purpose of business intelligence is to support the mission and goals of the institution by enabling fact-based decision making. An effective business intelligence solution can be used to: provide insight and measurement regarding strategic and tactical efforts provide the ability to see the big picture and to find the needle in the haystack support fact-based decision making provide rapid feedback regarding actions validate or discredit assumptions discover non-intuitive relationships

9 Business Intelligence (BI) Cont... Business Intelligence Techniques(2) There are huge varieties of BI solutions and techniques available AQL (Associative Query Logic): Analytical data processing tool that compared to OLAP is less time consuming and more machine driven. Score carding, Dash boarding and Information visualization: Scorecarding is a method that allows managers to get a broad view of the performance of a business while Dashboarding/ Information Visualization deal with visual representation of abstract data. Business Performance Management A tool for analyzing the current state of a business and for improving future strategies. DM (Data mining): Numerous methods for automatically searching large amounts of data for patterns and other interesting relations. Data warehouses - Logical collections of information with structures that favour efficient data analysis (such as OLAP). DSS (Decision Support Systems): Machine driven system that aids the decision making process in a business. Document warehouses: Instead of informing the business what things have happened (like the data warehouse does) the document warehouse is able to state why things have happened.

10 Business Intelligence (BI) Cont... Business Intelligence Techniques (2) EIS (Executive Information Systems): These systems are often considered as a specialized form of DSS with the purpose of facilitating the information and decision making needs of senior executives. MIS (Management Information Systems): A machine driven system for processing data and providing analysis reports for decision making and planning. In order to retrieve data the system has access to all communication channels in a business. GIS (Geographic Information Systems): A computer system for working with geographical data (e.g. satellite images) with editing, analyzing and displaying functionality. OLAP (Online Analytical Processing): OLAP is a tool for doing quick analytical processing of multidimensional data by running queries against structured OLAP cubes that is build from a set of data sources. Text mining: This task is generally referred to as the process of extracting interesting and nontrivial information/knowledge from unstructured text

11 Data Mining What is it? Data mining is the process of analyzing data from different perspectives and summarizing it into useful information - information that can be used to increase revenue, cuts costs, or both. Data mining software is one of a number of analytical tools for analyzing data. It allows users to analyze data from many different dimensions or angles, categorize it, and summarize the relationships identified. Technically, data mining is the process of finding correlations or patterns among dozens of fields in large relational databases. A major goal of data mining is to discover previously unknown relationships among the data

12 Data mining parameters include: Association - looking for patterns where one event is connected to another event Sequence or path analysis - looking for patterns where one event leads to another later event Classification - looking for new patterns (May result in a change in the way the data is organized but that's ok) Clustering - finding and visually documenting groups of facts not previously known Forecasting - discovering patterns in data that can lead to reasonable predictions about the future (This area of data mining is known as predictive analytics.) Data mining techniques are used in a many research areas, including mathematics, cybernetics, genetics and marketing. Web mining, a type of data mining used in customer relationship management (CRM), takes advantage of the huge amount of information gathered by a Web site to look for patterns in user behaviour.

13 Organizations are using data mining to help manage all phases of the customer life cycle, including acquiring new customers, increasing revenue from existing customers, and retaining good customers. By determining characteristics of good customers (profiling), a company can target prospects with similar characteristics. By profiling customers who have bought a particular product it can focus attention on similar customers who have not bought that product (cross-selling). By profiling customers who have left, a company can act to retain customers who are at risk for leaving (reducing churn or attrition), because it is usually far less expensive to retain a customer than acquire a new one.

14 Successful data mining There are two keys to success in data mining. First is coming up with a precise formulation of the problem you are trying to solve. A focused statement usually results in the best payoff. The second key is using the right data. After choosing from the data available to you, or perhaps buying external data, you may need to transform and combine it in significant ways.

15 How does data mining work 1. Classification (1) Classification is supposedly the most popular Data Mining tasks considering its broad application domain. Its main purpose is to classify one or more data samples that may consist of few or many features (dimensions). The latter case makes the classification task more complex due to the large number of dimensions. The actual number of classes is not always given or obvious in a classification task. Therefore, it is possible to distinguish between supervised and unsupervised classification. For supervised classification the number of classes is known along with the properties of each class. Neither of these is given in unsupervised classification which makes this task the more challenging one of the two.

16 1. Classification (2) The list below further exemplifies the use of the classification task. Is a given credit card transaction fraudulent? What type of subscription should be offered a given customer? What type of structure does a specific protein have? Is this customer likely to buy a bicycle? Why is my system failing?

17 2. Estimation Estimation is somewhat similar to classification, algorithm-wise. However, estimation does not deal with determining a class for a particular data sample. Instead, it tries to predict a certain measure for a given data sample. The list below further exemplifies the use of the estimation task. What is the turnover of a company going to be? What is the density of a given fluid? When will a pregnant woman give birth? For how long will this product work before failing? How much is a specific project going to cost?

18 3. Segmentation Segmentation basically deals with the task of grouping a given data set into a few main groups (clusters). The task of describing a large multidimensional data set (say customers) will therefore benefit from the use of segmentation. Moreover, many algorithm types can be used in segmentation systems. The list below further exemplifies the use of the segmentation task. How can a given buyer/supplier group be differentiated? Which types of ground does a given satellite image contain? Is a specific transaction an outlier? Which segments is a market based on? 5Which groups of visitors are using a given search engine?

19 4. Forecasting Forecasting is another important Data Mining task that is used for predicting future data values given a time series of prior data. Forecasting is a popular task often performed using simple statistical methods. However, forecasting done in the Data Mining domain uses advanced (learning) methods (e.g. Neural Networks, Hidden Markov Models) that in many cases are more accurate and informative than the standard statistical methods (e.g. moving averages). The list below further exemplifies the use of the forecasting task. What will the weather be like tomorrow? Will a particular stock price rise over the next couple of days? What are the inventory levels next month? How many sunspots will occur next year? How will the average temperature on earth evolve throughout the next 10 years?

20 5. Association Association deals with task of locating events that are frequently occurring together, this enables them benefiting from this knowledge. One of the most popular examples of association is probably Amazon.com s web shop. This web can or able to recommend related products to customers. The list below further exemplifies the use of the association task. Which products should I recommend to my customers? Which services are used together? Which products are highly likely to be purchased together in a supermarket? Which books are highly likely to be borrowed together in a library? Which dishes from a cookbook go well together?

21 Classes: Stored data is used to locate data in predetermined groups. For example, a restaurant chain could mine customer purchase data to determine when customers visit and what they typically order. This information could be used to increase traffic by having daily specials. Clusters: Data items are grouped according to logical relationships or consumer preferences. For example, data can be mined to identify market segments or consumer affinities Sequential patterns: Data is mined to anticipate behaviour patterns and trends. For example, an outdoor equipment retailer could predict the likelihood of a backpack being purchased based on a consumer's purchase of sleeping bags and hiking shoes.

22 6. Text Analysis Another key Data Mining task is text analysis. Text analysis has several purposes and is often used for finding key terms and phrases in text bits. The text analysis can convert unstructured text into useful structured data that can be further processed by other Data Mining tasks (e.g. classification, segmentation, association). The list below further exemplifies the use of the text analysis task. Which segments does a given mailbox contain? How is a document classified? Which subjects does a specific web page contain? How is a quick overview of multiple lecture notes from a classmate gained? Which terms are likely to occur together?

23 Data types In order to implement and use the different Data Mining tasks, knowledge about their input (data) is a necessity. Qualitative (also categorical, nominal, modal) Variables explicitly describe various properties of an object e.g. hair colour R v = (blonde, brown, brunette, red, etc...) or Boolean values R v = (0, 1). Note that qualitative variables have no intrinsic ordering i.e. there is no agreed way of ordering hair colours from highest to lowest.

24 Quantitative (also real, numeric continuous) Variables may describe properties such as age, profit, temperature. An m dimensional data set, where it for each dimensional holds R v = R, has a m dimensional real object space U = R m. Set valued Variables have multiple attributes e.g. variables for describing movies where each variable has 3 attributes (movie title, leading actors and year).

25 Ordinal Variables are similar to qualitative variables, but with a clear ordering of the variables e.g. the fuel level in a car given by 3 variables R v = (low, medium, high). Various quantitative variables are also regarded to be ordinal. Cyclic Variables are of a periodic nature e.g. the 60 minutes in an hour for which R v = Z/ (60Z). For cyclic variables standard mathematical operations such as addition and subtraction are not directly applicable - certain periodicity precautions are necessary when processing such variables. Seasonal A characteristic of a time series in which the data experiences regular and predictable changes which recur seasonally e.g. calendar year which R v = Z/ (12Z). Any predictable change or pattern in a time series that recurs or repeats over a one-year period can be said to be seasonal. Note that seasonal effects are different from cyclical effects, as seasonal cycles are contained within one calendar year, while cyclical effects (such as boosted sales due to low unemployment rates) can span time periods shorter or longer than one calendar year

26 It is also important to know the deference between structured data sets and unstructured ones. Examples of both structured and unstructured data Structured Data Dimensional Data Buyer demographics Supplier demographics Product properties Transactional Data Order headers with buyer, supplier, ship to address, etc. Order lines with product id, unit price, number of items, etc. Credit card purchases Unstructured Data Textual Data Descriptions of order lines Buyer comments A doctors notes

27 Different levels of analysis are available: Artificial neural networks: Non-linear predictive models that learn through training and resemble biological neural networks in structure. Genetic algorithms: Optimization techniques that use processes such as genetic combination, mutation, and natural selection in a design based on the concepts of natural evolution. Decision trees: Tree-shaped structures that represent sets of decisions. These decisions generate rules for the classification of a dataset. Specific decision tree methods include Classification and Regression Trees (CART) and Chi Square Automatic Interaction Detection (CHAID). CART and CHAID are decision tree techniques used for classification of a dataset. They provide a set of rules that you can apply to a new (unclassified) dataset to predict which records will have a given outcome. CART segments a dataset by creating 2-way splits while CHAID segments using chi square tests to create multi-way splits. CART typically requires less data preparation than CHAID. Nearest neighbour method: A technique that classifies each record in a dataset based on a combination of the classes of the k record(s) most similar to it in a historical dataset (where k 1). Sometimes called the k-nearest neighbour technique. Rule induction: The extraction of useful if-then rules from data based on statistical significance. Data visualization: The visual interpretation of complex relationships in multidimensional data. Graphics tools are used to illustrate data relationships.

28 Thank you