Forecasting, Prediction Models, and Times Series Analysis with Oracle Business Intelligence and Analytics. Rittman Mead BI Forum 2013

Transcription

1 Forecasting, Prediction Models, and Times Series Analysis with Oracle Business Intelligence and Analytics Rittman Mead BI Forum 2013 Dan Vlamis and Tim Vlamis Vlamis Software Solutions

2 Presentation Agenda Understanding classification and forecasting (predictions) Use of Geneva Forecasting engine in Oracle OLAP Holt-Winters and time series Parameter choices ARIMA forecasting algorithm in R Use Oracle R Enterprise Use of time dimension and time series functions in OBI

3 Dan Vlamis and Vlamis Software Solutions Founded in 1992 by Dan Vlamis in Kansas City, MO Developed/implemented more than 200 Oracle BI systems Specializes in ORACLE-based: Business Intelligence Analytic Options to Oracle DB (OLAP, Data Mining, Spatial) Data Warehousing Training and mentoring Expert presenter at major Oracle conferences (blog, papers, newsletters, services) Co-authored book Oracle Essbase & Oracle OLAP Beta tester for OBIEE 11g Reseller for Simba and Nokia map data for OBIEE HOL Coordinator for BIWA Summit 2013

4 Tim Vlamis Bio 20+ years experience in business modeling and valuation, forecasting, and scenario analyses Trainer for Oracle University Two-Day Data Mining Course Professional Certified Marketer (PCM) from AMA Active Member of NICO (Northwestern Institute on Complex Systems) Adjunct Professor of Business Benedictine College MBA Kellogg School of Management (Northwestern) BA Economics Yale University

5 Forecasting Today Predictions are the holy grail of BI systems and initiatives. Most all corporations have need for forecasting. Typical forecasting systems Are stand alone or from ERP (not integrated to BI system) Tend to use straight line or heuristic calculations. Not always integrated into the business. Are often tied directly to the budgeting process High level of angst surrounding forecasts.

6 Forecasting Should Should be integrated with rest of BI system. Should be another series of measures that are revealed in the context of historic information. Should be a part of the Common Enterprise Model. Should have visibility across functional areas and roles in corporations Should leverage most powerful calculation tools (database and BI system) Ideally adjusted based on an integrated view across corporate functions (marketing, operations, finance, etc.)

7 Forecasting Methodologies Rule-based heuristic (last period, last period +5%, etc.) Cross-sectional methodologies (point in time) Time series (time sequenced data series) Mixed models Averages (moving, weighted, etc.) Linear and Non-linear regressions (line fitting) Transforms, projections, min/max

8 Methodologies for Today OLAP Geneva Forecasting Engine Holt Winters for time series Oracle R Enterprise ARIMA ODM Classification and Regression (overview) OBIEE Time Series Functions (overview)

9 OLAP Geneva Forecasting Engine FCOPEN function -- Creates a forecasting context. FCSET command -- Specifies the forecast characteristics. FCEXEC command -- Executes a forecast and populates Oracle OLAP variables with forecasting data. FCQUERY function -- Retrieves information about the characteristics of a forecast or a trial of a forecast. FCCLOSE command -- Closes a forecasting context.

10 METHOD method AUTOMATIC best fit for the data. (Default) LINREG linear regression (y=a*x+b) is fitted to the data. NLREG1 nonlinear regression x'=log(x) and y'=log(y) a polynomial model between x and y(y=c*x^a). NLREG2 nonlinear regression x'=x and y'=ln(y) an exponential model between x and y(y=c*e^ax). NLREG3 nonlinear regression x'=log(x) and y'=y a logarithmic model between x and y(y=a*log(x)+b). NLREG4 nonlinear regression method ix'=1/x and y'=1/y an asymptotic curve (y=x/(a+bx)). NLREG5 nonlinear regression method x'=x and y'=ln(y/(k-y)) an exponential asymptotic curve (y=cke^ax/(1+ce^ax)). SESMOOTH single exponential smoothing method intended for short term forecasts of non-seasonal data. DESMOOTH double exponential smoothing method exponential smoothing is applied to both the series and the trend term. CROSTON Croston's Intermittent Demand method. used for intermittent data where more than half of the observations are zero HOLT/WINTERS triple exponential smoothing. used on seasonal data

11 Using Holt-Winters Triple Exponential Smoothing methodology Used for data suspected to be seasonal Needs multiple seasons Assumes regular periods Pre/post processing may be necessary (fiscal calendar 445, irregular holidays, Black Swans, outages, etc.)

12 Exponential Smoothing Methodology for smoothing data and preferencing more recent periods when doing time series forecasts. Similar conceptually to a weighted moving average Weights decline according to an exponential function. {1, (1-α), (1-α)², (1-α)³, } Higher values give more weight to more recent periods Single (weighted average of most recent observation and the most recent smoothed statistic) Double (trend either up or down) Triple (period effect)

13 FCSET Parameters ALLOCLAST {YES NO} ALPHA {MAX MIN STEP} decimal APPROACH {'APPAUTO' 'APPMA NUAL' BETA {MAX MIN STEP} decimal COMPSMOOTH {YES NO} CYCDECAY {MAX MIN} decimal GAMMA {MAX MIN STEP} decimal HISTPERIODS integer MAXFACTOR decimal METHOD 'method' MINFCFACTOR decimal MPTDECAY {MAX MIN} decimal NTRIALS integer PERIODICITY cycle-spec RATIO decimal SMOOTHING {YES NO} TRANSFORM {'TRNOSEA' 'TRSEA ' 'TRMPT'} TRENDHOLD {MAX MIN STEP} de cimal WINDOWLEN integer

14 Alpha, Beta, Gamma Setting Default Max is 0.3 Default Min is 0.1 Default Step is 0.1 (.05<= divisible value<=0.2) Greater value means nearer periods have more weight. Lower value means periods have more equal weight.

15 Recommendations Be careful of accepting the APPAUTO setting Be aware of Embedded total time dimensions Match HISTPERIODS with PERIODICITY for best results PERIODICITY cycle-spec is hierarchical from higher grain to lower Ex {52,7} 52 weeks in a year, 7 days in a week Ex {4,13,7} 4 quarters in a year, 13 weeks in a quarter, 7 days in a week Ex {12} 12 months in a year Months are challenging to incorporate with other periods

16 Case Study Using Oracle OLAP Forecasted values from Oracle OLAP made no sense Client trying to use Best Fit complicates study because don t know what method chosen Avoid tendency to inherit mistakes Problem in HISTPERIODS parameter Solution: set HISTPERIODS to number of data points Problem in forecasting on hierarchical dimension 12 month periods, 1 year period throwing off forecast Solution: LIMIT TIME TO TIMELEVEL PERIOD periods artificially inflate every 3 rd period Added 3 rd year average of 2 years

17 Example OLAP DML Forecast Program vrb _handle int Removed error handling and definition of temporary variables such as DJOFCST2_C_SEASONAL LIMIT DJOFCST2_C_MEASURE_DIM TO 'QTY_HW' _handle = FCOPEN('MyForecast') limit djotime_d2 to djotime_d2_levelrel eq 'PERIOD' SORT DJOTIME_D2 a DJOTIME_D2_END_DATE "Set forecast parameters for 'best fit' fcset _handle method 'HOLT/WINTERS' APPROACH 'APPMANUAL' SMOOTHING 'YES' MAXFCFACTOR 10.0 TRANSFORM 'TRSEA' - periodicity 12 histperiods 36 BETA MAX 0.5 "Execute the forecast - save seasonal and seasonal smoothed into the variables just defined FCEXEC _handle time DJOTIME_D2 INTO DJOFCST2_C_STORED - seasonal DJOFCST2_C_SEASONAL - smseasonal DJOFCST2_C_SMSEASONAL backcast DJOFCST2_C_QTY ALLSTAT "Close the forecast FCCLOSE _handle update commit return

18 Forecasts Did Not Make Sense

19 Forecasts Did Not Make Sense

20 Holt-Winters Forecast After Fix

21 Holt-Winters Vs. 3-Mo Moving Avg

22 Includes single, double, and triple exponential smoothing techniques. Includes linear and non-linear regression option. Does not include an auto-choice function. Non-linear regression transforms must be manually applied. Many other transform, calculation, and modeling capabilities in Essbase.

23 ARIMA Autoregressive Integrated Moving Average Powerful algorithm for series analysis and prediction Three parameters p, d, q Auto regression (how reliant series values are on previous series values). AR(0) is white noise. Integrated (degree of AR differencing, Random Walk) Moving average (smoothing function) ARIMA (1,0,0) = AR(1) ARIMA (1,0,1) = ARMA (1,1) Large number of potential models ( Know the name Rob Hyndman for ARIMA in R

24 Stationarity Processes with no growth related to time. Random walks are stationary. Necessary to difference non-stationary series before applying ARMA models. (ARIMA handles this through the Integrated term d of the p, d, q model parameters.)

25 Non-Seasonal ARIMA (p, d, q) φ B 1 B d γ t = c + θ B ε t ε t is a white noise process with 0 mean and variance σ 2. B is a backshift operator φ z is a polynomial of order p θ z is a polynomial of order q

26 Seasonal ARIMA p, d, q P, D, Q m Φ B m φ B 1 B D 1 B d γ t = c + Θ B m θ B ε t ε t is a white noise process with 0 mean and variance σ 2. B is a backshift operator Φ z is a polynomial of order p Θ z is a polynomial of order q

27 Forecast() package in R Includes methods: ets() auto.arima() Arima() arima() HoltWinters() StructTS() Produces the original series; ˆpoint forecasts; prediction intervals of specified coverage; the forecasting method used and information about the fitted model; ˆresiduals from the fitted model; ône-step forecasts from the fitted model for the period of the observed data.

28 Choosing an ARIMA model Auto.arima can be used for model choice. Manual model choice requires hypothesis testing and evaluation of results. Use minimum AIC to chose best model AIC = 2log L + 2 p + q + P + Q + k Compare AIC values to each other, absolute values carry no meaning

29 arima Demo

30 ARIMA vs. Holt-Winters Holt-Winters can be used for series that are seasonal and have a trend. (require order 2 differencing in ARIMA) Model selection can be complex in ARIMA and auto.arima selection may not be well understood. ARIMA best for stationary data series. ARIMA very powerful, but more to learn. Initial values more important in ARIMA (can have a big effect on predictions depending on model selected.) ARIMA provides confidence intervals

31 Time Series Functions in OBI 11g Very powerful, accessible capability Time dimension must be designated Query results must be exact to pull from cache Can be expensive in processing Make sure that unique keys are defined at each level ( Jan13 rather than Jan )

32 AGO function Defines a time-based offset Can nest multiple AGO statements (same level) Ago(<<Measure>>, <<Level>>, <<Number of Periods>>) Measure is a fact such as sales. Level is an optional term, default is set by the grain of the query (BY clause) or is specified in repository for level based measures. Number of periods is an integer specifying the offset value.

33 TODATE Time-based aggregation function. Calculates based on starting value to current. Can nest with AGO (same level) ToDate(<<Measure>>, <<Level>>) Measure is a fact such as sales Level is the time grain such as year or month

34 PERIODROLLING Defines a period of time contextually Performs an operation across a specified set of query grain periods PeriodRolling(<<Measure>>, <<Starting Period Offset>>, <<Ending Period Offset>>, <<[Hierarchy]>>) Measure is a fact such as sales Starting Period Offset is an integer value, use a minus sign ( -2 means 2 periods ago) Ending Period Offset defines the end of the period, use a zero for current period Hierarchy is an optional setting to specify which time hierarchy to use such as fiscal Use unbound for starting period offset to calculate total from beginning PeriodRolling uses either the query level grain of measure or the measure level for measure if it has been set in the Admin tool.

35 Oracle BI Trend Demo

36 Oracle Data Mining Oracle Data Mining is an option for the Enterprise Edition of the Oracle Database. A collection of APIs and specialized SQL functions. Includes a large number of specialized algorithms and built-in procedures. Makes use of many built-in capabilities of the Oracle Database ODM typically refers to Oracle Data Mining Copyright 2012, Vlamis Software Solutions, Inc.

37 Copyright 2011, Vlamis Software Solutions, Inc. Oracle Data Mining Algorithms Problem Algorithm Applicability Classification Regression Anomaly Detection Attribute Importance Association Rules Clustering Feature Extraction A1 A2 A3 A4 A5 A6 A7 F1 F2 F3 F4 Logistic Regression (GLM) Decision Trees Naïve Bayes Support Vector Machine Multiple Regression (GLM) Support Vector Machine One Class SVM Minimum Description Length (MDL) Apriori Hierarchical K-Means Hierarchical O-Cluster NMF Classical statistical technique Popular / Rules / transparency Embedded app Wide / narrow data / text Classical statistical technique Wide / narrow data / text Fraud Detection Attribute reduction Identify useful data Reduce data noise Market basket analysis Link analysis Product grouping Text mining Gene and protein analysis Text analysis Feature reduction

38 Copyright 2012, Vlamis Software Solutions, Inc. Classification Prediction model for non-continuous information Binary such as yes/no Limited set (low/medium/high) Involves supervised learning Prediction directed by a previously known dependent variable or target variable. Commonly includes three phases: Training Testing Scoring Results in predictive models that are applied to new data sets. In our example, we predict which prospects are likely to buy insurance.

39 Oracle Data Mining Demo

40 Oracle Test Drive Free to try out Oracle BI Go to Runs off of Amazon AWS Hands-on Labs based on Collaborate 2012 HOLs Test Drives for: Oracle BI BI Publisher Microsoft Excel against Oracle OLAP Oracle Data Mining Map Views in OBIEE Once sign up, you have private instance for 5 hours Available now

41 Thank You!