Turning Data into Actionable Insights: Predictive Analytics with MATLAB WHITE PAPER
Introduction: Knowing Your Risk Financial professionals constantly make decisions that impact future outcomes in the face of uncertainty and risk. Predictive analytics is about quantifying this uncertainty so that you can make calculated decisions with a certain degree of confidence. Predictive modeling techniques, based on rigorous statistical methodologies, are often used in financial analysis tasks ranging from time-series analysis and forecasting, risk classification, and default probability estimation to generalized data mining. This paper provides an overview of the challenges associated with predictive modeling, outlines a predictive modeling workflow, and shows how you can use MATLAB to address these challenges. Decision-Making Challenges in the Era of Big Data In the Harvard Business Review article You May Not Need Big Data After All, the authors note that corporations with a culture of evidence-based decision making have seen improvements in their business performance. Decision making in the modern organization is increasingly complex, requiring the near-continual analysis of ever-increasing amounts of data. As the volume, variety, and velocity of data grow, the analytical models we design to work on this data need to become nimbler. Additionally, we must be able to react quickly. Further, model designing is inherently a collaborative activity, with inputs from multiple individuals and teams, and the resulting models are often shared across groups. Finally, there is an increasing requirement for transparency within the finance-related functions of an organization. Within a financial institution, predictive analytics touches multiple stakeholders not just financial engineers and quantitative researchers, but also management, trading desks, and other groups such as controllers and internal auditors. Externally, it impacts regulators, clients, and business partners. Predictive Analytics Overview Predictive Modeling Predictive analytics starts with a business goal that involves decision making, possibly with the intent of making the process more systematic and less error-prone. Examples of such decision-making workflows include cash flow analysis, fraud detection for credit card companies, customer churn analysis, trading strategy selection, and adaptive portfolio allocation. Analysts make use of predictive modeling to understand the data and create mathematical models that generate actionable outcomes. These models are in turn passed over to the field associates to aid them in their day-to-day decision making. Uses of Predictive Models Predictive models are widely used in the finance function and industry. Some of these uses are: Forecasting prices or returns Analyzing the impact of market moves on instrument prices (sensitivity analysis) Stress testing a portfolio to predict the effect of crises and shocks Detecting anomalous and fraudulent credit card transactions Segmenting and targeting potential customers for marketing financial products WHITE PAPER 2
Case: MathWorks Tools Used to Predict Financial Crises in Emerging Markets Challenge Develop an economic model that would help predict and avert financial crises in emerging economies Solution Use MathWorks tools to develop a model that applies both linear methods and neural networks to the analysis of trends in currency demand over a selected period Because MATLAB is both powerful and easy to use, I felt confident that the Bank of Indonesia would be able to implement the MATLAB programs and use them as an early warning system for financial distress. Dr. Paul McNelis Georgetown University Results Models with greater predictive capacity A program for averting financial disaster New research tools for the next generation Predictive Modeling Workflow Predictive modeling is an iterative process (Figure 1). As part of this process, analysts may need to access data that comes from multiple sources and may be unstructured or stored in multiple formats. This data needs to be filtered and transformed in order to gain insights from it. A model is developed based on these insights and the analysts mathematical assumptions. During the development process, analysts may collaborate and share insights with others in the form of a report. These models are refined and revised repeatedly until their accuracy is deemed sufficient. Once models are developed, they need to be shared with decision makers to deliver their real value. This sharing process typically involves integrating the predictive models into the organization s production environment. It is important to choose a set of tools that enables rapid iteration, repeatability, maintainability, and reuse through automation to maximize the ROI for such workflows. WHITE PAPER 3
Access Research and Quantify Share Files.xls A B C Data Analysis and Visualization Reporting.doc.html PDF Databases Datafeeds BZ NG CL 71.92 5.332 81. Application Development Financial Modeling S=31; K=30 C=blsprice P=C S+K*ex Option 1 Option 2 NEXT Automate Applications A B C D A B C D A B C D Production.dll Automate Figure 1: The predictive analytics workflow. Developing Predictive Models Developing predictive models is as much art as it is science. Analysts need to understand the structure of the data first through multiple stages of exploratory analysis. This often leads to insights that can help shorten the list of mathematical approaches to try. Analysts also need to select and prepare predictors (perhaps a subset of the available data) that become inputs to the mathematical approaches. The set of predictors along with a particular mathematical approach is called a predictive model. Then, a trial-and-error process iterates through multiple models. Comparing different models to choose the best approach can be challenging. Every model needs to be trained by providing samples of known predictors and responses. A measure for accuracy of each model is also computed to be able to compare the different models. Model training can be a highly data- and computation-intensive process. One consideration is speeding up the process by utilizing all the available resources, including parallel computing on multicore machines, processing on GPUs, and farming out tasks to computer clusters or to the cloud. Depending on the computational needs, one or a combination of these approaches may be suitable. Integrating Models into the Organization Once a predictive model is finalized, new data is fed to produce predicted responses. For the predicted responses to be useful to the business, however, the models must be integrated into the operational environment, so that they can be accessed by end users and decision makers. This may entail integrating the predictive model into a desktop application or web/server based solution. WHITE PAPER 4
The Business Integration Gap Organizations often face challenges bridging the gap between the predictive modeling process and the subsequent integration process, which requires technical infrastructure for collaboration and coordination among different business units. This is typically a time-consuming process if not planned properly, and is often overlooked in discussions of predictive analytics. Organizational Challenges In financial institutions, predictive models are primarily developed by financial engineers and quantitative researchers (Figure 2). Other end users, such as traders and management, use the output of these operationalized predictive models through existing reporting systems or business applications. IT groups typically manage the production data pipeline that feeds into these predictive models, and integrate the deployed models into business applications. Financial engineers may be unfamiliar with the technical challenges IT engineers face, and IT engineers may not be familiar with the statistical requirements that the system must meet in order to ensure accurate predictive performance. End users and management are less technical but have their own sets of operational challenges. In addition, depending on the application, the model may also need to meet the needs of external audiences such as regulators, internal and external auditors, clients, and business partners. Successfully embedding predictive models into the organizational decision-making framework requires the coordination of all these stakeholders. Traders Regulators Financial Engineer Quant Group Managers Clients Other Groups Partners Figure 2. Predictive analytics stakeholders in a financial services organization. WHITE PAPER 5
The Challenge of Matching Models with Platforms Financial engineers develop predictive models using specialized statistical computation tools. IT groups manage business applications based on the existing IT infrastructure. This implies that predictive models developed by financial engineers must be often re-implemented in a form that is compatible with the target platform. Challenges of Functional Validation and Testing If the models are to be re-implemented on different platforms, a question of ensuring the functional equivalence of the new implementation arises. The need for testing also arises when integrating the models with production data pipelines for input and existing business applications for output. Bridging the Modeling and Integration Gap with MATLAB MATLAB bridges the gap between modeling and business integration in a number of ways. First, it provides an easy-to-use, efficient environment for model development for financial engineers. Next, it enables predictive models developed in MATLAB to be integrated with a variety of enterprise IT platforms, obviating the need for model re-implementation by IT engineering. Bridging the integration gap reduces the time needed to take a model from concept to implementation and improves overall efficiency of the business organization. MATLAB provides: An easy-to-use and efficient interactive environment for predictive modeling Visual tools for exploratory data analysis Ability to easily to evaluate and choose appropriate statistical and numerical algorithms Simple code parallelization to maximize resource usage for computation-intensive tasks Apps to help you get started (e.g., apps in Neural Network Toolbox, Curve Fitting app) for point-and-click workflows Multiple sophisticated algorithms Linear and non-linear regression Time-series analysis: ARIMAX/GARCH Machine learning techniques such as classification, clustering, and ensemble learning Integration with multiple data sources Files: text, CSVs, Microsoft Excel spreadsheets, flat files, and unstructured data files SQL databases and NoSQL data stores Hadoop clusters Deployment and integration options (Figure 3) Multiple formats for integration with production systems: standalone.exe files, Excel add-ins, C code,.net assemblies, Java classes, and COM/ActiveX DLLs WHITE PAPER 6
Clear separation of tasks between modeling process and deployment process Unified process for desktop and server-based deployment Multiple deployments from a single source model that fit the needs of various end users, easing the burden on IT groups Deployment Highlights Database Servers.exe Desktop Applications HADOOP COM Excel Spreadsheets Application Servers.NET C/C++ Client Front End Applications Web Applications Java CTF Batch/Cron Jobs Figure 3. MATLAB deployment options for predictive models. WHITE PAPER 7
Case: Horizon Wind Energy Develops Revenue Forecasting and Risk Analysis Tools for Wind Farms Challenge Develop revenue forecasts and quantify risk for wind farms across multiple geographic locations Solution Use MATLAB and MATLAB Compiler to develop and deploy an automated production system that analyzes historical, current, and forward-looking price and wind-level data Results The tools that we developed with MATLAB are much more reliable, scalable, and maintainable than our spreadsheet-based approach. We can sleep at night because we know the tools will work, we can add new capabilities and data inputs, and we can update the production system without getting IT involved. Manuel Arancibia Horizon Wind Energy Core process automated Standalone program seamlessly integrated with enterprise IT infrastructure Risk management improved, saving millions of dollars Training MATLAB is intuitive and easy to learn. However, you may be able to exploit its full potential and shrink your learning curve by taking advantage of the professional training offered by MathWorks. Training formats include: Public training (available worldwide) Onsite training with standard or customized courses Web-based training with live, interactive instructor-led courses Self-paced interactive online training MathWorks Training Services has more than 30 course offerings. Specialized courses in financial analysis, parallel computing, code generation, platform deployment, and other areas are available, along with introductory and intermediate training on MATLAB. Consulting In addition to training, you can also tap into a global team of experts that supports adoption and integration of MATLAB based solutions within your organization based on your specific needs and requirements. WHITE PAPER 8
Case: RWE Develops and Deploys an Automated System for Natural Gas and Power Trading and Risk Management Challenge Automated business processes for quoting gas contracts and hedging against price fluctuations Solution Engage MathWorks Consulting to develop and deploy to a production environment an automated pricing and risk management system that fits within the company s existing IT infrastructure Results MathWorks consultants were well-qualified, professional, and fast. They understood not only the technical issues but also the business goals, which is essential when working on a core business system. We got more than we expected from MathWorks Consulting. Dr. Norbert Tönder RWE Models created in minutes, not weeks 100% accurate results delivered Technical expertise applied to core business goals Conclusion Predictive analytics enables informed decision making in the face of uncertainty and risk through datadriven insights and predictions. To take full advantage of the predictive power such analytics provide, organizations must embed and fully integrate predictive models into their operational systems. MATLAB provides an interactive environment and a rich feature set to enable research and development of predictive models. Furthermore, through flexible deployment options, production-ready models can be integrated much more quickly into business systems while financial engineers retain control of the model itself, reducing the overall time required and resources spent to develop and manage such solutions. 2014 The MathWorks, Inc. MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See mathworks.com/trademarks for a list of additional trademarks. Other product or brand names may be trademarks or registered trademarks of their respective holders. WHITE PAPER 9 92212v00 07/14