Manage the Analytical Life Cycle for Continuous Innovation

Transcription

1 Manage the Analytical Life Cycle for Continuous Innovation From Data to Decision WHITE PAPER

2 SAS White Paper Table of Contents Introduction The Complexity of Managing the Analytical Life Cycle A Factory Approach to Analytical Lifecycle Management... 5 How SAS Can Help... 5 The Predictive Analytics Factory Concept in Action Data Preparation and Exploration Model Development Model Management Model Deployment Model Monitoring How SAS Is Different The Benefits of Analytical Lifecycle Management For More Information... 11

3 Manage the Analytical Life Cycle for Continuous Innovation Introduction This scenario might look familiar: The organization has nearly 120 analytical models in production to support marketing, pricing, operational risk, credit risk, fraud and finance functions. Analysts develop these models without formalized or standard processes across business units to store, deploy and manage the portfolio of models. Some models don t have any documentation describing the model s owner, business purpose, usage guidelines or other information necessary for managing the model or explaining it to regulators. Model results are provided to management with limited controls and requirements. Because different data sets and variables are used to create the models, results are inconsistent. There is little validation or back testing. Managers make decisions based on the model results they receive, and everyone hopes for the best. This was the scene at a South African financial institution, but it might look all too familiar in your organization. In a distributed and loosely managed modeling environment, it can be difficult to answer critical questions about the models the organization relies on for strategic and operational insights. Why is it taking so long to put models into production? How many models are in production? Who created them, and how are they used? When were they last updated, and how well are they performing? Where is the supporting documentation? The organization that can t answer those questions with confidence can t be sure its models are delivering on their promise. Analytical models are at the heart of critical business decisions for finding new opportunities or managing uncertainty and risks. So dozens, or even hundreds, of predictive models should be increasingly used in real-time decision making and in operational production systems. These models should be treated as the high-value organizational assets that they are. They must be created using robust and industrialstrength processes, and managed for optimal performance throughout the life cycle. IT and analytic teams need a repeatable and efficient process and a reliable architecture for creating and deploying predictive analytic models into production systems. In short, they must operationalize analytics. That ideal is not always the reality. Here s what actually happens in many organizations: Delays. Due to processes that are largely manual and ad hoc, it takes months to get a model implemented into operational production systems. In fact, it takes so long to move models through development and testing that they are stale by the time they reach production, or never get deployed at all. Poor results. Poorly performing models remain in production, leading to inaccurate results that lead to poor business decisions. There is no central repository of models, nor are there consistent metrics that determine when a model needs to be refreshed or replaced. 1

4 SAS White Paper Confusion. Organizations find themselves in reactive mode responding in a rush to deadlines from external agencies. Each group has a different approach for handling and validating a model, which results in unique reports with differing levels of detail for review. No one is quite sure why the champion model was selected or how a particular score was calculated. Lack of transparency. There is little visibility into the stage of the model or who touches the model and when as it goes through the analytic life cycle. Conflicting assumptions surface, so unbiased reviewers must be called in to validate models as they pass through each group. Loss of important model knowledge. With inadequate documentation for models, important intellectual property is in the mind of the model owner. When that person leaves, the knowledge is lost. Collectively, these inefficiencies diminish the value of the organization s predictive models and the results they deliver. The Complexity of Managing the Analytical Life Cycle Leading organizations recognize that analytic models are essential corporate assets, and they seek to create the best models possible. However, few fully manage all the complexities of the interactive and iterative model life cycle. It s a multifaceted task, because in a large organization, there may be dozens or even hundreds of models to manage through the key stages shown here: Problem identification. Business units such as marketing, fraud or credit risk specify the need, scope, market conditions and goal related to a business question they are trying to solve which will lead to the selection of one or more appropriate types of modeling techniques. Analytical data preparation. Depending on the business question and analyses in mind, this time-consuming step involves using specialized techniques to source, clean and prepare the data for optimal results. Data exploration. Explore all data in an interactive and very visual fashion to quickly identify relevant variables, trends and relationships that were not evident before. Model development. A skilled analyst or modeler builds the model using statistical, data mining or text mining algorithm software, including the critical capability of transforming and selecting key variables. Models need to be rapidly built using sample data or a complete set of data. Model validation and documentation. Once built, the model is registered, tested or validated, approved and declared ready to be used against production data. The centralized model repository stores extensive documentation about the model (such as input and output files), scoring code and associated metadata for collaborative sharing coupled with users authentication and version control for audit/tracking purposes. 2

5 Manage the Analytical Life Cycle for Continuous Innovation Model deployment. Once approved for production, the model is applied to new data to generate predictive insights. Model monitoring and assessment. The predictive performance of the model is monitored to ensure it is up-to-date and delivering valid results. If the model degrades, it is recalibrated by changing the model coefficients or rebuilt with existing and new characteristics. When the model no longer serves a business need, it is retired. EVALUATE/ MONITOR RESULTS IDENTIFY BUSINESS PROBLEM DATA PREPARATION Firms must rerun their analysis on new data to make sure the models are still effective and to respond to changes in customer desires and competitors. Many firms analyze data weekly or even continuously. DEPLOY MODELS COMPETITIVE ADVANTAGE DATA EXPLORATION Mike Gualtieri, Forrester Research Inc. The Forrester Wave : Big Data Predictive Analytics Solutions, Q VALIDATE MODELS ANALYTICAL MODELING TRANSFORM & SELECT Figure 1: The analytical life cycle. It is easy to imagine the many ways this process can get mired or derailed. Organizations often take months, sometimes years, to move through this end-to-end process: The needed data sources might be scattered across the organization. Structured and unstructured data may need to be integrated and cleansed multiple times to support a variety of analytical requirements. It may take a long time for models to be manually translated to another language for integration with operational systems. The organization might be slow to recognize when a model needs to be updated, so it forges ahead with decisions based on specious model results. Many of the steps in the analytical life cycle are iterative in nature and might require going back to a previous step in the cycle to add and/or refresh data. 3

6 SAS White Paper The net effect is that the models that are supposed to yield solid business insight instead lead to suboptimal decisions, missed opportunities and misguided actions. The desired result is basically the flip side of the scenarios described earlier. In an effective analytical environment, data is rapidly created and accessed in the correct structure for model development. Models are rapidly built and tested, and deployed into a production environment with minimal delay. Production models quickly generate trusted output. Model performance is constantly monitored, and underperforming models are quickly replaced by more up-to-date models. In short, analytics means more than creating a powerfully predictive model; it is about managing each of these lifecycle stages holistically for a particular model and across the entire portfolio of models. This is no easy feat. Consider that analysts don t just develop one model to solve a business problem. They develop a set of competing models and use different techniques to address complex problems. They will have models at various stages of development and models tailored for different product lines and business units. An organization can quickly find itself managing hundreds or thousands of models. Furthermore, the model environment is anything but static. Models will be continually updated as they are tested and as new results and data become available. The goal is to build the best predictive models possible, using the best data available. Predictive models are high-value organizational assets, and success requires more than relying solely on the technology element. Organizations must also closely look at the people and process elements. For example, it s important to constantly upgrade business and technical analytical skills to properly identify business issues and apply analytical insights into operational processes. The analytical life cycle is iterative and interactive in nature. Staff from different backgrounds and skills are involved at various stages of the process. For instance, a business manager has to clearly identify an issue or problem that requires analyticsdriven insights, then make the appropriate business decision and monitor the returns from the decision. A business analyst conducts data exploration and visualization and works to identify key variables influencing outcomes. The IT and data management team helps to facilitate data preparation and model deployment and monitoring. A data scientist or data miner performs more complex exploratory analysis, descriptive segmentation and predictive modeling. To get the best analytic results, organizations need to put people with the right skills in place, and enable them to work collaboratively to perform their roles. With the demand rising for predictive models, a structured approach enables an enterprise view on deploying models, embedding them into businesses processes and monitoring them over time. The growing complexity and magnitude of the task of managing potentially hundreds or thousands of models in flux puts organizations at the cusp of an information revolution. The old and inefficient hand crafted approach must evolve to a more effective factory approach. 4

7 Manage the Analytical Life Cycle for Continuous Innovation A Factory Approach to Analytical Lifecycle Management A predictive analytics factory formalizes ongoing processes for analytic data preparation, model building, model management and deployment with particular attention to the process of managing models. With the demand rising for predictive models, a structured approach enables an enterprise view on the organization s portfolio of models. With a formal model management framework, analysts can register, validate, deploy, monitor and retrain analytical models in a minimal amount of time. A predictive analytics factory makes it far easier to document models and collaborate across internal and external teams. There is a mechanism for feeding model results back into the process for continuous improvement. And it becomes clear which models are still adding value and which are no longer working and need to be retired. As the foundation for a well-oiled analytical life cycle, a predictive analytics factory supports critical capabilities that are lacking today, such as the ability to: Select, retain and evolve the right analytical infrastructure for each step in the life cycle. Promote collaboration and sharing of best practices, policies and processes. Provide more analytic bandwidth with the same resources. Support easily repeatable, reproducible projects with the right level of automation. Consider data preparation and data quality as core requirements for developing the most effective models. Provide secure, intuitive access to support various user roles. By bringing cohesion to a fragmented process, a predictive analytics factory enables more strategic thinking about models and how you can treat them as corporate assets. Analytics projects and talent can evolve from the current technical focus into a stronger focus on business drivers and understanding the problem in business terms. By starting with a decision in mind, the business and analytics teams are encouraged to think about how to operationalize the model, integrate it into businesses processes, and determine when it has outlived its original purpose. Expedite the management and deployment of best models into production. Apply analytics more pervasively to a broader range of decisions. Document models and collaborate across departments and internal agencies. Monitor models to know whether they still add value or need to be improved or retired. Gain transparency for audit purposes and compliance to regulatory requirements. Streamline analytical modeling processes to generate consistent and timely results. How SAS Can Help SAS provides all components for complete lifecycle management of analytical models: Model repository. A central, secure repository stores extensive documentation about the model, its scoring code and associated metadata. Modelers can easily collaborate and reuse model code, with their activities tracked via user/group authentication, version control and audit controls. Automated workflow. A Web-based workflow console enables the model management process to become more automated and collaborative. Users can track each step of a modeling project, from problem statement through development, deployment and retirement. 5

8 SAS White Paper Governance. Accountability metrics and version control status reports track who changes what, when control is passed from one area to another, and more. A centralized model repository, lifecycle templates and version control provide visibility into analytical processes and ensure that they can be audited to comply with internal governance and external regulations. Validation. Scoring logic is validated before models are put into production, using a systematic template and process to record each test the scoring engine goes through, to ensure the logic embedded in the champion model is sound. Deployment. Choose from multiple deployment options in batch or real time, depending on IT requirements to get rapid and timely insights out of the champion models. Model retraining. Users can quickly and efficiently create possible new candidate models with up-to-date data without leaving SAS Model Manager. Performance monitoring. As the champion model reaches test, stage and production lifecycle milestones, its status and performance metrics are pushed to subject matter experts through standard reporting channels to gauge a model s fitness for the business question at hand. You can also monitor and publish challenger models. Overall lifecycle management. All stages of a model s life cycle are coordinated in holistic perspective, using prebuilt and customer-defined templates aligned with the organization s business processes. A more efficient model management process enables organizations to manage a larger number of complex analytical models with a minimal amount of time and resources. With a formal model management framework, the best models get into production faster to start serving the business sooner. The organization can generate more models, and more sophisticated models, with a large variety of analytic methods with fewer resources. Analytical models are continually monitored and refined, so they are up-todate and accurate. The modeling process becomes more transparent, so it is easy to explain analytics-based decisions to regulators and business leaders. The Predictive Analytics Factory Concept in Action The model factory approach streamlines and plays a key role in the following stages of the analytical life cycle. Data Preparation and Exploration Data preparation. SAS is used to create extract, load and transform (ELT) routines that produce analytical data marts using just the required data from the database. The data is staged in the database for fast loading, transformed into a structure fit for model building, and summarized to create derived fields. Data exploration. SAS Visual Analytics can be used to augment the data discovery process and quickly zero in on areas of opportunity or concern, uncover unexpected patterns, examine data distributions, find the prevalence of extreme values, and identify important variables (those that are highly correlated) to incorporate in the model development process. 6

9 Manage the Analytical Life Cycle for Continuous Innovation PREDICTIVE ANALYTICS FACTORY SUPPORTS THE ENTIRE DATA TO DECISION LIFE CYCLE SOURCE / OPERATIONAL SYSTEMS DATA PREPARATION & EXPLORATION MODEL DEVELOPMENT MODEL DEPLOYMENT & MONITORING MODEL MANAGEMENT Model Development Analysts can build models using a variety of SAS tools that include a rich set of algorithms to analyze structured and unstructured data, such as: Supporting Technologies for SAS for Predictive Analytics Factory SAS Data Management Advanced SAS Enterprise Miner TM, which streamlines data mining to create accurate predictive and descriptive models based on large volumes of enterprisewide data. SAS Rapid Predictive Modeler, which auto-generates models through a workflow of behind-the-scenes data preparation and data mining tasks. SAS Text Miner, which provides a rich suite of tools for discovering and extracting knowledge from text sources. SAS High-Performance Analytics Server, which supports the power of inmemory processing to enable models to run very quickly against extremely large data sources. SAS Visual Analytics SAS Enterprise Miner SAS Text Miner SAS High-Performance Analytics Server SAS Model Manager SAS Scoring Accelerator Model Management When model development is complete, analysts register a model package that contains the model and all of its associated output and documentation. This package makes it easy to ensure that the right steps have been taken, and a suitable and robust model is released into the production environment. This model package enables organizations to standardize the process of creating, managing, deploying, and monitoring analytical models. 7

10 SAS White Paper Model Deployment Once a model has been reviewed, signed off and declared ready for production, it has champion status in SAS Model Manager. With the click of a button, the model is converted into a vendor-defined function (VDF) or registered as a DS2 program to execute inside the database. Scoring code, including transformations, is generated in SAS, Java, C, and PMML languages for deploying in SAS and other environments. Model execution. Model execution is centrally controlled using SAS Data Integration Studio jobs that control which data tables are used as scoring marts, the model used to score the mart, and the creation of a file that contains the scores. The scoring mart is created using in-database processing, and it resides in the database. The model execution job is scheduled to run at a specific time interval or initiated by a trigger. The model is executed from within SAS Data Integration Studio and runs directly in the database. Model Monitoring Once a model is in a production environment, and is being executed at regular intervals, the champion model is centrally monitored through a variety of reports, because its predictive performance will degrade over time. When performance degradation hits a certain threshold, the model can be replaced with a new model that has been recalibrated or rebuilt. Comprehensive analytical lifecycle management capabilities from data to decision make it possible for organizations to take advantage of sophisticated analytical techniques, a large number of analytical models, and a virtually unlimited number of variables and data volumes. Model recalibration or rebuild. When executing a model, the SAS Data Integration Studio job accesses the latest version of the analytical data mart. It can either recalibrate or rebuild a model, depending on the process. A model package is created and automatically registered in SAS Model Manager. Users can view the model in SAS Model Manager, and decide whether the model should replace the existing champion model. 8

11 Manage the Analytical Life Cycle for Continuous Innovation How SAS Is Different A flexible infrastructure supports multiple analytical disciplines (such as data mining, forecasting, text analytics and optimization) and analytical scenarios for greater agility. The ability to track model lineage from data source to analytic result provides essential governance, critical in situations that are regulated or have strict reporting requirements. Only SAS provides the ability to effectively manage models in the context of big data and high-performance analytics environments. SAS can manage large numbers of complex models that use advanced analytical techniques, virtually unlimited variables and extremely large data volumes. Easy-to-use model management and monitoring tools make it clear which models are still performing and are adding value, and which should be retired. The tightly integrated SAS Business Analytics technology is at the heart of the predictive analytics factory. It provides reliability, relevance and faster time to insights. With Web-based workflow capabilities, users can easily define custom processes, manage them through to completion, foster collaboration with notifications, and establish enterprise standards. Intuitive, graphical performance monitoring dashboards help track model performance across multiple projects, so teams can focus on projects that need the most immediate attention and avoid model decay. Interoperability with third-party modeling tools enables organizations to import, manage and monitor modeling assets created by SAS and other tools (e.g., PMML models, R) all together in a central repository. In-database scoring functions can be achieved with widely used databases such as Teradata, Aster Data, EMC Greenplum, IBM Netezza, IBM DB2 and Oracle. SAS also supports high-performance scoring and model performance monitoring on a Teradata or EMC Greenplum database appliance. Integration of model deployment processes with other operational processes enables the organization to easily manage, deploy and fine-tune models on demand, on a set schedule, or when triggered by external business events. Looking beyond the software, SAS also brings extensive technical and business expertise for pre- and post-sales support, to help organizations expedite time to value and improve return on investment. Predictive models use your data to tell you about the likelihood of some future event. Since nobody knows exactly what s going to happen in the future, managing predictive models is about managing the uncertainty of future outcomes across the organization. 9

12 SAS White Paper The Benefits of Analytical Lifecycle Management With a predictive analytics factory approach to analytical lifecycle management, the after scenario looks quite different from the usual modus operandi and creates a serious competitive advantage. A major financial institution in the UK determined that its cycle time from model initiation to model deployment wouldn t meet 21st-century expectations. The process was manual, error-prone, and resource-intensive and had little or no monitoring to identify model degradation. Working with SAS and data warehousing vendor Teradata, the organization built a flexible predictive analytics factory platform by integrating data management, model development and model deployment using indatabase technology. The platform harnesses the scalability of the Teradata environment for model scoring and uses the power of SAS Analytics to build models. With the new platform, more than 55 million records can be scored within Teradata many times during the day something that could never have been achieved with the former process. The time required to promote a model to a production environment dropped from three months to days. Data preparation time was trimmed by 40 percent. Analyst productivity jumped 50 percent. As more and more organizations are discovering, a predictive analytics factory approach delivers a host of benefits, such as: Efficient development. A predictive analytics factory uses integrated SAS components to reduce the modeling life cycle by eliminating redundant steps, and supports cohesion across the information management chain from data to decision management. Faster deployment. Operationalize information and analytical processes with minimum infrastructure and cost. For example, the conversion of scoring code into logic that is placed directly in the enterprise data warehouse happens automatically. This eliminates the time-consuming and error-prone manual process of translating the model. Faster scoring processes. Because the model is scored directly in the database, the model execution job takes advantage of the scalability and processing speed offered in the database. Jobs that used to take hours and days can now be completed in minutes and seconds. Active monitoring and management of models. The predictive analytics factory allows standard monitoring reports to be created and reviewed, so models can be kept up to date, delivering accurate results. Reduced risk. Consistent processes and technologies for model development and deployment reduce the risks involved in the modeling processes while supporting collaboration and governance among key business and IT stakeholders. 10

13 Manage the Analytical Life Cycle for Continuous Innovation Predictive models use your data to tell you about the likelihood of some future event. Since nobody knows exactly what s going to happen in the future, managing predictive models is about managing the uncertainty of future outcomes across the organization. That s an important enough purpose to deserve rigorous process controls a predictive analytics factory approach to analytical lifecycle management. For More Information Learn more about model management and monitoring: sas.com/modelmanager 11

14 About SAS SAS is the leader in business analytics software and services, and the largest independent vendor in the business intelligence market. Through innovative solutions, SAS helps customers at more than 60,000 sites improve performance and deliver value by making better decisions faster. Since 1976 SAS has been giving customers around the world THE POWER TO KNOW. For more information on SAS Business Analytics software and services, visit sas.com. SAS Institute Inc. World Headquarters To contact your local SAS office, please visit: sas.com/offices SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. indicates USA registration. Other brand and product names are trademarks of their respective companies. Copyright 2013, SAS Institute Inc. All rights reserved _S85014_0313