Transcription

1 Page 1 of 5 The Modeling Agency Newsletter Feature Article Predictive Analytics & Data Mining Why Most Projects Fail & What Really Works by Eric A. King President The Modeling Agency direct training fax send a message newsletter Receive articles, training schedule updates and industry announcements: The Right Approach Yields High Returns Predictive Analytics and Data Mining (PADM) has been a slowly but steadily growing practice within the broader field of Business Intelligence (BI). The machine learning technologies that drive PADM were developed many decades ago. But only the last two decades have presented adequate desktop computing power and software automation to make PADM a practical and highly valuable BI function in every medium and large business. It won t be long at all until every organization with more than $50M in annual revenue makes PADM a standard part of its BI platform. And the application areas are limitless not just across industries, but within nearly every practice area of an organization that collects transactional or behavioral data. It completes the operational view from historical queries and situational reports, to a reliable prospective view. This vision allows organizations to anticipate behaviors and proactively apply treatments that substantially expand competitive advantage, profits and savings. Imagine the view when leadership ultimately realizes how much more effectively they can drive when looking through the windshield as opposed to the rear-view mirror. Just how impactful can PADM be? My company recently led a large retailer in a predictive modeling project which reduced their catalogue mailing costs by 40% while increasing sales by 7%. A $50,000 investment in services saved their mailing operations $8M and increased sales by $7M. This was done by anticipating which customers would be most receptive and influenced by specific messaging based on available demographic and psychographic data, and targeting those who would perform most like their best customers. While the magnitude of the impact is larger than typical implementations mainly due to the retailer s high volume, the overall success of the solution was due to a well designed project -- not because of a sophisticated technical approach. And with a sound PADM project design, most implementations should realize and insist upon a 1000% ROI a ten-fold return on their overall PADM investment. It s that powerful, as evidenced by an ever-increasing number of case reports. There are two significant opposing forces influencing the pace of broad PADM adoption. Fortunately the forces for progress prevail and are gaining momentum. PADM will continue to grow not only because the majority of organizations are steadily proceeding down the BI development chain from data acquisition to storage, structure, quality, retrieval, exploration, visualization, dashboarding, and retrospective analysis but also due to the increasing flow of powerful case studies and industry reports that reveal significant rewards enjoyed by those who properly approach the practice. Yet, there are a number of limitations that have prevented PADM from taking off more quickly. PADM can be perceived as cryptic, intangible, complex and somewhat intimidating. Most do not know where to begin. Many have heard about or directly experienced PADM projects that failed and improperly blamed the overall practice. Leadership often considers PADM to be a risk-laden research activity. And those who forge ahead without first learning how to methodically approach what is essentially a discovery process will ultimately encounter any of a host of common yet fully avoidable pitfalls that plague PADM implementations.

2 Page 2 of 5 Why Most PADM Projects Fail Contrary to common perceptions, PADM projects rarely ever fail due to tactical or technical complexity issues. While the underlying science, methodologies and mathematics are indeed quite complex, modern commercial and open-source software does a commendable job of automating the model-building process. It s now quite easy for general business practitioners who do not have PhD s in math or advanced training in statistics to quickly create very good predictive models. Yet, most business practitioners approach PADM with a similar doomed approach. They start by purchasing data mining software, introducing available data, and expecting the model s results to make sense, be impactful, and be adopted. But they are not to blame. Investing in software and directly applying it to a business function works for the majority of IT and BI projects. Even the majority of self-proclaimed data miners fall victim to the same common pitfalls, as evidenced by implementation surveys and project outcomes. This survey of more than 750 data miners reveals a fundamental strategic flaw in their PADM project design. When asked How do you evaluate project success? more than half the respondents selected a technical or artificial metric of model performance. From Rexer Analytics 3rd Annual Data Miner Survey by Karl Rexer, PhD, Heather N. Allen, PhD and Paul Gearan Essentially, these practitioners chose to focus on building technically accurate models that answer the wrong questions, are not interpretable, not adapted to the operational environment, not reasonably implementable, not understood or appreciated by leadership, and don t meet organizational metrics that define success or failure. In a business environment, practitioners are not rewarded for technical accuracy, but for business impact. It s no wonder then why a corresponding number of projects (again over 50% when adding the first three rows in the illustration below) did not reach completion, could not calculate ROI, or didn t achieve positive ROI. From KDnuggets.com What Really Works in Predictive Analytics The vast majority of BI professionals do not realize that there are exceptional industry standard processes for PADM available in the public domain. Two of the most popular are SEMMA ( Sample, Explore, Modify, Model, Assess by SAS Institute) and CRISP-DM ( CRoss-Industry Standard Process for Data Mining ). CRISP-DM is vendor-neutral and more broadly adopted. CRISP-DM evenly applies emphasis across project-level strategic implementation and tactical model development with three phases each. The first two phases are purely strategic and are the most critical to overall project success in operational settings. The first two phases entitled Business Understanding and Data Understanding concentrate on conducting a comprehensive assessment of the overall environment, situation, team members, resources and objectives.

3 Page 3 of 5 CRoss-Industry Standard Process for Data Mining (CRISP-DM) Any organization interested in building more than an independent or pilot model should not overlook the importance of implementing every stage within these first two critical phases. The best cookbook available for the practice of PADM is available to everyone at no cost. My organization has formalized these initial two phases with corresponding reporting in a service package entitled Data Mining Project Assessment (DMPA). The DMPA not only develops a solid project definition and resulting strategic implementation plan, but it reports on situational findings including whether a client is even at the starting line and the steps to get there. The resulting blueprints include an overarching plan for building a process and sustainable practice, as well as specific next steps to implement the initial phase with fixed cost, timeframe and deliverables. Allow me to distill the experiences and recommendations imparted so far into a low-risk / highimpact approach to PADM. Those seeking the most effective and efficient way to approach and succeed in this high risk / high reward practice will benefit greatly from this procedural outline: 1. Start With Training: Whether you are a practitioner or leadership, training is essential for appreciating the unique nuances of this particular BI practice. Even those planning to outsource the effort will interact more effectively with their consulting team, after understanding PADM s risks, rewards, capabilities, limitations and standard process. There are two PADM courses on Global Knowledge s curriculum. One provides the tactical and methodological approach (CRISP-DM Phases III, IV & V) and the other is the strategic orientation (CRISP-DM Phases I, II & VI). The courses may be attended independently, or as a series providing a comprehensive view of PADM. Learn more about these courses by searching for Predictive Analytics on GlobalKnowledge.com 2. Conduct a Data Mining Project Assessment: While most organizations already have the capability to ultimately own, operate and maintain a resulting PADM practice internally with existing staff, it is recommended to have a seasoned data mining expert play the role of assessor and architect of the project design. The resulting findings and recommendations will set your organization on the right path, encourage early results reporting, and separate you from the vast majority who proceed down an elusive doomed path. 3. Implement Internally with External Mentorship: There is no reason that your existing business practitioners cannot ultimately build, operate and maintain a PADM practice. Likewise, you should not proceed without the guidance and oversight of a seasoned PADM project leader and architect. This mentor should work closely with your organization to assess, design and roadmap the project, then work collaboratively with your team to delegate and oversee the tactical implementation. When your team is responsible for implementing the process, your organization will build and retain its own knowledge capital while reinforcing the skills to maintain the iterative cycle through lifetime model management. Likewise, your external investment will be minimal, as the external consultant s time investment will be limited to assessment, design and mentorship. The majority of the overall labor can be undertaken internally with oversight. The benefits of this arrangement are numerous and compounding. You Can Do It, and Own It, and Reap Residual Rewards If you are in a medium or large business that has not made PADM a highly profitable part of your BI process, then the valuable prospective insights left hidden within your glut of data are no different than large denominations left hanging from low branches. It won t be long until PADM is a standard function within your BI practice. Why delay the rewards? Like any other BI practice, PADM is not a one-time implementation, but an ongoing process. As such, it would be highly inefficient to perpetually outsource the function. Not only is the talent involved far more expensive than nearly any other IT role, but once the process is designed and

4 Page 4 of 5 established, it s straightforward for general business practitioners to maintain the end-to-end process. With some training and initial mentorship to stand up your practice, you will be on your way to generating substantial ROI and case reports with your own internal predictive modeling factory. ABOUT THE AUTHOR ERIC A. KING After graduating from the University of Pittsburgh with a bachelor's degree in computer science in 1990, Eric joined NeuralWare, Incorporated, a neural network tools company, as a senior account executive. In 1994, he moved to American Heuristics Corporation (AHC): an advanced software technology consulting company in West Virginia specializing in artificial intelligence applications. At AHC, he performed as the director of business development for the commercial services division and started a training operation, The Gordian Institute. With the valued support of AHC, contractors, customers and family, in January of 2000 Eric founded The Modeling Agency (TMA) in the spirit of establishing long-term professional relationships. TMA provides guidance and results for those who are data-rich, yet information-poor. TMA produces a popular live webinar, "Data Mining: Failure to Launch."

5 Page 5 of 5 address One Oxford Centre 301 Grant St, Ste 4300 Pittsburgh, PA USA phone: fax: training: info@the-modeling-agency.com Copyright The Modeling Agency. All rights reserved. Privacy Statement.