Analytics Outsourcing: The Hertz Experience 4 Hugh J. Watson, Barbara H. Wixom, and Thomas C. Pagano

Transcription

1 EXCLUSIVELY FOR TDWI PREMIUM MEMBERS volume 18 number 4 The leading publication for business intelligence and data warehousing professionals Analytics Outsourcing: The Hertz Experience 4 Hugh J. Watson, Barbara H. Wixom, and Thomas C. Pagano Three Best Practices for IT and Business Users 8 in Big Data Projects Fern Halper Mainframes: The (Other) Elephant in the 10 Big Data Room Jorge A. Lopez Filling the Demand for Data Scientists: 13 A Five-Point Plan John Santaferraro Marketing IT to BI Users In-House: 19 The Importance of Small Talk Max T. Russell BI Training: Closing the Business Analytics 22 Gap at UT Austin Linda L. Briggs Overcoming Data Challenges with Virtualization 25 Nilesh Bhatti Big Data Management Platforms: Architecting 32 Heterogeneous Solutions Ravi Chandran BI Experts Perspective: Aligning Business 39 Strategy with BI Capabilities Alicia Acebo, Jim Gallo, Jane Griffin, and Brian Valeyko Implementing an Enterprise Data 46 Quality Strategy Nancy Couture Data Variety: The Spice of Insight 53 David Stodder

2 TDWI TDWI ONSITE ONSITE EDUCATION EDUCATION BI Training Solutions: As Close as Your Conference Room TDWI Onsite Education brings our vendor-neutral BI and DW training to companies TDWI worldwide, Onsite tailored Education to meet brings the specific our vendor-neutral needs of your BI organization. and DW training From to fundamental companies worldwide, courses to advanced tailored to techniques, meet the specific plus prep needs courses of your and organization. exams for the From Certified fundamental Business courses Intelligence to advanced Professional techniques, (CBIP) designation we plus prep courses can and bring exams the for training the Certified you need Business directly Intelligence to your team Professional in your own (CBIP) conference designation we room. can bring the training you need directly to your team in your own conference room. YOUR TEAM, OUR INSTRUCTORS, YOUR LOCATION. YOUR TEAM, OUR INSTRUCTORS, YOUR LOCATION. Contact Yvonne Baho at Contact or Yvonne Baho for at more information. or for more information. tdwi.org/onsite tdwi.org/onsite

3 volume 18 number 4 3 From the Editor 4 Analytics Outsourcing: The Hertz Experience Hugh J. Watson, Barbara H. Wixom, and Thomas C. Pagano 8 Three Best Practices for IT and Business Users in Big Data Projects Fern Halper 10 Mainframes: The (Other) Elephant in the Big Data Room Jorge A. Lopez 13 Filling the Demand for Data Scientists: A Five-Point Plan John Santaferraro 19 Marketing IT to BI Users In-House: The Importance of Small Talk Max T. Russell 22 BI Training: Closing the Business Analytics Gap at UT Austin Linda L. Briggs 25 Overcoming Data Challenges with Virtualization Nilesh Bhatti 32 Big Data Management Platforms: Architecting Heterogeneous Solutions Ravi Chandran 39 BI Experts Perspective: Aligning Business Strategy with BI Capabilities Alicia Acebo, Jim Gallo, Jane Griffin, and Brian Valeyko 46 Implementing an Enterprise Data Quality Strategy Nancy Couture 52 Instructions for Authors 53 Data Variety: The Spice of Insight David Stodder 56 BI StatShots BUSINESS INTELLIGENCE Journal vol. 18, No. 4 1

4 volume 18 number 4 tdwi.org EDITORIAL BOARD Editorial Director James E. Powell, TDWI Managing Editor Jennifer Agee, TDWI President Director, Online Products & Marketing Senior Graphic Designer rich Zbylut Melissa Parrish Bill Grimmer Senior Editor Hugh J. Watson, TDWI Fellow, University of Georgia Director, TDWI Research Philip Russom, TDWI Director, TDWI Research David Stodder, TDWI Director, TDWI Research Fern Halper, TDWI Associate Editors Barry Devlin, 9sight Consulting Mark Frolick, Xavier University Troy Hiltbrand, Idaho National Laboratory Claudia Imhoff, TDWI Fellow, Intelligent Solutions, Inc. Barbara Haley Wixom, TDWI Fellow, University of Virginia Advertising Sales: Scott Geissler, sgeissler@tdwi.org, List Rentals: 1105 Media, Inc., offers numerous , postal, and telemarketing lists targeting business intelligence and data warehousing professionals, as well as other high-tech markets. For more information, please contact our list manager, Merit Direct, at or Reprints: For single article reprints (in minimum quantities of ), e-prints, plaques, and posters, contact: PARS International, phone: , reprints@parsintl.com, Copyright 2013 by 1105 Media, Inc. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. Mail requests to Permissions Editor, c/o Business Intelligence Journal, 1201 Monster Road SW, Suite 250, Renton, WA The information in this journal has not undergone any formal testing by 1105 Media, Inc., and is distributed without any warranty expressed or implied. Implementation or use of any information contained herein is the reader s sole responsibility. While the information has been reviewed for accuracy, there is no guarantee that the same or similar results may be achieved in all environments. Technical inaccuracies may result from printing errors, new developments in the industry, and/or changes or enhancements to either hardware or software components. Printed in the USA. [ISSN ] Product and company names mentioned herein may be trademarks and/or registered trademarks of their respective companies. President & Chief Executive Officer Senior Vice President & Chief Financial Officer Executive Vice President Vice President, Finance & Administration Vice President, Information Technology & Application Development Vice President, Event Operations Chairman of the Board neal Vitale Richard Vitale Michael J. Valenti Christopher M. Coates Erik A. Lindgren David F. Myers Jeffrey S. Klein Reaching the Staff Staff may be reached via , telephone, fax, or mail. To any member of the staff, please use the following form: FirstinitialLastname@1105media.com Renton office (weekdays, 8:30 a.m. 5:00 p.m. PT) Telephone ; Fax Monster Road SW, Suite 250, Renton, WA Corporate office (weekdays, 8:30 a.m. 5:30 p.m. PT) Telephone ; Fax Oakdale Avenue, Suite 101, Chatsworth, CA Business Intelligence Journal (article submission inquiries) Jennifer Agee journal@tdwi.org tdwi.org/journalsubmissions TDWI Premium Membership (inquiries & changes of address) membership@tdwi.org tdwi.org/premiummembership Fax: BUSINESS INTELLIGENCE Journal vol. 18, No. 4

5 From the Editor Snow White s Seven Dwarfs happily whistled while they worked. These days, we whistle in amazement that all their work could be done by such a small team and wonder where we can find the skilled talent we need to maintain or expand our BI initiatives. Senior editor Hugh J. Watson, Barbara Wixom, and Thomas Pagano look at how Hertz used outsourcing to solve their resource problems. The authors describe what s driving outsourcing of BI staff and what projects the auto rental firm chose to outsource. John Santaferraro presents a five-point plan for filling your open data scientist positions. He touches on incentive programs, technology infrastructure, and the value of an enterprisewide culture of analytics. Linda Briggs describes a new program at the University of Texas at Austin that targets the business analytics gap plaguing many organizations. Perhaps your organization doesn t need more resources but rather needs to use its existing resources more effectively. Director of TDWI Research for advanced analytics Fern Halper looks at three best practices IT and business users can follow to work better together and achieve success in big data projects. Max T. Russell explains how IT professionals can build a stronger relationship with their user base by learning the art of small talk a simple way to build trust and respect and help the IT team play a bigger, more important role in an organization s BI efforts. Having the right tools and technology may also reduce the stress on resources. Nilesh Bhatti discusses the benefits and challenges of implementing data virtualization to help manage increasing data volumes. Organizations must be sure the data they manage is accurate, complete, and up to date. Nancy Couture looks at how best to implement an enterprise data quality strategy. TDWI s David Stodder looks at why organizations should harness the wide variety of data they collect. Getting work done isn t just a matter of human resources. Jorge Lopez explains how you get more done by leveraging mainframe data with Hadoop. Although they seem like an unlikely duo, Lopez offers some practical Hadoop use cases for mainframe users. Organizations can also run smoother when the business strategy aligns with BI s capabilities, which is the subject of our BI Experts Perspective column. We provide advice from Alicia Acebo, Jim Gallo, Jane Griffin, and Brian Valeyko. Are you working smarter? Do you whistle while you work? Let us know. We welcome your feedback and comments; please send them to jpowell@tdwi.org. BUSINESS INTELLIGENCE Journal vol. 18, No. 4 3

6 Analytics Outsourcing Analytics Outsourcing: The Hertz Experience Hugh J. Watson, Barbara H. Wixom, and Thomas C. Pagano Analytics is becoming increasingly important for many organizations. To address their need for advanced analytics, some firms are using outside organizations to help provide analytics expertise and capabilities. Hugh J. Watson is a Professor of MIS and a C. Herman and Mary Virginia Terry Chair of Business Administration in the Terry College of Business at the University of Georgia. hwatson@terry.uga.edu Barbara H. Wixom is a principal research scientist in the MIT Sloan Center for Information Systems Research at MIT. bwixom@mit.edu Thomas C. Pagano is director, business information and data warehouse systems, for The Hertz Corporation. tpagano@hertz.com We use the term analytics outsourcing in this article to refer to the use of any external organization to provide parts of or an entire analytics solution. Analytics outsourcing comes in a variety of forms. For example, your enterprise may hire a consulting firm to help implement a dashboard or scorecard system. You may choose a firm that provides fraud detection through software-as-aservice. You might select a third-party supplier of data that supports your company s CRM application. Outsourcing has been around for many years and is now a commonly accepted business practice. It started with less complex business processes such as call centers and has moved on to more knowledge-intensive processes such as analytics. This market is experiencing significant growth. Enterprises will spend an estimated $46.9 billion this year on analytics outsourcing, and IDC projects that spending will grow to $70.8 billion by 2016 (Zaidi and Dialani, 2013). We are in the initial stages of a case study with Hertz, a leader in the rental car industry, to understand their use of analytics. As you would expect, Hertz has long used analytics for pricing rental cars, forecasting demand, designing marketing campaigns, and so on. One current initiative is the use of analytics to better understand customers, communicate with them better through real-time analytics, and increase customer loyalty. 4 BUSINESS INTELLIGENCE Journal vol. 18, No. 4

7 Analytics Outsourcing In our interviews with key people at Hertz, we found many instances where the company outsourced its analytics, so we asked them to help us understand and describe the company s use of analytics outsourcing. First, however, let s consider some of the reasons why firms are turning to analytics outsourcing. The Drivers The reasons vary with the company and the application, of course, but they most often include a combination of the following: Competitive advantage. An outside firm may be able to help develop applications that create a strategic advantage. It is quite likely, though, that the outside firm will be willing to provide the same services to competitors, thus negating the advantage. In the long run, true competitive advantage can normally be achieved only if the company does most, if not all, of the analytics in-house. Organizational agility. There may be changes in market conditions, the emergence of new competitors, or changes in technology that require a fast response. An outside analytics provider may be able to act quickly. Core competency. A firm may choose to outsource those activities, such as analytics, that are not considered to be critical to success. Faster development. If a company doesn t have the hardware, software, or specialized skills needed for a particular project, an outside firm may be able to provide a solution more quickly. However, once the decision is made to invest in the required resources to do the project in-house, it may be possible to respond more quickly with changes or enhancements to the application later on or to develop new applications. Improved quality. Because they work in specific analytics areas and depend on delivering successful implementations, outsourcing firms may be able to provide higher-quality solutions than in-house personnel. Specialized skills. Analytics may require specific skills that are not available in-house, and it may be better to contract with a firm that employs people with the needed skills and experience. Analytics outsourcing firms have a high level of specific domain expertise because of their work with a large number of clients. Cross-industry experience. Some analytics outsourcing firms work across industries, and based on this experience, may be able to bring new approaches and technologies to a firm in a particular industry. Cost. Analytics often requires specialized hardware, software, and skills, and it may be less expensive to outsource these to a firm that is able to spread the costs over multiple companies. In the long run, however, the costs may favor an in-house implementation. Hertz Hertz is a leader in the rental car business with approximately 10,400 corporate, licensee, and franchise locations in North America, Europe, Latin America, Asia, Australia, Africa, the Middle East, and New Zealand. Hertz is the number one airport rental car brand in the U.S. and operates at 111 major airports in Europe. With its recent acquisitions of Dollar and Thrifty, Hertz offers rental cars across a variety of price points. Analytics at Hertz Hertz has a BI and data warehousing group that maintains the company s 2 TB data warehouse. The group includes developers in the U.S. and Europe who are responsible for queries, reporting, and multidimensional modeling. It also has BI analysts who are responsible for determining information requirements. Business units have additional analysts with deep domain knowledge; the units also employ a variety of analytical specialists. Many of the business units rely on analytics and use outside firms for analytical services. Consider the following examples, the external services used, and the reasons why. Data Warehousing A data warehouse has been in place at Hertz for many years, but as the amount of data, number of users, and the complexity of the analytics grew, it was no longer BUSINESS INTELLIGENCE Journal vol. 18, No. 4 5

8 Analytics Outsourcing meeting organizational needs. After an evaluation process, Hertz selected Teradata for its new platform along with Aprimo (a Teradata product offering) for CRM applications. Teradata s professional services staff was brought in to help with the customization of Teradata s logical data model for the transportation industry, create connections to query tools, and develop a semantic layer between Teradata and the tools. Teradata professional services were used because Hertz did not have the requisite skills in-house, but the long-term plan is to reduce its dependency by developing the needed expertise internally. Other third parties also contributed to the selection and rollout of the Teradata and Aprimo products. Gartner was consulted on the strategic direction for data warehousing, and LoyaltyOne (discussed later) contributed to the design of the data model Hertz implemented. Teradata s professional services were selected because of their in-depth knowledge of the Teradata and Aprimo products and their experience within and across different industries. The desire to decrease the use of the services over time is primarily due to cost considerations. Revenue Management and Pricing A rental car is a perishable good, much like an airline seat or a hotel room, in that it generates value only if it is used. In other words, a rental car sitting on the lot is not generating any revenue. A key to success in this industry is dynamically pricing cars so that revenues and profits are optimized. Pricing is a combinatorial, challenging problem because of the large number of locations, types of cars, possible rental dates, and other factors that affect pricing decisions. It is also an area where analytics has been applied for many years. Pricing systems operate using a combination of inventory data (what cars are available), demand forecasts (what cars are likely to be demanded), and mathematical programming techniques (what prices are optimal). Hertz handles pricing in-house but partners with an industry leader in revenue management systems to assist in developing and maintaining its various models. Working with this industry-leading firm allows Hertz to leverage the specialized skills, experience, and insight they have gained as they operate across industries. Customer Relationship Management For over 20 years, Hertz has used Brierley + Partners to drive its CRM efforts. Brierley specializes in loyalty programs, customer-centric marketing services, and analytics and customer research services. It also provides production/fulfillment services such as loyalty program design, customer relationship management strategy, technology, and creative services. Hertz s CRM relies on Brierley s production team, creative staff, technology people, data quality experts, and analytics specialists. Brierley s employees are essentially a part or extension of Hertz s CRM team. Hertz works with Brierley to set the CRM strategic direction and goals and works closely with Brierley daily on the execution of its CRM initiatives. For example, Brierley now uses the customer-centric Teradata data warehouse (Brierley used to maintain a similar data mart) to generate dashboards that show key metrics; the firm also performs customer analytics such as market segmentation analysis. Brierley works on Hertz s customer rewards program, including research and advice about issues such as whether a customer s points should be transferable to anyone else (they now are). Hertz works with Brierley because of the advanced skills and expertise of its people. For example, some of its analytics staff have Ph.D.s and years of CRM experience. Another reason is the ability to leverage Brierley s state-of-the-art technology. Hertz benefits from Brierley s expertise and their knowledge of the best practices of customers across many firms and industries. Finally, Brierley has dedicated resources to respond quickly to any issues or problems that arise. Hertz also uses LoyaltyOne to help with its CRM initiatives. LoyaltyOne specializes in customer insight and strategy, loyalty and marketing programs, and managing the customer experience. Hertz has worked with Loyalty- One in a variety of ways over the years, but the recent 6 BUSINESS INTELLIGENCE Journal vol. 18, No. 4

9 Analytics Outsourcing emphasis is on developing the technical requirements (working with Teradata and Brierley) to ensure Hertz will successfully execute its new CRM initiatives. Hertz works with LoyaltyOne because of its specific skills and experience. The firm was helpful in evaluating Hertz s data architecture and designing the customercentric data model for the data warehouse. Its people provide a valuable combination of technical and business skills and provide great ideas about what can be done with existing data. Conclusion Like many firms, Hertz uses a wide variety of analytics services providers. In many instances, the relationship has existed for many years. In the case of Brierley and LoyaltyOne, the major reasons for their continuing use include their specialized skills and experience, their ability to keep up with and provide the latest technology, and their knowledge of best practices across different firms and industries. Of course, good performance is also key. If outside providers fail to perform satisfactorily, they are unlikely to stay in business for long. Analytics outsourcers need to be competitive to keep their client base. In the case of Teradata, specific skills were needed to get started and will still be needed as new source systems and subject areas are added to the warehouse and the technology evolves. For cost reasons, most of the skills needed for operating the warehouse will be in-house. The IDC study projects a continuing rise in analytics outsourcing. It is difficult to forecast specific growth numbers, but based on the experiences at Hertz and other companies, it will be significant. Reference Zaidi, Ali, and Mukesh Dialani [2013]. Worldwide Business Analytics Services Forecast, IDC Report, April. BUSINESS INTELLIGENCE Journal vol. 18, No. 4 7

10 Big Data Projects Three Best Practices for IT and Business Users in Big Data Projects Fern Halper Fern Halper is director of TDWI Research for advanced analytics, focusing on predictive analytics, social media analysis, text analytics, cloud computing, and other big data analytics approaches. TDWI recently built a big data maturity model and benchmark assessment tool. The goal of the model is to provide guidance for IT and business professionals on their big data journeys. The model provides a framework for companies to understand where they are, where they ve been, and where they still need to go on their big data deployments. The model itself consists of five dimensions: organization, infrastructure, data management, analytics, and governance. A great feature of the TDWI Big Data Maturity Model is the interactive benchmark assessment. At the end of the benchmark survey, you can quantify the maturity of your deployment in an objective way, understand your progress, and identify what it will take to get to the next maturity level. We have identified five levels of maturity in the big data model: nascent, pre-adoption, early adoption, corporate adoption, and mature/visionary. Of course, organizations can be at different stages of maturity in each of the five dimensions, and most are. There is also a chasm that companies need to cross to get from early adoption to corporate adoption. You can take a look at the TDWI Big Data Maturity Model assessment tool and guide at tdwi.org/bigdatamaturity. As part of our research for the model, my co-author, Krish Krishnan, and I spoke to quite a few companies. Some are Internet-based, while others are traditional companies. They are at different stages of big data maturity. For instance, some are using relatively advanced analytics on huge amounts of structured data. Others are building out Hadoop clusters as a means of making high volumes of data storage more cost effective. Still others are primarily content-based businesses that are building out a big data 8 BUSINESS INTELLIGENCE Journal vol. 18, No. 4

11 Big Data Projects infrastructure and BI practice to support it. Only a small number could be considered mature or visionary. In our research, we discovered how important it is for IT and the business to work together to achieve success in their big data efforts. This principle was cited by a number of organizations as a key success factor, especially in the enterprise, but also for smaller, Internet companies as they start to grow. Here are just a few of the insights we heard related to business and IT working together. Insight #1: Business needs to help identify the big data opportunities In a TDWI survey conducted at a recent conference, we asked respondents about some of the challenges they faced around big data. The top response was identifying the right problem to solve. Many of the companies we spoke to echoed this. They stated that in order for big data and big data analytics to be widely accepted, an enterprise must find a problem that is worth solving. Chances are that problem is going to be articulated by the business. Therefore, as the seeds of a big data project start to germinate, it s important to get a business person involved so you can get their input and so they can help you articulate the problem in a way that business users will understand. This involves building relationships with the business. Insight #2: Funding must move out of IT for big data success In our discussions with companies, we asked several questions about funding. Some companies were at the experimentation and proof-of-concept (POC) phase and were funding the project out of an IT organization. However, those that were more mature stressed the importance of getting funding from outside of the CIO organization and moving it to a marketing or sales organization, for instance, so that the business has a vested stake in the game. One end user related a story about sitting down with executives (one at a time) and showing them what was possible with big data analytics. He was looking for someone who was asking questions about the data and analysis because this indicated that the executive was serious about big data. Of course, this can involve a lot of show and tell. The key is to demonstrate some wins that get people excited. Executives then bring others on. It is difficult for IT to sustain a big data effort alone. Insight #3: Data sharing is key In order for a company to build a big data ecosystem that drives business action, organizations have to share data. Collaboration is necessary for big data projects. Of course, there are many considerations involved in assembling big data, including people, processes, and technologies. However, sometimes companies get to a certain point with their big data programs where they have assembled large amounts data from across the company and then the ax falls because of company politics. Some companies pointed to the need for a chief data officer someone responsible for data usage and governance at the corporate level. In our research, we discovered how important it is for IT and the business to work together to achieve success in their big data efforts. Others stressed the need for a well-organized data governance program. As one person put it, Big data is a liability waiting to happen. Whose data was it? Whose data is it? Where is it going? How long will it last? These are important questions that people aren t asking. This is a clear case where business and IT need to work together to ensure that data can be shared as well as to put the policies and practices in place for this sharing to occur. Assembling a governance team should start early in the big data process, even if you re an Internet company that is more concerned about getting a product or service out the door than about governing the data that feeds the product or service. BUSINESS INTELLIGENCE Journal vol. 18, No. 4 9

12 mainframes and Big Data Mainframes: The (Other) Elephant in the Big Data Room Jorge A. Lopez Abstract With up to 80 percent of data originating on your mainframe, you can t ignore big data trends. In this article, we recommend steps to get started using Hadoop to leverage your mainframe data. Jorge A. Lopez is the director of product marketing for Syncsort. jlopez@syncsort.com At first sight, mainframes and Hadoop might seem like the most unlikely duo. One appeared in the late 1950s even before the PC while the other (to this day) hasn t reached its teenage years but is already bragging about managing big data. Much has been said and written about the death of mainframe computers, but the truth is, some of the largest organizations (think of the top telcos, retailers, insurance, healthcare, and financial organizations of the world) still rely on mainframes for mission-critical applications. When talking to these organizations, it s not unusual to hear that up to 80 percent of their corporate data originates on the mainframe. That is some serious big data, and organizations cannot afford to neglect it! That s why they are making the mainframe a core piece of their big data strategies. How can such organizations get started with Hadoop? What are some practical Hadoop use cases for mainframe users? Paving the Road from Mainframe to Hadoop These four practical steps can help you and your organization start off on the right foot to leverage mainframe data with Hadoop. 10 BUSINESS INTELLIGENCE Journal vol. 18, No. 4

13 mainframes and Big Data Step 1: Build knowledge and sensitivity about mainframe and Hadoop This may sound like the most obvious step, but you would be surprised at how often this aspect is overlooked. Despite the importance of both Hadoop and mainframe technologies, it s common to find that their advocates know little or nothing about each other. Therefore, before you get started, it s important to understand the intricacies behind these two technologies and the teams that support them. Some aspects to cover include: Understand the nature of existing applications running on mainframes versus Hadoop. Mainframe applications typically include a combination of batch as well as transactional processing (OLTP). Hadoop applications are mostly batch oriented but are more analytical in purpose. The most important difference is that mainframe applications are the most mission-critical and need to accurately and reliably operate 24/7. pedigree. Therefore, any sensible approach to leverage mainframe data in Hadoop needs to be examined carefully. Decisions must be made regarding what data is fair play and about a security infrastructure that guarantees secure data access and storage. Understand your mainframe SLAs and costs. Storing and processing data in the mainframe is expensive. The costs are relatively easy to quantify because mainframes are billed in terms of CPU utilization. This is important because before mainframe data can be leveraged, it needs to be moved and transformed. Depending on the amount of data and required frequency of data loads, moving data alone can be costly. For instance, a major bank won t be happy if customers have to wait longer at ATMs because IT is moving or copying a terabyte of data. Meticulous load scheduling and capacity planning can go a long way to avoid issues on this front. Identify critical data generated by mainframes. Big data and Hadoop initiatives may center around capturing and processing unstructured and semi-structured data coming from Web logs, social media, and other sources that is, the data that influences or leads to a transaction. Mainframes then process and capture the transactions, also generating critical data that provides reference and valuable context to big data. Similarly, the team will need to look at where the data transformations take place. Factors such as the amount of data that you need to transfer, SLAs, and mainframe CPU utilization are part of the decision, which usually comes down to a compromise between performance and costs. In addition, be aware that any additional thirdparty software on the mainframe will use more CPU cycles and thus increase your annual mainframe costs. Address security concerns. There s a reason why the mainframe is still around: it is quite possibly the fastest, most reliable data processing platform. There s more: it s highly secure. That s the kind of system you need when processing health records or financial transactions. Therefore, mainframe developers are very keen about the security, confidentiality, reliability, and integrity of transactions. On the other hand, Hadoop developers might be more interested in agility, finding relevant trends, right-time analytics, and scalability. After all, missing a single tweet or a Web click might not be such a big deal, but the slightest error in a financial transaction can send you and your CEO to jail. Needless to say, mainframe administrators will be very reluctant to allow access or even to install third-party software without a mainframe Step 2: Be clear about the business and IT objectives In most cases, mainframes are not going away. It s not difficult to see why after considering Step 1. Instead, Hadoop presents a tremendous opportunity to uncover valuable business insights from otherwise unused (or poorly used) data. By doing so, you ll be able to complement the capabilities of your mainframe with increased business agility, virtually unlimited reporting, and new analytics opportunities. Why not? You can maximize the return on your mainframe investment by selectively choosing the right home for the right workload (a phrase I learned from Shawn Rogers a recognized analyst and long-time contributor to TDWI). After all, not all of your data deserves the BUSINESS INTELLIGENCE Journal vol. 18, No. 4 11

14 mainframes and Big Data first-class treatment, right? More important, by offloading some of the less critical data and batch processing from your mainframe into Hadoop, you can lower costs, provide better access to mainframe data, and deliver better service for all your mainframe users. My advice: be clear about the value and expertise that mainframe admins and developers bring to the table. You will need them on your side in order to succeed. Step 3: Create a road map that gradually builds the skills of your organization Leveraging mainframe data in Hadoop is not easy. I know this is not what you want to hear, but that s the reality and you should probably beware of any person, website, or individual that tells you otherwise. Granted, some approaches are better than others and will make it easier, but it will never, totally and simply, be easy. Therefore, it s important to create a road map that allows you to gradually build the required skills within your staff, minimize risk, and capitalize on previous successes to gain more support. Here is a high-level road map that makes sense for many organizations: Create copies of selected mainframe data sets in HDFS. Combine this data with other data sources to enrich existing analysis and create new reports, dashboards, and visualizations. The main objective here is to uncover new insights and improve decision making. only in reducing mainframe costs, but also by actually preserving mainframe capacity for more critical workloads. In the end, this is about making the best use of your available resources so only specific workloads get the mainframe VIP treatment. Step 4: Create a cross-organization team and involve all stakeholders early in the game I recently met a mainframe customer who proudly told me how he shut down a big data initiative in less than 10 minutes due to security concerns. The Hadoop team had spent months working on the project without involving the mainframe group. Now, looking for final approval, they needed mainframe s blessing. There s not much more to say about this other than you really need to bring key mainframe and Hadoop stakeholders together. Otherwise, you can jeopardize your big data strategy. These are just some of the key, initial steps you need to follow when embarking on a big data strategy that include mainframes and Hadoop. If you have a mainframe, chances are a Hadoop initiative is closer than you think. Therefore, you may want to start implementing these steps, especially Step 1, sooner rather than later. Migrate mainframe data to HDFS. Once you ve built a certain level of skills and are familiar with the challenges, you can start, not by copying, but by migrating selected mainframe data to HDFS. Legacy data is usually a good initial target; then you can move on to other data sets. The benefits at this stage add up, so in addition to better insights, you can contain or reduce costs by offloading data from the mainframe into Hadoop. Offload batch processing. A large portion of mainframe processes involve sorts, copies, reporting, and other batch operations. That sounds like the ideal workload for Hadoop, which means offloading these processes to Hadoop can actually help on many fronts not 12 BUSINESS INTELLIGENCE Journal vol. 18, No. 4

15 Data Scientists Filling the Demand for Data Scientists: A Five-Point Plan John Santaferraro John Santaferraro is VP of solutions in the ParAccel Platform Group at Actian. He has 18 years of experience in big data, analytics, and business intelligence. john.santaferraro@actian.com Abstract Big data offers enterprises big opportunities. To make sense of all the information being produced and collected, enterprises have been turning to data scientists, those geeks with proficiency in parallel processing, MapReduce, petabyte-sized NoSQL databases, machine learning, and advanced statistics. There s just one hitch: a McKinsey report suggests that by 2018, a shortage of data scientists will emerge, ranging from 140,000 to 190,000 in the U.S. alone. How can enterprises cost-effectively prepare for their datadriven future in the face of this shortage? The solution is an internal program that provides the opportunity for existing data analysts, BI analysts, and business analysts to acquire the skills they need to become big data analysts. These big data analysts then perform the predictive and prescriptive analysis and discovery needed to innovate and compete effectively. Along with developing education programs, companies need to consider providing incentives for existing analysts to participate, reorganizing their analyst community to support big data analysts, deploying technology infrastructure to support analytics, and fostering an enterprisewide culture of analytics. Introduction The 2011 film Moneyball (based on the 2003 book by Michael Lewis) focuses on Oakland Athletics general manager Billy Beane s ultimate success in building a competitive professional baseball team using data instead of tired truisms and the instincts associated with years of baseball experience. For me, the real hero of the film is Peter Brand, the research geek and statistician (based on real-life baseball scout and executive Paul DePodesta) who is elevated from backroom obscurity to baseball BUSINESS INTELLIGENCE Journal vol. 18, No. 4 13

16 Data Scientists celebrity as Beane s assistant GM because of the success of his analysis. The story of Peter Brand is being repeated in company after company as the executive suite looks to data scientists to help them obtain the benefits promised by big data. According to the Harvard Business Review, Thousands of data scientists are already working at both start-ups and well-established companies. Their sudden appearance on the business scene reflects the fact that companies are now wrestling with information that comes in varieties and volumes never encountered before (Davenport and Patil, 2011). Data scientists are cropping up in companies of every size and in every industry and using predictive analysis and data mining for competitive advantage. Data scientists are practitioners of data science, which, according to Data Science: An Introduction, is an advanced discipline, requiring proficiency in parallel processing, map-reduce computing, petabyte-sized nosql databases, machine learning, and advanced statistics (Andrus and Cook, n.d.). Data scientists were originally found in only a handful of enterprises (such as LinkedIn, Twitter, and Facebook) that needed to mine their massive social media streams, as well as companies such as Netflix and Amazon that wanted to leverage predictive analysis to recommend movies and books to their customers. Today, however, data scientists are cropping up in companies of every size and in every industry and using predictive analysis and data mining for competitive advantage. Data Analysis Reveals Data Scientists to be in Short Supply Like baseball, the business world will be changed by the ascendency of data, but there s a problem with the sudden demand for data scientists. There aren t enough of them and the situation is getting worse. A McKinsey report suggests that by 2018, a shortage of data scientists will emerge, ranging from 140,000 to 190,000 in the U.S. alone (Manyika, et al, 2011). As the competition to hire these experts increases, so will their salaries. In the face of this shortfall and increasing cost pressure, should you take out an insurance policy by hiring data scientists now even if you aren t ready for them, just to make sure you won t be left behind? Possibly, but probably not. This is a very expensive strategy that may deliver benefits to your company in the long term, but other solutions are available. There are several steps you can take that will provide the data expertise you need when you need it and drive your transformation into an analytics-driven company. Businesses Already Have Analysts The first step is to look at your existing legions of analysts to identify those with the background, talent, and desire to increase their skill set and fill the data analytics positions you will eventually have. Let s review the analysts most companies are already hiring to see which ones could take on the analytics role if given the right training, incentives, and tools. Data Analysts Data analysts understand where data comes from and how it can be made useful for business users. They focus on capturing, understanding, cleansing, transforming, modeling, and loading data. They may also integrate multiple data sources into a single repository, such as Hadoop or a data warehouse. Most data analysts have taken computer science courses and have a solid grounding in mathematics, possibly including statistics courses. It makes sense to look at the pool of data analysts for those who may want to expand their skill set to be able to use analytics. 14 BUSINESS INTELLIGENCE Journal vol. 18, No. 4

17 Data Scientists BI Analysts Once data has been moved into data warehouses or into data marts by data analysts, business intelligence (BI) analysts perform the next level of data preparation. Although BI analysts generally have a solid understanding of data sources and types, their focus is on using BI tools such as MicroStrategy, Tableau, and QlikView to present information in a more user-friendly and visual way to make it accessible to business users. They typically generate static reports or create interactive reports or dashboards that let users drill down into the details. BI analysts are often thought of as report writers or dashboard builders, but many have a basic understanding of analytics and may have a desire to get additional training and expand their expertise in more advanced analytics. Business Analysts Business analysts generally don t have as deep an understanding of data sources and types as data and BI analysts, but they are able to transform the information through reports and dashboards into actionable insights for the business. For example, a supply chain business analyst uses the data to optimize supply chain processes, from sourcing materials for manufacturing through product distribution and point of sale. Customer business analysts understand the mathematics involved in segmentation, affinity, and optimization of an offer, and use data to increase the number of conversions and retain the most profitable customers. Because business analysts most directly tie the data to business insight and likely have already dabbled in data analytics they are particularly appropriate to be considered for expanding their roles. Understanding the Role of the Big Data Analyst All three of these analyst groups have generally been more comfortable with descriptive analysis, that is, with describing what has happened and what is happening. What businesses need today, however, is the ability to discover new patterns and anomalies, predict scenarios, and prevent negative business impact. Predictive analysis examines what will happen: who will respond to a specific offer, under what conditions are customers most likely to leave for a competitor, and what are the characteristics of the people most likely to commit fraud? Prescriptive analysis looks at recommendations for the next best offer or action. For example, what is the best offer to make in order to retain customers while maximizing margins? What price point is necessary to double or triple sales? If a supply chain source is disrupted by a natural disaster, what is the next best source for getting the right items to the right place at the right time for the least cost? Discovery the ability to discern something that no one else has seen or wondered about is also an essential capability. What are some new trends impacting my industry? Why are attitudes about my product or business changing? What are the latest fraud techniques hackers are using to break into networks, and what parts of my network are most at risk? It makes sense to look at the pool of data analysts for those who may want to expand their skill set to be able to use analytics. Advanced analytics, combined with new data sources and types, opens the door for a new crop of analysts big data analysts. This new breed of analyst uses advanced analytic techniques on large and diverse data sets to uncover hidden patterns, unknown correlations, and other useful information. The skills required include a basic understanding of analytics, data mining, statistics, and natural language processing what one might call Analytics 101. Big data analysts may not have to create a linear regression algorithm, but they should understand how linear regression works. They probably won t write advanced clustering algorithms to look for patterns, but they need to understand how pattern matching works and where it can be used. Big data analysts must understand a wide range of analytics use cases, such as golden path, pattern matching, triggers, events, affinity, and socially aware text analytics. BUSINESS INTELLIGENCE Journal vol. 18, No. 4 15

18 Data Scientists Big data analysts must also have a working knowledge of different types of data including conventional sources, Web and mobile application data, machine data, log data, and sensor data and they must also be able to understand and explore new types of data as they emerge. Only by doing so can they see the potential of integrating the different types of data to discover connections or arrive at insights that have never before been discovered. Equally important, big data analysts must understand the business not just general business principles, but the specific industry and organization they are a part of. Only by understanding these current practices and challenges will they be able to spot opportunities for optimization and competitive advantage. Big data analysts must understand a wide range of analytics use cases, such as golden path, pattern matching, triggers, events, affinity, and socially aware text analytics. Important capabilities to look for in a big data analyst candidate include: Fast thinking: Analytics is an iterative process: query, review results, tweak the query, iterate. To be productive, the big data analyst must be able to quickly analyze results, assess their impact or value, and formulate a new path of discovery. Innovative thinking: Big data analysis is all about thinking outside the box. It s an exploration a discovery process. Big data analysts must be open to completely new ideas and new ways of doing things. They must be able to see new patterns and detect anomalies and outliers, and they must be able to imagine the potential of combining data that hasn t been combined before. For example, does the weather impact the sale of items that one would not normally associate with the weather (such as consumer electronics or types of meat)? Does the rising cost of consumer goods reduce the use of power and cooling? Storytelling: A fundamental job of big data analysts is to put their insights into an understandable context for business users. They must be able to tell the story of the data. Whether it s discovering how the weather in one part of the country is affecting home prices in a seemingly unrelated geographic area or seeing a connection between fans of a certain artist or style of music and potential voters in an election, the big data analyst must be able to convince business users of the connection and its importance. The storytelling may need to be visual using tools such as Tableau or QlikView or in prose. People with both verbal and visual skills should be highly prized. Creating a Big Data Analyst Program and Becoming an Analytics-driven Enterprise You may be able to hire big data analysts, but the strategy that will deliver the greatest long-term benefit includes educating and promoting existing analysts with a program that fosters an analytics culture and helps transform your enterprise into a truly data-driven organization. Here are five tasks you can begin today to create such a program. 1. Create educational opportunities. Start by offering big data analytics education to all interested individuals technical and business staff alike. Think of your training program as an extended analytics center of excellence (ACE). Both business and technical teams will benefit from training in Analytics 101, developing a core understanding of how to use analytic functions. Furthermore, additional business education topics such as marketing, supply chain, or risk management will help data analysts expand their business acumen. As business users learn more about big data technology, they will be better able to understand how new tools and techniques can change the way they do business. In addition, they will learn to better communicate their needs to big data analysts and adapt their business processes to include embedded 16 BUSINESS INTELLIGENCE Journal vol. 18, No. 4

19 Data Scientists analytics or analytics-driven action. Some of the more technical business sponsors will quickly see the potential of analytics and may even choose to pursue the big data analyst position. Finally, by opening big data programs to all participants, organizations can foster a better general understanding of the value of big data and its impact on the way they operate and the services they provide. 2. Provide incentives for participants. Provide incentives for analysts to participate in relevant education programs and reward them for reaching milestones. With the shortage of data scientists and the increasing value of their contributions, companies that provide analytics training without incentive programs run the risk of training new talent for other companies to poach. 3. Reorganize to support big data analyst success. Historically, businesses have organized their analysts in one of two ways either tied to a particular business unit or grouped into a large pool of resources that are available to the business units as needed. Neither strategy is sufficient. Analysts tied to a particular business unit tend to be isolated and out of touch with big data technology. They tend to miss out on the collaboration that normally takes place among analysts and the water cooler conversations about new advances in technology and analytic techniques. In a similar way, analysts separated from the business units into a general resource pool often find themselves a step behind what is happening in the business units. They are never able to adequately understand the issues and challenges of the business units, and therefore can t provide the insights that are needed. They end up reacting to the needs of the business instead of being proactive in their approach to analytics. A much better organizational approach is a hybrid of these two strategies, where some analysts are tied to a business unit, working at a more strategic level, and yet are still a part of the greater community of analysts. Connection to the business puts common goals and objectives at the forefront of their minds, making all of their analytic efforts more strategic. Collaboration with the community of analysts opens them up to constant interaction and sharing of ideas. This approach offers the best of both worlds. The centralized analysts provide the business with a growing set of capabilities driven by new technology and innovation. Meanwhile, analysts within the business units use what they learn from the community to transform the way their colleagues do business. This hybrid approach is critical for companies that want to use big data not just to provide better answers to the same questions they ve always asked, but to truly explore the data and discover new insights. By opening big data programs to all participants, organizations can foster a better general understanding of the value of big data and its impact on the way they operate and the services they provide. As part of this reorganization, consider creating an analytics center of excellence, an informal mechanism for bringing business and technology staff together to develop a common vocabulary, share insights, and create opportunities for cross-pollination. The center of excellence provides a forum to enlighten business users on the power and possibility of the data and technology. At the same time, analysts are exposed first-hand to a better understanding of business processes, requirements, and desired outcomes. One of the most promising outcomes of this kind of collaboration is a road map that drives the increase of data-driven decisions and analytics embedded in business processes for real-time action. 4. Deploy infrastructure to support analytics. No matter how many incentives you put in place and how much education you offer, no big data analytics initiative can succeed without a technology infrastructure that supports unconstrained analytics on massive volumes of information. (Even if you eschew the entire big data ana- BUSINESS INTELLIGENCE Journal vol. 18, No. 4 17

20 Data Scientists lyst strategy and decide to hire data scientists tomorrow, you ll still need such a system for them!) Few existing database systems support unconstrained analytics, so most companies will need to add this technology. Instead of ripping and replacing existing systems, however, forward-thinking companies will look to add an analytic platform with the following capabilities: Embedded analytics: A library approach to analytics, with functions embedded in the analytic database, allows the big data analyst to run sophisticated analytics with a simple SQL call. They don t have to develop algorithms; they just need to know how to use them. Agile extension: Users should be able to easily incorporate new mathematical, statistical, and data mining functions into their analytics library without interrupting analyst productivity. Rapid iteration: The typical analyst follows an iterative process of discovery. High-speed execution of complex queries and access to many data sources allow an analyst to quickly tweak an algorithm or bring new data into the mix. Real-time access: Big data analysts frequently need access to new, time-sensitive data as soon as possible. An analytic platform must support some kind of on-demand access to data and high-performance processing. Extreme flexibility: The system must be flexible enough to allow analysts to run queries whenever they want, make changes to the query at will, and easily enrich or alter the data set to support the discovery process. 5. Foster a culture of analytics. After you have completed the previous four steps, you are in a position to foster a culture where analytics is valued by the entire organization. When it comes to making decisions, many enterprises value most a person s title and years of experience whether or not that person makes decisions based on facts. By fostering a culture of analytics, companies can begin to eliminate opinions, emotions, gut feelings, and ego from decision making so that executives base decisions on data. The impact of those decisions can be tracked, so future decisions continue to be based on the evolving reality. In the film Moneyball, the biggest obstacle to accepting data as the foundation for decisions was that it challenged the authority and influence of those with years of baseball knowledge. Today, every baseball team uses data and statistics as the foundation for building their teams, and it hasn t destroyed the game. Experienced baseball managers and scouts are still highly prized they just have more information available to them than ever before. A data-driven company will still need influential executives with years of experience, but it s time to arm them with the facts and insights they need to make the best reality-based decisions possible. The supply of data scientists will never catch up to the demand for data-driven decisions. The answer to what could amount to one of the most strategic challenges of this decade lies in helping existing analysts become big data analysts. References Andrus, Calvin, and Jon Cook [no date]. Data Science: An Introduction, Wikibooks. wiki/data_science:_an_introduction Davenport, Thomas H., and D.J. Patil [2011]. Data Scientist: The Sexiest Job of the 21st Century, Harvard Business Review. Manyika, James, et al [2011]. Big data: The next frontier for innovation, competition, and productivity, report from McKinsey Global Institute. insights/business_technology/big_data_the_next_ frontier_for_innovation 18 BUSINESS INTELLIGENCE Journal vol. 18, No. 4