A Better View PAGE 3

Transcription

1 Bring Dark Data Into the Light 12 Unused data can transform organizations analytics programs. Architected for Success 14 Teradata Analytic Architecture Services maximize the business value from analytical systems. See the Big Picture 17 Expand your business horizons with fast, easy enterprise access to Hadoop data. A Better View PAGE 3 The Teradata Unified Data Architecture

2 VIEW FROM HERE Best of the Best The Teradata Unified Data Architecture is the key to capturing, analyzing and acting on all types of data. There s no doubt that big data analytics is a big challenge for many organizations. Why? It comes down to two specific issues: integration and the last mile of delivery. Every day, organizations generate massive volumes of multi-structured data from a wide range of sources point-of-sale terminals, websites, social media, , voic , you name it. It can be tough to determine which platforms are best for which data types, and how those platforms integrate so you can make use of the data throughout your organization. The second challenge is what I call the last mile of delivery. It s one thing to gather and integrate all of the data, but it s quite another to extract value from it. Some newer technologies require very specialized talent to extract the value, and those people can be expensive to obtain and retain. Deploy and Deliver The Unified Data Architecture a collection of platforms, applications and tools for data management, processing and analytics addresses those concerns. Not only does it give you the data integration that s so important to creating business value, but it also provides for that last mile of delivery using common tool sets and standard business intelligence (BI) tools. With the Unified Data Architecture, you can effectively and quickly deploy big data analytics and deliver the resulting insights to the point in your business where decisions are made and value is gained. You can also leverage the BI and skill investments you ve already made so you can focus future BI investments on truly incremental analytics. Remove the Confusion The world is moving from a single integrated analytics platform toward a logical analytics platform, and it needs a roadmap. Even if the IT organization hasn t yet made that move, I would say the majority of users are already starting to build their own solutions to take advantage of newer tools and data that are being created. The Unified Data Architecture enables IT to talk with the business about how to integrate and deliver end-to-end solutions. A cornerstone to the logical analytics platform is still the Integrated Active Scott Gnau president of Teradata Labs Warehouse that enables real-time delivery of integrated intelligence to front-line decision makers. This is now critical because of the confusion in the market concerning new big data technology and vendors. It s even being labeled disruptive technology. That hype, along with today s very real business challenges, can make it difficult to see what the next move should be, which means it s all too easy to get off on the wrong path. With the Teradata Unified Data Architecture, you have a nice framework that removes the confusion and enables organizations to make proactive decisions about how they move forward with analytics. It integrates best-ofbreed technologies to support the new kinds of data that are available and delivers tangible results. T 2 l Teradata Magazine l Special Edition l Unified Data Architecture

3 A Better View INDUSTRY EXPERTS TOUT THE PROBLEM-SOLVING CAPABILITIES OF THE UNIFIED DATA ARCHITECTURE. by Colleen Marble Companies are being bombarded with more data and more types of data coming at them faster than ever before. The days of a controlled inflow of highly structured data are over, and a new world of multi-structured, high-volume data has begun. More data is better, right? Yes provided organizations have the skill sets and platforms to capture, analyze and standardize that data to guide them to make the best business decisions possible. But many companies are learning that their data types and diverse analytics needs exceed the existing capabilities of standard data warehousing or business intelligence (BI) solutions. The answer to this challenge is the Teradata Unified Data Architecture that brings together the processing power, storage capacity and analytic capabilities of multiple platforms into one cohesive environment. This enables users throughout the organization to extract value from all available data, giving key stakeholders actionable intelligence on which to base their critical decisions. >> TeradataMagazine.com l 3

4 The Unified Data Architecture enables companies to integrate and extract value from all types of data throughout the organization to empower users to do more across the enterprise. Look Forward, Not Back The market is evolving to a logical data warehouse. In this market, it s crucial for enterprises to be ahead of the competition with their vision and technology. Although BI platforms have claimed for years to provide actionable intelligence, the Unified Data Architecture takes the concept to the next level. The solution now makes it possible to ask any question of any data and get near real-time answers. In a traditional environment, it s simply too difficult to conduct that kind of in-depth discovery analysis in a timely manner given the volume, variety and velocity of today s data. CEOs and other business leaders run the risk of not having the right data at the right time to make fully informed decisions about highly complex business problems. Users are restricted to asking simple questions of data, not asking questions at all, or getting partial answers too late to take advantage of opportunities. They re unable to see and understand unique associations in the data to predict outcomes or resolutions. In the good old days, most companies had BI, not analytics, explains Evan Quinn, senior principal analyst, data management and analytics, for Enterprise Strategy Group. BI is typically look backward. Most of the data you re working with is structured, and there s not a lot of iteration or discovery around BI. The questions are relatively straightforward: How many people have met their sales quota this quarter? for example. Analytics, on the other hand, looks forward. With analytics, we may or may not know what the structure of the data is. We may be dealing with semi-structured data, and we may have to go through many processes to build analytic models, Quinn continues. It s no longer a question of What have we done? Rather, it s a question of What should we do? Trustworthy Results In this brand new era of analytics, the integrated data warehouse (IDW) becomes one of several platforms used to capture and process data to create actionable intelligence. The IDW continues to play a very critical role in a hybrid ecosystem; it s just not the center of it, explains Shawn Rogers, vice president of research at Enterprise Management Associates. There s a need for flexibility and a need for companies to try to align their data and their workloads with the best possible platforms. Within this unified environment, data is stored and processed on the platform or platforms that best meet user requirements for response time, long-term storage and access. Not only does this environment return answers faster than previous architectures, it integrates the data so users will have faith in the results. They know the solution takes into account all available information to deliver a panoramic view of the business. Consistency Across Platforms The integration enabled by the Unified Data Architecture allows for data governance and stewardship practices to be consistent across all data platforms simultaneously. Previous architectures were very monolithic, which meant that you had a single platform that was predicated on having a single source of truth, notes Tony Baer, principal analyst at Ovum. A unified data environment acknowledges that there are many analytics and many paths to getting what you need, and 4 l Teradata Magazine l Special Edition l Unified Data Architecture

5 ARCHITECTED FOR OPTIMAL VALUE The Unified Data Architecture is a proven, safe and cost-effective framework for smarter data management, processing and analytics that enables organizations to exploit all their data, regardless of structure. This collection of services, platforms, applications and tools helps organizations define and deploy an architecture that makes optimum use of available technologies in a way that unleashes the full value of data. you need to have the right platform for the right workload. But it s not just about availability, Baer adds. It gets you closer to being able to manage something such as data quality with a consistent policy even if you have different practices for carrying it out based on different data types and data requirements. You can more easily federate data between different platforms. The same is true for other key practices. If you have a UDA, you can also start applying backup, recovery, business continuity, security, compliance, data protection all of the things you need to run an ERP class application, Quinn points out. Those are going to be required for big data, and if you ve got everything more or less in one place, that s going to facilitate that process. Not only does this help across platforms, but it creates consistency. The Unified Data Architecture enables companies to integrate and extract value from all types of data throughout the organization to empower users to do more across the enterprise. Now s the Time The Unified Data Architecture addresses a wide range of business pain points that aren t easily solved by traditional data warehouse or BI environments. By integrating a variety of platforms into one solution, businesses can use the right platform for the right use case to deliver the best possible results in a timely and economical manner. The solution also makes it possible to apply governance, security, business continuity and other critical policies in a consistent manner across all data, regardless of platform. The key benefit, however, is that the Unified Data Architecture facilitates more complex data discovery, which in turn provides more actionable insights. It opens up new opportunities for flexibility and savings to the company, Rogers says. It allows you to do things you couldn t do before. And at its highest or most sophisticated level, it allows you to do more complex work. That benefit comes just in time. Big data is going to require a better understanding, better management and better security of your data, notes Quinn. The time to get on board with having a bigger picture is now. T BRIDGING THE IT/ BUSINESS GAP Using the Unified Data Architecture as a solution for big data analytics is gaining momentum. However, there s still a lot of work to be done to close the cultural divide between the business and IT sides of the house. It s great to talk about the volume, velocity, variety, veracity all the Vs but you re really living in a vacuum if you only talk about those things. That s the IT world, explains Tony Cosentino, vice president and research director at Ventana Research. When you really look at the business users, the guys who are tasked with insights for the organization, they re much more focused on what I call the Ws what is the data, so what are the inferences and implications of the data, and now what are the decisions that need to be made from that data. Cosentino suggests that the unified data environment is a marriage between the Vs (the IT side) and the Ws (the business side). The challenges are to overcome the cultural resistance to the integration of business and IT, and to meet executive expectations for what the value chain should be for the data. What good is an analysis if you don t know what s valuable at the end of the day? Cosentino asks. You can t find the needle in the haystack if you don t know what the needle looks like. The more we can build analytic centers of excellence that bring together business and IT to address this question, the better off we ll be. Colleen Marble has been writing about business, marketing and information management since TeradataMagazine.com l 5

6 Got to Have It! THE MOST POWERFUL ANALYTICS PLATFORM ON THE PLANET DELIVERS ANALYTICS ON ANY DATA TO REVEAL NEW OPPORTUNITIES. by Brett Martin Handling diverse data types is a challenge for most businesses, requiring the use of multiple technologies. That challenge is compounded by a spike in data creation rates from an ever-increasing range of sources. Organizations need to capture, store and analyze all types of information, including multistructured data from social media, texts, graphics, audio, machines and other sources. Making sense of this data requires powerful analytics and data warehousing tools to extract intelligence that gives businesses a competitive advantage. Teradata has a solution: The Teradata Unified Data Architecture allows for the transparent movement of data in and out of complementary systems. This unified data and analytics environment delivers a high-performance, innovative system that gives any user any analytics on any data. The solution enables organizations to leverage all data for new insights and new business opportunities. True Integration The Teradata Unified Data Architecture is the only truly FIGURE THE TERADATA UNIFIED DATA ARCHITECTURE integrated solution and is also the most powerful and complete analytics architecture available today. It integrates with and benefits from Teradata Database, Teradata Aster and open-source Apache Hadoop technologies. (See figure.) The result is a unified, highperformance architecture that aligns data warehousing, data discovery and data staging to unlock valuable insights for increased productivity, lower costs and new business value. The key components of the solution include: The Teradata Database The database provides a single source of consistent, centralized and integrated data. The integrated approach supports the highest business value through cross-functional analysis. Users can ask 6 l Teradata Magazine l Special Edition l Unified Data Architecture

7 the most challenging questions from determining complex trends and uncovering business anomalies to identifying customers for automated, custom Web offers and get quick answers. The Teradata Aster Database This delivers the patented SQL- MapReduce technology. Business users can run MapReduce functions using SQL, allowing data discovery through iterative analytics against structured and multi-structured data. Even users who don t have a deep understanding of MapReduce can still leverage the robust discovery analytics. Bridging the gap between the business language of SQL and the analytics of MapReduce lets users gain insights that can curb customer churn, increase the success of marketing campaigns and ultimately improve the organization s bottom line. Hadoop The Hortonworks Data Platform provides an appliance for loading, storage and refinement of data. Unstructured data with multiple formats can be quickly loaded into open-source Hadoop. This could be the perfect solution for storing semi-structured data, including weblogs, and performing data refinements such as sessionization and aggregation. Benefit From Any and All Data While these core technologies provide the heavy lifting, the value-add enabling software determines how well they re integrated. The software also controls how much of a burden is left on an organization s shoulders to get the technologies to work together to enable true business value. The Unified Data Architecture offers a rich set of software and services that provide administrators and end users with transparent data access, seamless data movement throughout the environment and one operational management view that includes single-source support. This truly integrated environment allows organizations to focus on creating business value, not on technology integration. The Unified Data Architecture simplifies processing across massive data sets. Organizations can quickly perform iterative analytics against a broad, deep set of data using SQL, SQL-MapReduce or non-sql languages and tools. With its combined capabilities, the Unified Data Architecture allows businesses to explore vast stores of traditional and new data. They can capture, store and analyze the data and turn it into actionable intelligence. Deriving meaning from all data allows companies to better understand and predict customer behavior to improve the customer experience. They also gain a panoramic view of their business and supply chain to improve forecasting and planning, answer questions and identify new revenue opportunities. The solution is also supported by a team of technical experts with extensive industry experience. The experts deliver solutions that remove the common obstacles organizations face, such as deploying and managing new systems, and providing accessibility to enterprise data. A True Business Advantage The Unified Data Architecture offers a breakthrough in analytics. It s the only platform to bring together multiple technologies the Teradata Database, Teradata Aster analytics platform and Hadoop and integrate them with value-add software and support. PROBLEM SOLVER A large global bank was struggling to reduce churn in profitable customer segments. Part of the problem stemmed from the bank integrating customer interaction data across multiple channels from numerous siloed repositories. Further complicating matters was the size of the data billions of records per month. The company turned to the Teradata Unified Data Architecture to help detect and prevent churn. The solution enabled the bank to build an enterprise view of all customer interactions and identify the most frequent paths to account closure. This solution allowed the bank to: > Pinpoint the event causing churn > Remove the event > Reduce churn among profitable customers by 5% The results were achieved using: > A Teradata Database-enabled enterprise data warehouse for historical customer transaction, profile and product information > A Teradata Aster Database to analyze and discover patterns to determine which actions likely caused churn > Apache Hadoop for loading, storage and refinement of data, and optimizing storage costs > Teradata Relationship Manager to make real-time decisions and offers to improve customer satisfaction and prevent account closures By combining the advantages of data warehousing, data discovery and data staging, the solution lets companies quickly and easily answer any business question, regardless of the type of data being analyzed. With the Unified Data Architecture, no data is too big or complex to benefit the business. T Brett Martin is the senior editor for Teradata Magazine. TeradataMagazine.com l 7

8 The Power of Integrated Analytıcs USE CASES DEMONSTRATE THE ABILITY OF THE UNIFIED DATA ARCHITECTURE TO DELIVER UNIQUE BUSINESS BENEFITS. by John Edwards Squeezing the maximum insight out of data collected from a rapidly growing number of sources should be a top priority for every organization. Yet finding an analytics framework capable of fully leveraging the value of both structured and unstructured data created by various applications and services can be a challenge. The Teradata Unified Data Architecture is the answer. This tightly integrated set of platforms, tools and services enables organizations to integrate and exploit all their data for competitive advantage. Here are just a few examples: Communications Telcos can obtain a panoramic view of each customer to better leverage transaction and interaction data and deliver next-generation customer experiences. Carriers are able to monetize data and their existing infrastructure, increasing profitability. Highly sophisticated marketing segmentation can identify key groups such as prime customers who pay bills on time, loyally renew service contracts and frequently purchase addons, such as extra minutes and more data capacity. Relevant and timely offers can then be crafted to retain profitable customers and to attract new customers possessing similar characteristics. > A Russian wireless provider delivered a 10% uplift in retention and $13 million in annual win-back of customers through social network analytics of SIM card data. > A European carrier uses micro targeting to focus on small customer segments with distinctive profiles. By accurately applying event-based marketing to offers, the carrier increased close rates by up to 200%. Financial Services Banks, credit unions, brokerages and other financial service organizations benefit from an integrated picture of customer activity across multiple channels (Web, branch, mobile and ATM), to 8 l Teradata Magazine l Special Edition l Unified Data Architecture

9 customize online banking services, mobile apps and ATM features to match customer behavior patterns and preferences. They can also capture new business by recognizing in-stream activity in near real-time and prioritizing the most relevant offers based on customer activity. > A financial institution increased profitability and reduced customer churn by 5% through identifying and then removing an event that was causing a high number of account closures. > A global bank achieved a 25% uplift in response rates to Web offers and a marketing return on investment (ROI) of 1,400% ($1 of marketing = $14 revenue). A > North American bank realized $7.5 million in incremental value by using a new best customer profile to target new accounts. A > Southeast Asian bank achieved campaigns that were 10 times more successful than before, gaining $961 million in new sales and realizing a 164% annualized ROI. Healthcare Integrating data from all patient sources (predictive behavior analytics, sensor data, pharmacy data, claim data, emergency room visits, etc.) helps these organizations deliver next-generation care management and dramatically reduce plan costs. > One provider uses analytics to help identify false positives. This helped reduce a list of 35,000 potential diabetic patients by 25%. A > health insurer detects fraud before a claim is paid with constant real-time monitoring. A payer s model run time was reduced from days or weeks to minutes, enabling appropriate investigations. > A pharmacy enterprise saved a healthcare organization $40 million over six months due to accurate and quick generic conversion when a popular brand drug went off-patent. Manufacturing Manufacturers can pinpoint weak suppliers, partners who fail to meet deadlines or companies that deliver inferior materials. Conversely, they can also identify suppliers that excel in speed, quality and value. The Unified Data Architecture also provides analytics for immediate and predictive decision support by reducing the time to identify, isolate and resolve manufacturing and operational issues. Enterprises are able to operationalize key findings and iterate quickly for continuous improvement, delivering reduced procurement costs and increased manufacturing quality. > A manufacturer improved yield by 1%, saving millions of dollars. Utilities These companies can strengthen customer satisfaction by enabling customers to manage and reduce their own utility usage. Utilities can also increase efficiency of theft investigations by analyzing interval usage data from smart meters, helping them detect suspicious usage behavior and pinpoint fraud with a much higher degree of accuracy. In addition, utilities can also identify customers with unusually high levels of consumption at unusual times to target for energy efficiency programs. > At a U.S. utility, an integrated view of data resulted in a $9 million cost savings and a reduction of 2,800 man-hours. > Another U.S. utility helps individual customers reduce their energy usage, with the goal to reduce total consumption by 1,000MW roughly the output of one power plant. > An investor-owned utility in the western U.S. transformed how it identifies energy theft, moving from a manual system to a datadriven one. The new analytical system detects theft incidents with 70% accuracy, up from 30% with the manual system. Take the Lead The Teradata Unified Data Architecture puts new analytics capabilities to work for organizations in any industry. Having the ability to rapidly and painlessly analyze large volumes of traditional and new data sources enables organizations to identify and capitalize on new opportunities, learn more about every touch point in their supply chain and better meet customer expectations through the value of integrated data. This intelligence enables organizations to progress from competing in their industry to leading it. T John Edwards has covered the technology industry for more than two decades. TeradataMagazine.com l 9

10 Benefit from Any and All Data HORTONWORKS VICE PRESIDENT SHAUN CONNOLLY DISCUSSES HOW AN INTEGRATED ENVIRONMENT DERIVES VALUE FROM DATA IN NEW WAYS. by Colleen Marble In the age of big data analytics, organizations are looking for ways to capture, analyze and act upon the enormous volumes of multi-structured data generated every day. What they re finding is that the best solution may not be a single solution at all. Instead, the Teradata Unified Data Architecture embraces the brawn of an enterprise data warehouse, the brains of a discovery platform and the breadth of a big data platform. Teradata Magazine spoke with Shaun Connolly, vice president of corporate strategy for Hortonworks, to learn more about what the Unified Data Architecture can deliver. What makes the Unified Data Architecture different from other analytics architectures? CONNOLLY: It s really a question of how you evolve your existing architecture to deal with the challenges of massive data volumes. How can you create a well-integrated system that derives value from data in ways that haven t been possible until now? Also, is there a compelling economic model that can be leveraged against existing skill and solution investments? The Unified Data Architecture realizes this vision by bringing together an enterprise data warehouse, a discovery platform and a big data platform. They re deeply integrated for not only efficient data access and sharing, but also for a robust operational experience which, in my opinion, is almost as important. 10 l Teradata Magazine l Special Edition l Unified Data Architecture

11 The Unified Data Architecture is purpose-built to deal with any and all forms of data. That really resonates, particularly with large enterprises... Shaun Connolly, vice president of corporate strategy for Hortonworks Does this offer benefits that aren t available from other solutions? CONNOLLY: Yes. It enables companies to pick the right data system for the right use case at an optimal cost-performance benefit for their data processing needs. For instance, if you have 10 years worth of data but you re not sure of its longer-term value, then a big data platform enables you to cost-effectively store that information while you decide what to do with it. On the other hand, you also have traditional data warehousing, reporting and BI [business intelligence] solutions that come into play for business-critical data that s highly structured and operationalized. The benefits extend into the area of multi-structured or unstructured data. Before, you may have talked about analyzing that data, but you couldn t access it. Now you can create a space to cost-effectively store, process and report on it in an appropriate way. How does this make it possible for organizations to obtain full value from data? CONNOLLY: Organizations need a good crawl-walk-run strategy, and the Unified Data Architecture spans all three phases. The crawl phase is about creating what we refer to as a data refinery. It captures a large data set and puts it all into one place. It doesn t matter whether that data is structured or unstructured. That refinery outputs to a data warehouse, where you can blend data for reporting, visualization, etc. The walk phase covers data exploration. People can mash up new data with existing data to identify emerging patterns. That might require you to bring together mobile data, Web data and transactional data so you can see patterns that influence, say, customer satisfaction and leverage that knowledge to improve the business. The run phase enriches the online application with advanced analytics to create highly customized experiences. Yes, you ll have batch processing, interactive querying and exploration, but you ll also have the ability to get those results into your online applications very quickly. Is it able to deliver insights faster than other solutions? CONNOLLY: Many companies share data between systems, and they do a lot of point-to-point system integration in order to distribute insights in a timely manner. The well-integrated Unified Data Architecture lets you automate a lot of that data flow and get it where it needs to be a lot faster than traditional methods. This is true regardless of format. Rather than spending a lot of time figuring out how to transform the data into a highly structured format, you can store it in its native format. Structure becomes somewhat irrelevant. It s only applied when you decide what insight you want to operationalize. Why should businesses consider the Unified Data Architecture if they already have an analytics solution in place? CONNOLLY: You want a logical architecture that lets you store and process unstructured, semi-structured and highly structured data within the system. It might be in the data warehouse or in Apache Hadoop, but it doesn t matter as long as it s in a place where you can conduct more sophisticated analytics. Many companies have already operationalized analytics in highly structured relational databases. But when you consider the multi-structured data coming out of sensors, mobile devices or digital videos, traditional analytics don t have a strong answer. The Unified Data Architecture is purpose-built to deal with any and all forms of data. That really resonates, particularly with large enterprises that are trying to integrate all their information and still use appropriate tools for particular data sets. T Colleen Marble has been writing about business, marketing and information management since TeradataMagazine.com l 11

12 Document repositories/ecm 85 BRING 54 % DARK DATA 52 % INTO THE LIGHT 52 % UNUSED DATA HAS THE POTENTIAL TO TRANSFORM Lack of maturity in ORGANIZATIONS ANALYTICS PROGRAMS. say 52 % they can only access structured big data data tooling sets say the Lack data of is support there but their tools can t for real-time make sense dataof it 25 % 13 % 39 UNSTRUCTURED % CHALLENGE Enterprises 38 report % Poor large data amounts qualityof unstructured 37 % 34 % 29 % 39 % 82 % Web behaviors, clickstreams External/public social media report they have too much data, not enough analysis or semi-structured data repositories they already analyze, monitor or query, or would like to analyze, monitor or query: Document repositories/ecm 85 % 82 % Don t have Web enough behaviors, in-house 54 % expertise clickstreams Unsure of External/public how to connect across 52 % all social of their media data sets Data is likely to be too dirty to use Big Data: Extracting value from your digital landfills, AIIM 39 % 38 % Lack of maturity in big data tooling OPPORTUNITIES ABOUND As organizations learn how to leverage unstructured content and connect across multiple repositories, users have the following concerns about their analytics tools: 37 % 34 % 29 % 54 % 52 % Web behaviors, clickstreams External/public social media Lack of support for real-time data Poor data quality Don t have enough in-house expertise Unsure of how to connect across all of their data sets Data is likely to be too dirty to use Big Data: Extracting value from your digital landfills, AIIM DEMAND FOR DATA 52 % Lack of maturity in Companies are interested big in myriad data tooling types of data for their analytics projects: 39 % Lack of support for real-time Relational data from 70 % data transaction systems 38 % Poor data quality 45 % Unstructured data/documents such as PDF, Word, Excel docs 37 % 34 % Social media data Semi-structured industry data MAKE SENSE OF ALL 70 DATA % Relational data from transaction systems Many enterprises 45 % Unstructured have plenty data/documents of but they struggle such to make as PDF, use Word, of it due Excel docs to the 37 % lack of capable tools: Social media data report they have too much 39 % 34 % data, not enough analysis Semi-structured industry data 25 % say they can only access structured data sets 13 % say the data is there but their tools can t make sense of it Balancing Opportunity and Risk in Big Data: Don t have enough A survey of enterprise 37 % priorities and strategies for harnessing big in-house data, Informatica expertise 34 % Unsure of how to connect across all of their data sets 29 % Data is likely to be too dirty to use 12 l Teradata Magazine l Special Edition l Unified Data Architecture Big Data: Extracting value from your digital landfills, AIIM Document repositories/ecm 85 % 82 %

13 LINKED UP 60 % Nearly 60% of companies would find it very useful to link structured and unstructured datasets they already have, but only 2% are able to do so. Big Data: Extracting value from your digital landfills, AIIM SOPHISTICATED ANALYTICS 56 % 56% of enterprises would consider the ability to do sophisticated analytics on unstructured content and data streams very valuable, including 18% who say it would be hugely valuable. Big Data: Extracting value from your digital landfills, AIIM DARK MATTER WASTED 23 % The vast majority of useful data is not used. 23% of the data in the digital universe would be useful if it was tagged and analyzed, but less than 1% is actually analyzed. The Digital Universe in 2020: Big data, bigger digital shadows, biggest growth in the Far East, EMC A growing amount of data is collected and stored by organizations in almost every industry, which is often driven by compliance and/or a mindset that we should keep everything and sort it out later. A report by Gartner, Market Trends: Big Data Opportunities in Vertical Industries, states that organizations see existing, underutilized dark data as one of the most immediate opportunities to transform their businesses. Here s a big data opportunity heat map categorized by industry: Volume of Data Velocity of Data Variety of Data Underutilized Dark Data Hardware Software Banking and Securities Communications, Media and Services Education Government Healthcare Providers Insurance Manufacturing and Natural Resources Retail Transportation Utilities Wholesale Trade POTENTIAL BIG DATA OPPORTUNITY ON EACH DIMENSION IS: Very hot (compared with other industries) Hot Moderate Low Very low (compared with other industries) Service Gartner (July 2012) Market Trends: Big Data Opportunities in Vertical Industries, Gartner TeradataMagazine.com l 13

14 SERVICES Architected for Success Teradata Analytic Architecture Services maximize the business value gained from analytical systems. by David R. Schiller, CCP, and Lance Miller Businesses are faced with decisions regarding the waves of data coming their way. They have valid concerns about how to manage the data and corresponding projects to maximize efficiency and gain optimum business value. What s needed is an architectural approach to effectively manage this data and turn it into valuable, insightful and actionable information. This approach helps align data to business needs, prioritizes projects and adjusts the project scope to address business priorities and pain points in the most effective and leveraged way regardless of the data size. Exploit All Data Teradata addresses the analytic challenges of managing growing volumes and diverse types of data through the Teradata Unified Data Architecture. The systems architecture unifies multiple forms of data and data-oriented technologies into an integrated, cohesive and transparent solution. This allows organizations to leverage the complementary values of the industry-leading Teradata Database, patented Teradata Aster SQL-MapReduce and opensource Apache Hadoop technologies. Different data types can be housed in a manageable environment using the best technology to exploit the data to its fullest potential. The Teradata Analytic Architecture Services enable the building of the architecture to deliver information to the business. These services play a critical role in designing and implementing analytic environments to support data and its related components so business users can get the answers they need, when they need them. 14 l Teradata Magazine l Special Edition l Unified Data Architecture

15 NEW CAPABILITIES VIA TERADATA ANALYTIC ARCHITECTURE SERVICES The Unified Data Architecture makes all information available so organizations can explore new opportunities, fully leverage existing ones, and address the myriad business needs and regulations they face. Teradata Analytic Architecture Services make this vision a reality by working with organizations to determine their business priorities, issues and other data-related needs, and how to handle them in the most efficient, effective manner. Streamlined Architecture The Teradata Analytic Architecture Services handle information through the same architectural principles that apply to all types of data, including multi-structured. This ensures a consistent methodology and approach to help expand or improve the analytic environment. A key benefit of having streamlined, integrated analytic architecture services is the ability to deliver a unified approach based on the business, information, applications and systems supporting the three major components of the Unified Data Architecture : > Data warehousing offers integrated and shared data environments to manage and deliver strategic and operational analytics to the business. > Data discovery provides the analytics to unlock insights from data, which needs to be performed with a technology that has rapid exploration capabilities through a variety of analytic techniques and is accessible by mainstream business analysts. > Data staging enables loading, storing and refining data in preparation for analytics. Other benefits of the streamlined approach include leveraging global best practices, reduced risk, and consistent, efficient and cost-effective project implementations. Business analytical challenges that organizations face include: > Inability to analyze data on a granular level, resulting in a lack of information on actuals, plans and forecasts > Analytic capabilities that do not meet business needs due to duplication of efforts, inconsistent information and an inability to perform what-if scenarios > Difficulty correlating customer satisfaction to labor and staffing > Lack of rigorous data standards, causing confusion and difficulty when comparing historical performance > Data access and security issues New capabilities enabled by Teradata Analytic Architecture Services: > Advanced analytics provides what-if capabilities using historical data and improved forecasts to predict, understand and plan business actions. > Location attributes enable organizations to analyze data and support marketing strategies based on shared common attributes. > Business intelligence dashboard delivers a snapshot of business performance. Supporting Services The Teradata Analytic Architecture Services help set up the systems architecture to meet each organization s unique needs. An infrastructure can be optimized to support each company s analytic requirements based on answers to questions about the business strategy, the data being gathered and analyzed, who needs to consume information and other business functions. The supporting and streamlined services include: > Analytic roadmap, a strategic consulting service, provides a structured framework to determine business priorities, the value of data and data-leveraging capabilities. It puts prioritized projects on a roadmap that shows each project s incremental business value and increased business capabilities. > Opportunity workshop defines the scope of a specific project, creating the conceptual architecture document used to outline facts about the project. > Scoping service establishes the project parameters with an emphasis on ensuring the business requirements are realized in the delivered solution. > Design and delivery service provides a complete, detailed solution architecture for a project. Reach Data s Full Potential Teradata addresses the challenge of growing volumes of diverse data types with the Unified Data Architecture. The architecture places data in a manageable environment and utilizes the best technology to exploit information to its full potential. The Teradata Analytic Architecture Services are the enablers providing this flexible approach to derive maximum business value from all forms of data. T David R. Schiller, CCP, has nearly 30 years of IT experience. He manages Teradata Professional Services marketing programs. Lance Miller manages the Teradata Professional Services/Customer Services marketing group. TeradataMagazine.com l 15

16 The Power of Interaction A discovery platform empowers organizations to capitalize on their analytic prowess. by Randy Lea Over the years data warehousing has moved from transactional analysis and report generation to being operational and mission-critical, actually driving a majority of businesses. With the introduction of big data, organizations now want to know more about interactions for behavioral analysis around customers, products, machines and supply chains. Facilitating this type of analysis calls for a discovery platform that can analyze all data non-relational, multistructured and structured as well as transactional data without requiring extensive data modeling, pre-prep of the data or stringent service level agreements (SLAs). It s not trying to balance the books, or reconcile down to the penny. And as long as the quality of the data is sound, its completeness can be good enough. Furthermore, both the data analyst and the data scientist who know the business and the data can execute an iterative process using multiple analytic types, such as SQL, MapReduce, graph or statistical functions, in conjunction with each other to discover unique insights. When the analysis encompasses a broader set of data including text, machine and sensor data, discovery Discovery Platform Requirements ALL DATA Nonrelational data Multistructured data Structured data OLTP DBMSs DISCOVERY Discovery platform Doesn t require extensive modeling Doesn t balance the books Data completeness can be good enough No stringent SLAs theory advances from transactions to interactions and empowers businesses to execute behavior analysis. Here s just one example of the value derived from analyzing interactions. U.S. healthcare providers receive a quality rating based on a five-point scale. Each point on the scale can be worth millions of dollars in business so negative consumer feedback can be very, very costly. With a discovery platform, it s possible to identify specific behavior that typically leads to a consumer complaint such as issues with office visits, billing mistakes or comments made during their ITERATIVE ANALYSIS ANALYTICS SQL MapReduce Statistical functions Graph Behavioral analytics Customer Product Machine Supply chain USERS Data scientist Data analyst call center interaction. Through a series of steps, analysis can determine when a patient may be headed down the path toward a complaint. Armed with that information, the healthcare provider can put in place appropriate interactions to try to influence a more positive pathway. In today s ultra-competitive environment, knowing all of your organization s interactions can provide a winning edge the discovery platform is your means to sharpen it. T Randy Lea is vice president of the Teradata Aster Center of Innovation for the Americas. 16 l Teradata Magazine l Special Edition l Unified Data Architecture

17 APPLIED SOLUTIONS See the Big Picture Expand your business horizons with fast, easy enterprise access to Hadoop data. by Arlene Zaima A Teradata integrated data warehouse (IDW) gives organizations in every industry a fresh, big-picture view of their business. Now, with the Teradata Unified Data Architecture, that view just got even bigger. The solution integrates opensource Apache Hadoop as part of its analytic data foundation, allowing organizations to store and access massive volumes of data with ease. Companies can expand the scope of their application, business intelligence (BI) and data mining implementations by extracting hidden jewels from the data housed in Hadoop. Those hidden jewels can be incorporated with other data in the secure data warehouse to provide richer, more detailed insights into the business. Access to Hadoop data is enabled by new capabilities within Teradata Studio and Teradata Database Extraction Made Easy Smart loader for Hadoop is a new feature in Teradata Studio that simplifies browsing Hadoop file systems within the new Hadoop View and provides bi-directional data transfer between Hadoop and Teradata systems. The smart loader is composed of wizards within the Hadoop View to automate and simplify Hadoop connection and transfer tasks. The smart loader consists of three elements. The first element, Teradata TeradataMagazine.com l 17

18 FIGURE HADOOP VIEW IN TERADATA STUDIO Hadoop View in Teradata Studio provides a tree browser of the Hadoop tables with a drag-and-drop interface for table transfers, and provides both a Transfer Progress View and a Transfer History View. Studio, runs on the user s laptop or personal computer, which makes a Java database connectivity (JDBC) connection to the second element, the Teradata Database. Once the connection is established, the data move is initiated on the third element, the Hadoop cluster. Teradata Studio executes the Teradata Connector for Hadoop, which is a set of application programming interfaces (APIs) and tools that support high-performance, parallel bi-directional data movement. A drag-and-drop interface in Teradata Studio allows business analysts to extend their self-service capabilities. They no longer have to rely on Hadoop programmers to extract data. Instead, analysts can easily experiment with and explore a combination of Hadoop and warehouse data in their secured and controlled Teradata Data Labs. Pain-Free Data Movement The Hadoop View in Teradata Studio provides a connection management interface to allow users to create, edit and delete profilers describing their Hadoop system. Users enter HCatalog, port and system credentials to establish a connection. Teradata Studio connects to the HCatalog to determine the location of the associated files within the Hadoop Distributed File System (HDFS) and to read the metadata associated with the tables and files. The Hadoop View displays a tree of the Hadoop database, schemas and tables to simplify navigation. Within this interface, users can move tables between the Teradata Database and Hadoop. In addition to the Hadoop View, Teradata Studio also provides the Transfer Progress View and a Transfer History View to monitor and manage current and past data transfers (as referenced in the tabs shown in the figure). Each entry consists of information about the transfer job, including: > Job name > Time stamp for start time > Source and destination > Job status > Duration > Number of rows transferred > Notes > Summary The Wizards of Data Transfer An import wizard supports data transfers between Hadoop and the Teradata Database. The wizard prompts the user for the destination table name and source delimiters. Default column names and types are provided, but can be overridden. As a default, no primary index (NOPI) tables are created to avoid skewing. 18 l Teradata Magazine l Special Edition l Unified Data Architecture

19 APPLIED SOLUTIONS Analysts can maximize the power of the Teradata Database and the simplicity of standard SQL to benefit from a big data storage and staging environment. To import data from Hadoop, Teradata Studio creates the target table in the Teradata Database by interpreting the HCatalog metadata and generating the appropriate CREATE TABLE SQL statement. Once the table is created, Teradata Studio executes the Teradata Connector for Hadoop within the Hadoop cluster. Data is imported directly from Hadoop to the Teradata Database across the Infiniband network for fast, seamless data movement. An export wizard supports data transfers between the Teradata Database and Hadoop. This wizard navigates users through the process, allowing them to change the default Hadoop table name, column names and types, delimiter and job name. Teradata Studio will invoke the Teradata Connector for Hadoop within the Hadoop cluster to export the data. Unlock the Value in Hadoop Data Business users need a simple way to unlock and analyze data stored in Hadoop without employing complex Hadoop MapReduce programming and distributed processing skills. They need technologies that allow a standard, easy-to-use business language like SQL to analyze data that has been captured or refined in the Hadoop environment. They also need the flexibility to use their standard BI and reporting tools against this data. To that end, Teradata SQL-H allows business users to easily leverage the data in Hadoop. Teradata SQL-H is a new query interface to analyze data from both Hadoop and the Teradata Database. It provides standard ANSI SQL access to Hadoop data, allowing applications and analysts to continue using standard interfaces to access external data in Hadoop systems. The solution can deliver unique benefits to businesses. For example, wireless communications service providers can improve the accuracy of customer attrition scores by enriching their data in the warehouse with call center records in unstructured formats, Web log data and other information. This integrated data can help analysts build more effective models with a higher level of precision. The companies can benefit from reduced churn, improved customer satisfaction, and targeted up-sell and cross-sell opportunities. Organizations can also use Teradata SQL-H to improve sales. Analysts track sales of products for key customer segments, and a trend may show growing sales in a segment called others, previously considered outliers to their customer segments. In-database analytic tools allow analysts to drill down into the detailed transactions across multiple channels of the other category to refine customer segmentation using the familiar query language SQL to reduce the learning curve, latency and staffing costs to exploit the trend. Empower Analysts Smart loader for Hadoop and Teradata SQL-H provide business analysts the opportunity to work in a self-service environment within the security of the Teradata Database where sensitive data can be enriched with data from a less secure source. These interfaces allow enterprise users to directly access and analyze vast amounts of Hadoop data without requiring complex programming or an understanding of the interworking of the Hadoop system. These tools let analysts leverage the full set of analytic implementations available to Teradata Database users against Hadoop data. Analysts can maximize the power of the Teradata Database and the simplicity of standard SQL to benefit from a big data storage and staging environment. T Arlene Zaima is a program manager for Teradata Integrated Analytics solutions, including Geospatial and Agile Analytics Data Lab. FREE DOWNLOAD Download your complimentary copy of Teradata Studio at Teradata.com. TeradataMagazine.com l 19

20 10000 Innovation Drive Dayton, OH teradatamagazine.com Unified Data Architecture and SQL-H are trademarks, and SQL MapReduce, Teradata and the Teradata logo are registered trademarks of Teradata Corporation and/or its affiliates in the U.S. and worldwide. Hadoop is a trademark of the Apache Software Foundation. Teradata continually improves products as new technologies and components become available. Teradata, therefore, reserves the right to change specifications without prior notice. All features, functions and operations described herein may not be marketed in all parts of the world. Consult your Teradata representative or Teradata.com for more information. Reproduction in whole or part of any material in this publication without written permission of Teradata Corporation is expressly prohibited. EB Copyright 2013 by Teradata Corporation All Rights Reserved. Produced in USA.