Cloudera Search and the Enterprise Hub

Transcription

1 Cloudera Search and the Enterprise Hub Version: Q

2 Table of Contents Introduction 3 The Cost of Data Silos 4 The Enterprise Data Hub and Search 4 Multi-Workload Search and the EDH 7 Inside Cloudera Search 7 Examples of Cloudera Search and an EDH 10 Conclusion 12 About Cloudera 13 2

3 Introduction Success in today s competitive market demands that enterprises embrace information at every opportunity and become informationdriven in decisions and thought. The enterprise data hub (EDH) has emerged as the transformative data management architecture to help enterprises discover the full potential of their data. An EDH provides a secure and cost-efficient place to store and analyze unlimited business data, from batch processing and interactive SQL to machine learning and stream processing. This architecture empowers enterprises to derive new insights and correlation while extending the value of existing investments. With an EDH, enterprises can leverage a large and unpredictable volume and variety of data economically through scalable and flexible storage and in place processing of data. An EDH provides visibility and technical acumen for a variety of business audiences, without incurring costly data movement, remodeling, or duplication found in conventional data management approaches. Cloudera Search plays an important part in realizing an EDH architecture and strategy and serves a multitude of use cases across the organization. Its native integration with Apache Hadoop and its ecosystem is key to future business innovation and insight. Search opens doors to powerful, yet approachable, aggregation and correlation of structured and unstructured data. Many large enterprises already employ Cloudera Search for their informationdriven objectives and applications. 3

4 Separate silos of data emerge for various business and operational reasons, from corporate acquisitions to distinct retention policies. These silos suffer from the high cost of data duplication and management and have limited data visibility characteristics common to singlepurpose, standalone systems. The Cost of Data Silos For most enterprises, their data management deployments have over time grown into siloed architectures of separate computing capabilities and storage functions. This data management separation has emerged for various reasons. Different teams with a variety of operational and retention policies manage the data, and often enterprise IT teams need to serve data to different audiences with varying degrees of interpretation, format, and access. For example, corporate acquisitions and restructuring can result in heterogeneous infrastructure and multiple data sharing needs. Siloed environments unfortunately come with the cost of maintaining the many copies and forms of data among the various computing capabilities. These costs include the transport of a growing volume of data between infrastructures to order to serve both the existing and new use cases encountered in today s business. Siloed environments also limit analytic access and visibility to only the subset of data that is within the silo, thus can stymie the opportunity to gain more valuable insight by correlating various disparate data sets. For example, a product management team might not recognize which products or services are generating more happy customers because the team is unable to link the unstructured data of their text-based customer surveys with their structured account data. Standalone search solutions are not the exception and constitute one of those silos, representing yet another piece of an enterprise IT teams data synchronization puzzle. Such solutions also demand duplicate cluster management and place additional operational demands on the IT operators and information security teams. Ultimately, these solutions limit business insight because they represent only part of the data, and users must turn to other computing capabilities for the full view. The Enterprise Data Hub and Search One of the core principles of an enterprise data hub is to dissolve the barriers of data silos and grant visibility and use across an unified data environment - where multiple, disparate business workloads execute directly on all data in one centralized location. As mentioned, an EDH brings scalable batch processing, interactive query and exploration, stream processing, and advanced analytics side-by-side within a single, maintainable infrastructure. Many of these computing capabilities serve critical business needs and user communities within and outside the enterprise. Yet, often these capabilities either require significant expertise, like advanced analytics or structured query language (SQL) reporting, or address important but defined scope functions, like end-of-day transaction aggregation, which limits the audiences served. Search is a critical capability for the enterprise due to its easy and intuitive approach to exploration and correlation of all types of data. Search as part of an enterprise data hub represents an important gateway to all the data, regardless of shape, size, and volume, for a broad set of business communities within an organization and has several common business drivers. 4

5 Broad and Rapid Insight Search lets business users get insight from enterprise data that otherwise might not be possible. An EDH, as a common storage substrate for all enterprise data in its original, full fidelity, can handle all types of data, from structured formats like relational data coming from a transactional system to unstructured forms like information found in research notes, analyst reports, and voice recording translations. Search, as part of the EDH, has access to the full scope of this data, and users can employ familiar tools like faceted navigation and full-text queries in their own language to aggregate and correlate across these data sets. Search also permits business analysts and operators to filter data sets of relevance through rapid, iterative queries and refinements, which allows these users to solve their mission critical tasks at the speed of thought. Streamlined Workflows Enterprise users can run search and other workloads, like SQL reporting and batch processing, on shared data sets within an EDH, which avoids the costly data transfer or duplication to purpose-built, siloed computing systems for these capabilities. Similarly, IT operators can readily index and serve data in the same infrastructure as other computing capabilities, which establishes more advanced data workflows. Using search within an EDH lets administrators tap into distributed, fault-tolerant, and high-throughput data processing frameworks both to simplify the scale-out of manual or bulk data preparation and indexing and to serve the immediate results. Expanded Audiences A well-managed enterprise data hub with integrated search opens data to a wider business audience within the organization. For most enterprises, a limited set of developers and analysts have the skills to write custom computing jobs with frameworks like MapReduce, let alone the understanding of how to use SQL and business intelligence (BI) tools to evaluate and investigate data and construct views and reports. The majority of the business communities within an organization are effectively shut out from the wealth of information represented by big data and an EDH because they do not have the skills or tools to access the information effectively. With search, all audiences gain a familiar and interactive method to exploring data and finding answers everyone knows how to search and thus an enterprise can easily bring big data and its value and insights to everyone within the organization. Distributed Computing Search Indexing Batch Processing Interactive SQL Data Data Data Data Shared Data Sets Figure 1: Streamlined Workflows and Distributed Computing in Apache Hadoop 5

6 COMMON USES OF SEARCH Data discovery and metadata search Enterprise search Web crawling and web search Document search Log and event search BI search ecommerce catalogs Social and graph search New media (video, audio, etc.) search search Threat detection analysis Data accuracy/quality assurance Data Exploration and Discovery The introduction of big data into an enterprise data management environment can quickly overwhelm conventional (i.e. relational) approaches and methods for data exploration; the vast majority of big data is unstructured in form and is a poor fit for the schemas of relational technologies. Search lets business users explore and discover data interactively without knowledge of schemas by relying on the intrinsic relationships and intersections of key elements and data points both within a given data set and in combination with other data sets. Business teams, from analysts and data scientists to line-of-business leaders and missioncritical staff, can use search to quickly and easily investigate and understand new data sets, discover correlations in data, and speed up data modeling and data discovery processes. Enterprise teams can offer this powerful capability with entity extraction that is driven by the language and concepts of the business audiences, which results in domain-specific and intuitive search experiences. Teams can also employ simple, yet powerful fuzzy matching functions to discover more accurate and comprehensive data sets. And these capabilities extend beyond historical and static data to real-time data sets, so business users can employ the same processes and bring the same navigation experience across multiple timescales of data. Archival Access Organizations realize that their historical data sets hold significant value and opportunities for their business in trend analysis or anomaly detection, for example but often are thwarted by the cumbersome and time-consuming processes required to restore these data sets from near-line and off-line archives for that analysis and reporting. Moreover, conventional systems might not have the additional capacity to accommodate this data. With a Hadoop-based EDH, enterprise IT teams can establish a highly cost-efficient system for capturing and storing data that not only meets enterprise archival or retention requirements, but also presents the business with opportunities to find value from this data without incurring the significant costs of retrieval, movement, and separate storage for analysis. Search within the EDH offers enterprise teams methods for on-demand processing over arbitrary data volumes at no incremental cost or complexity to the system, which both opens archival data to ad-hoc processing and accelerates midterm validation of results during development. In this capacity, search is an excellent tool for uncovering the unknown or potential value within archival data sets. 6

7 Multi-Workload Search and the EDH Search as a technical capability has broad application, and thus can be divided into several categories and solutions. Examples of search within an enterprise include log and event search, document management search, and web search. IT operators traditionally have realized many of these solutions as distinct, purpose-built systems. Cloudera Search is this broad computing capability realized in a Hadoop-based EDH, yet Cloudera Search is decidedly not a standalone solution. Cloudera Search is an integrated, flexible, and robust search solution that works directly with the other workloads provided by an EDH, like batch processing, interactive SQL, and advanced analytics. Given the depth and breadth of this integration with these other services, enterprise IT teams can easily employ Cloudera Search in tandem with these other capabilities to provide comprehensive solutions for a much wider range of search applications and use cases, all from within an existing data hub. In particular, administrators and developers find Cloudera Search to be a great match for multiple search applications that have overlapping or shared sets of data, like an engineering document repository and the associated product support call center dashboard. Cloudera Search is also well suited for situations where business requirements demand multiple, simultaneous computing capabilities in addition to search to render value and uncover insight from data, for example fraud investigation, advanced threat detection, and anomaly and data mining efforts. For many organizations where these multiple workload scenarios are commonplace, Cloudera Search is an ideal solution, because Cloudera Search shares not only the same data sets, but also the same physical infrastructure and system, resource, and security management frameworks. These features eliminate the typical data duplication and synchronization issues, while streamlining IT operations and maximizing overall data management investments. Inside Cloudera Search Cloudera Search is not strictly a single product; it is a fully open source search solution, built with the feature-rich and extensible Apache Solr project. Apache Solr, in turn, includes other open source projects like Apache Lucene and Apache Tika. The reasons for basing Cloudera Search on Solr are clear Solr has a mature code base with a very active developer community and is broadly adopted throughout many industries and used in many applications, making it the open standard for big data search. One example that demonstrates the strength and depth of innovation within the Solr project is the recent development of SolrCloud, which enables distributed, scalable indexing. SolrCloud was a key contribution that resulted in Cloudera s decision to include Solr as the search engine for its EDH. Solr also provides developers with open APIs that promote extension and innovation, and its flexible architecture offers many options for seamless integration with a broad ecosystem of existing enterprise tools, systems, and data formats. As mentioned, Cloudera Search is a solution that uses and integrates many services and capabilities of the Hadoop ecosystem to enable an EDH. These integrations provide IT teams with several critical capabilities required in building and operating a comprehensive search engine capable of handling the challenges of big data. 7

8 Real-time and Scalable Indexing Cloudera Search has various options for indexing data and thus has the flexibility and agility to handle a multitude of use cases. Cloudera Search works with Apache Flume to offer developers a straightforward and extensible method for streaming data, like log events and social media content, into search for real-time indexing. This approach simultaneously provides immediate data serving via search while capturing and storing the data within Hadoop for further analysis, processing, and archiving. These real-time capabilities also extend to Hadoop s NoSQL solution, Apache HBase; IT operators can send writes and updates in HBase directly to Cloudera Search for immediate indexing and serving with minimal overhead to the HBase service through the use of HBase replication. Enterprise developers building real-time applications with HBase, from ad serving engines to operational data stores, can now easily incorporate interactive search and facet-driven dashboards to their services with minimal impact to performance. While real-time data indexing greatly expands the general data visibility for many applications, Cloudera Search also provides the enterprise developer with traditional indexing tools built with the ubiquitous Hadoop batch processing framework, MapReduce. IT teams can employ this proven, high-throughput framework to enable linearly scalable indexing of any data within an EDH. This arrangement also lets end users submit on-demand, ad-hoc indexing jobs to explore data sets not currently included in existing search indices. As these options use Hadoop s common processing model, Cloudera Search s indexing activities share the consolidated infrastructure, resource, security, and system management with the other services within an EDH, which streamlines operations for administrators and the execution model for end users. Simplified Data Storage Cloudera Search stores its data and indices in the common data substrate of Hadoop (i.e. Hadoop Distributed File System or HDFS). This critical integration lets Cloudera Search benefit from the same unified management, control, and robustness that comes out of the box with Hadoop s native file storage, such as disk-to-cluster fault tolerance and data block replication as well as perpetual, full fidelity data. This integration also offers significant cost efficiencies for index storage, as indices are just data within HDFS, and thus avoids typical per-document and per-volume pricing models common to standalone search solutions. In addition, Cloudera Search takes advantage of the collocation of its data and indexes with its query and indexing services. Collocation eliminates the error-prone and expensive data duplication and data pipelines required to move data to and from separately managed storage systems for indexing and serving. IT administrators can also optimize and streamline data workflows involving shared services, including search, due to the collocation of data with the multiple computing services; within a Hadoop-based EDH, Cloudera Search is one of many services using the same data and resources within the cluster. Familiar Search Experience Cloudera Hue, an out-of-the-box user experience for Hadoop, includes a Cloudera Search GUI and GUI builder. The GUI is built on the standard Solr APIs and enables users to search data interactively, view result files, and perform faceted exploration. The availability of the GUI means that business users and developers can perform searches alongside all the other workloads and capabilities in Hadoop through the same, centralized application. The GUI builder allows non-it developers to construct meaningful and informative views of searchable data sets, from configuring the individual result fields and facet values to constructing the guided and visual navigation features. 8

9 Figure 2: Hue Search GUI Production Management and Visibility Cloudera Search is fully integrated with Cloudera Manager, so IT operators can simplify the deployment, configuration, and monitoring of all EDH services, including search, from a single entry point for administration. They can gain deep insight into metrics like service utilization, monitor system health, examine performance and system trends, and facilitate search index management and resource control while balancing the needs of different workloads, like batch processing and interactive SQL, on the cluster. Access Control Enterprise IT teams can extend their existing authentication systems and authorization policies to Cloudera Search. The authentication framework of Cloudera Search employs Kerberos for both intra-service and client-to-service communication to ensure IT operators have comprehensive and centrally managed authentication for all channels throughout their entire data management environment. Cloudera Search also integrates with Apache Sentry (incubating) for granular document-level authorization to individual indices. By consolidating Search authorization within Sentry, IT operators have a centralized system for managing authorization throughout the EDH data access workloads, like batch (i.e. Hive) and interactive SQL (i.e. Impala) in addition to search. Multi-Format Support One of the strengths of Apache Solr is its support for a wide and growing range of data formats. Cloudera Search, with its Solr integration, supports many standard file formats for indexing, e.g. XML, JSON, HTML, Microsoft Word, PDF, JPG, , audio, video, and others, by using supporting libraries like Apache Tika, which is part to the Solr project family. These features greatly ease data ingestion and open Cloudera Search to many existing enterprise search workloads. In addition, Cloudera Search has native support for Hadoop-optimized file formats and compression codecs, such as Snappy, Sequence Files, and Avro. By employing these open formats, IT developers can seamlessly use data sets between the various computing capabilities within an EDH, and data stewards can ensure easy reuse across these same capabilities, including search. 9

10 Simplified Data Processing Cloudera Search includes a collection of integrated tools and frameworks, one of which is Cloudera Morphlines, a Kite SDK library that provides common command-based data transformations. With Morphlines, search developers can simplify index configuration and replace custom Java programming with simple command calls to prepare and transform data, for example parsing, configuring, and indexing MBox files for search. These commands are both extensible and inclusive and can be used outside of search with other computing workloads in an EDH, thus promoting shared development and operations. Examples of Cloudera Search and an EDH As illustrated in the previous sections, Cloudera Search is a key capability of an EDH and addresses many business and technical challenges facing organizations across industries. The following examples highlight some of the uses where Cloudera Search accelerates an organization s time-to-action and time-to-discovery with its data and expedites an enterprise data hub strategy. Real-time Event Search and Advanced Pattern Extraction Many organizations recognize the strategic and tactical advantages of combining search and other workloads, like interactive SQL, simultaneously on the same data sets. For instance, IT operations and information security teams typically need quick drill-down and real time correlation capabilities for varied, multi-type log data to support rapid inspection and tactical exploration of current alerts and anomalies. This same log data, however, can also service the advanced analytics efforts of these teams, tackling subjects like capacity planning, long-term reporting, and historical trending. SLA-Bound search Nightly Batch Processing Real-time Data Real-time Data Real-time Data Real-time Data Calculated Baselines Figure 3: Search for Baseline and Real-time Analytics To meet these challenges, IT teams rely on an EDH as the foundation for advanced deployments like automatic anomaly detection. In these cases, developers match incoming data against nightly batch-calculated distributions built from the same data pool. While an enterprise IT team could optimize a standalone search solution to provide real time data for their mission-critical staff, they are faced with significant cost inefficiencies when scaling to larger volumes of data. Moreover, this conventional approach mandates that an IT team move or duplicate data to expose the same information to the other additional processing frameworks. An EDH answers multiple technical needs for the same data cost-efficiently and at scale and Cloudera Search serves as the primary delivery engine for SLA-bound audiences to this complex mix of baseline and real-time analytics. 10

11 360 Degree View Another common challenge for the information-driven organization is how to gain a better view and understanding of a particular subject or topic, like a patient, a client, or an order. While conceptually simple and straightforward, the details are often much less so, as a business typically needs to correlate numerous and diverse types of data that have limited or unclear intersection linking market surveys, support tickets, and s with sales trends, transactions, and usage logs. Yet successful correlation of these data points grants enterprises more insight into what makes a happy customer or an efficient supply chain. For example, customer support teams use the extraction and correlation across data sets within an EDH to discover how a purchased product and its associated resource and support interaction correlate with customer satisfaction and support ticket load. This mix of activity creates more accurate, actionable, and repeatable recipes for customer success. Cloudera Search is the primary application for this effort, both for interactive exploration of the various data sets during the design of the processing pipeline and for the in-process introspection of the intermediate results. This same search service and resources still serve the traditional applications and audiences the marketing, support, and sales staff with the original data sets, yet without needing additional search systems. Data Exploration Many forward-thinking organizations turn to social media for insights and answers to get a head start over the competition. For marketing firms, investment concerns, and retailers, the analysis of Twitter, Facebook, and other external data sources can yield new visibility into market segments. Cloudera Search aids the exploration of these new forms of information by allowing richer and faster modeling of the original, full fidelity data with clustering, faceted search, and fuzzy matching. These approaches allow analysts and end users the freedom to explore the information as needed, but with limited IT intervention or reliance on complex SQL queries. An EDH readily and efficiently stores all this variable, unstructured, yet highly valuable data due to the underlying scale out and schema-on-read characteristics of HDFS. This data is correlated with other data within the EDH; an EDH centralizes and accelerates not only the usual processing, categorization, and reporting, but also other advanced analytics, such as sentiment analysis, in this shared environment and across all of its data, unstructured or otherwise. This approach is most pervasive in trend analysis as well as impact analysis, as business users typically uncover better, more comprehensive results when analyzing multiple sources together and at significant depth and history. More innovative uses of this broad and deep analysis include risk and disaster prevention. Process Validation Organizations across many industries are turning to non-text information, like video, audio, and images, as another source of valuable data. In these cases, business teams need to improve the information received through or related to this new media, and IT teams typically seek ways to streamline both the metadata processing and decorating of these files, as well as to serve the combined data-metadata results. Some examples of the value of this metadata include the geographical or weather conditions at the time of capture of an image or video, the diagnostics reports and logs of the equipment generating the sensor reading, or downstream, post-processing details about the source content, like advanced image or voice processing. 11

12 All these scenarios benefit from an EDH, which combines a highly scalable processing platform well suited to metadata treatments with parallelized advanced processing. Cloudera Search, as one of the shared computing capabilities of an EDH, provides a critical tool for IT developers to explore in-flight and intermediate processing results. With search, developers and users can realize and validate procedure or algorithm results or identify errors and flaws at early stages, thus accelerating overall time-to-value. Cloudera Search also lets IT teams serve the final results of the processing to multiple end audiences with free text and faceted navigation, yet with no incremental cost or additional infrastructure for this function. Similarity Discovery Business analysts face a common problem often, they will have an important data element, yet need more data points to continue analysis, for example additional documents that share topics and themes with the source document or other genomes that are 80% similar to the subject s sequence. Finding these related data points could be complex and time consuming. Business users can accelerate their time-to-discovery by using an EDH as a very large data storage solution and executing fuzzy search or similarity searches across that entire data set with Cloudera Search. This approach significantly reduces the effort spent in manual introspection, particularly across unstructured, yet important, data sets. With the relevant data sets identified, analysts can then immediately process the information further with other workloads, without moving, copying, or regenerating data. The shared data and resources of a Hadoop-based EDH, coupled with the native and rich integration of Cloudera Search, gives these users analytical depth and flexibility with familiar and cost-effective tools. Conclusion Cloudera Search plays an important role in an organization s EDH strategy, serving a multitude of individual and multi-workload use cases. Its rich and unique native integration with the foundations of Hadoop and its ecosystem presents IT administrators and business users alike with opportunities for more streamlined workloads across larger and more diverse data sets. Cloudera Search and an EDH empower enterprise teams to breaking out of separate, disconnected data silos and discover bigger insights across all types of data, structured and unstructured. With Cloudera Search, enterprises enjoy the power, flexibility, and ease of use of a proven and unmatched search solution for their enterprise data hub. 12

13 About Cloudera Cloudera is revolutionizing enterprise data management by offering the first unified Platform for Big Data, an enterprise data hub built on Apache Hadoop. Cloudera offers enterprises one place to store, access, process, secure, and analyze all their data, empowering them to extend the value of existing investments while enabling fundamental new ways to derive value from their data. Cloudera s open source Big Data platform is the most widely adopted in the world, and Cloudera is the most prolific contributor to the open source Hadoop ecosystem. As the leading educator of Hadoop professionals, Cloudera has trained over 22,000 individuals worldwide. Over 1,200 partners and a seasoned professional services team help deliver greater time to value. Finally, only Cloudera provides proactive and predictive support to run an enterprise data hub with confidence. Leading organizations in every industry plus top public sector organizations globally run Cloudera in production. cloudera.com or Cloudera, Inc Page Mill Road, Palo Alto, CA 94304, USA 2015 Cloudera, Inc. All rights reserved. Cloudera and the Cloudera logo are trademarks or registered trademarks of Cloudera Inc. in the USA and other countries. All other trademarks are the property of their respective companies. Information is subject to change without notice.