MarkLogic Enterprise Data Layer

Transcription

1 MarkLogic Enterprise Data Layer MarkLogic Enterprise Data Layer MarkLogic Enterprise Data Layer September 2011 September 2011 September 2011

2 Table of Contents Executive Summary... 3 An Enterprise Data Layer... 4 MarkLogic-Enabled Enterprise Data Layer... 6 About MarkLogic Server... 8 Benefits of MarkLogic in EDL... 8 MarkLogic Server Capabilities...10 About MarkLogic Corporation...13

3 Executive Summary Enterprise Data Layers (EDLs) solve the problem of information scattered across silos, resulting in compounded challenges around ownership, control, and exploitation. It is an infrastructure that aggregates multiple sources of information to provide a set of services to aid in information management, provisioning of tools, and secure information sharing. Organizations deploy EDLs to greatly reduce implementation time and costs associated with developing new information products and services. MarkLogic Server is the right foundation for an enterprise data layer, combining best-of-breed technology across database management systems, application servers, and search engines. Some key benefits of MarkLogic in an EDL infrastructure include improved agility, higher performance and scalability, and lower total cost of ownership over systems based on traditional technologies such as relational databases. MarkLogic Server is architected to efficiently handle the types of information found in an EDL environment. MarkLogic Corporation has a proven track record of delivering solutions that help government organizations gain mission advantage and achieve mission success. Our vision is to give every organization the ability to make the most of their unstructured information and Big Data. 3 3

4 An Enterprise Data Layer Why an Enterprise Data Layer (EDL)? Too often agency processes result in critical information being scattered across silos, frequently in proprietary formats. The information ownership contract is broken, it is difficult for the agency to own or control the data, and it is sequestered in user-facing tools and managed by multiple system providers. An EDL can help resolve these issues by giving control of mission-critical information back to the data owner, making information manageable, portable, and easier to share. What is an Enterprise Data Layer? An EDL is an infrastructure that aggregates multiple sources of varying information into a consistent, welldefined view. It supports the management of mission-critical information, automatic provisioning of analyst tools, and secure information sharing. It is a paradigm shift from current integration approaches, placing mission critical information at the center of the IT portfolio. All capabilities and tools are generated, securely provisioned, and managed from the attributes defined in metadata. An EDL includes communities of interest (COIs) portals, information sharing environments, legacy integration, federated metadata catalogs, databases, and enterprise services. What Does it Do? An EDL supports secure information sharing by providing users with the ability to discover and request information through a collection of flexible, reliable, high-performance web services and web applications. It facilitates information access and sharing for the enterprise and for other systems that collect, process, exploit, analyze, and disseminate information and its associated metadata. As information evolves and new sources of information emerge, an EDL adapts to expose new access methods quickly, efficiently, and without compromising existing information-sharing relationships. Legacy applications integrate with the EDL using the standards-based services for ingest, search, federation, and alerting. What Are the Characteristics of an Enterprise Data Layer? Metadata defines mission-critical information and dictates how data is updated, searched, shared, secured, and managed Standards-based services are automatically generated allowing ingesting, updating, searching, federating, and securing data The EDL is entirely service-enabled and extends existing SOA environments by providing data services New data sources are quickly exposed and data sharing partners integrated, in days not months Queries return in sub-second time, combining multiple metadata types, including geospatial, temporal, full text, and structural, all at once Security is enhanced, ensuring appropriate access to data based on policies and metadata-defined properties Commodity hardware and virtualization environments are fully supported 4 4

5 What Are its Advantages? An EDL greatly reduces implementation time and costs associated with developing new information products and services. An EDL is purpose-built to accommodate a wide variety of information types, expose access to information through a diverse collection of access methods, and evolve in an agile manner as the needs of the enterprise change over time. An EDL serves as a central point for integrating new sources, storing mission critical information and providing secure information dissemination. In comparison, traditional systems with relatively terse structured or semi-structured field data support only limited discovery, and such systems are inflexible when confronted with new types of information or new access requirements (Figure 1). An EDL escapes these bounds, enabling discovery over full textual content, fielded data, geospatial and semi-structured information, and document attachments, with equal agility. Furthermore, an EDL can provide both automated and manual approval mechanisms for limited and complete release of requested information. Figure 1. Without an EDL, discovery is limited or expensive due to custom integrations. Systems commonly used today are limited to statically exposing data according to its fixed structure or schema. By contrast, an EDL sits at the center of information flows (Figure 2), which it dynamically manages using metadata properties and policies. Having a central point of control greatly reduces development efforts, breaks down the information silos, and provides secure access while facilitating data sharing. The ability to query and fuse disparate information from an EDL dramatically increases the speed with which new products and services can be delivered. An EDL streamlines the management, creation, and dissemination of information. 5 5

6 Figure 2. An EDL provides a central point of control to facilitate future application development. MarkLogic-Enabled Enterprise Data Layer MarkLogic Server is the right foundation for an EDL, combining best-of-breed technology across database management systems, application servers, and search engines. This combination provides unprecedented ability to scale and accommodate Big Data, including agility with varied forms of content, data, and metadata. That agility is most directly reflected in the ease of tailoring MarkLogic to new data and analytic requirements. Additionally, as a services-enabled technology, MarkLogic Server easily integrates into and complements SOA environments, being readily accessible via web services, including XML-RPC, REST, and SOAP. This also provides the capability to quickly deploy feature-rich web services within the existing service environment. Conceptual Architecture The conceptual architecture of the EDL is illustrated in Figure 3. The functionality is delivered across the following components: Information Domain. A community of interest (COI) is commonly defined as a group of users collaborating on a particular subject or corpus of information. Typically, COIs require some combination of data fusion and collaboration applications. The EDL extends this concept by providing automatic mechanisms to create, provision, manage, and secure the information domains. Information domains are automatically created, provisioned, and deployed from the domain definitions contained in the metadata catalog. Each domain provides: 6 6

7 o A data fusion and collaboration portal allowing a COI to search, edit, collaborate, and disseminate information associated with a corpus of information defined in the metadata repository. o Web services to ingest, search, and retrieve information. o The ability to participate in federated search with other information domains. o A secure environment based on the access permissions defined in the metadata catalog. o Access to information from legacy data store holdings. Data is accessed, transformed, or ingested to provide higher-level data services. o A virtualized storage container and database to store the information domain data. The container is provisioned and managed automatically. o An export tool to migrate and relocate information domains with their associated data to other environments. Additionally, all information contained in a domain can be exported in industry standard and vendor-agnostic formats. Metadata Catalog and Semantic Model. As described above, the EDL extends the functionality of a metadata catalog. The metadata catalog not only describes the enterprise information and where to find it, but also defines how it is secured and automatically managed. It holds the policies and parameters, allowing the provisioning of the information domains and all SOA data services. A semantic model provides common ground for federation and defines entities and their relationships. The metadata catalog and semantic model make the EDL agile and future-proof. They enable the extension of data schemas by only editing metadata. Metadata is used to capture user activities and operations on the enterprise data. This approach allows agencies to minimize rework, track individual contributions, and minimize the number of copies of the same dataset. Data Services SOA Integration. Web services are automatically provisioned and secured from policies defined in the metadata catalog. Services include search, ingest, alerts, and federation. This feature enables dynamic integration with the enterprise SOA and support for various federated data sharing initiatives. Automatic transformation services provide decoupling between internal and external schemas. Enterprise Services. The enterprise services layer provides common administrative tools, auditing, and logging. Unified security allows secure access to information based on attributes (ABAC) associated with external and internal users. 7 7

8 Figure 3: Conceptual architecture. About MarkLogic Server MarkLogic Server is a proven technology for intelligence systems that aggregate, exploit, and deliver large volumes of heterogeneous information. MarkLogic provides flexibility, performance, and scalability to power mission-critical systems. This includes extensive deployments in U.S. intelligence agencies, other government organizations, and commercial customers, demonstrating its viability as a solution for today s information challenges. Agencies face lower risk with new implementations based on MarkLogic Server as demonstrated by extensive, successful government deployments. As the core technology for an EDL, MarkLogic easily integrates into existing infrastructures, so it can be leveraged to enhance specific tasks without throwing out prior work. Benefits of MarkLogic in EDL Agility Agencies need to adapt as new requirements arise, and MarkLogic Server is easily tailored to handle new data types and analytic requirements (Figure 4). Support for standards such as XML and XQuery lets development teams leverage technologies designed to manage all information types: structured, semi-structured, and unstructured. Indeed, an EDL based on MarkLogic Server can help agencies to extend their existing systems investments. 8 8

9 Figure 4. The MarkLogic advantage increases as more inputs and outputs are required in the system. Performance and Scalability Customers running MarkLogic Server realize significant hardware cost savings as a result of higher performance. Optimized for low-cost, commodity hardware, MarkLogic Server s faster search and retrieval reduces hardware requirements to lower the total cost of ownership. In addition, the internal architecture is designed for environments similar to EDL infrastructures growing number of data sources, various flexible and rigid schemas, a mix of structured and unstructured data, etc. which also contributes to its performance advantage. When compared to a separate RDBMS and search engine infrastructure, MarkLogic Server enables a significant reduction in the required hardware servers due to much higher per-server performance. Figure 5. MarkLogic s superior performance and scalability in a customer performance test. 9 9

10 It can be difficult to predict future information requirements: how much information will need to be stored and accessed, the number of users who may need to access information, or the complexity of the queries users may want to run. One way to easily scale is to design a system where commodity hardware can easily be added and data evenly distributed without requiring a complex clustering framework. This type of clustered architecture is called a shared-nothing architecture, and MarkLogic Server was designed for deployment using this method. The goal of this architecture is to provide scalability by easily adding nodes to handle increases in query requests and information volumes (Figure 5). Total Cost of Ownership Major, mission-critical MarkLogic Server production implementations have demonstrated dramatic, quantifiable benefits. Organizations realize cost savings by replacing the high-end hardware required by traditional technologies with the commodity servers for which MarkLogic Server is optimized. Additionally, MarkLogic Server s architecture requires far less administrative overhead, freeing up additional human resources for mission-relevant tasks. To give an example of a customer who achieved lower TCO, a large federal government project with nearly 100 hardware servers, billions of documents, and hundreds of terabytes of information requires less than one full-time employee for database administration on MarkLogic Server. Ten or more database administrators could be required for a similar system based on a relational DBMS, leading to a prohibitive operation and maintenance cost. MarkLogic Server Capabilities Unstructured and Semi-Structured Information Most of the information created and managed by an organization is unstructured or semi-structured. It is critical for an EDL to be built on a database supporting unstructured information. MarkLogic Server was designed and built to offer unparalleled ability to store, search, and disseminate unstructured or semi structured information. It can index and search a large variety of documents from text to Microsoft Office formats and PDF. XML Centric and Schema-Agnostic MarkLogic is XML centric and schema-agnostic, making it the ideal tool to build an enterprise data layer, offering all the capabilities and tools to efficiently ingest, search, and store metadata and other XML documents. Ingesting and searching documents regardless of their schemas provide the necessary flexibility to manage evolving information. Rapid Information Loading MarkLogic Server is schema-agnostic, meaning it loads information as is without requiring a fixed schema. When comparing this to the requirements of RDBMS in which time-consuming data modeling, planning, and reformatting are required to fit information into defined structures, one can see significant administrative and development overhead is avoided with MarkLogic Server

11 ACID Transactions As required by an enterprise DBMS, MarkLogic Server implements a transactional system that adheres to the ACID (atomicity, consistency, isolation, durability) model to guarantee accurate answers to queries. MarkLogic Server also supports non-blocking read operations which promote efficiency by ensuring users will never have to wait on the read/write operations of others to guarantee returning the correct answer. High Availability Another key component of an enterprise DBMS is high availability, which assures users continuous access to information. MarkLogic Server ensures high availability through several mechanisms, including journaling, clustering, local and remote replication, and automatic failover. High Scalability with Shared-Nothing Architecture MarkLogic Server s shared-nothing architecture enables easy addition of commodity hardware to a cluster to scale across more information and/or more users. MarkLogic currently has customers with hundred-terabyte databases in production. Backup and Restore MarkLogic Server provides a facility for consistent database backup that allows administrators to consider their unique requirements when determining how much or how little they want to backup or restore. They have the flexibility to backup selected components of the database or the entire database, giving them the control needed to meet the specific recovery needs for each deployment. Secure Web-Based Administration MarkLogic Server features an easy-to-use, web-based administration interface to make system administration easy and straightforward. Administrators can complete a wide range of tasks, including changing configuration, managing databases, and managing users. Universal Indexes MarkLogic Server provides universal indexes which capture multiple data elements including full text, markup structure, structured data, and geospatial data. The broad capabilities of these indexes enable full exploitation of aggregated information. The universal index is what drives an array of productivity and analytic capabilities: Advanced Search Capabilities MarkLogic Server provides the search capabilities found in most enterprise search engines today, including advanced features such as faceted navigation, auto-suggestion, relevance tuning, and advanced international language processing

12 Real-Time Search MarkLogic Server also supports real-time search, which ensures information can be searched immediately as it is ingested, without indexing delay or latency. It eliminates the latency typically found in applications that rely on separate systems to handle storage and search. Real-Time Alerting Real-time alerting is the search technology for users who need information pushed to them as soon as it is available. Real-time alerting takes MarkLogic Server s extensive query capabilities and continuously applies them to an incoming feed of information. Millions of alerts can be created to enable a variety of immediate actions on the matching information, including delivery, categorization, enrichment, and transformation. Since the alerts are optimized for this type of search, they are far more efficient than mechanisms in systems that periodically run standing queries. Geospatial Index and Search MarkLogic Server enables organizations to identify geographical relevance in their information with geospatial search capabilities, usually combined with full-text search. MarkLogic Server supports a comprehensive set of query criteria including point search, radius search, latitude-longitude box search, and polygon search in which thousands of boundary points can be specified for polygons. XML Structure-Aware MarkLogic Server is XML structure-aware, so it can search information within XML elements, attributes, and values, as well as combinations of each. In addition, it can further enrich content with inline markup, thus enabling more precise search and analytics. Query Optimization Because database and text-type indexes are in the same software kernel, MarkLogic can provide best-of-both response, typically returning responses for complex queries in well under a second. International Language Support Since MarkLogic Server stores information internally in UTF-8, over 270 languages are supported. Advanced language processing is available for fourteen languages

13 About MarkLogic Corporation MarkLogic empowers organizations to make high stakes decisions on Big Data in real time. Customers trust MarkLogic for mission critical applications that drive revenue and growth through Big Data Analytics enabled by MarkLogic products, services, and partners. MarkLogic is a fast growing enterprise software company that has been providing solutions to the public sector and Global 1000 for nearly a decade. Operating at petabyte scale, MarkLogic Server is a next generation database for unstructured information that allows customers to outflank their competition by consistently getting to better decisions faster. MarkLogic is headquartered in Silicon Valley with field offices in Austin, Boston, Frankfurt, London, Tokyo, New York, and Washington D.C. For more information, please visit MarkLogic Corporation UK One Kingdom Street Paddington sales@marklogic.com Central London W DB Headquarters 999 Skyway Road, Suite 200 San Carlos, CA