Virtual Operational Data Store (VODS) A Syncordant White Paper

Transcription

1 Virtual Operational Data Store (VODS) A Syncordant White Paper

2 Table of Contents Executive Summary... 3 What is an Operational Data Store?... 5 Differences between Operational Data Stores and Data Warehouses... 5 Historical vs. Operational... 5 Historical Record vs. System of Record... 5 Transactional Integrity... 6 Query Profiles... 6 Applications of Operational Data Stores... 6 Operational Data Stores Traditional Approach... 7 Diagram: A Traditional Operational Data Store Environment... 8 Issues with Traditional Operational Data Stores... 9 Unnecessary Data Movement... 9 Proliferation of Pre-Defined Interfaces... 9 ETL Tools are Overkill... 9 Real-time Network Requirements are Too Demanding... 9 Transaction Integrity is Difficult to Manage... 9 Why Virtual? What is VODS? Syncordant s VODS Architecture Benefits of Syncordant s VODS No Data Replication Time Savings Trading Partner Data Exchange Exception Alerts Minimal Network Bandwidth Flexibility for New Data Sources No Pre-Defined Message Routes or Interfaces Access to Data over WAN and Internet Unlimited Scalability Conclusion About Syncordant Data Exchange About Syncordant Syncordant phone: sales@syncordant.com web: PAGE 2 of 14

3 Executive Summary Organizations depend on integrated corporate information, and now there is a new method that significantly reduces the time and expense of data exchange the Virtual Operational Data Store (VODS). Syncordant enables enterprises to assemble Virtual Operational Data Stores (VODS), at total costs of ownership that are far less than the costs of any effective alternatives. Virtual Operational Data Stores enable enterprises to quickly and economically overcome one their greatest challenges; the inability to readily access detailed data about operational changes that affect demand, supply, production and finance throughout their information ecosystems. Syncordant does for consumers of structured data what Internet search engines do for consumers of unstructured content. It seamlessly integrates sources of structured data across extended demand, production and supply and finance networks and information ecosystems. It enables information consumers to invoke ad hoc or persistent queries over any network, from any source that generates SQL queries. Queries are routed dynamically only to those data sources where relevant data is stored. Composite result sets are returned to users in Internet time and presented by the user interface, in the format required by the user. Enterprises can use Syncordant to get real-time visibility into all of the physical and logical events taking place in their extended demand, production, supply and finance networks. The purpose is to support departmental and enterprise reporting, business intelligence, analytics, and any sense-and-respond business processes. The need to share information among lines of business, business partners and customers has never been greater. In order to compete in today s complex business environment, operators of global retail supply-chains, business-to-business trading partner networks or financial service companies require ready access to accurate, composite views of all pertinent data throughout their information ecosystems. Operational information typically resides in many disparate systems throughout these ecosystems. Syncordant phone: sales@syncordant.com web: PAGE 3 of 14

4 Applications and databases are often designed to serve specific functional needs within an organization, but not to integrate and share their information. The Operational Data Store brings this disparate data together in a single resource. The Operational Data Store concept grew out of the need to provide both operational analytics and right-time access to relevant data across these heterogeneous systems. Data Warehousing emerged in the 1990 s as the most common means for enterprises to capture, manage and present the data they need to perform strategic planning and business analyses. As users demanded lower data latency and greater operational relevance of data, Operational Data Stores were conceived and deployed widely for a whole range of business problems that require more current information. Today, Operational Data Stores often are just as prevalent as Data Warehouses in corporate environments. Growing requirements for right time access to operational data add complexity and cost to an already expensive challenges involved in deploying traditional Data Warehouses. In addition to analytical processing requirements for operational reporting, Operational Data Stores often must have instant access to operational data all across geographically dispersed data source topologies and information ecosystems. With current technologies based on data warehousing architectures, point-to-point replication methods are required to move the current data to centrallymaintained physical Operational Data Stores. This necessitates pre-defining and hard coding message routes, developing custom interfaces, data integrity issues, capacity planning requirements, availability concerns, and everincreasing data elements and volumes. Syncordant s VODS combines networking and database technologies to provide timely and efficient access to information a single view of all pertinent information straight from the multiple discreet systems in which the data may reside. Syncordant provides a much faster and more cost-effective way to satisfy the objectives of Operational Data Stores. Rather than copying data to central repositories, Syncordant leverages a network based approach to create virtual operational data stores (VODS). A VODS enables applications to query operational data from where it resides, in the underlying operational applications, without the complexities of moving the data to central repositories. A VODS appears to information consumers as a single database that can be accessed through standard database interfaces from any technology that is capable of rendering standard SQL queries. The great advantage to deploying a network-based VODS is that it provides dramatic reductions in implementation time, costs and risks. VODS implementations are often measured in days or weeks, at cost savings up to 90% compared to building a centralized ODS. Syncordant phone: sales@syncordant.com web: PAGE 4 of 14

5 What is an Operational Data Store? ODSs integrate an organization s operational information in order to facilitate day-to-day operations, support tactical queries and analyze targeted business questions. Operational Data is defined as subject-oriented, integrated views of detailed data that may reside multiple operational information sources. Whereas Data Warehouses emerged for analytic processing for historical and trend analyses, Operational Data Stores are designed to address operational analytic requirements for knowledge workers to resolve real-time business issues. Differences between Operational Data Stores and Data Warehouses Operational Data Stores and Data Warehouses share many technology requirements, especially as those related to analytical processing. However, this is often where the similarities end. The following sections illustrate some of the differences. Historical vs. Operational Data Warehouses typically are intended to keep historical records of past transactions in order to do trend analyses and generate reports based on pre-determined metrics. Trend analyses compare one time period to others; so it is critical to have summary data that returns the same results every time the same question is asked. The inherent value of an ODS hinges on real-time representations of a business and its data. An ODS is different, in that analyses are geared towards understanding what is happening in the business currently, so timely corrective actions can be taken if they are required. Therefore, understanding what the current situation is of a particular shipment, sales order or fraudulent financial transaction is important, and the answer to that question will be different 30 minutes from now. Historical Record vs. System of Record Data Warehouses are intended to store historical records and aggregated reports based on a time slices in the past. The purpose of an ODS is to be the operational system of record, representing the business and data as closely as possible to real-time. In this sense, the tolerance for data latency in ODS environments is very small and sometimes nil. Data Warehouses typically are appropriate where data latency is acceptable in the weeks or months timeframes, since those are the time dimensions used for analyses. Syncordant phone: sales@syncordant.com web: PAGE 5 of 14

6 Transactional Integrity In Data Warehouse environments, data typically is loaded via bulk feeds that occur during low points in transaction volumes (e.g., during off-hour or overnight extraction cycles). For this reason, there is no need for transactional integrity at the Data Warehouse. An ODS often represents the system of record for a potentially wide range of business data; so synchronization and transactional integrity in an ODS environment are critical. Query Profiles Data Warehouses typically are intended to process analytics over complete sets of data, thereby providing aggregate and summary data for reports. Operational Data Stores typically provide some level of aggregation, though not nearly on the same scale as Data Warehouses. In fact, ODS queries frequently access only very small subsets of the data stored therein. Typically, a small fraction of the data loaded into an ODS is needed to service any specific query. The problem is that it is not possible to know in advance the specific data set that is required to service any specific query. So, any standard ODS must store all of the transactional data that s required to service all possible relevant queries. Applications of Operational Data Stores Traditional data exchange approaches depend on centralized coordination and administration that can be cumbersome and inefficient. A wide variety of applications leverage Operational Data Stores to fulfill their requirements. These applications require access to real-time, near real-time, or right-time operational data from a multiple, disparate operational systems to perform simplified analytics across relevant data sets. It is critical to have the data available at the time of specific events, particularly those that can materially affect an enterprise s operations, so analytics can be performed and remedy actions can be identified and executed intelligently. One example is in the application of fraud detection. Fraud detection methods may require access to customer buying patterns over large numbers of points of sale to detect fraud potential probabilistically. Fraud detection applications are useful only when fraudulent transactions are occurring. The usefulness of the intelligence provided by an Operational Data Store is greatest at the moment of purchase. Other applications of Operational Data Stores include real-time inventory visibility, marketing programs, production exception handling, customer account balances, operations load balancing, financial reconciliation and the like. The potential applications are wide and varied. Syncordant phone: sales@syncordant.com web: PAGE 6 of 14

7 Operational Data Stores Traditional Approach The most common Operational Data Store architectures are based on relational databases. Like all relational database applications, the challenges to implementing Operational Data Stores include significant initial investments in time and capital to incorporate new data sources. Traditional Operational Data Store approaches often require significant initial investments in development time and capital. In fact, the costs of designing and building an operational data store are sometimes prohibitive for many applications. Extraction, cleansing, transformation, and loading of data from multiple sources into Operational Data Stores are very expensive processes that require long and arduous development cycles. In addition, the trickle feed mechanisms that constantly send transactional updates to an ODS require custom coding and a robust network messaging infrastructures to handle the large loads of transaction information, further complicating ODS deployments. Moreover, Operational Data Stores are built to solve specific sets of problems, yet the analytical requirements placed upon them are everchanging. Every new data source that needs to be integrated into Operational Data Store requires custom-designed extraction, cleansing, transformation, and loading operations to support the new source. Many enterprises have deployed Operational Data Stores only to find them inflexible in incorporating additional data or data sources into the ODS data context. Syncordant phone: sales@syncordant.com web: PAGE 7 of 14

8 A Traditional Operational Data Store Environment Appl Data Appl Data Legacy ETL Bulk Feeds Ad Hoc Query, Reporting & Analysis Appl Data Trickle Feeds Operational Data Store Application Systems Appl Data Appl Data Appl Data Data Delivery to other Systems The diagram above depicts how Operational Data Stores typically are deployed today, with support for both trickle feeds from operational systems as well as bulk ETL loads from legacy data systems. Uses of an ODS commonly include querying for ad hoc reporting, application system queries, as well as data delivery for other operational systems that utilize the analytical capabilities of the ODS. Syncordant phone: web: PAGE 8 of 14

9 Issues with Traditional Operational Data Stores Unnecessary Data Movement For many ODS applications, it is not necessary to access entire data sets, For example, to support a specific customer service application, an ODS may access only a small fraction of all existing customer accounts in a given time period. With a traditional physical ODS, data for all customers is included, because it s not possible to know in advance which customers are going to call in. So, the use of physical ODS architectures often results in a large amount of unnecessary data propagation and movement. Proliferation of Pre-Defined Interfaces To deploy any ODS, large numbers of pre-defined and hard coded interfaces are required to feed current transactional updates. These interfaces typically are very difficult and expensive to develop and maintain. ETL Tools are Overkill ETL tools traditionally have been oriented toward bulk processing of large data sets, rather than the item-level updates common in ODS environments. For many applications where bulk loads are not necessary, ETL tools provide unnecessary overhead in processing trickle feed updates. Real-time Network Requirements are Too Demanding Message-based infrastructures that are used to propagate item-level updates to information are typically not very efficient, and require far higher network overhead as compared to bulk load methods. As a result, in order to support an ODS environment with close to real-time data latency, the network requirements often are too demanding for corporate infrastructures to support. This is why ODS projects often prove to be prohibitive for customers, and is a leading reason why standard ODS are inflexible to address additional data and data source exchange requirements. Transaction Integrity is Difficult to Manage Since Operation Data Stores often are used as systems of record for data shared between different applications, it is critical to maintain transactional integrity between these stores and operational systems. Depending on the numbers of references to the shared data, the transaction integrity requirements may include synchronization of large numbers of data sources with the common data in an ODS. Maintaining transaction integrity among so many sources is a very difficult problem to solve with traditional methods. Syncordant phone: sales@syncordant.com web: PAGE 9 of 14

10 Why Virtual? What is VODS? A virtual ODS provides direct access to the original sources of information, allowing the local owners of the data more control over the dispersal of the data. Local data owners can maintain control over their data on their premises while also automatically pooling that information with a larger data management system. Operational Data Stores have been built on centralized relational database management systems because, simply, there hasn t been a better way, until now. Syncordant employs a revolutionary architecture that provides much better native support for ODS requirements. Rather than relying upon centralized ODS servers, Syncordant accesses data at its sources, and services information consumers with unified views of real-time operational data from disparate sources via reporting, query and business intelligence tools, and enterprise applications. Because there is no physical server, the Syncordant created, in effect, the Virtual Operational Data Store (VODS). The most important attributes of the Syncordant Virtual Operational Data Store are its patented, network-centric architecture and distributed information processing power. Syncordant s architecture is a substantial paradigm shift that leverages patented and proprietary innovations in network search, routing and data access. Today s networks are very efficient at connecting network addresses, but provide no visibility into the data sources that reside on those networks. Syncordant adds data level visibility, thus providing network-based views into operational data. This eliminates the need to move data to central repositories. Instead, Syncordant employs intelligent network query techniques to access and retrieve data only those sources that hold the data required to answer specific queries. Syncordant phone: sales@syncordant.com web: PAGE 10 of 14

11 Architecture of Syncordant s VODS A Virtual Operational Data Store (enabled by Syncordant) The diagram above depicts the architecture of the VODS. Syncordant provides a set of distributed software components, known collectively as Syncordant data Exchange (SNDX) that run on commodity hardware. The components are as follows: Intelligent Data Services (IDS) IDS provide data source access, transform data from local to global representation, execute persistent queries, perform local joins and render alerts. They reside locally or in the cloud. Dynamic Query Routers (DQR) DQR intelligently route queries and results through the SNDX network environment, and provide reliable connectivity between all components. Query Processors (QP) QP provide a SQL environment for query, reporting and business intelligence tools and applications to access the VODS, issue queries, and get query responses. Administration Dashboard The Dashboard provides a simpleto-use interface to administer access control, map data sources, define transformations, and coordinate all of the distributed components of the VODS. Syncordant phone: sales@syncordant.com web: PAGE 11 of 14

12 Benefits of Syncordant s VODS VODS provides significant benefits compared to centralized data storage architectures for ODS requirements: 90% Cost Reduction - No Data Replication - With Syncordant, there is no need to replicate data. Syncordant accesses data at its sources via intelligent and efficient distributed query mechanisms. This results in a 90% cost savings of hardware, database software, and integration engineering time. Only the information needed to answer a query is extracted with Syncordant s VODS, therefore it is able to meet the tight real-time requirements even in the most dynamic situation. 90% Time Savings - With Syncordant, there is no need to create a physical ODS. Simply create a virtual schema, use the Administration Dashboard to map the local data sources, and information consumers can query in a matter of hours or days. Trading Partner Data Exchange - Syncordant s distributed architecture uniquely enables the most effective and economical integration of remote data sources for trading partner data exchange. Enterprises can get real time visibility into all the physical and logical events that are taking place in their demand, supply, production and finance networks. The result is superior support for sense-andrespond business processes such as Just-in-Time Inventory replenishment, realtime product quality monitoring, or credit card fraud detection. Exception Alerts - Syncordant s distributed information processing uniquely enables real time exception alerts across large numbers of remote data sources. Information consumers can invoke persistent queries on remote data sources and monitor to changes to the local data. Alerts are generated and delivered to users automatically, along with the relevant data, when changes to local data are detected that satisfy persistent query conditions. Network and CPU resource usage is minimized, since no persistent query polling is required from central servers. Effective operation of a virtual ODS is possible even in low network bandwidth situations because it only transfers the information that is needed to answer the query. Minimal Network Bandwidth - Network bandwidth is greatly minimized because there is no need to load operational data into a central repository. Network bandwidth is required only to route queries directly to local data source and return the results, or when persistent queries identify changes to local data that require sending alert notifications. Flexibility for New Data Sources - Syncordant is designed to be easily extensible and access additional data sources and data models. Adding new data sources DOES NOT interrupt Syncordant s production environment. Syncordant phone: sales@syncordant.com web: PAGE 12 of 14

13 No Pre-Defined Message Routes or Interfaces - Syncordant requires only a logical mapping between data models in order to access distributed and remote data sources. Syncordant provides an easy-to-use GUI environment that is accessible across a network to administer data access, transformations, and mappings. Access to Data over WAN and Internet - Syncordant is designed using a network paradigm; so, access to data across WANs, via the Internet and through firewalls is virtually seamless. Patented and proprietary network routing methods used in Syncordant are built to natively route to any source that is accessible on the network. Unlimited Scalability - Traditional ODS architectures are limited in the number of feeder sources that can be integrated. Syncordant provides unlimited scalability to operational sources as well as terabytes of indexed data, thereby meeting even the most demanding of ODS scalability requirements. Summary Comparison of Electronic Data Warehouses, Operational Data Stores and Virtual Operational Data Stores Virtual ODS implementations are often measured in days or weeks, at cost savings up to 90 percent compared to building a centralized ODS. EDW ODS Virtual ODS Data Residence Central Database with Central Database with Original Data Sources Replicated Data Replicated Data Cost Very High High Low Development Time Years Months to Years Weeks to Months Data Freshness Daily Minutes to Hours Seconds to Hours Data Access Technologies Batch-Oriented Programs and ETL Products Batch-Oriented Programs and ETL Products Access More than 200 Data Source Types in Place Summarization Yes Very Little None Query Complexity Large Complex Queries Small Targeted Queries Small Targeted Queries Schema Organization More Denormalized Both Normalized and Denormalized Both Normalized and Denormalized Corporate-Wide View Strategic Decision- Making View Tactical Day-To-Day View Tactical Day-To-Day View Data Owner Single Single Multiple, Local Control Network Bandwidth High High Low Syncordant phone: sales@syncordant.com web: PAGE 13 of 14

14 Conclusion IT departments are challenged by applications that demand access to more current data and operational analytics. The Operational Data Store is a construct that emerged to access heterogeneous operational data and address these requirements. Unfortunately, relational database management systems are expensive and time consuming to build, and are often overkill for some of the most basic enterprise applications. Syncordant has introduced a new technology that enables Virtual Operational Data Stores (VODS), thus eliminating the need to replicate data in central physical repositories. By accessing data in-place, Syncordant provides orders-of-magnitude improvement in cost savings and ROI. About Syncordant Data Exchange Syncordant Data Exchange is based upon extraordinary patented and proprietary advancements in network search, routing and data access science. It originally was conceived and developed by the US Defense Advanced Research Projects Agency (DARPA) on behalf of the US Army Communications and Electronics Command (CECOM). The military version is used to enhance precision, remote battlefield command and control. These advancements then were adapted for commercial use at Siemens Corporation s Technology-to- Business Transformation Center in Berkeley, CA. About Syncordant Syncordant is owned by IgniteIP, a private equity firm that focuses on the placement and monetization of intellectual property (IP) directly into industry. Syncordant. All rights reserved. Syncordant is a pioneer in combining network and database technologies to lead the emerging market for Enterprise Information Integration. The Syncordant Data Exchange provides right time access to information, without replication, regardless of how and where the data is stored. The names Syncordant and Syncordant Data Exchange as well as associated corporate logos are registered trademarks of Syncordant, Inc. Other company, product of services names may be trademarks or service marks of other corporate entities. Syncordant phone: sales@syncordant.com web: PAGE 14 of 14