(54) (71) (72) (73) (21) (22) (63) Applicant: Platfora, Inc., (US) Zhang, Fremont, CA (US) Assignee: Platfora, Inc., San Mateo, CA (US)

Transcription

1 US A1 (19) United States (12) Patent Application Publication (10) Pub. No.: US 2013/ A1 Eshleman et al. (43) Pub. Date: Aug. 29, 2013 (54) (71) (72) (73) (21) (22) (63) INTEREST-DRIVEN BUSINESS INTELLIGENCE SYSTEMS AND METHODS OF DATA ANALYSIS USING INTEREST-DRIVEN DATA PIPELINES Applicant: Platfora, Inc., (US) Inventors: John Glenn Eshleman, Mountain View, CA (US); Benjamin Mark Werther, Burlingame, CA (US); Kevin Scott Beyer, San Francisco, CA (US); Brian F. Babcock, Palo Alto, CA (US); YeWei Zhang, Fremont, CA (US) Assignee: Platfora, Inc., San Mateo, CA (US) Appl. No.: 13/861,205 Filed: Apr. 11, 2013 Related US. Application Data Continuation of application No. 13/408,872,?led on Feb. 29, 2012, now Pat. No. 8,447,721. (60) Provisional application No. 61/505,271,?led on Jul. 7, Publication Classi?cation (51) Int. Cl. G06F 17/30 ( ) (52) U.S. Cl. CPC..... G06F 17/30563 ( ) USPC /602; 707/600 (57) ABSTRACT Interest-driven Business Intelligence (BI) systems in accor dance With embodiments of the invention are illustrated. In one embodiment of the invention, a data processing system includes raw data storage containing raw data, metadata stor age containing metadata that describes the raw data, and an interest-driven data pipeline that is automatically compiled to generate reporting data using the raw data, Wherein the inter est-driven data pipeline is compiled based upon reporting data requirements automatically derived from at least one report speci?cation de?ned using the metadata. 41s s 424, 430

2 Patent Application Publication Aug. 29, 2013 Sheet 1 0f 17 US 2013/ A1

3 Patent Application Publication Aug. 29, 2013 Sheet 2 0f 17 US 2013/ A1 260 \ \V 210 \\ \ Bi Reparting Engine g Data Manager / \\ interest-driven Data Pipeiine // Compiier 7/ Data Warehouse 240 \ \ Raw Data Storage

4 Patent Application Publication Aug. 29, 2013 Sheet 3 0f 17 US 2013/ A1 300 \1 31 O

5 Patent Application Publication Aug. 29, 2013 Sheet 4 0f 17 US 2013/ A

6 . Patent Application Publication Aug. 29, 2013 Sheet 5 0f 17 US 2013/ A1 mmomtmg $ Q02 :omusmew 5mm

7 Patent Application Publication Aug. 29, 2013 Sheet 6 0f 17 US 2013/ A1 3mm mcmtonmm Ema wumtmg gmmmaonw mug/m meteqmm Ema / 353cm 68% cozmmgmg 32m U gm QQQ comumh?cm mg 3mm Sam howmhmmww

8

9 Patent Application Publication Aug. 29, 2013 Sheet 8 0f 17 US 2013/ A1 606

13 Patent Application Publication Aug. 29, 2013 Sheet 12 0f 17 US 2013/ A \\ \\ \ \\ \ \q

14 Patent Application Publication Aug. 29, 2013 Sheet 13 0f 17 US 2013/ A

17 Patent Application Publication Aug. 29, 2013 Sheet 16 0f 17 US 2013/ A1 1406

19 US 2013/ A1 Aug. 29, 2013 INTEREST-DRIVEN BUSINESS INTELLIGENCE SYSTEMS AND METHODS OF DATA ANALYSIS USING INTEREST-DRIVEN DATA PIPELINES CROSS-REFERENCE TO RELATED APPLICATIONS [0001] The current application is a continuation of US. application Ser. No. 13/408,872?led Feb. 29, 2012, which application claimed priority to US. Provisional Patent Appli cation No. 61/505,271,?led Jul. 7, 2011, the disclosure of which is incorporated herein by reference. FIELD OF THE INVENTION [0002] The present invention relates to business intelli gence systems, speci?cally interest-driven business intelli gence systems and methods of data analysis using interest driven data pipelines. BACKGROUND OF THE INVENTION [0003] Business intelligence refers to techniques for iden tifying, processing, and analyzing business data. Business intelligence systems can provide historical, current, and pre dictive views of business operations. Business data, gener ated during the course of business operations, including data generated from business processes and the additional data created by employees and customers, may be structured, semi-structured, or unstructured depending on the context and knowledge surrounding the data. In many cases, data generated from business processes is structured, whereas data generated from customer interactions with the business is semi-structured or unstructured. Due to the amount of data generally generated during the course of business operations, business intelligence systems are commonly built on top of and utilize a data warehouse. [0004] Data warehouses are utilized to store, analyze, and report data; for example, business data. Data warehouses utilize databases to store, analyze, and harness the data in a productive and cost-effective manner. A variety of databases are commonly utilized, such as a relational database manage ment system (RDBMS), such as the Oracle Database from the Oracle Corporation of Santa Clara, Calif., or a massively parallel processing analytical database, such as Teradata from the Teradata Corporation of Miamisburg, Ohio. Business intelligence (BI) and analytical tools, such as SAS from SAS Institute, Inc. of Cary, NC, are used to access the data stored in the database and provide an interface for developers to generate reports, manage and mine the stored data, perform statistical analysis, business planning, forecasting, and other business functions. Most reports created using BI tools are created by database administrators, and the underlying data base may be tuned for the expected access patterns. A data base administrator may index, pre-aggregate or restrict access to speci?c relations, allow ad-hoc reporting and exploration. [0005] Online transaction processing (OLTP) systems are designed to facilitate and manage transaction-based applica tions. OTLP may refer to a variety of transactions such a database management system transactions, business, or com mercial transactions. OLTP systems typically have low latency response to user requests. [0006] Online analytical processing (OLAP), a modi?ca tion of OLTP, is an approach to answering multidimensional analytical queries. OLAP tools enable users to analyze mul tidimensional data utilizing three basic analytical operations: consolidation (aggregating data), drill-down (navigating details of data), and slice and dice (take speci?c sets of data and view from multiple viewpoints). The basis for any OLAP system is an OLAP cube. An OLAP cube is a data structure allowing for fast analysis of data with the capability of manipulating and analyzing data from multiple perspectives. OLAP cubes typically are composed of numeric facts, called measures, categorized by dimensions. These facts and mea sures are commonly created from a star schema or a snow?ake schema of tables in a RDBMS. [0007] A snow?ake schema is an arrangement of tables in a RDMBS, with a central fact table connected to one or more dimension tables. The dimension tables in a snow?ake schema are normalized into multiple related tablesifor a complex schema there will be many relationships between the dimension tables, resulting in a schema which looks like a snow?ake. A star schema is a speci?c form of a snow?ake schema having a fact table referencing one or more dimension tables. However, in a star schema, the dimensions are normal ized into a single tableithe fact table is the center and the dimension tables are the points of the star. [0008] Returning to OLAP systems, measures are derived from fact tables, which are typically composed of the mea surements or data of a business process. Dimensions are derived from the dimension tables. In other words, a measure has a set of labels, where the description of the labels is described in the corresponding dimension. Two varieties of OLAP tools are commonly used: relational OLAP (ROLAP) and multidimensional OLAP (MOLAP). Both ROLAP and MOLAP are designed to allow analysis of data through the use of a multidimensional data model. [0009] ROLAP tools access the data in a relational database and generate SQL queries to calculate information at the appropriate level when an end user requests it. With ROLAP, it is possible to create additional database tables (summary tables or aggregations), which summarize the data at any desired combination of dimensions. While ROLAP uses a relational database source, generally the database must be carefully designed for ROLAP use. A database which was designed for OLTP will not function well as a ROLAP data base. Therefore, ROLAP still involves creating an additional copy of the data. However, since it is a database, a variety of technologies can be used to populate the database. One example of a ROLAP tool is the Pentaho BI Suite from the Pentaho Corporation of Orlando, Fla. [0010] MOLAP tools differ from ROLAP tools in that MOLAP tools often involve the pre-computation and storage of information in an OLAP cube. Most MOLAP solutions store this data as an in-memory multidimensional array, rather than in a relational database. This pre-processing and storage of data allows for fast query performance due to optimized storage, multidimensional indexing and caching, and automated computation of higher level aggregates of the data. However, the pre-processing and storage of data has some disadvantages, such as a long processing step, espe cially when dealing with large volumes of data. MOLAP tools traditionally have dif?culty querying models with dimen sions with very high cardinality or a large number of dimen sions. One example of a MOLAP tool is the Cognos Power play system from International Business Machines of Armonk, NY. [0011] Predictive analytics encompasses a variety of statis tical techniques from modeling, data mining and game theory

20 US 2013/ A1 Aug. 29, 2013 that analyze current and historical facts to make predictions about future events. Generally, When referring to business intelligence systems, the term predictive analytics is used to mean predictive modeling, scoring data With predictive models, and forecasting. SUMMARY OF THE INVENTION [0012] Businesses are increasingly capturing and storing machine generated data, such as server logs or records of user interactions With a system, resulting in the generation of extremely large amounts of data. Accordingly, machine-gen erated data is exposing many of the limitations of traditional BI systems that are not designed to handle such large volumes of data. The systems used to store such large volumes of data are typically high-latency and, therefore, provide very poor interactivity. Traditional business intelligence systems often utilize an in-memory processing model Where datasets are loaded into system memory for analysis from a data Ware house using a data pipeline. Existing systems utilizing this method require a signi?cant amount of labor by highly trained engineers and business intelligence analysts to build the data pipeline to populate the in-memory dataset based upon the raw data. Further, there is no active updating of the in-memory dataset once the data pipeline has been built. Changes or updates to the data pipeline typically involve further efforts from the supporting engineers and analysts and the end user typically does not have visibility concerning data that is not in the in-memory data set that may be available for analysis. [0013] Interest-driven Business Intelligence (BI) systems in accordance With embodiments of the invention are capable of managing huge datasets in a Way that provides a user With complete visibility into the available data and the ability to dynamically recon?gure the BI system s data pipeline to provide access to desired information. [0014] Systems and methods for interest-driven business intelligence systems in accordance With embodiments of the invention are illustrated. In one embodiment of the invention, a data processing system includes raw data storage containing raw data, metadata storage containing metadata that describes the raw data, and an interest-driven data pipeline that is auto matically compiled to generate reporting data using the raw data, Wherein the interest-driven data pipeline is compiled based upon reporting data requirements automatically derived from at least one report speci?cation de?ned using the metadata. [0015] In another embodiment of the invention, the inter est-driven data pipeline is automatically compiled based upon at least one change selected from the group consisting of a change to the reporting data requirements, a change to the metadata, and a change to the raw data. [0016] In an additional embodiment of the invention, the data processing system further includes source data storage con?gured to store source data, Wherein the interest-driven data pipeline is con?gured to create source data by perform ing an extract, transform, and load (ETL) process on raw data using a source model. [0017] In yet another embodiment of the invention, the data processing system further includes aggregate data storage con?gured to store aggregate data, Wherein the interest driven data pipeline is con?gured to generate aggregate data by performing aggregations using the source data. [0018] In still another embodiment of the invention, the interest-driven data pipeline is con?gured to generate aggre gations utilizing the reporting data requirements. [0019] In still yet another embodiment of the invention, the data processing system further includes an intermediate pro cessing layer, Wherein the intermediate processing layer includes a data manager con?gured to store data models and an interest-driven data pipeline compiler. [0020] In yet another embodiment of the invention, the intermediate processing layer accesses data Within the raw data storage. [0021] In still another embodiment of the invention, the interest-driven data pipeline is automatically compiled utiliz ing the interest-driven data pipeline compiler. [0022] In still yet another embodiment of the invention, the intermediate processing layer is con?gured to perform raw data intake comprising updating raw data and updating reporting data. [0023] In yet another embodiment of the invention, the data processing system is con?gured as an interest-driven business intelligence system including a business intelligence report ing engine, Wherein the business intelligence reporting engine includes reporting data storage con?gured to store reporting data. [0024] In still another embodiment of the invention, the business intelligence reporting engine comprises a user inter face con?gured to display data and receive requests for data. [0025] In still yet another embodiment of the invention, the interest-driven business intelligence system further includes report speci?cation storage con?gured to store report speci?cations, Wherein the interest-driven business intelligence system is con?gured to receive at least one report speci?ca tion. [0026] In yet another embodiment of the invention, the interest-driven data pipeline is con?gured to automatically generate reporting data requirements using at least one report speci?cation. [0027] In still another embodiment of the invention, the business intelligence reporting engine is con?gured to gen erate a report utilizing the report speci?cation and the report ing data. [0028] In still yet another embodiment of the invention, the interest-driven business intelligence system includes an inter est-driven data pipeline compiler, Wherein the interest-driven data pipeline is con?gured to convert the raw data into source data utilizing an extract, transform, and load (ETL) process, utilize the reporting data requirements to generate aggrega tion processes that are applied to the source data to generate aggregate data, and generate a data model based upon the reporting data requirements and to populate the data model using the aggregate data to generate reporting data; Wherein the interest-driven data pipeline compiler is con?gured to automatically compile the interest-driven data pipeline in response to the interest-driven business intelligence system receiving a report speci?cation modifying the reporting data requirements. [0029] In yet another embodiment of the invention, the interest-driven data pipeline compiler is con?gured to update the ETL process in response to the modi?cations to the report ing data requirements. [0030] In still another embodiment of the invention, the interest-driven data pipeline compiler is con?gured to gener ate new aggregations in response to the modi?cations to the reporting data requirements.

21

22

23

24

25

26

27

28

29