WHITE PAPER. Interactive Data Exploration With An In-Memory Analytics Engine INTELLIGENT BUSINESS STRATEGIES

Transcription

1 INTELLIGENT BUSINESS STRATEGIES WHITE PAPER Interactive Data Exploration With An In-Memory Analytics Engine By Mike Ferguson Intelligent Business Strategies May 2015 Prepared for:

2 Table of Contents Self-Service Visual Analytics Tools How They Work... 3 Ad-Hoc Query - Converting User Actions Into SQL Queries... 5 SQL As An Analytical Query Language... 6 Key Requirements For Interactive Data Exploration And Analysis... 7 Coping With Concurrent Interactive Data Exploration Using Qlik s QIX Engine... 9 How Does The QIX Engine Work And What Does It Allow You To Do?... 9 Stepwise Approach Loading the data Insight production and user interface development Insight consumption Multi-Table Data Model In-Memory, Compressed, Binary, Columnar Data Logical Inference and Calculation In The QIX Engine The Power Of Grey Complex Calculations Using Parameters And Dynamic Data Sets Conclusions Copyright Intelligent Business Strategies Limited, 2015, All Rights Reserved 2

3 SELF-SERVICE VISUAL ANALYTICS TOOLS HOW THEY WORK Interactive dashboard and visual data discovery tools have had the greatest business impact in recent years For many years the adoption of business intelligence (BI) tools in organisations was relatively flat. However over the last few years we have seen a growth in adoption of BI thanks to the emergence of self-service BI tools. Self-service BI is a broad term that does not necessarily refer to a single tool or BI platform component. It covers a several types of tools that include business query, interactive reports, dashboards, visual discovery and mobile BI software. Dashboards, visual discovery and mobile BI in particular have contributed significantly to higher rates of BI adoption in many organisations. However mobile BI is aimed at information consumers and while important it is not much use if there is no insight available to consume. For this reason, interactive dashboards and visual discovery technologies have had the greatest business impact because they have made it possible for business analysts to explore data and quickly produce insight on their own without the need for IT. These kinds of tools reduce time to value. They allow business to clear any IT backlog that may exist. They help improve business agility because they are easy to use, provide advanced visualizations (e.g. Graph matrices, bullet graphs, tree maps, heat maps etc.) and interactive dashboards that guide people to take the correct action. Also, they give business analysts access to multiple data sources and the ability to quickly share insights with decision makers by publishing them to the web and to mobile devices. So how do these products work? What is it that makes them so attractive? Interactive dashboard tools with visual data discovery capability can pre-load data into memory Data from multiple data sources can be used to create visualisations and dashboards Figure 1, shows an interactive dashboard building tool with visual data exploration capability. One key feature of these products is that they provide the ability to pre-load and integrate data from one or more data sources into memory via some type of ETL or scripting capability. In a sense this is like creating an in-memory data mart with compressed, columnar data. Data sources can include real-time streaming data, data at rest or a combination of both. Being able to query, analyse and visualise real-time streaming data allows business analysts to create operational dashboards with live visualisations for situational awareness. For analysis of real-time streaming data to work well, requires in-memory processing. For data at rest, IT personnel or a business analyst can work with the business to define an initial set of requirements to identify what data needs to be loaded into memory and then build the ETL job(s) or script do to that. Once the data is in memory, then users have the flexibility and freedom to explore, analyse and visualise data in any way they want in order to identify new insights. The most important insights created can then be brought together to create managerial dashboard applications that offer up integrated, actionable information on a single screen to monitor key aspects of the business. Once these dashboard applications are built, in-memory data makes it possible to share them among large numbers of concurrent information consumers who can then interact with the data at the speed of thought. Copyright Intelligent Business Strategies Limited, 2015, All Rights Reserved 3

4 Interactive Data Exploration Using An In-Memory Analytics Engine Interactive Dashboard Building Tool With Visual Data Discovery Business Analyst Information consumers Publish / Share Dashboard tool with visual data discovery In-memory data allows many concurrent users to interact with data Visual data discovery server with in-memory columnar storage In-memory data collaborate Data Integration Script / Tool SQL SQL Transaction systems DW RDBMS Consume / Enhance / Re-publish e.g. Dashboards personal & office data Business analysts can create interactive dashboard applications that can be shared with and used by many Predictive models RDBMS Copyright Intelligent Business Strategies ! Figure 1 Another approach to visual data discovery is shown in Figure 2. Here, the business analyst simply connects to one or more data sources, chooses what data to load into memory (e.g. using SQL) and is guided by the tool as to what dimensions and measure are discovered in the data. From there, they can start to mash together, and visualise data to produce new insight with further queries being issued to underlying databases as and when needed. Unlike the multi-user dashboard and visual data discovery environment on pre-loaded data in Figure 1, this approach is more of a single user data exploration and data visualisation environment where data requirements are less well known up front. Of course, multiple business analysts can be working concurrently but it is not like in Figure 1 where multiple users are sharing the same data on the BI platform. Nevertheless, once new insights have been identified, they can again be put in a dashboard and shared with information consumers who can then access the dashboard via the web or mobile device. Typical Use of Visual Data Discovery Tools Accessing Multiple Data Sources Some visual data discovery tools don t pre-load data into memory and are more aimed at single user data exploration Business Analyst community Publish / Share Visual data discovery tool Visual data discovery server with in-memory columnar storage Consume / Enhance / Re-publish e.g. Dashboards In-memory data SQL collaborate SQL Transaction systems DW personal & office data Predictive models Using a visual discovery tools with dashboard capabilities business analysts can mash-up data to create new insights and share these with information consumers Copyright Intelligent Business Strategies ! Figure 2 Copyright Intelligent Business Strategies Limited, 2015, All Rights Reserved 4

5 AD-HOC QUERY - CONVERTING USER ACTIONS INTO SQL QUERIES An interesting contrast to the data visualisation tools described above is when you look at how ad hoc business query works. Figure 3 shows a very common approach to ad hoc business query processing whereby a tool provides users with access to a relational DBMS via a business view (also known as a semantic layer or a metadata layer) that shields the user from having to navigate the physical data model in a database. SQL Based Business Query Tool With Business View(s) Business Analyst SQL based ad hoc query tools often use business views to hide complexity from business users and simplify access to data Business query tool BI Server Business view Publish / Share e.g. report Information consumers collaborate Consume / Enhance / Re-publish SQL DW or Data Mart Ad hoc query BI tools generate SQL to query underlying RDBMSs Some ad hoc query tools may only generate a subset of SQL dialect supported by the underlying RDBMSs Many factors dictate how well concurrent users can interact with data at the speed of thought Copyright Intelligent Business Strategies ! Figure 3 Here, metadata specifications are used to define dimensions and measures in the business view and how they map to the underlying physical structures in the data warehouse or data mart DBMS. The purpose of this is to hide complexity from business users and make it easier for them to navigate and analyse data. Business views are created before users start querying data. If necessary, multiple business views can be created and aimed at different business user communities to give them access to the data they need. Depending on the BI tool, pre-defined business views can connect to one or more data sources although only one data source is shown in Figure 3. Users then start creating ad hoc queries by selecting measures and dimensions in the business view, viewing tabular results and then visualising these using various charting options. As this occurs, the software converts users selections into one or more SQL SELECT statements before querying a physical data model in the underlying relational DBMS. It is fairly straightforward. The question is, can the underlying DBMS cope with a large number of concurrent users running ad hoc queries while also allowing them to constantly change their minds at the speed of thought? That depends on a number of things including: Whether or not the DBMS supports compressed columnar storage Whether or not the DBMS supports columnar data partitioning The physical design of the database e.g. whether the data model is a star schema design, what indexes have been created etc. Unlike Figure 1 however, the BI tool does not load data into memory. It is the DBMS that takes the strain as the number of users increases. Copyright Intelligent Business Strategies Limited, 2015, All Rights Reserved 5

6 SQL AS AN ANALYTICAL QUERY LANGUAGE All BI tools use SQL to retrieve data from relational DBMSs SQL supports a range of aggregate and analytical functions In all three examples discussed so far, all of them use SQL to retrieve data from relational DBMSs either to answer business analyst specific ad hoc queries generated by BI tools or to load data into memory for further analysis. Over the years, SQL has been extended to support richer analytical processing. What started out as a small number of aggregate functions back in the 1980 s with MIN, MAX, AVG, SUM and COUNT, is now much richer and includes aggregate as well as analytical functions that can be applied to data over a window. These include: AVG MEDIAN STDDEV COUNT MIN STDDEV_POP CUME_DIST 1 NTILE 2 STDDEV_SAMP DENSE_RANK PERCENT_RANK SUM FIRST_VALUE PERCENTILE_CONT VAR_POP LAG PERCENTILE_DISC VAR_SAMP LAST_VALUE RANK VARIANCE LEAD REGR functions MAX ROW_NUMBER SQL aggregate and analytical functions can be included in views to simplify queries that need to access relational data None or only a subset of SQL aggregate and analytical functions may be available when accessing data in non-relational data stores In addition ROLLUP and CUBE OLAP 3 functions can also be applied to GROUP BY clauses to produce cumulative aggregates such as subtotals and cross-tabulation values respectively. Overall therefore, SQL has been enhanced a lot in support of analytical workloads and it helps if BI tools can generate analytical SQL queries that are executed in parallel on columnar relational DBMSs. In addition, these kinds of functions can be hidden in relational views. In this case the BI tools that generate SQL don t need to know that analytical functions have been applied. However, if your data is not in a relational data store e.g. in a file, in Hadoop HDFS or a non-relational DBMS, then the chances are that either none or only a subset of these functions may be available in an underlying query language. Many of the SQL on Hadoop initiatives today are only ANSI SQL 92 compliant. That means that only the basic aggregate functions are supported. But is this all that is required to support interactive dashboards and visual data discovery? There are a lot more requirements that need to be defined. 1 ROW_NUMBER function, REGR functions, rank functions (i.e., RANK, DENSE_RANK, PERCENT_RANK, CUME_DIST) were introduced in ANSI SQL NTILE function, and window functions LAG, LEAD, FIRST_VALUE, LAST_VALUE were introduced in ANSO SQL CUBE and ROLLUP were introduced in ANSI SQL 99 Copyright Intelligent Business Strategies Limited, 2015, All Rights Reserved 6

7 KEY REQUIREMENTS FOR INTERACTIVE DATA EXPLORATION AND ANALYSIS Many more requirements need to be met to support interactive data exploration and analysis beyond just query generation Need to change data selections, filters, sorting and visualisations all at the speed of thought Add new hierarchires, groups and calculations Perform more advanced analysis Leverage powerful visualisations Create and publish interactive dashboards for consumption When you contrast ad hoc business query in Figure 3 with the needs of business analysts building interactive dashboards and doing visual data discovery in Figure 1, you begin to realise that the difference between them is not just about queries. Yes, analytical functions in SQL are important. However row store RDBMSs can struggle with a large number of concurrent analytical queries. But it is more than that. Users who need the kind of tool shown in Figure 1, want much more agility than just being able to run ad hoc SQL queries as described in Figure 3. They have many more additional requirements than need to be met in order to support interactive data exploration. These include the need to: Analyse user-defined subsets of data Add / remove data sources as and when required Change query data selections on the fly Change filters on the fly Change sorts on the fly Change visualisations on the fly Create their own hierarchies Throw away hierarchies Create their own groups e.g. from a set of products (binning) Add their own calculations Do complex calculations beyond the capabilities of SQL functions Test hypothesis using what-if analysis Perform statistical analysis Perform predictive analysis and forecasting on data Join relational to non-relational data Discover insights from data they didn t select Leverage powerful visualizations to quickly identify problems on a lot of data E.g. o o o Heat maps, graph matrix, network diagrams, geographic maps, small multiples etc. Spot outliers (dots outside of clusters on scatter plots) Identify major problems shown up on a heat map Use single step query and visualisation with automated charting Create user assembled dashboards that include: o Multiple visualisations Copyright Intelligent Business Strategies Limited, 2015, All Rights Reserved 7

8 Support large numbers of users concurrently interacting with data in dashboard applications Managers now want to do their own lightweight analysis on data As elements of analysis become available to the masses, the need for in-memory data grows A powerful underlying engine is needed to support growing numbers of users interacting with and doing their own lightweight analysis on data o o o o Multiple data sets Connected visualisations that work together in context Ability to alert from within the dashboard (with mobile in mind) Ability to act from within the dashboard (with mobile in mind) Publish dashboards to the web and to mobile devices - including for offline use Persist data in memory for multi-user interactive analysis on published dashboards Support multi-threading to cope with a growth in interactive users caused by increasing numbers of mobile users and not just web users In addition, the bar continues to be raised. Whereas, in the past only power users would be expected to investigate exceptions and do what if analyses, we now find that managers want to do it too. In other words, it is the consumers of published dashboards who want to do this. Therefore managerial dashboards are having to change to offer more than just summary data with operational alerting. The requirement now, is that as managers (information consumers) become more analytically savvy, they want to inherit some of the analytical power from the business analyst (information producer) and use it in their day job to be more effective. They want to do it for themselves. They may even want to annotate and republish the dashboard or a subset of it, which puts more of an emphasis on collaboration. Therefore, while data discovery and visualisation is still very much a business analyst need, we are seeing elements of this becoming democratised. It is being offered up to the masses of information consumers who are learning how to use these new analytical powers to become much more effective in the workplace. Furthermore, much like business analysts are who now have the ability to produce their own insights without the need for IT, we are seeing managers (managerial dashboards) and people in front line business operations (operational dashboards) not only becoming capable of leveraging the insight produced for them, but also going further and performing their own light weight analysis via interaction with the data on published dashboards. They want to put their own interpretation on things while maximising the value of insight to the business. When you look at it like this, you quickly realise that the maximising the value of visual data discovery and interactive dashboard applications is not just about SQL. It is way, way, beyond that. In the modern business, where reducing time to value is always important and improving effectiveness among the masses is becoming mission critical, the requirement is as much about having a powerful underlying engine that supports all the aforementioned key requirements as it is about SQL. With that in mind it is worth looking at what such an engine can and should deliver. Copyright Intelligent Business Strategies Limited, 2015, All Rights Reserved 8

9 COPING WITH CONCURRENT INTERACTIVE DATA EXPLORATION USING QLIK S QIX ENGINE Having understood the requirements, this section looks at how one vendor steps up to meet the requirements defined earlier in order to be able to process big data. That vendor is Qlik. Qlik has two products: QlikView and Qlik Sense are built on a common platform QlikView is a tool for building interactive dashboard applications for guided analysis Qlik Sense is a selfservice visual data discovery tool QlikView and Qlik Sense are both built on the same platform and therefore share the same analytics engine Qlik is a provider of a platform-based approach to visual analytics. It was founded in Lund, Sweden in It has approximately 35,000 customers in more than 100 countries with over 2,000 employees worldwide. Qlik offers two products: QlikView and Qlik Sense. QlikView is a Windows based in-memory product aimed at IT and power user business analysts who need to build interactive dashboard applications for rapid deployment across the enterprise. Its in-memory, associated search capability allows it to automate the identification of relationships in data so that business analysts can quickly join and analyse data at the speed of thought. They can then create visualisations and assemble them into multi-screen interactive dashboards that can be published for others to use. QlikView has a dashboard interface and can connect to multiple data sources including popular relational database management systems, applications like SAP, Sage, Salesforce.com, Oracle, and Infor Lawson, CSV files, NoSQL databases and Hadoop distributions from Cloudera, Hortonworks and MapR. Qlik Sense is Qlik s self-service visual data discovery tool for business analysts to explore and analyse data. It also includes storytelling capability to help information consumers step through insights produced by business analysts during data exploration. QlikView and Qlik Sense share the same QIX in-memory analytics engine. That is because QlikView, QlikSense and indeed QlikCloud are all based on a common platform, which Qlik are building out to cover self-service data visualization, guided analytics, reporting, alerting, collaboration and embedded analytics. In addition they are opening up the platform to others to build on top of it. HOW DOES THE QIX ENGINE WORK AND WHAT DOES IT ALLOW YOU TO DO? The QIX analytical engine supports rapid exploratory analysis of in-memory data The Qlik QIX engine is a multi-threaded engine that can fully exploit multi-core processors. It also has its own APIs to allow it to be embedded in other applications. It operates differently from any BI tool that continuously issues SQL-based ad hoc business queries at an underlying DBMS. It works in two ways: To facilitate rapid exploratory analysis and visualisation of in-memory data to build insight It also supports concurrent users accessing interactive dashboard aplications To facilitate large numbers of concurrent users accessing and interacting with dashboard applications to leverage insight produced from them by business analysts Copyright Intelligent Business Strategies Limited, 2015, All Rights Reserved 9

10 STEPWISE APPROACH The QIX engine supports a stepwise implementation approach to producing business insight Data from multiple sources is loaded into memory and converted into compressed binary format for performance Qlik s new self-service data preparation facility generates script to prepare and load data With respect to the former, the QIX engine allows organisations to implement a stepwise, iterative, rapid prototyping approach to producing insight that involves business users from the outset. The purpose of this approach is to fuel agility. Loading the data The first step is to define the data to be loaded. This is a step where IT or a power user develops a script to query data in underlying systems and load data into memory. This effectively creates an in-memory data mart something unique to Qlik. Running the script loads the data, which the QIX engine converts into compressed binary format (more on this later). To improve productivity, Qlik has a new drag and drop self-service data preparation capability, including Smart Data Load to generate scripts. This capability, due soon, removes the need to learn how to write scripts, significantly improving productivity and reducing time to value. One of the most powerful things that can be done in script development is to include set-level SQL statements directly in the script. This allows individual SQL statements to run against each table (or view) in an underlying data store without doing any joins. This means you can bring in all the data you need into memory. Unique column names need to be created during this process by renaming ambiguous columns using SELECT column AS.. Note that in an exploratory environment, calculations may not be known ahead of when interactive data exploration starts. For this reason the QIX engine has been written to encourage on-demand calculations after underlying data has been queried and loaded. Data governance can be controlled to drive consistency across inmemory data Qlik s new Smart Data Load facility can be used to create reusable data preparation services Trusted dimension data can be taken from MDM systems In order to remove any qualms in IT about data governance during this process, several things can be done which include: Using Qlik s new self-service data preparation capability to create data preparation services and publishing these for reuse by other business analysts. Doing this would enable organisations to decouple the loading of data from the production of visual insights and allow them to keep a catalog of data preparation services that business analysts could pick up and reuse. This means there would be no need to re-invent wheels to load data if the data services required are already built. Just re-use the data services required. It also addresses any fears that IT may have of self-service data integration destroying efforts to implement enterprise data governance Commonly defined dimensional data (e.g. customer, product, supplier etc.) can be created by accessing trusted, integrated master data in your underlying master data management (MDM) system and using this approach across all scripts. This practice guarantees consistent dimension data with common definitions across all Qlik in-memory data marts IT design the in-memory start schema data model to be used - including access to an MDM system for common dimension data Copyright Intelligent Business Strategies Limited, 2015, All Rights Reserved 10

11 Data can be taken from a data warehouse where it has already been cleaned and integrated. Trusted base data can shared across projects to avoid re-invention and improve consistency Creating reusable base datasets that can be governed and shared with others to avoid people re-inventing script to do this over and over again in potentially inconsistent ways. With the QIX engine, this can be done by creating headless applications that are effectively void of any visualisations. In this way, popular data (e.g. dimension data and transaction data) can be governed and prepared in advance for business analysts to use. It can be achieved by allowing a layered approach to script development to be set up whereby a base layer of scripts are developed to create re-usable base datasets which then become input to the next layer of scripts above them. A combination of these approaches Business analysts can focus on producing insight, building visualisations and interactive dashboards Insight production and user interface development This means business analysts can focus on the second step namely producing insight. Producing insight itself can happen in two ways By building interactive, dashboard applications with guided analysis each aimed at a community of users in a specific part of the business using QlikView By business analysts doing self-service data exploration and analysis on their own using Qlik Sense. Information consumers make use of interactive dashboard applications and interact with data Insight consumption The third step is the insight consumer who can make use of the applications to interact with the data. Note that even if QlikView applications are built without Qlik Sense in mind, it is possible to explore the data in a QVW file using Qlik Sense. MULTI-TABLE DATA MODEL Information consumers make use of interactive dashboard applications and interact with data This stepwise approach encourages rapid prototyping and shortens the cycle of development. In addition, because joins and pre-defined aggregates are not encouraged during script development, it opens up a broader set of in-memory data for freeform exploration and interactive analysis By bringing data into memory and resolving ambiguities in the data through unique column naming, a multi-table in-memory data model can be created from one or more underlying data sources. By doing this, the QIX engine can automatically determine relationships in the data prior to any exploratory analysis taking place. It assumes a snowflake or star schema dimensional model 4 that is optimised for analysis. Note that nothing is pre-calculated which means that any calculation can be performed on demand. In addition, because the relationships are known, the QIX engine will always calculate an aggregation in the right table thereby avoiding any risk of incorrect aggregation 4 The data model is not limited to a star schema. It can be any schema as long as it lacks circular references Copyright Intelligent Business Strategies Limited, 2015, All Rights Reserved 11

12 Aggregation errors such as doublecounting are automatically avoided errors such as double counting. Furthermore, re-calculation can also be performed on-demand of all in-memory aggregates created on top of the multitable data model at the speed of thought. IN-MEMORY, COMPRESSED, BINARY, COLUMNAR DATA All data is converted into compressed binary format with columnar storage A key differentiator of the QIX engine is its patented approach to compressing data. Data from each source data table is converted into two types of inmemory data structure: A set of symbol tables (one for each column in the original table) A compressed binary data table that replaces the original table A symbol table contains a binary value for each distinct value in a column in the original table. The binary value then replaces the original column value in the original data table. Only the distinct values in a column are kept This means that much more data can be loaded into memory Compressed columnar storage and binary indexing allows the QIX engine support large numbers of concurrent users interacting with data at the speed of thought By doing this for every column, the original data table is converted, column-bycolumn to a compressed binary data table with each row/column cell value replaced with a binary reference. These binary references, act like foreign keys pointing back to a corresponding distinct value in a columnar symbol table. The compressed binary data table therefore contains the same number of columns and rows as the original uncompressed data, but is now full of binary references that make it much more compact. Also, if two fields have the same name in two different tables (i.e. a relationship) then they have the same symbol table and binary representation. The result of this compression technique is that there are no data types for columns in the compressed data table. Absolutely everything is binary. It is the symbol tables that give the QIX engine its compressed columnar and binary indexing capability so that it can support concurrent interactive data exploration, on-demand calculation and concurrent interaction with dashboard applications at the speed of thought. From a server perspective, a 64-bit operating system with a large amount of memory will mean more data can be loaded into memory. This fits with a symmetric multi-processing (SMP) scale-up architecture. LOGICAL INFERENCE AND CALCULATION IN THE QIX ENGINE The Qlik QIX engine support logical inference and ondemand calculations One of the greatest challenges in interactive data exploration is to be capable of meeting speed of thought requirements from a large number of information producers and information consumers all concurrently accessing data. Requirements like changing query data selections on the fly, changing filters on the fly, changing sorts on the fly, adding your own calculations on the fly and repeatedly re-doing complex calculations beyond the capabilities of SQL aggregate and analytic functions are all a challenge. So, besides compressed binary, columnar data, how does the QIX engine support this? The answer is by using a two-step process as each user interacts with in-memory data. The Qlik QIX engine supports logical inference and ondemand calculations Step one is logical inference. Before a user selects anything, the assumption is all that data is in play. However, every time a user clicks on a value in a field (selects data), the QIX engine immediately calculates what distinct values in all related tables qualify as being relevant to the data selection. If you were in a crowded room of people who are all standing and holding a ticket with a number on it and the first digit of the number is selected and read out, then QIX logical inference is like saying would everyone who does not have that first digit on their ticket please sit down. It is about relevance. But it is not Copyright Intelligent Business Strategies Limited, 2015, All Rights Reserved 12

13 Would all data that is not needed please sit down Interactive Data Exploration Using An In-Memory Analytics Engine The QIX engine continuously minimises the data needed to satisfy a user s query with every click the user makes It also continuously determines the shortest path to the data needed happening just on one table. The QIX engine knows the relationships between data tables. Therefore it effectively determines (infers) and flags all distinct column values in all symbol tables selected by the user and all rows in each data table in the star schema that are relevant to the selection. The reason for doing this is so that it can show the user what is selected and what is not selected as also to allow it to ignore all irrelevant data in satisfying the user s query. The powerful thing about QIX logical inference is that it is continuously minimising the data needed to satisfy a user s query with every click he or she makes. Subsequent calculations therefore only ever take place on relevant data. If the user changes his or her mind, it simply re-determines the relevant data subset based on the new selection. Logical inference allow the QIX engine to always know which data in which tables to use and to always be able to determine the fastest path to the data. All calculations are done on-demand on relevant in-memory data highlighted by logical inference Step two is the calculation engine which does all calculations and aggregations on-demand on the relevant in-memory data highlighted by logical inference. Each element of a calculation on each relevant table is calculated before calculating final result. Formulae can be nested and filters added. Every time you select a filter it totally recalculates the hypercube. Remember, because the data is in memory in compressed binary format and has been minimised by logical inference, it is extremely fast. Trying to support lots of concurrent users interacting at the speed of thought when all data is uncompressed on disk in a row oriented database using SQL queries would take too long. Also because calculations are on-demand and in-memory, there is no need to pre-calculate aggregates or complicate the data model with aggregate in- data structures that need to be pre-loaded with data. It is all done at memory speed. THE POWER OF GREY The QIX engine also knows about data a user selected but which was then excluded due to a subsequent selection Another unique capability of the QIX engine is its ability to allow users to see data that was excluded from their analysis due to other column value selections they have made. An example of this is shown in Figure 4 where the user has selected a Bike Helmet product with a list price of $20 that is in stock. When the user selects in stock, the engine knows from logical inference that $30 Bike Helmets are available but are excluded from the selection because a list price of $20 has already been selected. Therefore a value was selected but then excluded by the selection of another field. The QIX engine can highlight insights that a user may be totally unaware of by allowing them to also see data they did not select Source: Qlik Figure 4 If when looking at sales of $20 Bike Helmets the user becomes aware of the fact that perhaps the higher margin $30 Bike Helmets should have been in stock then it could highlight further investigation in a different direction. The point about this is that a user can see what is selected but also what is selected and excluded whereas in the case of analysing the data via SQL Copyright Intelligent Business Strategies Limited, 2015, All Rights Reserved 13

14 Interactive Data Exploration Using An In-Memory Analytics Engine query, they would only see what they selected and would not see what was excluded. The only way to see this would be to run another query. However the problem is that if you don t see what you don t know, you may never know. Another example is shown in Figure 5 where a business analyst has looked at sales in 2014 for their most profitable products and is shown a list of customers who bought those products. However, customers who did not buy the products are highlighted in grey on the lower left the screen. Realising an opportunity, the business analyst can then select the excluded customers to identify organisations to be targeted in a new marketing campaign to sell them high margin products. Figure 5 This is a unique and very powerful capability Users can discover insight they were totally unaware of even when not looking for it during an analysis This power of grey is a very powerful concept. This is about surfacing insight you may be totally unaware of even when you are not looking for it. This is something unique to Qlik and is very valuable in interactive data discovery. It is made possible by the QIX engine maintaining two independent statuses for each and every field value. These are An input state - which is either selected or not selected An output state determined by logical inference which is either possible or not possible In terms of input state, a field value is either selected or not. If selected, logical inference then determines the relevant data. Therefore by tracking what is selected (the input state), the QIX engine can determine what is and is not possible (the output state). COMPLEX CALCULATIONS USING PARAMETERS AND DYNAMIC DATA SETS In addition to the capabilities already discussed, the QIX calculation engine also supports a wide range of aggregate, statistical and analytical functions that can be used in on-demand calculations on in-memory data. It also supports parameterised variables and set analysis. The former allows variables to be created than will be Copyright Intelligent Business Strategies Limited, 2015, All Rights Reserved 14

15 Interactive All calculations Data are Exploration Using An In-Memory Analytics Engine on-demand and inmemory and support parameterised variables recalculated every time you click and make a selection. These variables can be used in other formulae and also for what-if analysis. Set analysis dynamically determines the set of data to be used in a calculation and visualisation based on what a user selects Set analysis can be used to dynamically determine the set of data used in specific calculations of visualisations based on what a user selects. This is done using an expression that can be defined for: A calculation used in an individual visualisation A part of a calculation used in an individual visualisation A calculation on another visualisation that includes all data except that selected for a specific visualisation Other combinations It is often used to compare results of two different time periods even if a user selects only one period It is used to dynamically restrict or extend the data set used in a calculation or part of a calculation Set analysis is commonly used when you want to do a calculation outside your filter e.g. to compare results of two different time periods even if a user selects only one period. For example, if a user selects data to look at sales of a product category for the first quarter this year, set analysis could dynamically extend the dataset to calculate and visualise the sales of the same category for the same quarter last year to allow the user to compare results. Alternatively it could use a predictive analytic to extend the data set to allow a trend to be plotted beyond the current period selected. It could also be used to compare what has been selected against everything e.g. sales of a product against total market sales Multiple set analysis expressions can be used in the same visualisation by applying different filters on different aggregations within the same formula. This means that different parts of the formula are each calculated using a different set of data. Copyright Intelligent Business Strategies Limited, 2015, All Rights Reserved 15

16 CONCLUSIONS The QIX engine is aimed at supporting interactive data exploration and concurrent usage of guided interactive analytic applications It allows business analysts to focus on rapidly building interactive dashboard applications It also allows business analysts to explore subsets of data from multiple sources and answer tough business questions Highlighting insight you may be totally unaware of in combination with everything else makes it very well suited to interactive exploration and analysis Qlik s QIX engine differs significantly from BI tools generating SQL based queries. It is not aimed at replacing SQL databases but in allowing organisation to leverage SQL and go beyond it to meet the needs of a modern enterprise in supporting interactive data exploration and concurrent usage of guided interactive analytic applications. It is designed to allow: IT to focus on data storage, data governance, performance optimization and complicated requests Separation of data modeling from user interface development so that IT and business can work together to iteratively design data models, govern data and rapidly integrate trusted data to load into memory Business analysts focus on rapidly building their own interactive dashboard applications and eliminate backlog Large numbers of concurrent users in different parts of the business to consume these applications, interact with data at the speed of thought and do further lightweight analysis on the data to help them make operational and managerial decisions Allow business analysts who have a good understanding of the data needed, to explore subsets of structured data from multiple sources inmemory to discover and produce new insight as well as answer tough business questions Capabilities like holding data in compressed binary, columnar format, dynamic logical inference for every user on every click, on-demand complex calculations on dynamic data sets allow Qlik to meet almost all of the requirements highlighted for interactive data analysis and exploration. However, doing all this plus providing the ability to highlight insights you may be totally unaware of (via the power of grey ) makes it stand out as a serious contender to help organisations reduce time to insight. Copyright Intelligent Business Strategies Limited, 2015, All Rights Reserved 16

17 About Intelligent Business Strategies Author Intelligent Business Strategies is a research and consulting company whose goal is to help companies understand and exploit new developments in business intelligence, analytical processing, data management and enterprise business integration. Together, these technologies help an organisation become an intelligent business. Mike Ferguson is Managing Director of Intelligent Business Strategies Limited. As an analyst and consultant he specialises in business intelligence and enterprise business integration. With over 33 years of IT experience, Mike has consulted for dozens of companies on business intelligence strategy, big data, data governance, master data management and enterprise architecture. He has spoken at events all over the world and written numerous articles. He has written many articles, and blogs providing insights on the industry. Formerly he was a principal and co-founder of Codd and Date Europe Limited the inventors of the Relational Model, a Chief Architect at Teradata on the Teradata DBMS and European Managing Director of Database Associates, an independent analyst organisation. He teaches popular master classes in Big Data Analytics, New Technologies for Business Intelligence and Data Warehousing, Enterprise Data Governance, Master Data Management, and Enterprise Business Integration. INTELLIGENT BUSINESS STRATEGIES Water Lane, Wilmslow Cheshire, SK9 5BG England Telephone: (+44) Internet URL: [email protected] Interactive Data Exploration Using An In-Memory Analytics Engine Copyright 2015 by Intelligent Business Strategies All rights reserved Copyright Intelligent Business Strategies Limited, 2015, All Rights Reserved 17