GOVERNANCE OVERVIEW A QlikView Technology White Paper December 2011 qlikview.com
Table of Contents Overview 3 Application Governance 4 Development 4 Usage 9 Security 12 Data Governance 13 QVD layer 13 Metadata 15 Conclusion 18 Appendix: Governance Files 19 QlikView Governance Overview Page 2
Overview Governance of QlikView deployments is an important topic for any size deployment, but has particular significance for larger deployments involving teams of developers, large numbers of applications and changing data considerations. For QlikView system administrators, it s important to have a controlled development and data environment and to have a 360-degree view of their deployment. This paper discusses two principle aspects of governance of QlikView deployments: Application governance and Data governance. Application governance refers to both the development of applications as well as the usage of applications, in a deployment. Data governance refers to understanding what data is being used and what its lineage is. Who is this paper for? This paper is designed to provide QlikView Developers and IT professionals with a highlevel overview of the best practices and tools available to provide proper and consistent governance over QlikView deployments, small or large. QlikView Governance Overview Page 3
Application Governance This section summarizes the best practices and available materials to address questions around governance of developing and deploying QlikView applications as well as governance over the usage of applications in a deployment. It does not cover data governance; that is covered in the next section. DEVELOPMENT For the purposes of this paper, it s convenient to consider QlikView as an application development environment and, as such, one that is subject to the rigors of development best practices that exist within all software development environments. These best practices extend to considerations of basic application development best practices, handling teams of developers working on a project, application certification and deployment migrations. General Development Best Practices The document QlikView Best Practices Development (located here) provides a comprehensive overview of the best practices recommended for conducting any development of QlikView applications. It is a reference manual for QlikView developers who s areas of expertise range from data modeling to scripting to UI design. The document is designed to facilitate a much clearer understanding of the methodologies and practices that are optimal for producing highly usable, highly optimized and highly configurable QlikView applications, whether used by small departments or by large enterprises. A partial list of contents includes: UI Design, Scripting, Data Models, Variables, Security, Naming Standards and Testing and Certification. In addition to this document, QlikTech have produced a free QlikView Developer toolkit and accompanying video that outlines the design best practices that should be followed when producing a QlikView application. These can be found at http://demo.qlikview.com under the Video Demos tab. Every good development process should follow a checklist approach, to ensure that no critical pieces or processes are neglected. The document entitled QV Developer Checklist (located here) is a simple yet effective list of key areas that a developer should consider, ranging from data model performance to interface performance to scripting best practices. QlikView Governance Overview Page 4
Figure 1: Development Checklist Figure 1: Development Checklist Application development workflows are necessary within any development project and must be Application development workflows are necessary within any development project and must adhered to especially when dealing with environments that include Dev-Test-Pre-Prod-Prod type be adhered environments. to especially Having a workflow when in dealing place to determine with environments when an application that can/should include be Dev-Test-Pre-Prod- Prod promoted type environments. to a test server or Having a production a workflow server is important in place to the to success determine of any deployment. when an application can/should The document, be promoted App Dev Workflow to a test Scenarios server (located or a production here) provides server 4 example is important scenarios that to the success of any development deployment. teams could The follow, document, depending App on their Dev particular Workflow deployment Scenarios circumstances (located and here) provides 4 example governance scenarios infrastructure. that development teams could follow, depending on their particular deployment circumstances and governance infrastructure. 5 QlikView Governance Overview Page 5
Figure 2: Example Application Development Workflow Scenario Figure 2: Example Application Development Workflow Scenario Having an application certification process in place is important for any QlikView Having an application certification process in place is important for any QlikView deployment. As deployment. the usage of As QlikView the usage grows of QlikView within an organization, grows within QlikView an organization, applications QlikView can proliferate applications rapidly. can As proliferate with any such rapidly. environment, As with any the such quality environment, and relevance the of quality the applications and relevance can vary of the across the applications entire deployment. can vary As across a result, the it s entire important deployment. for the users As a of result, the applications it s important to have for the an users indication of as the to applications the quality of to the have application an indication and whether as to the it is quality the most of current the application version. In and short, whether what business users desire is to know whether an application has been certified or not by their it is the most current version. In short, what business users desire is to know whether an development team(s). Certification indicates that an application has followed standard development application methodologies, has been has been certified reviewed or not by by the their correct development people and team(s). has passed Certification functional indicates use tests. that an application has followed standard development methodologies, has been reviewed Certification is determined by the best practices and governance procedures in place within your by organization, the correct people however and there has exists passed a document, functional entitled use tests. Application Certification Process Certification (located here) is determined to help guide by organizations the best practices who are and considering governance a certification procedures process in place for their QlikView deployments. within your organization, however there exists a document, entitled Application Certification Process (located here) to help guide organizations who are considering a certification process for their QlikView deployments. 6 QlikView Governance Overview Page 6
Figure 3 : QlikView Application Certification Figure 3 : QlikView Application Certification Development Teams It s very common for a team or teams of QlikView developers to work in concert to produce and DEVELOPMENT TEAMS It s very maintain common QlikView for a applications. team or teams In these of types QlikView of environments developers it s to important work in to concert maintain strict to produce control and visibility over the various work products produced during a project by individual team and maintain QlikView applications. In these types of environments it s important to maintain members. Having a robust application control system (i.e. source control system) in place is critical strict to control ensure and developers visibility are over working the on various the most work current products and authorized produced version during of the a project application(s). by individual Source team control members. integration Having was introduced a robust in application version 11 of control QlikView. system With this (i.e. integration, source control system) development in place is teams critical can now to ensure integrate developers with a source are control working system on like the Microsoft most current TFS to and maintain authorized project version control. of the application(s). Source control integration was introduced in version 11 of QlikView. With this integration, development teams can now integrate with a source control system like Microsoft TFS to maintain project control. 7 QlikView Governance Overview Page 7
Figure 4: Source Control Integration in QlikView Figure 4: Source Control Integration in QlikView In addition, QlikTech QlikTech partner partner NOAD NOAD provides provides a complete a complete change management change management and and deploy workflow deployment solution workflow that solution is integrated that is integrated with QlikView, with QlikView, called called EQM4. For For more more information, go information, go to www.noadbi.com www.noadbi.com UPGRADES AND MIGRATIONS Deployment upgrades to new versions, as well as migrations to new server infrastructures Upgrades are a part of and a QlikView Migrations administrator s role. To help facilitate this process, a document entitled QlikView 11 Upgrade and Migrations (located here) has been produced as a stepby-step guide to moving from QlikView 9 and 10 versions to QlikView 11. It also includes Deployment upgrades to new versions, as well as migrations to new server infrastructures part of a QlikView administrator s role. To help facilitate this process, a document entitled best practices for migrating from a single server environment to a multi-server environment 11 Upgrade and Migrations (located here) has been produced as a step-by-step guide to with dispersed services. from QlikView 9 and 10 versions to QlikView 11. It also includes best practices for migratin single server environment to a multi-server environment with dispersed services. QlikView Governance Overview Page 8
Figure 5: QV11 Upgrade and Migrations document Contents Introduction 2 Section 1: General Upgrade and Migration Considerations 4 Section 2: Existing QlikView Server 9 6 Considerations 6 QlikView 9 Upgrade Scenarios 12 Upgrade QlikView Server 9 to QlikView Server 10 or 11 13 Upgrade QlikView Server 9 to QlikView Server & Publisher 10 or 11 13 Upgrade QlikView Server & Publisher 9 to QlikView Server & Publisher 10 or 11 15 Section 3: Existing QlikView Server 10 17 Considerations 17 QlikView 10 Upgrade Scenarios 17 Upgrade QlikView Server 10 to QlikView Server 11 17 Upgrade QlikView Server 10 to QlikView Server & Publisher 11 18 Upgrade QlikView Server & Publisher 10 to QlikView Server & Publisher 11 20 USAGE Section 4: Example Multi-Machine Scenarios 21 Understanding the usage of QlikView deployments is important for a number of reasons. Firstly, Section it s important 5: Re-locating to understand QlikView the Services system usage, from a hardware and bandwidth 30 perspective, so that QlikView administrators can adequately provision the correct hardware and network QlikView infrastructure Server 10/11 to support a growing deployment. Secondly, it s important 30 from an auditing and compliance perspective, especially in those industries where these Directory Service Connector 10/11 32 requirements are critical, such as healthcare, finance and government. QlikView WebServer 10/11 33 The QlikView Management Console (QMC) allows system administrators to manage QlikView deployments. One of the tasks that can be performed allows system administrators to determine which users that are allowed to access applications and data, which users are allowed to download applications and data and which developers are allowed to access the data packages (QVD files) used with QlikView Publisher. It also can control which applications (and their data) can be downloaded, and which cannot. QlikView Governance Overview Page 9
used with QlikView Publisher. It also can control which applications (and their data) can be downloaded, and which cannot. Figure 6: QlikView Management Console QlikTech has produced a tool that allows a system administrator to monitor the complete operations of their QlikView deployment in a visual and interactive way. The QlikView Ops Figure Monitor 6: (located QlikView here) Management is application Console build using QlikView that will read the log files from a QlikView Server and Publisher deployment and will provide the results to an administrator QlikTech in a visual, has interactive produced a environment. tool that allows The a system administrator administrator simply to configures monitor the the complete application operations to read the of appropriate their QlikView log deployment files and will in a display visual and the interactive results instantly. way. The Administrators QlikView Ops can Monitor analyze (located a wide range here) is of an factors, application including build (but using not QlikView limited that to): will read the log files from a QlikView Server and Publisher deployment and will provide the results to an administrator in a visual, Performance interactive characteristics environment. The (including administrator RAM simply and CPU configures usage) the application to read the appropriate log files and will display the results instantly. Administrators can analyze a wide range Task results of factors, including (but not limited to): Data - performance volumes generated characteristics (including RAM and CPU usage) User - task access results data Document - data volumes usage generated - user access data License usage - document usage - license usage 10 QlikView Governance Overview Page 10
This free utility enables a QlikView system administrator to both optimize their deployment but also Figure provide 7: them QlikView with an oversight Ops Monitor capability. application In addition, QlikView administrators very often need to provide auditing results to meet regulatory This requirements. free utility QlikView enables deployments a QlikView have system built-in administrator auditing capabilities, to both optimize where user their activity deployment can be but logged also provide down to them object-level with an oversight of interaction. capability. That is, audit logs can be written to establish who clicked on what, and when. Using the QlikView Systems Monitor application (located here), In addition, these logs QlikView can be read administrators and analyzed easily. very often need to provide auditing results to meet regulatory requirements. QlikView deployments have built-in auditing capabilities, where user activity can be logged down to the object-level of interaction. That is, audit logs can be written to establish who clicked on what, and when. Using the QlikView Systems Monitor application (located here), these logs can be read and analyzed easily. 11 QlikView Governance Overview Page 11
Figure 8: Audit log analysis of a QlikView deployment Figure SECURITY 8: Audit log analysis of a QlikView deployment Providing access to the right data for the right people is a critical component of any Security QlikView deployment. Equally as important is ensuring that no unauthorized access to data Providing is permitted. access QlikTech to the right has addressed data for the this right topic people in is its a comprehensive critical component white of any paper, QlikView QlikView deployment. Security Overview Equally Technology as important White is ensuring Paper that (located no unauthorized here). QlikView access provides to data is a permitted. variety of QlikTech mechanisms has addressed to safeguard this data topic and in its applications, comprehensive including white paper, integration QlikView with Security existing Overview Single Technology White Paper (located here). QlikView provides a variety of mechanisms to safeguard Sign-On (SSO) systems, integration with Active Directory and other LDAP providers, as well data and applications, including integration with existing Single Sign-On (SSO) systems, integration with as native Active row-level Directory security. and other Using LDAP a providers, combination as well of as these native methods row-level highlighted security. Using in the a video combination series QlikView of these Security methods Overview highlighted Video in Series the video (located series here) QlikView QlikView Security deployments Overview Video are Series made highly (located secure here) and QlikView controlled. deployments are made highly secure and controlled. 12 QlikView Governance Overview Page 12
- Source data tracking and lineage Data Governance This section summarizes the best practices and available documents to address questions around data governance within QlikView deployments. QVD LAYER Almost all medium to large QlikView deployments incorporate a so-called QlikView Data (QVD) layer to stage data for rapid reload into front-line QlikView applications. The QVD layer includes QVD files that contain highly compressed and efficient data models, often organized around key metrics such as time, department, function or other user-defined metric. QVD files are created by extracting data from sources such as databases, flat files, transactional systems and other QVD files. These QVD files are then used throughout a variety of different front-line applications. They also can change, based on data reload schedules and changing business requirements. Therefore, it is critical for a QlikView administrator to understand the data lineage of and changes made to their QVD layer. Figure 9: Multi-staged QVD environment 2011 Figure QlikTech 9: Multi-staged QVD environment The QVD Monitor application (located here) is a free utility designed to allow administrators to conduct proactive data monitoring of their entire QVD infrastructure. It provides a comprehensive view of the QVD layer, including (but not limited to): - Changes to QVD files QlikView Governance Overview Page 13
The QVD Monitor application (located here) is a free utility designed to allow administrators to conduct proactive data monitoring of their entire QVD infrastructure. It provides a comprehensive view of the QVD layer, including (but not limited to): Changes to QVD files Source data tracking and lineage Trend analysis of QVD sizes and usage Field-level lookup Reload date tracking This application allows an administrator to monitor QlikView QVD files and their associated metadata. They can track data such as load date/times, sizes, record counts, column counts, This application allows an administrator to monitor QlikView QVD files and their associated column names, and trends for each. A set of charts is included to highlight changes in data metadata. They can track data such as load date/times, sizes, record counts, column counts, and to help identify any anomalies or exceptions that might occur. column names, and trends for each. A set of charts is included to highlight changes in data and help identify any anomalies or exceptions that might occur. Figure 10: QVD Data lineage display Figure 10: QVD Data lineage display QlikView Governance Overview Page 14
Figure 11: QVD Data size trend monitor In addition to the white paper, two utilities exist that allow a QlikView system administrator to extract the metadata from a deployment and, once extracted, analyze it and monitor it for changes METADATA that occur over time. The MetaScanner application (located here) will scan through a deployment, The extracting topic of the metadata descriptive, is an administrative important and one structural and is usually metadata high that on exists the in list a deployment, of relevant and topics the MetaMonitor application (located here) will provide administrators a clear picture of their when discussing the governance of any BI deployment, whether with QlikView or another entire deployment. Examples include (but are not limited to): solution. As outlined in the comprehensive white paper on this topic, QlikView s Pragmatic - Application-level Approach to Metadata (located here), metadata management with QlikView is both optional and pervasive. o That Number is, a of QlikView objects system admin can chose to implement a metadata monitoring environment o Number of (or variables not, depending on their requirements), and if they chose to do so can tap o into Number metadata of QVD that sources is automatically created with any QlikView deployment. This pragmatic approach o Reload gives schedule system administrators all the flexibility they need, when they need it. o Usage stats 17 QlikView Governance Overview Page 15
In addition to the white paper, two utilities exist that allow a QlikView system administrator to extract the metadata from a deployment and, once extracted, analyze it and monitor it for changes that occur over time. The MetaScanner application (located here) will scan through a deployment, extracting the descriptive, administrative and structural metadata that exists in a deployment, and the MetaMonitor application (located here) will provide administrators a clear picture of their entire deployment. Examples include (but are not limited to): Application-level - Number of objects - Number of variables - Number of QVD sources - Reload schedule - Usage stats Deployment-level - Server info messages - - Deployment-level VM committed - RAM o Server usageinfo messages o VM committed - Session statistics o RAM usage - Publisher reload events tracking o Session statistics o Publisher reload events tracking Figure 12: Application-level statistics with MetaMonitor Figure 12: Application-level statistics with MetaMonitor QlikView Governance Overview Page 16
Figure 13: Deployment-level statistics with MetaMonitor Figure 13: Deployment-level statistics with MetaMonitor QlikView can also access the metadata from third party systems, provided there are sufficient QlikView access can also privileges access the and metadata API s from in place third on party the systems, third party provided systems there to are allow sufficient their metadata access privileges to be extracted. and API s By in place doing on this, the third QlikView party systems can inherit to allow the business their metadata rules, to calculations be and extracted. data lineage By doing information this, QlikView without can inherit having the the business create rules, them calculations again. Figure and data 14 shows lineage a information without having the create them again. Figure 14 shows a QlikView application that is QlikView application that is using metadata extracted from a Business Objects Universe using metadata extracted from a Business Objects Universe using a connector build by a QlikTech using partner, a connector DataRoket. build by a QlikTech partner, DataRoket. 19 QlikView Governance Overview Page 17
Figure 14: QlikView application using metadata from Business Objects Figure 14: QlikView application using metadata from Business Objects Conclusion This paper has outlined the available tools and documents for providing robust governance over QlikView This paper deployments. has outlined Topics the available such as tools application and documents governance, for providing including robust both development governance and usage over QlikView have been deployments. covered, along Topics with such governance as application of data. governance, For a better including understanding both of your own specific development governance and usage requirements, have been please covered, connect along with your governance local QlikTech of data. representative For a better to set up a understanding meeting. of your own specific governance requirements, please connect with your local QlikTech representative to set up a meeting. 20 QlikView Governance Overview Page 18
Appendix GOVERNANCE FILES All files are located in the QlikCommunity, located here (http://community.qlikview.com/community) QlikView Best Practices Development v0.5.pdf QV Developer Checklist.xls App Dev Workflow Scenarios.pdf Application Certification Process.pdf QlikView Ops Monitor.qvw QlikView Systems Monitor.qvw QlikView Security Overview Technology White Paper.pdf QVD Monitor.qvw QlikView s Pragmatic Approach to Metadata.pdf MetaScanner.qvw MetaMonitor.qvw International AB. All rights reserved. QlikTech, QlikView, Qlik, Q, Simplifying Analysis for Everyone, Power of Simplicity, New Rules, The Uncontrollable Smile and other QlikTech products and services as well as their respective logos are trademarks or registered trademarks of QlikTech International AB. All other company names, products and services used herein are trademarks or registered trademarks of their respective owners. The information published herein is subject to change without notice. This publication is for informational purposes only, without representation or warranty of any kind, and QlikTech shall not be liable for errors or omissions with respect to this publication. The only warranties for QlikTech products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting any additional warranty. QlikView Governance Overview Page 19