IBM Cognos Framework Manager: Modeling Best Practice Jeremiah Joseph IBM Cognos SWG
Agenda Model Design Principles Modeling for predictable results Modeling for maximum flexibility Modeling Concepts Relational modeling concepts Multidimensional modeling for analysis Query generation Modeling for Dynamic Query Mode (DQM) Architecture Working in DQM mode Modeling techniques for Classic and DQM modes 2
Model for consistent and predictable results Everyone works from the same, common business view Broad range of data can be made readily accessible to all users User self-service, independent of data source Common Metadata extends to all BI capabilities Common Business Model Do you really want users building reports against multiple databases, multiple metadata repositories? IT Tools Services API Shared Set of Purpose-Built Services Business Content Open Data Access Cognos 10 Message Sources Relational Sources OLAP Sources Modern and Legacy Sources 3
Model for Consistent and Predictable Results Understand the reporting requirements What kind of reports will your users be writing? Multiple languages, security requirements, etc.? Know your data Is the data source a reporting structure or operational Do you know which tables are fact tables, which are dimensions and which can act as both? Partner with DBA and Report Author Create purpose specific views Give report authors what they need Think of the report author when designing presentation view 4
Maintainable and Flexible Models Only import objects that you need Import additional objects as required Change as little as possible in the import view Changes at this level are universal Verify query item properties and relationships in import view Ensure that relationships are present and cardinality is correct for the reporting purpose Use cardinality to clearly define facts Check for multiple relationships Ensure that query items are correctly identified, e.g.: fact, identifier Incorrect usage can impact query results 5
Maintainable and Flexible Models Create a Business View Apply business rules in this layer custom calcs, filters, etc. Business friendly names that report authors and consumers will understand Provides a layer of insulation for implementing changes Create a Presentation View to hold shortcuts to business view objects Use of shortcuts ensure that any changes in business view will automatically be reflected in presentation layer Provides an additional layer of insulation for changes Provides modeler and opportunity to organize metadata Tailored structure for report authors that will be easy to navigate 6
Maintainable and Flexible Models Keep models to a manageable size Models with 1000s of objects can be difficult to navigate Models greater than 50 MB in size can cause performance issues for memory intensive operations e.g.: publish, analyze publish impact, verify model Significant improvements made in version 10.1.1 to improve performance Publish performance and scalability dramatically improved Avoid having one model that tries to serve all purposes Have separate models for each line of business or subject area Easier to organize and safer for backup and recovery 7
Agenda Model Design Principles Modeling for predictable results Modeling for maximum flexibility Modeling Concepts Relational modeling concepts Multidimensional modeling for analysis Query generation Modeling for Dynamic Query Mode (DQM) Architecture Working in DQM mode Modeling techniques for Classic and DQM modes 8
How Framework Manager Uses Cardinality Cardinality is used by the query engine to: Identify query subjects that behave as facts and dimensions 1..n cardinality implies fact data on the n side and dimension data on the 1 side Avoid double counting fact data Support loop joins in star schema models Cardinality is applied in the context of a query Query subjects may be facts or dimensions depending on the other query subjects included in query 9
What are Determinants? Feature first introduced in Cognos 8 used to provide control over granularity when aggregating Reflects granularity by identifying groups or subsets of data in a query subject Used to ensure that repeating data is aggregated correctly Are set during import based on unique key and index information in database Used by Framework Manager to determine levels in default hierarchy when auto-generating regular dimensions from query subjects 10
Determinants: When and how should I use them? When do I need to use determinants When dimensions connect to fact tables at levels of granularity that have repeating keys When there is a need to count or perform aggregate functions on a key or attribute that is repeated To avoid generating a distinct clause on distinct keys 11
Determinants and Granularity Use determinants to resolve issues caused by multiple levels of granularity Two levels of granularity MONTH_KEY repeats for each instance of DAY_KEY 12
Determinants and Granularity A determinant will tell the query engine that grouping should occur on MONTH_KEY Otherwise queries could double count SALES_TARGET for each day of the month 13
Recommendation: Use Model Advisor Check early and often Model Advisor is a useful tool for detecting common modeling issues Cardinality issues Potential join path conflicts Incorrect determinants Easy to configure to check as for as much or as little as desired Run after import to check for areas that need further investigation Run again periodically to identify potential issues as you work Direct links to problem area and Help 14
Dimensional Modeling for Report and Analysis Dimensionally modeled relational (DMR) constructs in Classic Query Mode Logical dimensional layer over top of a relational model Single measure dimension and one or multiple regular dimensions Enables drill-up/drill-down analysis in the studios Dimensional layer can be applied over any star schema model Not a requirement that the underlying data source be a star schema design Dimensions and measure dimensions are constructed from model query subjects or database query subjects At runtime, queries are run to spin a virtual cube on disk to simulate the OLAP style analysis capabilities 15
Regular Dimensions Regular dimensions created from model query subjects or database query subjects Define hierarchies to organize levels and define drill paths You can specify multiple hierarchies in a single dimension Define levels, business keys and caption Create from scratch using Dimension Definition dialog Use Merge in New Regular Dimension or Convert to Regular Dimension capabilities Dimension will be auto-generated based on query subject definition Default hierarchy created Levels derived from determinants set May need to be adjusted based on requirements 16
Measure Dimension Create measure dimension Collection of facts Create measure dimension using Dimension Definition dialog or Convert to Measure Dimension capability Add existing measures or create calculations Define scope relationships Dictates level at which measures are available for reporting Scope relationship will be detected by Framework Manager using the underlying join paths between query subjects and fact query subject Scope relationships can be defined and reviewed in the Dimension Map 17
Query Generation By default, Framework Manager will use Cognos SQL to create Query Subjects In Classic Query Mode, at runtime, native SQL optimized for the data source is generated and passed down through to the data source RQP (Relational Query Planner) plans the query and then passes it off to UDA (Universal Data Access) UDA component has a series of gateway files that control SQL generation for specific RDBMS (DB2, Oracle, MS SQL Server, etc) UDA generates the native SQL and executes it against the RDBMS Copy of the query engine local in the Framework Manager install Used for importing metadata, testing query subject, generating SQL and validation of expressions 18
Query Generation In Dynamic Query Mode, requests are sent to the Query Service for processing No query service running locally on the FM side Framework Manager will include pass the necessary elements in the model to Query Service on the server Result set and generated SQL for the query will be part of the response back to FM. This is the case testing objects and evaluating expressions 19
SQL Generation Use minimized SQL when possible Certain conditions will trigger as view SQL and override this setting: Determinants on a model query subject Relationships between model query subjects Filters, macros or calculations in data source query subjects As view SQL not always bad Sometimes required to force certain behaviors Can be necessary to ensure predictable query results Know your requirements and your data 20
Agenda Model Design Principles Modeling for predictable results Modeling for maximum flexibility Modeling Concepts Relational modeling concepts Multidimensional modeling for analysis Query generation Modeling for Dynamic Query Mode (DQM) Architecture Working in DQM mode Modeling techniques for Classic and DQM modes 21
Dynamic Query Mode A new Dynamic Query processing mode (DQM) to the existing query service designed to improve query performance Common Business Model IT Tools Admin Upgrade Model Services API Content Security Metadata Presentation Service Query Service Shared Set of Purpose-Built Services Scheduling Service Movement Service Event Service Cognos 10 Bus Dispatcher SOAP, XML System Content Metadata, Events Search Index, Audit Logs, System Metrics Streaming Service Open Data Access Security Service Calculation Service Optimized OLAP 64-bit in-memory High Performance Streaming Cache Content Shared Dimensions Personal Datasets Attachments, Annotations Initiatives, Metrics Plan & Report Saved Objects Message Sources Relational Sources Application Sources OLAP Sources Modern and Legacy Sources
Dynamic Query Mode Enhanced Java-based query execution mode introduced in 10.1 to support select OLAP sources Leverages JDBC access to supported data sources Support has been extended in 10.1.1 to include the following relational sources DB2, Netezza, Teradata, Oracle, MS SQL Server Can be used with 64-bit report server included in 10.1.1 to enable administrators to better leverage memory Has robust caching capabilities to help improve query performance In-memory caching of query plan and query results Security aware SQL generation optimization for relational sources 23
Dynamic Query Mode New implementation to support dimensional modeling of relational data sources OLAP over Relational Not the same implementation as DMR in compatible mode Simpler, more efficient queries than DMR Null suppression optimizations Master-detail optimizations available in reports Treats OLAP over Relational as a true OLAP source Enables report authors to fully leverage member functions and OLAP style reporting techniques Dramatic performance benefits available by leveraging caching capabilities 24
Working in Dynamic Query Mode Framework Manager 1.x, 8.x and 10.x includes a full copy of the data access stack Used for import, testing and evaluating query subjects, relationships, etc. Framework Manager machine requires DB client software installed With introduction of 10.1.1, Framework Manager enables users to work in Dynamic Query mode or Compatible mode Can enable DQM at the project level or package level Working in dynamic mode will leverage the always leverage the query service on the server Import, testing and evaluation of objects Compatible mode will continue to leverage the local query components CQE, UDA, etc. 25
Working in Dynamic Query Mode New metadata import mechanism available for DQM JDBC import that leverages RELMD service Nothing local required on FM environment Dramatic improvement in performance for very large imports New mechanism to test objects and evaluate expressions Client-server type model Framework Manager will send request to query service Generated query and result set will be returned On-demand evaluation of expressions in editor Option to turn off per-click evaluation in expression editor 26
Working in Dynamic Query Mode Can work in compatible mode and enable dynamic query mode for packages Can enable dynamic mode on a package by package basis Mechanism in place to ensure data sources are enabled Can switch an existing model to dynamic mode All objects in the model will be evaluated by query service For existing models, more efficient to enable dynamic mode on package basis to evaluate 27
Working in Dynamic Mode OLAP over Relational Some differences between DMR and OLAP over Relational OLAP over Relational much closer to true OLAP experience Member ordering DQM provides 2 mechanisms for member ordering Natural ordering Ascending order by member caption Not consistent in compatible mode (varies by query/data source) Sort specification Option available in Framework Manager to explicitly set» Set for metadata tree in studios» Set for data values returned 28
Working in Dynamic Mode OLAP over Relational Null suppression capabilities True OLAP style null suppression available Controls available at studio level Null handling in calculation Configuration available on server to have 0 substituted for nulls for mathematical calculations Existing proven practices for modeling apply to Dynamic Query Mode No departure from our tried and tested recommendations for modeling in Framework Manager 29