DirectQuery vs In-Memory in BISM Julie Koesmarno www.mssqlgirl.com Sydney SQL User Group MsSQLGirl.com 1 @MsSQLGirl
About Me 8+ years experience with SQL Server (since SQL Server 2000) MCITP 2008 Past projects: large scale 24/7 Data Integration, Data Management and Data Cleansing system Current project: Data Warehouse and Analysis Services Tabular Model project 2
Agenda Terminology DirectQuery vs In-Memory Hybrid Mode Out of Scope: Sharepoint Integration 3
Abstract The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing for complex queries to run very fast What are the advantages and disadvantages of using in-memory cache to store and query data? Is DirectQuery the answer to address in-memory cache mode limitations? What are the restrictions on DirectQuery tabular model? When can a hybrid mode be used? How would the selected mode can affect the Processing option during Deployment? In this session, all the above questions will be answered with live demos, comparing the two modes and the scenarios that are applicable to them. It also discusses how partition plays an important role in DirectQuery mode and choosing the correct processing mode when deploying a Tabular Model solution. @mssqlgirl 4
Tabular Model Architecture and Solution Terminology v 5
Vertipaq DirectQuery In-Memory Cached xvelocity 6
Tabular Model Architecture Third Party Application Excel Reporting Services Power View Tabular Model DirectQuery In-Memory SQL Server Databases Non SQL Server Databases Files OData Cloud Services 7
Tabular Model Architecture Third Party Application Excel Reporting Services Power View Tabular Model In-Memory SQL Server Databases Non SQL Server Databases Files OData Cloud Services 8
Tabular Model Architecture Third Party Application Reporting Services Power View Tabular Model DirectQuery SQL Server Databases 9
Tabular Model Solutions Authoring (SSDT) Deployment Option Database Properties Connectivity demo 10
DirectQuery vs In-Memory Execution v 11
Querying DAX DirectQuery SQL DAX / MDX In-Memory VertiPaq SE Process Data Access Query Data Sources 12
DirectQuery Security Query DirectQuery SQL Role DirectQuery Impersonation Impersonate Current User Default (Data Source login) Data Source Impersonation Windows User Service Account 13
In-Memory Security Query In-Memory Role Row Level Security VertiPaq SE Process Data Source Impersonation 14
Other Security Consideration? Identity flow in Sharepoint 2010 http://www.sqlpass.org/summit/2011/speakers/callforspeakers/sessiondetail. aspx?sid=2027 BISM file RSDS file Simple Tabular Model Connection String 15
DirectQuery vs In-Memory Design v 16
Formula Compatibility Semantic differences xvelocity in-memory analytics engine (VertiPaq) SQL Server No Calculated Columns in DirectQuery 17
DAX Functions Some DAX Functions are not supported in DirectQuery: No equivalent calculations in relational engine No equivalent SQL expressions Performance of converted expressions E.g. TOTALYTD, SAMEPERIODLASTYEAR More information on BOL: http://msdn.microsoft.com/en-us/library/hh213006.aspx 18
DirectQuery vs In-Memory Administration v 19
Processing Options (In-Memory) Mode Database Table Partition Process Default Process Full Process Data Process Clear Process Defrag Process Add Process Recalc Further reading: http://msdn.microsoft.com/en-us/library/hh758414.aspx 20
Partitioning in DirectQuery One partition in Tabular Model Can take advantage of Partitioning and ColumnStore Index technology on SQL Server 2012 Partitioning / ColumnStore management done at the source database level 21
Partitioning in In-Memory In-Memory can have multiple mutually exclusive partitions To eliminate unnecessary processing and processor load on the AS servers Process C u r r e n t Mar 2012 Feb 2012 Jan 2012 In-Memory Partitions 22
DirectQuery vs In-Memory Pros and Cons v 23
In-Memory - Pros ALL functionality of Tabular Model Calculated Columns, ALL DAX functions Row Level security xvelocity in-memory analytics engine Many choices of client tool Excel, Power View, SSRS, MDX client* Many supported data sources 24
In-Memory - Cons AS requires Memory & CPU resources Caching Processing Querying Diligent Partitioning and Processing Out of date data 25
DirectQuery - Pros Real time access Scalable SQL Server permission SQL Server optimisation Consistency One Partition to manage No processing required 26
DirectQuery - Cons Restricted DAX functions Restricted client tools: PowerView, SSRS No Row Level Security Unless implemented at the Source Database level No Calculated Columns 27
The best of both worlds Hybrid Mode v 28
Tabular Model Solutions DirectQuery Mode On Off Query Mode DirectQuery DirectQuery with In-Memory In-Memory with DirectQuery In-Memory Connection String: DirectQueryMode=DirectQuery DirectQueryMode=In-Memory 29
Which Hybrid Mode? Query Mode DirectQuery with In-Memory In-Memory with DirectQuery Excel Connection String DirectQueryMode = InMemory No need to specify DirectQueryMode PowerView always uses the default Query Mode (i.e. the first bit of the Query Mode) can t change this! SSRS works with both Query Mode and Connection String combinations 30
Partitioning for Hybrid Mode One Partition for DirectQuery Processing Option: Never process this partition Multiple Partitions for In-Memory (No Overlap) C u r r e n t Aug 2012 DirectQuery Partitions Jul 2012 Jun 2012 In-Memory Partitions 31
Implications Inconsistency Data stale data in Cache Calculations semantic differences xvelocity vs SQL Security DirectQuery Design Flexibility at run time / client tools In-Memory (Excel) or DirectQuery (Real time) http://www.mssqlgirl.com/hybrid-mode-in-tabular-bisemantic-model-part-1.html http://www.mssqlgirl.com/hybrid-mode-in-tabular-bisemantic-model-part-2.html 32
Wrap Up v 33
Tabular Model Modes In-Memory Mode (default) DirectQuery Mode DirectQuery only DirectQuery with In-Memory In-Memory with DirectQuery 34
Do I DirectQuery? SQL Server database is optimised Columnstore Index Partitioning Contains all required business rules High Volume Can import non-sql data into one SQL Server database which will be the source 35
Further Reading Cathy Dumas blog http://cathydumas.com/ SQLBI (Marco Russo & Alberto Ferrari) http://www.sqlbi.com/ DAX resources (with links to Tabular Model) http://social.technet.microsoft.com/wiki/contents/articles/1088.da x-resource-center.aspx Julie Koesmarno s blog http://www.mssqlgirl.com 36
Q & A signal@mssqlgirl.com http://www.mssqlgirl.com http://au.linkedin.com/in/juliekoesmarno @mssqlgirl 37
Thank you! v 38
MsSQLGirl.com 39 @mssqlgirl