Building a BI Solution in the Cloud Stacia Varga, Principal Consultant Email: stacia@datainspirations.com Twitter: @_StaciaV_
2 SQLSaturday #467 Sponsors
Stacia (Misner) Varga Over 30 years of IT experience, 17 years of BI experience Frequently speaks, writes, and teaches about Microsoft BI technologies Principal Consultant and Founder, Data Inspirations Data Platform MVP and SSAS Maestro Las Vegas User Group Chapter Leader and SQL Saturday Organizer Wants you to make sure you can understand what your data is telling you
Agenda The Current Microsoft BI Landscape What s New in SQL Server 2016 BI On Premises BI Client Excel and Power BI The Gateway to Your Data Making this Secure
Classic Data Warehousing Uses Just a Subset Data Warehouse & BI Solutions ETL 5
Power BI 1.0 Similar to the combination of SharePoint 2013 and Power View/Pivot Added Q&A Functionality Limited Mobile Functionality Cost prohibitive for many customers Tied to Office 365
SaaS Data Sources Power BI Current Architecture Power BI web browser With natural language query, visual exploration, etc. Azure- resident systems On- premise systems / other Azure SQL/DW Azure Stream Analytics HDInsight Corporate Data Sources AS Tabular Excel PBI Desktop Power BI AS Connector Power BI Personal Gateway Power BI Mobile Apps with dashboards, alerting and visual exploration Real time Dashboards using Push API 8
Infrastructure as a Service (IaaS) VM running SQL Server, Virtual Network (IaaS)
Platform as a Service (PaaS) SQL Database, HDInsight, Storage (PaaS)
Hybrid Cloud VM running SQL Server, Virtual Network (IaaS) SQL Database, HDInsight, Storage (PaaS) SSIS Server Domain Controller (OnPrem)
Software as a Service (SaaS)
Cloud- Only Model Office 365 Azure SQL Database Virtual Machine running SQL Server
Hybrid Model #1 Office 365 (SaaS) VM running SQL Server, Virtual Network (IaaS) SQL Database, HDInsight, Storage (PaaS) Data Management Gateway On-Premises Assets Client Domain Controller SQL Server Analysis Services
Azure Virtual Machines Benefits Shut down during non- business hours during development cycle Scale up as workloads increase IP Addressing Dynamic Static (although properties show as dynamic)
Azure Virtual Machines Data Warehouse Image Options Load an image from an on- premises source Use predefined image o Up to 500GB for SQL Server 2012 o Up to 1TB for SQL Server 2014
Azure Virtual Machines Performance Options o Spread I/O over multiple filegroups and data files o Use AlwaysOn Availability Groups in asynchronous replication mode o Implement Azure virtual load balancing o Configure read- only routing
Azure Virtual Machines BI Resources (SSIS, SSAS, SSRS) o Spread over multiple VMs to reduce latency and scale out as needed o Use same virtual network and affinity group as DW to keep all components together during maintenance
Hybrid Model #2
Hybrid Model #3 Office 365 (SaaS) SQL Database, HDInsight, Storage (PaaS) Data Management Gateway On-Premises Assets Client Domain Controller SQL Server Analysis Services
Hybrid Model #4 Office 365 (SaaS) Data Management Gateway On-Premises Assets Client Domain Controller SQL Server Analysis Services
Cloud BI
SQL Database One of the earliest Azure offerings (SQL Azure) Originally a subset of both T- SQL language and SQL engine Now nearly code complete with SQL Server Some features are ahead of current release (SQL 2014)
SQL Data Warehouse Saas Azure Public Cloud Office 365
Architecture Control Node Query Tools ETL Tools MPP Engine Data Movement Service Hadoop SQL Server TempDB Configs Shell Db Compute Node Compute Nodes Compute Nodes Data Movement Service SQL Server User Data
Where does SQL Data Warehouse fit? OLTP / DW workloads Lift and Shift Customer managed OLTP/ DW workloads Net new development Fully managed service DW workloads only Fully managed Dynamic compute scaling and pause 1GB- 1TB+ 1GB- 1TB 250GB PB+
Azure Data Lake
ETL Strategies Driver={SQL Server Native Client 11.0}; Server=tcp:myservername.database.windows.net;Database = AzureAdventureWorks_staging;UID=dbuser;Pwd=mypassword ; Data Connections ETL Design Patterns for the Cloud Data Subsets File Compression
Data Connections Virtual Machine or On- Premises SQL Database Driver={SQL Server Native Client 11.0}; Server=tcp:myservername.database.windows.net;Database= AzureAdventureWorks_staging;UID= dbuser;pwd=mypassword; ODBC ADO.NET
ETL Design Patterns for the Cloud SELECT AuditKey from [dw].[dimaudit] where TableName = @tablename and ExecStartDT = @execstartdate Use parameter names rather than? for ADO.NET Execute SQL Task Sensitive Data Lookup MERGE dw.dimproductcategory AS target USING (SELECT Name, rowguid, ModifiedDate, ProductCategoryID, AuditKey FROM tmp.scdproductcategory) AS SOURCE MERGE Use? for ODBC query but start parameter mapping with 1
Data Subsets select IsNull(Max(ProductAlternateKey), @minkey) from dw.dimproduct "SELECT [ProductID],[Name],[ProductNumber],[MakeFlag],[FinishedGoodsFlag]... FROM [Production].[Product] WHERE [ProductNumber] > '" + @[User::MaxKey] + "' ORDER by [ProductNumber]"
File Compression Blob storage SQL Database On- Premises SQL Server (or Alternate Data Center) SQL Virtual Machine Virtual machine file system
Azure Data Factory ETL in the Cloud ETL Service for Data Transformation in Azure Priced by service you pay for what you use Still immature relative to SSIS Less Tooling Fewer Supported Data Sources More Custom Code
Cloud vs On- Premises Short Term Use Rapid Scale Test Use Cases Pay as you go Internet data source On- Premises Large long term implementations Well known workloads Shared clusters Large initial investment 3 4
SQL Server 2016 BI New Features
On- Premises BI SQL Server 2014 SQL Server Excel 2013 SQL Server Reporting Services SQL Server Analysis Services Tabular Multidimensional SQL Server Integration Services SharePoint 2013 Excel Power Pivot Power Query Power Map Power View Power BI Desktop No native on- premises option Pyramid Analytics
SQL Server 2016 BI Enhancements SSRS HTML5 Sunburst and Treemap Charts Modern Report Builder Power BI Compatibility Datazen integration
SQL Server 2016 BI Enhancements SSAS Great deal of enhancements to DAX New Functions Parallel Processing for Tabular models Better performance for DirectQuery GUI support for Extended Events
SQL Server 2016 BI Enhancements SSIS OData v4 Support Enhanced Logging Lots of New Connectors AlwaysOn Support Incremental Package Deployment Support
SQL Server 2016 BI Enhancements Engine Revolution Analytics integration Columnstore Enhancements In- memory columnstore JSON Support
BI Clients
SQL Server Data Tools and BIDs Finally One Unified Toolkit in Visual Studio One Visual Studio for All the Versions and SKUs SSMS Support for DAX
Power BI Desktop Independent of Excel so it can evolve faster Has its own In- Memory engine Supports DAX Complete dashboard design experience Revs every month so stay on top of it Limited Support for On- Premises Solutions
Power BI in Excel 2013 Power Query Get, filter, and shape data Power Pivot Get, filter, and relate data, add calculations to model Power View Interactively analyze & map data model Power Map Interactively explore data model in 3D map
Excel 2016 Get and Transform (Power Query) Power Pivot 3D Maps (Power Map) Power View
Gateways Personal Gateway Enterprise Gateway SQL Server Analysis Services Connector Note Personal and Enterprise can run on same machine
Single Sign- On Active Directory Azure Active Directory Active Directory Federation Services DirSync > Azure AD Sync Services Active Directory Certificate Services
Active Directory and Azure IaaS Domain Controller (DC) in Azure supports authentication if network connection to on- premises DC AD Federation supports Single Sign- On for SQL Database, Azure Management Portal, and Azure applications
Azure Security Multi- tenant sever environment security features similar to Office 365 Some security elements managed by tenant VPN connections DNS servers Internet connectivity
Hybrid Model with PaaS PaaS Components SQL Database HDInsight Security managed by control of inbound IP addresses
Hybrid Model with IaaS IaaS Components Virtual Machine Virtual Network Security managed by VPN implementation
Virtual Network Supports access to IaaS resources Requires assignment to new or existing affinity group Optionally supports on- premises or public DNS server for name resolution
Azure VPNs POINT- TO- SITE VPN Certificate secures connection between on- premises and cloud resource Easy option for quick start POINT- TO- POINT VPN Option 1: Windows server running Routing and Remote Access Services Option 2: Approved VPN device on- premises Option 3: ExpressRoute
Summary Modern Business Intelligence has a lot of moving parts Power BI simplifies combining them SQL Server 2016 has big gains for BI It takes a village to make this enterprise ready
Security and Azure SQL Database http://tinyurl.com/h3jw5m3 Using Power BI in a Hybrid Environment http://tinyurl.com/phowgpf Whitepapers