Building a BI Solution in the Cloud

Transcription

1 Building a BI Solution in the Cloud Stacia Varga, Principal Consultant [email protected]

2 2 SQLSaturday #467 Sponsors

3 Stacia (Misner) Varga Over 30 years of IT experience, 17 years of BI experience Frequently speaks, writes, and teaches about Microsoft BI technologies Principal Consultant and Founder, Data Inspirations Data Platform MVP and SSAS Maestro Las Vegas User Group Chapter Leader and SQL Saturday Organizer Wants you to make sure you can understand what your data is telling you

4 Agenda The Current Microsoft BI Landscape What s New in SQL Server 2016 BI On Premises BI Client Excel and Power BI The Gateway to Your Data Making this Secure

5 Classic Data Warehousing Uses Just a Subset Data Warehouse & BI Solutions ETL 5

6

7 Power BI 1.0 Similar to the combination of SharePoint 2013 and Power View/Pivot Added Q&A Functionality Limited Mobile Functionality Cost prohibitive for many customers Tied to Office 365

8 SaaS Data Sources Power BI Current Architecture Power BI web browser With natural language query, visual exploration, etc. Azure- resident systems On- premise systems / other Azure SQL/DW Azure Stream Analytics HDInsight Corporate Data Sources AS Tabular Excel PBI Desktop Power BI AS Connector Power BI Personal Gateway Power BI Mobile Apps with dashboards, alerting and visual exploration Real time Dashboards using Push API 8

9 Infrastructure as a Service (IaaS) VM running SQL Server, Virtual Network (IaaS)

10 Platform as a Service (PaaS) SQL Database, HDInsight, Storage (PaaS)

11 Hybrid Cloud VM running SQL Server, Virtual Network (IaaS) SQL Database, HDInsight, Storage (PaaS) SSIS Server Domain Controller (OnPrem)

12 Software as a Service (SaaS)

13 Cloud- Only Model Office 365 Azure SQL Database Virtual Machine running SQL Server

14 Hybrid Model #1 Office 365 (SaaS) VM running SQL Server, Virtual Network (IaaS) SQL Database, HDInsight, Storage (PaaS) Data Management Gateway On-Premises Assets Client Domain Controller SQL Server Analysis Services

15 Azure Virtual Machines Benefits Shut down during non- business hours during development cycle Scale up as workloads increase IP Addressing Dynamic Static (although properties show as dynamic)

16 Azure Virtual Machines Data Warehouse Image Options Load an image from an on- premises source Use predefined image o Up to 500GB for SQL Server 2012 o Up to 1TB for SQL Server 2014

17 Azure Virtual Machines Performance Options o Spread I/O over multiple filegroups and data files o Use AlwaysOn Availability Groups in asynchronous replication mode o Implement Azure virtual load balancing o Configure read- only routing

18 Azure Virtual Machines BI Resources (SSIS, SSAS, SSRS) o Spread over multiple VMs to reduce latency and scale out as needed o Use same virtual network and affinity group as DW to keep all components together during maintenance

19 Hybrid Model #2

20 Hybrid Model #3 Office 365 (SaaS) SQL Database, HDInsight, Storage (PaaS) Data Management Gateway On-Premises Assets Client Domain Controller SQL Server Analysis Services

21 Hybrid Model #4 Office 365 (SaaS) Data Management Gateway On-Premises Assets Client Domain Controller SQL Server Analysis Services

22 Cloud BI

23 SQL Database One of the earliest Azure offerings (SQL Azure) Originally a subset of both T- SQL language and SQL engine Now nearly code complete with SQL Server Some features are ahead of current release (SQL 2014)

24 SQL Data Warehouse Saas Azure Public Cloud Office 365

25 Architecture Control Node Query Tools ETL Tools MPP Engine Data Movement Service Hadoop SQL Server TempDB Configs Shell Db Compute Node Compute Nodes Compute Nodes Data Movement Service SQL Server User Data

26 Where does SQL Data Warehouse fit? OLTP / DW workloads Lift and Shift Customer managed OLTP/ DW workloads Net new development Fully managed service DW workloads only Fully managed Dynamic compute scaling and pause 1GB- 1TB+ 1GB- 1TB 250GB PB+

27 Azure Data Lake

28 ETL Strategies Driver={SQL Server Native Client 11.0}; Server=tcp:myservername.database.windows.net;Database = AzureAdventureWorks_staging;UID=dbuser;Pwd=mypassword ; Data Connections ETL Design Patterns for the Cloud Data Subsets File Compression

29 Data Connections Virtual Machine or On- Premises SQL Database Driver={SQL Server Native Client 11.0}; Server=tcp:myservername.database.windows.net;Database= AzureAdventureWorks_staging;UID= dbuser;pwd=mypassword; ODBC ADO.NET

30 ETL Design Patterns for the Cloud SELECT AuditKey from [dw].[dimaudit] where TableName and ExecStartDT Use parameter names rather than? for ADO.NET Execute SQL Task Sensitive Data Lookup MERGE dw.dimproductcategory AS target USING (SELECT Name, rowguid, ModifiedDate, ProductCategoryID, AuditKey FROM tmp.scdproductcategory) AS SOURCE MERGE Use? for ODBC query but start parameter mapping with 1

31 Data Subsets select from dw.dimproduct "SELECT [ProductID],[Name],[ProductNumber],[MakeFlag],[FinishedGoodsFlag]... FROM [Production].[Product] WHERE [ProductNumber] > '" + "' ORDER by [ProductNumber]"

32 File Compression Blob storage SQL Database On- Premises SQL Server (or Alternate Data Center) SQL Virtual Machine Virtual machine file system

33 Azure Data Factory ETL in the Cloud ETL Service for Data Transformation in Azure Priced by service you pay for what you use Still immature relative to SSIS Less Tooling Fewer Supported Data Sources More Custom Code

34 Cloud vs On- Premises Short Term Use Rapid Scale Test Use Cases Pay as you go Internet data source On- Premises Large long term implementations Well known workloads Shared clusters Large initial investment 3 4

35 SQL Server 2016 BI New Features

36 On- Premises BI SQL Server 2014 SQL Server Excel 2013 SQL Server Reporting Services SQL Server Analysis Services Tabular Multidimensional SQL Server Integration Services SharePoint 2013 Excel Power Pivot Power Query Power Map Power View Power BI Desktop No native on- premises option Pyramid Analytics

37 SQL Server 2016 BI Enhancements SSRS HTML5 Sunburst and Treemap Charts Modern Report Builder Power BI Compatibility Datazen integration

38 SQL Server 2016 BI Enhancements SSAS Great deal of enhancements to DAX New Functions Parallel Processing for Tabular models Better performance for DirectQuery GUI support for Extended Events

39 SQL Server 2016 BI Enhancements SSIS OData v4 Support Enhanced Logging Lots of New Connectors AlwaysOn Support Incremental Package Deployment Support

40 SQL Server 2016 BI Enhancements Engine Revolution Analytics integration Columnstore Enhancements In- memory columnstore JSON Support

41 BI Clients

42 SQL Server Data Tools and BIDs Finally One Unified Toolkit in Visual Studio One Visual Studio for All the Versions and SKUs SSMS Support for DAX

43 Power BI Desktop Independent of Excel so it can evolve faster Has its own In- Memory engine Supports DAX Complete dashboard design experience Revs every month so stay on top of it Limited Support for On- Premises Solutions

44 Power BI in Excel 2013 Power Query Get, filter, and shape data Power Pivot Get, filter, and relate data, add calculations to model Power View Interactively analyze & map data model Power Map Interactively explore data model in 3D map

45 Excel 2016 Get and Transform (Power Query) Power Pivot 3D Maps (Power Map) Power View

46 Gateways Personal Gateway Enterprise Gateway SQL Server Analysis Services Connector Note Personal and Enterprise can run on same machine

47 Single Sign- On Active Directory Azure Active Directory Active Directory Federation Services DirSync > Azure AD Sync Services Active Directory Certificate Services

48 Active Directory and Azure IaaS Domain Controller (DC) in Azure supports authentication if network connection to on- premises DC AD Federation supports Single Sign- On for SQL Database, Azure Management Portal, and Azure applications

49 Azure Security Multi- tenant sever environment security features similar to Office 365 Some security elements managed by tenant VPN connections DNS servers Internet connectivity

50 Hybrid Model with PaaS PaaS Components SQL Database HDInsight Security managed by control of inbound IP addresses

51 Hybrid Model with IaaS IaaS Components Virtual Machine Virtual Network Security managed by VPN implementation

52 Virtual Network Supports access to IaaS resources Requires assignment to new or existing affinity group Optionally supports on- premises or public DNS server for name resolution

53 Azure VPNs POINT- TO- SITE VPN Certificate secures connection between on- premises and cloud resource Easy option for quick start POINT- TO- POINT VPN Option 1: Windows server running Routing and Remote Access Services Option 2: Approved VPN device on- premises Option 3: ExpressRoute

54 Summary Modern Business Intelligence has a lot of moving parts Power BI simplifies combining them SQL Server 2016 has big gains for BI It takes a village to make this enterprise ready

55 Security and Azure SQL Database Using Power BI in a Hybrid Environment Whitepapers