SQLintersection SQL123 SQL Server Monitoring is my Superpower David Pless David.Pless@Microsoft.com
Introduction Overview Key Performance Monitor Counters Creating Custom Perfmon Counters Wait Statistics Building a Enterprise Policy Management Framework solution from PBM SSRS Monitoring Dashboards and Operational Insights Technologies Covered Performance Monitor (perfmon) PowerShell SQL Server DMVs SSRS / Power BI Policy Based Management
Quick Poll What perfmon counters do you pay attention to? Do you perform regular performance baselines? What tools do you use to monitor your environment?
Key Perfmon Counters (Keep it Simple) Memory OS Available MBs / Process SQL Page Life Expectancy (Compare to Lazy Writer) Disk Logical / Physical Disk Avg. Disk sec/read /Write / Process CPU OS % Processor Utilization, % Privileged Time Process (SQL) - % Processor Utilization, % Privileged Time Network(*) Look for Errors, Discarded Packets, Drops; Bandwidth TempDB SQL Server: Transactions Free Space in TempDB(KB) Longest Running Transaction Time Version Store Data Be aware of Red Herring Counters like % of Disk Time, % Processor Time, and Buffer Cache Hit Ratio
Comparison Example: System Process to SQL Process
New SQL Server 2014 Perfmon Counters In Memory OLTP (XTP*) XTP Transaction Log XTP Transactions SQLServer:Databases Group Commit Time/sec XTP Memory Used (KB) Buffer Pool Extension* (examples) Extension in use as percentage SQLServer:HTTP Storage (examples) Reads/Sec Total Bytes/Sec SQLServer:Deprecated Features (8) Resource Governor Resource Pool Stats (examples) Avg Disk Read/Write IO (ms) Avg Disk Read/Write IO (ms) Base Disk Read/Write Bytes/sec Disk Read (Write IO/sec Workload Groups (examples) Disk (Write) Read IO/sec Disk (Write) IO/sec Disk (Write) IO Throttled/sec Active Requests Active Parallel Tasks Blocked tasks
Basic Performance Monitor Rules SQL Server Memory / Buffer Manager is the What and SQL Server Access Methods is the Why There is a simple rule.. Memory pressure leads to Disk and CPU pressure means there is a problem somewhere There is only one TempDB per SQL Server instance. If its sick, then SQL Server is sick You should never make a determination off of a single examination of perfmon counters (Baseline and Compare to Waits / DMVs) It is important to identify workloads (EOD/EOM/EOQ/EOY Nightly ETL, etc.) There is always something other than SQL Server engine running on the server (AV, SSIS, Monitoring, etc.)
T2 T1 Wait Statistics Execution Model Status: Running session_id 60 51 Runnable Queue (Signal Waits) Status: Runnable session_id 51 session_id 64 session_id 87 session_id 52 session_id 93 56 IO_Completion Runnable Running Runnable Runnable Runnable Runnable Runnable Wait Queue (Resource Waits) Status: Suspended session_id 73 session_id 59 session_id 56 session_id 55 session_id 60 LCK_M_S NETWORKIO CXPACKET Runnable RESOURCE_SEMAPHORE IO_Completion T0
Examining Wait Statistics T0 Wait Queue T1 Runnable Queue T2 Time Spent on Resources (suspended) Time Spent waiting on CPU (runnable) Total Time For Execution
Perfmon Correlations Wait Statistics Get Snapshots XEvents / System Health Can measure PLE, Foreign NUMA memory use, Wait Stats, perfobject_, etc. System Health Examine for the Captured Window DMVs Blocking Active queries sys.dm_exec_requests (Now) sys.dm_os_waiting_tasks (Now) sys.dm_exec_query_stats (Cache) sys.dm_os_wait_stats (Baseline)
Methods to Capture Perfmon (User Defined, Data Collection Sets) SQL Server Performance DMVs PowerShell SQLDIAG / PSSDIAG Produces.blg file for perfmon analysis Has the ability to capture SQL Trace Files (.trc) Captures Error Logs, blocking output, system health information Captures SQL Server Perf Stats Script (SQL Nexus) and allows custom diagnostics Extended Events (SQL Server related counters: Example: buffer_node_page_life_expectancy)
Demo Creating a Perfmon Template Creating a Repeatable Troubleshooting Solution
Demo Using PowerShell to Collect PERFMON Data From PowerShell to Historical Reporting
Demo Using SYS.DM_OS_PERFORMANCE_COUNTERS Leveraging DMVs to Consume Performance Data
The DBA Toolkit Preset Perfmon Templates targets to SQL Server and associated application servers Predefined DMV scripts focused on Wait statistics, expensive statements, concurrency, AlwaysOn Availability Group Health, and more Tools designed to help troubleshoot a SQL Server instance such as the SQL Performance Dashboard Reports A preconfigured SQLDIAG / PSSDIAG ready to capture system data for troubleshooting
Demo Leveraging the DBA Toolkit Examining Perfmon Waits, Currently Running DMVs for Performance (TempDB, async_network_io, Blocking)
PAL (Performance Analysis of Logs) PAL analyses counter logs (.csv or.blg) and creates an HTML report Free and public download available at http://pal.codeplex.com The PAL installation uses Windows Installer (.msi) The PAL 2.0 has the following prerequisites: Windows PowerShell 2.0 (free).net Framework 3.5 (free) Microsoft Chart Controls for.net Framework 3.5 (free) Not supported by Microsoft Written by Clint Huffman clinth@microsoft.com For support, use the forums at http://pal.codeplex.com Designed to be used for remote analysis PAL supports most of the major Microsoft products Export thresholds to a counter log template Threshold files are exposed and can be edited to fit your baseline
PAL Updates Completely rewritten as of 2.6 Much improved scalability and overall performance A threshold file that takes 1 hour on 2.4 would take 15~ minutes on 2.6 SQL Server Business Intelligence Coming Soon Review by John Desch and other BI experts SQL Server 2012 / 2014 Threshold File (New Release being Tested) Resource Governor Custom Counters Full Deprecation Tracking In Memory OLTP Buffer Pool Extensions
Demo Using (Performance Analysis of Logs) PAL Using PAL and Examining the Output
Leveraging Custom Counters in SQL Server SQL Server allows for 10 custom counters by default in SQL Server. These counters are visible in perfmon. Scenarios for leveraging custom counters: Monitoring the USER_TOKENPERM CACHE Monitor the maximum database VLF Count Components unique to your SQL Server environment For Example: The number of customer orders logged or the product inventory The number of planes in the air, versus on the ground, versus in maintenance, etc. The custom counters can then be rolled into other monitoring solutions from 3 rd Party products, SCOM, SQL Server Agent Alerts, and more!
Leveraging Custom Counters in SQL Server These counters could monitored and alerted on via SQL Server Agent alerts and/or SCOM. Usage: EXEC sp_user_counter(1 10) accepts an int value sp_helptext sp_user_counter3 CREATE PROCEDURE sys.sp_user_counter3 @newvalue int as DBCC SETINSTANCE ('SQLServer:User Settable', 'Query', 'User counter 3', @newvalue) Reference: SQL Server, User Settable Object http://msdn.microsoft.com/en-us/library/ms187480(v=sql.110).aspx http://blogs.msdn.com/b/askjay/archive/2010/02/18/creating-a-custom-performance-monitorcounter-for-sql-server.aspx
Demo Leveraging Custom Counters in SQL Server Tracking Blocking Chains, Monitoring the top VLF Count, and More
DEMOS(s): The Contoso Air Application Premise: Contoso Air is an Atlanta based airline company who stores their flight status in SQL Server. They are looking for a low overhead solution to be able to track the status of their aircraft while they are in the air and when they are being maintained as well as the associated costs.
DEMOS(s): The Contoso Air Application For this discussion consider the following user counters: User Counter 6 - Flying User Counter 7 - Grounded User Counter 8 - Maintenance User Counter 9 - Decommissioned
SQL Server Agent Alerting (Custom Counters)
SQL Server Management Studio Custom Reporting SQL Server Management Studio allows running custom reports SSMS interprets the.rdl and executes the reports along with the current list of built-in standard reports There are extended SSMS reports available to increase monitoring capability in SQL Server Example: The SQL Server 2012 Performance Dashboard Reports are Reporting Services report files designed to be used with the Custom Reports feature of SQL Server Management Studio. http://www.microsoft.com/en-us/download/details.aspx?id=29063 Limitation: You cannot prompt the user for parameters, use document maps, you must run the report from the context of the query in Object Explorer, and other limitations
Policy Based Management Building Blocks Overview of Central Management Server Overview of Policy Based Management Microsoft Best Practice Policies Creating Custom Policies Enterprise Policy Management Framework Expanding Report Solutions
Overview of CMS The CMS Store needs to be SQL Server 2008+ Uses Windows Authentication (For Server Registration) CMS cannot register itself* Stores metadata in MSDB Registration can be easily shared Registered Servers can be in more can one group
PBM - Terminology Facet Conditions Policy Target Category
Varies by Facet Very Few Can Do On Change PBM Evaluation Modes Prevents use of DDL Triggers Log Only goes to the Windows Event Log All Facets support On Demand and On Schedule Evaluation modes are stored in dbo.syspolicy_management_facet for each facet 4 = On Demand and On Schedule 6 = On Demand, On Schedule, and On Change: Log 7 = On Demand, On Schedule, and On Change: Log, and On Change: Prevent
Custom Policies ExecuteSQL() Facets cannot cover everything Expressed as T-SQL ExecuteSQL(string returntype, string SQLQuery) returntype Numeric, String, Bool, DateTime, Array and GUID sqlquery String that represents the query to be run Return Value The first column in the first row of the result set returned by the Transact-SQL query Example: ExecuteSQL ( Numeric, SELECT COUNT(*) FROM msdb.dbo.sysjobs ) <> 0
Custom Policies ExecuteWQL() Taps into WMI Expressed as WQL ExecuteWQL(string returntype, string namespace, string WQL) returntype Numeric, String, Bool, DateTime, Array, GUID Namespace Is the WMI Namespace to execute against WQL Is the string that contains the WQL to be executed Example: ExecuteWQL( Numeric, root\cimv2\power, SELECT ElementName from w32_powerplan WHERE isactive= true )
Demo Policy Based Management VLF Count, AlwaysOn Tests, and other Key Policies
Enterprise Policy Management Framework The Enterprise Policy Management Framework (EPM) is a solution to extend SQL Server Policy- Based Management to all versions of SQL Server in an enterprise Centralize and report on the policy evaluation results. SQL Server Reporting Services reports will deliver information from the centralized table. This solution will require at least one instance of SQL Server 2012+.
Demo Enterprise Policy Management Framework Walk through of an EPMF Deployment
Azure Operational Insights Analysis Service Enables IT Admins to search through machine data Gain deeper insight into each environment Interact with data through Intelligence Packs & Search Works against Windows Server 2008+ Requires a Monitoring Agent Operational Insights for Azure Virtual machine SCOM 2012 SP1 UR2+ Sign up at: http://opinsights.azure.com/
Operational Insights Deployment Options Connect Servers Directly Download Microsoft Monitoring Agent Install and Connect Agent to the Workspace Azure Portal Manage.WindowsAzure.com Create Operational Insights Workspace Enable Operational Insights in your VMs Integration with System Center Operations Manager Through operations console Can be completed in bulk through PowerShell
Growing list of solutions designed to help customers address common scenarios Currently all built by Microsoft and free for use The more you use, more data you acquire Some require configuration Operation Insights Solutions Gallery
Operational Insights Mobile Exposure We have started detailed planning work for both ios and Android releases.
Key Issues - Troubleshooting Operational Insights Difference in times between computers and the service Proxy issues between systems If you are using SCOM w/ OpMgr, make sure the MPs are downloaded and installed on the OpsConsole In OpsMgr check the Operational Insights Managed link to see machines being monitored Check Management Packs node by date to see OpsInsight Gallery Items For direct connect machines check your Microsoft Monitoring Agent Properties Master Blog Post: http://aka.ms/opinsightsonboardingtrouble
Demo Azure Operational Insights Walk through of an Operational Insights Active Deployment
Questions? Don t forget to complete an online evaluation on EventBoard! SQL Server Monitoring is my Superpower Your evaluation helps organizers build better conferences and helps speakers improve their sessions. Thank you!
References Microsoft SQL Server 2012 Performance Dashboard Reports http://www.microsoft.com/en-us/download/details.aspx?id=29063 Loading Data With PowerShell http://www.sqlservercentral.com/articles/powershell/65196/ out-sql Powershell function - export pipeline contents to a SQL http://blogs.technet.com/b/sqlthoughts/archive/2008/10/03/out-sql-powershellfunction-export-pipeline-to-a-new-sql-server-table.aspx http://blogs.technet.com/b/heyscriptingguy/archive/2010/11/01/use-powershell-tocollect-server-data-and-write-to-sql.aspx SQLDIAG / PSSDIAG http://sqlnexus.codeplex.com/wiki/pageinfo?title=sql2005perfstatsscript
References Command-Line Options for Faster Access to Performance Data http://technet.microsoft.com/en-us/magazine/ff723776.aspx Compare Multiple Log Files in Performance Monitor http://technet.microsoft.com/en-us/library/cc748917.aspx Customizing the Default Counters for Performance Monitor http://www.sqlskills.com/blogs/erin/customizing-the-default-counters-for-performance-monitor/ Performance Analysis of Logs http://pal.codeplex.com SQL Server, User Settable Counters http://msdn.microsoft.com/en-us/library/ms187480(v=sql.110).aspx http://blogs.msdn.com/b/askjay/archive/2010/02/18/creating-a-custom-performance-monitorcounter-for-sql-server.aspx
Operational Insights References Main Page: http://azure.microsoft.com/en-us/services/operational-insights/ TechEd Session: http://channel9.msdn.com/events/teched/europe/2014/cdp-b343 Team Blog: http://blogs.technet.com/b/momteam/ Twitter: @OpInsights Windows Phone App: http://aka.ms/opinsightsapp Blog - Connect Servers Directly: http://aka.ms/opinsightsdirectconnect Blog - Enable Operational Insights for Azure VMs: http://aka.ms/opinsightsazurevms Blog - Search How-To: http://aka.ms/opinsightssearch Product Feedback: http://aka.ms/opinsightsfeedback