Enterprise GIS: Designing, Testing, & Monitoring

FedGIS Conference February 24 25, 2016 Washington, DC Enterprise GIS: Designing, Testing, & Monitoring Jim VanOstenbridge, Solution Architect jvanostenbridge@esri.com Andrew Sakowicz, Enterprise Implementation Team Lead asakowicz@esri.com Martin Hamann, Senior Systems Architect mhamann@esri.com

Please Take Our Survey! Download the Esri Events app and find your event Select the session you attended Scroll down to find the survey Complete Answers and Select Submit

Agenda Design Best Practices System Tools Overview Case Study Questions

Best Practice: Conceptual Design Patterns of GIS Business Behavior System of Engagement (Demand) Location Enablement Community Engagement System of Record (Supply) Visualization Field Mobility Analysis Data Management Complete and Integrated Web Mapping Platform Geo-enable decisions through integration of GIS with external systems and services Get feedback and make informed decisions Disseminate information where and when it is needed Get information into and out of the field Empower users to discover, use, make & share maps on any device, anywhere, and at any time. Operations Development Transform data into actionable information Collect, organize and exchange data Traditional and IT-Centric GIS

SDKs / APIs APPS Best Practice: Conceptual Technical Design Solutions Location Enablement Community Engagement Visualization (Operational Awareness) Field Mobility Data Management Analysis Patterns of Business Behavior Users Groups Items Tags (Content and Capabilities) Web Maps Web Layers Tools Files SDKs / APIs Data Management Analysis Data and Storage Initial design for fulfillment of SLAs I N F R A S TR U C TU R E Identifying affinity for content, capabilities, and platform components SDKs / APIs P O R TA L LOAD BALANCING Visualization HIGH AVAILABILITY Next step: Translating conceptual architecture into logical and physical technical architecture PUBLICATION STRATEGY PRODUCTION STAGING WORKLOAD SEPARATION DEVELOPMENT A S S O C I A TE D B E S T P R A C TI C E S IT Governance Project Prioritization Workforce Development Security Enterprise Systems SDKs / APIs Categorically aligning business needs with technology E XTE R N A L S YS TE MS A N D S E R V I C E S Salesforce.com Microsoft Dynamics SAP HANA IBM Cognos MicroStrategy Microsoft Office Excel PowerPoint Other Social Weather Traffic IoT Sensors

Best Practice: Leverage System Tools As a Practice System Designer System Monitor System Test Software tools to help plan, test and monitor a system implementation. Maintain SLA s Transparency into system Tune system stability & availability Identify bottlenecks Reduce risks Optimize spend Improved capacity planning

System Tools System Designer Enterprise GIS System Architecture Andrew Sakowicz System Monitor End-to-End GIS Monitoring System Test Load Testing for GIS

Definitions

Performance Speed, e.g. response time (seconds)

Scalability The ability to increase output and maintain acceptable performance

Capacity The maximum level of output the system can produce, e.g. X cars/sec X maps/sec At capacity Over capacity

Bottleneck Resource(s) limiting the performance or capacity Not bottleneck Think of : Lanes -as CPU processor Toll -as ArcGIS Server instances Cars -as map requests bottleneck

Process and Tools

Relationship between System Tools User Load CPU% Capacity models Performance Tests

System Tools framework System Tools are not just tools Tool Patterns Discipline

Infrastructure Capacity Planning

Provide sufficient hardware resources Most systems are CPU bound GIS Systems are bound by: 1. CPU - typically 2. Memory when large number of services 3. Disk Image Service, Synchronization 4. Network low bandwidth deployment 5. Poorly configured virtualization can result in 30% or higher performance degradation Most well-configured and tuned GIS systems are CPU bound.

System Designer Solution Architecture design methodology Gathering requirements Designing Capacity: CPU, Network, Memory Reporting

Performance Testing

Testing process Application GIS Services Infrastructure: Hardware and Software

Required skill set Configuration, Tuning, Testing Tuning Configu ration Testing

System Test Tool features GIS Test Automation ArcGIS Services - Mapping - Feature Service - OGC - Geocoding - Image Service - Network Analyst - Geoprocessing - Tile Cache Application GIS Services Infrastructure: Hardware and Software Application Testing Discipline relevant report

Test tools feature comparison Tool Cost Learning Curve OS Metrics GIS Data Generation GIS Test Automation Load Runner High High Windows/Linux No No Visual Studio Medium High Windows No No JMeter Free High Requires additional plugin No No System Test Free Low Windows/Linux Yes Yes Tech Support by Esri PS as part of consulting support

Performance testing Value Identify bottlenecks Determine system capacity Demonstrate performance SLA

Demo System Test

System Monitoring

Monitoring Enterprise GIS Challenges Multiple administrators Multiple disparate monitoring/diagnostic tools Data collected in a reactive fashion: on demand and for limited time Correlation of data with different timestamp is difficult ArcGIS administrators do not have access to all tools, data and reports Challenging to quickly identify the root cause and take appropriate measures

Standards for effective GIS monitoring Many excellent monitoring tools on the market Few provide GIS dashboards System Monitor can be used as reference implementation

Enterprise GIS effective monitoring PIECE of mind with System Monitor Network Hardware Web Server Proactive Integrated - Dashboards across all tiers End-to-End ArcGIS Server Geodatabase RDBMS - All tier monitoring Continuous - %Coverage provided Extendable - Custom queries

Monitoring Value Proactive validation: Configuration Resource Utilization Usage Trends Performance SLA Uptime SLA

System Tools download http://www.arcgis.com owner:enterpriseimp Show ArcGIS Desktop Content

System Tools show case https://systemmonitoring-emcs.esri.com/portal

Case Study NOAA IDP GIS Martin Hamann

NOAA National Weather Service IDP GIS Program Disseminate NOAA geospatial data and products in industry-standard formats on appropriate infrastructure Platform as a Service (Paas) available to NOAA Data Stewards 24 x 7 High Availability (HA) infrastructure for critical weather and decision support datasets Service Level Agreement (SLA) Dedicated staffing and resources with tiered support model across GIS, IT, and weather information\forecasting domains

Requirements and Challenges High availability Fault Tolerance Scalability in a context of event driven traffic spikes Performance Security Interoperability Continuously updating data feeds Hosted content

Technical and Organizational Complexity Hybrid architectures integrating COTS and Open Source technologies RHEL virtualized data center managed using Puppet Labs software Domain expertise, system accesses, and roles are split between multiple organizations GIS content often developed externally and provided to the IDP GIS Technical Team for reconfiguration and implementation on IDP infrastructure Tiered support teams distributed across multiple domains (IT, GIS, Weather)

Live Data Service Update Frequency Performance Workflow Nominal Map/Image requests (Est) Peak Map/Image requests (Est) Watches\Warnings and Advisories Daily Global Precipitation 1 minute \ 10 minutes Daily Light Vector 10K Hr 100K Hr Light Raster 1K Hr 10K Hr Radar (1x1 km base reflectivity) 5 minutes Light Raster 20K Hr 200K Hr HRRR Heavy Raster UNK UNK Hurricane Tracks/Wind/Surge 15 minutes 1 hour 10 minutes or less Light Vector 1K Hr 100K Hr AHPS gauges 15 minutes Light Vector 1K Hr 10K Hr Flood Outlook Product CPC Weather Hazards Quantitative Precipitation Forecast (QPF) Daily Daily 15 minutes Light Vector Light Vector Light Vector 1K Hr 10K Hr 1K Hr 10K Hr 100K Hr 10K Hr Weather Features Sea Surface Temp NDFD Wind Velocity Forecasts Daily Daily 1 hour Light Raster Light Raster Light Raster 1K Hr 1K Hr 1K Hr 10K Hr 10K Hr 10K Hr

Lots of Moving Parts

Testing and Monitoring Onboarding QA of new services provided by external sponsors System architecture and capacity planning - Service level unit testing - Site/System level load testing - Scenario / failover testing Monitoring KPI to inform system configuration tuning and capacity planning decisions Monitoring alerts to identify, prevent, and quickly resolve outages on critical decision support services - Computing Infrastructure - GIS web services - Live data feeds must ensure data currency Continuous monitoring provides shared awareness that allows for effective communication and coordination between different stakeholders

Tools Big Brother (NOAA internal system of record) Nagios Check MK (SA s system of preference ) System Monitor System Test JMeter mxdperfstat

Big Brother NWS System of Record Tightly coupled to operational protocols/procedures 24 x 7 watch desk monitoring all operational systems For a system to attain operational status it must be integrated into Big Brother and pass a 30 day long term stability test Relatively basic interface but extensible framework for developing monitors and alerts using a variety of programming or scripting languages

Big Brother

Nagios - Check MK The data center SA s system of choice for infrastructure monitoring Some alerting/reporting configured but primarily an ad-hoc information source for investigating issues Enabled by default on most servers Views tied to ldap role/group membership based on least privilege access Rich set of default collectors for server and file system metrics but API for developing custom collectors has a learning curve and was only accessible to small group

Nagios - Check MK The data center SA s system of choice for infrastructure monitoring Some alerting/reporting configured but primarily an ad-hoc information source for investigating issues Enabled by default on most servers Views tied to ldap role/group membership based on least privilege Rich set of default collectors for server and file system metrics but API for developing custom collectors has a learning curve

Nagios - Check MK

System Monitor Includes preconfigured GIS with domain specific Key Performance Indicators (KPI) Is usually more accessible and configurable for GIS technical teams Configurable reports that allow side by side viewing of system metrics alongside GIS metrics

System Monitor GIS Server/Service Key Performance Indicators (KPI)

System Monitor (Reports) Tr/Sec Free

System Monitor (Reports) CPU Mem

System Test Individually test services that we expect to receive high traffic to define a service throughput baseline given a certain min/max instance configuration Load test groups of servers and services to establish a system baseline to given a specific set of services and servers (e.g. a half stack test of max throughput for a set of services associated with a scenario such as a tropical storm or major snowfall) Investigate/analyze performance for services that have been alerting as slow or nonresponsive in Big Brother Develop test extents/parameter files to be exported using custom python scripts so that they can be configured in long running system wide load tests performed by the SA s using JMeter

System Test

JMeter NWS data center is primarily a linux environment SA s ultimately responsible for system wide load testing were more comfortable with Linux tools Limited access to Windows systems Conducted system wide load testing using JMeter (with parameter input generated in System Test) and cross referenced results to System Test

JMeter

mxdperfstat Been around for a while and still very useful Easy to batch script All service sponsor provided map documents were tested using mxdperfstat to identify cases where poor configuration could impact server performance

mxdperfstat 1.6 seconds may not seem horrible but this service is expected to see very heavy traffic and accounts for 6 instances on 8 servers. Improvements to the cartography could make a noticeable difference in overall server performance and/or free resources for other services

FedGIS Conference February 24 25, 2016 Washington, DC Implementing Enterprise GIS: Questions Jim VanOstenbridge, Solution Architect jvanostenbridge@esri.com Andrew Sakowicz, Enterprise Implementation Team Lead asakowicz@esri.com Martin Hamann, Senior Systems Architect mhamann@esri.com

Don t forget to complete your digital session survey Download the Esri Events app!

FedGIS Conference February 24 25, 2016 Washington, DC Implementing Enterprise GIS: Thank You! Jim VanOstenbridge, Solution Architect jvanostenbridge@esri.com Andrew Sakowicz, Enterprise Implementation Team Lead asakowicz@esri.com Martin Hamann, Senior Systems Architect mhamann@esri.com

FedGIS Conference February 24 25, 2016 Washington, DC Implementing Enterprise GIS: Designing, Testing, & Monitoring Jim VanOstenbridge, Solution Architect jvanostenbridge@esri.com Andrew Sakowicz, Enterprise Implementation Team Lead asakowicz@esri.com Martin Hamann, Senior Systems Architect mhamann@esri.com