ArcGIS for Server: Administrative Scripting and Automation Shreyas Shinde Ranjit Iyer Esri UC 2014 Technical Workshop
Agenda Introduction to server administration Command line tools ArcGIS Server Manager Overview of the administrative API Automation Popular scripting workflows - Publishing of services - Setting up permissions - Extracting popular extents - Monitoring services and databases - Scaling and statistics
Introduction to server administration ArcGIS Server Installation Media Provision hardware Select a deployment pattern Keep it running Software Authorization ArcGIS Server Setup
The administrator s toolbox ArcGIS Server Administrator Directory ArcGIS Server Manager Console tools
Command line tools Located under /arcgis/server/tools A collection of most popular tools Categories - Administrative credentials - For resetting administrative accounts - SSL/certificate management - Backup and restore - Managing cache Lock the tools down using OS file permissions!
Demo Command line tools Esri UC 2014 Technical Workshop ArcGIS for Server: Administrative Scripting and Automation
ArcGIS Server Manager Browser based interactive application Located at: http://server.domain.com:6080/arcgis/manager Simple, easy-to-use Ability to manage your entire server site - Even though you are accessing Manager on a specific server node Great for common workflows!
Demo le Tour de Manager Esri UC 2014 Technical Workshop ArcGIS for Server: Administrative Scripting and Automation
ArcGIS Server Manager under the hood Simple HTTP requests to the administrative API
Overview of the administrative API Entire server administration is an API - If ArcGIS Server Manager can use it, so can you! RESTful - GET and POST of JSON data Well organized and well documented Lots of samples Scriptable Designed for automation!
The administrative API Create and manage your services. Start or stop them as wanted Manage system properties, web adaptors, server directories. System environment Services Machines + Clusters Add new nodes to grow capacity. Remove them to shrink. Organize machines into clusters. Get fine grained statistics on your GIS services. Admin API Statistics (new at 10.3) Manage all your database connections through a single interface. Databases Security Harvest logs for auditing, understanding usage patterns and troubleshooting. Logs Configure GIS tier or web tier authentication. Manage accounts, roles and permissions on services. Configure SSL.
Why automate? Common repeated workflows - Plot popular extents for the data Scheduled actions - Restarting services when data is updated - Build map caches during low demand by moving machines to caching cluster Reactive actions - Adding capacity (server nodes) to a cluster when demand increases - Send email notifications when errors are logged Works with all programming/scripting languages that support HTTP!
Where should you begin?
Demo Publish Services Esri UC 2014 Technical Workshop ArcGIS for Server: Administrative Scripting and Automation
Motivation Publishing many services Moving services between Dev, Staging, Production environments - Backup/restore is an option but sometimes you just want to republish services Can publish from Desktop and Manager Tedious if you have lots of services Automation is your friend!
Task : Publish a list of services listed in a file Help system has a script that uses ArcPy ArcPy is an python library that complements the Admin REST API Provides many convenience functions
Caveats Works great, just need a few lines of code However, you need ArcPy on the machine Script publishes service definitions sequentially - For every SD - Upload - Publish With many SDs to publish this can be a very slow process Publishing ~100 services took hours
We can do better Run the script from any machine - No ArcPy dependency Exploit concurrency - Parallel uploads - Parallel publishing If your script is amenable to concurrency, exploit it by all means
Publishing a service with an SD Two step process - Upload the SD to the Server - QUICK - Invoke the publishing tool - INTENSIVE
Script design Upload Publish SD SD SD SD Upload UpldID UpldID UpldID Publish Upload
Comparison results Speed up of ~3 ArcPy publishing script DIY multi-threaded publishing script
Demo Starting a service Esri UC 2014 Technical Workshop ArcGIS for Server: Administrative Scripting and Automation
Options Manager Admin Script Start Geometry Service
Anatomy of an Admin script Main function - Drives the logic Internal utility functions - Get Token - Make HTTP requests - Parse JSON responses
Demo Apply Permissions Esri UC 2014 Technical Workshop ArcGIS for Server: Administrative Scripting and Automation
Task : Apply permissions to services Permissions are the way to secure your service Do it in Manager Tedious for lots of services. Must automate!
Demo Show them previously published services in Manager Show them roles Run the script Discuss the script
Demo Health check for services Esri UC 2014 Technical Workshop ArcGIS for Server: Administrative Scripting and Automation
Using the report API, Get all the folders get all the service instances. Print report Using Geo-services REST API to test availability.
Demo Health check for databases Esri UC 2014 Technical Workshop ArcGIS for Server: Administrative Scripting and Automation
Use finditems to search for all enterprise geodatabases Use validatedataitem to test database connectivity. Print output
Demo Popular extents Esri UC 2014 Technical Workshop ArcGIS for Server: Administrative Scripting and Automation
Demo Elastic scaling Esri UC 2014 Technical Workshop ArcGIS for Server: Administrative Scripting and Automation
Task : Dynamically add machines to your site Add machines to reduce latency and increase throughput Dynamically add them when a latency threshold is breached De-allocate machines when traffic is less
Demo Start with a single node site Make service requests and watch latency (we ll use JMeter) Run a script that adds a machine to a cluster Watch the latency drop
Statistics in 10.3 (Sneak peak)
Statistics at 10.3 Server is always collecting statistics - Avg response - Number of requests - Number of timeouts -... Report defines the resource, metric and interval - Previous day, last 7 days, last 30 days - Custom time window Server chooses intelligent data point intervals in the reports REST API to harvest all this information
Python is great!!
Short detour Most modern languages (like Python) support Threads Some languages provide concurrency as a language feature - Go - Erlang - Clojure - Scala Suited for concurrent programming
Go Invented at Google About concurrency Lightweight threads (goroutines) Typed thread-safe communication and synchronization (channels) Readable concurrent code
Erlang Invented at Ericsson Functional language Light weight processes Can spawn 100s of thousands of them Concurrency via message passing Actors model
Python is great!! Don t get us in trouble, we don t want to start a rebellion J
Thank you Please fill out the session survey: First Offering ID: 1146 Second Offering ID: 1344 Online www.esri.com/ucsessionsurveys Paper pick up and put in drop box
Questions?
Resources GIS Tutorial for Python Scripting Esri Press, 2014 Python Scripting for ArcGIS Esri Press, 2013 Paperback and e-book Paperback and e-book Just released! Offers several hands-on tutorial exercises. Good reference text