ArcGIS for Server: Administrative Scripting and Automation

Transcription

1 ArcGIS for Server: Administrative Scripting and Automation Shreyas Shinde Ranjit Iyer Esri UC 2014 Technical Workshop

2 Agenda Introduction to server administration Command line tools ArcGIS Server Manager Overview of the administrative API Automation Popular scripting workflows - Publishing of services - Setting up permissions - Extracting popular extents - Monitoring services and databases - Scaling and statistics

3 Introduction to server administration ArcGIS Server Installation Media Provision hardware Select a deployment pattern Keep it running Software Authorization ArcGIS Server Setup

4 The administrator s toolbox ArcGIS Server Administrator Directory ArcGIS Server Manager Console tools

5 Command line tools Located under /arcgis/server/tools A collection of most popular tools Categories - Administrative credentials - For resetting administrative accounts - SSL/certificate management - Backup and restore - Managing cache Lock the tools down using OS file permissions!

6 Demo Command line tools Esri UC 2014 Technical Workshop ArcGIS for Server: Administrative Scripting and Automation

7 ArcGIS Server Manager Browser based interactive application Located at: Simple, easy-to-use Ability to manage your entire server site - Even though you are accessing Manager on a specific server node Great for common workflows!

8 Demo le Tour de Manager Esri UC 2014 Technical Workshop ArcGIS for Server: Administrative Scripting and Automation

9 ArcGIS Server Manager under the hood Simple HTTP requests to the administrative API

10 Overview of the administrative API Entire server administration is an API - If ArcGIS Server Manager can use it, so can you! RESTful - GET and POST of JSON data Well organized and well documented Lots of samples Scriptable Designed for automation!

11 The administrative API Create and manage your services. Start or stop them as wanted Manage system properties, web adaptors, server directories. System environment Services Machines + Clusters Add new nodes to grow capacity. Remove them to shrink. Organize machines into clusters. Get fine grained statistics on your GIS services. Admin API Statistics (new at 10.3) Manage all your database connections through a single interface. Databases Security Harvest logs for auditing, understanding usage patterns and troubleshooting. Logs Configure GIS tier or web tier authentication. Manage accounts, roles and permissions on services. Configure SSL.

12 Why automate? Common repeated workflows - Plot popular extents for the data Scheduled actions - Restarting services when data is updated - Build map caches during low demand by moving machines to caching cluster Reactive actions - Adding capacity (server nodes) to a cluster when demand increases - Send notifications when errors are logged Works with all programming/scripting languages that support HTTP!

13 Where should you begin?

14 Demo Publish Services Esri UC 2014 Technical Workshop ArcGIS for Server: Administrative Scripting and Automation

15 Motivation Publishing many services Moving services between Dev, Staging, Production environments - Backup/restore is an option but sometimes you just want to republish services Can publish from Desktop and Manager Tedious if you have lots of services Automation is your friend!

16 Task : Publish a list of services listed in a file Help system has a script that uses ArcPy ArcPy is an python library that complements the Admin REST API Provides many convenience functions

17 Caveats Works great, just need a few lines of code However, you need ArcPy on the machine Script publishes service definitions sequentially - For every SD - Upload - Publish With many SDs to publish this can be a very slow process Publishing ~100 services took hours

18 We can do better Run the script from any machine - No ArcPy dependency Exploit concurrency - Parallel uploads - Parallel publishing If your script is amenable to concurrency, exploit it by all means

19 Publishing a service with an SD Two step process - Upload the SD to the Server - QUICK - Invoke the publishing tool - INTENSIVE

20 Script design Upload Publish SD SD SD SD Upload UpldID UpldID UpldID Publish Upload

21 Comparison results Speed up of ~3 ArcPy publishing script DIY multi-threaded publishing script

22 Demo Starting a service Esri UC 2014 Technical Workshop ArcGIS for Server: Administrative Scripting and Automation

23 Options Manager Admin Script Start Geometry Service

24 Anatomy of an Admin script Main function - Drives the logic Internal utility functions - Get Token - Make HTTP requests - Parse JSON responses

25 Demo Apply Permissions Esri UC 2014 Technical Workshop ArcGIS for Server: Administrative Scripting and Automation

26 Task : Apply permissions to services Permissions are the way to secure your service Do it in Manager Tedious for lots of services. Must automate!

27 Demo Show them previously published services in Manager Show them roles Run the script Discuss the script

28 Demo Health check for services Esri UC 2014 Technical Workshop ArcGIS for Server: Administrative Scripting and Automation

29 Using the report API, Get all the folders get all the service instances. Print report Using Geo-services REST API to test availability.

30 Demo Health check for databases Esri UC 2014 Technical Workshop ArcGIS for Server: Administrative Scripting and Automation

31 Use finditems to search for all enterprise geodatabases Use validatedataitem to test database connectivity. Print output

32 Demo Popular extents Esri UC 2014 Technical Workshop ArcGIS for Server: Administrative Scripting and Automation

33

34 Demo Elastic scaling Esri UC 2014 Technical Workshop ArcGIS for Server: Administrative Scripting and Automation

35 Task : Dynamically add machines to your site Add machines to reduce latency and increase throughput Dynamically add them when a latency threshold is breached De-allocate machines when traffic is less

36 Demo Start with a single node site Make service requests and watch latency (we ll use JMeter) Run a script that adds a machine to a cluster Watch the latency drop

37 Statistics in 10.3 (Sneak peak)

38 Statistics at 10.3 Server is always collecting statistics - Avg response - Number of requests - Number of timeouts -... Report defines the resource, metric and interval - Previous day, last 7 days, last 30 days - Custom time window Server chooses intelligent data point intervals in the reports REST API to harvest all this information

39 Python is great!!

40 Short detour Most modern languages (like Python) support Threads Some languages provide concurrency as a language feature - Go - Erlang - Clojure - Scala Suited for concurrent programming

41 Go Invented at Google About concurrency Lightweight threads (goroutines) Typed thread-safe communication and synchronization (channels) Readable concurrent code

42 Erlang Invented at Ericsson Functional language Light weight processes Can spawn 100s of thousands of them Concurrency via message passing Actors model

43 Python is great!! Don t get us in trouble, we don t want to start a rebellion J

44 Thank you Please fill out the session survey: First Offering ID: 1146 Second Offering ID: 1344 Online Paper pick up and put in drop box

45 Questions?

46 Resources GIS Tutorial for Python Scripting Esri Press, 2014 Python Scripting for ArcGIS Esri Press, 2013 Paperback and e-book Paperback and e-book Just released! Offers several hands-on tutorial exercises. Good reference text

47