Advanced ArcSDE Administration for SQL Server Shannon Shields Tony Wakim

Workshop Format Three topics What's New In-depth Database Administration Trouble-shooting / Performance Selection varies year to year Please let us know if there is a topic you'd like to see next year

Topics What's New in 9.3 SQL Server 2008 spatial types Database backups (for increased job security) How to backup and recover your geodatabase Diagnosing locking issues

SQL Server 2008 Spatial Types GEOGRAPHY GEOMETRY 'CLR' types (available in all editions that use.net framework) Create by defining column data type as 'GEOGRAPHY' or 'GEOMETRY' Large set of spatial methods (functions) OGC and extended Microsoft-provided methods Methods are called on instances MyTown.STContains(CoffeeShop)

GEOGRAPHY data type Stores ellipsoidal geometries Lat,Long coordinates referenced to a known Geographic Coordinate System Specifically, locations on the earth's surface Methods will return true distance/area measurements Each GEOGRAPHY instance uses a predefined SRID EPSG code (European Petroleum Survey Group) Query sys.spatial_references_system for full list Default is 4326 (WGS_1984) No single instance can exceed the size of a hemisphere Minimum bounding circle

Geometry data type Stores planar (x,y) coordinates Calculations are planar Coordinate system is arbitrary SRID does not correspond to any particular coordinate system SRID can be any value from 0 to 99,999 Defaults to 0 Appropriate for projected or unknown coordinate systems

What is all this about SRID? Spatial Reference Identifier Associates geometries with coordinate reference Associated with individual geometries, not entire tables geography::stgeomfromtext('point(37.603936-122.084552)', 4326) Methods comparing two geometries will return NULL results if SRID of inputs do NOT match

ArcSDE Implementation Creating data dbtune keywords control storage Registering 3 rd party data Structure t Storage configuration parameters CAD storage Spatial index Current Status

Data: M, Z, CAD, etc. Geometry and Geography support Z and M storage ArcSDE does not currently support Z and M with Geography Technical limitation Working on a solution that performs well Implemented and maintained by ArcGIS: CAD, Topology, Terrains, Geometric Networks, Network Datasets, Annotation, Dimensions, Raster, Subtypes, Domains, Relationship classes etc.

Creating data with ArcGIS GEOMETRY_STORAGE Dbtune parameter controls the type of geometry used Keyword Config g_ string DEFAULTS GEOGRAPHY GEOMETRY WKB_GEOMETRY SDEBINARY GEOGRAPHY GEOMETRY OGCWKB Additional keywords to support creation of Networks, Terrains & Topologies with GEOGRAPHY and dgeometryt types

Registering 3 rd party tables with ArcGIS Must register external tables with ArcSDE to see spatial content sdelayer o register -t t <storage_type> either ih GEOGRAPHY or GEOMETRY -R R SRID to associate with existing ArcSDE spatial reference or use other Spatial_Ref_Opts (x/y offsets, cluster tolerance, projection ID or file) to generate new C:\>sdelayer o register l contours,shape e nslc+ -t GEOGRAPHY C ID,SDE x -180,0,11930464.69,0.000000167638 -G 4326 D UC...

Registration and SRID ArcSDE SRID Unique record in SDE_spatial_references table GEOGRAPHY SRID Valid EPSG code from sys.spatial_references_system table GEOMETRY SRID User-defined SDE_spatial_references AUTH_SRID & AUTH_NAME columns Populated with GEOGRAPHY or GEOMETRY SRIDs

Registration Workflow Easy workflow for registration Use ArcGIS to create a 'template' feature class with correct coordinate reference Get ArcSDE SRID from template Use sdelayer o describe Use R <SRID> option when registering SQL Server table Registration ti will not automatically ti build a spatial index Assumption is that existing data would/should already have a spatial index

Registration rules All SQL Server SRIDs must be the same Cannot have different coordinate references in the same layer All geometries must be the same entity type Cannot mix points, lines, polygons in the same layer Must have suitable ObjectID column Registration can create one for you Use existing column (int, not null, unique) ID NAME 1 Cowichan 2 Fraser 3 Sooke SHAPE

Structure: Storage configuration parameters New dbtune DEFAULTS parameters B_MS_SPINDEX, A_MS_SPINDEX Sets SQL Server spatial index parameters GEOM_SRID_CHECK Creates a check constraint on spatial column to ensure consistent SRID values MyLocation.STSRID = 4326 Default is 1 (on) If inserts are NEVER performed at SQL level this can be disabled GEOMTAB_% Controls CAD side table storage

Structure: CAD storage CAD entity type is automatically enabled if data created using ArcGIS Desktop / ArcObjects Enables storage of true curves etc. Must enable manually y( (c entity type) if data created with ArcSDE CAD features are stored in a side table SDE_GEOMETRY<layer_id> Spatial column contains densified version of CAD curves Once table is populated, all spatial queries will join to CAD table If table is not populated, join will not occur Use caution if editing geometries outside ArcGIS Only ArcGIS can maintain the CAD feature

Spatial Index GEOMETRY and GEOGRAPHY spatial index Database index based on b-tree Can have up to 249 spatial indexes on a single spatial column ArcSDE will create a single, default spatial index Based on config_string value of A/B_MS_SPINDEX in dbtune Specify density (number of cells) per level Specify cells per object ArcSDE will provide default bounding box (current layer extents) ArcSDE returns index type 'SPIDX_MSSQL' and -6

Current Status SQL Server 2008 is not in final release Estimated Q3 2008 ArcSDE 9.3 works with SQL Server 2008 CTP6 and RC0 but IS NOT certified. ESRI will certify SQL Server 2008 when it is final Until then please test, but do not use in production 9.3 SP1 will contain several fixes Improvements to registration process

Topics What's New in 9.3 SQL Server 2008 spatial types Database backups (for increased job security) How to backup and recover your geodatabase Diagnosing locking issues

Five excuses not to backup your databases Nothing is going g to happen to my data It will take too long My data is read-only no one edits it Backups take too much space I have copies of all my original data

Five reasons why you MUST backup your databases 1. Natural disasters, media or hardware failures, mistakes 2. Backing up a database takes less time than: Reloading all your data Explaining to your boss where all the data went Finding a new job... 3. Read-only data needs to be backed up at least one (see reason #2) 4. Disks are much cheaper than data. Always factor backup space requirements into hardware capacity planning. 5. All edits will be lost if you have to reload original data instead of restoring a backup.

Backup planning 1. Determine how much data you can afford to lose 2. Work out a restore strategy based on acceptable data loss and data usage patterns How often does data change? What changes? All data, only certain tables Are modifications consistent or does editing volume vary over time? 3. Implement backups defined by restore strategy 4. Test, test, and test again Test and verify backups Test restore procedures 5. Re-evaluate evaluate and modify as needed

ArcSDE Geodatabases Active repository tables Heavily edited data, e.g. versioned feature classes Read-only data, e.g. large raster stores Edited Data Read-only Data

Managing read-only data in a read-write database Full database backups simplest way to prevent data loss Consume largest amount of disk-spacespace Take longest time to complete Not efficient for databases with significant amount of read-only data Partial backups Introduced at SQL Server 2005 Backup of read-write filegroups and optional read-only filegroups Work on any recovery model (simple, full etc)

Configuring geodatabases for partial backups 1. Create filegroups Segregate data onto read-write and read-only filegroups 2. Create dbtune keywords that direct data to correct filegroup %_STORAGE and %_INDEX_% parameters Add 'ON' argument WITH FILLFACTOR = 100 ON FG1 By default indexes are stored on same filegroup as base table 3. Load data using new dbtune keywords Can manually move existing indexes/tables to different filegroups DROP INDEX MOVE TO CREATE INDEX WITH DROP EXISTING 4. Mark specific filegroups as Read-only

Performing partial backup of geodatabase Start with an initial backup of all data Then on schedule, perform partial backups according to backup plan Partial backups contain: Primary filegroup All read-write filegroups Optional read-only filegroups (must specify) Must use BACKUP statement with READ_WRITE_FILEGROUPS argument Include FILEGROUP argument for inclusion of read-only filegroups Can make differential partial backups too

SQL Server geodatabase backup hints Use default settings for ArcSDE logfiles Creates logfiles in tempdb (9.2 and later) Logfile activity is NOT recorded in geodatabase's transaction logs You must backup your entire geodatabase Multiple database instances where data is spread over multiple databases with central (sde) repository present additional backup/restore complexity SDE database repository tables record activity in all related databases E.g. States, Lineages, Versions, etc Therefore MUST synchronize backup and restore of all databases in the geodatabase

Topics What's New in 9.3 SQL Server 2008 spatial types Database backups (for increased job security) How to backup and recover your geodatabase Diagnosing locking issues

Diagnosing Locking Issues What is a lock? How do locks affect workflow; Blocking issues Some tools used to identify locks Activity it Monitor SQL Server Profiler sp_lock / sp_who stored procedures Sys.dm_os_waiting_tasks

What is a lock? Locks are used to ensure data integrity when multiple users are accessing data at the same time Locks block other transactions from modifying resources that are used by current transaction Most common types of locks Shared Exclusive

Resources that can be locked Data Page Database Row Table Extent = 8 pages

How do locks affect workflow? Users might have to wait for a lock to be freed from a resource before being able to use it. The user is Blocked from using this resource. Lock escalation: : When a group of locks at a lower level is promoted to a lock at a higher level Deadlocks: : A deadlock occurs when two or more tasks permanently block each other from acquiring a lock on a resource

Demo: ArcMap non-versioned editing

How to check for blocking Activity Monitor: : A SQL Server tool that shows: List of processes and locks by process or by object SQL Server 2008 adds Resource waits, Recent Expensive Queries, Data File I/O sp_lock _ / sp_who stored procedures sys.dm_os_waiting_tasks

Using SQL Server Profiler to detect deadlocks Graphical Interface to Monitor Activity in an Instance Can capture data about events you specify Data captured can be saved for future reference

Demo: Monitoring locks and Deadlocking

Other related sessions to attend ArcSDE Configuration, Direct Connect and Upgrade Strategies Tue 8:30-9:45 (8) & Thu 3:15-4:45 (14B) Using SQL and Spatial Types with the Geodatabase Wed 10:15-11:3011:30 (4) & Thu 3:15-4:30 (4) Enterprise Geodatabase Tuning Tips and Tricks Tue 1:30-2:45 (6D) & Thu 8:30-10:15 (6D) Geodatabase Editing Workflows Intro Wed 8:30-9:45 (6C) & Thu 1:30-2:45 (6C) Geodatabase Editing Workflows Advanced Wed 10:15-11:3011:30 (6C) & Thu 3:15-4:30 (6C)

Questions? Thank you for attending