Challenges and Success of Migrating to an Enterprise Database in York County, PA PA GIS Conference June 16, 2015 Wade Gobrecht York County Planning Commission Andrew Ross GeographIT Source: Can You Draw All 50 US States?www.thrillist.com 1
Source: Map of USA United States Population Density Mapwww.mapofusa.net GIS in York County Prior to Enterprise Implementation Different s Different Software Versions Redundant Datasets potentially different and no standards Datasets e mailed or distributed on CDs quarterly or Annually No Public/Municipal access to data other than manual data extracts access to limited data on parcel web map Planning Commission Assessment 911/EMA Ag Preserve Human Services Economic Development Conservation District Different Same Different Same Same Different Different 2
Goals Goal: Design a system that enables each department take advantage of the coordinated data efforts and new standards while allowing them to work independently with their current systems and setup. Ultimate Goals Enterprise Wide Adoption of Official, Up To Date GIS Datasets Promote Enterprise Wide Data Sharing Consolidate Redundant Datasets (500+ down to 200) Adopt State Of The Industry Software and Data Standards Timeline 2006 Study completed by Penn State and Advanced Skills Center Recommended a GIS Department or a GIS Director in Commissioner s office Voted no by Commissioners 2006 York County GIS Advisory Council Created for Coordination 2010 Grant Funded Enterprise GIS Implementation Plan completed by GeographIT 2012/2013 Enterprise GIS Implementation Started Hardware / Software Updates Educated staff Data Investigations Database Needs and Design Data Migration Data Replication to other departments Applications Created 2014 Enterprise Database Deployed 3
Background Enterprise GIS Implementation Plan Creation of GIS Management Committee Development of Strategies Hardware, Software Re Purposing & Upgrades Identification of Critically Important Datasets Data Stewardship and Distribution Overcoming Technical Challenges ( & Licensing) Migration Procedures Budgetary Estimates Background Enterprise GIS Implementation Plan Inventory of Existing Data and IT Infrastructure, Hardware, Software, Licenses Inventory of GIS Business Practices at YC Agencies Inventory of Existing & Required Skills 4
Background Enterprise GIS Implementation Plan Human Services Agricultural Preservation MS SQL Server Express for Ag Preservation and SDE replication, may not be Cadastral Shape Files needed depending on bandwidth Two way update scripts SDE Copy of all GIS data from Planning MS SQL Server Express for SDE Copy of Parcel data SDE replication, may not be from Assessment needed depending on bandwidth SDE Copy of all GIS data from Planning SDE Copy of Parcel data 1 ArcView or ArcEditor 9.x from Assessment 1 ArcView 3.x 1 ArcInfo or ArcEditor 9.x 10 Mbit, to be upgraded to 1 Gbit during 10 Mbit, shared renovations with nursing home, soon will be 50 Mbit EMA / 911 DB Transactional Replication or Mirroring RDBMS + Storage ArcGIS Server = + ArcSDE + Dedicated Data RDBMS + Storage Server for Production = Geo-database / s Dedicated Data This is where EMA Server for WEB Sites GIS data is Database level maintained replication backs up Production Data SDE Copy of Parcel data from Assessment Server (including SDE replicas) SDE Copy of all GIS data from Planning Switch / Firewall / 2 MS SqlServer Reverse Proxy 1 ArcGIS Server Enterprise 3 ArcView 9.x DMZ 1 ArcEditor 9.x WEB Server 3 ArcInfo 9.x Soil Conservation Economic Devel. Corp. Conservation Projects Shape File & Access DB for Plan & Practices MS SQL Server Express for SDE replication Two way update scripts SDE Copy of all GIS data from Planning MS SQL Server Express for SDE replication SDE Copy of Parcel data from Assessment SDE Copy of all GIS data from Planning SDE Copy of Parcel data from Assessment 1 ArcEditor 9.x 2 ArcView or ArcEditor 9.x 1 ArcView 3.x 1 ArcEditor 9.x Cable Modem 6 Mbit down 1 Mbit up Internet Administrative Center Parcel Data One way Maintained in ArcInfo update scripts Workstation Format Administrative Center Backbone 4 Gbit 4 Gbit Assessment Office Planning Commission Planning Commission Switch / Firewall 1 Gbit 1 GBit DMZ Firewall / Reverse Proxy WEB Server 2 MS SqlServer 1 ArcGIS Server Enterprise 2 MS SqlServer 3 ArcInfo 9.x 1 ArcView 3.x 3 ArcInfo 9.x & 9 ArcView 3.x can be replaced by web apps. RDBMS + Storage RDBMS + Storage = = Dedicated Data Backup Server for Server for Parcel Dedicated Data Production Geodatabase Database level Server This where Parcel replication should data is maintained insure that SDE level Copy of Planning SDE Copy of all GIS data is included data from Planning DB Transactional Replication or Mirroring Total of 15 GIS Wkstns Total of 21 GIS Wkstns RDBMS + Storage ArcGIS Server = + ArcSDE + Dedicated Data RDBMS + Storage Server for Production = Geo-database / s Dedicated Data This is where all GIS Server for WEB Sites data is maintained Database level replication backs up SDE Copy of Parcel Production Data data from Assessment Server (including SDE SDE Copy of EMA /911 replicas) data DB Transactional Replication or Mirroring 14 ArcView 9.x 3 ArcInfo 9.x 4 ArcView 3.x 1 ArcPad 7.x 1 Lizardtech Geoexpress 1 Hazus Background Enterprise GIS Implementation Plan Vector Data Migration Process Overview Investigation and Requirements Design Instantiation and Loading Testing YES Found Issues? NO Production Environment Data Cleanup 5
Enterprise GIS Implementation Collaborative Effort led by YCPC Based on Enterprise GIS Implementation Plan With Participation of: All Other Concerned YC Agencies GIS Consultant: geographit Implementation Strategy IT Upgrades and Data Migration Follow Separate Overlapping Tracks Data Migration: Start with Critically Important Data Layers: Land Base (Parcels) Transportation (Road Centerlines) Major Participation of Stakeholders in Data Model Redesign Establish Templates and Procedures for Use with All Other Data Layers Script Complex Data Migration Procedures Simplify Re Deployment as New Hardware Becomes Available 6
Land Base Data Parcels Maintained in Original Format by Data Steward (Assessment) Tile Based Coverage Model Reformatted for Distribution on Recurrent Basis by Automated Scheduled Procedure Seamless Geodatabase Procedure includes incorporation of CAMA Data Data Steward to Switch to Seamless Geodatabase at Future Time Transportation Data Road Centerlines Maintained by Emergency Services Fields added in for Routing (speeds, one ways) Format Must Remain Compatible with CAD System Intermediate step to manually add data until CAD is upgraded Script will handle Schema Differences until testing is done 7
Data Distribution Challenges All YC Agencies Are Not On Same Data Exchange Must Go Over Internet Low GIS Licensing Level Further Restricts What Agencies Can Do With Geodatabase Replication Data Distribution Solutions Coalesce All Data onto Central Server at YCPC and Redistribute from there Minimize Use of Geodatabase Replication Use Python Scripts and SFTP Server to Move Packaged Sets of Feature Classes 8
Data Distribution Solution: Circular Data Exchange Between YCPC and YC Agencies More Robust than Replication in Some Ways Failed Sync Operation Self Repairs During Subsequent Exchange Cycle Without Manual Intervention Non Replication Exchanges Not Affected By Schema Changes Non Replication Exchanges Can Be Adjusted to Add or Remove Feature Classes Without Complex Resets York County Enterprise Depts. Pushing updates into Central Database 911 SFTP Server at 911 Enterprise Database Planning Commission Web Services Internet Applications and County Staff Desktops Econ All. Benefits Ag Pres Assessment Future Data Updates YCCD Hum. Serv. 1. Data updated Weekly 2. Data Integrity 3. County data standards developed 4. Eliminate Data Duplication 5. Data/Tools available to County Departments and Municipal staff 9
Scripts, Scripts and more Scripts Land Base Scripts Builds LandBase Feature Dataset(FD), Feature classes within FD, sets up domains and populates data. Also used to update Parcels and related features on a monthly basis. Replication Scripts Runs on Central database to push out data changes. Run in Departments to grab new data changes and copy to their system. Sync Scripts Sync Edit database to production database nightly Compress Scripts Compresses edit database on a weekly basis. Start/Stop Services Scripts Part of compress scripts but can also be run separately Scheduled Tasks (scripts) 10
Accomplishments New Enterprise Database Standards (Naming Conventions, Fields) Domains Replication Next Steps Create web apps using services from new database Fully integrate data replication with 911 and Assessment 11
Questions??? Contact Information: Wade Gobrecht, GISP Chief, Information Systems York County Planning Commission wgobrecht@ycpc.org 717 771 9870 Andrew Ross Senior GIS Analyst GeographIT aross@geographit.com 717 399 7007 12