Geotechnical Data Sharing and Electronic Data Exchange at Minnesota DOT. Derrick D. Dasenbrock 1, M. ASCE, P.E.

Transcription

1 Geotechnical Data Sharing and Electronic Data Exchange at Minnesota DOT Derrick D. Dasenbrock 1, M. ASCE, P.E. 1 Minnesota Department of Transportation, 1400 Gervais Ave, Maplewood, MN, USA PH (651) ; dot.state.mn.us ABSTRACT: The Minnesota Department of Transportation (Mn/DOT) has advanced over 25,000 soil borings and CPT soundings for state transportation projects; there is increasing interest in using this information for private projects, studies, and new construction. Boring and sounding information is stored electronically in project databases. These discrete project databases are then used to populate a composite database for a web-based GIS application, developed to make the data more accessible to Mn/DOT personnel and the general public. Interfaces and query tools have been developed to create a self-service environment for users to find, select, and download information. Mn/DOT is participating in the development of the DIGGS (Data Interchange for Geotechnical and Geoenvironmental Specialists) data exchange standard. In addition to electronic Adobe PDF boring logs, users of the ArcGIS application can also obtain DIGGS [draft] compliant XML files for use in DIGGS-ready applications. Development aspects of the Mn/DOT geotechnical database and the applications associated with the data storage and exchange are described. INTRODUCTION The Minnesota Department of Transportation (Mn/DOT) has been performing geotechnical site investigations, since To date, Mn/DOT personnel and consultants under contract to Mn/DOT have advanced more than 25,000 soil borings and soundings. Electronic storage of completed log data began in 1992 using a MS-DOS database system; this system was upgraded to a MS-Windows environment in Today, roughly 30% of logs have complete information stored in a database format; another 50% have been scanned and are electronically available in an Adobe PDF format. Location and general attribute information is available for about 98% of all borings. At present, boring information is entered based on time and project need. In the interim, older logs are being scanned and stored as electronic documents. Recognizing the increasing importance of accessing historic geotechnical data, Mn/DOT began developing specifications for the development of a GIS application to access geotechnical data in The intent was to allow internal and external users to search for borings and soundings using interactive maps and query tools. A web-based ESRI (Environmental System Research Institute) ArcIMS application was developed by a consultant to Mn/DOT in the summer of 1

2 2003 to allow point attribute data to be queried by a variety of properties including project number, structure number, and boring number and display this information on United States Geological Survey (USGS) topographic maps, aerial photos, or a Mn/DOT base map. A new GIS application, with enhanced functionality, has been created, adopting the new Mn/DOT GIS BaseMap. Improvements have been made in the data loading, background mapping, and data presentation formats. Additional map layers are also in development for the presentation of other geotechnical information, assets, and features. In the new system, users can obtain Adobe PDF files of complete boring and sounding logs or can output point-based information to the screen or MS Excel files. The user can select one of 6 report styles to best suit their information needs about the point information. Using the new draft DIGGS data interchange standard, Mn/DOT is also providing boring log and sounding information in an XML (Extensible Markup Language) format. This information can then be manipulated easily with a variety of software packages that are DIGGS compatible. With the new web based system, individuals can easily search for geotechnical data relevant to their project, research, or other need; data can be easily queried in a self-service fashion by keying in search parameters or by navigating on a state map. Users can print logs or download information without traveling to the Mn/DOT geotechnical offices. Additional history about Mn/DOT s geotechnical database development is described by Dasenbrock (2006). APPLICATION DEVELOPMENT A group of Mn/DOT personnel familiar with the development of web-based applications and geospatial data, held a series of meetings beginning in the spring of 2003 to define the scope of work for developing a geotechnical ArcIMS application. Boring information and existing database content was reviewed. Specifications for functionality and user interface organization were developed. Concepts for queries, reports, map navigation, selection tools, links, and other issues of the user interface were discussed. Different mapping layers, entities, boundaries, labels, colors, and other aspects were also explored. In addition to the interface and overall application development, a method to populate a new geotechnical database, referenced by this application, was included as part of the application scope. A basic querying tool was developed, as was a simple output report. The output report contained links to a PDF file of the associated logs. This file is either a scanned copy of the original log or a PDF output log generated by the gint software. In some cases PDF files do not yet exist, although gaps in data availability are slowly being fixed as they are encountered. After using the 2003 application, it was recognized as a powerful tool for searching Mn/DOT geotechnical data; it also quickly showed that a substantial portion of the older data contained erroneous information, including the coordinate location, highway number or asset number or type. The work to correct the erroneous data has been ongoing since the application roll-out. In the summer of 2005, the functionality for referencing and presenting map backgrounds was substantially improved. Entering historic data, creating electronic images of old borings, and exporting current logs to Adobe PDF files is ongoing. In 2006, the Office of Materials, of which the geotechnical section is a part, began an effort to update the user interface and other functional components of the geotechnical GIS database system to use the framework of the Mn/DOT interactive BaseMap, described in the following section. The Mn/DOT interactive BaseMap can be found on Mn/DOT s external web site at: 2

3 Mn/DOT BASEMAP HISTORY The BaseMap includes information about transportation features, as well as boundary information, stream and lake locations. The Mn/DOT web based GIS BaseMap is a recent development, although components of it have been created over the past 40 years. In 1979, Mn/DOT began using computer aided drafting to convert hand drafted Mn/DOT cartography into electronic files. Minnesota's 1,745 USGS 1:24,000 scale quadrangles were digitized to obtain seamless digital map coverage. By 1985, the seven-county St. Paul-Minneapolis Metropolitan Area was completed and a set of 50 edge matched CAD files containing all public road centerlines, railroads, political/administrative boundaries, the Public Land Survey System and surface water was produced. In the late 1980's, Mn/DOT began to research the potential offered by the exploding use of Geographic Information System (GIS) technology for its own needs and services. A series of internal task forces and consultation with the [Minnesota] state Land Management Information Center (LMIC) resulted in a 1992 decision to incorporate the CAD digitizing into a statewide seamless basic core of geography providing a means for relating Mn/DOT spatial data to other spatial data. The GIS software chosen was ESRI s Arc/Info. Digitizing continued, to complete statewide 1:24,000 scale coverage within 18 months. Mn/DOT also switched from using State Plane Coordinates, zone specific, NAD83, to Universal Transverse Mercator (UTM) Minnesota extended Zone 15, NAD83 coordinates. In 1996, Mn/DOT produced its first State of Minnesota GIS BaseMap on a CD-ROM. In 2003, Federal Geographic Data Committee (FGDC) compliant metadata was also provided for each individual feature data set. The metadata presents the basic characteristics of the data resource and information on data sources, and information related to geographic and database elements (Mn/DOT 2007). Mn/DOT GEOTECHNICAL PROJECT DATABASE ARCHITECTURE Originally, only point attribute information was stored in an electronic format, through a mainframe application. In the 1980 s information was entered into electronic systems that created dot-matrix log printouts; it is unclear if this information was stored electronically. In 1992, information was entered into an early DOS version of the gint software package; a MS Windows version is in use today. Depth based boring information from this point forward was entered into project databases; point-based attribute information was entered both there and in the mainframe application, which by this time had been migrated to a Borland Paradox database. This database was later migrated again to a MS Access structure and then to a MS Excel table, where it presently resides. Approximately 670 projects were entered into MS-DOS databases from 1992 to 1997 when a MS Windows version of the product was purchased. When a new file structure was implemented, the legacy MS-DOS files needed to be converted to the new format. This process was combined with an error checking and validation program and took several years of intermittent work, as time permitted, to update all of the old project files to the same file structure. Boring and sounding data is principally grouped by either State Project Number or by Minnesota State Highway Control Section, which may contain a number of discrete projects. The latter system of grouping is generally used in rural areas with smaller projects or in those situations where data from a number of independent projects is related. Boring and sounding 3

4 information for each of these project groups is entered and stored in an independent database. There are roughly 3500 project databases, organized by County and State Project Number. Until 2003, point attribute information was entered into both the MS Access database and the gint database where depth-based information was also stored. This allowed for easy searching and querying among all borings in the MS Access database. After 2003, the MS Access database was only periodically updated with exports from the gint project databases. The ArcGIS system has replaced the legacy MS Access system in general function. All the current exploration borings and soundings, as well as the historic investigation data now reside in gint project databases. These databases are now used to populate the ArcGIS database. The legacy MS Excel table is now used principally to locate records that have known errors or omissions and have not been uploaded to the ArcGIS system. Recent Project Database Enhancements In part due to an increase in the number of cone penetration test (CPT) soundings advanced for recent projects, there have been several enhancements and utilities added to the Mn/DOT gint project database environment. In November of 2006, all borings and soundings were loaded into gint project database files. Legacy and current files were all updated to a new database structure; this structure included some substantial revisions to better organize similar data and create new tables and fields for in-situ testing information. Several utilities have also been incorporated recently, including coordinate and CPT data importing routines, a coordinate conversion system to populate county coordinate, latitude/longitude, and UTM fields. A system to provide automated public lands survey legal descriptions, based on coordinate locations, has recently been implemented. Project Database Design and Electronic Data Storage The database design has changed significantly since the first borings were entered into the electronic format in the early 1990 s. Using built-in translation tools, data transfer was not particularly difficult for most of the transitions. The most cumbersome transition was migrating from DOS to MS Windows; this required significant data conversions, data validation, and error checking. The current project database format has 21 tables containing depth-based geotechnical information. The structure contains data from Mn/DOT s geotechnical borings, CPT soundings, and DMT pushes. Table 1 shows the tables in each project database and those that are available in the current electronic output formats through the ArcGIS interface. The Mn/DOT geotechnical section has been maintaining the project level databases and the legacy database internally since they were introduced. In addition, lab data, geophysics data, and in-situ testing data are presently kept in data files associated with specialty software. This information is not recorded in the project databases and is not yet available through the ArcGIS application, although this is an area for future development. A development project is underway with a geotechnical lab testing software vendor to create an interactive viewer that will allow original lab data files to be accessed through either the gint database or the ArcGIS web application. The lab data can then remain resident in the original software files and is not replicated unnecessarily in the gint database or the ArcGIS database. This also means that data, if updated or changed, only needs to be updated in the original data files. It is anticipated that users will eventually be able to print copies of the lab data in a similar method to printing copies of boring and sounding logs. 4

5 Table 1. Mn/DOT Geotechnical Database Tables and Present Data Availability by Electronic Log, GIS Application Downloadable Table Output, and DIGGS XML output. gint Project Database Boring/Sounding Tables Electronic Logs/ Adobe PDF GIS Application screen output or downloadable MS Excel file output DIGGS XML file output Project X X X Point X X X Log X X X Stationing X X X Sample X X Classification X X Legal Description / Health Dept X X X Lab Testing X X Rock Core Breaks X X Remarks X X Piezometer Information X X Database X Field Test Data * X Electronic Data X CPT Data X X CPT / DMT Settings / Project Info CPT / SPT Output In-Situ Tests * ** CPT Seismic Data * ** DMT Data * ** Push-In Piezometer Data * ** * The output from these specialty applications is in development, and is expected to become available through the web-based application by At present logs indicate that specialty investigation methods were used and that data from these tests is available on request. ** The availability of data from these tests will depend on the ability of the initial information and the ability to convert the information into the DIGGS format (still in development at the time of this publication). Parameters in these tables are used to adjust the output of collected data on computer generated field logs and do not contain test data or borehole/sounding data. GIS APPLICATION FOR GEOTECHNICAL DATA The GIS application system consists of a web server, an ArcIMS web application server, and an ArcSDE server application [used to access multi-user geographic databases stored in relational database management systems (RDBMSs)], an Oracle database with the boring and sounding data, the data loader system to take the gint data and populate the Oracle database, and the related databases with base maps, background maps, and other feature class information. Figure 1 shows the system of databases and applications used to provide content to the user. Data Loading to the ArcGIS Application For a time, two different systems were needed to load data into the ArcGIS system. Data was loaded from the single legacy point attribute MS Access database and the multiple individual gint project databases. With the successful migration of all of the point-based information to 5

6 individual, project-based, gint database files, the legacy database is no longer needed to import older project information into the ArcGIS system. The current process involves loading the data into 17 Oracle database tables by executing Pervasive Software mapping tools. A reporting table is created to contain data for reporting (shown in Table 1) through SQL procedures. Figure 1. Diagram of the new Mn/DOT Foundation Boring ArcGIS application A new universally unique identifier (UUID) field was added at the project and point level tables to relate project table data to all points within that project (as the point table did not inherently contain a reference to the parent project table). Current plans are to purge the data in the ArcGIS application every two months and upload new data from the project databases. Queries, Other Features, and Background Maps In addition to the boring and sounding point features, the system also shows highways, bodies of water and other information including city and county names. Users can select a state base-map, 6

7 USGS digital raster graphic (DRG) topographic quadrangle maps or digital orthophoto quadrangle (DOQ) maps (aerial photos), as backgrounds. The Mn/DOT geotechnical ArcGIS application has already proven very useful. The system has been used frequently to find borings by searching for their unique record number, structure number, and combinations of these and other parameters. Figure 2 shows an example of several borings found using the Mn/DOT ArcGIS application; a DOQ map is selected as the background. Figure 2. A screen shot of the new Mn/DOT Foundation Boring ArcGIS application WEB BASED DATA VIEWING AND EXCHANGE As an increasing number of Mn/DOT projects now involve facility reconstruction, the usefulness of the legacy data is steadily becoming more important, as is the need to be able to search for this data effectively. There is also increased interest in making geotechnical information more available to Mn/DOT staff, other units of government, researchers, geotechnical practitioners, project contractors, and the public. Geotechnical data has often been difficult to share easily due to the number of proprietary data storage systems (for logging, lab testing, and in-situ monitoring, offered by a number of different providers). In general, the recipients of transferred data do not have access to the software used to create, view, and edit the data. DIGGS In addition to providing summary point-level data and Adobe PDF logs, there was also an interest to be able to exchange boring and sounding data electronically to more easily share data. 7

8 Mn/DOT is a member of the Geotechnical Management Systems Group for the development of DIGG.S and is making progress toward providing boring and sounding information using the new standard. The development of the DIGGS standard, and the roles of participating organizations are described by Lefchik (2006). When the first DIGGS standard becomes available, Mn/DOT will create DIGGS compliant XML files. These files will then be available through the ArcGIS application in addition to the Adobe PDF electronic boring logs and the MS Excel files of the point-based data. Data Conversion to XML/DIGGS Through a conversion application within the gint database environment, gint project files can be converted into the DIGGS XML standard. Little additional effort will be required to create the data interchange files based on the existing project databases. A correspondence file will be used to map the Mn/DOT data into the DIGGS data structure; the program automates the creation of the XML files with the correct nomenclature and syntax. Features of the XML format and its use in the exchange of geotechnical database information are described by Caronna (2006). CONCLUSIONS Mn/DOT has developed an ArcGIS application that makes most agency geotechnical borings web-searchable and printable. As part of recent improvements, point-based data is also available as downloadable in a MS Excel file format. Boring and sounding data will be converted to the new DIGGS data interchange standard and this information will be available through the web based ArcGIS application. Additional areas for development include the addition of 2-D geophysical data, lab data, and monitoring/instrumentation data. The overall effort to develop the data sharing and exchange system has resulted in an increase in data quality, an improvement in overall process automation, and significantly improved data sharing techniques. Using the ArcGIS interface, records can now be obtained in real-time in a variety of formats. REFERENCES Caronna, S., (2006) Implementing XML for Geotechnical Databases JTC2 Workshop on Geo- Engineering Data: Representation and Standardisation, IAEG Congress, University of Nottingham, United Kingdom ( Dasenbrock, D. D., (2006) Mn/DOT s On-Line Geo-Spatial Borehole/Sounding Database Development. Geotechnical Engineering In the Information Technology Age, Proc., Geo- Congress 2006, Feb 26-March 1, 2006, Atlanta, GA, Edited by DeGroot, DeJong, Frost, Baise (CD-ROM), ASCE, Reston, VA. Lefchik, T. E., and Beach K.(2006) Development of National Geotechnical Management System Standards for Transportation Applications Geotechnical Engineering In the Information Technology Age, Proc., Geo-Congress 2006, Feb 26-March 1, 2006, Atlanta, GA, Edited by DeGroot, DeJong, Frost, Baise (CD-ROM), ASCE, Reston, VA. Mn/DOT (2007), Mn/DOT BaseMap History Minnesota Department of Transportation agency website 8