Geotechnical Data Management

Geotechnical Data Management The History, Development and Future of the AGS Data Format Stephen Walthall Chairman AGS Data Format Subcommittee Bechtel

Today s Presentation Introduce you to the AGS Data Format to provide background to data transfer to describe the format briefly to present the data dictionary to reveal the investment required to make the system work to illustrate the development of AGS-M and an XML version to set the scene for later presentations (this is not a software demonstration)

The 2 Golden Rules of Data Entry Q: Data entry is a very time consuming part of our industry. How can we cut down on this costly exercise? A: Follow these 2 rules. Rule 1 - ONLY DO IT ONCE! Rule 2 - Get someone else to do it 90% of Geotechnical companies break these rules almost everyday to show you how just watch the example data journey.

The Data Journey Paper Reporting Site Exploration Data Entry 1 On site the engineer will log entries in to the site log book. As it is often raining short hand is often used to speed the process up! Data Entry 2 At the end of the day the engineer will translate their shorthand and copy (rekey) the details out neatly. They then fax the details to the office. Data Entry 3 The Office will take the faxed information and re-key it into a borehole logging program or spreadsheet. That s 3 times just to get a borehole log printed out!

The Data Journey Paper Reporting Site Exploration Laboratory Testing Data Entry 4 A paper list of samples is sent to the laboratory who re-keys sample data into their system

The Data Journey Paper Reporting Site Exploration Laboratory Testing SI Presentation Data Entry 5 The engineer will take the hard copy data from the laboratory and the site work rekeying some of it and formatting it for the report. 3 copies of the SI data are then bound and sent to the client in a big lorry. The SI stage is now complete and we have already used up five data entry lives. But this is just the start of the journey - just wait and see what happens now!

The Data Journey Paper Reporting Site Exploration Laboratory Testing SI Presentation Data Entry 6 Now the extra time really starts to build up. The designer needs to produce design charts for critical parameters. To do this review the report and re-key the relevant data into their chosen graphing package. Engineering Analysis Data Entry 7 Every new plot the engineer needs to make requires them to re-key the data. A thorough design can require a large number of plots and will result in large amounts of time just re-keying in data.

The Data Journey Paper Reporting Site Exploration Laboratory Testing SI Presentation Engineering Analysis Data Entry 8 Final design sections and drawings are required then the draftsman will effectively re-key the data as they draw it. Data Entry 9 Any modelling or GIS work that is needed would again mean the re-key of this data CAD Presentation, 3D modelling and GIS

The Data Journey Paper Reporting Site Exploration Laboratory Testing SI Presentation Engineering Analysis CAD Presentation, 3D modelling and GIS Local or national Archive Data Entry 10 The final destination is to a national or local archive system. This is often an off site, paper based filing system. Summary information of the boreholes is often re-keyed into an index system so that the location information can be retrieved later. When the data is retrieved for further analysis the process starts again with yet further rekeying of data. This is an extreme case, Data Entry 10 times, but you will recognise some stages that your organisation is currently wasting TIME and MONEY doing. Lets look at the journey again this time using a standard data transfer format

The Data Journey Electronic Reporting Site Exploration Data Entry 1 (and only 1!) Whilst on site the engineer logs the data into a hand held unit. At the end of each borehole the data is converted to a standard format file and emailed back to the office Data TRANSFER 1 The office takes the standard format file and imports it into their borehole logging program or spreadsheet.

The Data Journey Electronic Reporting Site Exploration Laboratory Testing Data TRANSFER 2 A standard format file is sent to the laboratory who imports the sample data into their system Once the laboratory work is complete a standard format file containing the results is produced and sent with the hard copy records.

The Data Journey Electronic Reporting Site Exploration Laboratory Testing SI Presentation Data TRANSFER 3 The engineer will takes the standard format file from the laboratory and the standard format file from the field work and manipulates the outputs for their report. As well as the hard copy output the engineer also sends the data in a standard file format to the client. All good so far but just wait and see what happens now!

The Data Journey Electronic Reporting Site Exploration Laboratory Testing SI Presentation Data TRANSFER 4 Now the gains really starts to build up. To produce the design charts the engineer simply imports the standard format file into their program and presses the print button Engineering Analysis

The Data Journey Electronic Reporting Site Exploration Laboratory Testing SI Presentation Engineering Analysis CAD Presentation, 3D modelling and GIS Data TRANSFER 5 Final design sections and drawings can now be automated by importing the standard format file into an automated CAD program Data TRANSFER 6 The standard format file can be easily imported to GIS and modelling packages without re-keying any of the information

The Data Journey Electronic Reporting Site Exploration Laboratory Testing SI Presentation Engineering Analysis CAD Presentation, 3D modelling and GIS Data TRANSFER 7 Again the standard file format can be imported directly into the archive system. But more importantly is can be retrieved IMMEDIATELY and can be used without re-keying of any information This is the perfect scenario but if you implement just a few of the data transfer stages you will save TIME and MONEY. Local or national Archive

Your next project? A fragmented paper based system Streamlined electronic transfer Site Exploration Laboratory Testing SI Presentation OR Site Exploration Laboratory Testing SI Presentation Engineering Analysis Engineering Analysis CAD Presentation, 3D modelling and GIS CAD Presentation, 3D modelling and GIS Local or national Archive Local or national Archive

Life before the AGS Data Archive (Consumer) 1 1 Data User (Constructor) 2 2 3 3 4 4 5 5 1 2 3 4 5 1 2 3 4 5 Data Producer (Investigator) Data Receiver (Designer)

Life with the AGS Data Archive (Consumer) 1 1 Data User (Constructor) 2 2 3 3 4 4 5 5 1 AGS 1 2 3 4 5 2 3 4 5 Data Producer (Investigator) Data Receiver (Designer)

What is the AGS format? First edition published in 1992 3 rd edition published 1999, available from the AGS web site. A supplementary document AGS-M was published in 2001 AGS is The Association of Geotechnical and Geoenvironmental Engineers, a trade association based in the UK It is used extensively on large and small projects in the UK and in at least 15 countries world wide.

What is the AGS format? A Data Dictionary and some Rules A series of defined headings (fields) that describe all aspects of site investigation and geotechnical and geoenvironmental work. These headings are arranged in tables with all like data grouped together (groups). All AGS files are text based!

The Rules 26 Rules describe the format All data is text Units are given in the <UNITS> line Only transfer Engineering Data NO fields which can be derived from others Rules given for new groups, fields and abbreviations Pick lists for common abbreviations

The Group Names for Geotechnical Data HOLE (Reference Point Information) GEOL (Stratum Description) SAMP ( Information about the Sample) CLSS (Classification test results) TRIG (Triaxial Test Results) Example of table hierarchy TRIX (detailed test results)

Example of the format "**PROJ" "*PROJ_ID","*PROJ_NAME","*PROJ_LOC","*PROJ_CLNT","*PROJ_ENG","*PROJ_CONT","*PROJ_DATE","*PROJ_AGS","FIL "<UNITS>","","","","","","dd/mm/yyyy","","" "7845","Trumpington Sewerage","Trumpington","Trumpington District Council","Geo-Knowledge International","Lithosphere Investig "**HOLE" "*HOLE_ID","*HOLE_TYPE","*HOLE_NATE","*HOLE_NATN","*HOLE_GL","*HOLE_FDEP","*HOLE_STAR","*HOLE_LOG" "<UNITS>","","m","m","m","m","dd/mm/yyyy","","" "TP501","TP","523196","178231","61.86","3.25","21/07/1999","ANO","FS002" "BH502","IP+CP","523142","178183","58.72","15.45","22/07/1999","ANO","FS003" "**GEOL" "*HOLE_ID","*GEOL_TOP","*GEOL_BASE","*GEOL_DESC","*GEOL_LEG","*GEOL_GEOL","*GEOL_STAT","FILE_FSET" "<UNITS>","m","m","","","","","" "TP501","0.00","0.25","Friable brown sandy CLAY with numerous rootlets (Topsoil)","101","TS","A","" "TP501","0.25","1.55","Firm brown slightly sandy very closely fissured CLAY with some fine to coarse subrounded gravel. Medium sp vertical rough desiccat","","","","" "<CONT>","","","ion cracks with concentrations of rootlets. (Weathered Boulder Clay)","261","WBC","B","" "TP501","1.55","3.25","Stiff grey closely fissured CLAY with a little fine to medium subrounded gravel and rare sandstone cobbles (Bo "BH502","0.00","0.30","Friable brown sandy CLAY with numerous rootlets (Topsoil)","101","TS","","" "BH502","0.30","2.60","Firm brown very closely fissured CLAY with a little fine to medium subrounded gravel (Weathered Boulder Cl "BH502","2.60","5.75","Stiff grey slightly sandy closely fissured CLAY with some fine to coarse subrounded gravel (Boulder Clay)","2 "BH502","5.75","15.45","Dense becoming very dense yellow brown very sandy fine to coarse subrounded GRAVEL (Glacial Gravels)" "**SAMP" "*HOLE_ID","*SAMP_TOP","*SAMP_REF","*SAMP_TYPE","*SAMP_BASE","*SAMP_DATE","*SAMP_TIME""*GEOL_STAT "<UNITS>","m","","","m","dd/mm/yyyy","hhmmss","","" "TP501","1.00","1","D","1.00","","","B","" "TP501","1.00","2","B","1.30","","","B","" "TP501","2.50","3","B","2.75","","","C","" "BH502","1.00","1","U","1.45","","","","FS058" "BH502","1.50","2","D","1.50","","","","" "BH502","3.00","3","U","3.45","","","",""

Group Names Group Name Contents Parent Table ABBR Abbreviation Definitions New CBRG CBR Test - General SAMP CBRT CBR Test CBRG CDIA Casing Diameter by Depth HOLE New CHEM Chemical Tests SAMP Del CHIS Chiselling Details HOLE New CHLK Chalk Tests SAMP CLSS Classification Tests SAMP CMPG Compaction Tests General SAMP CMPT Compaction Tests CMPG CNMT Contaminant and Chemical Testing SAMP CODE Chemical Testing Codes New CONG Consolidation Test - General SAMP CONS Consolidation Test CONG CORE Rotary Core Information HOLE Example of the 68 groups in AGS3 plus 4?groups in AGS-M

Group Names (full list) Group Name Contents Parent Table ABBR Abbreviation Definitions CBRG CBR Test General SAMP CBRT CBR Test CBRG CDIA Casing Diameter by Depth HOLE CHEM Chemical Tests SAMP CHIS Chiselling Details HOLE CHLK Chalk Tests SAMP CLSS Classification Tests SAMP CMPG Compaction Tests General SAMP CMPT Compaction Tests CMPG CNMT Contaminant and Chemical Testing SAMP CODE Chemical Testing Codes CONG Consolidation Test General SAMP CONS Consolidation Test CONG CORE Rotary Core Information HOLE DETL Stratum Detail Descriptions HOLE DICT User Defined Groups and Headings DISC Discontinuity Data HOLE DPRB Dynamic Probe Test DPRG DPRG Dynamic Probe Test General HOLE DREM Depth Related Remarks HOLE FILE Associated Files FLSH Rotary Core Flush Details HOLE FRAC Fracture Spacing HOLE FRST Frost Susceptibility SAMP GAST Gas Constituents SAMP GEOL Stratum Descriptions HOLE GRAD Particle Size Distribution Analysis Data SAMP HDIA Hole Diameter by Depth HOLE HOLE Hole Information HPGI Horizontal Profile Gauge Installation Details HOLE HPGO Horizontal Profile Gauge Observations HPGI ICBR In Situ CBR Test HOLE IDEN In Situ Density Test HOLE INST Single Point Instrument Installation Details HOLE IOBS Single Point Instrument Readings INST IPRM In Situ Permeability Test HOLE IRDX In Situ Redox Test HOLE IRES In Situ Resistivity Test HOLE ISPT Standard Penetration Test Results HOLE IVAN In Situ Vane Test HOLE MCVG MCV Test General SAMP MCVT MCV Test MCVG POBS Piezometer Readings PREF PREF Piezometer Installation Details HOLE PROB Profiling Instrument Readings PROF PROF Profiling Inst Installation Details HOLE PROJ Project Information PRTD Pressuremeter Data Test PRTG PRTG Pressuremeter Test Results General HOLE PRTL Pressuremeter,Individual Loops PRTG PTIM Hole Progress by Time HOLE PTST Laboratory Permeability Tests SAMP PUMP Pumping Test HOLE RELD Relative Density Test SAMP ROCK Rock Testing SAMP SAMP Sample Reference Information HOLE SHBG Shear Box Testing General SAMP SHBT Shear Box Testing SHBG STCN Static Cone Penetration Test HOLE SUCT Suction Tests SAMP TNPC Ten Per Cent Fines SAMP TRIG Triaxial Test General SAMP TRIX Triaxial Test TRIG UNIT Definition of<units> and CNMT_UNIT WETH Weathering Grades HOLE WSTK Water Strike Details HOLE

How do I use AGS Data? Initially most files were created by hand or with inhouse software Data Files are now produced and used by commercially available software. A list is available on the AGS web site. (www.ags.org.uk) For training and demonstration, files can be made by converting spreadsheets

AGS files and spreadsheets

What can I do with AGS Data? Borehole logs Data analysis - graphs and reports Site plans Sections Searches Reports 3D modelling and contour plots Archives Others????? All without re-entering data

Data,Software & Systems Systems Software Data

Implementation 1 Developed by the Geotechnical Industry The industry accepted the need slowly Software houses and in-house teams developed the programs Clients specified the format Investment is large Engineers and Geologists use the format New applications develop

Implementation 2 Training of Engineers and Geologists Information for Managers and Clients Demonstration projects International Projects Committee to maintain and develop the format and protect users interests Bulletin board on web site Liaison with other initiatives

Software with AGS compatibility Program AGS File Manager ALF Contam Data System CPT-pro DataSystem 7 GEODASY GEOVIEW gint AGS gint LogWRITER+ gint HoleBASE III INCLI-pro KeyAGS KeyGeoView KeyHOLE KeyLAB MonitoringPoint PocketSI SID TECHBASE WinLoG Category Data validation Data validation Geoenvironmental database systems Insitu testing Laboratory testing (soil) Database systems (with log production) Geographical information systems Checker Data validation Database systems (with log production) Professional Database systems (with log production) Database systems (with log production) Instrumentation Data validation, Geographical information systems Database systems (general) Laboratory testing (soil) Instrumentation and monitoring Field data collection Database systems (with log production) Database systems (with log production) Borehole log production

Software with AGS compatibility

Instruments Electronic Development 1 AGS-M Monitoring System Data Logger Written Records Electronic Paper Software Collection System AGS Data File AGS Data File Another Software Application AGS Data File Another Software Application Lab Data Field Data Site Investigation System Paper or electronic Paper or electronic Software Collection System AGS Data File AGS Data File Another Software Application AGS Data File Another Software Application

The Group Names for Monitoring Data AGS-M HOLE (Reference Point Information)?MONP ( Information about the Monitoring Point)?MONR (Readings of ALL types)?icct (Chemical Test Results)

The Group Names for Monitoring Data AGS-M Group Name Contents Parent Table PROJ Project Information HOLE Reference Point Information?MONP Monitoring Point HOLE?MONR Monitoring Point Reading MONP?ICCT Insitu Contamination and Chemical Testing MONP?TREM Time Related Remarks ABBR Abbreviations CODE CNMT and ICCT 'Pick' List Codes DICT User Defined Groups and Headings FILE Associated Files UNIT Definition of <UNITS>

Development 2 Development of hand held field data capture systems on PDA s - PocketSI, Beclogger British Geological Survey (BGS) database Highways Agency (HA) Geotechnical Data Management System (HAGDMS) Web based projects boreholes.com and monitoringpoint.com for the management and distribution of information Specialist software for management and processing of geotechnical data, eg CPT-pro

Development 3 Work in progress:- Bilingual versions for descriptions ASTM D2488 based fields for GEOL_GEOL CPT information by the Dutch Chemical contamination testing has been updated

Development 3 XML XML development is required Enables new computer developments Use of the existing data dictionary Preserves the investment to date Maintains current format where required Data dictionary is FULLY tested after 12 years of practical use Will be invisible to most users A number of XML versions have been trialled

Acknowledgements I would like to thank the following for their assistance in the production of this presentation Dr. Roger Chandler of Key Systems Tim Spink of Mott Macdonald Dave Patterson of the Highway Agency The data format committee of the Association of Geotechnical and Geoenvironmental Specialists Data journey concept used with permission from Electronic geotechnical data formatting for the future R.J. Chandler, R.Hutchison Proceedings of the 12 th regional conference for Africa on soil mechanics and geotechnical engineering. Durban 1999

Conclusions The AGS format has proved to be sound over 12 years+ in use. The Data Dictionary has been tried and tested, it is flexible and has future potential. Implementation is a long and frustrating process but once achieved is worthwhile. The investment by companies is significant and needs protection. The format has already enabled significant software development. An XML version will permit even more software in the future and by using the same data dictionary will preserve the work to date. Liaison with others throughout the world is in progress.