Brief Technical Report on a Workshop on Integrating Predictive Models into the CRM Process

Transcription

1 Brief Technical Reprt n a Wrkshp n Integrating Predictive Mdels int the CRM Prcess Legacy Resurce Management Prgram, Prject David W. Cushman, Christpher L. Nagle, and Michael Heilen Prepared with funding frm the Legacy Prgram fr Air Frce Air Materiel Cmmand, Martyn D. Tagg, Principal Investigatr Submitted t: Paul R. Green, Ph.D. Headquarters Air Cmbat Cmmand 129 Andrews Street, Suite 102 Langley AFB, VA and Cecilia Brthers Legacy Prgram CRM Specialist 1225 Suth Clark Street, Suite 1500 Arlingtn, VA March 2010

2 Technical Reprt n the Wrkshp n Integrating Predictive Mdels int the CRM Prcess INTRODUCTION Since 2001, several studies were funded via the Department f Defense s (DD s) Legacy Resurce Management Prgram that investigated the integratin f mdeling int DD cultural resurce management (CRM) cmpliance. The Advisry Cuncil n Histric Preservatin and the Natinal Cnference f State Histric Preservatin Officers have been cnsulted regarding this integratin. T date, these studies have demnstrated that predictive mdels can prvide a strng scientific fundatin fr decisin-making abut archaelgical resurces. These studies have als demnstrated that fr predictive mdels t be sufficiently strng t be used in cmpliance, they must address issues f data quality and statistical mdeling and fully utilize the capabilities f gegraphic infrmatin systems (GIS) technlgy. The mdels used in these studies n DD lands have cnsistently shwn weaknesses that culd be easily vercme with a better understanding f hw past use f landscapes culd be cnceptualized in GIS. The key t imprving mdels, which was identified as part f an earlier Legacy prject (#01-167) in 2001, is training. The purpse f the current prject was t rectify this deficiency by develping and delivering a pilt wrkshp dedicated t imprving technical aspects f predictive mdeling and identifying key prcesses fr integrating mdeling in CRM cmpliance. The wrkshp was delivered n August 10, 2009, at the 2009 Sustaining Military Readiness Cnference in Phenix, Arizna. This reprt describes hw this training was develped, the results f delivering the wrkshp, and recmmendatins fr future delivery f this wrkshp t DD CRM and Natinal Envirnmental Plicy Act (NEPA) staff in the future. BACKGROUND Three DD Legacy prjects n predictive mdeling, in additin t an nging Envirnmental Security Technlgy Certificatin Prgram (ESTCP) prject (# ), served as the fundatin fr the pilt wrkshp. The Legacy prjects included #01-167, which evaluated the strengths and weaknesses f predictive mdels used at fur DD installatins, and Legacy prject (#03-167), which invlved a wrking cnference f DD managers, State Histric Preservatin Office (SHPO) representatives, tribal representatives, and mdeling experts. The wrking cnference reviewed the results f Legacy prject # and made recmmendatins n hw t best utilize mdeling in installatin cmpliance prgrams. The third Legacy prject (#06-167) crafted an utline f a Sectin 106 (f the Natinal Histric Preservatin Act [NHPA]) prgrammatic agreement fr integrating lcatinal predictive mdels with installatin CRM cmpliance and planning. This prject als created the structure f a mdel that prgrammatically evaluates the significance f archaelgical sites, rather than using a site-by-site, case-by-case apprach t evaluatin. The nging ESTCP prject builds n these three Legacy prjects and will result in the develpment f prgrammatic agreements fr fur demnstratin installatins, fully integrating predictive mdeling with installatin CRM cmpliance prgrams. An additinal prduct f the ESTCP prject is Web-based guidance n the develpment and use f predictive mdels and their integratin int an installatin s histric preservatin cmpliance prgram. These Legacy prjects and the ESTCP demnstratin prject shwed that: Predictive mdeling can be used fr NHPA and NEPA decisin-making. Mst large DD installatins d nt use predictive mdels as a tl fr NHPA r NEPA cmpliance. Few DD CRM managers have a clear understanding f hw these mdels are built and updated r hw they can be integrated int the envirnmental review and cmpliance prcess; and few DD 1

3 GIS specialists understand archaelgical data well enugh t design spatial databases that can prvide infrmatin t develp strng predictive mdels, which can then be used as a cmpliance tl. Given these findings and bservatins, there is clearly a need fr training n the develpment, refinement, and validatin f predictive mdels, s they can be effective envirnmental and histric preservatin cmpliance tls. Many installatins have predictive mdels that culd be imprved and then used fr cmpliance activities, if nly installatin persnnel had a greater understanding f the mdeling prcess and GIS techniques and the rle f mdeling in cmpliance. The current pilt wrkshp n integrating predictive mdels int the CRM prcess fills this need. PILOT WORKSHOP As a first step, the SRI Fundatin (SRIF) and Statistical Research, Inc. (SRI), develped a participants wrkbk, cntaining a curse utline and fur training units: (1) building predictive mdels, (2) validating predictive mdels, (3) refining and updating predictive mdels, and (4) using mdels as an envirnmental and CRM cmpliance tl. Next, SRI and SRIF delivered the wrkshp at the Sustaining Military Readiness Cnference, held in Phenix, Arizna, n August 10, During the wrkshp, participants received practical experience n hw t develp predictive mdels and were prvided prblem-slving applicatins using bth real-wrld DD examples and hypthetical scenaris. The first part f the wrkshp invlved a hands-n sessin, using laptp cmputers and GIS prgrams, n hw t build archaelgical predictive mdels, validate mdels, and update mdels with new infrmatin. The secnd wrkshp cmpnent fcused n incrprating predictive mdels int the Sectin 106 and NEPA cmpliance prcess. The wrkshp was partly underwritten by a dnatin f ArcGIS and Spatial Analyst GIS sftware licenses frm ESRI. Specific tpics addressed during the first part f the wrkshp included: Defining the Universe Hw much area shuld be cvered by the mdel? Shuld the installatin be mdeled as a single unit r shuld it be divided by physigraphic features? Military activities? Prpsed impacts? Creating Independent Variables What is a prxy variable? What are the prs and cns f selectively defining envirnmental variables that may relate t human behavir as ppsed t using every envirnmental variable pssible? What envirnmental GIS layers exist and where d yu find them? Creating Dependent Variables What are we mdeling? The difference between GIS shape files and raster cells and archaelgical sites. Can we mdel all sites at nce r d we split them by time and culture? Can we mdel histrical perid sites? What abut buried sites? Mdeling Techniques What is the difference between a Blean, weighted, and a lgistic regressin mdel? What is agent-based mdeling and when is it apprpriate? Validating the Mdel What is a jackknife? Hw d we measure the pwer f a mdel and hw much cnfidence we shuld have in it? Is statistical verificatin enugh? Hw imprtant are independent data? Updating and Refining Mdels Are we interested in a mdel r in mdeling? Hw t incrprate new data int a mdel? Shuld yu refine the mdel incrementally (i.e., as each new survey is entered int the database) r peridically (e.g., every year)? The secnd part f the wrkshp examined the use f predictive mdels as a tl fr Sectin 106 cmpliance, fcusing n the fllwing issues: Anticipating the kinds f prperties likely t be encuntered in an Area f Ptential Effect, s that efficient prperty identificatin, evaluatin, and management strategies can be develped as required under Sectin 106 f the NHPA. 2

4 Anticipating the csts f archaelgical survey needed t satisfy the identificatin requirement under NHPA. Facilitating the use f phased identificatin f archaelgical sites as a means t streamline Sectin 106 decisin-making. Designing an archaelgical sample survey in rder t cllect sufficient infrmatin t identify histric prperties and t assess effects as required under Sectin 106. Evaluating the integrity and Natinal Register eligibility f archaelgical sites. Redesigning undertakings t avid/minimize adverse effects n significant resurces, thus preserving sites and reducing csts. Assisting with tribal cnsultatin by prviding infrmatin early in the Sectin 106 prcess n ptentially sensitive areas r resurces that may be f special cncern. Facilitating the develpment f Prgrammatic Agreements t streamline DD installatin-wide cmpliance with Sectin 106. In the cntext f NEPA cmpliance, this secnd part f the wrkshp als lked at: Linking envirnmental planning with Sectin 106 cmpliance by integrating ptential archaelgical site lcatins with infrmatin n the spatial distributin f natural resurces, knwn and mdeled. Streamlining cnsideratin f archaelgical sites under categrically excluded undertakings. Facilitating the scping f ptential impacts/issues fr a prpsed actin under NEPA and infrming the public and ther stakehlders abut the ptential fr impacts. Selecting alternatives fr detailed study in cmpliance with the NEPA regulatins. Characterizing the affected envirnment fr EAs and EISs under NEPA withut having t physically inventry every alternative. One instructr taught the first part f the wrkshp, and anther instructr taught the secnd part. At the end f the pilt wrkshp, participants were asked t cmplete a wrkshp evaluatin frm. The cmments and ratings prvided by these frms, in additin t the instructrs bservatins made during the delivery f the wrkshp, were subsequently used t revise the wrkshp materials. WORKSHOP RESULTS AND RECOMMENDATIONS At the utset, it had been decided that the maximum number f wrkshp participants wuld be 24. Therefre, SRI rented 24 laptp cmputers, in additin t 2 spare laptps. A ttal f 24 individuals fficially registered fr the wrkshp. Hwever, there were an additinal 24 walk-ins t the wrkshp, and 12 f these individuals stayed and participated in the entire wrkshp, whereas the ther 12 came and went. SRI and SRIF were nt able t cntrl wh participated in the wrkshp, given the verall lgistics established by the Sustaining Military Readiness Cnference rganizers. The fllwing is a breakdwn f the wrkshp participants by service: US Army, 18; US Air Frce, 17; US Navy, 5; US Marine Crps, 3; US Army Natinal Guard, 3; and Others (unidentified), 2. Twenty-tw participants filled ut evaluatin frms fllwing the wrkshp. Many f these participants nted that this type f wrkshp had great value. Based n the cmments received, and the instructr s evaluatins, the fllwing fur issues were identified and need t be addressed befre this wrkshp is delivered in the future: There needs t be a limit n the number f participants, and the wrkshp shuld be pen t nly individuals with specific skills and wrk experience 3

5 Mre than ne instructr is needed t deliver the first part f the wrkshp. A minimum f tw in- structrs is required, r an instructr and an assistant. The wrkshp shuld be expanded t 2 full days Mre real-wrld examples are needed t illustrate hw mdeling can be used t streamline the reg- ulatry cmpliance prcess As nted abve, SRI and SRIF had little cntrl ver the number and type f participants wh culd attend the wrkshp during the cnference. As a result, the cmputer and GIS sftware skill levels f the participants varied tremendusly. This slwed dwn the pace f the hands-n cmputer-based training and resulted in teaching individuals with n practical experience with GIS. Fr future wrkshps, the number f participants shuld be limited t individuals. The first part f the wrkshp shuld fcus n GIS technical specialists, with their DD CRM and NEPA cunterparts sitting next t the specialist as they wrk. In this way, the CRM and NEPA staff will have at least a basic understanding f the requirements fr mdel develpment, validatin, and refinement. The secnd part f the wrkshp shuld fcus n DD CRM and NEPA staff, with the GIS specialists als participating s they can understand the cntext fr their GISbased mdeling effrts. In terms f instructrs, it is recmmended that the instructr fr the first part f the wrkshp have the assistance f secnd instructr r a GIS analyst/technician wh can wrk with the participants at their cmputers while the primary instructr prvides the verall lessns. As nted abve, the 1-day frmat fr the wrkshp was inadequate fr presenting the necessary lessn cntent. In revising the wrkshp, SRI and SRIF have redesigned the structure and the cntent f the wrkshp t fit a 2-day frmat. Several participants nted in their evaluatins that the data preparatin exercises in part ne f the wrkshp cnsumed t much time. Accrdingly, SRI and SRIF revised these lessns t decrease the time spent n data preparatin and nw include mre finished GIS data layers. This revisin will make it pssible t avid much r all f the data preparatin and prceed directly t mdel building. Finally, t address the cmments n the lessns dealing with hw t use predictive mdeling t streamline the regulatry cmpliance prcess, SRI and SRIF revised the wrkshp t include mre real-wrld examples frm DD and ther federal agencies. Given the psitive respnse frm the pilt wrkshp participants, SRI and SRIF recmmend that the revised wrkshp shuld be made available t DD installatin CRM and NEPA staff, and the cnsultants and ther rganizatins that wrk with these staff in the cntext f bth NEPA and Sectin 106 cmpliance prgrams. Lcatins fr delivering the revised wrkshp might include natinal Air Frce and Army CRM meetings, prfessinal meetings such as the Sciety fr American Archaelgy and the Sciety fr Histrical Archaelgy, and ther military cnferences that attract significant numbers f DD CRM prfessinals. If the wrkshp is delivered at these types f natinal meetings, the wrkshp instructrs must have the ability t restrict the number and type f participants. It als may be useful t deliver the wrkshp at reginal DD meetings that bring tgether persnnel frm installatins with similar envirnments and cultural settings. Fr example, the wrkshp can be tailred fr desert-based installatins in Califrnia, Arizna, and Nevada. The wrkshp can als be tailred fr a specific, large installatin with cmplex military missins and extensive cultural resurces, such as Frt Bliss in Texas and New Mexic. 4