Optimizing Anti-Terrorism Resource Allocation

Transcription

1 Optimizig Ati-Terrorism Resource Allocatio Steve R. Hayes ad Thomas George Kaampallil School of Iformatio Scieces & Techology, Pe State Uiversity, Uiversity Park, PA Lawrece L. Larso Marie Corps Research Uiversity, Pe State Uiversity, Uiversity Park, PA Nitesh Garg Departmet of Computer Sciece & Egieerig, Pe State Uiversity, Uiversity Park, PA Sice sprig of 2002 we have bee workig o a methodology, decisio model, ad cogitive support system to aid with effective allocatio of ati-terrorism (AT) resources at Marie Corps istallatios. The work has so far bee focused o the military domai, but the model ad the software tools developed to implemet it are geeralizable to a rage of commercial ad publicsector settigs icludig idustrial parks, corporate campuses, ad civic facilities. The approach suggests that ati-terrorism decisio makers determie mitigatio project allocatios usig measures of facility priority ad mitigatio project utility as iputs to the allocatio algorithm. The three-part hybrid resource allocatio model preseted here uses multi-criteria decisiomakig techiques to assess facility (e.g., buildig, hagar) priorities, a utility fuctio to calculate atiterrorism project mitigatio values (e.g., protective glazig, wall coatigs, ad stad-off barriers) ad optimizatio techiques to determie resource allocatios across multiple, competig AT mitigatio projects. The model has bee realized i a cogitive support system developed as a set of loosely coupled Web services. The approach, model, ad cogitive support system have bee evaluated usig the cogitive walkthrough method with prospective system users i the field. I this paper we describe the domai, the problem space, the decisio model, the cogitive support system ad summary results of early model ad system evaluatios. Itroductio The recet history of terrorist attacks o both military ad civilia targets at home ad abroad suggests a eed for icreased emphasis o effective implemetatio of mitigatios to protect idustrial facilities. The effects of attacks Accepted February 26, Wiley Periodicals, Ic. Published olie 2 December 2004 i Wiley IterSciece ( DOI: /asi such as the 1996 truck bombig of the Khobar Towers housig complex, the 1998 U.S. embassy bombigs i Keya ad Tazaia, ad recet attacks o expatriate housig complexes i Saudi Arabia may have bee dimiished had more effective facility ati-terrorism mitigatios bee istalled. Ati-terrorist mitigatios are geerally expesive, however, ad decidig what, how, where, ad whe to allocate resources to protect critical ifrastructure is a difficult problem. We have bee workig o a methodology, decisio model, ad cogitive support system to aid with more effective allocatio of ati-terrorism (AT) resources at Marie Corps istallatios. Though our work has so far focused o the military domai, the methodology, decisio model, ad supportig tools are geeralizable to a rage of commercial ad public-sector settigs icludig idustrial parks, corporate campuses, ad civic facilities. Ati-terrorism facility mitigatios are desiged to protect people ad property from asymmetric attack by otraditioal hostile orgaizatios. Ati-terrorism mitigatios iclude measures such as blast-resistat glazig (widows are the major cause of ijury ad death i most bomb attacks), wall-hardeig coatigs, feces, vehicle stad-off barriers, ehaced lightig, video surveillace, ad guards. Allocatig available fuds across differet ati-terrorism mitigatio projects, with the exceptio of guards, is a project selectio ad capital budgetig problem. Though a rage of welldeveloped techiques to address such problems are used i both the private ad public sectors, ad are widely researched, the ati-terrorism domai presets some special problems that make may of these techiques usuitable. Chief amog these are the eed to assess trade-offs betwee the cost of mitigatios, the fiacial beefits of protectig valuable facilities ad other assets, ad the value of the people protected whe mitigatios are applied. At the begiig of the paper, we provide some backgroud o project selectio ad capital budgetig techiques JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY, 56(3): , 2005

2 commoly employed. We the describe the domai of atiterrorist resource allocatio ad the hybrid decisio model assembled to address some of the special problems that arise i the domai. I the ext sectio we describe the cogitive support system that has bee costructed to realize this decisio model ad give summary results from early field evaluatios of both the model ad system. We coclude with a discussio of some aveues for further work i the AT resource allocatio problem space. Backgroud Project selectio is a critical task for may orgaizatios icludig govermet fudig agecies, uiversities, research istitutes, ad techology ad capital-itesive compaies. It is a complex decisio-makig process cosistig of multiple stages, with multiple groups of decisio makers, multiple ad ofte-coflictig objectives, ad high risk ad ucertaity i predictig the future success ad impacts of differet project combiatios (Ghasemzadeh & Archer, 2000). Amog the methods employed to address project selectio problems are ustructured peer review, scorig, mathematical programmig, ecoomic models, decisio aalysis, artificial itelligece, ad portfolio optimizatio (Herikse & Trayor, 1999). Multi-attribute utility methods are the basis for a umber of project selectio methods. Barabasoglu ad Pihas (1995), for example, describe a capital budgetig process usig the Aalytic Hierarchy Process (AHP) for resource allocatio. The method uses AHP for prioritizig the differet resources, ad uses a liear program for allocatig limited budget resources. Korhoe ad Walleius (1990) describe a dyamic decisio support system for solvig multiple liear programmig problems by usig the weightig techique of AHP. Liberatore (1987) focuses o structurig the decisio hierarchy so that the AHP ca be successfully implemeted i a capital budgetig decisio problem. Thursto (1991) demostrates the applicatio of multi-attribute decisio makig for selectig the best desig alterative, couplig optimizatio methods with multi-attribute aalysis. The utility of each desig alterative is evaluated, ad the oe that maximizes utility is chose. Greier, Fowler, Shuk, Carlyle, ad McNutt (2003) describe how to itegrate AHP with a 0 1 iteger portfolio optimizatio problem for supportig decisio-makig activities i the military domai, which provides a screeig process i selectig ew weapos developmet projects. Efforts to make these techiques more usable for a wider rage of stakeholders typically ivolve developmet of computer-aided decisio support systems to support project selectio tasks (Bard et al., 1988; Kocaoglu & Iyigu, 1994; Ghasemzadeh & Archer, 2000). Accordig to Liberatore (1987) most orgaizatios use project selectio ad justificatio tools that embed stadard fiacial aalysis methods such as Cost Beefit Aalysis (CBA) or Discouted Cash Flow (DCF). Orgaizatios i the commercial sector typically use oe of four methods for capital project rakig: payback, et preset value (NPV), regular iteral rate of retur (IRR), ad modified iteral rate of retur (IRR*) (Brigham, 1989). Proctor ad Caada (1992) provide a extesive summary of literature dealig with capital budgetig processes based o the discouted cash flow approach. These methods focus o the fiacial retur obtaied from a give course of actio. Though terrorist attacks geerally do have a fiacial cost, ad potetially impact the ability of a orgaizatio to geerate future reveue, the beefits of applyig specific AT mitigatios also iclude protectig people ad providig a eviromet i which they ca feel safe ad secure, ad esurig that the facilities platform is available to support the orgaizatio s missio. Though the fiacial costs of differet AT mitigatios are quite easy to derive, the beefits of these mitigatios are much more difficult to determie give the low probability that a give mitigatio will actually be used, ad the itagible ature of their may beefits. The methods ad tools described i this paper are oe approach to effectively maagig the AT resource allocatio problem. Traditioal methods for project selectio ad capital budgetig have ofte bee judged iadequate because they fail to accout for oquatifiable itagible factors ad to establish the lik betwee the capital ivestmet decisio ad orgaizatioal priorities (Sulliva & Smith, 1990). Otley (1999), for example, poits out that budget cotrol mechaisms ofte hide or make implicit the orgaizatioal priorities ad capabilities that they represet ad idetifies a series of challeges to be addressed whe assessig true performace of available resources. Amog the most relevat of these to the case of ati-terrorism resource allocatio are: How are budgets figures related to strategic goals? How are resources allocated (budgeted) relative to strategic goals? Amog the approaches sometimes used to address such issues is Value Focused Thikig (Keeey,1994),which ivolves iterative decompositio of a istitutio s high-level values ad objectives ito successively more detailed ad lower-level objectives. Empirical studies of VFT suggest that broader ad high-quality decisio alteratives (decisio opportuities i the VFT lexico) are idetified whe VFT rather tha traditioal,alterative-focused thikig approaches are employed (Keeey,1994). Aother approach is the Aalytic Hierarchy Process (AHP; Saaty,1980),which has bee used i cojuctio with more traditioal,fiace-based methods to solve project selectio ad capital budgetig problems (Varey,Sulliva,& Cochra,1985; Wabalickis,1987). Liberatore (1987) for example,describes a case i which AHP was successfully itegrated ito the capital-budgetig decisio process. Ati-Terrorism Project Plaig ad Allocatio It is importat that desigers of cogitive support systems esure that the support they provide helps aswer the right questios with respect to user requiremets i the domai 300 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY February 1, 2005

3 (Parker, 2001). Scearios are oe useful method for helpig to egotiate ad idetify the cocrete uses of a give decisio model, ad for showig how techology ca be itegrated i the task domai to the best effect. Scearios take the form of arratives that describe the details of behaviors, tasks, ad techology support i a desig space (Carroll, 2000; Carroll & Rosso, 1992). Scearios help to both evisio ew techology support ad to evaluate how well evisioed or implemeted techologies afford useful ad usable advatages i their domai of use. The followig are amog the key scearios drivig desig of our methodology, model, ad cogitive support system ad are provided to help portray how the model ad cogitive system are plaed for use. Allocatig a Ati-Terrorism Budget Over Multiple Locatios A Uited States Marie Corps (USMC) headquarters facilities plaer apportios a global budget across a geographically dispersed, fuctioally heterogeeous istallatio set. I this sceario, the plaer attempts to optimize orgaizatioal security ad capabilities by fudig mitigatios i support of critical worldwide activities. The facilities plaer begis with a madated AT budget, assesses each idividual istallatio s cotributio to these orgaizatioal objectives, assesses localized threats, ad allocates resources accordigly. Allocatig a Ati-Terrorism Budget Over Multiple Facilities at a Sigle Locatio The ati-terrorism team at a particular istallatio must allocate a fixed AT budget across facilities with varyig priorities ad cocers. The budget icludes ot oly the amout allocated by USMC headquarters but also additioal fuds budgeted by the local commader. A crossfuctioal team of persoel from public works, operatios, ad security recocile diverse protectio perspectives ito a sigle prioritized list of facilities. Mitigatio projects are the fuded by priority util the budget is exhausted. Plaig ad Justifyig a Ati-Terrorism Budget (Budget Programmig) A ati-terrorism team is resposible for idetifyig the AT requiremets for a give istallatio. Properly prioritized requiremets are a budget-programmig tool that the team ca use to justify resource requests. Whe a set of such requests is compiled by USMC headquarters across all istallatios, it represets a sapshot of the resources eeded to achieve a specific level of protectio across the etire service. Resource Allocatio Decisio Model We coceptualize the AT resource allocatio problem as cosistig of three major compoets: prioritizig facilities FIG. 1. AT Resource allocatio problem space. ad other assets to be protected, idetifyig the relative utility of differet mitigatios ad mitigatio projects (the latter are combiatios of mitigatios), ad allocatio of available resources (moey, time, people) to protect the highest priority facilities with those mitigatios providig the highest utility, subject to the costraied resources. A high-level view of this coceptualizatio of the AT plaig problem space is depicted i Figure 1. Decisio Model Ratioale Desigig a model for ratioal decisio makig i a complex domai such as AT plaig ivolves resolvig the eed for accuracy i the results of the model with the cost of capturig the data eeded for this icreased accuracy. Models desiged to provide perfect solutios ofte require perfect data to be useful. These may fail i practice whe eeded data are uavailable or the cost ad effort eeded to obtai them is urealistic. A sigificat challege to developmet of techiques ad tools to aid ati-terrorism decisio makers is the rage of differet data that potetially ifluece the decisio process. The cost of idetifyig,collectig,ad the mergig or fusig this data to make it comprehesible to decisio makers must be traded off agaist the icreased accuracy of the system solutios as the amout of available data icreases. A key teet i the desig of the decisio model ad cogitive system described here is that the system be useful with varyig data availability. I other words,eve with a icomplete data set the decisio maker should be i a positio to make decisios of a higher quality tha whe o data ad o cogitive support system are available. The approach take here to resolvig this trade off is to decouple or compartmetalize the problem-solvig uits or modules of the system such that each provides stad-aloe support to the AT decisio maker. For example, i cases where specific mitigatio project utility data are uavailable, JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY February 1,

4 the model s prioritizatio piece ca still be used to idetify which facilities are most importat to protect. If prioritizatio data are uavailable the the mitigatio project utility piece ca still be used to give a assessmet of the relative impact of differet mitigatio projects. Ati-terrorism plaers ad decisio makers may also use these model parts separately to perform rough assessmets of a istallatio s AT profile. Two factors ofte used to determie where ad whe these mitigatios will be applied iclude assessmets of facility vulerability ad threat. Efforts to measure facility vulerability are drive by factors such as locatio of the facility relative to a aveue of attack (e.g., public roadways), the actual or symbolic importace of the facility, ad whether or ot the type of the facility is oe historically subject to terrorist attack. However, the actios of terrorist orgaizatios are iheretly upredictable ad adaptive; usig vulerability to drive the allocatio of AT mitigatios may oly chage rather tha elimiate terrorist targets. The perceived threat to a facility is ofte a fuctio of specific, timesesitive itelligece data, but the latecy of costructio improvemet projects ad the relative rigidity of most capital-budgetig processes mea that decisio makers caot respod quickly to specific threat data. Both of these perspectives also focus solely o the curret status of the facility to be protected ad fail to accout for the impact of differet AT mitigatio projects, i other words, how the situatio improved oce mitigatios are applied. Facility Prioritizatio The first part of the AT resource allocatio problem is facility prioritizatio. The primary objective of this compoet is to prioritize a set of idetified assets a set which itself is idetified as all possible assets to protect, or some subset created usig heuristics based o a set of agreedupo rakig criteria. The two most widely used methods i multicriteria rakig problems are the Aalytical Hierarchy Process (AHP; Saaty, 1980) ad the Simple Multi-Attribute Ratig Techique (SMART; Edwards, 1977). Other alterative focused methods scorig methods are Simple Attribute Weightig (SAW), Weighted Product Method (WPM), ad Techique for Order Preferece by Similarity to Ideal Solutio (TOPSIS), ad ELECTRE (Hwag & Yoo, 1981). The Aalytical Hierarchy Process supports the icorporatio of both objective, quatifiable data ad the more subjective qualitative data ito the decisio process (Saaty, 1980). The Aalytical Hierarchy Process also facilitates the support for the differet phases of the decisio-makig process itelligece, desig, ad choice (Simo, 1960). The process ca also be used for idividual ad group decisio makig, a meas for coceptualizig ad commuicatig about the problem domai, which supports the buildig of a commuity with a shared visio. To prioritize facilities, decisio makers must choose from amog alteratives based o potetially coflictig objective performace measures. For example, a decisio maker must make tradeoffs betwee the populatio desity of a give facility, ad the extet to which loss of the facility impedes the orgaizatio s missio. Determiig the appropriate tradeoffs may require subjective value judgmets, based o idividual ad/or orgaizatio values ad expertise. Though some huma experts may be able to rak alteratives without beig able to give a explicit rule for trade-off values, various techiques have bee developed to help weight the importace of each of their objectives i a more explicit, quatifiable maer. The Aalytic Hierarchy Process (Saaty, 1980) is used to determie priority criteria ad to assess alteratives with respect to those criteria. The AHP ivolves decomposig a decisio problem ito a hierarchy cosistig of decisio goals at the top, criteria for selectig amog alteratives at the secod level, ad alteratives for selectio at the bottom of the hierarchy. Each of these three levels may themselves be further decomposed ito criteria categories each with a set of related criteria, or the subgoals that help clarify ad specify the operatioalizatio of the highest level objective. As prescribed i AHP, the facilities prioritizatio process begis with idetificatio of the criteria that will be used to develop the facility rakig. The AHP leads decisio makers ad domai experts through a series of pair-wise objective comparisos to determie the relative weights of differet prioritizatio criteria ad through assessmet of each idetified alterative with respect to each criterio. The AHP method ca also provide feedback to decisio makers o their valuatios usig sesitivity aalysis, cosistecy checks, ad simulatios showig the implicatios of differet objective weightigs. I additio, it provides a platform for discussio ad cosesus buildig amog a group of experts over the relative importace of each criterio ad alterative. Amog the criteria i the Marie Corps istallatio plaig domai are: Populatio desities Missio importace Cost to replace Time to replace Whether alterative facilities are available The pair-wise comparisos cetral to the AHP approach may be performed iteratively by all stakeholders i a give decisio cotext to arrive at cosesus values. The AHP has bee used for solvig capital ratioig ad resource allocatio problems by covertig the results of a AHP model ito sigle objective, maximizatio-type liear programmig problems (Ramaatha & Gaesh, 1995). I the defese domai, for example, a relatively complex, twolevel AHP hierarchy has bee used as the maximizatio basis for a iteger program (kapsack algorithm) that helps filter a set of proposed weapos systems projects ito a subset of those to be fuded for further research ad developmet work. I Barbarosoglu ad Pihas (1995), the 302 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY February 1, 2005

5 resource allocatio problem of the Istabul Water ad Sewage Authority is formulated usig AHP to help quatify the itagible, social factors that must be cosidered to most effectively allocate World Bak loa fuds to a set of competig waterworks upgrade projects. A weakess of the AHP approach is that the umber of idividual AHP comparisos required ca become uwieldy as the umber of criteria ad/or alteratives beig cosidered icreases,ad this issue is exacerbated whe multiple stakeholders egage i the compariso process. Variatios of the AHP approach exist that somewhat ameliorate this issue,but with trade-offs i cosistecy ad/or accuracy. These will be explored further i the ext phase of the project. Mitigatio Project Utility The secod compoet of the AT decisio model is a beefit-cost assessmet of differet AT mitigatio projects. This part of the model is a expressio of the moderated beefit-cost ratio of a give mitigatio as applied to a give istallatio asset (e.g., a buildig). For capital budgetig projects i the public sector, where ofiacial elemets may play a key role i determiatio of overall project utility, the beefit-cost ratio is a widely used measure to assist decisio makers (Au & Au, 1992). Cost beefit aalysis (CBA) is the pricipal aalytical framework used to evaluate public expediture decisios to allocate scarce resources i a more efficiet way. It was developed durig the 1930s whe the U.S. govermet had embarked i several costly costructio programs, such as dams i the West, ad eeded CBA to justify these expeditures to the public ad to Cogress. Cost/Beefit Aalysis ca be carried out usig oly fiacial costs ad fiacial beefits ad it is possible to iclude itagible items withi the aalysis. As a value must be estimated for these, it brigs a elemet of subjectivity ito the process. A AT example, it determies the relative beefit of applyig a hardeig mitigatio to the glazig of a certai buildig relative to the utility of applyig other mitigatios, or of the utility of applyig mitigatios to a differet buildig. A early challege to the developmet of this model is idetificatio of the factors or attributes that should be cosidered as part of the calculatio, or utility fuctio. Amog the elemets curretly icluded i calculatio of mitigatio project utility are the followig: Beefits People protected Equipmet protected Facility protected Costs Mitigatio direct costs Mitigatio idirect costs Utility Factors Mitigatio effect ratig Mitigatio iteractio effects Risk data Vulerability data The purpose of calculatig mitigatio project utility is to evaluate the et beeficial output achieved by providig differet mitigatios to a facility or other assets. What makes the utility fuctio so difficult to model is that it requires specificatio of the relatioship betwee the beefits of protectig lives ad the beefits of protectig buildigs ad equipmet. This specificatio may be a requiremet i the military domai,where the missio importace of buildigs ad equipmet caot be completely discouted with respect to huma life,but it may be the case that i civilia domais the umber or people protected domiates or trumps all other beefits derived from a give AT mitigatio project. To address this coudrum we are relyig especially o the use of sesitivity aalysis visualizatio tools to make apparet to decisio makers the structure ad cotet of the utility calculatio behid each mitigatio project. Further studies with users of the decisio model ad cogitive support system will,we hope,help tue this model further such that it more accurately reflects these importat relatioships. Resource Allocatio The optimizatio egie for resource allocatio is a iteger program. The iteger program is used to maximize the mitigatio utility with the budget ad mitigatio project costs as the costraits. The problem is solved as a iteger program, sice partial allocatio of budgets is ot allowed. Two differet approaches have bee suggested for similar applicatios (Ramaatha & Gaesh, 1995). I the first, the AHP priorities are used as coefficiets i the objective fuctio of the liear program formulatio. I the secod, the beefit-cost ratios are used as the coefficiets. We combie the AHP ad the beefit-cost ratio to form the coefficiet to the optimizatio problem usig the istallatio budget ad the sum of mitigatio costs for each project as the costraits. The solutio to this yields project allocatios with the highest aggregate utility to the highest priority assets. For AT resource allocatio it is required that a particular mitigatio project is either selected for resource allocatios or ot selected at all, i.e., there is o partial allocatio of resources. Icorporatig 0 1 iteger programmig is the methodology for solvig these types of problems. A 0 1-iteger program is used to maximize the mitigatio utility with the budget ad mitigatio project costs as the costraits (Kyparisis,Gupta,& Ip,1996). These types of 0 1 iteger programmig models, commoly referred to as kapsack problems, are based o the premise that the decisio maker wats to defie a selectio that provides optimal value while meetig a specific costrait, a budgetary costrait i this case (Greier et al.,2003). JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY February 1,

6 TABLE 1. Example raw scores for prioritizatio criteria. Cost Hours Alteratives Populatio Time Missio Cost Hours Alteratives Populatio Time Missio Facility Prioritizatio Example The first step i the USMC AT/FP resource allocatio model ivolves idetifyig facilities to be cosidered for protectio ad rakig them based o a agreed-upo criteria set. To prioritize facilities,decisio makers must choose from amog alteratives based o potetially coflictig objective performace measures. The ext step i the AHP process is to perform a pair-wise compariso amog the differet criteria to arrive at a criteria weight score. Decisio makers make comparisos of each pair of criteria usig a 9-poit scale; a represetative set icludes the five below. Equally importat 1 Moderately more importat 3 Strogly more importat 5 Very strogly more importat 7 Extremely more importat 9 Itermediate scores 2, 4, 6, 8 Usig the 9-poit scale, the decisio maker must determie the importace, preferece, or likelihood of each elemet whe compared to every other elemet at the same level of the problem space hierarchy. For example, a value of 1 would mea two criteria are equally importat, while a value of 7 meas oe criterio is very strogly more importat tha the other. Raw scores for the example criteria set are show i Table 1. Oce the pair-wise comparisos have bee completed, the criteria weight score is computed. This is calculated as follows. Each colum is summed ad the each cell value (colum) is divided by the total to get the ormalized values. For the colum Cost, the total is 20, ad the ormalized value for the first cell (cost cost) is 1/20. After the ormalized values are calculated, the average row sum of ormalized values is calculated. This value is the criterio score for that criterio (Table 2). TABLE 3. to cost. Example raw prioritizatio scores for alteratives relative Traiig Maiteace Supply Admi Housig Utilities Traiig Maiteace Supply Admi Housig Utilities The ext step is to perform a compariso of each alterative pair with respect to each criterio. Table 3 above shows the alterative pair-wise compariso for the Cost criterio. The pair-wise comparisos for the alteratives are similar to the criteria priority weightig. The alterative weight score is computed for each criterio. Oce the etire alteratives weight scores for each criterio is computed, the alterative priority score is calculated as a matrix product of the criteria score ad alterative weight scores. Table 4 below gives the priority weight scores of each of the alteratives. From the table it ca be see that maiteace has a higher priority tha other facilities like traiig ad supply. Mitigatio Project Utility Example A base project mitigatio utility is a additive fuctio ad is calculated as follows: u Adjusted utility value calculated above each project c Mitigatio cost for each project b Beefits for each project (replacemet cost) a b Raw mitigatio project utility a utility factors a c The raw mitigatio utility, is thus the ratio of beefit to cost added to the utility factors for each project. This process is cotiued across all available projects. Utility factors iclude exteral factors like risk data, threat data, etc. These are quatified as accelerators ad decelerators to the systems. (1) TABLE 2. Criteria weight score. TABLE 4. Asset priority score. Criteria Criteria score Asset class Asset class priority Cost Hours Alteratives Populatio Time Missio Maiteace Traiig Supply Housig Utilities Admi JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY February 1, 2005

7 Raw mitigatio utility is the utility value that is obtaied as a result of the beefit-cost ratio calculatio. This value has to be ormalized i the ext step to avoid domiace of mitigatio utility value over the others. As there ca be sigificat differeces i the mitigatio project utility values for differet mitigatio projects depedig o the beefit cost ratios, it is possible that a particular project utility will be domiat. To avoid this situatio, the mitigatio project utility values are ormalized as follows for each project: Normalized mitigatio project utility where total umber of projects. Sice differet types of facilities (e.g., buildigs, hagars, etc.) have associated priority weightigs from the prioritizatio part of the model, the ormalized mitigatio project utility is added to the facility priority value. The sum of the priority weightig ad the ormalized mitigatio project utility is the adjusted utility value for each project. Adjusted Utility Value, u Normalized mitigatio utility value (for the project) AHP priority value (for the asset class of the project). Table 5 gives the result of the above calculatios for the utility. Resource Allocatio Example Facility priority ad mitigatio utility are used as iputs to a commercial-off-the-shelf (COTS) algorithm that optimizes allocatio of available AT fuds across idetified mitigatio projects. Allocatios are made subject to the mitigatio project cost costrait with respect to the istallatio AT budget. The geeral form of the optimizatio ca be summarized as follows: u Adjusted utility value calculated above each project B Total available budget for allocatio x Project allocatio amout where is the umber of projects subject to: TABLE 5. Raw mitigatio project utility for the project a Raw mitigatio project utility Max a a c i x i B u i x i x i 50, 16 Mitigatio project utility. Mitigatio project Asset class Beefit-cost ratio Utility ATFP-Proj2 Maiteace ATFP-Proj3 Admi ATFP-Proj4 Utilities E-4 (2) (3) TABLE 6. Budget allocatio. Asset Mitigatio Project Asset priority Weighted Project Budget project utility class (AHP) utility cost allocated ATFP-Proj Maiteace ATFP-Proj Admi ATFP-Proj4 1.8E-4 Utilities , The applicatio of Equatio 3 yields the results for the allocatio of budget to the various mitigatio projects. The results for a test case with a budget of $65, are summarized below. Table 6 shows the allocatios made to the differet projects. ATFP-Proj2 ad ATFP-Proj3 have the maximum values for the weighted utility. Each of these projects is allocated the costs required to fud these projects. ATFP-Proj4 has a much higher cost, ad it caot be fuded with the available $65,000 budget. Thus, there is o partial fudig for a project. Cogitive Support System Desig Ratioale Cogitive support systems are systems desiged to aid users with kowledge work i complex domais. Cogitive support systems for decisio makig focus o problem structurig, data collectio ad orgaizatio, ad applicatio of algorithms for decisio aalysis, mathematical programmig ad optimizatio, stochastic modelig, simulatio, ad logic modelig (Sprague & Carlso, 1982). Cogitive support systems therefore may embed a great deal of kowledge about how a orgaizatio, ad groups ad idividuals withi the orgaizatio, operate withi a give domai (Mahiem, 1986; Turba & Watkis, 1986). Studies of huma, aturalistic decisio-makig have ucovered a rage of maladaptive behaviors that appear to be germae to humas (Klei, 1998; Tversky & Kahema, 1974). The theory of bouded ratioality, for example, ackowledges the huma cogitive dimesio of real-world decisio scearios ad describes a process by which people strive to make decisios that are merely satisficig, or good eough (Simo, 1957). Much cotemporary decisio research focuses o the situated ature of decisio makig. Other research poits to the highly adaptive ature of huma decisio-makig ad suggests that how decisios are made is depedet o both the cotext ad sceario, ad o the attributes of the idividual makig the decisio (Crozier & Rayard, 1997). A key objective the i the desig ad developmet of cogitive support systems is the extet to which such systems support careful cosideratio of all available iformatio ad use that iformatio appropriately i decisio-makig tasks. Whe desiged, implemeted, ad used appropriately, cogitive support systems ca sigificatly improve the quality of a orgaizatio s decisiomakig (Bhargava, Sridhar, & Herrick, 1999). The system developed to support the approach ad implemet the decisio model was developed based o three JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY February 1,

8 key desig criteria: usability, usefuless, ad deployability. All of these criteria ceter o the idea of a user-cetric desig process. Each elemet of the decisio model has bee implemeted as a discrete Web service, allowig elemets to be used aloe (e.g., for asset prioritizatio, or determiatio of mitigatio project utility) or i combiatio as required for more complex scearios ad decisio objectives. The system architecture is Web-based, distributed, ad lightweight allowig it to support reliable, pervasive access icludig access from mobile devices. Web services are discrete uits of software fuctioality that are deployed to a URI ad are discoverable either directly or through a Web-based broker such as UDDI ad describable usig stadard, XML-based otology laguages such as WSDL. The attractio of Web services is their ability to itegrate disparate systems, regardless of the platform o which they are deployed or the laguage i which they are writte. Web services combie best practices from compoet-based developmet with emergig support for autoomous ad semi-autoomous eterprise applicatio itegratio (EAI). Extesible Markup Laguage (XML) is utilized as the basis for a rage of applicatio-specific dialects icludig SOAP, WSDL, RDF, ad UDDI that provide a meas for stadardizig data formattig ad ecodig, therefore facilitatig autoomous data exchage. Figure 2 shows how Web service techology has bee used to implemet the AT cogitive support system. We have implemeted the system usig IBM s Web Services Developmet Toolkit (WSTK 3.2). The developmet ad deploymet platform is Apache Tomcat. We use the ope-source database maagemet system, mysql for persistet data. The liear programmig fuctioality required for budget allocatio is implemeted usig a COTS software compoet, the Lido Java API versio 2.0. Busiess logic for the core decisio processig was developed usig Java (JSDK1.4.1) ad the Web-based user iterface (presetatio logic) was developed usig Java Server Pages (JSP). FIG. 2. Cogitive support system Web services architecture. This loosely coupled, compoet-based architecture allows flexible use, combiatio, ad recombiatio of the decisio elemets. This simplifies extesios to ad maiteace of the compoets ad allows distributio of system processig parts across multiple servers to achieve load balacig or fail-over capabilities. The prioritizatio, project utility, ad budget allocatio services are relatively geeric, the Web-based compoet model supports buildig ew presetatio logic to re-badge back-ed fuctioality for differet applicatios, for example, a cogitive support system to help choose amog research ad developmet projects could be assembled easily usig these existig parts. The Web services model also eables importat future ehacemets to the systems, especially the use of itelliget software agets to act as assistats ad mediators betwee the huma user ad the differet services. Model ad System Evaluatio Formal evaluatios of the AT decisio model ad cogitive support system have bee performed usig the cogitive walkthrough method. The cogitive walkthrough (CW) method is a evaluatio techique desiged to ivestigate the usability ad comprehesibility of a system early i the developmet process (Polso, Lewis, Riema, & 1992; Wharto, Riema, Lewis, & Polso, 1994). A key advatage of the CW method is that it is derived from a cogitive theory of how users work through a computer-supported task (Kah & Prail, 1994). The method is based o a theory of exploratory learig (Polso et al., 1992), which posits that system users form goals, explore the actios available to them to make progress towards their goals, ad cotiually assess whether the actios they take lead towards achievig their idetified goals. Of particular iterest i the case reported here was idetificatio of poits at which users metal models ad their maer of performace of a give task differed from the model implemeted i the system. These poits form the basis for further aalysis to determie whether either system chages or user chages to existig task practices might be implemeted to more closely alig cogitive support capabilities with the task that is to be accomplished. The cogitive walkthrough method ivolves observig ad recordig the behavior of systems users as they work through a decisio sceario usig a cogitive support system. The objective of the method is to uderstad the cogitive fit, or comprehesibility of the system relative to the iformatio ad iformatio processig requiremets of the decisio maker. Of particular iterest are the breakdows that occur durig system use, i other words, where ad whe the system fails to adequately support the users model of the decisio domai. We are particularly cocered that the system s overall usefuless ot rely o a sigle, iflexible model of the AT resource allocatio task. Such prescriptive, rigid tasks models are both difficult to specify for a rage of users ad use scearios ad brittle over time as the cotext i which such systems are used evolves. Therefore, 306 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY February 1, 2005

9 the focus o the aalysis was how each major system fuctio (facility prioritizatio, mitigatio project utility, ad resource allocatio) supported each of the decisio maker s major subtasks. The CW method ivolves idetificatio of a set of tasks to serve as the basis for each participat walkthrough. I the ATFP CSS the baselie task is allocatio of a budget amog competig mitigatio project alteratives. To clarify, this task ca be further decomposed ito the followig set of subtasks: 1. Facility Prioritizatio a. Idetificatio of prioritizatio criteria b. Relative weightig of prioritizatio criteria c. Idetificatio of alterative facilities for mitigatio d. Assessmet of idetified alteratives with respect to each of the idetified prioritizatio criteria (AHP) e. Review ad adjustmet of calculated facility priorities 2. Mitigatio Project Utility a. Idetificatio of mitigatio projects b. Specificatio of mitigatio project costs c. Specificatio of mitigatio project beefits i. Persoel protected ii. Key equipmet protected iii. Facility replacemet cost d. Specificatio of mitigatio project beefit/cost accelerators (for example, facility vulerability, facility risk, facility s curret coditio) e. Review ad adjustmet of calculated mitigatio project utility values 3. Mitigatio Project Budget Allocatio a. Calculatio of mitigatio project budget allocatio b. Review of calculated mitigatio project resource allocatios Cogitive walkthroughs were coducted at six Marie Corps istallatios over 6 days, 3 days at istallatios i Califoria i the sprig of 2003 ad 3 days at istallatios i North Carolia ad Virgiia i the fall of the same year. Walkthroughs were performed with a total of 21 prospective system users icludig ati-terrorism officers, provost marshals ad other military police, public works officers (Navy egieers), base operatios officers, ad civilia facilities plaers. I additio, the model ad system have udergoe 13 iformal focus group desig reviews with seior Marie Corps ad other Departmet of Defese persoel. These desig reviews were used especially i the early phases of the project, primarily as a meas to clarify ad refie system requiremets. Oce a complete workig versio of the system was available, a more formal study of system usefuless ad usability commeced usig cogitive walkthroughs with potetial users i the field. After aswerig a set of basic questios related to their role ad experiece i the AT domai, walkthrough participats were asked to work through a AT resource allocatio sceario familiar to them usig the task outlie above as a guide. Walkthroughs lasted from 45 to 120 miutes ad were geerally uiterrupted, focused sessios. The sectio below outlies the most sigificat issues that emerged from the walkthroughs ad where possible outlies future work to address idetified issues. Model ad System Use Scearios Essetial to the desig of the decisio model ad cogitive support system is uderstadig the differet scearios of use where they are likely to be applied. May reviewers reacted to the stadard task sceario preseted i desig reviews by expressig that their ways of workig ad the cotext i which they perform AT plaig tasks ofte differ from that used as the baselie. For may the AT plaig process is a ew priority for their positio ad may are still i the process of workig through the best approach to addressed AT plaig ad allocatio scearios as they arise. I this sese developmet of the AT cogitive support system is ot a matter of developig a system to best fit a well-uderstood, preexistig task,but oe of co-evolutio of the users domai uderstadig ad the tools to support their work. This result suggests that the de-coupled architecture employed i cogitive support system developmet may provide sigificat beefit i allowig flexible recombiatio of the differet decisio compoets to map more closely to disparate scearios. Goig forward we pla to further our uderstadig of potetial use scearios through broader ad more i-depth desig ad system reviews with potetial users ad other stakeholders. Our ultimate goal is to allow system users to specify some key attributes of the plaig ad allocatio sceario at had, ad the use a software aget to actively costruct ad lead the user through a task model that itegrates uderlyig system compoets ito a sceario-specific applicatio. Explaiig ad Justifyig System Results Participats i both cogitive walkthroughs ad iformal desig review frequetly discussed the importace of how ad why the system arrived at a particular set of resource allocatio recommedatios. This is a difficult problem i a complex, hybrid decisio model sice the elemets of a give solutio iteract i a variety of ways. A importat goal i developmet of a cogitive support system is workig towards the best fit i cases where this fit is ecessarily less close, for example, whe the techology is ot able to support a particular iformatio eed or iferece process, it is importat that the system has the ability to provide ot oly what ad how iformatio about the workigs of the system but also why it works the way that it does. I decisio systems exposig the ier workig of a model ad system is commoly achieved through sesitivity aalysis. I a complex, hybrid decisio model such as that described here, sesitivity aalysis is complicated by the broad rage of data ad algorithms combied i the solutio. I respose to this result we are focusig primarily o the desig ad developmet of a miimalist traiig module (Carroll, 1990) ad a itegrated software assistat, a help facility, to guide users to a better uderstadig of how the system works. Also, JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY February 1,

10 ehaced sesitivity aalysis capabilities, i the form of pop-up dashboards that expose the uderlyig data ad itermediate results for a give fial result, are beig desiged ad developed. Problems Iheret Whe Placig a Value o Huma Life Valuatio of huma life ad assessig the trade-offs betwee cost of facilities ad equipmet, ad potetial casualties is a sigificat complicatig factor i the AT resource allocatio domai. Utility models that iclude aspects related to the value of huma life itroduce uique difficulties i a otherwise ratioal mathematical model. This problem is ot uique to the AT domai, there is little agreemet across studies as to the best way to value the lives ad ijuries to people (de Blaeij, Florax, Rietveld, & Verhoef, 2000; Takeuchi, 2000). Much of the work i this area is highly domai ad cotext specific, suggestig that orgaizatios adoptig AT resource allocatio approaches will eed to address this problem usig criteria derived from their local value systems. I the case of AT plaig ad resource allocatio at the USMC, these local value systems relate closely to commad priorities ad the ature of a give istallatio s missio. Availability of Model Data A sigificat barrier to usability of the AT/FP cogitive support system is the overhead required to maually populate required data such as asset classes ad assets, the utility factors associated with a asset (people, asset replacemet cost, ad the cost of equipmet housed at the asset), ad stadard AT/FP mitigatios. Several participats requested that the system provide a stadard set of AT/FP mitigatios with costs ad effect ratigs. This will allow the system to act as a kowledge base of AT/FP mitigatio iformatio ad ease the task of creatig mitigatio projects. The ease of use ad usefuless of the system would be ehaced cosiderably if these data were made available i the system through automatic data loads. Aalytical Hierarchy Process Scalability ad Usability The AHP criteria ad alterative weightig user iterface presets sigificat challeges for ew users. May reviewers foud the matrix display used to weight prioritizatio criteria ad alteratives cofusig. Though the matrix may be the most efficiet approach to represetig this data, we are cosiderig ways to provide more atural user iterface mechaism for iputtig prioritizatio weights. Participats highlighted the difficulties iheret i the AHP prioritizatio process whe cosiderig relatively large umbers of criteria ad alteratives. A well-kow problem with the AHP prioritizatio approach is that as the umber of criteria ad/or alteratives icreases, providig pair-wise compariso values to populate the model becomes laborious. I cases of large umbers of criteria ad/or alteratives other prioritizatio methods such as SMART become attractive, eve though decisio fidelity is reduced. We are explorig ways i which the system could support methods for simplifyig pair-wise comparisos as the umber required icreases. Model Termiology Much of the termiology used to label key model etities was obscure to may of the reviewers. For example, the term Problem Space, which describes a abstract etity that acts as a placeholder for a budget amout, was cofusig to may reviewers. Ufortuately, i may cases where terms were obscure or cofusig, o sigle, clear alterative was idetified to replace the term with somethig more familiar. This suggests that customizable labels for model ad system etities would improve the usability of the system. Coclusio The decisio model ad decisio support system preseted i this paper provides a rigorous yet practical approach to allocatig ati-terrorism resources i military,commercial,govermet,ad other public-sector domais. Though the desig of the model ad cogitive support system have emerged from the military,specifically Marie Corps requiremets,the relative flexibility of the approach is easily adapted to omilitary domais. A cetral differece betwee military ad omilitary AT resource allocatio is the issue of acceptable risk ad acceptable losses. Though oe might assume that i omilitary domais acceptable risks ad losses approach zero,i fact,society frequetly makes implicit ad explicit judgmets of acceptable risks ad losses,for example,i medical research allocatios,trasportatio safety programs,ad i seismic retrofits for earthquake damage mitigatio. The approach described here combies the rigor of formal decisio processes while at the same time explicitly ackowledgig the limits huma decisio makers. Decisio model tuig ad evaluatio of cogitive support system performace ad usability is a ogoig process. Additioal system cogitive walkthroughs are plaed with each icremetal versio of the system. Also plaed is developmet to ehace applicatio iteroperability so that stadig data such as facilities lists,idividual facility populatios,ad stadard mitigatios ca be loaded ad made available to systems users without additioal data etry. We expect that the Web services architecture employed i costructio,with its XML-ative data iterchage formats,will ease this importat ad ofte time cosumig task. Results of model ad system evaluatios to date suggest a umber of improvemets that we pla to make goig forward. These iclude developmet of sesitivity aalysis ad explaatio facilities to help expose how ad why the system makes the resource allocatio recommedatios that it does. We are curretly researchig approaches to provide users with itelliget assistace to aid with problem structurig, data gatherig, ad iterpretig of the solutios 308 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY February 1, 2005

11 proposed by the system. Though the loosely coupled decisio model ad cogitive support system architecture meas that useful results ca be obtaied eve whe all data eeded for a perfect solutio are either uavailable or too expesive to capture, eve i cases where this data is available the sheer volume of data potetially required by idividual decisio makers meas that iterfaces to existig Marie Corps legacy systems will be required to help automate data etry. Fially, we are plaig cotiue work with cogizat groups i the Marie Corps o idetifyig a base of scearios to represet the rage of model ad system use cases. Refereces Au, T., & Au, T. P. (1992). Egieerig ecoomics for capital ivestmet aalysis (2d ed.). Eglewood Cliffs, NJ: Pretice Hall. Barbarosoglu, G., & Pihas, D. (1995). Capital ratioig i the public sector usig aalytic hierarchy process. The Egieerig Ecoomist, 40(4), Bard, J., Balachadra, R., & Kaufma, P. E. (1988). A iteractive approach to R&D project selectio ad termiatio. IEEE Trasactios o Egieerig Maagemet, 35, Bhargava, H. K., Sridhar, S., & Herrick, C. (1999). Beyod spreadsheets: Tools for buildig decisio support systems. Computer, 32(3), Brigham, E. F. (1989). Fudametals of fiacial maagemet. Hisdale, IL: The Dryde Press. Carroll, J. M. (1990). The Nurberg fuel: Desigig miimalist istructio for practical computer skill. Cambridge, MA: MIT Press. Carroll, J. M. (2000). Makig use: Sceario-based desig of humacomputer iteractios. Cambridge, MA: MIT Press. Carroll, J. M., & Rosso, M. B. (1992). Gettig aroud the task-artifact cycle: How to make claims ad desig by sceario. ACM Trasactios o Iformatio Systems, 10(2), Crozier, R., & Rayard, R. (1997). Cogitive process models ad explaatios of decisio makig. I R. Rayard, W.R. Crozier, & O. Sveso (Eds.), Decisio makig: Cogitive models ad explaatios (pp. 5 20). New York: Routledge. de Blaeij, A., Florax, R.J.G.M., Rietveld, P., & Verhoef, E. (2000). The value of statistical life i road safety: A meta-aalysis. Tiberge Istitute Discussio Paper TI /3. Retrieved December 15, 2003, from Edwards, W. (1977). How to use multi-attribute utility measuremet for social decisio-makig. IEEE Trasactios o Systems, Ma, ad Cyberetics, 7(5), Ereback A.-L.,& Hook,K. (1994,April) Usig cogitive walkthrough for evaluatig a CSCW applicatio. Paper preseted at Coferece o Huma Factors i Computig Systems,Bosto,MA. Ghasemzadeh, F., & Archer, N. P. (2000). Project portfolio selectio through decisio support. Decisio Support Systems, 29, Greier, M. A., Fowler, J. W., Shuk, D. L., Carlyle, W. M., & McNutt, R. T. (2003). A hybrid approach usig the aalytic hierarchy process ad iteger programmig to scree weapo systems projects. IEEE Trasactios o Egieerig Maagemet, 50(2), Herikse, A. D., & Trayor, A. J. (1999). A practical R&D project-selectio scorig tool. IEEE Trasactios o Egieerig Maagemet, 46(2), Hwag, C. L., & Yoo, K. (1981). Multiple attribute decisio makig. Berli: Spriger-Verlag. Kah, M. J., & Prail, A. (1994). Formal usability ispectios. I J. Nielse & R. L. Mack (Eds.), Usability ispectio methods (pp ). New York: Wiley. Keeey, R. L. (1994, Summer). Creativity i decisio makig with value focused thikig. Sloa Maagemet Review, 35(4), Klei G. (1998). Sources of power: How people make decisios. Cambridge, MA: MIT Press. Kocaoglu, D. F., & Iyigu, M. G. (1994). Strategic R&D project selectio ad resource allocatio with a decisio support system applicatio (pp ). I Proceedigs of the 1994 IEEE Iteratioal Egieerig Maagemet Coferece. Piscataway, NJ: IEEE. Korhoe, P., & Waleius, S. (1990). Usig qualitative data i multiobjective liear programmig. Europea Joural of Operatios Research, 48, Kyparisis, G. J., Gupta, S. K., & Ip C. (1996). Project selectio with discouted returs ad multiple costraits. Europea Joural of Operatios Research, 94(1), Liberatore, M. J. (1987). A extesio of the aalytical hierarchy process for idustrial R&D project selectio ad resource allocatio. IEEE Trasactios o Egieerig Maagemet, 34(1), Mahiem, M. L. (1986). Creativity support systems for plaig, desig, ad decisio support. Microcomputers i Civil Egieerig, 1, Otley, D. (1999). Performace maagemet: A framework for maagemet cotrol systems research. Maagemet Accoutig Research, 10, Parker, C. (2001). A approach to requiremets aalysis for decisio support systems. Iteratioal Joural of Huma Computer Studies, 55, Polso, P. G., Lewis, C., Riema, J., & Wharto, C. (1992). Cogitive walkthroughs: A method for theory-based evaluatio of user iterfaces. Iteratioal Joural of Ma Machie Studies, 36, Proctor, M. D., & Caada, J.R. (1992). Past ad preset methods of maufacturig ivestmet evaluatio: A review of the empirical ad theoretical literature. The Egieerig Ecoomist, 38, Ramaatha, R., & Gaesh, L. S. (1995). Usig AHP for resource allocatio problems. Europea Joural of Operatioal Research, 80(2), Saaty, T. L. (1980). The aalytic hierarchy process. New York: Wiley. Simo, H. A. (1957). Models of ma. New York: Wiley. Simo, H. A.(1960). The ew sciece of maagemet decisios. New York: Harper & Row. Sprague, R. H., & Carlso, E. D. (1982). Buildig effective decisio support systems. Eglewood Cliffs, NJ: Pretice Hall. Sulliva, A. C., & Smith, K. V. (1990). Capital budgetig practices for factory automatio projects (Workig Paper No ). West Lafayette, IN: Ceter for Maagemet of Maufacturig Eterprises, Purdue Uiversity. Takeuchi, K. (2000). A meta-aalysis of the value of statistical life (Workig paper F ). Istitute of Social Scieces, Meiji Uiversity, Japa. Retrieved December 15, 2003, from Thursto, D. L. (1991). A formal method for subjective desig evaluatio with multiple attributes. Research i Egieerig Desig, 3, Turba, E., & Watkis, P.R. (1986). Itegratig expert systems ad decisio support systems. MIS Quarterly, 10(2), Tversky, A., & Kahema, D. (1974). Judgemet uder ucertaity: Heuristics ad biases. Sciece, 185, Varey, M. S., Sulliva, W. G., & Cochra, J. K. (1985) Justificatio of flexible maufacturig systems with the aalytic hierarchy process. Paper preseted at the 1985 Aual Iteratioal Idustrial Egieerig Coferece, Norcross, GA. Wabalickis, R.N. (1987). Justificatio of FMS with the AHP. Joural of Maufacturig Systems, 7, Wharto, C., Riema, J., Lewis, C., & Polso, P. (1994). The cogitive walkthrough method: A practitioer s guide. I J. Nielse & R.L. Mack (Eds.), Usability ispectio methods (pp ). New York: Wiley. JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY February 1,