DepartmentofSolids,Materials,andStructures FacultyofEngineering TelAvivUniversity TelAviv69978 YoramReich

YoramReichLayeredmodelsofresearchmethodologies June,1994 DepartmentofSolids,Materials,andStructures FacultyofEngineering TelAvivUniversity TelAviv69978 YoramReich ArticialIntelligenceinEngineeringDesign,Analysis,andManufacturing(AIEDAM) SpecialissueonResearchMethodology,editedbyReich,Y. yoram@eng.tau.ac.il 1994,8(4):263-274 Israel techniquestosolvingproblemsinengineeringdesign,analysis,andmanufacturingis poor.theremaybemanyreasonsforthisstatusincluding:unfortunateheritagefrom AI,pooreducationalsystem,andresearchers'sloppiness.Understandingthisstatusis ThestatusofresearchmethodologyemployedbystudiesontheapplicationofAI Abstract ofworldviewsofscience.theseworldviewsarecombinedwitharesearchheuristicsor aprerequisiteforimprovement.thestudyofresearchmethodologycanpromotesuch paperintroducesconceptsfromthephilosophyofscienceandbuildsonthemmodels understanding,butmostimportantly,itcanassistinimprovingthesituation.this researchmethodology.thislayerdmodelcanservetoorganizeandfacilitateabetter understandingoffuturestudiesofresearchmethodologies.thepaperdiscussesmany researchperspectivesandcriteriaforevaluatingresearchtocreatealayeredmodelof oftheissuesinvolvedinthestudyofaiandaiedamresearchmethodologyusingthis layeredmodel. AIEDAM,1994,8(4):263-274 1

YoramReich conjectures,policies,positions,sourcesofideas,traditions,andthelike whetheror nottheyarejustiable tomaximumcriticism,inordertocounteractandeliminateas muchintellectualerroraspossible?(bartley,1962,p.140) Howcanourintellectuallifeandinstitutionsbearrangedsoastoexposeourbeliefs. June,1994 1Introduction ofmethods. fundamentalinformationthatispartoftheliteralmeaningofthetermmethodology:thetheory theresultstoguidethedevelopmentofcadtools.whilesuchdescriptionimportant,itmisses performedinresearchprojects;forexample,conductingobservationalstudiesofdesignersandusing Thetermmethodologymeansdierentthingsfordierentresearchers.Someresearchersequate Inthispaper,wediscusstheliteralmeaningofthetermandsomeofitsinterpretationsthatare methodologywithmethod.indoingso,researchmethodologybecomessynonymoustotheactivities relatedtothestudyofarticialintelligencetechniquesinsolvingengineeringdesign,analysisand manufacturingproblems(aiedam).clearly,theissuesinvolvedinstudyingtheoriesofanykind arerelevant.therefore,centraltothisdiscussionarequestionssuchas(bunge,1983,p.1): thephilosophicalbranchthatdealswiththetheoryofknowledge.giventheapparentinsoluble Thesequestionsarestudiedinthephilosophyofscience,andmorespecicallyinepistemology (6)Isthereaprioriknowledge,andifsoofwhat?(7)Howareknowledgeandaction contributetohisknowledge?(4)whatistruth(5)howcanwerecognizetruth?[...] related?(8)howareknowledgeandlanguagerelated? (1)Whatcanweknow?(2)Howdoweknow?(3)What,ifanything,doesasubject study;ifweobservethenumberofpapersdiscussingthemyths,legends,andfallaciesassociated logic(cox,1992),ormachinelearning(buntine,1990))wemaystarttoappreciatewhatsucha studycanoer. justforphilosophersdoaddressthesequestions?bartley'squoteprovidessomemotivationforsuch bystudyingresearchmethodology?whydowethinkthatitisworthwhileforresearchersandnot withsomeaitopics(e.g.,ai(fox,1990),expertsystems(liebowitz,1987;mettrey,1992),fuzzy statusofthesequestions,whyareweattemptingtodealwiththem?whatdowehopetoachieve ofthemcannotbeknownapriori.rather,thepaperaimstoprovidethebackground,motivation, andaframeworkfororganizingandunderstandingstudiesofaiedamresearchmethodology. studyarediscussedatvariouslevels.thepaperdoesnotattempttodelineateallissuessincemany focusesontheresearchmethodologyofaiandaiedam(section4).theissuesinvolvedintheir AIEDAMresearchmethodologyiscriticaltoresearchquality(Section3).Subsequently,thepaper Thispaperstartsbyintroducingbasicconceptsfromthephilosophyofsciencethatarerelatedto researchmethodology(section2).thepaperexplainsthroughvariousargumentswhythestudyof AIEDAM,1994,8(4):263-274 2

Methodology,immediatelyconnoteswithphilosophy.Wedonotdealwiththephilosophyofscienceinthispaper,butwewillborrowfromitssourcesasnecessarytoexplainissuescentralto YoramReich 2Philosophicalpreliminaries June,1994 researchmethodology.inparticular,wediscussthreeissues:theevolutionarynatureofresearch 2.1Theevolutionoftheconceptofresearchmethodology methodology,amodelofaworldview,andtwoexampleworldviewsthatweemploy. itsmeaningevolvescontinually.wecandemonstratethisbytracingitsmeaningsinceleonardo Wehavebeenreferringtoresearchmethodologyasthetheoryofmethods.Asanyotherconcept, questforknowledgeofallkinds,butforpracticalpurposes.whileheusedexperiments,theywere davinci'stime.leonardo(1452-1519)wasanartistandanengineerwhocombinedthetwoinhis Leonardo'spropositionswereadoptedpartiallybyDescartes(1561-1626)andBacon(1596-1650). ofmechanisticinquiry,andtheprimacyofmathematicalexplanations. WhileDescarteselevatedmathematicalreasoningasthesolesourceofknowledge,Baconadvocated propositionsthatcapturethreeideascentraltoscience:theessenceofempiricism,theimportance hisideas(gille,1966).interestingly,atthebeginningofhisnotebooks,leonardodiscussedthree notsystematic;hismeasurementswereimprecise;andhelackedadequatelanguageforexpressing ofthetwothatisverydierentfromwhatwewouldexpectofresearcherstodaybutnonetheless, onethatassociatedhimwiththeinventionofthescienticmethod(pitt,1992).similarly,newton foradeductiveinductionoftheoriesfromacomprehensivesetofobservationsofsomeempirical phenomena.nobodyseemedtohavefollowedeitherinpractice.forexample,theircontemporary variousstagesinwhichthestatusoftheorieschangedfromentitiesthatcanbededuced,conrmed, refutedtobeingargumentsjustiedbysomestatisticalinference(giere,1984). (1643-1727)employedmathematicalreasoningbutaddedunprovedassumptionswhentheyhelped toexplainnaturalphenomena.overtheyearsthetermresearchmethodologyevolvedthrough Galileo(1546-1642)hadmainlyemployedgeometricformalizationandthoughtexperiments,amix man-madeartifactrangingfromtheoriestomundaneobjectssuchasutensils(petroski,1992). suchasafterthedevelopmentofquantummechanics.suchevolutionnecessarilyhappensinany whenevertheworkinginterpretationofthetimewouldnolongerbemeaningfulorappropriate Bunge(1983)calledit,prescriptiveepistemology hasevolvedthroughtheyears.itevolved Iftheseprescriptionsevolve,thanatbesttheyareworkinghypothesesofwhatshouldbedone Clearly,researchmethodology theprescriptionsortheacceptablewaysofdoingscienceoras AIEDAM,1994,8(4):263-274 descriptiveenterprisewithpracticalconsequencestoresearch. toachievesomeresearchgoals.thus,wecanstudytheevolutionofhypothesesthatresearchers employandtheirutilityinachievingresearchgoals.thismayturnresearchmethodologyintoa3

Researchmethodologycanbedescribedasacollectionofmethodsfordoingresearchandtheir YoramReich 2.2Amodelofaworldview June,1994 by,theworldviewitserves.inthispaperwemodeltheconceptworldviewasapositionaboutthree Ontologydealswiththenatureofthethingsweknowabouttheworldorthenatureoftheworld. issues(guba,1990b):ontology,epistemology,andmethodology. interpretations.ifweaskwhatdodierentworldviewsofscienceadoptastheirresearchmethodologyandwhy,wewillndthatresearchmethodologyisintimatelyconnectedwith,andconstrained alsocreatecompletearticialsettingswhichwestudy;forexample,acadsystemwithitsuser. theseinstruments(i.e.,ishermeneutic).further,wedonotjustdevelopscienticinstrumentsbut canbedetectedormeasuredonlybythem.consequently,ourrelationwiththeworldisthrough cannotbesurethataphenomenonwewishtostudyinrelationtothesettingreallyexists. Althoughwecanberathercondentthatthearticialsettingexistssincewehavecreatedit,we areectionofourmanipulationoftheworld?"toillustratetheconict,considerthatmuchof Acentralontologicalquestionis\doweknowthingsaboutthe`real'worldorisourknowledge scienceisbasedontheuseofscienticinstrumentsthatallowusaccesstothosephenomenathat tionedsomeofthecentralepistemologicalquestionsintheintroduction.answerstothesequestions mayadvocatethatfactshaveaprimestatuswhileothersmayclaimthattheoriesarethereection Epistemologydealswiththerelationbetweenhumansandtheirknowledge.Wehavealreadymen- ofnature.athirdpositionmaystatethattheoreticalconceptsaremeaningfulonlyiftheyinvolve someactivitysuchasmeasuring,andafourthpositionmayclaimthatwearriveatknowledgeby participatinginsocialprocesses.theseepistemologicalpositionstogetherwiththoseaboutontology,implyadoptingdierentresearchmethodsandtheirinterpretationsasillustratedinsection withquestionssuchas:howisresearchplannedandexecuted,howaretheoriescreatedandtested, andhowarethetestsinterpreted. 2.3. Methodologydealswiththemethodsforcreatingknowledgeabouttheworldandtheinterpretation ofthisknowledgeinlightoftheontologicalandepistemologicalpositions.methodologyisconcerned Thereareatleasttwoconceptsinsciencethatresemblethismodelofaworldview:Kuhn'sparadigm (Kuhn,1962)andLakatos'researchprogramme(Lakatos,1968).Aparadigmismorefuzzythan themodelofworldviewdescribedherein;itisacorpusofconceptsandtheoriessharedbyagroup ofscientists.inperiodsof\normalscience"scientistsemployvariousimprecisecriteriatoguide theirresearch.whencertainanomaliesarediscovered,paradigmsmayclash,leadingtorevolutionarychangesthatcannotbeexplainedonthegroundofempiricalevidenceorrationalization.in theseauxiliaryhypothesesusingsomestandards.amajordierencebetweenthesemodelsisthe conceptualizationofachangeinscience:accordingtokuhn,changecomesthrough\irrational" contrast,aresearchprogrammehasahardcorewhichiskeptxedandauxiliaryhypothesesthat protectthehardcorefromanycontradictingevidence.theworkofscientistsisthedevelopmentof revolutions,whereasaccordingtolakatos,throughtheuseofacceptableevaluationcriteriaand AIEDAM,1994,8(4):263-274 4

Somequestionsareinterestingorevenmeaningfulonlywithinaparticularworldviewandsome Aworldviewhasaprofoundinteractionwiththeresearchquestionswechoosetoinvestigate. YoramReich judgment.1 June,1994 machinethink?"isimportanttomainstreamaibutisinconsequentialtotheviewofaiasan aforementionedincluding:training(indoingresearchandpractice),ethics,andvalues.wewill researchquestionslimitourhorizontoseekalternativeworldviews.forexample,thequestion\can notdealwiththeseissuesherebutsee(guba,1990b)forsuchdiscussion. 2.3Scientismandpracticism engineeringdiscipline.aconceptofworldviewisalsolinkedtoahostofissuesotherthanthethree Theremaybeseveralworldviewsofscience.Thereare,however,twoworldviewsthatoutlinethe rangeofpossibleworldviews:scientismandpracticism.2 Therstandmostprominentinscience,engineering,aswellasinotheracademicdisciplines(even thoughwehavewitnesseditsdemiseinphilosophy),isscientism.itrepresentstheessenceof worldviewslikerationalism,positivism,post-positivism,andlogicalempiricism.evenifthereare Thepositionofscientismaboutthethreeissuesisasfollows(Guba,1990a): majordierencesamongthesepositions,atameta-theoryleveltheyarethesame(weimer,1979). Ontology: Methodology:Experimental/manipulative hypothesesarestatedinadvanceandaresubjectedtotestundercarefullycontrolledconditions.theresearcheradoptsa worldthroughtheemploymentofappropriatemethodology. eectfree-contextlaws.bydiscoveringtheselawsscienceachievesitsgoal distantposition,thusachievingvalue-freeknowledge. topredictandcontrolphenomena(whethernaturalorotherwise). Epistemology:Objectivist researcherscanacquireobjectiveknowledgeaboutthereal Realist realityexists\outthere."realityoperatesaccordingtocause-and- Practicismistherivalworldviewthatcapturestheessenceofperspectivessuchasactionresearch, ofthestepsbecomescandidateforamethodologicalstudy;althoughthestructureisassumedto Figure1outlinesacommonmodeloftheresearchmethodologyofscientism.Theexecutionofeach begiven.figure1:researchmethodologyofscientism(adaptedfrom(schumm,1991)) manyvariationsonscientismandpracticism(includingvariationsontheirnames),somethatdependontheparticular AIEDAM,1994,8(4):263-274 anddainty,1991)forthreeexamples.thepresentinterpretationisdierentthanthoseinsomeaspects. disciplineandsomeonthepersonalinterpretationofwriters.see(guba,1990b;reasonandrowan,1981;smith 2Rowan(1981)mentionsalistof19perspectiveswhichcanbegroupedintofewerworldviews.Also,thereare 1TherearealsootherinterpretationsofKuhnandLakatos'views(Weimer,1979). 5

criticaltheoryandconstructivism).thepositionofpracticismaboutthethreeissuesisasfollows (Guba,1990a): YoramReich participatoryactionresearch,human-centeredengineering,orcriticalconstructivism(ahybridof June,1994 Ontology: Relativist realityexistsinthemindofpeopleandwithinacertainvalueladentheoreticalframework.byinteractingwiththeworld,peoplecan reconstructtheirperceptionofitintheirmind.whentheinteractionin- benothingbutasubjectivistinteractionwiththeworld.toavoidmisuses ofsubjectivismacriticalmethodologymustbeadopted. volvestechnologicalororganizationalchanges,thegoaloftheinquirymay beachievingimprovedpractice. Methodology:Criticalhermeneutic/dialectical Realityisconstructedthroughtheidenti- Epistemology:Criticalsubjectivism sincetheoriesaboutrealityarevalue-laden,therecan Practicismhaveanexplicitstandonthevalue,ethics,andhumannatureissues,especiallywhen naturewouldbedeterministicunderscientismandvoluntaryunderpracticism.3 theyareinterpretedinthecontextofsocialsciences.forexample,inmanagementsciencehuman cationofmultiple(includingcontradicting)constructionsandtheircritical comparison,thusimprovingthegroundsformakinginformedchoicesbetweenconstructions. ordeletionofinuencingfactorsorrelations. torevisionsormethodologicalstudieswhichcanrevisethecompletemodelincludingtheaddition alized,andinteractiveviewofresearch.eachoftheseinteractionsorinuencesmaybeasubject Figure2outlinesamodeloftheresearchaccordingtopracticism.Itpresentsaholistic,contextu- Scientismandpracticismarediametricallyopposingworldviewmodels.Yeteachofthemisa another.suchcontradictionwouldoccurinamodelwitharealistontology,objectivistepistemology,anddialecticmethodology.theconsistencyisalsomanifestedintable1whichsummarizes thetwoworldviews. Figure2:Acontextualizedmodelofpracticism(adaptedfrom(Smith,1991)) self-consistentmodelinthattheirontology,epistemology,andmethodologydonotcontradictone Theevolvingandsocialnatureofmethodologydoesnotemptytheusefulnessofsomeprinciples 3WhystudyAIEDAMresearchmethodology? isdeterministic. AIEDAM,1994,8(4):263-274 forevaluatingscientictheories(kuhn,1987)nordoesitmeanthatmethodologyismerelyanart thatisnotamenabletosystematicstudy.thenextsectionsdiscussthisinmoredetail. 3AIfoundations,beingrootedpartlyinmanagementsciencethroughSimon'sviews,assumethathumannature 6

1981)) YoramReich Table1:Asummaryoftwoworldviews(adaptedfrom(SmithandDainty,1991;ReasonandRowan, Dimension ScientismWorldview PracticismJune,1994 Researcher'srelationship tosetting Validationbasis Researcher'srole Sourceofcategories Aimofinquiry Apriori UniversalityandgeneralizabilitySituationalrelevance Onlooker Measurement,logic,reliability,externalvalidity Detachment,neutrality Interactiveemergent Actor Experiential Immersion meaning acquired Typeofknowledge Natureofdataand Statusofscienceasaeld ofknowledge Factual,contextfree cumulative,reductionistic Universal,theoria,precise,causal, Interpreted,contextually Particular,praxis,imprecise, Valuecontent multiplecausation,problematic,holistic Aimofscience Predictionandcontrol Privileged,progressive,autonomousNotseparatedfromother Valuefree Promotionofhuman Valueladen eldsofknowledge development embedded (e.g.,whatisdesign?)whileothersstressthepracticalrelevanceofresearch(e.g.,howcanweaid design?).thesedieringobjectivesareoriginatingfromthetwoperspectivesofresearchdiscussed 3.1Generalmotivation worldviewsentail,itisclearthateachoftheseobjectiveshasitsownsuitableeectivemethod First,weobservethattherearedieringviewpointsabouttheroleofAIEDAMresearch:some researchersthinkthataiedamresearchisaboutgaininganunderstandingofsomephenomena insection2.3:scientismandpracticism.recallingthedierentmethodologiesthateachofthese ofinquiry.itthusbecomesusefultostudywhichtechniqueismostsuitabletoachieveaspecic researchobjectiveandinwhatcircumstancesisiteective. Thisdebatecannotberesolved.Intheseconddebate,someresearchersarguethatresearchcannot bydefendingaitechnology,arguingthataihasimpactedpractice(e.g.,thecommentsbyhayes, leadtomimickinghumanintelligence(wilkes,1992))whileothersquestionordismisssuchclaims Thedierencesbetweenunderstandingandpracticalrelevancebecomeapparentintwopossible debates.inonedebate,someresearchersarguethatresearchhasledtotheunderstandingof ResearchCouncil,1991)thusquestioningtheunderstandingachievedbyresearch(Reich,1992). somephenomenawhileotherscommentthatresearchtoolsarenotusedinpractice(national thesecondclaimdoesnotaddressthecriticismandtherstclaimdoesnotarguewiththesecond. Forexample,independentofwhetherWilkesanalysisiscorrectornot,technologicalorpractical leadtotheunderstandingofsomephenomena(e.g.,wilkes'claimthatpresentairesearchcannot Novak,andLehnerttoWilkespositioninCACM,35(12):13-14).Suchdebateismeaninglessbecause AIEDAM,1994,8(4):263-274 7

YoramReich impactcanprovidenoevidencethataicanmimichumanintelligence(westandtravis,1991),nor doeswilkesquestionsuchimpact.clearly,givendierentworldviews,thereisnoconsensusonthe ofthedierentworldviews.bothproblemscouldbenetfromastudyofresearchmethodology. statusofresearchandthecriteriaforitsevaluation,andthereisaconfusionbetweenthepositions June,1994 researchmethodologyofmanydisciplinesrelevanttothestudyofaiedamincluding:social Second,welearnthatthereisanincreasingnumberofcritiqueson,anddieringviewpointsabout, sciences(guba,1990b;palumboandcalista,1990;reason,1988;whyte,1991),management science(argyris,1980;smithanddainty,1991),informationsystems(bjerknesetal,1987;floyd critiquesarerootedinobservationsthatscientism themodelofsciencethatdrivesmuchof etal,1992),andvariousbranchesofengineering(addis,1990;reich,1992;vincenti,1990).these contemporaryresearchmethodology isawed,andthatmodelingscienceasacontextualized activities;andparticipatinginsocialactivitiessuchasattendingacademicmeetings.asapractical enterpriseismoreakintothepracticeofscience.thesecritiquesdisplaythetensionbetween activity,researchcanbenetfromthekindofreection-in-actionsofundamentaltothesuccessful selectionofresearchquestions;thesolicitationoffunding;theplanningandexecutingofresearch scientismandpracticismandtellusthatmodelsofaiedamresearchmustbereecteduponsince practiceofdierentpractitioners(schon,1983).clearlyresearchers,asotherpractitioners,reect Third,noonecandenythatresearchisaverycomplexanddemandingdesignactivityincluding:the someofthesemodelsmightbedecient. whilepursuingtheiractivities,forexample,inthefaceofanimpasseintheirresearch.the studyofaiedamresearchmethodologycanprovidemuchinformationforresolvingsuchimpasses. Itisinstructivetoremindthatreection-in-actionwasconstruedasanexplanationofpractical thesourceofpracticalcompetence.unfortunately,reectionisforeigntoscientism(habermas, competencethataroseoutofacritiqueoftechnicalrationalitywhich,accordingtoscientism,is 1971)becausescientismforcesitsfollowersto\objectivate"themselvesoutfromstudies.Again, scientismandpracticism. SomeresearchersindisciplinesoverlappingAIEDAMhavealreadyreectedpubliclyupontheir theseperspectivesonthesourceofpracticalcompetencearemanifestationsofthetensionbetween developersandusers(e.g.,participatorydesign(pd))maybeaprerequisiteforprojectssuccess oncomputer-supportedcooperativework(cscw)havelearnedthatacollaborativeresearchof etal,1992;mcdermott,1994);andresearchersworkingoninformationsystemsand,inparticular, (Floydetal,1992;Mulleretal,1992).Theseexamplesatleastsuggestthatsimilarapproaches commentedthatmuchusercontrolisrequiredforthesuccessfuldevelopmentofsystems(marques (Carroll,1991);researchersworkingonthedevelopmentofknowledge-basedexpertsystemshave reectedonthelackofinuenceofcognitivepsychologyresearchonthedevelopmentofhcisystems researchactivities.forexample,researchersworkingonhumancomputerinteraction(hci)have Inspiteoftheneedtostudyresearchmethodology,itislackingfromAI(CohenandHowe,1988; mightbenecessary,oratleastuseful,tomeetaiedamresearchgoals. tiquesof,airesearchmethodology,evenbyprominentresearchers(bundy,1990;hallandkibler, Ullman,1991;Reich,1992),orAIEDAM.Thereare,ofcourse,numerousstudieson,andcri- McDermott,1981;WestandTravis,1991)engineeringdesign(NationalResearchCouncil,1991; AIEDAM,1994,8(4):263-274 8

YoramReich 1985;McDermott,1981;PartridgeandWilks,1990).Nevertheless,thenumberofthesestudies orcritiquesisnegligible.inmanycases,thesecritiquesarebasedonthestudyofairesearch etal,1994;reich,1993a).notethatwhilesuchreectionsareconsistentwithsomeworldviews, notablypracticism,theyareoutsidetherealmofscientismbecausecase-studymethodsareneithercontrollednorobjectiveand,certainly,controlledexperimentscannotbeperformedatthe overscientismisitsinferiorstatusamongresearchers:themajorityofresearchersinalldisciplines subscribe,intheory,tovariantsofscientismwhichtheyassociatewithamaturediscipline(bailey, levelofresearchprojectsifonlyforresourceconstraints.scientismcannotacceptasituationthat projects(ritchieandhanna,1984)oraiedamresearchprojects(cohenandhowe,1989;fenves June,1994 contradictsitsmethodofinquirythus,attemptstoignorethestudyofresearchmethodology. Sofar,wehavemotivatedthestudyofresearchmethodologybygivingabiasedexpositionoftwo perspectivesofscience.personalpreferenceaside,thereasonforgivingpracticismsomepreference disciplines(kerrandpipes,1987;schon,1983;reich,1992). 1992;Dixon,1987;NazarethandKennedy,1993),althoughinpractice,theymayemploydierent perspectives.scientismisalsopredominantintheeducationofengineersandresearchersinmost 3.2Alayeredmodelofresearchmethodology Theconceptofresearchmethodologytranscendsthelevelofworldviewtoincludeatleasttwo additionallevels.thesecondlevelincludesheuristicsfordoingresearch:thesearemethodsfor modelingandsolvingproblemsinaparticularmanner.theyareparticularlyusefulforguiding researchandsomeheuristicscanserveseveralworldviews.therefore,researchersmaychooseone ormanyheuristicsdependingonthecontextoftheirresearchasdisplayedinfigure2.theyshould thecreationoftheories.ineachresearchworldviewtherecanco-existdierentheuristicsfordoing becareful,however,tomakesurethattheirheuristicsarecompatiblewiththeirworldview.useful heuristicsinclude: Cognitivescienceperspectiveisinformedbyinsightfrompsychology.Theworkon Decisionscienceperspectiveattemptstoaugmentthedecienciesofhumandecisionmaker,suchaspsychologicalbiases.AsystemconsistingofahumanandanAItoolis case-basedreasoningoriginatedfromthisperspective. Thethirdleveldealswithspecicissuessuchasthemethodsforevaluatinghypothesesandthe Softwareengineeringperspectiveattemptstocreatesoftwarefordoingspeciedtasks. Systemscienceperspectiveattemptstoviewaprojectwithinalargersystemthatis Exploratoryprogrammingisaprimeexampleofthisheuristic. expectedtofunction.thisviewcanleadtothedevelopmentofembeddedsystems. expectedtoperformbetterthanhumanexperts(levi,1989). criteriaforsuchevaluations(adelman,1991;adelmanetal,1994;cohenandhowe,1989). AIEDAM,1994,8(4):263-274 9

stractionandrefertothemodelsofresearchmethodologiesaslayeredmodels.table2depictsthe ticshavebeenpracticedoraremeaningful.similarly,notalltripletsofworldview/heuristics/specic YoramReich Giventheselayers,wecannotlimitthestudyofresearchmethodologytoaparticularlevelofab- issuesaremeaningful.someoftheissuesrelatedtotherstandsecondlayersareoutlinedin(hall threelayerseachwithseveralexamples.notallthecombinationsofworldviewsandresearchheuris- June,1994 etal,1994;cohenandhowe,1988) andkibler,1985),andissuesrelatedtothethirdlayerarediscussedin(adelman,1991;adelman Researchheuristics(sources oftheoriesorhypotheses) Layer Worldviews Table2:Alayeredmodelofresearchmethodology Examples Humancentered Decisionscience Formalmethods Cognitivescience Practicism Softwareengineering Scientism Specicissues(evaluationor goodnesscriteria) Practicalrelevance Parsimony Formalrepresentation Systemsscience hypothesesortheoriesandexperimentalevidenceorfacts.amongsttheperspectivesistheone experimentaltesting.wehavealreadymentionedseveralperspectivesontherelativestatusof Inordertodemonstratetheimportanceofstudyingresearchmethodologyletusexamineanissue 3.3Hypothesesandtheirexperimentaltesting entangled. Tostartwith,somehypothesesareirrefutableoruntestable.Forexample,Duhem(1982)discussed thatviewstheoriesasprimaryandanotherthatviewsfactsasprimary.but,therelationismore thattranscendsthethreelayersofresearchmethodology:therelationbetweenhypothesesand alwaysbefreetoassumeorbelieveitistrue.induhem'swords,suchhypothesis\cannotberefuted observeonlyrelativemotions,wewillneverbeabletotestpoincare'sprinciple.therefore,wewill Poincare'shypothesisthat\thecenterofgravityofanisolatedsystemcanhaveonlyauniform byexperimentbecausetheoperationwhichwouldclaimtocomparethemwiththefactswouldhave rectilinearmotion."unfortunately,theonlytrueisolatedsystemistheuniverse,butsincewecan ofinteractionwithevidenceorfacts. AIEDAM,1994,8(4):263-274 nomeaning."(emphasisintheoriginal,p.166)thisplacespoincare'sprincipleoutsidethescope 10

YoramReich Whilesomehypothesescannotbetested,otherscould.Nevertheless,itisuncertainthatsuchtesting wouldbebenecialtoscienticprogress.forexample,inthecaseofthetheoryofthermodynamics, fact,signicanttheorizingbygalileo(feyerabend,1975)andothersprogressedinthefaceof someexperimentsweredistracting(truesdell,1982).therearethereforeconditionsunderwhich experimentalevidencemightbeignoredandcontrastinghypothesesretained(agassi,1975).in June,1994 (Hooke,1983;Shvyrkov,1987).Sometimesuselessresultsareportrayedassuccessful,whilein rivalhypotheses.butusingstatisticalinferenceisnotwithoutitspresuppositionsormisuses employingstatisticalinferenceondatagatheredfromcontrolledexperimentsforselectingbetween contradictingevidence. analyzedtorevealsomeusefulinsight(brinbergetal,1992). othersituationsresultsseeminglyuseless(e.g.,fromconfoundedstatisticalexperiments)canbe Inspiteofthiscomplexrelationbetweentheoriesandfacts,theacceptablepracticeistheone (BlumbergandPringle,1983).Similarly,theideasrelevanttoworldviewsotherthanscientismmay notworkforprojectsthatareintunetoscientism(sloane,1991). example,controlledexperimentsmaynotworkifoneadheretoparticipatoryactionresearch(par) worldviewsmustbeawarethatcompletelydierentissuesmaybepresentintheirresearch;for fromthemainstreamworldview scientism.weshouldnotethatresearcherssubscribingtoother Sofarthisdiscussionwasaimedatpromotingthestudyofresearchmethodologyusingideas orfalsifytheories.indeedwecannotdoitinprinciplebecauseanyempiricalevidencecouldbe andusingitinevaluatinghypothesesisnottrivial.thus,weneednotbehardpressedtojustify incorporatedtosaveanyhypothesis.thisdoesnotmeanthatweneedtoadoptaconventionalist reectedandimprovedupon,bymembersthatpledgeallegiancewithanyworldview. empiricalevidencedemonstratesthatexperimentaltestingmustbeusedcarefully,studied,and Fromthisdiscussion,itisclearthattheprincipleoftestingbyexperiments,interpretingthedata, approachsayingthattheoriesaresimplyselectedbyconvention;nordoesitmeanthatexperimental 4ThestudyofAIandAIEDAMresearchmethodology testingcanbeeradicated,thusinvitingsloppiness.thecomplexrelationbetweenhypothesesand theoreticstageitpermitsmuchexploratoryresearchwithinformaltesting.aiallowsdoingscruy itisyoungandithas\hutzpa."aiisayoungdisciplineandasonethatissupposedlyinapre- FromallresearchdisciplinesAImustbemostreectiveuponitsmethodologyduetotworeasons: researchinascruymannerandevenadvocatesthatsuchstylemaybeasourceofpower(lenat whatotherdisciplinesdreamofmaking(sharkeyandbrown,1986) theprimeonebeingthatof andbrown,1984;mcdermott,1981).aiismostvulnerablebecauseitmakesclaimsfarbeyond TheneedforreectionisnottakenseriouslyinAI.First,AIhasevolvedreallyonlyoneserious mimickinghumanintelligence. AIEDAM,1994,8(4):263-274 11

YoramReich paradigm(newell,1983).althoughotherworldviewsexist,mostnotablytheoneofappliedai,or AIasanengineeringenterprise(Bundy,1990),theyarenotseparatedfromthedominantworldview bydetailingthedierences simplyrecallwilkes'positionabouthard-coreaiandhayes',novak's, andlehnert'sresponsesdiscussedinsection3.ithadbeensaidthataimethodologyisamessthat needstobesortedout(bundyandohlsson,1990);thatairesearchersoftenmakethemistakes June,1994 ComputersandCognition(WinogradandFlores,1986)thatchallengedtheunderlyingassumptions oftheirprecedingpeers(mcdermott,1981);thatairesearchersarehyper-sensitivetocriticism, especiallythatdealingwiththefoundationofai(westandtravis,1991);orthatairesearchersdo toaresearchprogramme:ithasanunchallengedhardcorethatstatesthat\aphysicalsymbol ThisstatusandattitudemakesAIlookssimilarinstructure(althoughnotinitswayofevolution) ofai(westandtravis,1991).4 exampleisthereactionoftheaicommunitytowinogradafterthepublicationofunderstanding notwanttodealwithphilosophicalissuesoriginatingfromai(partridgeandwilks,1990).aprime 1976,p.116)andvariousauxiliaryhypothesesthataredevelopedtoprotectit.Independentof systemhasthenecessaryandsucientmeansforgeneralintelligentaction"(newellandsimon, whetherthehardcorecanorcannotbetestedseriously,forexample,throughsometuringtest (andwhethersuchtestistrulyatestofthehypothesis(halpern,1987)),itisacceptedwithout search,aswellastheneedformethodologicalstudies.but,thestudyofaiedamresearchmethod- ologyismorecriticalthanthatofaibecauseinengineeringthephenomenaunderinvestigationor standingthisissueisthestatisticaltechniquebootstrapthateliminatesmanyassumptionsrequired forclassicalstatisticaltechniquesandthatwasmadepossiblebytheavailabilityoffastcomputing cognitivescienceresearch Pylyshyn(1991)acknowledgesthatthisapproachhasprovedfruitless. ofitsingredients,involvingthethreelayersofthemodelofresearchmethodology.first,researchers thishardcoreentailthatarenotarticulated.6researcherscanevolveanalternativeresearch canquestionthehardcorebytryingtounderstanditssourceorwhataretheconsequencesthat HavingestablishedtheneedtostudyAIresearchmethodologyingeneral,wecanelaborateonsome situation. AIEDAMresearchborrowsmuchofitslegitimacy,approaches,andtheoriesfromgeneralAIre- fromthemisusesofterminology(mcdermott,1981)thatperpetuatevariouslegends,myths,and fallaciesrelatedtoai.5webelievethatthestudyofairesearchmethodologywillimprovethis muchselfcriticismandmaintainedwithoutsupportingevidence(dietrich,1990).italsosuers power(efronandtibshirani,1991).anegativeexampleisthedesiretobuildhcitoolsbydoing theinquirycontextisalwayschangingduetoengineeringadvances.apositiveexampleofunder- thebookproclaims.asmallsampleofsourcesonthisissueincludes(bernstein,1992;farias,1989;wolin,1990). oftheirrelationtohimbeinganaziandignoringtheramicationthatthismighthaveonthenewunderstanding searchcombinatoricsbyapplyingsituationalknowledge;"\aienhancessearchthroughtheuseofopportunism;" aboutaihard-core.forexample,fox(1990)consideredthefollowingfacts:\searchisacoreaiconcept;"\aireduces \knowledgerepresentationisacoreaiconcept;"and\aiknowledgerepresentationextendsquantitativemodelsby paradigm.suchparadigmisalreadyemergingintheformofaiasapracticalengineeringdiscipline. abstractionanddierentiation."thewaythefactswerestated,theyonlydealtwithaitechnology,notunderlying assumptions. 4Notthatthisbookisfreeofcriticism.MymajorcriticismistheutilizationofHeidegger'sideaswithoutthinking 5SomeofthepapersdiscussingthosemythswerewrittenbyprominentAIresearchersbuttheydidnotponder AIEDAM,1994,8(4):263-274 6Someanswerstothesequestionscanbefoundin(Waring,1991;Weizenbaum,1976;Winograd,1990). 12

YoramReich Second,researcherscanstudythepracticeofAIresearchersandtheoutcomesoftheirresearch Third,researcherscanstudyspecicdetailedissuesinpursuingdierentresearchmethodssuchas projectstobetterassociateresearchperspectivesorheuristicswiththeirsuccessesorfailures. variousmethodsofevaluation.thefollowingsubsectionsbrieydiscusstheseissues. June,1994 4.1Questioningthehardcore:Programsashypothesesandtheirexperimental atasolutiontoaproblemwhereexpertscontendthatthereisnotenoughevidencetoreacha testingoftheoriesthroughtheirbehavior aformofturingtest.supposethatasystemarrives nottwellthemodeloftheoriesinscience.theyarehardtoevaluate,forexample,considerthe ItiscommontoperceiveAIprogramsasresearchhypothesesortheories.However,AIprogramsdo conclusion.\isthesystembrilliant?orprecocious(pople,1985)?"didthetheoryreceivepositive itisalmostimpossibletoverifysuchstatement.consequently,theperceivedexplanatorypowerof uncoverthisway,onehastolookinsidetheprogram,notmerelyatitstrace,andaiprogramsare toobigtobecomprehensible.thus,evenifaclaimismadethataprogramimplementsatheory, support?theanswersdependonthewaythesystemarrivedattheconclusion.butinorderto programsisnotrealized(sharkeyandbrown,1986). IfAIprogramsaretoocomplextobeparsimoniousandtoohardtounderstand;iftheirbehavior ishardtojustify;andiftheyrarelygeneralizetobeapplicabletomanysituations;cantheir behavioratleastbereplicated?theaisolutiontothischallengeisthereconstructionofai programs.unfortunately,therearehardlyanyexamplesofrationalreconstructionoflargeai programs(campbell,1990).toillustrate,eventhesuccessfulreconstructionofprotos(bareiss, 1989)intoCL-Protos(Dvorak,1988)doesnotcountasareconstructionbecauseitwascarried outwithinthesameresearchgroupthatdevelopedit;therefore,besidethewrittenmaterialthat task.atruereconstructionissupposedtorelyonlyonwrittendocuments;otherwise,howcan documentsthetheorythatprotosembeds,theoriginaldevelopersassistedinthereconstruction thattheirbehaviorcouldbeexperimentedwithbyotherresearchers(reich,1991).inthiscase, anybodyclaimthatitssourceofpowerifthewrittentheory. Toexacerbatethesediculties,thedescriptionofAIprogramsisdecient:languageismisused paradigmforaithatmayrelievesomeoftheseproblemsevenifnoteliminatethem:aiasan CL-Protosisonegoodexample. TheprecedingdiscussionsuggeststhatAIprogramsarenotreallytheories.Thereisadierent (McDermott,1981),andomissionsanderrorsaremadeinreporting(Grabiner,1986;Ritchieand engineeringdisciplinewhoseaimistobuildpracticaltools.forsuchaworldview,theoriesneed Hanna,1984).Ifprogramsarehardlyreconstructed,atleasttheyneedtobemadeavailableso notbeeasilytransferable,theycanbecontextdependent.however,theneedfortheirproper evaluationisstillcentral. AIEDAM,1994,8(4):263-274 13

scientismandpracticismandhowtheserolesguideairesearch.scientismcontendsthatcomputers YoramReich 4.2Analternativeparadigm:AIasanengineeringdiscipline Tobetterunderstandthealternativeparadigmconsidertherolethatisassignedtocomputersin June,1994 canmimichumanthinking.thetheoreticalargumentsforthiscapabilitytendtobephilosophical Eurisco(Dietrich,1990;RitchieandHanna,1984),andBACON(Chalmersetal,1992;Dietrich, claims.forexample,seethecritiqueof:thelogictheoristandgps(dietrich,1990),amand whiletheempiricalworkcanonlydealwithsimpliedbehaviororevenfailtodemonstrateits 1990;Grabiner,1986). Incontrast,theengineeringapproachhadsuccessessuchas:MACSYMA,DENDRAL,R1/XCON, objectiveisonlyonepartofpracticism.toillustrate,considerthedevelopmentofvariousgeneral Adoptingapracticalobjectivedoesnotguaranteeresearchsuccess.Primarily,sinceapractical tothisviewpoint;theempiricalworkdealswithbuildingsystemsanddemonstratingtheirpractical andothers(dietrich,1990;dymandlevitt,1994;tomiyama,1994).theengineeringviewpoint AItools(orshells),suchasKEE,ART,orNexpert.AccordingtoFikes,thesetoolshavehardly eectiveness.itbecomescriticaltodeterminewhatconstitutessuchacceptabledemonstration. simplyarguesthatcomputerscanbecomeeectiveengineeringtools.therearenograndtheories practicismarecontextuallyembeddedandparticular,ratherthancontextfreeorgeneral.therefore, theactualpracticeofbuildingapplicationsisnotsurprising. Practicalsuccessimpliesasignicantunderstandingofthetargetapplicationdomain.Agood them(hayes-rothandfikes,1991).lookingattable1wecanseethattheresearchproductsof accordingtohayes-roth,isthebuildingofmanyadhocsystemsthathavelittlesharingbetween beenusedforbuildingrealapplications(hayes-rothandfikes,1991).thecommonpractice, cisinglessinvolvementinpracticemayleadtopracticallyirrelevantresearch(fenvesetal,1994; Reich,1994).Practicismdemandsresearchersimmersioninpractice.Thisimmersionmaybeapproximatedortakedierentformssuchas:revisingtheeducationalsystemsothatresearchers domain(i.e.,targetusersofthetools)tocontroltheproduct(marquesetal,1992;mcdermott, ich,1994)orcollaborative(steinberg,1994)projects;orlettingthoseexpertsintheapplication 1994).Anotherviewarguesthatthebuildingofpracticaltoolsrequiresprofessionalsystembuilders havesignicantlymorepracticaltraining(dymandlevitt,1994);engaginginparticipatory(re- understandingcanleadtomajorsuccess(e.g.,thenite-elementprogramstress),whileexer- (Hayes-RothandFikes,1991).Itisunclearwhatistherolethatthisviewassignstootherpotential contributorstothedevelopmentoftools. 4.3Researchperspectives/heuristics:Sourcesofhypotheses AIEDAM,1994,8(4):263-274 InSection3.2wediscussedthelayeredmodelofresearchmethodology.Acombinationofaworldviewandaresearchheuristicorperspectiveprovidesmuchoftheguidanceforresearch.Halland 14

tiveai,andempiricalai.theseperspectivesaresomecombinationsofworldviewsandresearch YoramReich Kibler(1985)discussedveAIperspectives:performanceAI,constructiveAI,formalAI,specula- heuristicsandarenotjustworldviews.7 June,1994 inmultiplecombinations): ExamplesofthesecombinationsthatfollowTable2are(notethatsomeexamplescouldbeplaced Scientism Cognitivescience:hardcoreAI,e.g.,Soar(Lairdetal,1987).Thiscombination Scientism Formalmethods:logicaspresentedin(GeneserethandNilsson,1987).Notethat issimilartohallandkibler'sempiricalai. formalai. wellapracticist formalmethodperspective.thiscombinationissimilartohallandkibler's byremovingtheintroductorychapterofthisbook,thetechniquesdescribedinitcanserve Scientism Systemsscience:vision(Marr,1982).Thiscombinationhavesomeoverlapwith Scientism Humancentered:thiscombinationisincompatible. Scientism Softwareengineering:e.g.,CYC(Lenatetal,1986).Thiscombinationcontains HallandKibler'sformalAI. Practicism Cognitivescience:case-basedreasoning(Pearceetal,1992). Practicism Formalmethods:proofplanning(Lowe,1994)orqualitativephysics(Tomiyama, 1994). HallandKibler'sspeculativeAI,andpartofconstructiveAI. Practicism Softwareengineering:buildingperformancetools,e.g.,MACSYMA(Moses, Practicism Systemscience:integrationofAIwithothersoftwaretoolsandhardware(Pardee etal,1990),orrobotics(brooks,1991). 1971),situatedsoftwaretools(Marquesetal,1992;McDermott,1994),orcomputational acontinuousimprovementofresearchpractice. isboundtodriveresearcherstoconductmethodologicalreectionbecauseitsattainmentrequires depictedareofthoseprojectswhosestatedobjectiveisdemonstratingpracticalrelevance.thisgoal Fewofthecombinationsinvolvingpracticismhaveactuallybeentestedinpractice,yettheexamples designaids(steinberg,1994).thiscombinationincludeshallandkibler'sperformanceai. withscientism. AIEDAM,1994,8(4):263-274 thathechangedhisresearchheuristicfromcognitivesciencetosoftwareengineering,whilemaintainedhisadherence 7Toillustrate,whentheymentionthatFeigenbaumshiftedfromempiricaltoconstructiveAI,theyreallymeant 15

YoramReich 4.4Specicissues:Goodnessorevaluationcriteria ofaiprojectsseemstraightforward(bundy,1990): TherecanbemanywaystoassessthequalityofAIprojects.Somecriteriaforassessingthequality June,1994 Parsimony:butwehavealreadyclaimedthatmostAIprogramsdonotsatisfyit; Power widerangeofapplicability(e.g.,countertoaphdprogram):butthiscontrastswith Clarityisaprerequisite:butAIprogramsarenotclear,sometimesnottotheirdevelopers; Correctness(behaveasintended):butcanthisbeassessed?,justconsiderthevastamount Completeness(benishedandwork):butthiscanhardlybedoneunlessthetargetpractical Hayes-Roth(Hayes-RothandFikes,1991)andwithourpreviousanalysis; domainisfullyunderstood; Interestingly,eventhoughthislistseemsimpossibletosatisfybyanyresearchproject,itmaystill beconsistentwithpracticismandleadtopracticalsuccessduetothelastitem.fromapracisist Commercialsuccess:wecanhardlyquarrelwiththis. ofliteratureonthevalidationofexpertsystemswhichcannotguaranteecorrectness;and Therearemanyspecicissuesanddetailsthatplaydierentrolesintheevaluationofresearch henandhowe,1988;reich,1993b).independentofallthedetails,therulethatmustbefollowed projects,;detailsonsuchissuescanbefoundelsewhere(adelman,1991;adelmanetal,1994;co- improvingpractice,thenastudythatillustratessuchimprovementshouldbefurnished.ifthepurposeistounderstandhowdoesthehypothesesfunctiongivendierentparameters,thencontrolled experimentsmustbeperformed.ifthepurposeorresearchisthecreationofanimplementable cannotbesuchclaimwithoutsuchdemonstration). theory,oneshouldcreatecomputermodelsofthehypothesisandshowthatthecomputermodel Theanalysesoftestsalsodependontheobjectiveofresearch:statisticalinferenceissuitablefor controlledexperimentsandmorecomplexanalysismaybeneededforassessingpracticalimprove- indeedimplementsthetheory.thereisnoneedtotestthepracticalityofthetheory(butthere isthatthewayahypothesisistesteddependsontheanswersoughtfromthetest.ifthepurposeis perspectivethislistmaybeseenasanidealthatcanhardlybeattained.8 AIEDAM,1994,8(4):263-274 Lowe,1994). ments(e.g.,qualityofpractice,timetoperformpracticalactivities,revenueofpractitioners). 8Bundyandco-workersattempttoadheretotheseprinciplesbythedevelopmentofproofplanning(Bundy,1988; 16

YoramReich 5Summary Thispaperhascoveredmanygrounds,andinapartialmanneronly.Wehavediscussedthe June,1994 conductingmethodologicalstudiesofaiedamresearch.wehaveillustratedthecomplexityof philosophicalbackgroundofsciencicresearchmethodologyandpresentedseveralmotivationsfor theissuesinvolvedinsuchstudiesbydiscussingairesearchmethodologyanditsproblemsandby describingaframeworkthatcanhelporganizeandunderstandsuchstudies.wehopethatthis isnotasummarybutthepreviewofacontinuousreectionthatwillaiedamimproveresearch practice. References Addis,W.(1990).StructuralEngineering:TheNatureofTheoryandDesign,EllisHorwood,New Adelman,L.,Gualtieri,J.,andRiedel,S.L.(1994).\Amulti-facetedapproachtoevaluating YorkNY. Agassi,J.(1975).ScieneinFlux,D.ReidelPublishingCompany,Dordrecht. Adelman,L.(1991).\Experiments,quasi-experiments,andcasestudies:Areviewofempirical expertsystems."articialintelligenceinengineeringdesign,analysis,andmanufacturing, Argyris,C.(1980).InnerContradictionsofRigorousResearch,AcademicPress,NewYork,NY. andcybernetics,21(2):293{301. 8(4). Bailey,M.T.(1992).\Dophysicistsuscasestudies?Thoughtsonpublicadministration."Public AdministrationReview,52(1):47{54. methodsforevaluatingdecisionsupportsystems."ieeetransactionsonsystems,man, Bjerknes,G.,Ehn,P.,andKyng,M.,editors(1987).ComputersandDemocracy:AScandinavian Bareiss,R.(1989).Exemplar-BasedKnowledgeAcquisition,AcademicPress,Boston,MA. Bartley,W.W.I.(1962).TheRetreattoCommitment,A.A.Knopf,NewYork,NY. Blumberg,M.andPringle,C.D.(1983).\Howcontrolgroupscancauselossofcontrolinaction Bernstein,R.J.(1992).TheNewConstellation:TheEthical-PoliticalHorizonsofModernity/Postmodernity,MITPress,Cambridge,MA. research:thecaseofrushtoncoalmine."journalofappliedbehavioralscience,19(4):409{ Challenge,GowerPress,Brookeld,VT. 425. AIEDAM,1994,8(4):263-274 Brooks,R.A.(1991).\Newapproachestorobotics."Science,253(5025):1227{1232. Brinberg,D.,Lynch,J.,andSawyer,A.G.(1992).\Hypothesizedandconfoundedexplanations intheory."journalofconsumerresearch,19(2):139{154. 17

YoramReich Bundy,A.(1988).\Theuseofexplicitplanstoguideinductiveproofs."InProceedingsofthe9th Bundy,A.andOhlsson,S.(1990).\ThenatureofAIprinciples:AdebateintheAISBQuarterly."InPartridge,D.andWilks,Y.,editors,TheFoundationsofArticialIntelligence:A Sourcebook,pages135{154,Cambridge,UK,CambridgeUniversityPress. June,1994 Bundy,A.(1990).\WhatkindofledisAI?."InPartridge,D.andWilks,Y.,editors,TheFoundationsofArticialIntelligence:ASourcebook,pages215{222,Cambridge,UK,Cambridge ConferenceonAutomatedDeduction,pages111{120. Buntine,W.(1990).\Mythsandlegendsinlearningclassicationrules."InProceedingsofAAAI- Bunge,M.(1983).TreatiseonBasicPhilosophy,Volume5,EpistemologyandMethodologyI: UnderstandingtheWorld,D.ReidelPublishingCompany,Dordrecht. UniversityPress. Carroll,J.M.,editor(1991).DesigningInteraction:PsychologyattheHuman-ComputerInterface, Campbell,J.A.(1990).\ThreenoveltiesofAI:Theories,programsandrationalreconstruction."InPartridge,D.andWilks,Y.,editors,TheFoundationsofArticialIntelligence:A Sourcebook,pages237{246,Cambridge,UK,CambridgeUniversityPress. 90(Boston,MA),pages736{742,MenloPark,CA,AAAIPress. Chalmers,D.J.,French,R.M.,andHofstadter,D.R.(1992).\High-levelperception,representation,andanalogy:Acritiqueofarticialintelligencemethodology."JournalofExperimental 9(4):35{43. CambridgeUniversityPress,Cambridge,UK. Cohen,P.R.andHowe,A.E.(1988).\HowevaluationguidesAIresearch."AIMagazine, andtheoreticalarticialintelligence,4(3):185{211. Cox,E.(1992).\Thegreatmythsoffuzzylogic."AIExpert,7(1):40{45. Dietrich,E.(1990).\Programsinthesearchforintelligentmachines:themistakenfoundationof Cohen,P.R.andHowe,A.E.(1989).\TowardAIresearchmethodology:Threecasestudiesin evaluation."ieeetransactionsonsystems,man,andcybernetics,smc-19(3):634{646. Duhem,P.(1982).TheAimandStructureofPhysicalTheory,PrincetonUniversityPress,Princeton,NJ. Dixon,J.R.(1987).\Onresearchmethodologytowardsascientictheoryofengineeringdesign." AI."InPartridge,D.andWilks,Y.,editors,TheFoundationsofArticialIntelligence:A Sourcebook,pages223{233,Cambridge,UK,CambridgeUniversityPress. ArticialIntelligenceforEngineeringDesign,AnalysisandManufacturing,1(3):145{157. Dym,C.L.andLevitt,R.E.(1994).\OntheevolutionofCAEresearch."ArticialIntelligence Dvorak,D.L.(1988).\GuidetoCL-Protos:Anexemplar-basedlearningapprentice."Technical ReportAI88-87,ArticialIntelligenceLaboratory,TheUniversityofTexasatAustin,Austin, TX. forengineeringdesign,analysis,andmanufacturing,8(4):(inpress). AIEDAM,1994,8(4):263-274 18

YoramReich Farias(1989).HeideggerandNazism,TempleUiversityPress,Philadelphia,PA. Efron,B.andTibshirani,R.(1991).\Statisticaldataanalysisinthecomputerage."Science, Fenves,S.J.,Garrett,J.H.J.,andHakim,M.M.(1994).\Representationandprocessingof 253:390{395. June,1994 Floyd,C.,Zullinghoven,H.,Budde,R.,andKeil-Slawik,R.,editors(1992).SoftwareDevelopment Fox,M.S.(1990).\AIandexpertsystemsmyths,legends,andfacts."IEEEExpert,5(1):8{20. Feyerabend,P.K.(1975).AgainstMethod,NewLeftBooks,London,UK. andrealityconstruction,springer-verlag,berlin. designstandards:abifurcationbetweenresearchandpractice."inproceedingsofthe1994 StructuresCongress(Atlanta,GA),NewYork,NY,AmericanSocietyofCivilEngineers. Genesereth,M.R.andNilsson,N.J.(1987).LogicalFoundationsofArticialIntelligence,Morgan Giere,R.N.(1984).UnderstandingScienticReasoning,Holt,RinehartandWinston,NewYork, Gille,B.(1966).EngineersoftheRenaissance,MITPress,Cambridge,MA. Grabiner,J.V.(1986).\Computersandthenatureofman:Ahistorian'sperspectiveoncontroversiesaboutarticialintelligence."BulletinofTheAmericanMathematicalSociety, Kaufmann,LosAltos,CA. 2ndedition. Habermas,J.(1971).KnowledgeandHumanInterests.TranslatedbyJ.J.Shapiro,BeaconPress, Guba,E.G.,editor(1990).TheParadigmDialog,SagePublications,NewburyPark,CA. Guba,E.C.(1990).\Thealternativeparadigmdialog."InGuba,E.G.,editor,TheParadigm 15(2):113{126. Hall,R.P.andKibler,D.F.(1985).\Dieringmethodologicalperspectivesinarticialintelligence."AIMagazine,6(3):166{178. Dialog,pages17{27,NewburyPark,CA,SagePublications. Boston,MA. Halpern,M.(1987).\Turing'stestandtheideologyofarticialintelligence."ArticialIntelligence Kerr,A.D.andPipes,R.B.(1987).\Whyweneedhands-onengineeringeducation."Technology Hayes-Roth,F.andFikes,R.(1991).\Interviewsbyb.chandrasekaran."IEEEExpert,6(5):3{14. Hooke,R.(1983).HowtoTelltheLiarsfromtheStatisticians,MarcelDekker,NewYork,NY. Review,(October):36{42. Review,1(2):79{93. AIEDAM,1994,8(4):263-274 Kuhn,T.S.(1987).\Objectivity,valuejudgment,andtheorychoice."InKourany,J.A.,editor, Kuhn,T.S.(1962).TheStructureofScienticRevolution,TheUniversityofChicagoPress, CA,Wadsworth. ScienticKnowledge:BasicIssuesinthePhilosophyofScience,pages197{207,Belmont, Chicago,IL.

YoramReich Lakatos,I.(1968).\Criticismandthemethodologyofscienticresearchprogrammes."Proceedings Laird,J.E.,Newell,A.,andRosenbloom,P.S.(1987).\Soar:anarchitectureforgeneralintelligence."ArticialIntelligence,33(1):1{64. June,1994 ofthearistoteliansociety,69:149{186. Lenat,D.B.andBrown,J.S.(1984).\Whyamandeuriscoappeartowork."ArticialIntelligence, Lenat,D.,Prakash,M.,andShepherd,M.(1986).\CYC:usingcommonsenseknowledgeto 23(3):269{294. Liebowitz,J.(1987).\Commonfallaciesaboutexpertsystems."ComputersandSociety,16(4):28{ Levi,K.(1989).\Expertsystemsshouldbemoreaccuratethanhumanexperts:Evaluation proceduresfromhumanjudgmentanddecisionmaking."ieeetransactionsonsystems, 33. Man,andCybernetics,SMC-19(3):647{657. overcomebrittlenessandknowledgeacquisitionbottlenecks."aimagazine,6(4):65{85. Lowe,H.(1994).\Proofplanning:AmethodologyfordevelopingAIsystemsincorporatingdesign Marques,D.,Dallemagne,G.,Klinker,G.,McDermott,J.,andTung,D.(1992).\Easyprogramming:Empoweringpeopletobuildtheirownapplications."IEEEExpert,7(3):16{29. press). issues."articialintelligenceforengineeringdesign,analysis,andmanufacturing,8(4):(in McDermott,D.(1981).\Articialintelligencemeetsnaturalstupidity."InHaugeland,J.,editor, Marr,D.(1982).Vision:AComputationalInvestigationintotheHumanReresentationand McDermott,J.(1994).\Situatingsoftwareartifacts.".PresentedattheAISeminar,Schoolof ProcessingofVisualInformation,W.H.Freeman,SanFrancisco,CA. Mettrey,W.(1992).\Expertsystemsandtools:mythsandrealities."IEEEExpert,7(1):4{12, MindDesign,pages143{160,Cambridge,MA,MITPress. ComputerScience,CarnegieMellonUniversity,Pittsburgh,PA,April,1994. Muller,M.J.,Kuhn,S.,andMeskill,J.A.,editors(1992).PDC'92:ProceedingsofTheParticipatoryDesignConference(Cambridge,MA),ComputerProfessionalsForSocialResponsibility, PaloAlto,CA. Advantage,NationalAcademyPress,Washington,DC. 14(8):548{560. Moses,J.(1971).\Symbolicintegration:Thestormydecade."CommunicationoftheACM, February. NationalResearchCouncil(1991).ImprovingEngineeringDesign:DesigningForCompetitive AIEDAM,1994,8(4):263-274 Nazareth,D.L.andKennedy,M.H.(1993).\Knowledge-basedsystemverication,validation,and 162. testing:theevolutionofadiscipline."internationaljournalofexpertsystems,6(2):143{ 20

YoramReich Newell,A.andSimon,H.A.(1976).\Computerscienceasempiricalinquiry:Symbolsandsearch." Newell,A.(1983).\Intellectualissuesinthehistoryofarticialintelligence."InMachlup,F.and CommunicationoftheACM,19:113{126. Manseld,U.,editors,TheStudyofInformation:InterdisciplinaryMessages,pages187{227, June,1994 Palumbo,D.J.andCalista,D.J.,editors(1990).ImplementationandThePolicyProcess:Opening Pardee,W.J.,Sha,M.A.,andHayes-Roth,B.(1990).\Intelligentcontrolofcomplexmaterials UpTheBlackBox,GreenwoodPress,NewYork,NY. processes."aiedam,4(1):55{6. JohnWiley&Sons,NewYork. Partridge,D.andWilks,Y.,editors(1990).TheFoundationsofArticialIntelligence:ASourcebook,CambridgeUniversityPress,Cambridge,UK. \Case-baseddesignsupport:Acasestudyinarchitecturaldesign."IEEEExpert,7(5):14{ Petroski,H.(1992).TheEvolutionofUsefulThings,Knopf,NewYork,NY. Pearce,M.,Goel,A.K.,Kolodner,J.L.,Zimring,C.,Sentosa,L.,andBillington,R.(1992). Pople,H.(1985).\Evolutionofanexpertsystem:FromInternisttoCaduceus."InDeLotto,I. Pitt,J.C.(1992).Galileo,HumanKnowledge,andtheBookofNature:MethodReplacesMetaphysics,KluwerAcademicPublishers,Dordrecht,Holland. 20. Pylyshyn,Z.W.(1991).\Someremarksonthetheory-practicegap."InCarroll,J.M.,editor,DesigningInteraction:PsychologyattheHuman-ComputerInterface,pages39{49,Cambridge, SagePublications,NewburyPark,CA. UK,CambridgeUniversityPress. andstefanelli,m.,editors,aiinmedicine,pages179{208,amsterdam,elsevier. Reason,P.,editor(1988).HumanInquiryinAction:DevelopmentsinNewParadigmResearch, Reason,P.andRowan,J.,editors(1981).HumanInquiry:ASourcebookofNewParadigm Research,JohnWiley&Sons,NewYork,NY. Reich,Y.(1993).\ThedevelopmentofBridger:Amethodologicalstudyofresearchontheuse Reich,Y.(1991).\Bookreview:Exemplar-BasedKnowledgeAcquisition,byRayBareiss.AcademicPress,1989."MachineLearning,6(1):99{103. EDRC12-51-92,EngineeringDesignResearchCenter,CarnegieMellonUniversity,Pittsburgh,PA. Reich,Y.(1992).\Transcendingthetheory-practiceproblemoftechnology."TechnicalReport AIEDAM,1994,8(4):263-274 Reich,Y.(1993).\Thestudyofdesignresearchmethodology."JournalofMechanicalDesign, ASME.(acceptedforpublication). issueonmachinelearningindesign. ofmachinelearningindesign."articialintelligenceinengineering,8(3):217{231.special 21

YoramReich Reich,Y.(1994).\WhatiswrongwithCAEandcanitbexed."InPreprintsofBridging Ritchie,G.D.andHanna,F.K.(1984).\AM:AcasestudyinAImethodology."Articial thegenerations:aninternationalworkshoponthefuturedirectionsofcomputer-aided Engineering,Pittsburgh,PA,DepartmentofCivilEngineering,CarnegieMellonUniversity. June,1994 Rowan,J.(1981).\Adialecticalparadigmforresearch."InReason,P.andRowan,J.,editors, Schon,D.A.(1983).TheReectivePractitioner:HowProfessionalsThinkinAction,Temple Intelligence,23(3):249{268. Schumm,S.A.(1991).ToInterprettheEarth:TenWaystobeWrong,CambridgeUniversity HumanInquiry:ASourcebookofNewParadigmResearch,pages93{112,NewYork,NY, JohnWiley&Sons. Smith,London,UK. Shvyrkov,V.V.(1987).\WhatHarvardstatisticiansdon'ttellus."Quality&Quantity,21(4):335{ Sharkey,N.E.andBrown,G.D.A.(1986).\Whyarticialintelligenceneedsanempirical Press,Cambridge,UK. Sloane,S.B.(1991).\TheuseofarticialintelligencebytheUnitedStatesNavy:Casestudyof UK,ChapmanandHall. foundation."inyazdani,m.,editor,ai:principlesandapplications,pages260{291,london, Smith,C.N.andDainty,P.,editors(1991).TheManagementResearchHandbook,Routledge, afailure."aimagazine,12(1):80{92. 347. Tomiyama,T.(1994).\FromGeneralDesignTheorytoknowledgeintensiveengineering."Articial Steinberg,L.(1994).\ResearchmethodologyforAIanddesign."ArticialIntelligenceforEngineeringDesign,Analysis,andManufacturing,8(4):(inpress). London,UK. Truesdell,C.(1982).\Thedisastrouseectsofexperimentupontheearlydevelopmentofthermodynamics."InAgassi,J.andCohen,R.S.,editors,ScienticPhilosophyToday:Essays IntelligenceforEngineeringDesign,Analysis,andManufacturing,8(4):(inpress). Vincenti,W.G.(1990).WhatEngineersKnowandHowTheyKnowIt:AnalyticalStudiesFrom Ullman,D.(1991).\CurrentstatusofdesignresearchintheUS."InProceedingsofICED-91 AeronauticalHistory,JohnsHopkinsUniversityPress,Baltimore,MD. (Zurich),Zurich,Heurista. inhonorofmariobunge,pages415{423,dordrecht,d.reidelpublishingcompany. AIEDAM,1994,8(4):263-274 Waring,S.P.(1991).TaylorismTransformed:ScienticManagementTheorySince1945,The Weimer,W.B.(1979).NotesontheMethodologyofScienticResearch,LawrenceErlbaum, Hillsdale,NJ. UniversityofNorthCarolinaPress,ChapelHill,NC. 22

YoramReich West,D.M.andTravis,L.E.(1991).\Thecomputationalmetaphorandarticialintelligence:A Weizenbaum,J.(1976).ComputerPowerandHumanReason:FromJudgementtoCalculation, reectiveexaminationofatheoreticalfalsework."aimagazine,12(1):64{79. Freeman,SanFrancisco,CA. June,1994 Winograd,T.andFlores,F.(1986).UnderstandingComputersandCognition:ANewFoundation Wilkes,M.V.(1992).\Articialintelligenceastheyear2000approaches."Communicationsof Whyte,W.F.,editor(1991).ParticipatoryActionResearch,SagePublications,NewburyPark, theacm,35(8):17{20. Wolin,R.(1990).ThePoliticsofBeing:ThePoliticalThoughtofMartinHeidegger,Columbia Winograd,T.(1990).\Thinkingmachines:Cantherebe?Arewe?."InPartridge,D.and Wilks,Y.,editors,TheFoundationsofArticialIntelligence:ASourcebook,pages167{189, UniversityPress,NewYork,NY. ForDesign,AlbexPublishing,Norwood,NJ. Cambridge,UK,CambridgeUniversityPress. AIEDAM,1994,8(4):263-274 23

YoramReich Acontextualizedmodelofpracticism(adaptedfrom(Smith,1991)) Researchmethodologyofscientism(adaptedfrom(Schumm,1991)) FigureCaptions June,1994 AIEDAM,1994,8(4):263-274 24

June,1994 YoramReich 1. Preparation Training, literature review 2a. Observations 2b. Data collection 3. Induction Statement of problem: Primary hypothesis Revision 4. Deduction Secondary hypothesis generation 5. Testing Data collection, analysis Rejection 6. Acceptance Explanation, prediction, control AIEDAM,1994,8(4):263-274 25

June,1994 YoramReich Previous research experiences Education Age Personality Sex Culture Family The researcher Researcher s norms Research institution Research questions Research perspective Research methods Research context: - resource - evaluative - procedural - operational Research output AIEDAM,1994,8(4):263-274 26

YoramReich Asummaryoftwoworldviews(adaptedfrom(SmithandDainty,1991;ReasonandRowan, 1981)) TableCaptions June,1994 Alayeredmodelofresearchmethodology AIEDAM,1994,8(4):263-274 27

YoramReich Researcher'srelationship tosetting Validationbasis Dimension Measurement,logic,reliability,externalvalidity cumulative,reductionistic Universal,theoria,precise,causal, Interactiveemergent Actor Experiential Detachment,neutrality ScientismWorldviewImmersion PracticismJune,1994 acquired Researcher'srole Sourceofcategories Aimofinquiry TypeofknowledgeApriori UniversalityandgeneralizabilitySituationalrelevance Onlooker meaning Natureofdataand Statusofscienceasaeld ofknowledge Factual,contextfree Interpreted,contextually Particular,praxis,imprecise, Valuecontent multiplecausation,problematic,holistic Aimofscience Predictionandcontrol Privileged,progressive,autonomousNotseparatedfromother Valuefree Promotionofhuman Valueladen eldsofknowledge development embedded AIEDAM,1994,8(4):263-274 28

YoramReich Researchheuristics(sources oftheoriesorhypotheses) Layer Worldviews Examples Humancentered Decisionscience Formalmethods Cognitivescience Practicism Softwareengineering Scientism Specicissues(evaluationor goodnesscriteria) Practicalrelevance Parsimony Formalrepresentation Systemsscience June,1994 AIEDAM,1994,8(4):263-274 29