A New Approach for Protocol Analyss on Desgn Actvtes Usng Axomatc Theory of Desgn Modelng Shengj Yao and Yong Zeng * Concorda Insttute for Informaton Systems ngneerng Concorda Unversty 455 de Masonneuve West, CB-40-6, Montreal, Quebec H3G M8, Canada * -mal: zeng@cse.concorda.ca Abstract Although varous desgn methodologes have been developed to help desgners generate desgn concepts and desgn deas, they must be appled by desgners. Desgners play an mportant and crtcal role n delverng a successful and nnovatve desgn. To understand how desgners thnk n solvng a problem durng the desgn process wll help us develop a new desgn methodology that can accommodate desgner s performance. Ths paper proposes a new protocol analyss approach to studyng desgner s cogntve behavour durng the desgn process by usng the axomatc theory of desgn modelng. The conventonal protocol analyss approach, whch ncludes transcrpts, segmentaton, encodng and experment results analyss, largely depends on the experence of the people who conduct the analyss. In ths research, the concept of desgn state, derved from the axomatc theory of desgn modelng, s used to gude the entre protocol analyss process. The consstency of analyzng subject s protocol s checked by three operators. Ths paper reports our prelmnary study of protocol analyss on desgn actvtes. Indepth report of ths research wll be gven n our future work.. Introducton Desgn s a subject that has been studed for many years. A lot of desgn theores and methodologes have been developed to assst desgners to conduct desgn actvtes [-8]. Whle those desgn theores and methodologes provde some gudelnes and procedures for desgners, they leave a lot of flexblty and space for desgners to perform. The desgn solutons are vared wth dfferent desgners. An nnovatve desgn soluton s often delvered by a creatve desgner. Dasgupta ndcated n hs paper [9], desgn agent s freedom and capacty to use hs knowledge contrbute sgnfcantly to the desgn creatvty. One promsng approach to coverng the dscrepances between tradtonal desgn methodologes and desgner s freedom s to study desgners cogntve processes when they are workng on a desgn problem. Our ultmate goal s to develop robust desgn tools to enhance the creatvty and nnovaton of human desgners n makng desgn decsons. To reach ths goal, we must understand desgner s thnkng and mental process and what factors affect a desgner s decson. In ths research, a protocol analyss method s developed to understand what factors are mportant n stmulatng desgners to generate creatve deas, by studyng desgner s thnkng process and problemsolvng behavor. Protocol analyss s a major technque n current research for attemptng to dentfy the role of desgner s thnkng and reasonng n the desgn process [0-6]. Ths method s to study subject s mental processes n the performance of tasks by recordng ther spontaneous thnkng aloud and subsequently segmentng the runnng commentary nto the dscrete atomc mental operatons that the subjects have used n the accomplshment of the tasks [7]. There are two major approaches for protocol analyss: retrospectve report and concurrent verbal report. The approach of concurrent verbal report s that desgner s nstructed to thnk-aloud and talk-aloud concurrently whle performng a desgn task. The approach of retrospectve report has two stages. At the frst stage, the desgners are left alone to conduct ther desgn tasks wthout any nterference. Then at the second 43
stage, the desgners are asked to recall ther desgn process and are remnded to answer some questons f they mss some nformaton. Concurrent reports have ther advantages n trackng desgner s thnkng behavor at any tme n the desgn process. However talkng aloud mght nfluence the subject s percepton and thnkng process. Lloyd, et al. [8] argued that although concurrent verbal reports can reveal some aspects of desgn thnkng, there are many types of desgn thnkng that reman mpervous to concurrent verbalzaton requrng dfferent methodologes for analyss. Suwa, et al [9] employed retrospectve report to reveal the perceptual nteractons between a desgner and hs own sketches because they thnk the talkng aloud concurrently may nterfere wth partcpant s percepton durng ther sketchng actvtes. In retrospectve protocols, desgners mght recall ther desgn process partally and selectvely, but ths effect can be reduced by showng the desgners the vdeo that recorded hs/her desgn process and by askng them some questons as memory cues. In our research, the retrospectve report s employed to record desgner s actvtes. The desgners are frst asked to thnk to solve a desgn problem and draw sketches to llustrate ther desgn process f necessary. Then n the ntervew sesson, the desgner wll be asked to recall the desgn process and answer some questons f they mss some nformaton. The rest of ths paper s organzed as follows. Secton ntroduces the axomatc theory of desgn modelng. Secton 3 llustrates the experment set-up. The experment analyss s explaned n Secton 4. Secton 5 presents a summary and the future work of ths research.. Axomatc Theory of Desgn Modelng Our protocol analyss method s developed based on axomatc theory of desgn modelng [8]; hence, t s essental to brefly ntroduce ths theory before we dscuss our protocol analyss. The axomatc theory of desgn modelng [8] provdes a formal approach that allows for the development of desgn theores followng logcal steps based on mathematcal concepts and axoms. The detals of axomatc theory of desgn modelng can be found n [8]. A key concept n the axomatc theory of desgn modelng s the structure operaton, whch can be defned as follows: Structure operaton, denoted by, s defned by the unon of an object O and the nteracton of the object wth tself. O = O (O O) () where O s the structure of object O. The structure operaton provdes the aggregaton mechansm for representng the object evoluton n the desgn process. Based on the structure operaton, the concept of product system can be defned. A product system s the structure of an object (Ω) ncludng both a product (S) and ts envronment (). Ω = S,, S[ S = Φ] () where Φ s the object that s ncluded n any object. The product system ( Ω) can then be expanded as follows: Ω= ( S) = ( ) ( S) ( S) (S ) (3) where and S are structures of the envronment and the product, respectvely; S and S are the nteractons between the envronment and the product. A product system can be llustrated n Fgure. Fgure. Product system The desgn process can then be represented by desgn governng equaton [0], Ω + = D ( Ω ). (4) The desgn problem evolves along the evoluton of the product. At each stage of ths evoluton process, the desgn problem s defned by ts current product system Ω, whch s called the state of the desgn. The components of product system Ω keep on changng whle conflcts exst n Ω. Ths evoluton process of desgn can be llustrated as n Fgure. state of desgn: Ω t t 0 Ω 0 Ω t Ω t n Ω n tme: t Fgure. voluton of the desgn process The evoluton process of desgn can be stated n the followng theorem [0]. Theorem of Dynamc Structure of Desgn Problem. In the desgn process, desgn solutons to a desgn problem may change the orgnal desgn problem, f the desgn solutons are dfferent from ther precedents, ether by refnement or by alteraton. 44
Ths nature of the desgn problem was orgnally proposed by Zeng and Cheng as the recursve logc of desgn []. Ths logc ndcates that desgn s a process recursvely generatng desgn solutons and the knowledge to evaluate the solutons. Ths result was confrmed by Roozenburg's research []. Later Maher, et al. [3] combned the gene concept wth the recursve desgn process and ponted out that the problem space and the soluton space co-evolve together, wth nterchange of nformaton between the two spaces. Now desgn has been wdely recognzed as an evoluton process wth the change of the problem state and the soluton state. Consderng the theorem of dynamc structure of desgn problem, we wll dentfy each desgn state n the desgn process to analyze the subjects experment data. Based on the axomatc theory of desgn modelng, we antcpate that n the desgn process, desgners can fgure out more envronment components, deeper relatonshps between envronment components and the product. Desgn problem: to desgn a ltter-dsposal system for the passenger compartment. Ths system should be convenent for the passengers to depost garbage and meanwhle t s easy for the cleaners to collect the garbage. The structure of the passenger compartment s as follows. Asle 3. xperment 3. Desgn Problem Desgn problems should be selected based on ther potental demands on dfferent aspects of desgn sklls. They should be practcal, solvable, easly understood, sutable for desgners wth dfferent levels. The desgn problem we have used n ths paper s adapted from the research [4]. Ths desgn problem s to desgn a new ltter-dsposal system n a passenger compartment located n the trans of NS (Dutch Ralways). The orgnal descrpton for ths desgn problem can be found n [4]. Snce the subjects n our experment are not famlar wth the tran structure of Dutch Ralways, we have modfed the orgnal desgn problem to make t more understandable to our subjects; the revsed descrpton of the desgn problem s shown as below. 3. Subjects Subjects play an actve role n our experments. Seven graduate students from mechancal engneerng, electrcal engneerng, and computer engneerng partcpated n our experment. We refer to the seven subjects as S to S7. We consdered all the subjects as experenced desgners snce they have had varous desgn experence from 5 to 0 years. All of the seven subjects have been traned n one or two desgn methodologes. In ths research, we wll use the desgn process of the subject S as an example to llustrate our protocol analyss method. You are asked to generate a desgn concept for the desgn problem above. You need to wrte down all your deas, concepts and draw the sketches to llustrate your solutons n your desgn process. 3.3 Procedure Our experment has two stages: desgn sesson and retrospecton sesson. Desgn sesson refers to desgner s problem-solvng process and retrospecton sesson refers to desgner s retrospectve process. In the frst stage, subjects stay n a quet room to solve the problem wthout any nterference. If they have any queston about the problem or need any further nformaton, they can ask the expermenter. In ths experment, we set up three webcams to record the whole desgn process. One webcam s set up for general recordng of the whole pcture of the subject and hs or her actons. The second webcam s set up for trackng the subject s subtle expresson by recordng hs/her facal expresson. The thrd webcam s set up to record the subject s gestures durng the 45
desgn process. The three webcams can record the audo and vdeo nformaton. The experment set-up s shown n Fgure 3. Fgure 3. xperment Set-up Subjects are requred to draw and wrte anythng they want to solve the desgn problem on a WACOM tablet screen usng the tablet pen wth the help of the software Procreate Panter Classc. Ths software s a knd of tool to support desgners to draw or wrte anythng on the screen. Before subjects begn to solve the problem, they are traned for 5-0 mnutes to use the tablet pen and the tablet screen to draw sketches n order to feel as f they were usng pen and paper. The actvtes done on the tablet screen by subjects are recorded by the software My Screen Recorder. Ths software can record anythng that subjects draw or wrte on the screen of the computer. Subjects actons and the screen actvtes are montored by the expermenter n the desgn process. When subjects fnsh solvng the problem, the expermenter wll understand desgner s general actons and ntentons. In the second stage, we need to determne the subject s motvaton towards the test stuaton and ther own desgn. The subject s asked to recall and report what he/she was thnkng at each step of hs/her desgn process by watchng the vdeos of hs/her actons and the vdeos of hs/her screen actvtes. If the subject msses some nformaton, he/she wll be remnded or asked to clarfy hs/her problem-solvng behavors. Meanwhle, the three webcams are used to record the whole retrospectve process from dfferent perspectves. Then the protocol data wll be organzed from the vdeo and audo meda and screen recordng vdeo for further analyss. 4. xperment Analyss After we conduct the experment, we need to collect and organze the data from the experment. Frst we need to transcrbe what subjects spoke n the retrospectve process from the vdeos. Then the transcrbed data wll be segmented and encoded for further analyss. The consstency and the accuracy of the protocol analyss need to be checked to ensure that the experment analyss s more objectve and more robust. The desgn solutons are evaluated based on the analyss usng the axomatc theory of desgn modelng. Then the results wll be further conducted for statstcal analyss. 4.. Transcrpton The frst step of the experment analyss s to transcrbe all the words spoken by subjects durng the retrospectve stage nto text documents. The text documents are taken as the ntal transcrpts. Then the ntal transcrpts are checked by a second operator to ensure the accuracy of the ntal transcrpt. The text documents orgnally transcrbed from subjects usually contan some vague and nconsstent nformaton. Moreover, subjects may not gve enough nformaton to explan ther thnkng process just by showng ther ntentons. We also need to add some annotatons to explan ther non-verbal ntentons to make the transcrpt more consstent. So a thrd operator s needed to flter the unnecessary nformaton and add some annotatons nto the transcrpts. If there are any dscrepances of the transcrpts between the three operators, they wll dscuss to get the fnal agreement. Then we wll get the formalzed transcrpts. A part of the formalzed transcrpt of the subject S s shown below. Frst, I got your desgn problem. The frst pont of the desgn problem s that I want to make clear what the thng to be desgned s. We made clear that we need to desgn a garbage bn. Then I thnk what the envronment for ths garbage bn s. In whch place should t be put? Lke ths coach car or sleepng car? Then I consder the structure of the coach car. Now I consder the convenence for the cleaner to pck up the garbage bn. Cleaners walk along the asle. Then I thnk to put the garbage bn under the table. It wll affect the movement of the passenger s legs f the garbage bn s put under the table. So the only place s under the seats. Put here (under the seats). These are seats, tables (pont to the sketches on the screen). Ths place (under the seats) s not convenent for pckng up. Here, I feel t s not good. So, I consder puttng along the sde of the seats close to the asle. 4.. Segmentaton and encodng After we get the formalzed transcrpts, we need to dvde the transcrpt of each subject nto segments and 46
then the segments are encoded usng our codng scheme. In ths research, the transcrpt s dvded nto separate desgn states. A desgn state represents a sngle ntenton of the desgner. From Secton, we know that a desgn state s defned by the structure of envronment components, the structure of product and the relatonshp between envronment components and product. So n segmentaton, a desgn state can be dentfed as any changes of envronment components, any changes of product descrptons, or any change of the relatonshp between envronment and product. An example of segmentaton s gven below by segmentng the above example of the formalzed transcrpt. Desgn State : Frst, I got your desgn problem. The frst pont of the desgn problem s that I want to make clear what the thng to be desgned s. We made clear that we need to desgn a garbage bn. Then I thnk what the envronment for ths garbage bn s. In whch place should t be put? Lke ths coach car or sleepng car? Then I consder the structure of the coach car. Desgn State : Now I consder the convenence for the cleaner to pck up the garbage bn. Cleaners walk along the asle. Then I thnk to put the garbage bn under the table. It wll affect the movement of the passenger s legs f the garbage bn s put under the table. Desgn State 3: So the only place s under the seats. Put here (under the seats). These are seats, tables (pont to the sketches on the screen). Ths place (under the seats) s not convenent for pckng up. Here, I feel t s not good. So, I consder puttng along the sde of the seats close to the asle. ncodng s the most mportant part to analyze subject s protocol. xstng protocol studes n devsng codng scheme frst dvde subjects protocols nto varous desgn actons and encode desgn actons nto some specfc categores and subcategores. However those codng schemes are developed accordng to the specfc desgn problem. If the desgn problem s changed, the codng scheme wll also be changed. Snce our encodng scheme s developed based on the axomatc theory of desgn modelng, whch mples the logc of desgn, t wll enable us to systematcally code desgner s actons. Desgn process proceeds wth the evoluton of desgn states. So we wll take each desgn state as the basc encodng unt. Then a desgn state wll be decomposed nto envronment components, product, and the relatonshps between them. A desgn state (DS) s represented as Ω. The structure of the product and the envronment are represented as S and, respectvely. An envronment may contan several envronment components = e. Some envronment components could be modfed n dfferent desgn states; therefore, the modfed envronment components wll be encoded as m(e ). The relatonshp between the envronment and the product s encoded as B = ( S ) (S ). If the relatonshps or the envronment components are modfed n dfferent states, they wll be encoded as m(b ) and m(e j ), respectvely. An example of the encodng s explaned below. They are summarzed n. Desgn State : = ( S ) = e =coach car Ω B =where the garbage bn should be put n the coach car S =garbage bn Desgn State : = ( S ) e 3 = e e e3 e =cleaners, Ω =table B =the convenence for the cleaner to pck up the garbage bn S =garbage bn s put under the table Desgn State 3: = ( S ) Ω 3 3 3 4 e 5 =seats 3 = e e e3 e4 e5 e =passenger, B 3 : affect the leg of the passenger when put under the table and not convenent for pckng up rght under the seats. S 3 =garbage bn s put along the sde of the seats close to the asle. 4.3. Consstency Snce the experment analyss ncludng transcrpton, segmentaton, and encodng was conducted by people and t mght contan subjectveness, we need to assess the correctness and the consstency of the protocol analyss. In general, the consstency can be assessed n two ways. One way s to deal wth the protocol data by the same person for several tmes and evaluate them repeatedly, lke the Delph method [5]. The other way s to deal wth the protocol data n parallel by several people and then coordnate ther results. In our study, the second j j 47
method wll be used to evaluate the experment analyss. There are three operators to conduct the experment analyss n our research. The frst operator conducts the experment analyss to get the ntal transcrpt, segmentaton and encodng. Then the second operator wll check the results of the frst operator and mark the results f he or she has a dfferent opnon. Fnally the thrd operator wll coordnate the results of the two operators and make the fnal agreement for the experment analyss. In ths way, the consstency of our protocol analyss approach s mantaned. No DS DS DS3. An example of encodng : envronment = e e =coach car e = e e e =cleaners e 3 =table = e e 3 3 4 3 e e e e =passenger 4 e 5 =seats 5 S : product garbage bn garbage bn s put under the table garbage bn s put along the sde of the seats close to the asle. B : relatonshp where the garbage bn should be put n the coach car? the convenence for the cleaner to pck up the garbage bn affect the leg of the passenger when put under the table and not convenent for pckng up rght under the seats. 4.4. Measurement of desgn soluton The desgn solutons generated by the subjects need to be evaluated for comparsons. Prevous research often used experts to set up some crtera to evaluate the desgn solutons. To avod the subjectveness, our study uses envronment-based desgn [6] to analyze the desgn problem. All possble envronment components and relatons between envronments and product wll be lsted. The potental conflcts n the desgn requrements wll be dentfed. Then the desgn solutons wll be measured based on the full analyss of the desgn problem usng envronmentbased desgn. The desgn solutons wll be ranked accordng to how they match the analyss of the envronment-based desgn. On the other hand, the desgn solutons wll be measured accordng to the novelty and the value, whch are the characterstcs of creatvty. Three assessors wll evaluate the desgn solutons separately. The average measurements from the three assessors wll be used as the fnal results. 4.5. Statstcal analyss We assume that the creatve desgner n hs/her desgn process wll consder more envronment components, relatonshps and there are more transtons of evaluatng envronments between dfferent desgn states. Ths assumpton wll be valdated by the statstcal analyss. The statstcal analyss s based on some quanttatve values and wll be conducted from the desgn processes of the seven subjects. The followng quanttatve values wll be summarzed from the experment results. - Number of envronment components n each state - Number of envronment-product relatonshps - Number of repeated envronment-product relaton - Number of concepts generated - Tme spent n the desgn process and n each desgn state - Tme spent n delverng a key concept - Number of transtons/transton rate of evaluatng the envronment, the product and ther relatonshps - Number of conflcts. Ths analyss s stll on-gong. The results wll be reported n the future. 5. Conclusons and Future Work Ths paper presented a protocol analyss approach for studyng desgn actvtes. The axomatc theory of desgn modelng s ntroduced as the bass of the protocol analyss approach. The experment set-up s llustrated. We also nvestgated the experment analyss ncludng transcrpton, segmentaton and encodng, consstency, the measurement of desgn solutons and the statstcal analyss to be conducted. Ths paper s a prelmnary study of protocol analyss on desgn actvtes; future work needs to be done to measure and compare the experment results of dfferent subjects. The comparsons between dfferent desgners wll be checked to get observatons and suggestons for gudng desgners to make creatve desgn decsons. 6. Acknowledgement Ths work s partally supported by NSRC (Grant number RGPIN 9855). We would lke to thank Hongln Yang and Imran Ur Rehman who asssted wth data collecton and analyss for ths research. We 48
also would lke to express our thanks to all subjects of the experment for ther partcpaton. 7. References [] Altshuller, G.S., Creatvty as an xact Scence: the Theory of the Solutons of Inventve Problems. Studes n Cyberbetcs, ed. F.H. George. 984: Gordon and Breach Scence Publshers. [] Braha, D. and O. Mamon, A Mathematcal Theory of Desgn. 998: Kluwer Academc Publshers. [3] rtas, A., M.M. Tank, and T.T. Maxwell, "Transdscplnary engneerng educaton and research model". Transacton of the SDPS: Journal of Integrated Desgn & Process Scence, 000. 5(4): p. -. [4] Hubka, V. and W. der, Desgn Scence. 996: Sprnger. 3. [5] Maxwell, T.T., A. rtas, and M.M. Tank, "Harnessng complexty n desgn". Transacton of SDPS: Journal of Integrated Desgn and Process Scence, 00. 6(3): p. 63-74. [6] Pahl, G. and W. Betz, ngneerng Desgn: A Systematc Approach. 988: Spnger-Verlag. [7] Suh, N., The Prncples of Desgn. 990: Oxford Unversty Press. [8] Zeng, Y., "Axomatc theory of desgn modelng". Transacton of SDPS: Journal of Integrated Desgn and Process Scence, 00. 6(3): p. -8. [9] Dasgupta, Creatvty n Innovaton and Desgn. 994. [0] Jn, Y. and P. Chuslp, "Study of mental teraton n dfferent desgn stuatons". Desgn Studes, 006. 7(): p. 5-55. [] Atman, C.J., M.. Cardella, J. Turns, and R. Adams, "Comparng freshman and senor engneerng desgn processes: an n-depth follow-up study". Desgn Studes, 005. 6(4): p. 35-357. [] Atman, C.J., J.R. Chmka, K.M. Bursc, and H.L. Nachtmann, "A comparson of freshman and senor engneerng desgn processes". Desgn Studes, 999. 0(): p. 3-5. [3] Suwa, M., T. Purcell, and J. Gero, "Macroscopc analyss of desgn processes based on a scheme for codng desgners' cogntve actons". Desgn Studes, 998. 9(4): p. 455-483. [4] Gero, J.S. and T. Mc Nell, "An approach to the analyss of desgn protocols". Desgn Studes, 998. 9(): p. -6. [5] Akn, O. and C. Akn, "On the process of creatvty n puzzles, nventons, and desgns". Automaton n Constructon, 998. 7: p. 3-38. [6] Cross, N., "Descrptve models of creatve desgn: applcaton to an example". Desgn Studes, 997. 8(4): p. 47-440. [7]http://www.nformatcs.susx.ac.uk/books/compute rs-and-thought/gloss/node.html. [8] Lloyd, P., B. Lawson, and P. Scott, "Can concurrent verbalzaton reveal desgn cognton?" Desgn Studes, 995. 6(): p. 37-59. [9] Suwa, M. and B. Tversky, "What do archtects and students perceve n ther desgn sketches? A protocol analyss". Desgn Studes, 997. 8(4): p. 385-403. [0] Zeng, Y., "nvronment-based formulaton of desgn problem". Transacton of SDPS: Journal of Integrated Desgn and Process Scence, 004. 8(4): p. 45-63. [] Zeng, Y. and G. Cheng, "On the logc of desgn". Desgn Studes, 99. (3): p. 37-4. [] Roozenburg, N.F.M., On the logc of nnovatve desgn, n Research n Desgn Thnkng, N. Cross, K. Dorst, and N.F.M. Roozenburg, dtors. 99, Delft Unversty Press. p. 7-38. [3] Maher, M.L. and H.-H. Tang, "Co-evoluton as a computatonal and cogntve model of desgn". Research n ngneerng Desgn, 003. 4: p. 47-63. [4] Dorst, K. and N. Cross, "Creatvty n the desgn process: co-evoluton of problem-soluton". Desgn Studes, 00. (5): p. 45-437. [5] Lnsone, H.A. and M. Turoff, The Delph Method Technques and Applcatons. 975, Addson-Wesley, Readng, MA. [6] Zeng, Y., B.Q. Yan, B. Chen, and S.J. Yao. A Theoretcal and xpermental Study on Desgn Creatvty. n The Inaugural CDN Desgn Conference. 004. Montreal, Canada. 49