Transcription

1 Cometencies of Excetional and Non-Excetional Software Engineers Richard T. Turley Colorado Memory Systems, Inc. James M. Bieman Colorado State University Published in Journal of Systems and Software, 28è1è:19-38, January Key words: software engineering, large software develoment, software teams, knowledge and skills of software engineers, software roductivity, software sychology Running title: Cometencies of Software Engineers Abstract The attributes of individual software engineers are erhas the most imortant factors in determining the success of software develoment. Our goal is to identify the rofessional cometencies that are most essential. In articular, we seek to identify the attributes that diæerentiate between excetional and non-excetional software engineers. Phase 1 of our research is a qualitative study designed to identify cometencies to be used in the quantitative analysis erformed in Phase 2. In Phase 1, we conduct an in-deth review of ten excetional and ten non-excetional software engineers working for a major comuting ærm. We use biograhical data and Myers-Briggs Tye Indicator test results to characterize our samle. We conduct Critical Incident Interviews focusing on the subjects exerience in software and identify 38 essential cometencies of software engineers. Phase 2 of this study surveys 129 software engineers to determine the cometencies that are diæerential between excetional and non-excetional engineers. Years of exerience in software is the only biograhical redictor of erformance. Analysis of the articiants Q-Sort of the 38 cometencies identiæed in Phase 1 reveals that nine of these cometencies are diæerentially related to engineer erformance using a t-test. A ten variable Canonical Discrimination Function consisting of three biograhical variables and seven cometencies is caable of correctly classifying 81è of the cases. The statistical analyses indicate that excetional engineers èat the comany studiedè can be distinguished by behaviors associated with an external focus behaviors directed at eole or objects outside the individual. Excetional engineers are more likely than non-excetional engineers to maintain a ëbig icture", have a bias for action, be driven by a sense of mission, exhibit and articulate strong convictions, lay a ro-active role with management, and hel other engineers. Authors addresses: R. Turley, Colorado Memory Systems, Inc., 800 S. Taft Ave., Loveland, CO RICKTURL.COMEMSYS@CMS SMTP.gr.h.com, è303è , Fax: è303è ; J. Bieman, Deartment of Comuter Science, Colorado State University, Fort Collins, CO bieman@cs.colostate.edu, è303è , Fax: è303è Coyright cæ1993 by Richard T. Turley and James M. Bieman. Permission to coy without fee all or art of this material is granted rovided that the coies are not made or distributed for direct commercial advantage, the coyright notice and the title of the ublication and its date aear, and notice is given that coying is by ermission of the author. Direct corresondence concerning this aer to: J. Bieman, Deartment of Comuter Science, Colorado State University, Fort Collins, CO 80523, bieman@cs.colostate.edu, è303è , Fax: è303è

2 1 Introduction We reort on a study of the diæerences between individual software develoers. This study is based on the remise that excetional software engineers exhibit diæerent skills which they aly to the roblems of software engineering. These unique skills can be identiæed by careful study of exerienced software engineers. Further, once these skills are recognized, we hoe that they can be transferred to the software engineering community at large through formal training rograms ëkelley and Calan 1993ë. Thus, additional software engineers can be taught these valuable skills. Our overall goal is to identify the skills, techniques, and attributes that diæerentiate between excetional and non-excetional software engineering erformance. Much eæort has been laced in the develoment of engineering aroaches to software develoment such as software tools, coding ractices, and test technology. But the overwhelming determiner of software roductivity and quality is still ersonnel and team caability. Boehm found ersonnel and team caability tobe twice as imortant as the next most imortant roductivity factor ëboehm 1981ë. By studying excetional rogrammers, the individual caabilities that most inæuence erformance can be identiæed ëcurtis 1981ë. Most research into the develoment of software focuses on the individual only to the extent that individuals are members of a larger develoment eæort. Although the team is a critical comonent in software develoment, most research misses a fundamental oortunity toidentify and exloit the roven ability ofhighly talented individual contributors. Weinberg noted the lack of research on individuals observing that ëour rofession suæers under an enormous burden of myths and half-truths." ëweinberg 1971ë. The industry has a great lore about the factors aæecting software roductivity, but few facts are known. Bohem also cites a 25-to-1 ratio between the most roductive and least roductive software develoers and a 10-to-1 diæerence in their error rates ëboehm 1988ë. If the ersonal attributes of these most roductive individuals can be understood, a number of exciting oortunities resent themselves: æ Understanding the characteristics of the most successful software develoers could lead to the imrovement of all software develoers. æ Once the characteristics are understood, it may be ossible to develo seciæc toolsets and aids to further increase the roductivity of these individuals. æ A valuable criterion for the selection of software develoers may be discovered. Brooks suggests the ëuse of great designers" as one of æve romising aroaches to imrove software develoment roductivity ëbrooks 1987ë. One of Boehm's seven basic rinciles of software engineering is to use ëbetter and fewer eole" ëboehm 1983ë. Tyical exerimental aroaches to studying individuals in software develoment start with an individual's exerience and rejudices about software develoment ëbrooks 1975ë. A technique for imrovement is roosed, imlemented and tested ëshneiderman 1976, Curtis et al 1979ë. The results of these exeriments are then analyzed and often valuable results are achieved. This study follows a diæerent aroach. We start with rofessional software develoers who are acknowledged for their software ability. Our focus on the to individual contributors breaks with the traditional emhasis on the team. We seek to enhance the value of teams by ensuring that each individual is oerating at eak roductivity. A relevant theory of excetional erformance does not exist. Thus, we take an observational rather than a theory-driven aroach. Our examination of the successful ractices of software engineers should lead to the develoment of a theory. We do not make a riori assumtions 2

3 concerning the relative imortance of diæerent engineering activities. For examle, we do not weight earlier life cycle activities such as requirements analysis heavier than later activities. Such a ranking scheme would interfere with the objectivity of the study. Our aim is to determine the attributes that are necessary for excetional erformance, so that the erformance of all software engineers can be imroved. We reort the results from a two hase study designed to determine the essential cometencies of rofessional software engineers ëturley 1991ë. In Phase 1 we identify these cometencies via the Critical Incident Interview technique. In Phase 2 we diæerentially relate these cometencies to engineer erformance. Phase 1 corresonds to the qualitative ortion of the research in which the cometencies associated with the job of software engineering are ærst uncovered. Phase 2 corresonds to the quantitative ortion of the research in which the cometencies discovered in Phase 1 are validated and considered on a diæerential basis between excetional and non-excetional erformers. The remainder of this aer is organized as follows. Section 2 describes Phase 1 in detail, and Section 3 resents the details of Phase 2. Section 4 contains a discussion of the imlications of the results. We review related work in Section 5, and our conclusions are given in Section 6. 2 Phase 1 In Phase 1, we identify critical rofessional cometencies through an in-deth analysis of a small samle of excetional and non-excetional software engineers. We use a biograhical questionnaire and a Meyers-Briggs Tye Indicator èmbtiè test ëmyers and McCaulley 1985ë to characterize our samle. We conduct Critical Incident Interviews to identify the signiæcant cometencies of software engineering. 2.1 Phase 1 Subjects Subjects are drawn from æve commercial research and develoment laboratories at three diæerent sites of a single comany. The software engineering environments varied greatly between the laboratories. While all of the laboratories used software rocess standards and software engineering tools, the seciæc methods diæered. The subjects develo alications in test and measurement, embedded ærmware, and comuter aided design. We use two matched subject ools with 10 subjects in each of the excetional and nonexcetional ools. The subjects are matched by time in current organization. Thus, if an excetional engineer with four years in the current organization is identiæed, a second non-excetional engineer with four years exerience in the same organization is added to the study. This aroach controls for the eæect of the organization on the individual's erformance. The study does not attemt to control any other factors, since all are ossible contributors to excetional erformance. All subjects are rofessional software develoment engineers from a major US cororation èreferred to as The Comany for rorietary reasonsè with a minimum of two years of exerience in develoing software. The Comany isafortune 500 comany involved in the design, manufacture, and suort of single and multi-user comuter systems. This comany is large enough and has enough exerienced ersonnel to be caable of excetional software engineering. Each subject has successfully comleted a roject released to the end user. Table 1 summarizes the oulation from which the Phase 1 study articiants are drawn. The èsw Engineers reresents the number of software engineers in the æve comany laboratories. The èstudy Particiants indicates the number of engineers that were selected for the study. The è Studied Excetional SW Engineers is the ratio of the number of excetional software engineers studied to the total number of software engineers in the oulation. The oulation reresents a samle of organizational units in The Comany. 3

4 Phase 1 Poulation Summary Total èsw Engineers 150 èstudy Particiants 20 è of Excetional SW Engineers Studied 10 è Studied Excetional SW Engineers èout of total SW Engineersè 6.7è Table 1: Phase 1 Poulation Summary Subjects are selected by a rocess in which managers identify the to erformers in their organization. Managers were asked to identify an excetional èto 5è of the organizationè and non-excetional erforming air of individuals. The air should have sent the same amount of time in the organization. As a result of this rocess, manager bias is an inherent art of the research design. Excetional software engineers are those identiæed as excetional by managers. Managers may be biased in favor of ëromotability to management" skills. If so, romotability will be a comonent of what managers consider excetional. We do not see this as a roblem, since managers should be the best judge of what is considered excetional behavior from The Comany's ersective. In addition, we havenoway to searate, a riori, ëromotability" behavior from ëengineering behavior." Vessey also used manager assessment as a method èthe ëex ante" methodè for identifying exerts ëvessey 1985ë. Conducting Critical Incident Interviews is quite labor intensive. As a result, the samle size is fairly small. With this samle we are able to erform an evaluation giving us a rich set of qualitative information. These initial results can be validated through further studies of larger samles using closed end survey instruments. 2.2 Biograhical Proæle A biograhical questionnaire is used to evaluate the subject ool. The questionnaire validates that subjects reresent exerienced rather than naive rogrammers, and that subjects include a valid cross-section of develoers covering diæerent language use, target alications, and develoment environments. The questionnaire requests information concerning education, on the job training, exerience, languages used, and methods emloyed. We ænd that: æ 75è of the subjects are male; 25è are female. The 3 to 1 ratio is consistent with ublished reorts that women constitute only 30è of the emloyed comuter scientists ëpearl et al 1990ë. æ The mean age of the subjects is years. æ The mean number of degrees held is è of the subject hold a Bachelors degree as the highest degree, 30è hold a Masters degree, and one subject è5èè earned a Ph.D. æ The mean number of training hours comleted er subject in the two years receding the study is hours. Comleted training ranged from zero to 306 contact hours. æ Subject resonses to the question of describe the software engineering methods and tools that you use now or in the ast in your job varied too greatly to be very useful. æ Subjects had worked at The Comany a mean of 7 years in software engineering, ranging from 2 to 15 years. 4

5 èn=20è Years at Comany Mean Std Range in Software Dev Excetional í15 Non-Excetional í7.5 Table 2: Years at Comany in Software Diæerential The data were slit between Excetional and Non-Excetional subjects and comared. The biograhical data were analyzed for statistical signiæcance at the.05 level when studied on a diæerential basis. The Fisher's Exact Test was used to comare nominal variables with only two values èe.g. genderè. The t-test was used to comare the means of ordinal values èe.g. training hoursè. Since this was such a small samle, we did not exect any signiæcant diæerences between the Excetional and Non-Excetional grous. However, Years at Comany in Software are signiæcantly related to Excetional Performance with the 2-tail t-test calculated value of with a signiæcance level of.007. This signiæcance demonstrates that although subjects were matched for total exerience in the current organization, they were not matched for Years at Comany in Software. Table 2 shows the diæerential information concerning years in The Comany in software. The demograhic analysis indicates that, with the excetion of the exerience variable, no demograhic data were signiæcantly diæerent between the excetional and non-excetional sub-samles in this small samle of 20 subjects. Thus, for examle, the software engineering environment was not a signiæcant factor in distinguishing between the excetional and non-excetional subjects èerhas because both excetional and non-excetional subjects work under the same environmentè. The lack of other statistically signiæcant diæerences indicates exerimental control of the other variables, the uniformity of the samle, or the weakness of the identiæed diæerences. 2.3 Myers-Briggs Cognitive Style Tye Indicator èmbtiè The MBTI is a tool for determining sychological tye ëmyers and McCaulley 1985, Shneiderman 1980ë. We use it to determine if tye diæerences exist between excetional and non-excetional engineers. Through a questionnaire, the MBTI comutes a score for four contrasting ersonality airs: æ extrovert vs. introvert æ sensing vs. intuitive æ thinking vs. feeling æ judging vs. ercetive The urose of the MBTI is to identify, from self-reort of easily recognized reactions, the basic references of eole with regard to ercetion and judgement ëburos 1989ë. The four references are assumed to interact in comlex nonlinear ways to roduce one of 16 sychological tyes with diæerent attributes ëisachsen and Berens 1988ë. The MBTI can rovide a continuous score for each of the four reference scales allowing for statistical analysis of signiæcant diæerences ëmyers and McCaulley 1985ë. A detailed descrition of the MBTI aears in ëshneiderman 1980ë. All 20 subjects comleted the Myers-Briggs Tye Indicator èmbtiè test. Figure 1 shows the distribution of the Phase 1 study articiants according to one of 16 ersonality tyes. Eighteen out of twenty subjects exhibit the Introvert tye. The Introvert tendency is consistent with the 5

6 Sensing Tyes Intuitive Tyes With Thinking With Feeling With Feeling With Thinking ISTJ ISFJ INFJ INTJ Introvert 3 Judging ISTP ISFP INFP INTP Introvert 33 3 Percetive ESTP ESFP ENFP ENTP Extrovert Percetive ESTJ ESFJ ENFJ ENTJ Extrovert 3 Judging = 1 Non-Excetional Subject 3= 1 Excetional Subject Figure 1: Myers-Briggs Tye Indicator èmbtiè Results. Kagan-Douthat study of students learning Fortran which found a tendency towards introversion in higher erforming rogramming students ëkagan and Douthat 1985ë. We also found a that 17 out of 20 subjects exhibited the Thinking tye èrather than Feeling tyeè. This result is consistent with broader studies that ænd that comuter secialists exhibit the thinking reference 67è of the time ëmyers and McCaulley 1985ë. The MBTI test uncovered an interesting tendency for excetional engineers to favor the Introvert, Thinking tye. The most frequent classiæcation for excetional erformers is the INTJ èintrovert, Intuitive, Thinking, Judgingè tye. The INTJ tye occurs in only 10è of the male college graduates ëmyers and McCaulley 1985ë. Hence these excetional engineers diæer from the èmaleè oulation at large. 1 Only one of the excetional engineers is classiæed as an Extrovert tye. The non-excetional engineers exhibited more varied ersonality tyes. Only two of the nonexcetional engineers exhibited the INTJ tye. Six of the non-excetional engineers exhibited a combination of the Introvert and Judging tyes. Like the excetional grou, only one of the non-excetional engineers is classiæed as an Extrovert tye. A statistical analysis via the t-test of the diæerential MBTI scores revealed that there were no signiæcant diæerences between the scores of the excetional and non-excetional subgrous. This indicates that either ersonally tye is not a good redictor of erformance or that the samle size is too small. We do not use the MBTI test in Phase 2 because of the inconclusive results in Phase 1 èdue to the small samleè, and because the MBTI test is exensive and very time consuming for a larger samle. 1 The data given by Myers and McCaulley ëmyers and McCaulley 1985ë is unfortunately limited to males. 6

7 2.4 Interview Process Each Critical Incident Interview was conducted in a rivate room at the subject's work site. Each interview was tae-recorded, and the recordings were transcribed for later use. The interviews began with casual conversation followed by a descrition of the scoe of the research and the general æow of the interview. The interview followed the basic structure and ractices deæned in ëhewlett-packard 1989ë. A tyical interview began with an introduction similar to the following one taken from the transcrit of one of the interviews: What I'd like you to do is start oæ by thinking about a time which reresents for you erhas your ersonal best associated with software engineering in whatever form, so be it software develoment, software maintenance, testing, whatever it is, but a time at which you feel you were at your ersonal best, and when you've got one of those situations in mind, give me kind of a broad overview, a æfty word summary overview which is, how did you get involved in the situation, who were the other layers, what was the nature of the task, and then we'll come back and we'll walk through it ste by ste in gory detail to ænd out exactly what you did in each case of that task. The subject would then describe an incident and the interviewer would robe for clariæcation or increased deth of resonse. The interviewer used robes, oen-ended questions, questions of clariæcation, and reæective listening to kee the articiant onthe subjects of interest. The only way that the interviewer tried to direct the conversation was to rovide additional clariæcation or to move on to other toics. The subject generally described two to three signiæcant incidents in the course of one two hour interview. When each incident was comleted, the subject was asked to describe the critical skill or cometencies which were essential to the successful comletion of the task. At the end of the discussion of the subject's incidents, the subject was asked to describe the list of essential cometencies for an excetional software engineer. 2.5 Analysis of Critical Incident Interviews The Critical Incident Technique attemts to discover the critical job requirements that have been demonstrated to make a diæerence between success and failure ëflanagan 1954ë. The technique was introduced during World War II in the Aviation Psychology Program to study combat leadershi and ilot disorientation. The technique has since been reæned and alied to measures of erformance, measures of roæciency, training, selection, job design, equiment design, and leadershi. Protocol Analysis is used to translate the verbatim coy of an interview to a generalized set of cross-transcrit results ëericssson and Simon 1984ë. A formal rocess rovides a record of the analysis and allows identiæed relations to be tied to seciæc utterances in the original transcrits ëweber 1985, McCracken 1988ë. We used the Protocol Analysis technique described by McCracken ëmccracken 1988ë. Each written transcrit was reviewed and highlighted to identify tasks, incidents, cometencies, selfdescribed skills, and identiæed cometencies for excetional erformance. Each transcrit was reviewed individually to identify consistent themes which could be generalized as cometencies for that individual. After each transcrit was reviewed individually, the set of transcrits was examined to identify cometencies which aear across multile transcrits. These cometencies were generalized and reworded as required to emhasize the similarities. Great care was taken not to over-generalize or distort the original meanings. A set of behaviors was identiæed based uon all of the the transcrits and served as a detailed exlanation of the intent of the cometency. At this 7

8 oint, original transcrit text was retained and attached to the cometency as further deænition. A ænal ass allowed the combination of related cometencies into a single cometency. All of the analysis to this oint was done blindly. The transcrits were tagged with an identiæcation number and the analyst did not know the name of the subject. Further, the analyst did not know if the transcrits were from an excetional or non-excetional subject. The next ste of the rocess was to count the number of subjects exhibiting an identiæed cometency from each of the excetional and non-excetional grous. Those cometencies exhibited by few subjects were droed from further consideration. In general, at least three subjects had to identify a cometency before it was retained. However, if one excetional and one non-excetional subject identiæed a cometency, it was also retained. 2.6 Identiæed Cometencies The 20 Critical Incident Interviews yielded a massive amount of data. Each interview lasted an average of two hours. Hence, the full set of data consists of 40 hours of taed interviews. The transcrition of these taes roduced over 200,000 words for just the subject resonses. Derived Cometencies A total of 27 cometencies were derived from the analysis of the subjects descrition of their own role in seciæc incidents. These cometencies are identiæed by marking the skills, knowledge, or ersonal attributes alluded to while describing their own role in the incidents. Self-Described Cometencies Subjects were also asked to name the skills, knowledge, or ersonal attributes most imortant in heling them achieve their success in the described incident. The subjects were romted for this resonse by a very oen-ended question. Hence the relies are resumed to be the cometencies considered most signiæcant by the study articiants. Each subject enumerated those cometencies that they felt most contributed to their own success. All summary lists for each of the 20 subjects were combined into a single list of cometencies. Related cometencies were merged to form a single cometency. The number of subjects, both excetional and non-excetional, exressing the cometency was noted. The cometencies mentioned most frequently were retained for future analysis. Many of the cometencies cited by engineers as being imortant to their own success, are, in fact, the same cometencies identiæed from the analysis of the transcrits. Manager Described Cometencies Another set of cometencies was created by asking the managers of the subjects: What are the Knowledge, Skills, or Attributes that diæerentiate your excetional erformers from your non-excetional erformers? These are the same managers who classiæed the subjects in their organization as excetional or non-excetional. Sixteen diæerential cometencies were identiæed by the æve managers in the study. There was no further discussion with these managers to rovide further elaboration on these cometencies. Many of these cometencies are similar to those identiæed by the analysis of transcrits or cited by engineers as those leading to excetional erformance. 8

9 Summary of Cometencies Table 3 summarizes the cometencies identiæed most frequently from the multile sources. The Derived category refers to those cometencies extracted from the analysis of the interview transcrits. They reresent those areas which the subject chose to discuss during their narration about their exeriences. The number in this column records the number of subjects that described behaviors related to this cometency. The Self-Described column records the number of subjects that oæered the listed cometencies when were asked to describe the skills, knowledge, and attributes associated with their successful erformance on rojects. The Manager records how many of the æve managers cited the listed cometencies as those that diæerentiate between excetional and non-excetional erformers in their organization. The cometencies derived from the rotocol analysis are considered to be more imortant than the cometencies oæered directly by the engineers or managers. This is because this study is based on the notion that behaviors associated with high erformance are the unit of study. And it is through the interviews that subjects demonstrate these behaviors. We consider cometencies that are validated by multile sources to be more imortant than cometencies that come from only one source. Anumber of cometencies were identiæed by the subjects andèor managers, but were not included in the set of cometencies that will be used for further research. These cometencies were rejected because few eole identiæed the cometency, oritwas not validated by multile sources. The identiæed cometencies rovide an alternative view of the job of software engineering. Rather than an antisetic alication of formal software methods, we ænd a broad mix of knowledge, ersonality, and attitude involved. In addition to the exected technical skill cometencies èuse of Prototyes, Automates Tests, Reuses Code, Uses Code Reading,...è we ænd ersonality èsense of Fun, Lack of Ego, Willingness to Confront Others, Perseverance,...è and attitude èpride in Quality, Strength of Convictions, Bias for Action, Desire to Imrove Things,...è emerge as signiæcant factors in the engineering rocess. The cometencies were analyzed on a diæerential basis using Fisher's Exact Test with a 2- tail robability. The score used for this test was the number of subjects that described behavior exhibiting a articular cometency. Only one of the cometencies exhibited signiæcant diæerences between excetional and non-excetional subjects. There was a signiæcant diæerences between the grous with a 2-tail comuted signiæcance level of for the Use of Prototyes cometency. None of the remaining cometencies exhibited signiæcance at the 0.05 level or better. Although most of the cometencies cannot be used to distinguish between the excetional and non-excetional subjects based on this small samle of 20 subjects, the derived cometencies oæer a unique view of the necessary skills of rofessional software engineers. For comlete a descrition of all of the identiæed cometencies see ëturley 1991ë. 3 Phase 2 In Phase 2, we use the identiæed cometencies, larger samles, and objective survey instruments to detect signiæcant diæerences between excetional and non-excetional software engineers. Our objectives are to determine which cometencies identiæed in Phase 1 are diæerentially related to erformance, and determine if a simle redictor of erformance exists. We develo a redictive model that uses the cometencies to redict whether a articular engineer will be ranked as excetional or non-excetional. 9

10 Self- Cometency Derived Described Manager 1. Team Oriented Seeks Hel Hels Others Use of Prototyes WritesèAutomates Tests with Code Knowledge Obtains Necessary TrainingèLearning LeveragesèReuses Code Communicationè Uses Structured 8 8 Techniques for Communication 10. Methodical Problem Solving Use of New Methods or Tools Schedules and Estimates Well Uses Code Reading Design Style Focus on User or Customer Needs Resonse to Schedule Pressure Emhasizes Elegant and Simle Solutions Pride in Quality and Productivity Pro-activeèInitiatorèDriver Pro-active Role with Management Driven by Desire to Contribute Sense of Fun Sense of Mission Lack of Ego Strength of Convictions Mixes Personal and Work Goals Willingness to Confront Others Thoroughness SkillsèTechniques Thinking Desire to DoèBias for Action Attention to Detail Perseverance Innovation Exerience Desire to Imrove Things Quality Maintaining a ëbig icture" viewè 1 3 Breadth of View & Inæuence n =20 n=20 n=5 Numbers indicate the frequency that cometencies are identiæed as follows: Derived: extracted from interview transcrits. Self-Described: oæered by subjects as most imortant cometencies. Manager: oæered by managers as diæerential cometencies. Table 3: Essential Cometencies 10

11 Poulation Summary Total Surveys Distributed 275 Total Resonses 129 Resonse Rate 46.9è è Excetional Resonses 41 è Non-Excetional Resonses 88 è Excetional 31.8è è Non-Excetional 68.2è Table 4: Phase 2 Poulation Summary 3.1 Phase 2 Subjects In Phase 2, we seek to validate the Phase 1 results against a broader oulation. Thus we exand the samle of excetional and non-excetional software engineers both in quantity and diversity. Matching for time in the organization is not required since the breadth of subjects is exected to eliminate the relevance of diæerences in exerience. In addition, the deænition of ëexcetional" was widened to include the to 30è rather than the to 5è of engineers. This widened deænition allows a more even mix of excetional and non-excetional engineers in the study. Allowing more subjects to be deæned as excetional is a conservative aroach we increase the risk that a cometency will not be identiæed as diæerential. The resulting increase in the relative number of excetional engineers in the subject ools also aids the statistical analysis. As in Phase 1, all subjects are software engineers emloyed by The Comany. We did not use atwoyear minimum exerience criterion as in Phase 1. Managers were asked to distribute surveys to their entire lab on a diæerential basis 70è of the surveys are distributed to non-excetional erformers and 30è to excetional erformers. The determination of excetional versus nonexcetional was again made by the managers. Managers were allowed to distribute excetional surveys to slightly more than 30è of their lab based on their judgment of erformance. Managers were instructed to kee the diæerential nature of the survey conædential. A total of 275 survey instruments were distributed to engineers working in nine divisions of The Comany at three sites. The engineers articiate in the develoment ofævetyes of software alications test & measurement, embedded ærmware, CAEèCADèCASE software, grahics, and oerating systems. 3.2 Descritive Statistics Each survey acket contained a letter of instruction that outlined the assignment and clearly indicated the voluntary nature of the study. Each acket included a Biograhical Questionnaire and a set of Q-Sort cards. The acket also included a re-addressed return enveloe for returning the comleted survey. The results were thus blind in that we did not know the names of study articiants or their corresonding rating. The Biograhical Questionnaire used in Phase 2 was nearly identical to the one used in Phase 1. Some minor changes were made for book keeing uroses. A Results of Sorting section was added to cature the results of the Q-Sort activity. The total number of surveys distributed, resonses, and distribution between resonses from excetional and non-excetional engineers are indicated in Table 4. Only four of the Phase 2 resonses were incomlete in some of the major indeendent variables. We consider only valid surveys in our analysis of each variable. As a result, there is some variation in the reorted number 11

12 of samles n. The resonse rate of nearly 50è indicates the level of interest in this information at the comany studied. The samle of 129 articiants rovides suæcient statistical ower to comlete the study. The resonse rate for excetional and non-excetional erformers was similar, since 30è of the surveys were distributed to excetional erformers and 31.8è of those returned were from this grou. We collected descritive statistics using a Biograhical Questionnaire to ensure that the Phase 2 samle is similar to that of Phase 1 We also analyzed the data for normalcy so that subsequent statistical stes will be valid. Phase 2 subjects can be described as follows: æ 78.9è of the subjects are male; 21.1è are female. This distribution is similar to the Phase 1 mix and reæects the reonderance of males in Comuter Science. æ The mean age is years and is comarable to the Phase 1 mean age of years. æ The subject ool is well educated with over 53è holding two or more degrees, and 40è have a Master's degree as their highest degree. Over 74è hold at least one degree in Comuter Science, while only 35è of Phase 1 subjects held degrees in Comuter Science. æ The mean number of training hours comleted in the rior two years was 102, versus 117 hours for Phase 1 subjects. æ Subjects had worked in The Comany for a mean of 6 years in software engineering, comared to a mean of 7 years for Phase 1 articiants. We analyzed the descritive data to determine if there are signiæcant diæerences between the excetional and non-excetional grous. The Chi-Square test is used to test for statistically significant diæerences between nominal variables such as gender. The t-test is used to test for signiæcant diæerences between ordinal variables. Diæerences are considered statistically signiæcant when the calculated 2-tail signiæcance level is less than We found only three statistically signiæcant biograhical variables associated with excetional erformance: Years at Comany in Software, Total Years in Software, and Total Years Worked. This is consistent with the Phase 1 result indicating that Years at Comany in Software is a diæerential attribute. 3.3 Q-Sort The Q-Sort method is used to assist subjects in ranking the cometencies identiæed in Phase 1. Q Methodology encomasses the Q-Sorting Technique, which is designed to rovide ractical means for subjects to sort and researchers to analyze large lists of items ëmckeown and Thomas 1988ë. The method stresses the individual's ercetion of value in a set of statements as the actual data under study. The technique has a long history being ærst romoted by Stehenson in the 1930's. His text continues to be a signiæcant reference on the technique ëstehenson 1953ë. Using Q-Sort, a subject is asked to rank order a set of items against a seciæc condition of instruction. The ordering is quasi-normal in that it asks subjects to lace the item in one of a limited number of bins or iles. The number of items is exected to far exceed the number of iles. Each ile maintains a seciæc relationshi to the other iles. The number of items to be laced in each ile is meant to be roortional to a roughly normal distribution of the items. For examle, if there are ten items to distribute across æve iles, the ærst ile will have one item, the second ile will have two items, the third ile will have four items, the fourth ile will have two items, and the æfth ile will have one item. This arrangement aroximates a normal distribution. Critical to the sorting is the condition of instruction. A subject may rovide a radically diæerent sorting based uon the instructions given. For examle, a subject could be instructed to sort 12

13 DEFINITION I value the synergy of grou eæorts and invest the eæort required to create grou solutions, even at the exense of my individual results. Key Behaviors æ I balance the strengths and weaknesses of other team members. æ I romote constant communication among team members using techniques such as brainstorming sessions, travel, hone calls, , or just being hysically close to the rest of the team. æ I recognize synergy of grou eæorts and invest ersonal time and energy to leverage it. Item è1 Figure 2: Q-Sort Card for Cometency è1, Team Oriented. cometencies based uon è1è the order which most relates to being excetional, or è2è the order based on the subjects own behavior on the job. We would exect a diæerent result deending on which instructions are given. We emhasize that the criterion for sorting the cometency cards is the subject's self reort of his or her own behavior. Using Q-Methodology, a Q-Sort task is normally comleted by a subject with the hel of the researcher. We used a simliæed aroach to the Q-Sorting task to allow subjects to comlete the task on their own. Each subject received a set of Cometency Cards with one cometency listed on each of æ 5 00 index cards. èfigure 2 shows a cometency card for Cometency è1, Team Oriented.è A set of Pile Marker Cards is also included in order to romt subjects to create the correct number of iles and to include the correct number of cards in each ile. Further, the Pile Marker Cards include romts to remind subjects of the deænition of the continuum across which the cometencies are sorted. The directions that subjects followed in comleting the Q-sorting exercise are given in Figure 3. Study articiants sorted a set of 38 cometencies into a quasi-normal distribution of seven iles. Each ile was assigned an integer value from zero to six. Zero means Least Like My Behavior while six means Most Like My Behavior. For each survey, the Q-Sort item was assigned the integer value associated with the ile that the subject laced it into. We calculated the mean Q-Sort for the full samle of both excetional and non-excetional engineers. We also calculated the skew and kurtosis numbers which indicate that all Q-Sort items are normally distributed. A t-test comarison of means for each of the Q-Sort Cometencies is given in Table 5. The means are calculated searately for excetional and non-excetional erformance and tested for diæerence. The two means are considered diæerent when the calculated signiæcance level is less than These entries are denoted by ææ in Table 5. The table is sorted by the mean scores of the excetional resonses. The Delta column reresents the number of laces that a articular cometency moves in its rank order when sorted by excetional means rather than sorted by the full samle means. Nine cometencies show statistically signiæcant diæerences in the mean values reorted by the excetional and non-excetional engineers. Thus 24è of the 38 cometencies are related to the diæerence in erformance of excetional and non-excetional engineers. The æve cometencies 13

14 Cometency Sorting Exercise The objective of this exercise is to determine which job cometencies identiæed in Phase 1 research best characterize the Comany's Software Engineering oulation. You will sort these cometencies based on how well they describe your behaviors on the job, esecially when you're erforming at your best. Try to think of the best software exerience you've had and use that to guide selection of which attributes best describe your behavior on the job. 1. Be sure that you have a clear desk or table to work on before you start. You will be lacing 3 æ 5 cards in one of 7 iles so you need sace to sread these out. Find the sulied ile markers in the enveloe and lay these out on your table in order from number6onyour left to number 0 on your right. These ile markers are annotated to remind you that column 6 reresents those cometencies that are most like your behavior and column 0 reresents those cometencies that are least like your behavior. 2. Read through all 38 cometency cards to become familiar with them. 3. Sort all of the cards into 3 iles of anynumber of cards. Place to the left the cards which include the cometencies which best describe your behavior in the rocess of software engineering. Place to the right those cards which include cometencies which least describe your behavior in the rocess of software engineering. Place those cards with cometencies about which you are unsure in the middle ile. 4. During the sorting you will sread the items in iles under the ile markers, while maintaining the general left-center-right relationshis. 5. Select the 2 items that most strongly relate to your behavior on the job as a software engineer. Think in articular about those time which have been a ersonal best for you. Place these two cards under the column marker labeled 6. The order of these cards under the marker is not imortant. All will receive the same score. 6. Now select the 2 items that least reæect your behavior on the job as a software engineer. Place these under the column marker labeled Continue in this way, alternating between the left and right sides of the distribution, lacing the indicated number of cards below each column marker. Feel free to move any card at any time should you change your mind about which cometencies are most closely related to your actual behavior. All that matters is that the right number of cards eventually are found beneath each column marker. Try not to take too long agonizing over the lacement ofany one card. Your ærst imulse for lacing the card is robably the best. If it hels, you can jot a short hrase that catures the essence of the cometency directly onto the card as a romt to use in sorting. 8. Review your grouings to be sure that they accurately reæect your behavior while comleting your software engineering assignments. Move any cards you wish to better reæect which cometencies most aly to you doing your job. Now record the item identiæcation numbers found in the lower right hand corner of each card in the aroriate column on the back of the Biograhical Questionnaire. If you have any questions, don't hesitate to give me a call at to ask for hel. Figure 3: Q-Sorting Instructions given to Comany Particiants. 14

15 èitems sorted by mean score of excetional resonsesè XP NXP Sig. Mean Mean Test Level Delta Cometency èn=40è èn=85è Value è2 Tailè 0 Concern for ReliabilityèQuality Focus on UserèCustomer Needs Thinking Pride in QualityèProductivity Emhasizes Elegant & Simle Solutions Driven by Desire to Contribute ææ +8 Mastery of SkillsèTechniques ææ +20 Hels Others ææ -1 Innovative Maintains ëbig icture" view ææ -2 Enjoys Challenge of Assignment í Has Fun Seeks Hel From Others ææ 0 Lack of Ego Prior Exerience Attention to Detail Pro-activeèInitiatorèDriver Team Oriented LeveragesèReuses Code Desire to Imrove Things Perseverance Strength of Convictions ææ +13 Pro-active Role with Management ææ +4 Schedules and Estimates Well Methodical Problem Solving WritesèAutomates Tests with Code Driven by a Sense of Mission Use of New Methods or Tools Uses Decomosition Design Style Desire to DoèBias for Action Obtains Necessary TrainingèLearning Uses Code Reading Use of Prototyes Possesses Unique Knowledge Mixes Personal and Work Goals Thoroughness - Methodical, Cautious Willingness to Confront Others ææ +1 Structured Techniques for Communication Resonds to Schedule Pressure by ææ Sacriæcing Parts of Design Process XP = Excetional Subject, NXP = Non-Excetional Subject. Diæerences are considered statistically signiæcant when the calculated signiæcance is less than These instances are in bold rint and are denoted by ë ææ ". Table 5: Diæerential Q-Sort Cometency Resonses, T-Test Results 15

16 which have a higher mean for excetional erformers and the behavior andèor attitudes of engineers that exhibit each cometency are brieæy described as follows: 1. Hels Others: sends a signiæcant amount of time assisting others in the comletion of their tasks or inæuencing broad organizational direction. These engineers act as lab-wide consultants for rocess or roduct issues; they review, direct, or inæuence the work of other engineers; they teach engineering skills to other engineers. 2. Pro-active Role with Management: ro-actively attemt to aæect roject direction by inæuencing management. These engineers discuss issues concerning other engineers with their managers; they attemt to set roject direction and make roject decisions by inæuencing their managers; they romote roduct ideas through demos or selling of ideas to management. 3. Exhibits and Articulates Strong Convictions: exhibits and articulates strong beliefs and convictions, and acts in accordance with these beliefs, even when they are counter to seciæc management direction. These engineers act in accordance with their beliefs rather than acting solely on their assignment; they risk their erformance ranking in an eæort to secure the best solution; they argue forcefully for a seciæc oint of view. 4. Mastery of Skills and Techniques: mastered the skills and techniques necessary for good software design and imlementation. These engineers have a strong technical and software develoment background; They are comfortable with multile software design and imlementation techniques; they have very strong software develoment skills. 5. Maintains ëbig Picture" View: sees the overall situation rather than focusing on details in an attemt to inæuence the roject direction. These engineers remain aware of what other engineers are doing and suggest ways to better achieve roject objectives; they try to be sure that roject goals make sense, and work to change them if necessary; they try to æt their roject into the broader scheme of division rograms. The four cometencies which have a higher mean for non-excetional erformers are: 1. Seeks Hel from Others: ro-actively seeks the assistance of others in learning, researching, designing, understanding, debugging, or checking results. These engineers ask revious develoers to exlain their designs; they ask other engineers to critique or evaluate their designs; they survey others to create lists of alternatives. 2. Resonds to Schedule Pressure by Sacriæcing Parts of Design Process: In resonse to schedule ressure, these engineers are forced to rovide incomlete documentation; they do not have time to adequately insect or test the roduct; they will not rototye or adequately design risky arts of the roduct. 3. Driven by Desire to Contribute: values the sense of accomlishment which comes from making a direct contribution. These engineers seek assingments where they can contribute and feel rewarded by the chance to contribute. 4. Willingness to Confront Others: confront others when necessary to ensure a good design or roduct solution. These engineers will not let a conæict simmer and will oenly confront another erson in order to resolve a roblem; they will raise a tough issue of conæict with another engineer to their manager in an eæort to have the conæict resolved. 16

17 Of articular interest, the Use of Prototyes is not diæerential according to the Q-Sort, although it was diæerential in Phase 1 with a signiæcance level of We oæer two ossible exlanations for this discreancy. A signiæcance level means that there is a 1è chance that the relationshi found in Phase 1 was just the result of chance, and Use of Prototyes was not really diæerential. The actual lack of signiæcance of the Use of Prototyes cometency was demonstrated by the larger samle used in Phase 2. Another exlanation is that the criteria for determining excetional erformance were signiæcantly diæerent between Phase 1 and Phase 2. In Phase 1 the excetional grou was to be in the to 5è of the organization, while in Phase 2 the excetional grou included the to 30è. The diæerences between the excetional and non-excetional grous are likely to be diminished by relaxing of standards for selecting excetional engineers. The urose of Phase 2 was to quantify and conærm the qualitative results of Phase 1. The Phase 2 results, generated from a larger samle and a broader deænition of excetional erformance, show that the Use of Prototyes cometency can not be used eæectively to distinguish between excetional and non-excetional engineers. Thus, in this case, the signiæcance indicated in Phase 1 could not be veriæed in Phase 2. Both excetional and non-excetional engineers indicate that they do not resond to schedule ressure by sacriæcing arts of the design rocess. The Resonds to Schedule Pressure cometency was ranked 38th last by excetional engineers and 37th next to last by non-excetional engineers. Although both grous ranked this cometency very low, the diæerence in ranking roved to be statistically signiæcant. Non-excetional engineers are more likely to rovide inadequate documentation or inadequate testing when the schedule gets tight. Somehow the excetional engineers are able to avoid this tra. A few of the results are counter-intuitive. For examle, there is not a signiæcant diæerence between the excetional and non-excetional grous in their use of new methods and tools and their view of the role of innovation. In both cases the non-excetional engineers ranked these cometencies slightly higher than the excetional grou. A ossible exlanation for the counterintuitive results is that there may be a discreancy between how engineers view their own activities and an evaluation of an outside observer. Excetional and non-excetional engineers may view themselves equally in terms of innovation as described in the cometency cards: ëi am innovative in my solutions to roblems. I like to create alternatives that are both creative and ractical. I have creative ideas and solutions to roblems." Yet an outside observer might rank the excetional and non-excetional engineers quite diæerently. Both the excetional and non-excetional engineers engineers rank innovation highly innovation is ranked 9th by the excetional engineers and 8th by the non-excetional engineers èout of 38 cometenciesè. 3.4 Discriminant Analysis We erformed a discriminant analysis of the full set of non-correlating variables, and then erform the analysis using a set of fewer variables. First we analyzed the correlations of variables, since highly correlated variables cannot be used in the analysis. Cross-correlations of the biograhical variables demonstrates that age and exerience variables are highly correlated. This is exected since engineers who are older will tend to have more exerience. We assume that exerience rather than age is the imortant variable here. Since not all of these variables can be used in subsequent analysis, we select Total Years in Software as the most aroriate variable. This choice is consisting with rior literature ëchrysler 1978ë. There is also a natural high correlation between the Number of Degrees Cometed and the Highest Degree Comleted. We use the Highest Degree Comleted in subsequent analysis. None of the 38 17

18 VARIABLE Gender Highest Degree Held Comuter Science Degree? Engineering Degree? Math Degree? Training Hours Total Years in Software Total Number of Languages 38 Cometencies Table 6: Retained Variables for Discriminant Analysis cometency variables were correlated with each other or with the biograhical variables at a level of 0.60 or better. Hence all are used in the subsequent discriminant analysis. The variables that will be used in the discriminant analysis are shown in Table 6. These variables were entered into a stewise discriminant analysis using a 24 ste rocess with the results shown in Table 7. Table 7 shows that 49è of the variance è1, çè can be exlained by the 20 variables in the Canonical DiscriminantFunction following the analysis. A more signiæcant result is demonstrated in Table 8 where we ænd that the function comosed of the 20 variables in Table 7 is able to correctly classify 86è of the cases collected in this study. As a ractical reænement, the discriminant analysis was rerun over the same variables, but only allowing the ærst 10 variables of Table 7 to enter the Canonical DiscriminantFunction. This was an attemt to create a more tractable redictor function which can be more readily used in ractice. The full discriminant analysis in Table 7 shows that after the ærst 13 variables entered the discriminant function, subsequent variables exlained less than 1è of the remaining variance. Thus we ænd a ractical cutoæ for additional variables. Further, the eleventh and thirteenth variables entered èuses Prototyes and Maintains ëbig icture" view cometenciesè were subsequently removed from analysis. This is an indication that the eleventh, twelfth, and thirteenth variables are not imortant to retain for further analysis. Hence the analysis included only the ærst 10 variables in the Canonical Discriminant Function. Table 9 gives the classiæcation results for the reduced case of 10 variables. The function of 10 variables is able to correctly classify over 81è of the cases collected in this study. The ten variable function is nearly as eæective as the full twenty variable function in classifying the excetional and non-excetional cases. The total variance exlained by these ten variables is 41è. The Hels Others cometency exlains 16è of the variance in the samle. The Total Years Software Exerience variable exlains another 8è of the variance. The Bias for Action cometency exlains 4è of the samle variance. Each of the seven remaining variables exlains less than 3è of the variance of the samle. In the ten variable function, seven of the variables are cometencies; the remaining three are biograhical variables. Four of the cometencies in the function were found as diæerential using the t-test. Thus, three of the cometencies are diæerential in the ten variable Canonical Discriminant Function, but are not diæerential using the t-test. These three diæerential cometencies and the behavior andèor attitudes of engineers that exhibit each cometency are brieæy described as follows: Two of these diæerential cometencies have a higher mean for excetional erformers: 18

19 Action Vars Wilks' Cometency ècè or Ste Add Delete In Lambda Sig. Biograhical Variable èbè Hels others ècè Total years Software exerience èbè Driven by bias for actionèurgency ècè Total languages used rofessionally èbè Willingness to confront others ècè Exhibits and articulates strong convictions ècè Perseverance ècè Driven by sense of mission ècè Resonds to schedule ressure ècè Math Degree Held? èbè Uses rototyes to assess design ècè Schedules and estimates well ècè Maintains ëbig icture view" ècè Uses structured techniques for communication ècè Team oriented ècè Engineering degree held? èbè Takes ride in quality and roductivity ècè Total training hours ècè Uses code reading ècè Focuses on user or customer needs ècè Maintains ëbig icture view" ècè Uses rototyes to assess design ècè Writesèautomates tests in arallel ècè Uses methodical roblem solving aroaches ècè Table 7: Full Discriminant Analysis Summary Table Predicted Grou No. of Membershi Actual Grou Cases 0 1 Grou Non-excetional 88.0è 12.0è Grou Excetional 17.5è 82.5è è of ëgroued" cases correctly classiæed: 86.18è 125 Cases were rocessed. 0 Cases were excluded for missing or out-of-range grou codes. 2 Cases had at least one missing discriminating variable 123 Cases were used for rinted outut Table 8: Full Discriminant Analysis Classiæcation Results 19

20 Predicted Grou No. of Membershi Actual Grou Cases 0 1 Grou Non-excetional 81.9è 18.1è Grou Excetional 20.0è 80.0è è of ëgroued" cases correctly classiæed: 81.3è 125 Cases were rocessed. 0 Cases were excluded for missing or out-of-range grou codes. 2 Cases had at least one missing discriminating variable 123 Cases were used for rinted outut Table 9: Limited 10 Variable Discriminant Analysis Classiæcation Results 1. Desire to DoèBias for Action: driven by a bias for action and sense of urgency in comleting assignments. When faced with a tough roblem, these engineers do not hesitate to get started and develo the required caabilities as they go; they are results oriented and want tomake rogress on a regular basis; they ush themselves to achieve results quickly. 2. Sense of Mission: driven by a sense of mission and clearly articulate goals to achieve a seciæc result. These engineers create and articulate clear and seciæc goal statements; they drive the roject to achieve seciæc goals. One of these diæerential cometencies has a higher mean for non-excetional erformers: 1. Perseverance: methodical, organized, and cautious in their work. These engineers make sure that all aths are covered in their design and roblem solving; they work slowly and carefully to avoid making mistakes. 4 Discussion We conclude from our evaluation of Phase 2 results that exerience is indeed a signiæcant redictor of erformance. This is articularly true when the exerience is in software engineering and the exerience is received at the comany where a subject still works. It seems that either comanies reward the exerience at their own comany more, or the exerience at the comany is more relevant to the tasks of that comany. The exerience variable by itself is not a satisfying redictor of erformance. Exerience alone is only able to correctly classify 63è of the 123 ècomleteè cases from this study. Two other biograhical variables enter into the ten variable Canonical Discriminant Function, Total Languages Used Professionally èwhich might be considered a ëbreadth of exerience" variableè, and Math Degree Held?, with both variables associated with excetional erformance. However, the cometencies are of major imortance in classifying the engineers using either the Canonical Discriminant Function or the t-test. The cometencies can be organized into four categories, Task Accomlishment, Personal Attributes, Situational Skills, and Interersonal Skills as shown in Table 10. The cometencies in 20