Splitting the Organization and Integrating the Code: Conway s Law Revisited

Transcription

1 Spitting the Organization and Integrating the Code: Conway s Law Revisited James D. Herbseb and Rebecca E. Grinter Be Laboratories, Lucent Technoogies 263 Shuman Bvd Napervie, Iinois USA {herbseb, beki}@research.be-abs.com ABSTRACT It is widey acknowedged that coordination of arge scae software deveopment is an extremey difficut and persistent probem. Since the structure of the code mirrors the structure of the organization, one might expect that spitting the organization across time zones, cutures, and (natura) anguages woud make it difficut to assembe the components. This paper presents a case study of what indeed turned out to be the most difficut part of a geographicay distributed software project, i.e., integration. Coordination probems were greaty exaggerated across sites, argey because of the breakdown of informa communication channes. The resuts impy that muti-site deveopment can benefit to some extent ti-om stabe pans, processes, and specifications. The inherenty unpredictabe aspects of projects, however, require communication channes that can be invoked spontaneousy, by deveopers, as needed. These resuts shed ight on the probems and mechanisms underying the coordination needs of deveopment projects generay, be they co-ocated or distributed. Keywords Coordination, coaboration, quaitative research. systems integration, 1 INTRODUCTION Geographicay distributed software deveopment has become a business necessity for many goba corporations. This imperative is being driven by the need to ocate resources in a country for marketing purposes, the acquisition of foreign divisions in mergers and buy-outs, and the promise of goba round the cock deveopment. Despite the necessity, and perhaps even desirabiity cf geographicay distributed deveopment, it is extremey difficut to do successfuy (see, e.g., [IO]). Unanticipated coordination breakdowns appear fi-equenty to ead to deay, inefficiency, and frustration. Permission to make digita or hard copies of a or part ofthis work thr persona or cassroom USC is granted without fee provided that topics are not made or distributed for profit or commercia advantage and that copies bear this notice and the fu citation on the first page. To copy otherwise, to repubish, to post on servers or to redistribute to ists. requires prior specific permission and/or a fee. ICSE 99 Los Angees CA Copyright ACM 1999 I /99/05...$5.00 In this paper we report research that begins buiding a better understanding of why geographicay distributed, deveopment is so difficut to coordinate. Specificay, we present a case study of the integration phase of a geographicay distributed deveopment effort. By integration we mean a the work that is necessary to assembe the product from its components. We were pointed towards integration as a topic by an initia series of interviews that focused on occasions where the muti-site character of the project was most disruptive. The next section reviews the iterature on coordination in software deveopment and the various kinds of mechanisms that serve to maintain coordination. Section 3 describes the sites invoved and our empirica research methods. Our resuts are presented in section 4, and our concusions in section 5. 2 COORDINATION, PREDICTION, AND COMMUNICATION The coordination of sofsvare deveopment work has been a focus of attention within research for a ong time. One of the frst peope to suggest its significance was Mevin Conway [3]. Specificay, what Conway said was that the structure of the system mirrors the structure of the organization that designed it. Conway s Law - as it has become known - was the first expicit recognition that the communications patterns of an organization eft an indeibe mark upon the product buit. Pamas went on to carify how the reationship between organization and product occurs within s0thm-e deveopment. His definition of a modue as a responsibiity assignment rather than a subprogram ceary shows that dividing a software system is simutaneousy a division of abor [ 121. It is the division of abor, among different individuas, that creates the need for them to coordinate, to aign their efforts, in the production of software. Both Pamas and Conway focused on the structure of the product, which provides an important foundation for the coordination of work. Product structure addresses the question of what is to be deveoped by individuas or sma groups. This, however, is ony one of severa dimensions on which deveopment projects must be coordinated. A project must aso, for exampe, determine how the product 85

2 is to be deveoped (i.e., the deveopment process). Processes can aso determine work assignments, as when an intermediate product is handed off between two groups, e.g., for different stages of design, coding, or testing. Again, we-defined boundaries, in this case between process steps, are critica in breaking up work so that it can be assigned to different groups. In addition to what is deveoped, and how the work is done, when various project miestones are to be achieved (usuay aid out in a pan), and who wi do the work (often contained in a stafing profie) are aso critica to project coordination. Projects can strugge bady or fai atogether if these other critica dimensions of project coordination are not attended to. Around the same time that Parnas and Conway were describing how coordination was fundamentay part of software deveopment work, Brooks was aready noticing how difficut it was in practice. Brooks Law says that if a project is ate then adding more peope to the deveopment wi sow the work down further [2]. Brooks argues that one of the reasons this aw hods is that the addition of more peope creates communications overhead, because af the project coordination required. A few years ater, Curtis [4] documented the fact that communication and coordination was one of the most difficut and pervasive probems in arge-scae software engineering. The chaenge of coordinating deveopment projects has not gone unnoticed by the software deveopment researchers and practitioners who have primariy worked on soutions that provide expicit and visibe mechanisms for coordination. These soutions incude activities such as panning, defining and foowing a process, carefuy managing requirements and design specifications, measuring process characteristics, reguar status meetings to track progress, impementing a workfow system, and so on. They are generay imposed by management, athough they require cooperation from everyone to succeed. The software industry has increasingy recognized the importance of these sorts of practices, as evidenced by such things as the increasing adoption of the Capabiity Maturity Mode for Software [6, 131 and increasing attention to software processes [ 1, I. These kinds of soutions faciitate the coordination of deveopment by providing a shared understanding of what the purpose, process and desired outcomes are. They give a the members of the team a common direction. However, pans and processes are most usefu ony to the imits of one s abiity to anticipate. One can certainy pan for various risks and contingencies, and can have dieerent favors of the process for different circumstances. But there wi aways remain many decisions that cannot be made ahead of time, unanticipated probems, detais to be fied in, mistakes to be corrected and recovered from [ 18, 193. The abiity to predict wi neary aways be much better deveoping the nth version of a product than deveoping the first version, but predictabiity wi aways have its imits, and various coordination-preserving adjustments wi aways be needed. For this reason, a number of vita eements of software deveopment work such as the exercise of individua ski, habitua but unrecorded patterns of human activity, creative handing of unanticipated conditions and events, and the use of informa persona networks, are hard to represent expicity in pans and processes [9, 161. Work that is unpredictabe to some degree, and hard to represent expicity in pans and processes, is impossibe to avoid in software deveopment. The usua and perhaps most effective approach to deaing with the imits of predictabiity is to avoid over-engineering a process description or a pan. Beyond a certain eve of detai, those doing the work must be trusted to have the skis to do it appropriatey, handing exceptions and coordination issues as needed. Interference from pans and processes that anticipate incorrecty is thereby argey avoided. Yet these essentia outside-the-pan actions are a serious potentia threat to project coordination for wo reasons. First, they often need to take account of others,actions, so as not to interfere with them. Second, they need to be communicated to everyone potentiay tiected, in order to curtai rippe effects. Previous, empirica studies of the coordination of software deveopment suggest that deveopers use informa communication channes [4, 5, 7, 14, 151 to address this threat. Informa communications channes are outside the officia reporting smcture of a project. They are simpy deveopers access to other deveopers, managers, testers, and anyone ese they need to interact with during the deveopment process. Unike the expicit mechanisms described above, they are usuay invoked by those doing the work; without requiring management authorization, and perhaps without management s knowedge. These channes hep deveopers fi in the detais of work, hande exceptions, correct mistakes and bad predictions, and over time mange the rippe effects of previous decisions and actions. Using informa communications channes, or even unofficia roes such as boundary-spanner [4] is a critica compement to expicit coordination mechanisms. These communications channes have not received the same kind of attention in software engineering as the more visibe coordination mechanisms have. Nevertheess, previous research in other areas has shown their critica importance (e.g., [20]). Changes in the way work is done that disrupt these informa communications channes often have disastrous effects (e.g., [17]). What appear to be merey casua conversations around the water cooer often serve to informay exchange the kinds of information and experience that are critica to project coordination. Since communication between distant sites is rnore difficut than communication within a singe site, this anaysis suggests that mutisite deveopment wi disrupt informa communication channes to a greater or esser extent. We woud expect apses in coordination to become visibe at the point where the products of ongoing work am finay 86

3 brought together, or recomposed [5]. In order to better understand the oss of coordination in muti-site deveopment, we conducted a quaitative study of the integration process during the deveopment of a arge, software-intensive teecommunications product. The site and the empirica methods used are described in the next sections. 3 SITES AND METHODS In this section we describe the sites of study, incuding some background on the products buit. We aso discuss how the work is divided among sites. We concude with a description of the methods used to anayze and coect the data. Site Geographicay distributed software deveopment is pervasive among most arge companies, incuding Lucent Technoogies. We chose one division of the company for this study, which is part of a arger project to examine a aspects of coordinating muti- site deveopment work. We are studying this department for three reasons. First, the department was wiing to host researchers and provide us with access to deveopers, documents and other resources. Second, they work in an area of teephony that is both technicay compex and growing rapidy in market demand. The product that they buid is a rea-time system, and the software contros embedded hardware eements, making the design and deveopment work chaenging. In addition this product competes in an aggressive market which brings its own pressures to deveopment work. We fee that if we can better support this kind of work, we wi find insights into other compex domains. Finay, the department engages a number of cross-site coaborations, both within the deveopment of their product as we with other divisions of the corporation, and other companies. We were interested in exporing the varieties of coordination required in the deveopment process. In this study we focus on two of those ocations, one in the UK and one in Germany, where the department does a arge share of its deveopment work. These two sites exchange information frequenty and make decisions that require cross-site synchronization. The German site had existed for a number of years, and the peope there had considerabe experience working together on simiar systems. However, it had not previousy participated in cross-site deveopment where the architecture is spit. The UK site was new, with no existing reationships to any other Lucent site. The department aso has interactions with other divisions of the company because the product must interact with other technoogies. Many of these technoogies are buit in the United States so the deveopers coordinate work with these other sites. These US sites had not previousy worked together, nor had they worked before with the UK or German sites. In a cases, the coaborations span different anguages, cutures, and many time zones, making them more difficut. Methods The study began with an initia set of 10 interviews with managers and technica eads. The purpose of these interviews was to gather information about what peope in the department fet the chaenges of mutipe site deveopment are. They identified integration as one area where they had experienced difftcuties, so we conducted a second round of 8 interviews where we focused on those probems expicity. Data anaysis foowed quaitative protocos [8]. We transcribed a the interviews and then reviewed the materias for specific events within the integration activities. We then ooked for causes, and outcomes, as part of buiding up a rigorous expanation of what happened during integration. In addition to the primary interview materia we had a number of secondary sources avaiabe to us, that heped us to earn about the probems and verify some of the information we gathered. Specificay, we reviewed documents reated to the deveopment process. We were aso given access to a retrospective of the deveopment process that the sites carried out at the end of their fmt reease. These documents and other archiva sources heped us to earn about the deveopment context in which the deveopers found themseves working on cross-site integration. 4 RESULTS 4.1 Limits of expicit coordination mechanisms The means of coordination that figured most prominenty in the integration phase of this project were the integration pan, 0 component interface specifications, software processes, and documentation. In this section, we expore how these were used to coordinate work, and how the imitations of these mechanisms were exposed in this project. 4. I. I The Integration Pan As with any deveopment effort, a pan for integrating the system was devised. In this case it consisted of 40 steps, that woud bring eements of the overa product together. The initia pan was not cosey foowed because it was based on severa assumptions and predictions that turned out to be in error. Dependence on the overa deveopment pan. For the integration pan to succeed, the components to be integrated had to be avaiabe at the right time. The project suffered from many of the usua difftcuties of panning software deveopment, such as changing requirements, staff turnover, and extreme schedue pressure. Add to this the virtua impossibiity of predicting the effort and timing of a new product being deveoped in a new organization, and it is no surprise that the components were not ready for integration on the schedue described in the pan. The 87

4 origina pan turned out, in retrospect, to be very optimistic. As the deveopers strove to adjust to the reaities of the project, they took an approach one deveoper described in this way, We chopped and changed as things became ready. Deveopers reported that the pan changed weeky. As the project progressed, the deveopers augmented the documentation to hep them dea with the unpredictabiity, keeping very detaied records, for exampe, of exacty what steps they had taken, and exacty what fies went into each buid so that they coud quicky back them out if something went wrong. Substrate environment. Some deveopers concuded, in retrospect, that the pan missed a critica pre-step, i.e., buiding the substrate environment on which the product sits. This product reies on a number of substrate software and hardware technoogies, buit within other divisions of the corporation, and by other companies. Making the product work invoved ensuring that it ran on top of these substrate technoogies, and so prior to testing the software, the environment had to be assembed. As it turned out, the pan did not adequatey take account of the difficuty of this. In addition to sowing down the testing whie some deveopers constructed the environment, it further compromised the integration effort because none of the deveopers were sufftcienty famiiar with these substrate technoogies and had not given sufficient attention to aigning their efforts with them during deveopment. Perhaps the biggest uncounted for chaenge of assembing the substrate environment was the new reationships that the deveopment team had to forge. In their own work the deveopment group spans two primary sites. The deveopment of the substrate invoved working with a the usua partners but aso required the deveopers to get parts from the United States. These substrate technoogies took time and energy to assembe, and often probems were very subte and difficut to spot remotey. This added further time onto the integration process Interface Specifications Specification of interfaces is a critica part of any deveopment effort, especiay when it is spit among a number of deveopment teams. This project used a contemporary soution to support the specification of their interfaces, OR131Xm (based on CORBA). Interfaces were specified primariy by event tracing, or fence diagrams that showed sequences of messages among processes. The use of ORBX/OMT aowed each deveopment group to deveop their own part of the product, reassured that the agreements among the different interfaces hed. In the process of buiding their pieces the deveopers aso buit simuators to represent the other components that their code woud need to interact with. Even before integration got under way, it became cear that the interface specifications Ty OR3IX is a trademark of IONA Technoogies. acked many essentia detais, such as message type, return types, and assumptions about performance. In many cases, the incompete specifications aowed the deveopers to proceed with different assumptions about what the other components were doing, and it was not unti the initia attempts to make the pieces work together that these aternate assumptions were exposed. 13y etting deveopment groups write simuators to represent others code the discrepancies among assumptions had remained hidden during unit testing. Not a conficts were hidden unti integration. In many other cases, deveopers reaized that conficts among assumptions were ikey to occur, and they contacted deveopers responsibe for components with which they shared an interface, and worked out informa re:fmements to the specification. It is not at a cear that this sort of incrementa refmement coud be eiminated, since the refmements were often based on knowedge and understanding that came from the design work itsef. The informa refinements were sometimes, but not usuay, recorded in the documentation. Ony be examining the code coud one infer the information. This caused difficuties on a number of occasions, particuary when the origina deveoper eft and a new deveoper, unaware of the informa specification refinement, took over. It was aso very probematic in the test phase, when many bug reports were generated by tests that vioated thke informa agreement. One deveoper, for exampe, reported that there was an informa agreement that, for performance reasons, another component woud verify a data it sent to his component. The testers, however, not knowing about this agreement submitted a series of bug reports based on tests which sent the component bad data. Much time was wasted with this and simiar probems with undocumented interface refinements. The deveopers aso had to manage interfaces among the product and its substrate technoogies. Differences in assumptions about what the product wanted and what the substrate coud provide persisted in part beciause no-one was using the rea substrate in their deveopment work. When integration surfaced these differences the deveopers faced additiona chaenges of finding the right peope to contact who were organizationay and physicay remote. In many cases probems were soved by hosting a substrate deveoper on-site for a period of time. This not ony resoved probems during their stay, but gave the product deveopers vita contacts into these other divisions Process The deveopers used a number of processes in their work to hep organize and structure the deveopment environment. These processes evoved as the project did. However, during integration a number of weaknesses with the processes were uncovered. One response to the mutipe site probem wabs to isoate each site from another in certain ways. For exampe, the architecture of the product was divided so that the 88

5 components did not cross sites, each group worked on separate pieces of the product. This separation aowed two change management processes to evove independenty, one for each site. The advantages of working this way were obvious initiay, changes got into oca buids quicky, and during these initia phases deveopers at both sites got timey information about their code. Given the conditions that the project faced, for some time this was a sensibe operating strategy to aow the rapid deveopment of the product. It did not come without costs though. When it came time to finay integrate both sites work, the processes in pace created additiona effort for those responsibe. For exampe, changes coud be found at either sites and ogged into both change management systems, fixed, and with a sequence where the code was merged together in between that woud mean the same probem was futed twice usuay creating new bugs. To prevent this, changes needed to be ogged at both sites, and then the responsibiity assigned to one person. However, the numbers for the same change varied between the sites, so it became incumbent on the integration team to know both numbers for the same change. Finay, over time the processes for buiding the product aso became different, so a buid that worked at one of the sites, coudn t be made to compie at the other. The extent of these compications certainy was not foreseen, nor perhaps coud have been foreseen, when the parae databases were set up. The obvious soution to this probem was to consoidate the process at one site, which happened. However, this in turn ed to a series of chaenges. Especiay difficut was getting timey feedback about the resuts of the buid at the other site. This was resoved in part by sending deveopers from the remote site to the integration site. However, whenever deveopers from the remote site came to the integration site, they ost their abiity to work in their own deveopment environment, which remained back at the od site. So the remote deveopers faced a choice, go to the centra site and find probems, or stay remote and fix them. This sowed down the deveopment effort quite seriousy. Change Contro Board. A change contro board (CCB) is another mechanism for controing the changes made to software. Their roe is to examine each change request that comes in and decide whether it shoud be futed, review the probem to ensure its vaidity and sometimes to assign a person to work on the revision. The organization under study had a oca CCB at each site which coordinated with a project-eve CCB. The discussion in this subsection concerns ony the project-eve board. Initiay, the CCB was amost excusivey managed at one of the two sites. This meant that the members of the CCB were famiiar with their code, but coectivey knew much ess about the work at the other site. In fact, it was hard to know anything about the remote sites software in advance. The members knew about their software because it had existed as a product before this particuar deveopment effort. The software at the other site was competey new. So at first, the CCB did not need to span the sites, and their ack of knowedge was not an issue because there was nothing to know. Over time as the site began to buid their software and the product evoved from a one-site egacy system to a mutipe site revision, the ack of knowedge became a probem. When change requests were made for the new code, no one on the board had enough famiiarity to make decisions about how best to proceed with the change. Furthermore they coud not predict how changes in the code that they did know woud impact the software they did not understand. So the deveopers of the new code found that they got hit with probems stemming ti-om changes made to other pieces of software that probaby woud not have been impemented in that way if anyone had understood their code better. Since the probem had a gradua onset, the CCB was for a considerabe time unaware of the extent of the probems their decisions were causing. The soution to the probem was straightforward. An architect from the other site was added to the CCB. He was abe to bring a broad and deep knowedge of the design of code deveoped at the other site to bear on CCB decisions, and the probems were argey aeviated. Sharing and evoving processes. Another process chaenge invoved evoving practices that both groups of deveopers shared coectivey. One case was a system that one site devised for debugging their code. They used a series of numbers that represented different kinds of probems in the code, so that when the system broke the deveopers understood why. However, to the other site of deveopers this system of numbers was as incomprehensibe as the bugs in the code. The process was not understood at a, and it took considerabe time to become famiiar with how the numbers worked and what they were teing the deveopers about the state of the code. 4. I. 4 Documentation Documenting the software process is something that a number of researchers continue to argue for. The rationae in this case is that reiabe and accurate documentation can hep the process by supporting access to accurate and reiabe information. This project experienced a number of chaenges in documenting their process and decisions rationay though. First, the technoogies that were supposedy there to hep with documentation did not meet expectations. Specificay, the technoogies to support the deveopment of code from design documents coud not hande the compexities of the appication domain. Eventuay this probem was soved by abandoning the toos, and thus the first break between the designs and the code was taken. From this point on deveopers woud have to manuay update the design documentation as we as their code to refect their changes. Under time pressure to buid the system, the deveopers proceeded with coding, and sowy the code diverged fi-om 89

6 design. However other peope were sti reying on that documentation to design their own components and test suites. Over time, and especiay in integration, a these inconsistencies came to ight. Testers pointed to documentation as demonstration of why code was faiing certain tests, other deveopers pointed to documents that described behavior that they had assumed was sti exhibited by the code. A of these inconsistencies had to be resoved as part of buiding a working product. This was further compounded by a shift in the roe of documentation. In the beginning documents had been focused on recording the design of the system. In ater phases of the product documentation became oriented around the change management process. So instead of having descriptions of the components and their behavior, the documentation was organized around the changes and updates to the system. This kind of documentation does not revea the overa architecture and behavior of a component et aone the product easiy. The deveopers were eti with one accurate source of documentation -- the code itsef. And as the origina designers of the system eft, the soiware itsef became increasingy the source of their system understanding. Yet, omissions in documentation, which can simpy arise from a careess mistake, or because it is amost impossibe to competey describe a system, can create serious probems in deveopment. This project faced one such probem with some documentation from the one of the substrate technoogies provider. In this case the omission invoved ensuring that header fies contained certain nonobvious detais. As this information was not incuded in the documentation, the headers were not correct, and it took weeks to figure out that this was the probem, simpy because it was something that woud never have occurred to anyone to think of in absence of the documentation Summary In hindsight it seems easy to see that any of these things might have been probems that coud have been resoved with better pans. This is not however the case. The essence of the difficuties with these pans, specifications, processes, and documentation, is the fundamenta chaenge of predicting the highy variabe process of sotware deveopment. Requirements change, other reated deveopment efforts proceed and impact the work that you do, systems seem to provide support but present subte chaenges to work with, and so forth. A projects manage these variabiities as part of day to day software deveopment work by ad hoc communication, informa agreements, testing assumptions, and so on. As we argued in section 2, none of these coordination mechanisms can work without aowing for the fiing in of detais, handing exceptions, coping with unforeseen events, and recovering from errors. This section has presented a number of exampes of how pans, specifications, and processes were modified or augmented in order to aow the work to proceed. What perhaps makes this project more chaenging than others is the fact that mutipe sites have to notentia to disrunt the conditions necessarv for these adjustment techniques. This is the topic of the next section. 4.2 Barriers to informa communication It was cear from the interviews that spitting the organization across sites presented barriers to informa communication during the project, causing serious probems. The primary barriers which ed to coordination breakdowns were Lack of unpanned contact, Knowing who to contact about what, Cost of initiating contact, Abiity to communicate effectivey, and Lack of trust, or wiingness to communicate openy. Our findings in each of these areas are discussed in the foowing sections I Lack of unpanned contact In a typica co-ocated deveopment effort, project members run into each other in the haway, at the coffee machine, in the cafeteria, and esewhere, on a frequent basis. They discuss many things, incuding many that have nothing to do with project work. But they aso bring up the project as a topic of discussion, especiay things that are currenty on their minds, or are causing them some concern. This is not necessariy expicity to get hep, or to formay notify others of specific events, but rather just the usua sort of friendy exchange, common in the workpace. These sorts of unpanned contacts seem to be surprisingy important in keeping projects coordinated. For exampe, one interviewee tod us of an incident in which, during the course of such an unpanned discussion, it came to ight that he and a co-worker were making contradictory assumptions about what board woud have a particuar digita signa processing chip. They were abe to resove the issue in a matter of a few minutes. But had they not discovered this difference in assumptions as eary as they did, this minor probem coud have become extremey costy. What makes this and simiar incidents significant is that the participants were not aware of any need to communicate. There was absoutey no reason either one knew of, even if communication coud be initiated very easiy, to contact the other person to discuss the project. Many conficts in assumptions foow this pattern, because project members are ofen unaware of the assumptions they are making, or that others might be making conficting assumptions. This form of coordination is predicated on reativey frequent unpanned communication: during the course of which reevant information and important issues may come to ight. In our interviews, there were a number of probems that seem to be the resut of a genera ack of information. Deveopers at one site, for exampe, had deve.oped a very handy step tracing too that providing intorrnation on 90

7 memory usage, CPU usage, and so on. Whie deveopers at the same site used the too extensivey, those at the other site did not know of the too s existence for months. It often took weeks to dea with probems that coud have been soved very quicky with the too. The fact that deveopers at one site were aware of the too and those at the other site were not, makes it seem quite ikey that unpanned contacts payed a significant roe in disseminating awareness of the too. Overa, severa consequences seemed to fow t?om the reative ack of unpanned contact. One was to make it much ess ikey for conficts and issues to be recognized. If a deveoper is aware that an issue exists, it is possibe to take action to correct it. But since unpanned contact seems to be one of the primary mechanisms for bringing issues to ight, many more conficts went unrecognized unti ater in the deveopment. Another consequence was just the genera ack of background information across sites, i.e., how they work, what issues are most pressing to them, how they typicay communicate with each other, site-specific vocabuary, and the responsibiities, expertise, and reationships among those at the other site Knowing who to contact about what Deveopers often reported great difficuty in determining the appropriate person to contact at the other site. If there was a need to coordinate on an interface specification, fm exampe, there was no straightforward way to find out who was responsibe for the component on the other site. Deveopers found severa workarounds for this probem, athough none was entirey satisfactory. One was to ook for cues buried in the documentation, such as names at the bottom of web pages containing specifications. Often those whose names were isted as authors were the correct person, or coud point to the correct person. Another frequenty used workaround was to contact a system architect at the other site. Architects were known to have a very broad knowedge of the system and who was doing what. Once some of the deveopers had spent a significant amount of time at the other site, these individuas became contact peope or iaisons. A visitor from the UK, for exampe, woud often be used by those at Germany to hep them to figure out who they shoud get hod of. When these peope returned to their own sites, they aso often acted as the first point of contact to peope at the other site. In addition, peope at their own site regarded them as something of an expert on the other ocation, and woud often come to them with questions about who was doing what, and about how things worked at the other site. This, of course, imposed a significant cost on the iaisons, particuary in the earier days when there were very few peope with cross-site experience Cost of initiating contact. When deveopers are co-ocated, contact can generay be initiated quite easiy. There are many cues about who is around, how avaiabe they are (e.g., is the door open?), and so on. If someone s office is ony a few feet away, one need expend itte effort to stro down the ha for a chat. Perhaps more significanty, it is sociay comfortabe to do so, since you know them we, know how to approach them, and have a good sense of how important your question is reative to what they seem to be doing at the moment. For deveopers at d&rent ocations, the cost of initiating contact was often much higher. Who is avaiabe. One difftcuty is simpy determining if someone is avaiabe. If they do not, e.g., answer the phone, they coud be momentariy tied up, or in the midst of a crisis, or it coud be a hoiday, or they coud be on a vacation (of many weeks). Unike the same-site case, it is not easy to determine if the person being caed is ikey to be avaiabe. Our interviewees reported it often took many attempts, over many days and often invoving severa peope, to contact someone at the remote site. In the mean time, progress was often hed up. Time dfirence. There is ony one hour time d&ence between the two sites, so one woud not think that woud make much difference. But this can be very deceptive. There is an hour ost at the beginning (or end) of each day, but since typica unch time was dispaced by an hour, that meant that another hour for each site was unavaiabe. Add to that the fact that deveopers at the German site tended to start earier and finish earier in the day, and an additiona hour or two of potentia time overap was ost. These time differences meant that something that coud be handed in a matter of minutes for a same-site deveopment woud often have to wait at east unti the next business day. Reduced responsiveness. Peope who know each other reativey we, and know that they wi be deaing with each other frequenty in the future tend to be more responsive to each other s needs and requests. Individuas at different sites often seemed reativey unresponsive, e.g., not answering e-mai or voice mai prompty. This again makes it much more costy to initiate contact, since a singe message is ess ikey to be effective. Severa of our interviewees beieved that it was difftcut to accuratey gage the importance of a message from another site. Often, they did not understand the context we enough to determine why the question was being asked, or to see why it was an important request and not merey arbitrary or irreevant. Consequences: Three consequences of the high cost of initiating contact were mentioned. First, deveopers did not try to communicate as frequenty as they woud have had they been co-ocated. They were more incined to take the risk that significant coordination issues woud not arise if they did not check assumptions, etc. Peope aso reported that they were not consuted on decisions made at the other site that a&cud them. Second, cyce time was increased. Even when messages were answered prompty, deveopers beieved that resoution was fin more ikey to stretch into the next day. Worse, it often took severa days, rather than minutes or hours, to make the right 91

8 contact. Finay, issues had to be escaated more ofen, if an adequate and timey response was not forthcoming Abiity to communicate efectivey Once the right person has been identified and communication is initiated, information must be conveyed in a reativey compete and undistorted way for communication to effectivey support coordination. What can t be seen. The most obvious obstace to communication is simpy not being abe to see the same things, have access to everything in the environment. This probem took a variety of forms. For exampe, one engthy probem invoved a hardware and software component f?om an interna contractor. It was not behaving propery, and the suppier coud not dupicate the probem, even though they had identica hardware and coud access the software remotey to ensure it was correct. However, they coud not see a fauty firmware chip, and the probem was very difficut to sove. Another exampe iustrates a very di&rent form of this probem, in a deveoper at one site coud not see what a tester at another site was doing. As the deveoper expained it, the documentation essentiay said, eave bank if you want to get a the fauts on the system and this tester was typing in b--a-n-k, the actua word bank into the system and that s it, a ot of probems occurred on that end. The deveoper coud not dupicate error, naturay, and finay, after severa weeks of trying to straighten it out, went to the other site, where the probem was spotted immediatey. Communication technoogies. Face-to-face meetings, panned and unpanned, are the most common form of communication among co-ocated deveopers. When this communication channe is removed or greaty attenuated by geographicay distributed deveopments, deveopers a~ forced to fa back on other communication media which they used with varying degrees of success. Most of the native Engish speaking deveopers found the phone to be a use medium for one-to-one communication. It often required considerabe patience, but they generay found that issues coud be resoved. It was particuary effective for asking very specific questions, ess so for reasoning about hypotheticas. The non-native Engish speakers, on the other hand, seemed to regard these same teephone conversations as much ess effective. As one described it, it s hard to expain something to someone you don t know in your second anguage. They aso found that conversations tiequenty became very emotiona, and took a great dea of time and energy. Some deveopers reported that it was very difficut to get everyone together who was needed in order to sove a probem. If everyone was at the same site, the peope invoved coud gather in an office or conference room and reach a concusion quicky. But when they were distributed, someone generay had to contact peope one at a time, and get bounced around from one to another. Probems such as determining which component generated an error that propagated through a number of other components were very hard to resove this way. Conference cas, on the other hand, i.e., cas invoving more than two peope, and often 6-10 or even more, tended to be ess than satisfactory. They were adequate fbr reativey simpe discussions, and for status meetings, but were thought to be inadequate for contentious issues or for substantia technica discussions. As one deveoper said, every conference ca I waked out of, if I ask somebody what do you understand from it, and they say, I don t know. E-mai was the preferred means of communication for many, especiay the non-native Engish speakers. The advantage of e-mai was that one coud take time to think, to be very care in the anguage one used, and coud do research if necessary before responding. It seemed to take :much of the pressure off non-native speakers communicating in a essfamiiar anguage. They were abe to overcome some of the imitations of this text-ony medium by constncting text diagrams, or simpe diagrams buit from text characters. They aso attached other text documents, such as og fies, to messages. There were a number of difficuties surrounding the distribution and use of various kinds of documents. Distribution was made particuary difficut by,the fact that there were both UNIX and PC patforms in use, and a variety of appications used for editing documents. Even when deveopers used the same patform and appications, there was often a version mismatch so that, as one deveoper reported, they usuay ended up in a secretary s office faxing a document to someone at eveg muti-site meeting. Using documents, or coaborativey viewing fies, presented many probems. Deveopers generay found it very difficut to ook at code together with someone on the teephone. They coud not point to paces in fine code, or scro the other person s screen to a point of interest. DifJerent cutures and anguages. The sites represented different cutures in at east two senses, and both of these infuenced how they interacted. Most obviou&y, the sites are in di%rent countries with distinct nationa cutures. These cutures differ in many subte ways. One that was mentioned very frequenty was the more direct communication stye of the Germans a$ compared to the British. A German interviewee mentioned that Germans are accustomed to caing someone up and irnmediatey saying, e.g., there is a probem with your qode. The British, on the other hand, tend to expect more in the way of a greeting, and more of an indirect poite form for suggesting there might be an error in their code. Such diffinces made it difficut for the Germans and British to work together without confusing or irritating each other. There are aso differences in how Germans and British customariy regard hierarchy and process. Whie this is 92

9 ceary an oversimpification, one might say that the Germans are more comfortabe with a detaied process that specifies the steps in a deveopment fairy competey and precisey. The British, on the other hand, are more comfortabe with processes specified ony at a reativey high eve. Simiary, the Germans have a greater tendency to take hierarchica reationships a bit more seriousy, expecting and receiving a greater degree of direction from managers and supervisors. These di&ences often ed to misunderstandings about, for exampe, whether a deveoper coud simpy take the initiative to change something, or whether a change required a manageria decision or a process exception. Discussions around such issues ofen ed to feeings that the person on the other end was either being obstinate and rue-bound, on the one hand, or a bit of an anarchist and disrupter on the other. This did not hep to foster cooperation. The sites aso represented two distinct engineering cutures. The German technica staff was experienced in rea-time systems and teecommunications, whie the British staff, in addition to being generay younger and ess experienced, tended to be more UNIX-oriented. These differences otten made it difficut to communicate, since they thought about probems differenty and used different vocabuaries. Consequences. One deveoper estimated that any. necessary discussions for a sma change invoving ony one site coud generay be resoved within an hour. The same change, if trivia or neary so, woud probaby take a day if two sites were invoved, and severa days or more if the change was non-trivia. Athough the primary consequence seemed to be cyce time, some deveopers mentioned that the difficuty of communication aso infuenced the way in which they went about modifjing the code. They strove to make absoutey minima changes, regardess of what the best way to make the change woud be. This was because they were so worried about how hard it woud be to repair the probem if they broke the system Wiingness to communicate openy: Trust As the two primary sites began to work together, there was initiay itte trust between peope at different ocations. There was concern about hoding onto work, worries that one site might be cosed down and everything moved to the other. There was aso itte sense that they were partners, cooperating toward the same ends. This manifested itsef in uncharitabe interpretations of behavior, as when, for exampe, someone woud say, we can t make that change, it was oten interpreted as we don t want to make that change, whether it woud benefit the overa project or not. The situation improved consideraby over time, and visits across sites seem to have been pivota. As one deveoper noted, they just did not seem to be abe to make progress unti they had worked together face to face. Primariy because of difficuties encountered during integration, about a haf-dozen deveopers traveed i-om the UK to Germany (where the centra test site was ocated and where the buid was done) for significant periods of time, ofen months. After working together, the reationships between the sites began to change. As one deveoper said, things eased a ot when we met these peope face to face instead of over teephones and e-mai. We worked much coser, and resoved things much quicker as we. The change seems to have arisen from severa sources. For one thing, the differences in communication stye, e.g., the reative directness of the Germans, was seen in context. The British deveopers saw that they spoke to each other this way, and it was not intended to be, nor interpreted as, rude or insuting. In a simiar fashion, they became accustomed to other cutura differences, and were ess mystified or offended by behavior that had seemed strange or out of pace. Other factors eading to these changes seemed to be the perception that Now there s ess of a wa between the UK and Germany, and we can see that they re in basicay the same boat as us. By working together, face to face on a daiy basis, a sense of common goas and purpose was eventuay estabished. Peope began to give more charitabe interpretations of ambiguous behavior. Rather than assuming that disagreements were arbitrary, each side was more ikey to assume that the other side was acting out of a genuine concern about the wefare of the project. Disagreements sti arose, but the context of the discussion was achievement of a common goa. As one deveoper said, I earned to have a ot of patience, and we got through it quite we. Time spent at the other site famiiarized each party with the terminoogy and probem-soving stye of the other. They earned to communicate much more effectivey, and understood the context underying the concerns expressed by peope at the other site. As mentioned above, the visits aso created a number of de facto iaisons who understood the other site very we, and coud act as points of contact for interactions between sites. They coud provide information about who was responsibe for what, how to get things done, coud interpret information that seemed cryptic, and so on. This continues to be a very important roe, but it was especiay so when ony a few peope had spent appreciabe time working face to face with peope at the other site. Consequences. The ack of trust ed to a reuctance to share information, for fear that if work was abe to move between sites, one site might be cosed down. It aso caused very uncharitabe attributions to be made whenever disagreements arose. This initiay caused hard feeings, and probaby introduced considerabe deay in resoving probems that spanned sites, since each side tended to assume the other was just being difficut, rather than trying to understand the concerns behind their position. 5 CONCLUSIONS In this paper we have shown that mutipe site deveopment works against informa communication channes by creating 93

10 geographic boundaries among deveopers who need to work. together. Simpe things ike meeting in haways and knowing who to ask about a probem, are more than simpe peasantries of deveopment, they are a vita part of the coordination of this kind of work. Since designs never exhibit perfect moduarity and are never error-tie, process execution is rarey fawess, and the word is never competey predictabe, informa communication wi be essentia to maintain project coordination. There are, however, some sound ideas on how to keep the need for cross-site communication to a minimum. It was pointed out many years ago [12]that a good design is one in which design decisions about each component can be made in isoation from decisions about other components. As we noted in the introduction, it foows that good organizationa design shoud mirror product structure [3, 121 in order to minimize the need for communication and coordination among groups. Geographic distribution of those groups is just an extreme case where it is even more important to reduce the need for communication, and correspondingy more important to have a good design. As fundamentay sound as this anaysis is, we beieve there is ampe evidence indicating it is necessary to extend it in two major ways in order to account for the compexities of communication and coordination of reaword deveopment projects. First, product structure is ony one of severa domains in which dependencies arise, and hence is ony one of the domains in which coordination is required. When, for exampe, intermediate work products are handed off between groups, it is necessary fa both to have a cear idea of what steps have and have not been carried out at that point. This generay requires a common understanding of a defined deveopment process. Large projects are aso repete with tempora dependencies that have major impications for resource panning and ontime deivery. Good design is a powerfu coordination mechanism, but it is by itsef insufficient. In addition to coordination on the basis of what is being deveoped, it is aso essentia to coordinate when, how, who, and where. Things ike project pans, defined processes, sta%ng profies, and so on, serve as coordination mechanisms in these domains. The second extension was anticipated by Conway (athough it seems to have received very itte attention in the ast 30 years). Even though communication needs am the primary consideration in organizing a design project, Conway adds this criterion creates probems because the need to communicate at any time depends on the system concept in effect at that time [3] (p.31, emphasis added). In other words, the voatiity of the design over the course of the project imits the degree to which it is possibe to optimize the project organization. This astute observation appies to a of the coordination mechanisms, not just product design. To the extent the origina design, pan, deveopment processes, and so on are unstabe, substantia communication between teams and across organizationa boundaries wi be required. This is precisey what geographicay distributed orgrmizations do east we, since, as we have seen, communication is greaty attenuated across sites. Severa essons for muti-site deveopment faow. First, reduce the need for cross-site communication as much as possibe: Attend to Conway s Law: To the extent possibe, assign work to different sites according to the greatest possibe architectura separation in a design that is as moduar as possibe. To the extent possibe, ony spit the deveopment d we-understood products (or parts of products), where pans, processes, and interfaces are estabished and ikey to be very stabe. Instabiity wi greaty increase the need for communication. Second, take a possibe steps to overcome thie barriers to informa communication. Front oad trave, i.e., don t postpone using the trave budget, bring peope who need to communicate together eary on. A other means of communication wi work better once deveopers, testers, and managers have some face-to-face time together. Pan this trave in order to create a poo of iaisons. Give the eary traveers the expicit assignment of meeting peope in a variety of groups at the other site, and earning the overa organizationa structure. Try to send gregarious peope who wi enjo:y this roe. When they return, make it known they can hep with cross-site issues, and free up some of their time to do so. Invest in toos to make it easier to find organizationa information, check avaiabiity of peope, imd to have more effective cross-site meetings, both panned and spontaneous. ACKNOWLEDGMENTS We woud ike to thank a the deveopers who gave their time wiingy to answer our questions about integration. We woud aso ike to thank Dorene Brumme for the timeiness of her hep with this study. REFERENCES [] B. Boehm, Anchoring the Software process, IEEE Softiare, Vo. 13, No. 4, 1996, pp. 73. [2] F.P. Brooks Jr., The Mythica Man-Month, Datamation, Vo. 20, No. 12, 1974, pp [3] M.E. Conway, How Do Committees Invent? Datamation, Vo. 14, No. 4, 1968, pp [4] B. Curtis, H. Krasner and N. Iscoe, A Fied Study cf the Software Design Process for Large Systems, Communications of the ACM, Vo. 31, NO. 11, 1988, pp [5] R.E. Grinter, Recomposition: Putting It A Back Together Again, Proc. ACM Conference on Computer 94

11 Supported Cooperative 1998, pp. Work (CSCW 98), ACM Press, [6] J. Herbseb, D. Zubrow, D. Godenson, W. Hayes and M. Pauk, Software Quaity and the Capabiity Maturity Mode, Communications of the ACM, Vo. 40, No. 6, 1997, pp [7] R.E. Kraut and L.A. Streeter, Coordination in Software Deveopment, Communications of the ACM, Vo. 38, No. 3, 1995, pp [8] M.B. Mies and A.M. Huberman, Quaitative Data Anaysis: An Expanded Sourcebook, Sage Pubications, Inc., Thousand Oaks, Caifornia, [9] R.R. Neson and S.G. Winter, Evoutionary Theory of Economic Change, Harvard University Press, Cambridge, MA, [o] M. O Hara-Devereaux and R. Johansen, Gobawork: Bridging Distance, Cuture, and Time, Jossey-Bass, San Francisco, CA, [ ] L. Osterwei, Software Processes are Processes Too, Proc. 9th Conference on Softiare Engineering, 1986, pp. [12] D.L. Pamas, On the Criteria to be Used in Decomposing Systems into Modues, Communications of the ACM Vo. 15, No. 12, 1972, pp [13] M. Pauk, B. Curtis, M. Chrissis and C. Weber, Capabiity Maturity Mode for Sofmare (Version I. I), Technica Report, CMU/SEI-93-TR-024, Pittsburgh, Software Engineering Institute, Carnegie Meon University, February, [14] D.E. Perry, N.A. Staudenmayer and L.G. Vota, Peope, Organizations, and Process Improvement, IEEE Softiare, Vo. 11, No. 4, 1994, pp [15] J.M. Pickering and R.E. Grinter, Software Engineering and CSCW: A Common Research Ground, Softiare Engineering and Human-Computer Interaction: ICSE 94 Workshop on SE-HCI Joint Research Issues, R.N. Tayor and J. Coutaz ed., Springer-Verag, Heideberg, 1995, pp [16] M. Poanyi, Tacit Dimension, Peter Smith Pubications, [17] P. Sachs, Transforming Work: Coaboration, Learning, and Design, Communications of the ACM, Vo. 38, No. 9, 1995, pp [ 181 K. Schmidt, Of Maps and Scripts: The Status of Forma Constructs in Cooperative Work, Proc. Internationa ACM SIGGROUP Conference on Supporting Group Work GROUP 97, ACM Press, 1997, pp [19] L. Suchman, Pans and Situated Actjons: The Probem of Human-Machine Communication, Cambridge University Press, Cambridge, UK, [20] L.A. Suchman, Office Procedure as Practica Action: Modes of Work and System Design, ACM Transactions on Ofice Information Systems, Vo. 1, No. 4, 1983, pp