Distributed Development With Perforce Software Tony Vinayak Perforce Software
Introduction Not too long ago, the term distributed development did not exist. Every developer working on a project had to be in the same building, often with offices located along the same hall. Managers were of the belief that communication breaks down at about a hundred feet. Today, this kind of centrally located development is rare, even strange. The pervasiveness of broadband Internet access allows developers to work in small offices and even at home. Today s corporate software development teams epitomize the global village: business users in New York present requirements to design team in San Francisco that drives the development team in UK and QA team in India. Add modern day methodologies like Agile Development to that, and you get processes that are not only geographically dispersed but also highly interactive, requiring real-time exchange of digital assets between team members. Distributed software development projects are fast becoming the norm, and coordinating efforts of far flung development teams is shaping up to become one of the biggest challenges faced by enterprise IT today. It is hard to imagine any sizeable development project to be managed without an underlying software configuration management (SCM) tool. This paper outlines the business needs that an SCM tool should be able to fulfill to support distributed development, the common approach adopted by tool vendors over the years, and the innovative architecture implemented by Perforce Software. Requirements for Distributed Software Development Phone calls, emails, instant messages, zip files while they are all essential to promote healthy and timely communication amongst the distributed teams, an ideal SCM system should enable all team members, regardless of their location, to get real-time access to the repository that houses source code and other project-related documents (the depot ). This would enable Joe in San Francisco to immediately see the code changes submitted by Jane in the UK, and avoid duplication of effort or possible conflicts. To support development across multiple locations, the SCM tool should be capable of supporting empirical requirements such as: 1. Common Repository - The depot should provide a consolidated, up-to-the-minute view to all users. This applies both to data (the versioned files) and the meta-data (who has which files checked out, who made what changes, etc.). This is best accomplished by centralizing all data into a single repository, and making it available across the network to all users. 2. Network Tolerance - The connectivity between users and the depot should be network-friendly to overcome any network performance issues. This is essential from a usability standpoint, so that a user with dial-up connection at home is not looking for ways around having to adopt the SCM procedures only because it is going to take too long to check-in or checkout the code. The SCM tool should be able to adapt to possible network latency and bandwidth saturation. These issues can arise even across multiple offices in the same city. Network hiccups shouldn t be disruptive. If for some reason a user s link to the depot is unavailable, it should not impair the user s ability to continue being productive. If there is temporary outage in the network link 2
between New York and San Francisco, the users in San Francisco should still be able to continue working on their own set of files without being bombarded with obtrusive errors because the depot in New York is unavailable. And when the network connectivity is restored, it should be easy to reconcile off-line changes with the depot. 3. Deployment Versatility It is quite typical for a large company to have several development groups that work in heterogeneous technical environments and observe different development methodologies and workflows. After all, one size does not fit all. The SCM tool should lend itself to such varying needs. It should ideally be available on all major hardware and OS platforms, and interoperable with all popular IDEs and third-party tools. Moreover, instead of enforcing its own out-of-the-box development workflows, the tool should adapt to the development team s best practices. SCM Architecture for Distributed Development Over the years, there have been two major schools of thought when it comes to connecting distributed development teams to the depot: Replication: setting up multiple depots at various geographical locations, and keeping them in sync via replication. This allows remote sites to have their own local depot. Central Repository: setting up only one instance of the depot, and all local and remote users connect to it. Replication Approach This has been one of the more popular approaches for making depots available to global audience. It involves setting up multiple code depots across the network, and then keeping them in sync with each other via real-time or scheduled replication 3
Figure 1: Master depot in New York is replicated to depots in other locations Each instance of the depot can serve its own set of local users, and any change made by any user is propagated to all other depots either in real time or in batch mode. While this approach brings repository closer to remote developers, it suffers from several shortcomings: o Real-time replication is an intricate computational process, requiring careful implementation and administration. By its very nature, data synchronization demands uncompromising connectivity between participating nodes on the network. Any vulnerability during transactional updates between the nodes could render data inconsistent, resulting in an inaccurate view of project status for the team members. o Replication solutions are often network bandwidth intensive, resulting in poor overall performance o For each instance of the depot on the network, administrative procedures like backups, security, and master/slave designations are required, resulting in overall higher cost of ownership o Since each depot can work individually, it can easily induce conflicts. For example, if Jane in London wants to modify a critical business function, she has no way of knowing whether someone else in another location, connected to another instance of the repository, is also modifying the exact same file, until the time the two depots reconcile with each other via replication. This could result in duplicate effort, or time lost reconciling changes. This is the inherent pitfall of relying on multiple, replicated repositories, since different locations can and will work independently of each other. 4
Perforce Approach Perforce Software bypasses the intricacies of replication by basing its solution around a centralized depot. This approach provides simple yet powerful architecture to cater to different scenarios of collaborative software development like: Shared development, when local and remote developers need to work together as a cohesive unit with no practical distinction for geography Code sharing, when local and remote developers work as separate teams that need to share their digital assets on an ongoing basis Shared Development With Perforce To support shared development across multiple locations, there is one and only one Perforce Server housing the depot, and several Perforce Proxy Servers (P4P). Proxy servers act as caching agents and are setup local to the remote teams. Figure 2: Shared development: remote locations use Perforce Proxy for sharing development in real-time P4P is completely transparent to users. Jane in London would willingly believe that she is connected directly to the central depot in New York, as she is able to see real-time updates of all other team members. An instance of P4P could be set up to support all the users in the London development group. Every time Jane or one of her co-workers in the London office requests a file from the central repository in New York, chances are that the requested version of the file is already cached locally by P4P, saving on file transmission across the WAN. So Jane gets her file instantly, while reducing overall 5
processing burden on the central Perforce Server. Subsequently, when Jane submits her changes, P4P automatically forwards the modified files to the central depot. All the versioned files and the meta-data are stored in the depot in New York, so that there is only a single point of administration. Typically it only takes a part-time administrator to manage the central Perforce Server. Code Sharing With Perforce Often times, collaborative development implies sharing just the code and not the process. Figure 3: Files can be shared selectively between two totally separate Perforce Depots The development team in San Francisco has subcontracted part of the development to a third party called CodeBlazers. For all practical purposes, CodeBlazers is an external entity, so not only are they not required to work in real-time with the main development team in San Francisco, the corporate security guidelines mandate that they do not have direct access to the main code depot. However, periodically the version of code developed by CodeBlazers needs to be incorporated into the main product by the development team. Instead of having to ship code via email attachments or CDs, Perforce Software provides Remote Depot feature, which automates such code-drops. As shown in Figure 3, CodeBlazers set up their own separate instance of a Perforce Server. To Joe in San Francisco, CodeBlazers depot appears as a remote depot, allowing him to import changes directly into his own depot. There can be a lot of flexibility in identifying the code to be imported, for example: All the changes related to a specific bug fix Just the incremental changes made in a specific period of time A new code branch created by CodeBlazers Any subsequent changes Joe makes to the imported code are not automatically propagated back to the remote depot; that is by design of the remote depot feature. Of course if Joe and CodeBlazers did want to automatically share each others changes they would be characterized as shared development instead of code sharing, and should connect to the same centralized depot as mentioned in the previous section. 6
Benefits of Perforce Architecture Perforce Software has been implemented by thousands of companies around the world. Large companies have successfully deployed Perforce to support their distributed development teams. P4P is easy to implement and administer and yields manifold benefits to the development teams: 1. Real time collaboration By virtue of architecture, Perforce Server and P4P provide real-time access to the central depot to all users, without having to worry about publishing data to other nodes on the network. All users connected directly to the Perforce Server or to P4P have access to the same depot at all times. So if Jane in London wants to checkout a file, she would know right away that it was just checked out by Joe in San Francisco. 2. Network friendly Perforce uses TCP/IP as the communication link between Perforce Server, P4P and Perforce clients. This makes it easily routable across WANs and through firewalls. End-to-end data transmission between Perforce Server and Perforce clients can be compressed to boost performance, which comes to the rescue when working over dial-up connections. Bear in mind though that if the connection between P4P and the central Perforce Server is disrupted, then users connected to P4P will not get access to the metadata. They could still continue working on files stored in their own workspace, and subsequently reconcile changes with Perforce Server when connectivity is restored. 3. Versatile deployment P4P can be deployed on a variety of hardware and OS platforms, and is interoperable with Perforce sever and Perforce clients running on all other supported platforms. It does not impose any inordinate hardware resource requirements. To cater to varying process and workflow requirements of distributed teams around the globe, Perforce Server supports highly customizable features like triggers and notifications, rather than enforcing any specific paradigm. P4P works equally well with binary files, which is good news for the artistically inclined among you editing those image artifacts from the comfort of your lovely home up the hill. In fact, Perforce adapts itself well to several different remote development scenarios such as: At-home developers, who could connect directly to the central Perforce Server (presumably via VPN or SSH). They could also benefit by setting up an instance of P4P on their home network, and doing nightly refresh of a pseudo-workspace resulting in all the necessary files to be pre-cached for the work day ahead. On-the-road users, who could simply connect directly to the central Perforce Server via the available Internet connection. Perforce client/server architecture and TCP/IP protocol lends itself to remote connectivity. Perforce also provides a Browser-based interface (P4Web) for those moments when you are in an Internet café and cannot resist finding out about the latest developments in your project from a public computer. Thin Client for read-only users: P4Web is also a good solution for granting read-only access to internal users who might have occasional browsing interest in the project (e.g. marketing team members wanting to have a look at product documentation) without having to install Perforce client software for them. Distributed (or off-shore) development teams, who could set up their own local instances of P4P and benefit from its caching without having to worry about any 7
additional administrative chores. A desirable side effect of caching is off-loading of processing load from the central Perforce server. Multiple development teams that share code as part of their internal release cycles. This is typically the scenario in large companies where heterogeneous development groups develop software independently of each other and set up their own instances of Perforce Server. By leveraging the Remote Depot feature they are able to discretely and systematically share code with each other. 4. Low Administrative overhead P4P is available on a variety of hardware/os platforms, and requires very little or no ongoing administration. Chores like user administration and backups are required to be performed only at the central Perforce Server. P4P lives a rather silent, transparent life on the network. Conclusion Perforce Software provides a simple yet powerful solution for supporting distributed software development teams. It has been deployed successfully around the world by hundreds of companies to help thousands of local and remote users to collaborate in realtime on an ongoing basis. Perforce Proxy Server is lot simpler to implement and administer than replication-based solutions. Perforce Proxy Server and Remote Depot support is part of Perforce Software suite, and is available at no additional licensing cost. For more information on Perforce Software, please visit our website at http://www.perforce.com. 8