StarTeam Best Practices Getting the most from StarTeam

Transcription

1 Getting the most from StarTeam Randy Guck Chief Scientist, StarTeam Borland Software May, 2008

2 Table of Contents 1. Introduction Welcome to StarTeam! StarTeam Has Grown Up! What s New? Overview Introduction to ALM What is Version Control? Centralized Repository Versioning Focus on Text Files VCS Operations Containers Security What is Software Configuration Management? Introducing the Artifact Branching Merging Sharing and Cheap Copies Change Tracking Development Streams Change Rules What is Application Lifecycle Management? Collaboration Process Improvement Traceability Orchestrating Distributed Teams StarTeam Basics The Big Picture Standard Architecture Overview StarTeamMPX Components Core Concepts Containers: Configurations, Projects, and Views Artifacts Artifact Versioning, Branching, and Merging Items and Sharing Links and Process Items Labels Promotion States Miscellaneous Concepts Guidelines for Deploying StarTeam Why Your Mileage will Vary What is a Large Configuration? Borland Software Page 2 of 90

3 4.3. What Happens When a Configuration Gets Too Large? Deploying Multiple Small Configurations on a Single Machine Deploying Medium Configurations Deploying Large Configurations Clustering for High Availability Supporting Distributed Teams Best Practices for Configuring StarTeam Server Configuration: How Many Do You Need? Consider Business Unit Divisions Leverage StarTeam s Support for Distributed Teams Don t Partition for Internal/External Access Plan for Growth Splitting-up Large Configurations Project Configuration: Defining a Project s Scope Keep Projects Autonomous Example 1: A Simple Client/Server Application Example 2: An Independent Client Module Example 3: A Complex Application Suite Handle Cross-Project File Dependencies Support Cross-Project Activities View Configuration: Understanding View Types and Roles View Types View Roles Main View: Home Base for Artifacts Activity Views: Isolated Team Work Areas Release View: For Post-Release Maintenance Work Sandbox View: For Small Teams or Individuals Build View: Read-Only Windows for Build Scripts View Do s and Don ts Best Practices for Common Processes Change Management within a View Scenario #1: Working in a Small Team Scenario #2: Preventing Merge Conflicts Scenario #3: Using Process Items Effective Use of Labels Scenario #1: Daily Builds Scenario #2: Change Builds Miscellaneous Tips Change Management across Views Overview of VCM When to use each VCM Merge Type Rebase Merges Promote Merges Replicate Merges Borland Software Page 3 of 90

4 Using VCM with a Simple Project Using VCM with Activity and Sandbox Views Using VCM in High Concurrency Projects VCM and Process Items Process Items Should Be Single-Purpose Where Should Process Items Be Created? Simplified Work Status Reporting Administrative Processes Performing Online Backups Managing Security Monitoring Server Performance Tuning Server Performance Maintaining Configuration Health Maintaining a Test Server References Glossary Borland Software Page 4 of 90

5 1. Introduction 1.1. Welcome to StarTeam! Whether you re a StarTeam newcomer or an experienced veteran, welcome! The goal of this document is to bring you tips, techniques, and practices that help you use StarTeam successfully in your environment. The information presented here supplements the standard StarTeam guides and manuals, providing how to advice not found elsewhere. These best practices are cultivated from years of experience helping customers get the most from StarTeam. We hope this information helps your organization as well StarTeam Has Grown Up! Back in 2006, StarTeam celebrated its 10-year anniversary. As most of us that have worked on large software systems know, this is no small feat. Most sophisticated applications succumb to software entropy or simply aren t valuable enough to survive even 10 years. We like to think that StarTeam reached its first decennial birthday because we ve listened to you, our customers, and we ve worked hard to meet your needs. We re not surprised that Joel Spolsky says, Good software takes ten years get used to it. [Spolsky] StarTeam began as a software configuration management (SCM) tool for small teams on Windows a niche back in StarTeam was the first SCM product to combine traditional file version control with defect tracking in an integrated, graphical product. Over the years, StarTeam has expanded on all fronts: features, platforms, performance, and scalability. StarTeam is now used by international companies, allowing thousands of globally distributed users to share a common repository. Enterprises now use StarTeam to integrate application lifecycle management (ALM) activities in truly global spanning organizations What s New? Since 2006, StarTeam has seen some significant changes and improvements to some fundamental aspects of the product. The 2006 release was a watershed release for more reasons than just its coming of age. It is one of the biggest releases in recent years, introducing many important new features such as: Atomic transactions: StarTeam provides a new multi-artifact update capability that supports full ACID 1 transactions. For example, you can check-in multiple files, link them to a change request, mark the change request fixed, and attach the files to new revision label, all in one atomic transaction. View compare/merge: Known as VCM, this new change management facility provides a powerful way to propagate changes within StarTeam. VCM elevates StarTeam change management to a whole new level. 1 ACID properties are the yardstick of database transactions: atomicity, consistency, isolation, and durability. Borland Software Page 5 of 90

6 Integrated visual file differencing and merging: StarTeam provides a new graphical file comparison and merging tool, tightly integrated with StarTeam clients at key touch points to greatly improve file management. Unlimited file size: Building on the highly-scalable Native-II vault introduced with StarTeam 2005, the 2006 release allows individual file revisions to be unlimited in size 2. Combined with StarTeam s equal proficiency at managing text and binary files, this feature opens-up whole new media types that can be managed with StarTeam. New client customization features: The StarTeam cross-platform client (CPC) provides new customization features, allowing each user to see exactly what they want. Client usability improvements: The StarTeam CPC provides numerous new features and usability improvements in managing folders, navigating to referenced items, customizing properties dialogs, and more. Expanded platform support: StarTeam components are now available on more platforms with almost no platform-specific restrictions: the StarTeam server now supports Linux and VMWare ESX; HP-UX is a supported client platform; and virtually all clients are supported on all platforms. And since StarTeam 2006 we have made even more progress up to the latest release of StarTeam Some of the key advancements include: Improved Performance: Newly added MPX Object Caching allows for an even greater performance improvement for accessing and working with non-file items (CR s, Tasks, Requirements and Topics). In addition we improved Borland Search by providing parallel support on the indexing servers, as well as significantly improved the performance of list controls on custom forms (APE s). Improved Usability: It is now possible to compare content within the same view with the VCM (not just between different views/branches). We ve added support for non-derived views within the VCM. We also added a new type of Enhanced Process Links and Process Tasks to the way StarTeam handles process items and automatic link creation. Process Guidance & Support: We ve tried to improve the way the tool provides guidance on process automation by for example creating the ability to prevent the creation of Advanced Views (prevent floating views) and also by allowing users to select cross project process items when doing their work, thus making it easier to separate change-control projects from those where the actual software assets are being stored and managed. Platform updates for StarTeam 2008: The platform support situation is of course always evolving and the latest changes includes Microsoft Vista support (for the Cross Platform Client, SDK, 2 Well, technically files are limited to a maximum of bytes. But, at over 9.2 x bytes (that s 9 exabytes, if you must know), we consider this virtually unlimited. Borland Software Page 6 of 90

7 command line, and bulk check-out utility), 64-bit Oracle DB support and SQL Server 2005 support (for both the StarTeam server and the Datamart). Some platforms were obsolete: Oracle 9 DB, SQL Server 2000, Linux RH 3 (for the CPC) and Solaris 9 (for the CPC). For a complete list of new features for StarTeam 2008, see the What s New topic in the StarTeam Help documentation or the Administering and Using StarTeam manual. Since the release of StarTeam 2006 and up to the very latest StarTeam 2008 release, we have developed some new techniques and practices to pass along. StarTeam now provides whole new ways to accelerate software development Overview This document updates and supersedes the documents StarTeam Configuration Best Practices and StarTeam Performance & Scalability document, both dated December, Best practices are updated to reflect capabilities of the StarTeam 2006 release and new experience garnered from customer successes and even some mistakes since then. This document provides guidelines on how to use StarTeam in several categories: deployment, configuration, short- and long-term operation, and administration. To help you decide which sections you should read, here s a summary of what s in each one: Introduction to ALM: This section explains foundation concepts on version control, software configuration management, and application lifecycle management. If you re new to StarTeam or these concepts, this should help you understand where StarTeam fits in the software management landscape. StarTeam Basics: This section provides an overview of core StarTeam concepts: containers, artifacts, items, links, and so forth. If you re new to StarTeam, this section is a must. Guidelines to Deploying StarTeam: This section is designed to help you plan new StarTeam deployments or evolve existing ones. It covers topics such as hardware configuration alternatives, high availability concepts, and optimizing StarTeam for distributed teams. IT managers and administrators will definitely want to read this section. Best Practices for Configuring StarTeam: This section provides considerations for how you should set-up and organize the primary StarTeam containers : configurations, projects, and views. This information describes the best techniques that other customers have successfully used. Best Practices for Common Processes: This section gets to the heart of the question how should I use StarTeam? We provide our best advice on daily operational topics such as change management, security administration, and backups. For many users, the last section will be the most important. So if you can t wait, start there. For some topics, we summarize material that is discussed in greater depth elsewhere and point to those documents for more details. In other cases, this document covers important topics in depth. Borland Software Page 7 of 90

8 2. Introduction to ALM Application Lifecycle Management (ALM) is an important evolutionary step in the effort to successfully manage software development. ALM is a process in which StarTeam plays a central role. In order to understand what ALM is, this section describes what came before it and why it has become important. Although you can think of version control, software configuration management, and ALM as a capability hierarchy, the lines between them are blurred by that fact that some applications (including StarTeam) provide benefits from each of these layers. Read this section if you re new to these concepts or want to see how StarTeam draws from all of them What is Version Control? When the first developer accidentally wiped out some changes while saving a file, version control was born. Late that night while working under the glow of a teletype, this same user created the world s first version control system (VCS). 3 The need for version control becomes readily apparent to anyone who at some point realizes that an older version of a file is the one they want now. And, when many people work on the same set of files, other VCS requirements become quickly apparent: identification, locking, security, and so forth. At its core, the goal of version control 4 is to store and maintain an accurate history of changes for a set of files. The first version control system is generally agreed to be the Source Code Control System (SCCS), developed in the early 70 s. The popular Revision Control System (RCS) was later developed in the 80 s. It was superseded by the open source VCS known as the Concurrent Versioning System (CVS) and other CVS-compatible or partially-compatible follow-on systems such as CVSNT and Subversion. The features most important to a VCS are summarized below Centralized Repository The main purpose in life of a VCS is to store file revisions. To make them accessible to multiple users, file revisions are typically stored in a repository. The repository may consist of a set of OS-level files, a database, or both. Putting the repository on a centrally-accessible server has many advantages. It can be made accessible throughout the company and even externally with sufficient security controls. Each user can copy exactly the file revisions they want to their workspace. Backing-up and securing the repository is easier when it is centralized. And so forth. 5 3 The credit for the first VCS goes to SCCS, created in Since it was developed for a mainframe, we assume a teletype was involved. For you kids, a teletype is an early computer terminal. Think typewriter with a long roll of paper. No, they don t actually glow in the dark. Now, don t ask what a typewriter is. 4 Version control is also known as revision control, so the term revision control system applies as well. 5 For completeness, we must point out that peer-to-peer networking introduced an alternative to the centralized repository model. In lieu of a central authority, file revisions are copied from workstation to workstation as needed (like free music ). This model has been used successfully with small teams, but it introduces difficult issues in terms of scalability, control, security, backup, and so forth. We won t discuss it further here. And just for the record, we had nothing to do with that free music stuff. Borland Software Page 8 of 90

9 Versioning Since a VCS may have multiple versions of a single, conceptual file such as foo.java, we need a way of distinguishing them. To do this, every new revision added to the repository is assigned a sequential version number. Depending on the VCS, the first revision s version number may be 0 or 1; each modification thereafter increments the version number. Each file is independently numbered. Figure 1 shows a zerobased version sequence. Initial revision (file added) Update revisions (file modified) File: foo.java Version numbers Figure 1: Version numbering An industry convention is to refer to the most recent revision available for a file as the tip revision Focus on Text Files Because they were born from the need to manage source code, VCS s typically focus on text files. This allows them to use deltas to leverage the line-oriented nature of text files to minimize storage space. Nontext files are typically not supported at all or not supported well. (For example, binary files may experience performance problems, size limits, or other issues.) A VCS also does not provide version control for change requests, tasks, and other non-file artifacts. As we will discuss later, we need features from SCM applications to manage non-file artifacts VCS Operations To allow users to interact with the repository in a controlled manner, a VCS provides basic operations such as the following: Add: Add a brand new file to the repository. Since a VCS is aware of folders, you tell the VCS to add the file to the appropriate folder. Get: Fetch a specific file revision, copying it to the local workspace. A get does not lock the file; hence the VCS may mark your local file copy as read-only so you re reminded not to modify it. Check-out: This is a get with lock. That is, the VCS copies the file to your local workspace (if it s not already there) and locks it in the repository, preventing others from performing a check-out. (However, other users can still get the file.) When you have the lock, the VCS typically makes your local file copy modifiable. Check-in: After a file is modified locally, this operation posts the new revision to the repository, assigning it the next version number. When you check-in, your lock is typically released at the same Borland Software Page 9 of 90

10 time so that someone else can perform a check-out. This means that your local copy may be marked read-only again since you no longer have the lock. As you can see, locking is a key VCS capability. Acquiring a lock during check-out and releasing it at check-in avoids the possibility that two users both think they have control of the file. Otherwise, if two users both modified the same file and then checked-in their changes, the second user could wipe-out the changes made by the first user. As we ll see later, allowing multiple users to work on the same file at the same time is an important capability. Such parallel development requires additional features such as branching and merging. Although some VCS s support parallel development, it is typically not their strong suit. We ll probe this topic more under SCM Containers For some reason, having everyone on planet earth store their files in a single repository doesn t scale well. Consequently, VCS s allow multiple repositories, sometimes referred to as instances. Within an instance, a VCS may provide additional containers such as projects, subprojects, or views. 6 These containers allow you to separate files within a repository based on application, module, business unit, or other criteria. VCS s also provides tags (or labels) that you can apply to an arbitrary set of file revisions. A tag is a snapshot that identifies files related to a specific purpose (e.g., the 2.1 release build). A VCS typically supports a client-side container called a workspace. In addition to providing a welldefined area for check-in/check-out operations, a workspace allows a VCS to compute the status of each file: which ones have been modified since they were checked-out, which ones are out-of-date, and so forth Security Minimally, a VCS must support repository-level security, restricting the users who can access the repository. Some VCS s rely entirely on OS-provided, file-level security for all access. Some VCS s offer finer-grained access control on lower-level containers (projects, folders, and individual files). Access control specifies which users and user groups can access these containers. Correspondingly, permission control specifies what each user or group can actually do: read, add, update, delete, etc What is Software Configuration Management? At your company, everyone just works on one release at a time, right? And when it ships, you forget about it and move on to the next release, yes? No, we don t know anyone who works that way either. Very few companies have the luxury of worrying about only one version of an application at any given time. If you re like most companies, you have a current release in the field, a new release under development, and several older releases still being supported. If you find a bug in the current release, you may have to fix it in two older releases, but sometimes the fix isn t quite the same in each release. And tomorrow, you may need to rebuild an old field test release so you can duplicate a problem found there. Oh, 6 Amazingly, some VCS systems have no container other than an instance! Borland Software Page 10 of 90

11 and your boss has this great idea to start three new releases in parallel, each delivering functions 3, 6, and 9 months out. And surprise, you re going to need another service pack for that last release. In short, you need software configuration management (SCM). The primary purpose of SCM is to allow parallel development. The term SCM was well-chosen as it aptly describes the activities of parallel development: Software: The focus of SCM is not just source files, but all artifacts needed to develop software applications. This includes all files required to produce software resource files, build scripts, thirdparty libraries. But software development often involves artifacts that may not naturally exist as files: requirements, defects, enhancement requests, tasks, and so forth. SCM allows you to manage these artifacts well. Configuration: SCM seeks to provide precise definitions of specific artifact configurations. For example, you can create a configuration that identifies an approved set of requirements or all the files used to create a field test build, including the compilers, build scripts, and other tools. Reproducibility of specific configurations is important for traceability, auditing, and in some cases even regulatory purposes. Management: This means that the flow of changes to existing configurations and the creation of new ones are controlled and tracked. SCM allows you to define rules or workflow that control what can be changed by who. As you can see, SCM requires the features of version control, but it adds a significant layer on top. Let s examine the key features in this layer more closely Introducing the Artifact We ve already used the term artifact, but let s single out what it means to SCM. A typical software development lifecycle requires the development, evolution, and management of things other than source files such as requirements, models, graphics, change requests, schedules, tests, and so on. The term artifact refers to the generalization of objects that can be versioned, branched, merged, etc. Because SCM emphasizes management of all artifacts required to produce software, it must address management of more than source files. Some SCM systems accommodate non-file artifacts by serializing them into files and applying file-based version control. Other SCM systems support non-file artifact types directly, providing type-specific behavior for storage, versioning, merging and so forth. Yet another approach is to provide an SCM solution as an integration layer on top of multiple, lower-level tools such as VCS, defect tracking, and change management applications. How artifact generality is addressed varies greatly from one SCM system to another, but the goal of SCM is to manage all artifacts Branching Branching allows an artifact s version stream to be forked. Each fork can be independently modified, receiving its own versions. There s a big difference between copying an artifact and branching an artifact. Although copying allows each copy to be modified independently, the SCM system does not know that Borland Software Page 11 of 90

12 copied artifacts are related to each other in the repository. In contrast, with branching, the SCM system retains special knowledge about an artifact s branches. This information supports things such as intelligent revision comparison and three-way merging (discussed in the next section). An example of branching and merging is shown in Figure 2. Main branch New branch created from main branch version Revision 3 on the lowest branch merged to both upper branches New branch created from main branch revision Figure 2: Branching and Merging When a new artifact is added to the repository, a main branch is started and new revisions are added to it. At any time, a parallel child branch can be started. In this example, one child branch is created from main branch version 1, and another child branch is created from main branch version 2. Within a branch, the version number starts over (at zero in this example). New revisions are applied to a specific branch, incrementing the version number on that branch but not affecting other branches. Branching is needed to support parallel development on files. However, there are advantages to allowing non-file artifacts to branch as well. For example, if a defect artifact can be branched, the two branches can be used to track fixes to the same defect that exist in different releases. Not all SCM systems support branching for non-file artifacts Merging Inevitably, a change on one branch will need to be propagated to another branch. In Figure 2, version 3 of the artifact on the lowest branch is applied to both of the parent branches. However, you can t just copy a revision from one branch to another branch this could wipe-out changes that are specific to the target branch. Instead, what you want to do is merge the changes from the source to target branch. 7 More specifically, SCM systems store synchronization information that allows three-way merging. The three parts of a merge operation are (1) the source revision containing changes to be propagated, (2) the target revision that will be modified, and (3) the last source revision common to the source and target branches, known as the common ancestor. 7 Note, however, that overwriting the target artifact with the source revision instead of merging it is sometimes the desired effect, for example with binary files. Borland Software Page 12 of 90

13 For files, merging is typically done by passing these three file revisions to a three-way file merge tool. The tool compares both the source revision and target revision to the common ancestor revision and determines two important things: (1) what changes appear in the source revision only that should be propagated to the target revision, and (2) what changes appear in the target that may conflict with changes in the source revision. In many cases, merging detects no conflicts, so the merge tool automatically propagates the source changes to the target revision. When conflicts are detected (and sometimes even when none are), the merge tool typically displays the differences to a user who can review the differences, resolve conflicts, and approve the final result. The merge tool then creates a result file reflecting the target file updated with changes. The SCM system adds the result file to the target revision s branch, creating a new revision. In Figure 2, revision 3 on the lower child branch was merged to both of the upper branches. When it was merged to the main branch revision 3, it created main branch revision 4, which contains the merge results. (The arrow points to the revision that was created as a result of the merge.) For this merge, the common ancestor between the two branches is main branch revision 2: it is the most recent revision that both branches had in common. When the lower child revision 3 is merged with upper child branch revision 2, upper child branch revision 3 was created. For this merge, the main branch revision 1 is the common ancestor. For files, there is more to merge than contents. In an SCM system, files have other properties such as name and description. Merging a file requires merging these properties as well, for example to propagate a name change. Non-file artifacts that branch also require merging in order to propagate changes Sharing and Cheap Copies Over time, you will have a lot of artifacts especially files and some files will have a lot of branches. Consider the effect of containers : if you have a lot of teams, software components, and releases, you will need a lot of independent projects, subprojects, or other containers to support parallel development and separate maintenance. Often the same files will be needed in each of these containers. How do you get the files you need to each of these containers? Forcing every file to branch in order to get a unique branch in every possible container could be a lot of branching, which is expensive. SCM systems address this problem with a technique known as cheap copies. This involves creating references to files in a new container. Similar to Unix links, this happens without actually copying the files themselves (that is, their content or their history). Unlike Unix links, however, the first time a file is modified via a new reference, it is branched. For this reason, cheap copies are also referred to as copy on write sharing. Cheap copies are an important SCM feature to support efficient branching with large projects Change Tracking When we develop software, we tend to work in specific units of work: implement a feature, fix a bug, add a unit test, and so forth. Often a single change involves updates to multiple files. Although it is useful to see the history of individual files, sometimes we want to see why a certain file revision was created and what other files were modified for the same reason. Tags can help, but what we really want is a change container into which artifacts can be placed that are related to the same unit of work. Borland Software Page 13 of 90

14 Lots of such change containers are used by different SCM systems, going by names such as activities, change sets, change packages, jobs, and so forth. What s common to these various facilities is that they allow use to see what was actually changed as well as the work artifacts that support the reason for the changes: requirements, defects, tasks, etc. Moreover, SCM systems typically allow the related artifacts to be placed inside the container and then applied or committed in an atomic transaction. The use of atomic transactions ensures that the entire set of changes is applied or none of them Development Streams So far, we ve talked about containers that hold sets of artifacts: projects, subprojects, folders, etc. We ve also talked about branches as parallel version streams for individual artifacts. In practice, we need to develop or maintain sets of artifacts related to a specific long-term activity: new feature development, bug fixes, a service pack release, etc. In other words, we need a container where the branch of every artifact it contains is dedicated to the corresponding activity. Whether they are for new development or maintenance, we ll call this intersection of containers and dedicated artifact branches a development stream. Think of a development stream as a place where we may apply a continuous stream of work-specific changes as described in the previous section. In some SCM systems, a container type is the same as a development stream. For example, a subproject may be used to separate artifacts belonging to different applications, but a subproject may also be the mechanism used to create multiple development streams for the same application. In other cases, an SCM system may provide separate mechanisms for containers and development streams Change Rules It is not enough to simply capture and organize artifact revisions. SCM systems typically provide a way to control which users can make changes and when. (After all, we re talking about software configuration management.) Security rules provide part of the solution, but security typically does not consider an artifact s state. Consequently, some SCM systems allow you to define change rules that affect how certain changes can be made. An example rule may be that a defect can only be marked fixed by someone in the QA group. Another rule may be that files cannot be added or modified unless a task or other work item is selected as the context, allowing the SCM system to record that context along with the file changes What is Application Lifecycle Management? SCM is not the only process that occurs in software development. Most organizations also use a combination of requirements management, modeling, coding, build management, test management, deployment, and other processes. Although fundamental SCM concepts such as artifacts, versioning, and change management thread throughout these processes, managing the end-to-end lifecycle requires a layer above them all. Application lifecycle management (ALM) is the coordination of all activities in the software development lifecycle. Compared to SCM, whose focus is the management of specific branches, components, and 8 Confusingly enough, such a development stream is sometimes called a branch, not to be confused with the version branches of individual artifacts. Borland Software Page 14 of 90

15 artifacts, ALM broadens the management scope to entire project lifespans and finished goods : complete applications delivered for a specific business goal. ALM is a discipline aimed at improving efficiency and reducing errors. As such, it can be practiced independently from the tools used in any specific development phase. However, practicing ALM is a laborintensive process without support from the tools used in the lifecycle. The following sections describe some of the key features that an ALM solution must address Collaboration If you had to summarize ALM in a single word, you could say that it s all about collaboration. Software development management has evolved from the VCS focus of managing bits to the SCM focus of managing change to the ALM focus of managing processes. Since processes are the activities of people and tools, this means ALM facilitates the collaboration between people and tools. ALM seeks to coordinate people on the same team and those on different teams involved in different project phases. For example, ALM provides a way for QA people to not only see business requirements but to communicate with business analysts. One collaboration issue that ALM must grasp is information timeliness. As our teams get bigger, our application portfolios grow, and our processes become more sophisticated, we cannot depend solely on information pull. That is, we cannot rely on users monitoring large sets of information to discover work they need to address. Time-sensitive information must be pushed on an event or exception basis to appropriate people and tools. ALM solutions provide event-based notifications in a variety of ways such as and publish/subscribe messaging frameworks. An important use case for ALM collaboration is progress reporting. Team leads, project managers, and executives need to see the big picture via customized reporting, analytics, and even data mining. By integrating tools, artifacts, and people, ALM gives managers the oversight they need for monitoring activities and making mid-course corrections Process Improvement If collaboration is the short answer to what is ALM about, process improvement is the short answer to what is ALM s goal. In different organizations, process improvement may be known as process enforcement, reengineering, or enactment. Regardless, the goal of ALM is the same: improve the predictability, quality, and efficiency of software projects. Said another way, ALM strives to reduce process variance, minimize defects, and expend resources more intelligently. Sometimes process improvement is motivated by regulatory pressures (FDA, EMEA, Sarbanes-Oxley), internal evolution towards software development maturity models (COBIT, CMMI, ITIL), or a combination of these. It is important to note that ALM does not promote a specific development process. Rather, it provides a framework for which a chosen methodology can be monitored and managed. In fact, large organizations will often want to use multiple methodologies simultaneously. Which process is appropriate for a given project depends on the team size, its maturity, application size and complexity, and other factors. Whether the methodology is waterfall, RAD, agile, or a custom process, the role of ALM is to allow the methodology to be fine-tuned to reflect lessons learned. Over time, this ensures that teams follow established best practices. Borland Software Page 15 of 90

16 The role of a specific ALM solution, then, is to support the specific processes that the organization wishes to use. This has a number of implications: Automation: As much of the development lifecycle should be automated as possible. Every step that can be codified improves conformance and reduces process errors. Customization: An organization must be able to customize artifact workflows, event-based applications, and even artifact types to support the processes they use. Even end-user tools should be customizable to fit the needs of specific end-user roles. Extension points: It must be possible to add new tools to the ALM framework and integrate them with other tools. Process improvement by managing interactions across development phases is highly dependent on the ALM factor discussed next: traceability Traceability Traceability is an important function that ALM fulfills above phase-specific tools. It entails the linking of artifacts to represent specific relationships: requirements to design artifacts, defects to modified files, parent tasks to child tasks, test cases to requirements, etc. Traceability is important to prevent divergence of the final application from the initial requirements it lets us to follow the original target through its entire lifecycle. It is important that traceability is bi-directional. For example, a business analyst may want to know when test cases are written that verify specific requirements. Conversely, a QA engineer writing test cases must be able to review requirements in context to his or her work and know when they change. (Arguably, everyone in the lifecycle needs to correlate their work back to the appropriate requirements.) By spanning tools and development phases, traceability gives us greater schedule predictability, quality control, and process auditing Orchestrating Distributed Teams Increasingly, we live in a world of globally distributed development (GDD). Even small- to medium-size companies find themselves with software teams in different time zones and on different continents. Whether they are internal groups, contractors, or off-shored development, distributed teams are becoming commonplace. While distributed teams yield benefits such as follow-the-sun development, they impose special collaboration problems as well: Information integrity: How do you provide distributed teams access to up-to-date requirements, source code, defect reports, test cases, and other artifacts? How do we prevent contradictions and ensure one version of the truth? Process compliance: How can you ensure that remote teams are following corporate development practices? Borland Software Page 16 of 90

17 Scalability: How do you scale the information infrastructure to accommodate thousands of distributed users who need to access millions of artifact revisions, potentially representing terabytes of data? High availability: Given that software development has become a mission-critical business operation, how do you keep information available to practitioners on a 24 x 7 basis? Administration: How do you backup, secure, and monitor repositories in a distributed topology? In the pre-alm era, we d expect to find each team independently addressing these issues on a geographicor project-specific basis. We d probably find a hodgepodge of solutions with greatly varying degrees of attention to each issue. When we elevate our focus to ALM, we expect these issues to be addressed uniformly across the entire development organization. Borland Software Page 17 of 90

18 3. StarTeam Basics In this section, we ll review StarTeam basics: its architecture, key components, and core concepts. We ll map StarTeam s terminology and features to the VCS, SCM, and ALM concepts described in the previous section. If you re new to StarTeam or want to brush-up on the essentials, read on The Big Picture If you just read section 2, you may be wondering: how would we classify StarTeam? Is it a VCS tool, an SCM application, or an ALM solution? Well, you have to read the rest of this document and we ll reveal the answer on the last page. Oh, you re one of those who reads the last page when the suspense is too much? All right we won t make you wait: here s how we classify StarTeam: StarTeam supports all of the features of a VCS: a centralized repository, versioning for all file types (not just text), all basic VCS operations, different container types, and fine-grained security. StarTeam is also a complete SCM application: it supports file and non-file artifact types; branching and merging; artifact sharing ( cheap copies ); change tracking; development streams; and change rules. StarTeam supports enterprise ALM foundation features in the following ways: Collaboration: A topic artifact supports threaded discussions, and event-driven notifications are provided via and publish/subscribe messaging. Read-only audit artifacts are automatically generated to provide a detailed log of all data modifications. Customization: StarTeam provides customizable workflow, forms, and artifact properties. An SDK supports custom event-driven applications and the integration of new tools. Bundled graphical clients provide per-user customization features. Traceability: A flexible link facility allows arbitrary artifacts to be related within the repository. Distributed teams: StarTeam uses a unique architecture that allows thousands of users worldwide to access a single instance without replication. Borland Search is a search engine that provides secure searching across multiple StarTeam (and CaliberRM) repositories. StarTeam Datamart is a data mining application that provides multi-repository analytics and custom reporting. StarTeam is integrated with other Borland ALM products (e.g., CaliberRM, Silk, Tempo, and Together), popular IDEs (e.g., JBuilder, Eclipse, Delphi, Visual Studio), and other repository-based products (e.g., HP Quality Center). In short, we can t say that StarTeam is an all-in-one ALM solution, but it is far more than a VCS or even an SCM system. Companies use StarTeam because they are building ALM, and StarTeam provides a strong foundation. Borland Software Page 18 of 90

19 3.2. Standard Architecture Overview The core components of StarTeam architecture are depicted in Figure 3. StarTeam Clients Network StarTeam Server Cross- Platform Client StarTeam SDK Database Server Administrator StarTeam SDK LAN, WAN, VPN, etc. command API StarTeam Server process Vault Hive 1 Files Command StarTeam Line SDK Interface TCP/IP Hive 2 Files Custom Application StarTeam SDK Figure 3: Overview of the StarTeam Architecture As you can see, StarTeam employs a client/server architecture. The Cross-Platform Client (CPC), Server Administrator, and Command Line Interface are example bundled StarTeam clients. StarTeam clients use the freely available StarTeam SDK, so you can write custom applications that have access to the same features as the bundled clients. The SDK is fully featured in Java,.Net, and COM flavors, allowing you to write custom applications for any environment. A single StarTeam client can have multiple sessions to any number of StarTeam servers. All StarTeam clients connect to a StarTeam Server process using TCP/IP, so virtually any kind of network can be used: LAN, WAN, VPN, or the public internet. StarTeam uses a proprietary protocol called the command API, which supports compression and multiple levels of encryption. The command API has been optimized to support high performance, automatic reconnect, delta check-out for slow connections, and other important features. A single deployment instance of StarTeam is known as a server configuration, usually shortened to just configuration. The persistent data of a configuration consists of a database and a vault and is managed by a single StarTeam Server process. The database holds all metadata and non-file artifacts, whereas file contents are stored in the vault. The database can be Microsoft Desktop Engine (MSDE), full SQL Server, or Oracle, and it can reside on the same machine as the StarTeam Server process or a separate machine. Borland Software Page 19 of 90

20 In its SCM capacity, StarTeam s vault is a critical component that affects performance and scalability. In contrast to the traditional delta storage technique, StarTeam s vault uses an innovative (patent pending!) architecture designed for scalability, performance, high availability, and dynamic expandability. Today, customers are storing up to a terabyte of data in a single StarTeam vault, but it was designed to store content up to a petabyte and beyond. Within the vault, files are stored in containers known as hives. A hive is a folder tree containing archive and cache files on a single disk volume. Hives can be dynamically added on existing or new disk volumes, thereby allowing virtually unlimited capacity. StarTeam stores each file revision in a separate archive file in a manner that minimizes space usage as well as duplicate content. Amazingly, StarTeam s vault uses less space than delta-based storage! 9 The StarTeam database and vault can be backed-up dynamically, while the server is in use. This supports 24 x 7 operations that want to minimize down time. Other high availability features and techniques are discussed later StarTeamMPX Components The standard architecture described in the previous section represents the minimal components present in a StarTeam instance: a StarTeam Server process managing a vault and a database and one or more StarTeam clients. With just these components, all basic StarTeam functionality is available. Like all client/server architectures, as the number of clients grows, the server could potentially become a bottleneck. In fact, the scalability of many client/server systems is entirely limited by this bottleneck. Other client/server systems address scalability by deploying multiple instances and replicating information between them to attain synchronization. StarTeamMPX (or simply MPX) is a unique solution to client/server scalability. MPX is a publish/subscribe messaging framework that pushes update events that contain metadata and data to clients. It is optional because it is not required for basic StarTeam functionality. However, when MPX is activated, it improves StarTeam server scalability and improves StarTeam client responsiveness. Basic MPX requires the addition of a single extra component, known as the Message Broker. The Message Broker s role is illustrated in Figure 4. 9 In certain cases where it is more economical to send file deltas to clients instead of full versions, StarTeam generates and caches delta files. However, in most cases sending full versions is more economical. Borland Software Page 20 of 90

21 Client Client command API StarTeam Server process Database Client event API Vault Client update events MPX Message Broker Figure 4: Role of the MPX Message Broker. The Message Broker is a messaging process that uses an event API to receive updates from the StarTeam Server process. The Message Broker broadcasts encrypted messages containing updated artifacts. StarTeam clients subscribe to subjects and receive only messages relevant to them. By receiving updates as soon as they occur, StarTeam clients do not need to poll for updates or refresh information they have cached, significantly reducing the demand-per-client on the StarTeam server. This improves server scalability, but it also improves client responsiveness since updates are received within seconds after they occur. Messages broadcast by a Message Broker benefit clients with active sessions. However, for files MPX offers an optional Cache Agent process that manages its own persistent cache. Cache Agents can be deployed at geographic locations, allowing clients to fetch file contents from the nearest Cache Agent, preventing the need to fetch this content across a longer (and potentially slower) network connection. MPX Cache Agents are illustrated in Figure 5. Borland Software Page 21 of 90

22 Remote site Server site Client Client command API StarTeam Server process Database Client Vault file checkouts Remote Cache Agent Client Remote Message Broker update events broker-tobroker forwarding Root Message Broker update events file checkouts to local clients Root Cache Agent Figure 5: MPX Cache Agents In this example, a Root Cache Agent is deployed network-near to the StarTeam Server process. A Root Cache Agent directly accesses the StarTeam vault, providing local clients with an alternate path to the vault for checking-out files. This reduces demand on the StarTeam Server, enhancing its scalability. This example also shows a Remote Message Broker and a Remote Cache Agent deployed at a remote site. Utilizing broker-to-broker forwarding, each update event is forwarded once to the Remote Message Broker, which then broadcasts it to local clients. Files are streamed to the Remote Cache Agent, which stores them in an encrypted private cache. StarTeam clients network-near to the Remote Cache Agent can check-out files at any time, leveraging the local high-speed network instead of pulling content across the WAN. This further reduces demand from the StarTeam Server while improving remote client responsiveness. Though not shown in this example, MPX offers many other benefits for distributed organizations: An optional companion to the Message Broker is the Multicast Service, which broadcasts messages to local clients using IP multicast. This broadcast protocol further reduces network traffic in large client communities. Borland Software Page 22 of 90