NEESit Software Configuration Management Process

NEESit Software Configuration Management Process 1 1 NEES Cyberinfrastructure Center, SDSC Last Modified: 2005-02-09 Version: 1.1 Acknowledgment: This work was supported by the George E. Brown, Jr. Network for Earthquake Engineering Simulation (NEES) Program of the National Science Foundation under Award Number CMS-0000000. Visit http://it.nees.org/ for more information.

NEES Software Configuration Management Guidelines 2 of 12 Table of Contents Revision History...3 About This Document...3 1 Introduction...4 2 Configuration Items...4 3 Revision Control...6 3.1 CVS Commit Guidelines...8 4 Change Management...8 4.1 Fogbugz...8 4.2 Change Management Guidelines...9 5 Build Management...9 5.1 Development...9 5.2 Staging (Baseline)...9 5.2.1 Fogbugz Updates...10 5.2.2 Nightly Builds, Regression Tests, and QA...10 5.3 Production...10 6 Release Management...12 7 Software Coding Practices...12 List of Figures Figure 1: Resolving Fogbugz Issues...11 List of Tables Table 1: Document CI's...4 Table 2: neespop CI's...5 Table 3: neesdaq, neestpm, and NTCP/MATLAB CI's...6 Table 4: NEEScentral CI's...6 Table 5: CVS Repositories and Modules...7 Table 6: Fogbugz Priority Levels...8

NEES Software Configuration Management Guidelines 3 of 12 Revision History Date Version Description Author(s) 9/7/2004 0.1 Initial version for inclusion in statement of work 1/29/2005 1.0 First release 2/9/2005 1.1 Extended section on build management. Other minor edits also included About This Document The purpose of this document is to provide a methodology for tracking and controlling change for releases of NEESgrid software and services. Software development efforts for NEESgrid are highly distributed, with code being developed by NEESit staff, their contractors (Michigan, Oregon State, and Berkeley), the System Integration team, equipment site staff, NEESR researchers, and international collaborators. Every line of code added to the software repository adds to the overall maintenance and increases costs. Therefore, this distributed development effort must to be carefully managed in order to produce a reliable, usable, and maintainable software distribution to meet the needs of the earthquake engineering community. The overhead involved with configuration management can be a cause of irritation for everyone involved. The objective of the processes defined in this document is to improve group productivity while minimizing the impact on individual productivity. Therefore, these processes try to straddle the thin line between chaos and stifling bureaucracy. This document focuses on issues related to configuration management. Refer to the NEESit Software Development Process (TR-2005-001) document for detailed information on NEESit software engineering practices. Suggestions, questions, and comments regarding these guidelines are encouraged. Please send email to swhitmor@sdsc.edu or it-support@nees.org.

NEES Software Configuration Management Guidelines 4 of 12 1 Introduction The configuration management processes addressed in this document are as follows: configuration identification: identifying the objects which require tracking revision control: the process of tracking all objects (project plans, documentation, source code, test plans, etc) and keeping track of the changes to and versions of all of these objects over the course of their lifetime build management: the process of assembling all software components into a working system change management: the process of carefully tracking software deliverables to ensure that they match the current scope of work as well as the task of evaluating the addition of unplanned changes to the release release management: the process of putting a software release into production 2 Configuration Items Configuration items (CI s) are objects that are carefully tracked and managed throughout their lifetime. CI s for NEESit include those involving documentation (Table 1) and those involving software (Table 2, Table 3, and Table 4). Table 1: Document CI's Type CI Name/Description Software Management NEESit Software Development Process NEESit Configuration Management Process Documentation for Software Releases Software Project Plans Software Quality Plans, Test Plans, and Test Reports End-User Documentation All documents at http://it.nees.org/documents

NEES Software Configuration Management Guidelines 5 of 12 Type NEESgrid specific source code/supporting files 3 rd Party Source Distributions Binary distributions Table 2: neespop CI's CI Name/Description DAQ components NEESgrid Account Management Tools NTCP CASA CHEF NFMS NMDS NEESport Simulation Portal SDSC-CA Client Tools Apache HTTPd Big Brother Creare DataTurbine (RBNB) Jakarta Jetspeed Jakarta Tomcat NMI Condor-G Globus Toolkit GPT GSI-OpenSSh MyProxy Network Weather Service neespop gzipped tar files from releases Coming soon

NEES Software Configuration Management Guidelines 6 of 12 Type neesdaq source/support files Table 3: neesdaq, neestpm, and NTCP/MATLAB CI's CI Name/Description Server daemon Client library Example code Testing code NTCP control code MiniMOST-1 control and DAQ programs neestpm distribution NJZTPM-<version>.tar 1 NTCP/MATLAB source/support files MATLAB MOST simulation programs NTCP/MATLAB plugin NTCP server NFMS UploadAgent NTCP/MATLAB support programs Security Files Source distributions Binary distributions neesdaq, neestpm, and NTCP/MATLAB gzipped tar files from releases Coming soon Type NEESit source/support COTS applications Table 4: NEEScentral CI's CI Name/Description Central Repository Authentication Services Fogbugz enhancements Fogbugz Perforce 3 Revision Control Revision control involves the use of tools which keep track of revision history of CI s, allowing changes to rapidly evolve. Such tools provide the ability to retrace changes made to CI s and are useful for keeping a backup of work in progress. NEESit s revision control system also requires the ability to manage a large team of distributed software developers. NEESit s current revision control system is CVS. Within the next year, NEESit will migrate to Perforce for internal management of the production source repository, which provides powerful interfaces for tracking changes to CI s. External collaborators will continue to use existing CVS repositories, with each repository kept in sync via automated tools created by the NEESit team. 1 NEESit stores the neestpm tar file in CVS but not the source code

NEES Software Configuration Management Guidelines 7 of 12 The NEESit software distribution currently consists of four CVS repositories accessible from cvs.nees.org. These four repositories were originally created to support the distributed development environment of the System Integration Team 2 / For anonymous access to the repository, the CVS_RSH and CVSROOT environment variables must be set. A template for accessing CVS via a Bourne shell environment is shown as follows: CVS_RSH=<path to ssh at your site> CVSROOT=:pserver:anonymous@cvs.nees.org:/${REPOSITORY_NAME} cvs co ${MODULE_NAME} Repository and module names currently supported are listed in Table 5. Note that many of the modules listed in this table are no longer of use. Repository Names CVS/neesgrid-chef CVS/neesgrid-matlab CVS/neesgrid-neespop disks/cvs/neesgrid Table 5: CVS Repositories and Modules Module Names casa-ogsi, chef, chefncsa2, dmd-dist, lib-gt3, libnees-dmd, nees-dmd, nees-dmd-archives, nees-java2, neespop-chef, neesport-common, neesport-services, nfms-ogsi, nmds-ogsi main amie, bb, chef, collab, daq, foundry, gpt, grid, httpd, main, mdist-p1, neesam, nfms, nmds, nsds, ntcp, patch, rbnb, rbnb-nees, tomcat, var AxisPTZ, dataq-194, dndtester, fake_daq, flog, foundry, GreenWork, grid, JavaServer, jcamera, jclient, jphoto, JunkModule, JunkModules, jusb, lv-examples, lv-programs, mdist-p1, modules, nclient, nees-adxl, nees-di, NEESTelepresence, nsds, nsdsdriver, nsds_info_providers, nsds-service, nsds-webservice, nsds-webservice-gt3f, nsds-webserviceold, nsds-webservice-src, ntcp, NTCP, ntcp_plugins, ogsa-pre-alpha2, orb2nsds, RBNBDataViewer, server16, turbine Anyone may access CVS anonymously but NEESit authorization is required to write to any CVS repository. To request this access, please email it-support@nees.org. Over the year, NEESit will merge the four CVS repositories into a single repository and will work on simplifying the directory structure. In addition, the modules listed in Table 5 which are no longer in use will be removed. 2 The SI is the team that developed NEESgrid for three years ending October 2004.

NEES Software Configuration Management Guidelines 8 of 12 3.1 CVS Commit Guidelines Developers are encouraged to commit changes to the repository often. This gives the developer a backup of their work and a method for backing out changes. For large development efforts, developers are encouraged to tag their code with their own unique names to avoid side effects of other software development efforts. When committing a change to the repository, please include a brief but informative comment. Project management and QA staff will track all CVS commits. 4 Change Management Change management focuses on procedures used to track and evolve items over the course of time. NEESit uses a commercial product called Fogbugz to assist in this process. 4.1 Fogbugz Any issue or feature request communicated to user support via phone, email (it-support@nees.org), or http://bugs.nees.org is tracked by NEESit s Fogbugz issue tracking system. Fogbugz is also used to track issues identified by internal staff. When a new issue is submitted to Fogbugz, support staff determines if the issue is with the software and if so, assigns issue to software development management, who will assign a Fogbugz priority level (Table 6). If the issue is to be included in an upcoming release, it is assigned to a member of the software development team. Table 6: Fogbugz Priority Levels Priority Description 1 - severe Show stopper immediate patch to the software required Critical part of functionality does not work 2 critical Functionality significantly affected Workaround exists but is hard to use Must fix for the next patch 3 high Workaround exists Performance problems Usability issues Probable inclusion for the next patch 4 medium Feature will be required within the next year Feature is not currently needed as often Probable inclusion for the next major release 5 low Causes unease Minor disturbance 6 minute May not be worth fixing When the developer receives the assignment, the estimated amount of time required to fix/implement the issue (in days or hours) should be entered for the given issue. Refer to section 5.2.1 for detailed information on how to resolve Fogbugz issues.

NEES Software Configuration Management Guidelines 9 of 12 4.2 Change Management Guidelines Overall direction for reviewing and approving change requests will be obtained from the NEES Change Management Committee (to be formed). Refer to NEESit Software Development Process (TR-2005-001) for more detailed information. Until the NEES Change Management Committee has been formed, NEESit will follow the following general guidelines for managing change request: 1) For a change request to be approved, the features requested must be needed for year 1 NEESR, pre-neesr, or other user experiments unless directed otherwise by NEESinc. 2) All software development efforts must be in response to a bug/new feature request entered into NEESit s Fogbugz tracking system (http://bugs.nees.org). 3) The Fogbugz case must be assigned priority 1-3 to be included in the next release or patch 5 Build Management NEESit has three different build environments for releases: development, staging (or baseline), and production. Each build environment, which is supported on a different computer system, is described in this section. 5.1 Development Developers are encouraged to utilize this system for their own test and development efforts. The system simulates four different development machines (neesdev0, neesdev1, neesdev2, and neesdev3) via a commercial software package called VMWare. Each of the virtual machines communicates with each other as if they were on a real network. In addition, different operating systems and NEESgrid deployments may be installed on each machine. 5.2 Staging (Baseline) The staging baseline is the software produced for the next release that has been tested by the developer and is ready for QA to test. Software developers should make every effort to execute their own tests prior to moving their code to the staging baseline. The baseline is more tightly controlled and therefore is of higher integrity, enabling QA staff to run integration tests well before the release date. The baseline is defined within the repository by a tag. For example, code development for an upcoming 3.3 software release might be tagged r3.3_alpha1 for the first alpha release. The NEESit Builds and Packaging Engineer will communicate these baseline tag names during the alpha and beta stages of software development. Code that has the baseline tag will be built and deployed on the NEESit staging machine where QA and other developers may access it. Software developers should only tag new code with the baseline tag once it meets the following conditions: the latest baseline code has been checked out of CVS (cvs update) the new code builds to completion (make) the new code has passed the developer s unit tests (make test)

NEES Software Configuration Management Guidelines 10 of 12 all other components in the distribution (with the baseline tag) build to completion the builds and packaging engineer has been informed of any modifications that need to be changed in the deployment process If any of the above conditions cannot be met, then the developer should continue to work within their own private workspace on the development system. It is important to reiterate that all of the above conditions need to be met before applying the baseline tag. Failure to do so will lead to time wasted by other members of the development team. 5.2.1 Fogbugz Updates Once a baseline tag has been applied to new code, developers need to update any issues in Fogbugz which have been resolved. This will let QA know that the issue will be ready for testing on the staging system. The process for updating fogbugz is as follows: Click on the Resolve button for the issue (refer to Step 1 in Figure 1) Provide a detailed description for the issue (Step 2 in Figure 1) o Give information on what was implemented o Include details on how to test the new code o Include information on any changes required to the deployment process (otherwise QA will not be able to test) o If needed, give details on documentation updates Specify the amount of time spent on the issue Click on the OK button In the next screen, assign the issue to QA (Jinghong Gao) (Step 3 in Figure 1) 5.2.2 Nightly Builds, Regression Tests, and QA Once tagged, the baseline code in the software distribution will be used in an automated nightly build (to be implemented) of the entire NEES distribution. Failures in the nightly build process will be communicated to development staff and must be resolved as soon as possible. If this is not possible, then the change will be rolled out of the nightly build. A regression test suite is in development for the NEES software distribution. When ready, these tests will be automatically executed after a successful nightly build. Regression test failures will be forwarded to the developers and need to be resolved in a timely fashion. Once the build is completed and regression tests pass successfully, the baseline distribution will be deployed to a staging area where QA will test bug fixes and new functionality. This staging area will be made available to the all software developers who want to run additional tests. 5.3 Production The NEESit production system is used to test release candidates prior to release. During other stages of the software development process, this system will have the last production software release deployed on it.

NEES Software Configuration Management Guidelines 11 of 12 Step 1 Step 2 Step 3 Figure 1: Resolving Fogbugz Issues

NEES Software Configuration Management Guidelines 12 of 12 6 Release Management To be written in a future version of this document 7 Software Coding Practices This section provides a list of best practices for NEESit software development. This section will be moved to its own document when the author gets a chance to do so. 1. Get familiar with the Gnu Coding Standards and use them in your development. (See http://www.gnu.org/prep/standards/) 2. Don t reformat code. This is a really bad thing to do because it makes diffs hard to understand and apply. Real changes get hidden in the mass of reformatting. 3. Use the same coding style as the code you are editing. This is a corollary to the previous last guideline. It is easier for people reading the code if it uses consistent layout rules throughout, so when you are editing someone else's code the code you add should be in the same style. 4) If in doubt, lay down a tag. Tags are useful for pinning down a particular version of the code, e.g. one that is being run in service, or just before a big change or import. They are also used to identify branches. Tag names should be short and meaningful, like variable names. 5) Consider any upgradeability ramifications of your change. a) If you want to build a new and clean API consider deprecating the old procedure and making it invoke the new one. b) If you are changing the database you must provide an upgrade script 6) Comment code liberally. You may think your change is obviously but someone else might not agree four years from now. 7) Don t mix tabs. In fact, don t use tabs at all. NEESit is standardizing to 2 characters per virtual tab. Please use this if you can for all new development. 8) Keep it simple. There is no use in implementing an elegant solution if the person maintaining the code can t understand it. 9) Keep maintainability in mind at all times. 10) Use javadoc for all Java API s and pod for PERL. 11) Try to maintain 80 character lines for all UNIX code. 12) Never rush to commit something. Before committing double check with cvs diff what exactly you are committing. 13) Commit one cohesive bug fix or feature change at a time. Don't put a bunch of unrelated changes into one commit. 14) Keep code simple, adhere to conventions, and use comments liberally.