PSFC/JA-13-23 MDSplus Automated Build and Distribution System Fredian T.W., Stillerman J.A.*, Manduchi G.** * Plasma Science and Fusion Center, MIT ** Consorzio RFX, Euratom-ENEA Association, Padova,Italy May, 2013 Plasma Science and Fusion Center Massachusetts Institute of Technology Cambridge MA 02139 USA This work was supported by the U.S. Department of Energy, Grant No. DE-FC02-99ER54512. Reproduction, translation, publication, use and disposal, in whole or in part, by or for the United States government is permitted.
MDSplus Automated Build and Distribution System T.Fredian 1, J. Stillerman 1, G. Manduchi 2 1 Massachusetts Institute of Technology, 175 Albany Street, Cambridge, MA 02139, USA 2 Consorzio RFX, Euratom-ENEA Association, Corso Stati Uniti 4, Padova 35127, Italy Support of the MDSplus Data Handling System has been enhanced by the addition of an automated build system which does nightly builds of MDSplus for many computer platforms producing software packages which can now be downloaded using a web browser or via package repositories suitable for automatic updating. The build system was implemented using an extensible continuous integration server product called Hudson which schedules software builds on a collection of VMware based virtual machines. New releases are created based on updates via the MDSplus cvs code repository and versioning are managed using cvs tags and branches. Currently stable, beta and alpha releases of MDSplus are maintained for eleven different platforms including Windows, MacOSX, RedHat Enterprise Linux, Fedora, Ubuntu and Solaris. For some of these platforms, MDSplus packaging has been broken into functional modules so users can pick and choose which MDSplus features they want to install. An added feature to the latest linux based platforms is the use of package dependencies. When installing MDSplus from the package repositories, any additional required packages used by MDSplus will be installed automatically greatly simplifying the installation of MDSplus. This paper will describe the MDSplus package automated build and distribution system. Keywords: Data Acquisition Systems, Data Management, Data Formats, MDSplus Introduction MDSplus[1] is a collection of libraries and applications used for providing access and storage of scientific data. It provides a wide range of functionality including: Data acquisition from measurement devices; organization of the storage of both experimental measurements, analysis results and various metadata associated with those data items Access to the data and metadata from a wide variety of programming languages and utilities Remote data access using a variety of transport mechanisms. Originally developed on the OpenVMS platform in the 1980 s, MDSplus has since grown in functionality and has been ported to numerous computing platforms. MDSplus is used by scientists and engineers worldwide mostly in the field of fusion energy research. Maintaining the MDSplus software on a large number of computing platforms and operating system versions is no easy task. To ensure that users of the software on each of these platforms can take advantage of bug fixes and new features in a timely fashion, the MDSplus 1
developers constructed an automated build and release system. Each night the MDSplus software code management software is queried to determine if any changes have been made to the software. If changes are found new releases of the MDSplus software are built for nearly a dozen different computer operating systems. These new releases are made available to MDSplus users via the MDSplus web server via file downloads or package repositories making it possible to update MDSplus packages using some of the same tools used for updating operating system software. This paper will discuss how this build and distribution system was implemented. Current Computer Platforms Supported MDSplus distribution kits are currently available for the following computing platforms: RedHat Enterprise Linux Versions 5 and 6, 32-bit and 64-bit (download and repository) RedHat Fedora Core Versions 15, 17 and 18, 32-bit and 64-bit (download and repository) MacOSX Lion 10.5 (has been known to work on 10.6 through 10.8) (download) Ubuntu Versions 11 and 12, 32-bit and 64-bit (download and repository) Microsoft Windows Windows 7, 32-bit and 64-bit (download) Solaris Version 11 (download and repository) Automated Build Control To manage the nightly builds for the MDSplus software we are using a software package called Hudson Extensible Continuous Integration Server [2]. This package provides the ability to configure build jobs to be dispatched to various server systems which check for MDSplus code updates and perform the software builds on those servers. Hudson keeps track of the code updates found which triggered the builds, the log files associated with the builds and the completion status of the builds. The developers can be notified if any problems were detected during these builds via email. The continuous integration server has many useful features such as defining dependencies between jobs so the jobs can be scheduled to occur at fixed intervals or after another job completes. Hudson also has the capability to start and stop virtual machines such as those provided by products such as VMware vsphere and Oracle Virtualbox. This enables the building of software for several different operating systems using a single host computer. Each nightly build of MDSplus is performed with over 30 build jobs spread over more than 10 different computers or virtual machines. Hardware and Virtual Machine Configuration To perform MDSplus builds on multiple platforms we are using a collection of servers in a virtualization environment utilizing VMware vsphere 5.0[3]. The current environment consists of two servers, a free-standing Dell Precision Workstation 690 with a Xeon 5150 dual core 2
processor with 18 GB of RAM. The second server is a rack-mounted Dell PowerEdge R410 server with dual quad core Xeon X5672 processors and 32 GB of RAM. Two new servers are being added to the configuration. This virtualization environment is being used for many functions at the Plasma Science and Fusion Center at MIT in addition to the MDSplus build system. Each night, various virtual machines are powered on to build MDSplus and then powered back down after completion. Configuration of new virtual machines can be completed to add additional build platforms as needed with minimal effort. A backup product called Veeam[4] which is optimized for backing up VMware virtual machines is used to protect the contents of the virtual machines. Packaging and Versions For the linux based distributions the MDSplus software package has been split into functional pieces so you only need to install the software modules that you wish to use. Currently the list of packages include: Camac - Libraries, device support and utilities used when doing data acquisition with hardware based on the CAMAC data acquisition hardware standard. D3D - Scripts used at General Atomics in San Diego, CA for accessing data on the D3D fusion experiment. Devel - Object libraries and include files needed for developing compiled libraries and utilities which call into the MDSplus internal routines. Epics - Interface modules to link the Epics process control system with MDSplus. GSI - Libraries and configuration files to utilizing Globus Security Infrastructure in MDSplus client/server communications. HDF5 - Libraries and utilities for moving data between HDF5 and MDSplus storage systems. IDL - Libraries and procedures to access MDSplus from the Interactive Data Language product distributed by Exelis Visual Information Solutions. JAVA - Libraries and utilities to enable access to MDSplus via Java based applications. KBSIdevices - Device support modules developed and used by the Korea Basic Science Institute. Kernel - MDSplus core libraries, utilities and configuration settings used by all other packages. Labview - Libraries and modules providing an interface between the National Instrument s Labview product and MDSplus Matlab - Libraries and modules providing an interface between Mathworks Labview product. MITdevices - Device support modules developed and used at the Massachusetts Institute of Technology s Plasma Science and Fusion Center. Motif - Libraries and applications based on the Motif X Windows users graphical interface. MsSql - Libraries to provide an interface between MDSplus and Microsoft SQL Server databases. 3
PHP - Libraries to access MDSplus data for producing web cgi scripts using the PHP cgi utility. Python - MDSplus module for the Python language. RFXdevices - Device support modules developed and used at the RFX experiment in Padova, Italy MDSplus - Catch all package for installing the complete MDSplus system. When using the MDSplus software repositories, the new MDSplus packages contain module dependencies such that any additional software packages needed for a particular package will automatically be installed along with the requested package. The new MDSplus build and distribution system provides for three distinct levels of releases: stable, beta and alpha. The stable releases may incorporate bug fixes as soon as they are available but no new features are added until those features have been released in the beta and alpha versions and tested. Similarly the beta releases may incorporate bug fixes but no new features are added until they are tested in the alpha releases. The alpha releases represent the most recent changes to any of the MDSplus code modules. Package releases are identified with the major classification of stable, beta or alpha as well as major,sub and minor release numbers. For instance a package might have the name: mdsplus-beta-kernel_5.0.0_i386 indicating that it is a beta release of the kernel with version number 5.0.0 for a 32-bit computer system. New software build scripts written in Python were developed to be used by the build system to automate the checking for code modifications to either the stable, beta or alpha branches of the MDSplus code version control system (currently CVS). If a change is detected, a new stable, beta, or alpha build is executed and tested and if successful a new release with the minor version number incremented is automatically made available for download via the MDSplus web site or through the installation repositories. On platforms where the MDSplus installation package has been broken into subpackages, only the affected subpackages will have their minor release versions incremented. Development scripts were also developed to simplify the promotion of alpha releases to beta and beta to stable and incrementing the sub or major portions of the version numbers. These operations are done manually by the developers when new functionality has been added and tested. Downloads and Repositories The MDSplus web site has been modified to provide access to all the installation kits being built by the automated build system. Users can download individual installation kits from the web site or install repository installation kits which will configure the system to access MDSplus repositories directly. System administrators of systems such as RedHat and Ubuntu are usually very familiar with the use of repositories as this is how system updates or new package installs are generally performed. Utilities such as yum (Redhat systems) and apt-get (Ubuntu systems) 4
can then install and/or update MDSplus packages using the same type of command that operating system installation and updates are performed. Conclusions The new MDSplus Automated Build and Distribution System has been in service for nearly a full year now although some additional platform support has been added more recently. The nighly build feature reduces the workload on the developers and hopefully provides the user community with quicker updates and new feature releases on all the platforms. There has been surprisingly little feedback from the user community on the increased availability of installation kits and updates. Hopefully this is an indication that things are working well and fulfilling the needs of MDSplus users. I suspect that many sites are still installing MDSplus packages manually and do not routinely update their systems via the repositories. As users begin using these facilities perhaps we will learn more about how well the new system is functioning from a user point of view. There is always room for improvements. As each release is constructed there is a set of regression tests done on the installation of the packages and the operation of several of the key functions provided by the MDSplus core system. More thorough test coverage would better guarantee the quality of the releases but as usual designing better test scenarios is often as difficult and time consuming as developing the functions that they are supposed to test. Another area of improvement will likely come after we gain experience in release management. Currently there is no formal procedure for determining the appropriate time for major release updates where the alpha and beta releases get moved on to the stable versions. This will be more critical as sites begin to perform automatic software updates on production systems. The use of continuous integration technology in conjunction with virtual machine capabilities has greatly improved our ability to produce MDSplus distribution kits for a wide variety of computing platforms. MDSplus bug fixes and new features can now be distributed to the MDSplus community in a fraction of the time and with much less developer effort than ever before. You can view the MDSplus Hudson setup at http://hudson.mdsplus.org [1] MDSplus, http://www.mdsplus.org [2] Hudson Extensible Continuous Integration Server, http://hudson-ci.org [3] VMware vsphere, http://www.vmware.com/products/datacentervirtualization/vsphere/overview.html [4]Veeam Modern Data Protection, http://www.veeam.com/ 5