Automated job scheduling Bob Eldering, Software Engineer JIVE December 17, 2012 Contents 1 Introduction 2 2 Design 2 2.1 Goals............................... 2 2.2 First stage............................. 3 3 Usage 3 4 Implementation 7 4.1 Code................................ 7 4.2 Databases............................. 7 5 Extension: FlexBuff support 7 1
1 Introduction In current correlator operations, job scheduling revolves around an object known as: the folder. The folder serves as a container for all information about an experiment. For example, it contains print outs of the VEX file, tape log and station logs. It describes how the operators should correlate the experiment. Receiving the folder is a sign for the operators that they can start correlating. Results of clock searching are added to the folder. When all data has been correlated, the folder is passed on to the support scientists. Before correlation can actually start, the operators have to check if all disks required are present. This is done manually using a web page for the paternoster database. This operation process works fine for now, but might get a bit more complicated in the near future with new developments for the NEXPReS project. These developments include: evlbi experiment will be recorded at the station and/or locally (see http://www.jive.nl/~jive_cc/sin/sin28.pdf). This means that after the initial evlbi correlation, the experiment might be recorrelated with the disk from the local or remote recording, or even from a combination of those recordings. Mixing evlbi and disk-only recording in an experiment. Some stations have no high bandwidth connection to JIVE. These stations might be included in an evlbi experiment by including them in the recorrelation phase only. So for these cases the evlbi portion of the experiment uses a subset of the available stations for the experiment. When 4Gbps data acquisition systems are installed at stations, we will need to select a subset of those channels to be able to correlate that on the Mark4 correlator. The 4Gbps data might be correlated later on the software correlator for example. This document describes a system to help automate correlator job scheduling, to be able to handle these new developments. 2 Design 2.1 Goals The goal of the system is to inform the user when a target is ready for correlation. A target in this context is a subset of an experiment, subset in that a target defines a set of stations, channels and time ranges to be correlated. One could add properties like the priority in the correlation queue to a target. The most common examples of targets would be the 2
whole experiment for normal production and one particular scan for clock searching. Given this definition of a target, we need a system that allows user to : Create and edit targets. Track the status of a target and its media components. When a target is ready to be correlated, the system should clearly show this to the user and offer help to gather/setup the media. 2.2 First stage As said in the introduction, our current situation is simple compared to what is expected to be required in the near future. As it does not make much sense to define the details of the automated job scheduling for systems that do not exist yet, the first stage of the design is rather simple but should allow for extending to accommodate these new systems. In our current situation there are 2 types of experiments: For disk experiments we can use the tapes database to track disk and combine this with the paternoster database. We can even use bank_set queries to Mark5 play back units to improve the resolution of the disk tracking. For pure evlbi we cannot do much right now. Except waiting for the start time and maybe try to ping the remote machines. To make the system useful out of the box, it will create a default target for every experiment, which is just the whole experiment. If it is possible to get a list of sources suitable for clock searching the system could even create another default target for clock searching. 3 Usage To start the automatic job scheduling tool: type start_job_scheduling.sh on ccsops as user jops. The first version of the software is shown in Figure 1. In the left portion of the screen, all active targets are shown. Here, active means that the target s experiment is not released to the PI yet. The following six properties of each target are displayed: Experiment: simply the name of the experiment on which the target is defined. 3
Figure 1: Main window Name: a descriptive name of the target. The system automatically makes two targets named prod and clock search. Media: whether all media are available for correlation. For disk experiments this means that all disks are in the building. It has two possible values: present (color coded green) and waiting (color coded red). Status: this field can be used to put a target on hold manually. By default targets are on hold (color coded red). When all preparation for the experiment are done, it can be set to ready (color coded green). Profile: a target can be linked to a profile (see http://www.jive. nl/~jive_cc/sin/sin22.pdf for a description of profiles) to allow it to be run (semi-)automatically. By default no profile is linked to a target, which makes the field turn red, when a profile is selected it turns green. Mounting: to be able to run a target, not only do the media need to be available, they also need to be mounted on a playback unit (MarkV). This field shows whether all required media are mounted (it is coded green if so, red otherwise). At the right, details about the target selected on the left are shown. Most prominent is the table showing which media are available and mounted. The columns in the table represent time ranges as defined by the selected target. The rows represent stations. A field in the table shows the required medium for that time range and station. When it is available it is coded yellow, if it is also mounted it is coded green, otherwise it is coded red. To reduce the size of the table, consecutive columns that have the same field entries are merged together. The other four properties of a target are shown above the table. Finally, a bar with control buttons lives at the bottom of the screen: 4
Edit: pops up a new target edit window shown in Figure 2. This Figure 2: Target edit window window allows the user to edit the stations, channels and time ranges to be correlated for the selected target. The window also has a radio button group with the option all scans and any scans. When any scan is selected, the media and mounting field in the overview table will turn green if any of the selected scans has the media of all required stations respectively available and mounted. If all scans is selected, this has to be true for all selected scans. At the top of the window, the user can change the target name and select a profile to be linked to the target. The edit button next to the profile selection box will pop up another window. This window is similar to the main window of the program described in http:// www.jive.nl/~jive_cc/sin/sin22.pdf and is shown in Figure 3. If a profile is selected, only the stations and channels used by that profile are available. Delete: this will simply delete the selected target. 5
Figure 3: Profile edit window Set ready/put on hold: this will toggle the status (as previously described) of the selected target. Reload: will do a reload of the database. Should only be required if new experiments are entered into the database. Run: this button will start a correlation job for the selected target. It is only available if all fields of the target are coded green (as shown in Figure 4). Figure 4: A target is selected that is ready to be correlated Create target: this will create a new target for the experiment selected in the drop down box next to this button. It will pop up the same window as the edit button. 6
4 Implementation 4.1 Code This program is implemented in python, using Qt for the GUI elements. The code lives in /ccs/opt/ccc_scripts/ccc_python/job_scheduling and uses many of the other packages defined in ccc_python. Basically, the three parts of the window each live in their own module. As do the pop-up windows. These GUI modules are tied together in the module gui.py. Next to the GUI modules there is a module to handle database transactions: database.py. However, one of the queries executed by this module turned out to be too slow for practical purposes. Changes to the database schema and even the engine are still being researched. But to make the program usable for now, the query has been given its own module (query_implement.py), where some of the logic has been implemented in python. This module makes a temporary table, which causes the program to take quite some time to start up. 4.2 Databases The job scheduling program uses several databases to gather the information that needs to be displayed. The experiment and correlator_control databases are used to query for information like schedules, stations and channels (see http://www.jive.nl/~jive_cc/sin/experiment database.pdf and http://www.jive.nl/~jive_cc/sin/control database.pdf). The profile database is used to link targets to run profiles. The media database gives information about disk being in house. And, while not a database, the reservation server is used to see which media are mounted (see http://www.jive.nl/~jive_cc/sin/sin21. pdf). 5 Extension: FlexBuff support As described in section 2.2, the job scheduling software is expected to be extended when new systems come online. This section describes the changed implemented to accommodate the FlexBuff as a data source. The FlexBuff is a COTS machine with many disk, designed to send and receive data at high speeds, see http://www.jive.nl/dokuwiki/ lib/exe/fetch.php?media=nexpres:2011-02-28_wp8-d8.2.pdf for details. As the same data might be recorded on multiple media (for example on a remote and local FlexBuff), we need to give the user the option to 7
Figure 5: The media selection options. The user can select the data input to use for the whole experiment or, if required, fine tune this selection per station. select which media to use. This option is added to runjob as shown in Figure 5. This figure shows a new subsection in the configuration section called Select data input type. Selecting a medium type here will change the selection for all station. Clicking configure will pop up a new window, shown on top of the runjob window, where the user can set the medium type per station. So it is possible to configure an experiment to use both conventional disk packs and the new FlexBuff. The possibility that data is stored on multiple media is also reflected in the job scheduling software. When no profile is selected yet for a target, all data sources for an experiment are shown in the target details section, see Figure 6. For this experiment, two dummy FlexBuff recording have been added to the database, one for each of the stations Hn and Jb. The color of the box with the data sources reflects the worst status of all data sources. In this case the disks JIVE-059 and OSOD-062 are at JIVE, but not mounted. Therefor the color of the boxes with these data sources is yellow. When a profile for a target is selected. The job scheduling software will filter the data sources according to the selection made for that profile. This is shown in Figure 7. This figure shows the same experiment with the same data sources, but now a profile is selected which uses FlexBuffs for all stations. The stations Hn and Jb have this medium available. However the rest of the stations do not, therefor red empty boxes are shown whenever these stations are scheduled. In addition to supporting the FlexBuff, the job scheduling software will now also automatically run a target when it is ready. Every five minutes 8
Figure 6: Stations Hn and Jb have multiple data sources per scan. Figure 7: A profile which uses FlexBuffs for all stations is selected. the database is checked for available media and the status of the target is updated. 9