Workflow-Management with flowguide

Technical White Paper May 2003 flowguide IT-Services Workflow-Management with flowguide (TM) Improving Quality and Efficiency by Automating your Engineering Processes Betriebskonzepte Sicherheitslösungen Linux-Cluster-Computing Systemmanagement Linux-Migration Rechenlastverteilung Open Software Software-Consulting Backup-Lösungen science + computing ag Hagellocher Weg 71-75 72070 Tuebingen, Germany Workflowmanagement phone +49(0)7071 9457-0 fax +49(0)7071 9457-211 www.science-computing.de info@science-computing.de creating IT solutions Strömungsvisualisierung FEM-Preprocessing Engineering Portale

Overview This paper outlines how to improve the engineering process using flowguide. It introduces the basic concepts and illustrates the use and benefits of flowguide. Since it takes over repetitive tasks, automating processes improves efficiency. It reduces the incidence of mistakes, yields higher quality and reduced time to market. flowguide is a flexible and powerful software tool specially designed for automating engineering processes. It allows to properly define processes, to easily maintain them, and to make them available to other users. The knowhow is captured and long living. Computations can be reproduced and documented. flowguide performs complex computations which are executed repeatedly by one or more persons. It automatically executes large computations as needed for automated optimization or robustness evaluation. It is a powerful tool if these computations are distributed among several computers which can even run with different operating systems such as Windows or Unix. It smoothlessly integrates batch queuing systems. flowguide is especially designed for engineers in large and small organizations, but is equally suitable for all other fields in industry and science wherever fixed processes are executed several times. Installations can range from one to many hundreds of users. This whitepaper is written for users as well as for engineering and IT managers who want to understand how flowguide works and how it can be used to improve the design process. page 2 of 13

Background Typical engineering work involves recurring tasks as pre-processing, conversions of data, moving data to other computers, computations, post-processing and communications to databases. Since design almost always is iterative, the same procedure is repeated many times, often by many engineers. Todays demand for more design cycles in less time increases the need to have a flexible and powerful software for process automation. This software must resolve the problems of often used script-based solutions, which over the long run are expensive to maintain and vulnerable to changes in the computing infrastructure. The list of requirements to such software is long. First of all it must be possible to define a clear interface for the computing task specifying the parameters and results of a computation. A Graphical User Interface (GUI) is often requested for filling in the parameters. This GUI provides the possibility for a free formulation of the flow of work independent of the operating system. Workflows have to be portable with little effort to other operating systems or have to run cross platforms. It shall be flexible and easy to maintain to adapt to changes in the computing environment, of the application software, or of the workflow itself. Furthermore it must be easy to use and allow to monitor and control running tasks. The execution of subprocesses on other computers or via batch queuing systems has to be possible in typical engineering environments. For these purposes science+computing ag has developed the workflow management tool flowguide. Its design is based on the analysis of existing problems and solutions for workflows in CAE departments of the automotive industry. flowguide seamlessly integrates third party applications and can be used wherever similar problems occur, as in almost all engineering fields. page 3 of 13

The Design Process with flowguide In conventional engineering design an engineer repeatedly executes various computer programs manually. flowguide describes this task as a workflow and executes this workflow automatically. The engineering design process with flowguide differs in many ways from the manual design process. The major difference is a split of roles between the workflow administrator, who defines and maintains the workflows, and the workflow user, who performs computations using the workflows. workflow-administrator workflow-user workflow-editor monitor quick-gui configuration flowscript Workflow-Engine PDM A core element of flowguide is flowscript, an XML-based language for describing workflow schemes. Although flowscript can be edited manually, the workflow administrator usually creates flowscript by using the graphical workflow editor. With the Editor the workflow administrator creates new workflows, reads existing flowscript files (even if written manually), edits them, and writes workflows in flowscript format as output. The workflow administrator typically is a system administrator who is supported by an experienced engineer. page 4 of 13

The interface for the workflow user is the Quick-GUI. This input mask is automatically created from the workflow definition and he user fills in the parameters for a specific execution of a workflow in the Quick-GUI. The user can start workflows directly from the Quick-GUI and can monitor and stop them by using the workflow monitor. The workflow user is typically an engineer who wants to run a calculation. A workflow defined in flowscript describes the process. Execution details as where it is to run, names of servers, or names and paths of applications can be kept separate from the workflow in 'workflow-configuration' files. Using this option makes it very easy to maintain a workflow. The flowguide 'workflow-engine' is responsible for the execution of a workflow. Depending on the workflow configuration the workflow-engine either executes the workflow itself, or sends it or some subworkflows to other computers, using a remote shell or a batch queuing system. Shorter design cycle. The engineers are released from time-consuming tasks. Easy to use processes. Your processes are easy to use for others which are not familiar with the details. Clear process definition and reproducibility. The process, the input parameters and the program versions used are clearly defined and you are able to reproduce them at any time, even if they were originally performed by someone else. Enforcement of company policies. page 5 of 13

Workflow Editor and flowscript: Describing the Workflow A workflow requires a proper description of what is to be done. This is done by flowscript which is a computer language for describing the flow of work. With the workflow editor you can easily use all the powerful and flexible features of flowscript, without having to worry about the XMLsyntax of flowscript. flowscript follows an object oriented approach. Each workflow can have a set of variables. Workflows have states as ready, running, done, or failed, and a set of methods which are run if a workflow changes its state (e. g. if it is started, oder stopped). Generally, a workflow itself can contain other workflows as subworkflows. All variables have certain predefined types, such as String, Double, Integer, Boolean, File and FileList. You can also define own types, or use arrays of variables. You can define a variable as input or output variable of a workflow, or you can restrict it to internal usage. The input and output variables define the interface of a workflow. If it is used as top-level workflow (one that is not a subworkflow), flowguide automatically creates a panel from the interface description. Before the workflow is executed the workflow user fills in the input parameters in this panel the so-called Quick-GUI. The interface variables are also used to exchange variables with subworkflows. Subworkflows can either be workflows defined in flowscript which can have subworkflows themselves, or they are some of the predefined atomic workflows or basic workflows. Atomic workflows delivered with flowguide perform elementary operations as copying files, execution of shell commands, applications, shell and Python scripts. The execution of commands and subworkflows are controlled by conditions and rules. To deal with abnormal situations, as unsuccessful runs of applications, there is the concept of exception handling. Free definition for the flow of work Re-use of workflows within other workflows Easy definition of workflows page 6 of 13

Quick-GUI: Easy Use of Workflows The Quick-GUI allows to set and manage the input parameters for workflows and to control the workflow execution. Furthermore, the Quick-GUI presents the output of a workflow. The Quick-GUI is a user panel for filling in the input parameters of a workflow. This panel is created very quickly from the workflow interface description. For specific needs there is the option to add specific layout or description information to variables, which influences their visual appearance in the Quick-GUI. Before workflows are started, the workflow user can save the parameter sets to parameter files together with a textual description. This allows to easily reproduce old calculations or to reuse them in order to make parameter variations. All parameter sets are organized within projects, which allows to keep the overview when doing many runs. Parameter files are text files in XML-syntax. They can be edited manually or be created by other computer programs. You can then start flowguide in batch-mode with these parameter files. This is the most simple way to integrate flowguide workflows in other programs or scripts. More sophisticated possibilities exist, as explained later. Efficient and easy use of workflows Other persons can use your workflows Less mistakes when filling in the input parameter Reproducible computations The focus is on the design, not on the computations Easy integration of workflows in other processes, if needed page 7 of 13

Workflow Monitor and Logging: What Happens? The visual workflow monitor gives an overview over all executed workflows. It shows all currently and past running workflows and their subworkflows, including their states and execution details. It informs you about what a workflow is doing right now, and where problems occurred. Locally executed tasks as well as jobs sent to a batch queuing systems are monitored and can be stopped. In addition, information about execution details is written to log files. These files contain valuable information for workflow users and particularly for workflow administrators if troubleshooting. Log messages are written in XML-syntax together with meta information such as their importance, time, or source. They can also contain output or error output of applications. System-wide as well as user-specific log files for each workflow run exists. Monitoring of running computations Control of running workflows Easy tracking of problems Statistical information of workflows can be generated Setting Execution Details: Workflow Configuration and Execution Strategies flowguide distinguishes between the process of work and the details of its execution. The process itself rarely changes, but the execution details are usually subject to regular changes. Compute servers change their names, the version or installation path of applications might change, or you may want to execute the flow or parts of it via a batch queuing system on a compute server instead of computing on the local workstation. Workflows are to be used by other departments which use another computer infrastructure, or, most challenging, your processes run on Unix and you want to port them to Windows or the other way round. flowguide is designed to make these changes as easy as possible, without spending much time to adjust the workflows. It provides two related concepts to separate execution specific information from the process description: Workflow configuration and Execution page 8 of 13

strategies. Workflow configuration means to define configuration parameters in separate configuration files. These files typically contain information as host names, application paths, or the execution strategy to be used. Workflows read these files when they are opened. To ease administration, several workflows can share configuration files. Workflow configuration is typically done by the workflow administrator, but they can allow workflow users to override some parameters, if desired. An Execution Strategy defines how and where a workflow is executed. flowguide knows several ways how to execute a workflow. The standard way is to execute a workflow by the Workflow Engine. This gives flowguide full control over the execution of the workflow. Workflow-Management with flowguide: Whitepaper But often parts of a workflow are sent to another computer or they are Execution strategy 'LSF' (batch queuing system) executed by a batch queuing system. If demonstrated in example workflow CmdLineLSF a workflow follows some restrictions, it can be translated to a shell or Perl script. These scripts can be sent to any other computer and executed there even if there is no flowguide installed. This kind of execution is called external execution. Several sub-methods of this execution strategy allow to easily run external workflows on the local host, using a remote shell as rsh or ssh, or via batch queuing system. flowguide automatically transfers the input and output parameters of the workflows across the computers. It also provides a workflow for easily copying files across them. Easy administration of workflows Remote execution of workflows possible Simple workflows can be executed on computers without flowguide installed Cross-platform computations are possible Portability of workflows across platforms page 9 of 13

Implicit Parallelism: Rule-Based Execution of Commands The workflow administrator can create workflows by just adding commands or subworkflows in sequential order, possibly also including if/else conditions or loops. But for real-life examples of workflows, this is often not sufficient. For example, if you have the four tasks A, B, C, and D. First A is to be done, then B and C - if possible simultaneously -, finally D. There might be reasons not to execute B and C consecutively but in parallel: It might save time, or, as an example, C should periodically copy intermediate results which B produces to another directory. In addition to the procedural description of flows, flowscript knows the concept of rulebased execution, which is much more appropriate for the description of workflows than other concepts. Each action is performed, if some condition for it is fulfilled. In this example the rules for B and C say: run if A was successful. The rule for D is: run if B and C were both successful. This is all you have to do for a parallel execution of workflows. Even for arbitrary complex workflows, flowguide detects what can be done in parallel. You do not have to care about the hassle of multitasking. Programmers might already know the principle of rule-based execution from 'makefiles', which are used to describe dependencies between program files. Conditions for rules typically contain the states of workflows or conditions of variables or expressions. If the state of a workflow changes to 'failed', a different action can be executed as when successful. Easy and natural formulation for tasks Automatic parallelization even of complicated workflows Writing Workflows with Python flowguide comes with a built-in Python interpreter 1 and allows to execute Python commands (evaluations) on flowguide variables. These commands can range from simple operations on numbers or concatenations of strings to arbitrary complex commands or calls of Python functions. For more complex operations, it provides a very powerful atomic workflow which allows to include Python scripts directly in a flowguide workflow. These Python workflows are seamlessly integrated into flowguide and allow to exchange variables in both directions with flowguide. Input and output variables of Python workflows are directly accessible as variables in these scripts, which makes exchanging variables very easy. The included Python interpreter also gives access to the extensive Python library, which provides methods for string handling, mathematics, operating system commands, network programming, data conversion and many more. In contrast to shell scripts, all what is done in Python runs on Windows, Unix/Linux and all other operating systems.. 1 More precisely, it uses the Java-implementation Jython, see http://www.jython.org page 10 of 13

The very popular and easy to learn scripting language Python offers almost unlimited possibilities for writing workflows. However, the use of Python is an option for those who are interested. You can use flowguide without any knowledge about Python. Full power of Python included in workflows Easy exchange of variables between flowguide and Python scripts Access to the extensive Python Library in workflows Portable independent of operating system Unlimited Extensibility: Writing Workflows in Java flowguide is completely written in the programming language Java. flowguide also allows to write workflows in Java. Java workflows have all possibilities of flowscript workflows. Opposed to Python workflows they can have all kinds of subworkflows. Most of the basic workflows delivered with flowguide are written in Java. Java workflows are an open interface to flowguide which provides the full flexibility to tightly integrate it with other applications, such as optimization software. Via the Java Native Interface (JNI) you can directly connect libraries and programs written in other languages such as C or C++. Writing workflows in Java is an easy task for Java programmers. Simply derive a class from a special base class, define the interface variables, and implement just the methods you need. Full power of the Java language available for workflows with unlimited possibilities Access to all Java libraries, and via JNI-Interface to C and C++ libraries Open interface to tightly integrate other applications in Java or the other way round page 11 of 13

flowguide Server With the flowguide server, workflows are easily available on other computers. While the workflow engine resides on the server, the Quick-GUI and the workflow monitor run on the client computer. Workflow installation and configuration is all on the server, which releases the need for workflow administration on the client. After workflows are started, clients can disconnect and reconnect at any time to check the status of the workflows. As with the standalone version of flowguide, the workflow engine can execute workflows on the same host or send them to other servers or to a batch queuing system depending on the execution strategy. flowguide server is an excellent tool to distribute workflows within a department or everywhere within the LAN. Together with the 3rd-party tool EnginFrame, flowguide offers a solution to provide workflows through the web for easy use with a webbrowser within the company or all over the world. Client Client Client flowguide - Server Client PDM Clients can be removed (Laptop) or rebooted (Windows) while workflows keep running Easy way to run software on other platform (e. g. client is Windows, while applications run on Unix) Simplified administration of workflows since it is all on the server page 12 of 13

3rd Party Tools with flowguide Workflow-Management with flowguide: Whitepaper flowguide's open interfaces allows to easily connect to other applications or to integrate flowguide in other applications. This section presents two applications, which are seamlessly integrated with flowguide and are an ideal completion of it. Optimization and Robustness Analysis with OptiSLang When automating design computations the demand for automating parts of the design itself also arises. The product OptiSLang is a software platform for optimization and stochastic analysis, which is fully integrated in flowguide. It offers gradient methods, stochastic search strategies, as well as response surface methods for the solution of general parameter optimization. A variety of statistic methods is available for robustness analysis, the finding of sensitive parameters and instable system answers. OptiSLang provides simple-to-use predefined workflows for defining the optimization problem, as well as for optimization or stochastic analysis and for postprocessing. The integration in flowguide allows OptiSLang to use complex environments for the solver runs in heterogeneous networks. Even "non-optimization specialists" can use multidisciplinary optimization and stochastic analysis in an easy and safe manner. Technically, OptiSLang is a workflow within flowguide, and it can be used to optimize or analyze a design by using any other workflow, starting from trivial calls to a single application to complex workflows which are distributed among several platforms. EnginFrame - Executing Workflows through the Web The flowguide server distributes workflows to other computers within your LAN. EnginFrame goes one step further and distributes it through the Web. The client only needs a web-browser. Providing workflows through the web is a matter of seconds, after the software is installed. flowscript provides a clear interface for all workflows. The same interface definition which is used to create the Quick-GUI can also be used to automatically create all necessary description files for EnginFrame. You only have to call a translation program, and the workflow is accessible through the web. Workflow-Engine Firewall EnginFrame Server page 13 of 13