Best Practices in Model Development and Maintenance Adam Rose Product Manager, XP Solutions

Transcription

1 Best Practices in Model Development and Maintenance Adam Rose Product Manager, XP Solutions adopted from Best practices for software development projects Mike Perks Solution Architect, IBM Software Forward: This paper is intended to provide best practices and general guidelines in the development and maintenance of hydraulic models. Each project, client, organization and group is different and as such, there is no singular best way to develop and maintain a model. However, there exist some useful and standardized approaches to assist the developer or architect of a model (or series of models). While there exist numerous resources on the technical development of a model, I have found fewer resources on the meta development of the model: that is, the ways and means of actually creating a model that are best suited for lasting success and easier maintenance. One of the more robust areas of best practice developments is in the software industry. As such I patterned my discussion around one of the more simplified and straightforward resources I have used in my own software development work. About the Author: Adam Rose is the Water and Wastewater Products Manager for XP Solutions, a global producer of software for a sustainable environment. He is a registered professional engineer (1), a certified floodplain manager (2), a GIS professional (3), and was certified as a Project Management Professional (4). He has volunteered at state, regional and national levels for organizations such as AWWA and WEF and was part of the review team responsible for the AWWA M32 Manual: Computer Modeling of Water Distribution Systems. Prior to joining XP Solutions he spent more than a decade at private consulting firms developing potable water, reuse water, wastewater, storm water and atmospheric models. Prior to consulting he spent time at the Texas Natural Resource Conservation Commission (now TCEQ) and completed a master s thesis related to aerosol behaviors

2 How to use this document: This document chiefly serves the internal stakeholders of an organization. A hydraulic model does not exist in a vacuum, but rather within the structure(s) of an organization. The table below lists the different internal stakeholders, their typical model related interests and an assigned level. Throughout the document you will see reference to this level to provide guidance on the most useful audience of the topic. Stakeholder Interest Level Company Improve bottom line improve efficiency and manage risk 4 Group Build capability gain revenue/market share and customers 3 Project 1 Project 2 Project Managers Project 3 Project 4 2 Project 5 Staff Project 1: Task 1 Task 2 Project 2: Task 4 Project 4: Task 1 Project 5: Task 1 Task 2 Task 3 1 You may find more value from finding the level that is of most pressing need and spending more effort on those topics. Perhaps you will find effort from utilizing a capability maturity model on one or more areas to see the greatest room for improvement. Or perhaps you find it best to copy this document and make changes to it to make it your own guidebook into the future. Just as there is no one way to structure a model, there is no one way to use the resource. This work is licensed under a Creative Commons Attribution ShareAlike 4.0 International License.

3 Sections in this Document Item Level Name User Notes 1 4 Define the Development Process 2 4 Measure Development Process Progress 3 3 Define Model Interaction 4 3 Define the Architecture & Design of the Hydraulic Model 5 2 Project Management of the Model 6 2 Hydraulic Modeling Requirements 7 2 Testing & Validation 8 2 QA/QC & Peer Review 9 2 Project Maintenance 10 2 Project Close Out 11 1 Construction of the Model

4 Number 1 Item Define the Development Process Level 4 Summary The definition of the actual, overall process(es) you utilize is important both for consensus as well as continual improvement. Explanation This is the overall way to handle everything related to model maintenance practices, not to be confused with the technical approach/best practices to build a model (which is also covered in this document). A user defined version of this document would serve as a good starting point to a development process. There are three different but related aspects typically found in during a modeling effort: Technical lifecycle: this may or may not be tied to a modeling platform or a physical system. This is the overarching pattern and technology used to perform a successful project by using a specific model or collection of models. Project lifecycle: this is the entire project, of which the hydraulic model may make up a small or significant portion. Often the larger project will be the driver for the client/stakeholders, with other constraints that will drive the hydraulic model in different directions than if the model existed in a vacuum. Model lifecycle: this is the complete development of the model for only a specific project. It is important to choose the appropriate development process to the project at hand because all other project activities are derived from the process. It is likely that this process will need to diverge for different major physical systems: the industry standard naming conventions and model building workflows will likely be different for a stormwater model versus a potable water model (not to mention the ways that most modeling software will handle technical workflow issues like scenario management and object inheritance).

5 Number 2 Item Measuring Progress of the Development Process Level 4 Summary Develop an independent way to measure the success of all of the parts of the development process. Explanation One can measure the overall model development process against an industry standard, such as the Capability Maturity Model (CMM) from the Software Engineering Institute at Carnegie Mellon University. On a scale of 1 5 (ad hoc to well defined), most processes operate at level 1 initially. It is of note that processes will not progress unless actively developed: ad hoc processes may gain in efficiency but will not by themselves evolve appreciably. The CMM/CMMI approach is a high level approach, so while it will be suitable to most development processes, it will also require effort to make the model fit the bounds of the process. Similar approaches have been made by other organizations, such as the AWWA WLC Audit Software (1), which scores a utility not only on the quantity of water loss but also on the quality of the data that supports the result. Different data quality scores drive different initial effort for the utility using the program. This is just one example of pre defining the scoring metrics to help assist the development of an overarching approach to model development and maintenance. Many organizations already maintain some kind of structure to score the progress of a process or approach perhaps as part of a project management or human resources initiative. While these approaches will require modification to fit the technical requirements of model development, they often already include language that is indicative of the overarching culture and requirements of the organization. At a minimum, it is suggested that the user provide some scoring metrics for the items in the development process (whether they are from this document or generated in house): one goal should be to continue to progress the processes in the document until maximum efficiency is reached. 1

6 Number 3 Item Define Model Interaction Level 3 Summary Formally define how all model stakeholders will interact with the model as well as input and output data Explanation Model interaction can be subdivided into four areas: Type Input Model Output Meta Interaction Users & Data Versions Tools 1,2 1,2,3 1, Names and responsibilities of users. This is a practical, brief list of individuals responsible for model data scrubbing, model input, model review, model postprocessing, and any other steps in the modeling workflow. The list should, at minimum, include the name, start/end (if applicable) dates, and tasks of every user. This is especially helpful for longer projects or projects with several interconnected users. 2. Logs of data inputs and model outputs. For data inputs this is a list of the location, provider, quality, notes and responsible user for all input data. Note that there can be significant data that is of value to the model but is not explicitly included in the model. For model outputs this should include the location, applicable model attribution that generated the data, user and other notes related to the output. This is especially important for results that cause a project decision, even if those results are not explicitly included in a deliverable. If possible this should be tied to #1. 3. Logs of current model state. This should include the file name of the model as well as any other useful, platform dependent information, such as a longer description of scenarios and parameters as well as other information that may be of forensic value. If possible this should be tied to #1. 4. Software Version Control. The software version and computer hardware used to run the hydraulic model should be stored. Changes to either software or hardware can produce slight model output variations or model performance changes. This can also be called configuration management: knowing the state of all artifacts that make up your system or project and managing the state of those artifacts.

7 Number 4 Item Define the Architecture and Design of the Hydraulic Model Level 3 Summary Define an area of best practices for your specific physical model: conventions, standards, best practices, etc. Explanation The time for developing the final hydraulic model architecture and design is best set during lower than normal active model development this will reduce the bias impressed by the most recent modeling activities. There will be some standards that are likely platform (and perhaps physical system) independent, and there will be some standards that will be set to meet the demands of the modeling platform used. The name, version and any extensions of these platforms should be kept as part of this step. Some organizations expand specific parts of this item and develop technical excellence programs or internal or external training on specific technical aspects of modeling building. This helps with uniformity of model building and can also assist with both ongoing staff training and initial staff onboarding. It should be noted that any such development must be maintained to stay relevant to the organization and the software. While the total and complete list of design features will vary based on the organization, some commons features include: 1. Naming conventions inside models as well as file naming conventions for models. 2. Design patterns scenario/parameter binding 3. Default values (and their basis) 4. Lists of best/industry/company practices. The AWWA M32 Manual is an example for water distribution system modeling. 5. Regulatory requirements or standards, such as minimum pressure in a water system, minimum velocity in a sanitary sewer system, or minimum pipe diameter in a storm water system.

8 Number 5 Item Project Management of the Model Level 2 Summary How to manage the hydraulic model is related but typically different than how to manage the project that relies on the model results. Explanation Project management is one key to a successful project, and most successful projects have either a detail project management plan or a successful manager that utilizes a number of heuristics. However, most hydraulic models are represented as abbreviated and simplified steps in the context of an overall project management plan. The differences of an organization and individual projects make development of an effective, comprehensive project management guideline beyond the scope of this document. However, there are some basic considerations that should be reviewed and addressed. The figure above qualitatively represents the progress of a hydraulic model in the context of a project. The first section is typical of a model building phase, and will progress rapidly assuming required data is available for model construction. A large amount of time often passes with relatively little additional model development this can be because of review time or development of other deliverables (reports). When it is time to complete the project there is often a final push to incorporate comments and final data. The development of the management approach should consider the impacts of the duration and slope of each section. For example the fast finish of many projects leaves little time or budget for plan do check act (PDCA) or lessons learned behaviors.

9 Number 6 Item Hydraulic Modeling Requirements Level 2 Summary Defining the requirements for the project explicitly is not often done but can be quite useful for a variety of reasons. Explanation Gathering and agreeing on requirements is fundamental to a successful project. This does not necessarily imply that all requirements need to be fixed before any architecture changes or model development is done, but it is important for the modeling team to understand what needs to be built. This type of data might exist in a project management document but this is a more technical application (one that could be templated and copied into future projects if practical). Requirements often change throughout the project but should be documented for final project review and for planning future projects. Requirements can be divided into internal and external. Example internal: driven by model revelations Example external: driven by client expectations Quality requirements are broken up into two kinds: functional and non functional. Example functional: what model needs to do (address CIP needs, validate historical system response, etc) Example non functional: the standards of model the model does what it needs to do (calibration level, documentation, number of exhibits, etc) Functional Non Functional Internal Defined only in general terms as part of the process definition (Item 4) Heavily defined as part of process definition (Items 3 7) External Based on contract Rarely included in contract This approach can be good for disputes in budget, schedule, accuracy, or other project constraints.

10 Number 7 Item Testing and Validation Level 2 Summary Prepare model benchmarks to support accurate and confident results Explanation This step should not be confused with quality control or quality assurance. Testing is not an afterthought or cutback when the schedule gets tight. It is an integral part of model development that needs to be planned. It is also important that testing is done proactively; meaning that review points are planned before modeling starts, and further reviews are planned while the model is being designed and built. Examples might include: Initial model results not substantially above other model results indicative of too large a timestep Some modeling platforms support efficient and error logging goals for these kinds of metrics should be planned and met, as practical, during the model development and not as part of a final model review. Examples might include: Continuity error less than 2% Convergence with less than 50 trials This is something that the main developer of the model should be responsible for maintaining and reporting, even if the developer is not responsible for the development of the benchmarks themselves. Some organizations may combine this item with Item 8 in the form of larger checklists of model validity.

11 Number 8 Item QA/QC and Peer Review Level 2 Summary Define the formalized ways that a model is reviewed and certified as acceptable for a project. Explanation It is important to review other people's work. Experience has shown that problems are eliminated earlier this way and reviews are as effective as or even more effective than testing. If the entire framework developed as part of this document is complete, it should all be reviewed as part of the modeling project. Note that most of the framework would only be reviewed as part of an ongoing review and should not appreciably, negatively impact the project schedule or budget (if done consistently). To enforce consistency and to ensure efficient review, many organizations opt for checklists as a means for quality control (and at times quality assurance). By their very nature these types of review documents will vary by physical system (water distribution systems will require different parameters than river systems). The workflows that modelers develop are also often defined, to some extent, by the modeling platforms utilized during the project. As such these differences should be documented as part of the checklists (or other suitable review formats). Many organizations that develop and support software will have some form of review documentation and/or checklists to support efficient use of their software.

12 Number 9 Item Project Maintenance Level 2 Summary There are additional requirements to support the success of long duration modeling projects Explanation Best practices for maintaining model for longer periods. Although a subjective term, this is intended for projects that span multiple calendar years and may not receive direct user support for extended periods. It should be noted that the practices outlined in this section are still applicable for all models, but might not fit project constraints based on the client and the organization. The hallmark of long duration projects is additional care taken towards the inevitable change in users, software (or at least software version), hardware, and input data. There is no simple method more effective to combat these additional challenges than documentation. The technological approach to documentation will vary by organization, but at its heart should consider at least the following: The responsibility assignment matrix, relative to major milestones and decisions in model develop. This matrix includes those that are responsible, accountable, consulted, and informed of decisions (more information related to this method can be found online). More detailed data input and model output reporting. Some organizations consider a modified chain of custody to enforce sufficient documentation of data input. Logs of access to models on the file server, including dates and times of edits and authenticated users. It should be stressed that proper initial design and project execution will assist with this level of documentation. The level of effort required to maintain a project to this level of detail is seldom budgeted in a project, but as the duration of a project increases the risk of failure* increases. Above all, a long term project is subject to forces beyond the control of the model developer, model manager or project manager. As such the processes defined as part of this item, or as part of this entire exercise, must be comprehensive and flexible enough to absorb the change. *in this respect failure is defined as reaching a state in the modeling process where the definable structure (or substructure) of the model cannot be located, such as: not being able to substantiate fire flow requirements, not having defined wet weather sanitary sewer unit flows, or not being able to locate the user or file that supplied the final development extents of a river model.

13 Number 10 Item Project Close out Level 2 Summary Defining lessons learned and appropriately filing the hydraulic model are import parts of every project Explanation As is evidenced with by the graph in Item 5, the final steps to project closeout could be accomplished separately hydraulic model. This item is developed specifically to capture the lessons learned for the project and to ensure that the final steps in the project are defined sufficiently for future reference. As these are two different goals, they are described separately below. Last Minute Changes These changes can/will create disconnect between project and software that will persist into document storage if time is not allowed to make complete changes. A first step is to set up development documentation to make changes most efficient, but time must be allotted, even if after project submitted, to make final changes to make model and deliverable consistent. At a minimum some form of log that defines the variation between the hydraulic model and final deliverables should exist: many times engineering judgment will allow for recommendations and conclusions to diverge slightly from model output based on final parameter and/or client changes. Lessons Learned A format and location of modeling lessons learned is encouraged for every organization. This can be as specific and technical as normalized demands/flows, as generalized as best formats for output graphics, or even related to ideas for project management of future projects (scope, schedule, fee or otherwise). Organizations could also choose to create temples or global databases. The goal is generate content that will help the organization better handle similar projects into the future.

14 Number 11 Item Construction of the Model Level 1 Summary The actual construction of the model Explanation The discussion of actual model construction is not covered until the end of this document, and that is by design. While there is no substitute for skilled modelers, and there is no checklist or format that will replace the skill and judgment of a hydraulic modeler, it should be noted that there are many steps that can and should occur prior to the first steps a hydraulic model development. At the end of the day the experienced modeler should be given some free reign to develop the hydraulic model in the way that they best see fit. Any model specific guidance should be provided here, but most overarching model requirements should be defined in other portions of the development document. Examples might include (perhaps taken from project specific information like contracts or stakeholder requirements): Develop a Monte Carlo simulation to support random water demands Use a specific antecedent moisture condition as part of a sanitary sewer model Assume no maintenance when determining roughness as part of a river model Use the municipal standard 2D grid size when running an urban stormwater model