A Case Study in Software Enhancements as Six Sigma Process Improvements: Simulating Productivity Savings Dan Houston, Ph.D. Automation and Control Solutions Honeywell, Inc. dxhouston@ieee.org Abstract This case study illustrates the application of Six Sigma process improvement to software upgrades justification. Specifically, software use was simulated based on business process maps and data provided by users. The simulation results were combined with upgrade investment estimates to calculate the return on investment and justify the upgrades. 1. Introduction Information technology (IT) has become recognized as an essential component of business, providing the platform for a wide array of business processes. When considered in this context, IT enhancements may result from process improvement projects. IT software upgrades can often be justified in terms of savings due to increased productivity. However, in a Six Sigma organization, the DMAIC method and a host of tools can be brought to bear on business process improvement through software enhancement. This paper presents such a case study, the Configurator ROI project, relating the application of Monte Carlo simulation as a Six Sigma tool to identifying productivity savings in the deployment phase of a software lifecycle. Information technology software enhancements are performed for many reasons, including better customer service, better decision support, higher productivity, and so forth. Return on investment (ROI) is ordinarily calculated in terms of a focused objective for savings. Many of these enhancements either provide support for, or implement improved business processes, but the ROI is typically produced solely from a financial analysis, and not necessarily in the context of a process analysis. 2. Six Sigma DMAIC Six Sigma is a quality improvement program that looks at processes with a view to analyzing process steps, determining what process elements most need improvement, developing alternatives for improvement, then selecting and implementing one. It relies on a variety of qualitative and quantitative tools, emphasizing the use of data and statistical analysis within a method called DMAIC, an acronym for the names of its five phases (Define, Measure, Analyze, Improve, and Control) (Table 1). Six Sigma projects are typically selected for their potential savings in improving any process, whether it is in production, administration, engineering, or services. A Six Sigma project typically begins with a high level definition of a process, using a diagram to specify the process boundaries, inputs, outputs, customers, and requirements. In the measure phase, a process metric is selected and used to baseline the current performance of the process. In the analysis phase, the process is analyzed, usually with a process map and a failure modes and effects analysis (FMEA), but may include other types of analysis. The process map shows each process step with its inputs and outputs and provides the basis for either a FMEA or a quantitative, usually statistical, analysis. Areas for improvement are pinpointed and alternatives are generated and evaluated. Once an improvement option is selected Phase Define Measure Analyze Improve Control Table 1. DMAIC Method for Process Improvement Steps Identify an opportunity and define a project to address it. Analyze the current process and specify the desired outcome. Identify root causes and proposed solutions. Prioritize solutions; select, plan, validate, and implement a solution. Develop a plan for measuring progress and maintaining gains.
and implemented, the project enters the control phase. In this phase, a plan is established for monitoring and controlling the process to ensure that gains are maintained. The use of the DMAIC method may vary between projects. For example, the Measure and Analyze phases of this project ran concurrently rather than sequentially. Also, a proposed solution may emerge early in the Measure and Analysis phase, leading to an emphasis on planning and implementation in the Improve phase. Such was the case in the Configurator ROI project. Consequently, this paper focuses on the Measure and Analyze phases in which a simulation based on a process map provided the justification for an IT enhancement. 3. Configurators and the Configurator ROI Project Honeywell Industry Solutions, a division of Honeywell International, produces and services control systems for industrial process such as refining, power generation, and paper production. These complex control systems consist of a multitude of components controllers, sensors, monitoring devices, network cabling, signal converters, and so forth that must be configured to each customer s specifications. Over the years a number of software packages have been developed for configuring the various Honeywell control systems, each configuration package supporting a control system serving a particular market. A product configurator is essentially a rule-based package that specifies all the necessary components for a proposed control system, prices the components, then provides a set of reports, including a bill of materials. Salespersons and estimators use the configurators to develop proposals for customers. Each configurator was predicated on different business processes. With corporate globalization, the inefficiency of using multiple processes and software programs for system configuration became evident. In April 2001, a division-wide study of configurator requirements was undertaken with a Voice of the Customer (VOC) process. This process solicited input from sales, marketing, manufacturing, engineering, and information technology departments, analyzed it, and produced a set of high-level requirements for a comprehensive product configurator. This case study discusses the follow-on project, Configurator ROI, in which detailed requirements were specified and used as the basis for estimating ROI for the proposed upgrades to the TotalPlant Configurator (TPC), the primary existing product configuration system. Figure 1 illustrates the Measure and Analyze phases of the Configurator ROI project. Calculation of the ROI was broken down into two parts, the investment calculation and the productivity savings calculation. The investment calculation was performed by (1) identifying differences between the current configurator tools and the configurator requirements derived from the VOC process, (2) proposing engineering solutions to close the gaps, (3) using a quality function deployment matrix to evaluate the solutions against the requirements, and (4) estimating the cost of each solution. As the Configurator ROI project began, the solution was undecided. Two alternatives were available. These involved upgrading either one of the two leading Current Tools System Requirements Identify Gaps Estimate Upgrade Costs of Each Alternative Calculate ROI To-Be Process Map Survey System Users for Productivity Data Simulate To-Be Process Forecast Productivity Savings As-Is Process Map Simulate As-Is Process Fig. 1. Configurator ROI Project Diagram
Configuration and Proposal Order Entry Bill of Materials Fig. 2. Primary Functions of a Complete Configurator configurators, TPC or ebob, to address all of the requirements elicited during the VOC process. These two systems addressed, to varying degrees, the requirements of a complete system for configuration and proposal, order entry and Bill of Materials (BOM) for manufacturing (Figure 2). Though ebob provided much more flexibility in delivering a BOM and its data was automatically transferred into the order processing system, its configuration capabilities were largely manual because it lacked a rule base and expert system engine. TPC, on the other hand, had a well-designed and up-to-date rules base and expert engine, but needed more BOM flexibility and order entry capability. During the process of estimating the cost of each solution, it became clear that upgrading ebob would cost roughly twice the upgrade cost of TPC. From that point on, the project focused on the expected ROI for upgrading TPC. The productivity savings calculations were produced using a Monte Carlo simulation of existing productivity and comparing these results with those of a simulation of expected productivity after the upgrades. The remainder of this paper dwells on the details of data collection and simulation. 4. Simulation Approach Although many groups benefit from a product configurator, the primary beneficiaries are the estimators who develop proposals and provide quotes to salespersons and customers. This group of configurator users has defined a complex process for developing proposals. The paths of the proposal development process depend on the size of the proposed system, whether engineering services are to be provided, and the degree of additional consulting required. The Configurator ROI team obtained the process map for the proposal development process (fourteen 11 x17 pages) and used it to identify the process flow directly affected by configurator usage. The elements of the proposal development process specific to configurator usage were extracted into the process map of Figure 2. The process shown in Figure 3 branches depending on the number of personnel required for developing a proposal. The top branch is taken when a Account Manager develops a proposal without assistance. When Sales Cost Consultants are used, Engineering and Installation (E&I) services may or may not be proposed. When E&I are not being proposed, a Cost Team may or may not be required. If E&I services are being proposed, other services may or may not be required. In addition to the branches shown in Figure 3, the simulation had to account for various proposal sizes. Three proposal sizes were distinguished: small (< $100K), medium ($100K to $1M), and large (>$1M). A Monte Carlo approach to simulation was chosen for two reasons. First, the feedback loops in the process map are only around single or paired steps in series. This suggests that the combination of process and feedback loop can be simplified to only the process steps by treating the duration of each process step as an aggregate duration rather than discrete multiple durations. Secondly, the process steps of interest lie on independent paths. The lack of path dependencies means the process has no timing requirements and a simulation of it does not require a calendar. A simple Monte Carlo approach to simulating this process requires only two types of random variables: discretely distributed branching for proposal size, and continuously distributed effort applied in each process step simulated. Although the Configurator ROI team had access to simulation tools, other groups interested in the method and results do not. Accessibility to Monte Carlo simulation examples is highly desirable in the Industry Solutions business because simulation is not widely used for modeling business and development processes, but its value is recognized by the Six Sigma organization. Taking advantage of the simplicity of this simulation problem, Microsoft Excel, which is standard for desktops in the division, was chosen as the simulation vehicle. The additional effort required for programming the simulation in Excel rather than in a simulation program was justified by its accessibility to other teams and its usefulness for teaching simulation in Six Sigma courses.
SALES PURSUIT AM Only or Use SCC? AM Only AM prepares proposal using TPC/eBOB to obtain standard TPS hardware and software model numbers and list pricing. AM provides the TPS prj file / e.bob index number to SCCA for placement in job file and on server. Use SSC. of jobs. of jobs Is all required info included? SCC finalizes TPC and/or e.bob, creates block drawings, researches new products and secures other cost items. SCC obtains Honeywell hardware costs through HIET Data Report for TPS hardware or e.bob for PlantScape hardware. Modify proposal?. of jobs. of jobs Is this an E&I opportunity? Is Costing Team required? SCC reviews and completes TPC/eBOB file and saves the following reports: Bill of Material, Calculated Values, Service Data, HIET Data.. of jobs Other services required? SCC finalizes initial TPC/e.BOB file submitted by TPAM, creates block drawings, researches new products and generates other costing information. Modify proposal? Add'l SCC resources required?. of jobs Customer clarifications? Finish AM = Acount Manager SCC = Sales Cost Consultant E&I = Engineering & Installation Fig. 3. System Proposal Process Map for Configurator Usage 5. Data Collection Data had to be collected for two configurators: TPC and ebob. TPC was used for configuring TPS systems and ebob was used for configuring PlantScape systems. The project sought to eliminate one of the configurators and consolidate support for both TPS and PlantScape in the other. By the time the simulation was being developed, TPC appeared to be the better configurator for upgrading, and the decision was made to estimate the productivity savings and ROI for this choice. Data for the simulation was produced from three sources (Table 2), a proposal manager questionnaire, a survey of estimators, and the 2001 PlantScape order entry data. Because proposal size is a significant factor in the effort required to produce a proposal, the simulation model had to provide branches for proposal size and the probability of branching was based on the Table 2. Sources of Model Data TPS Proposals PlantScape Proposals. of Proposals in 2001 Proposal Manager Questionnaire 2001 PlantScape Order Entry Data As-Is Effort Estimator Survey: TPC data Estimator Survey: ebob data To-Be Effort Estimator Survey: TPC data Estimator Survey: TPC data
Table 3. Number of TPS Proposals (last 12 months) Job Size Personnel Involved Small Medium Large AM only AM+SCC, n-e&i AM+SCC+CT, n-e&i AM+SCC+OS All fraction of each proposal size produced in 2001. For TPS proposals, these fractions were calculated from the numbers of TPS proposals of each size passing through each configurator usage path (Figure 3), as provided by the proposal group manager in response to a questionnaire (Table 3). Because PlantScape is a much smaller system, proposals for it are entered by many people, so the 2001 PlantScape Order Entry datafile was used to determine the number of PlantScape proposals of each size. The process map of Figure 3 was validated after the results of the proposal manager questionnaire were received. It was found that the Account Manager-only branch was used very infrequently and only for small proposals. Therefore, this branch could be eliminated from the simulation model with an insignificant effect on the results. The validation exercise also revealed that, despite what the proposal process map indicated, the use of a Costing Team made no difference with regard to configurator usage effort. Therefore, the numbers of proposals for these two paths were merged. These simplifications left only three steps to be simulated for each proposal size. Table 4. Estimator Data Form for As-Is Effort Job Size Process Step Configurator Small Medium Large SCC finalizes TPC and/or ebob, creates block drawings, researches new products, secures other cost items, and saves the following reports: Bill of Material, Calculated Values, Service Data, HIET Data. TPC ebob SCC obtains Honeywell hardware costs through HIET Data Report for TPS hardware or ebob for PlantScape hardware. TPC ebob SCC finalizes initial TPC/eBOB file submitted by TPAM, creates block drawings, researches new products and generates other costing information. TPC ebob Table 5. Estimator Data Form for To-Be Effort Job Size Process Step Small Medium Large SCC finalizes TPC, creates block drawings, researches new products, secures other cost items, and saves the following reports: Bill of Material, Calculated Values, Service Data, HIET Data. SCC obtains Honeywell hardware costs through HIET Data Report for TPS or PlantScape hardware. SCC finalizes initial TPC file submitted by TPAM, creates block drawings, researches new products and generates other costing information.
The second survey solicited data from thirteen estimators about the amount of effort actually spent on each of the three steps for proposals (Table 4) and about the amount of effort that could be expected after the TPC upgrades described in the survey (Table 5). The proposed upgrades included enhancements for facilitating the use of TPC, as well as providing support for systems other than TPS, such as PlantScape. Continuous distributions were derived from the effort data obtained in the estimator survey using an input analyzer. Distributions having closed form inverse transforms were used: discrete, triangular, Weibull, exponential, and uniform. Excel formulas were written for inverse transforms of the effort distributions. 6. Random Number Generation Using Excel introduced a special problem for simulation because the quality of Excel s random number generator (RNG) is debatable [1]. However, the quality of any RNG s random number streams may be highly dependent on the choice of seeds. Use of seeds is desirable not only for common random numbers in reducing variance when comparing scenarios [2], but for choosing those seeds that produce the best pseudorandom streams. The Excel RNG was tested with 10,000 seeds using empirical tests for both uniformity (Chi-square and serial tests in two and three dimensions) and independence (runs-up) on streams of three different lengths, 600, 6000, and 60000 numbers. These stream lengths were chosen based on the stream lengths needed for the simulation model. The best seeds were selected using composite rankings for the tests across the four empirical tests and across the three stream lengths. Seed Rank = R ij R ij = rank for the i th empirical test and the j th stream length Each selected seed produced the three different streams for which each empirical test null hypothesis could not be rejected α=.05. A different random number stream was employed for each random variable because a high number of simulation runs was expected, making independent streams desirable. Since Excel s RNG lacks the capacity to sample multiple random number streams alternatively, streams were generated by a Visual Basic script and stored, one stream per spreadsheet. Fully synchronized common random numbers were used to reduce variance in the difference between the two cases, allowing calculation of variance from the difference of each pair of As-Is and To-Be runs. 7. Simulation Model An Excel worksheet was allocated for each of the four simulation models: TPC As-Is, TPC To-Be, ebob As-Is, and ebob To-Be. The first two columns of Table 6 are an excerpt from TPC As-Is, showing how the model works for a single process step and one proposal. The Description column in Table 6 explains the numbers in the second column. Table 6. Sample of Simulation Model: As-Is Proposal 1 in the AM & SCC <& OS>, E&I Path Path: AM & SCC <& OS>, E&I Description Proposal. 1 Row contains consecutive numbers. Fraction medium proposals (FracMed).186 Constant for this row. Fraction large proposals (FracLar).814 Constant for this row. Random number for branching proposal size (RN6).79504 VB script reads a stored random number stream and enters this number. Step 3: configuration Random Number for proposal prep effort (RN7).64468 VB script reads a stored random number Effort distributions by proposal size stream and enters this number. Medium 0 =IF(RN6<FracMed, InverseTransformForMedium(RN7), 0) Large 15.655 =IF(RN6>=FracMed, InverseTransformForLarge(RN7), 0) Total Effort for All Proposals on this Path 4692 Sum values in Medium and Large rows. Total Effort for all paths 19478 Sum of Total Effort values for all paths.
Proposals are numbered sequentially in the Proposal. row. The number of proposals for a path is number of proposals reported for that path by the proposal manager. The fraction of each size proposal (FracMed and FracLar in Table 6) is also calculated from the proposal manager s report (this path did not have any small proposals). For each process step, two random numbers were used: one for branching on proposal size and the other for calculating the effort in the process step. A row is allocated for the effort distribution corresponding to each process step in the path. The path shown in Table 6 had medium and large proposals, so a row was allocated for each of the effort distributions, one for medium proposals and one for large proposals. The branching pseudorandom number determines which of the effort distributions is used. Subject to this condition, the proposal effort is calculated using the effort pseudorandom number in an inverse transform. The effort figures for each path are summed and the effort sums across paths are totaled. The simulation is run by a Visual Basic (VB) script. The script populates each model, calculates the worksheet, and stores the total effort on a Results worksheet. Each year represents all the proposals for 2001. The scripts produce 1000 runs on each of the four models. 8. Results The As-Is and To-Be runs for TPC were paired through the use of synchronized common random numbers, so the difference between each pair of runs was taken to obtain the expected savings in the To-Be scenario due to TPC enhancements that would facilitate its use. The same was done with the As-Is and To-Be runs for ebob, however the To-Be ebob runs used TPC effort distributions since ebob would be upgraded to support PlantScape proposals. The outputs and savings are normally distributed. Figure 4 shows the normal probability plots for TPS proposals made in TPC and Figure 5 shows the PlantScape proposals produce now in ebob and later in TPC. 99 Savings Percent 95 90 80 70 60 50 40 30 20 10 5 To-Be As-Is 1 0 10000 20000 Person-Hours Annually Fig. 5. rmal Probability Plots for TPC (TPS Proposals) Effort: As-Is, To-Be, and Savings
Percent 99 95 90 80 70 60 50 40 30 20 10 5 Savings To-Be As-Is 1 4000 9000 Person-Hours Annually 14000 Fig. 6. rmal Probability Plots for ebob (PlantScape Proposals) Effort: As-Is, To-Be, and Savings Taking the two savings together TPC upgrades to facilitate its use and upgrades to support PlantScape proposals the mean projected savings is 6600 person-hours annually with a standard deviation of 757 person-hours. Five of the proposed upgrades would save one to two person-years of effort in producing product proposals (95% confidence interval on range). The cost of the upgrades would be reclaimed in 1.3 to 2.0 years due only to increased proposal productivity. In addition, variation in the configuration effort required for proposals would be decreased 65% (indicated by the slope of the plots). References [1] B.D. McCullough and B. Wilson, On the accuracy of statistical procedures in Microsoft Excel 97, Computational Statistics and Data Analysis, 31 (1999), pp. 27-37. [2] Averill M. Law and W. David Kelton, Simulation Modeling and Analysis, 2 nd ed. McGraw-Hill, New York, 1991. 9. Conclusions and Next Steps Simulation of software system usage, based on data elicited from regular system users, provided a sound, quantitative basis for estimating productivity increases and justifying upgrade investments. The next stage of the project calls for measurement of actual savings and use of a control plan for monitoring variation in improved steps of the process.