The Value of Application Audits Douglas Welker

Transcription

1 The Value of Application Audits Douglas Welker Payoff One of the best ways to promote system controls is to audit applications during development. Unfortunately, many application audits never survive the development cycle. This article explains how to effectively use application audits to correct control problems before a system enters production. A commitment to effective application controls is required of the auditor and the development project team. Introduction The need for an audit is not always apparent at the outset of a project, especially when the development process has been documented with neat flowcharts of steps from a standard methodology. However, as the project matures, the need for an audit may become only too apparent. As project demands increase and budgets are expended, milestones are often missed and completed applications do not function as planned. In response to these pressures, the project team may reduce the scope of the project, ignore system requirements, and fail to produce planned deliverables. The development methodology is viewed as an impediment, and so, in whole or in part, it is jettisoned. As steps and deliverables of a methodology are eliminated, the project team loses mechanisms that promote the development of effective system controls. Technical specialists that develop security controls may not be available in such weak development environments. Because of time pressures and limited resources, the project team usually focuses its remaining efforts on developing basic system functions rather than system controls. It is exactly in this type of weak development environment that an audit can be most effective. Although such an environment can present significant obstacles to auditing applications under development, this article demonstrates the means for overcoming these obstacles. Justifying the Application Audit The decision to conduct an application audit should be based on a reasonable expectation that the audit will contribute to the quality of an application's controls. Nothing else justifies the overhead and development delays that may result if an audit comment leads a team to make system changes. To justify the audit, the following issues should be examined: Does the development process discount the value of application controls? For example, the development methodology is no more than a set of guidelines, project controls are weak, and design review are not required during the development process. Is the audit's objective to incorporate effective controls into an application before its implementation? Does this objective take priority over other objectives, such as reviewing methodology controls or developing related audit software?

2 Is the application intended to help the company achieve significant business goals? Does the application perform a critical processing function? Is the project team unfamiliar with application control issues? If a distinct design phase occurs before the programming and implementation phases, can the audit be scheduled for the design phase? Will sufficient design information be available before implementation to form the basis for control reviews? Is the auditor aware of the need to address control concerns so that solutions can be applied before implementation? Does the auditor understand that failure to address these concerns before implementation may reduce the value of the audit? Is audit management committed to promoting the importance of application controls? For example, will it: Schedule the time and staff needed to do the job? Give priority to evaluating controls rather than diverting EDP auditors to such nonaudit tasks as programming and office automation support? Advocate application controls through substantive reporting, challenges to unsatisfactory responses, and follow-ups on commitments to improve controls? If the answers to the first four questions are affirmative, the audit is warranted but may not necessarily be feasible. If the answers to the fifth and sixth questions are affirmative, the audit is feasible. If all answers are affirmative, the audit can and should be performed. Five Steps to a Successful Audit To ensure that applications contain appropriate controls, auditors should evaluate system designs during the development cycle by identifying meaningful sources of control information, evaluating that information, and providing the results of their analysis to the project team to improve controls before system implementation. Beyond reporting weaknesses, auditors should promote controls as an integral part of application development despite methodologies and project practices that often fail to promote controls. The following steps provide an approach for accomplishing these objectives. Step 1: Setting Audit Objectives and Scope Audit goals should be clearly defined to help auditors focus their review. An audit's value can be diluted if the auditor attempts to pursue every control issue encountered during the project. For example, it may be tempting to include project and development methodology controls in the audit scope; however, addressing both types of controls during a system controls audit can be inefficient for several reasons. First, project and methodology controls are often the result of company policy. If the auditor cites control weaknesses, the project team may ignore them to adhere to company

3 policy. Senior management, sensing that these control problems are isolated, may choose not to act on them either. More can be accomplished if these control concerns are documented over several audits and presented separately to promote changes in company practices. In the meantime, the auditor can achieve more by correcting control weaknesses specific to the audited system. Second, auditors may not be able to address system control issues adequately if they spend time on other topics. Application control issues are interrelated; research into one issue frequently provides answers relevant to others. Application controls are not related to project and methodology controls, however; time spent in research on project controls will not benefit the analysis of system controls. Because of this lack of synergy, an auditor may not be able to cover all three areas within the assigned schedule. Third, auditors should set reasonable goals, especially during the first few audits. System auditing during the development process is uncommon. Project, user, and IS managers may not know what to expect from such audits. To demonstrate the value of this type of review, auditors should set achievable goals for the early audits. This may mean concentrating on application control audits before addressing methodology and project control concerns. Last, the auditor may be tempted to include audit software and program development as part of the audit scope. In truth, the best time for improving system controls is not the best time for writing audit programs and software. Controls should be evaluated and improved during the functional design phase, whereas audit software and programs should be developed during the technical design phase, after the design is completed. The auditor should avoid such diversions and concentrate on the auditing of system controls. Step 2: Selecting an Application If a company is involved in several systems development projects, the need for audit support may outstrip the available resources. The application selection process should ensure that companies get the best return on their audit investment. Auditors should select projects on the basis of the following key factors. Corporate Strategy. Certain systems are implemented as part of a company's strategic plan. In a manufacturing company, for example, management may decide to differentiate itself from its competitors by providing superior customer service. To achieve this objective, it may choose to develop a new order management system. By contributing to the design of effective controls for this system, the auditor can support the success of this strategy. Significance of Controls. Controls are essential for applications in which information first enters the data processing system. For example, the auditor can contribute to the integrity of data throughout the system by promoting effective controls of a new order entry application. Controls Experience of the Project Team. Auditors with a financial background may assume that accounting applications deserve the most audit support. However, financial system project teams usually include people experienced with controls. Auditors should provide the most support to teams with

4 limited controls experience, such as sales, marketing, and inventory teams responsible for developing order and inventory management systems. In selecting applications to audit, auditors should also consider the estimated level of investment for each project and the relative sensitivity of the data for each application. Step 3: Timing It is most cost-effective to improve system controls during the functional design phase of the development project. Although feasibility studies may contain design information, they generally do not contain information that is detailed enough to support meaningful control reviews. Audits can be performed during the technical design and programming phases, although it is expensive to implement recommended changes after programming has begun. The auditor should evaluate controls during the functional design phase, when improvements can be made by changing design documents rather than reprogramming and retesting software. Controls should be evaluated on a timely basis to ensure that weaknesses are addressed before systems are implemented. During the early stages of a project, managers have more time and flexibility in handling control issues. As the project progresses, schedule slack evaporates and resource commitments tighten; resources that were once available for correcting control problems may now be assigned to more basic systems development issues. Timely reporting can make the difference between merely documenting weaknesses and facilitating improvements to system controls. Step 4: Evaluating Controls A successful audit depends on identifying meaningful sources of information for evaluating controls. Unfortunately, controls may not be explicitly described in obvious sources, such as project plans. Auditors may need to base their evaluations on project documentation not directly related to controls. The emphasis should be placed on documentation that is available before the end of the functional design phase. Project Plans. Project plans can give early indications of potential control problems and should include steps for implementing system security. However, even the most comprehensive plan may not include such information. If the developers of the plan are not familiar with the need for security controls, the plan may not specify the required tasks. If this is the case, the auditors should advise the project team. For example, if the project plan does not include steps for establishing security control and administrative capabilities, the auditor should recommend that the plan be modified as follows: Control requirements must be defined and reviewed to ensure that they are consistent with the company's internal control requirements. Controls must be designed, tested, and implemented. Administrative procedures and controls must be documented. Administrators must be trained.

5 Similarly, the plan should include steps relating to such issues as data conversion, service recovery, data retention, and system testing. The absence of a step may be inadvertent, but it may also indicate that the team does not deem the controls important and is not planning to develop them. Inappropriate task assignments may signal weak controls. For example, a plan for service recovery of a system that serves two departments assigns a representative from only one department. If effective service recovery depends on coordination between both communities, the auditor might suggest that representatives of both departments, along with a technical team representative, coordinate system recovery requirements, tasks, and controls. The plan for service recovery should then be amended to include all three parties. In summary, auditors should evaluate the project plan for steps that are either missing or that are defined in vague or contradictory terms. They also look for steps that do not relate to specific deliverables or are not assigned to the appropriate individuals, teams, or business units. Business Partner Agreements. Agreements between business partners can be reviewed to determine whether project teams or business partners are responsible for controls relating to packaged software and applications that interface with external systems, such as Electronic Document Interchange. Agreement reviews are important because most project teams pay little attention to these documents. If teams are not aware of an agreement's content, they are more likely to make incorrect assumptions about interorganizational control responsibilities. They may mistakenly think that the partner will assume responsibilities that are not spelled out in the agreement in order to retain the other partner's business. The auditor should correct this assumption so that the project team is clearly aware of its responsibility in ensuring effective controls. Agreements can be written to establish accountability for such control areas as: Software, hardware, and data security. Software development and change management. Service recovery. User and technical training. If control responsibilities are not specified, the auditor can recommend that they be included in the contract. If an application uses EDI, for example, the auditor could suggest that the EDI partnership agreement take into account the following control issues: Data transmission validation conventions to establish what constitutes authorized and binding electronic messages. Acknowledgment protocols to ensure transmission accuracy and completeness. Data security control responsibilities. Service recovery responsibilities.

6 Regardless of the format or level of detail in an agreement, project teams should be advised that if a contract does not explicitly make a partner responsible for a control, the project team becomes implicitly responsible. Design Documentation. System design descriptions are the most obvious source of information for control evaluations. However, project teams create this documentation at the end of the design phase primarily as input to the next development phase: programming. This documentation is not generally available for evaluation during the design phase. Consequently, the auditor may be able to provide a project team with evaluations only early in the next phase. Some project plans allow a few weeks after the functional design phase for controllers and user department representatives to review and comment on system designs. At this point, the auditor may be able to deliver evaluations before the next phase begins. The auditor can supply more timely control evaluations by reviewing design information as it is completed instead of at the end of the phase. When possible, the auditor should attend design review sessions, known as walkthroughs, in which prospective users and technical support people are given the opportunity to critique the design. Auditors can use walkthroughs to informally learn about control processes and evaluate them before designs are finalized. Regardless of the information source, the auditor should not expect to find clear-cut functional descriptions of controls. Teams concentrate on developing each system's core functions, and design documentation reflects this. Control information may be implied in the design but not stated explicitly. As a result, identifying control weaknesses in a system's design is a challenge. Step 5: Issuing a Timely Report The value of application audits hinges on the timely delivery of control information. An auditor contributes to system quality only if project teams receive the information in time to improve controls before implementation. Control information is useless if it is provided when system changes are no longer feasible. Indications of control weaknesses often surface throughout the project; if the auditor waits to issue the report until a preponderance of evidence has been obtained, it could be too late to incorporate changes. The following example illustrates the consequences of delaying reports on control concerns: A project plan has been issued that contains a step for developing system security controls. The auditor, wishing to evaluate these controls, asks for the task's deliverable. The person assigned to the step claims that the work is being performed by a specialist in the IS department. The specialist claims responsibility only for designing an interface to the security software, not the security software itself. The individual responsible for this task is never identified; therefore, the detailed deliverable for the task is never obtained. A control concern is not reported because the auditor is unable to obtain documentation that evidences actual control weaknesses, although the auditor receives assurances that the issue is being addressed by a group of people. Instead, the auditor waits for clear-cut deliverables that will provide a comfortable basis for evaluation. Over time, security weaknesses become obvious, but management no longer has the resources or time to correct the problems before implementation. As a result, the system enters production with inadequate security controls.

7 An appropriate degree of audit conservatism is desirable, but too much reduces the value of the audit. Waiting can ensure that control comments are substantive, but if they are supplied when improvements are no longer practical, these comments will amount to no more than unconstructive criticism. Conclusion Even if the auditor consistently provides timely and useful control information, project teams may still be skeptical about its value. Teams may be accustomed to development environments in which system controls are not considered important. In addition, project teams may receive conflicting messages about the importance of controls from several sources: Methodologies that do not prescribe controls development. Project procedures that do not require controls development. Management that assigns priority to meeting deadlines and developing basic system functions rather than achieving adequate internal controls. A project team will not take an audit report on controls seriously if the development environment places little emphasis on controls. In this case, auditors should assert the importance of application controls and, when necessary, attempt to change relevant department policies and procedures. Application audits must be managed differently from audits of established business functions. Whereas conventional audits may be completed in several weeks, system reviews can take months to finish. The level of audit activity can vary according to the amount of auditor involvement requested by the project team. Productivity may not be measurable solely in terms of reports and control comments. If an auditor helps the project team resolve control concerns during systems development, for example, the audit report may document few unresolved weaknesses. In summary, audits put substance into the commitment to effective application controls. Without audits, project teams will receive the wrong signal about what priority management in general attaches to security controls. Author Biographies Douglas Welker Douglas Welker, CISA, is an applications auditor with Dow Chemical Co. in Midland MI. Previously, he audited applications for WW Grainger, Inc., and performed data center reviews for Coopers & Lybrand.