Application Testing: Not Just for IT Auditors
Huntington Ingalls Industries Who We Are Over a century designing, building, overhauling and repairing ships for the U.S. Navy, the U.S. Coast Guard and world navies The United State's sole industrial designer, builder and refueler of nuclearpowered aircraft carriers One of only two U.S. companies capable of designing and building nuclearpowered submarines Have built over 40 percent of the U.S. Navy s current surface combatant fleet 2
Huntington Ingalls Industries Our Application Landscape Hundreds of distinct applications used for design, engineering, inventory, personnel, planning, scheduling, production, billing, accounts payable, etc. Reviews of business processes and production operations are conducted by internal audit as well as internal QA, external auditors, DCAA, DCMA, Naval Reactors, OSHA, SOS, DoD Security, etc. All these reviews include at least some review of application controls. If you rely on a computer program, someone will be interested in the application controls. 3
Your Presenter for This Session Richard Fowler, CIA, CRMA, CISA, CFE, CICA, GIAC, etc. 19 years of internal audit experience Senior Audit Specialist with Huntington Ingalls Industries Previous audit & IT audit experience with Northrop Grumman, Virginia Information Technologies Agency, Virginia Social Services, Circuit City, SunTrust Bank and Crestar Bank 16 additional years of engineering, quality control, computer programming, network security, and program management Program Chair of the ASUG Internal Controls Special Interest Group (SIG); ASUG Financials Community Facilitator ISACA Application Controls Community Leader Also a member of IIA, ASQ, IIC, IISFA, SANS Follow my discussions on LinkedIn, ASUG, and ISACA 4
Key Points Learn different ways to identify application controls Learn how to assess the significance of specific controls Go through a number of common applications to identify critical controls Learn how to effectively test application controls based on their design Incorporate these application control tests into an integrated or stand-alone audit program. 5
Key Points Almost every audit can incorporate application control testing. By incorporating this type of IT audit into an otherwise regular audit, we will be conducting an integrated audit. Integrated audits don t require multiple auditors - all they need is an integrated auditor. You can do it! 6
1. Identifying application controls You can t test what you can t find Different approaches are OK there s no one best way General approach can be used on any and all applications But first things first You Your subject knowledge Your expectations 7
1. Identifying application controls What are application controls? First, we need to know What are applications? This is a quiz SAP? Yes. Oracle? Trick question Oracle Database? No. Oracle Financials? Yes. SQL? No. Word? Yes! Apple itunes? Yes! 8
1. Identifying application controls What are application controls? Well, first we need to know What are applications? Consider the OSI layers: 1. Physical layer (copper, fiber, wireless, etc.) 2. Data layer (bits & bytes being broadcast) 3. Network layer (packets being transmitted) 4. Transport layer (TCP/IP or HTTP for example, the protocols used) 5. Session layer (your computer/network login & the ports being used) 6. Presentation layer (databases, n-tier functions, web back ends) 7. Application layer (yes, this is what we re interested in!) Remember that applications are the highest of the OSI layers and exist where the user interfaces with the data (that s the end-user, not the sys-admin or DBA). We want to minimize user errors. OK, we ve got the what, so what s next? 9
1. Identifying application controls What are application controls? We still need definitions. Second, Where are applications located? On your computer On a mainframe or server (2-tier, 3-tier, n-tier) On the Internet At a vendor (SaaS) On your mobile device (yes, there s an app for that ) In the cloud (also SaaS) 10
1. Identifying application controls So now that we know what qualifies as an application and where we can find them, let s see what can we do to identify a control. In general, what are controls? A process designed to provide reasonable assurance regarding the achievement of objectives in three categories: Effectiveness and efficiency of operations Reliability of financial reporting Compliance with applicable laws and regulations Anyone familiar with these three categories? 11
1. Identifying application controls 12
1. Identifying application controls 13
1. Identifying application controls So application controls would be a subset of these controls The objective of application controls is to help ensure that: Input data is accurate, complete, authorized, and correct. Data is processed as intended in an acceptable time period. Data stored is accurate and complete. Outputs are accurate and complete. A record is maintained to track the process of data from input to storage and to the eventual output. 14
1. Identifying application controls So, we ve covered definitions of applications, internal controls, and the big topic of the day, application controls. Let s look at some general types of application controls: o o We can break this down into the common categories of People, Process and Products. Let s focus on the Process part. 15
1. Identifying application controls We ve seen the big picture, let s drill down to some details. Input controls: Data checks and validations Automated authorization, approval, and override Automated SOD Pending items 16
1. Identifying application controls Output controls: General ledger and sub-ledger posting Update authorization Report distribution Receipts, invoices, statements Storage controls: File transfer controls (check sums, record count, etc.) Data transmission controls (acknowledgement, error reports, etc.) Data backup and recovery 17
1. Identifying application controls Processing controls: Automated file identification and validation Automated functionality and calculations Management overrides Data extraction, filtering, and reporting Interface balancing Automated functionality and aging Duplicate checks 18
1. Identifying application controls Monitoring controls: Audit trails Logged data (changes, access, etc.) Log reviews Configuration settings Licensing 19
1. Identifying application controls 20
2. Assess the significance of controls Not all controls have the same weight 21
2. Assess the significance of controls This is essentially a risk assessment for an application We want to identify all the controls (or as many as possible) so we can determine which are more critical We want to test all the critical controls for an app, and as many other key controls as we can given the resources allocated for the review GTAG-8 has guidance on the risk assessment process 22
2. Assess the significance of controls Not really any different from any other audit planning process. To start, we need to ask 4 key questions: 1. What are the biggest organization-wide risks and main audit committee concerns that need to be assessed and managed while taking management views into account? 2. Which business processes are impacted by these risks? 3. Which systems and applications are used to perform these processes? 4. Where are processes performed? 23
2. Assess the significance of controls What are some potential risk factors for a single app? Data criticality Management reliance Time sensitivity of data Processing complexity Change management Configuration stability Legal requirements (privacy, national security, financial, etc.) 24
3. Critical controls of common apps SAP, Oracle, PeopleSoft, JD Edwards, & other ERP Primavera, Preactor, Quintiq, other scheduling tools AutoCad, Catia, Siemans, other engineering/design Inventory & warehousing management Procurement systems Insurance, risk management, investments Retail / POS systems, banking systems, healthcare Contracts & legal resource data Legacy and mainframe applications etc. 25
3. Critical controls of common apps SAP 26
3. Critical controls of common apps Access (should be limited for production systems) Segregation of duties (there should be roles assigned) Edit checks (numeric fields vs. text, value limits) Embedded calculations (amount due, depreciation, tax) Change controls Configuration controls Tolerance limits Report distribution Data encryption, data classification, and other privacy controls 27
4. Testing application controls Types of testing Tests of application controls / tests of control design Compliance testing Substantive testing / test of effectiveness Tests of transactions Tests of balances Analytical review procedures Sample testing vs. total population testing Familiar? It s just like any other audit 28
4. Testing application controls Tests of controls / tests of design Recall that application controls are designed to mitigate risks (of error, misstatement, fraud, etc.) In testing, the risk is more important than the control However, the control is easier to identify You remember, we just did it!! We ll be looking at some examples shortly 29
4. Testing application controls Tests of compliance / substantive testing Some controls are mandated by law, regulation, or internal procedures Testing is needed to verify that the intent of the control is being met, not just that the control exists. Substantive tests can use sampling Samples may be judgmental, haphazard, or random Using CAATs, we can also sample at 100% 30
4. Testing application controls Access control limits those who can run the app Risks Too many people means slow response times Too many people means data leakage Too many people means uncontrolled changes Others? All these risks can be addressed by one control But does this control work? Is it effective? 31
4. Testing application controls Access control limits those who can run the app Test the control Does it exist? How can we tell? Unique user IDs (no default IDs enabled) Passwords Password rules, depending on data criticality Access reviews (app owner, mgmt, etc.) Is it effective? How can we tell? 32
4. Testing application controls Edit checks limits the data that can be input Risks Typos can produce erroneous results Unlimited data can preclude proper classification Whoops wrong account, wrong rates, etc. Others? All these can be addressed by this control type But does it work? 33
4. Testing application controls Edit checks limits the data that can be input Test the controls Is it working? What does it need to do? Minimize typos (0 vs. o, etc.) Limit data entry to valid options Is it effective? How can we tell? Is there a better process available? Are there overrides available? 34
4. Testing application controls Data encryption, data classification, and other privacy controls protect stored data Risks Safe Harbor, EU Data Directive PCI DSS requirements In the US, HIPAA, Dodd-Frank, GLB Proprietary data protection Others? Check out http://www.dlapiperdataprotection.com All these risks can be addressed by one control But does it work? 35
4. Testing application controls Data encryption, data classification and other privacy controls protect stored data Test the control Does it exist? Who can we ask? DBA System Administrator Application owner / Business process owner Is it effective? How can we tell? Type of encryption Regulatory requirements met 36
4. Testing application controls Embedded calculations a key processing control Risks Financial report misstatements Incorrect payment of taxes Overpayment of invoices Others? All these can be addressed by this control type But does it work? 37
4. Testing application controls Embedded calculations a key processing control Test the controls Assume the controls (the calculations) exist Is this a valid assumption? Are they effective? How can we tell? Review outputs, reports, summaries, etc. Re-perform critical calculations 38
5. Creating an audit program Generic application control audit programs are available IIA s GTAG-8 ISACA s CobiT AuditNet s program library Corporate Executive Board (CEB) Audit Directors Roundtable (ADR) and their Audit Reference Center (ARC) Google can be used to find others Also check out the user community of the IIA, ISACA, and related groups on LinkedIn to request specific information or examples 39
5. Creating an audit program First things first Is this a stand-alone application control audit? Or is it part of an integrated audit? As a stand-alone audit, the audit scope and objective will be better defined. You will be looking at a single application. As part of an integrated audit, you may be looking at several applications in a limited review (you can t look at every part of an ERP system and also review supporting COTS applications and spreadsheet controls) 40
5. Creating an audit program A bit about integrated audits Application control reviews are ideal elements of an integrated audit. Almost every process, whether financial or operational, relies in part on an application. Whatever is being tested one way can have an IT test included. Examples: Travel accounting audits look at expense reimbursement, approvals, duplicate/false receipts, and compliance Add application controls to verify data entry edit checks, access controls, database monitoring & payment approval 41
5. Creating an audit program Examples: Material management audits look at receiving, inventory, warehouse management, issuance, returns & scrap Add application controls to verify data entry edit checks (quantity=po order), cycle count sample configuration, RFID scanning lookups, scrap designation access, etc. Financial reporting audits look at balance sheets, P&Ls, management approvals, executive reviews, etc. Add application controls to verify spreadsheet calculations, management override access, automated reconciliations, and access to preliminary & critical reports 42
5. Creating an audit program Application objectives what is it trying to do? Objective risks what could go wrong? Mitigating controls how do we keep things right? Control tests how do we know the controls are there? Substantive tests are the controls effective? What else do we need to know for an audit? 43
5. Creating an audit program Objective risks what could go wrong? The risks are based on the business objectives, i.e., what we want to the application to do: Timely processing latency issues, capacity planning gaps Valid configuration insufficient testing, lack of documentation Accurate calculations outdated rate tables Log changes logs not retained, logs not reviewed, logging unneeded data 44
5. Creating an audit program Mitigating controls how do we keep things right? The controls are based on the risks: Latency issues daily network performance reports Insufficient testing test plans are reviewed by process owners and IT Outdated rate tables Monthly rate table updates Logs not reviewed Weekly review of critical logs 45
5. Creating an audit program Control tests how do we know the controls are there? The tests are based on the controls: Daily network performance reports Verify daily reports are produced and reviewed Test plans are reviewed by process owners and IT Verify test plan review and concurrence Monthly rate table updates Verify updates, verify source, validate calculations Weekly review of critical logs Verify log reviews are performed (logs are no good if they re not used) 46
5. Creating an audit program Substantive tests are the controls effective? The substantiation is based on the test and criticality: Verify daily reports are produced and reviewed Random sample of 25 reports, validate the reviews Verify test plan review and concurrence Review 5 test plans and verify concurrence Verify updates, validate calculations Verify 2 monthly updates to rate tables; re-perform a rate calculation from each period. Verify log reviews are performed Determine how often logs are reviewed and by whom; determine what attributes are looked for 47
Presentation Summary Application controls exist in all applications Some controls are more critical than others, and now you know how to find them Once found, you can test the control for design and for effectiveness More reliance can be placed on substantive tests, but the tests of design are usually sufficient. Learn how to most effectively test application controls based on their design 48
Thanks for attending! Mountains of Change - You can be the change agent for your audit group! Oceans of Opportunities - Your next audit can be an integrated audit! 49