Audit Data Analytics. Bob Dohrer, IAASB Member and Working Group Chair. Miklos Vasarhelyi Phillip McCollough



Similar documents
INTERNATIONAL STANDARD ON AUDITING 520 ANALYTICAL PROCEDURES CONTENTS

Pentaho Data Mining Last Modified on January 22, 2007

Data Analytics Working Group Update

Analytical Procedures

Data Analytics Working Group Update

Auditing Standard ASA 520 Analytical Procedures

Audit Data Analytics Alert

Qi Liu Rutgers Business School ISACA New York 2013

SOLUTION: AUDIT AND INTERNAL REVIEW, MAY 2014

INTERNATIONAL STANDARD ON AUDITING 401 AUDITING IN A COMPUTER INFORMATION SYSTEMS ENVIRONMENT CONTENTS

ANALYTICS CENTER LEARNING PROGRAM

Electronic Audit Evidence (EAE) and Application Controls. Tulsa ISACA Chapter December 11, 2014

CUSTOMER RELATIONSHIP MANAGEMENT (CRM) CII Institute of Logistics

International Standards on Auditing (ISA) and their Use for Second Level Control of European Territorial Cooperation Programmes

BIG DATA What it is and how to use?

UPS battery remote monitoring system in cloud computing

Data Mining + Business Intelligence. Integration, Design and Implementation

Is a Data Scientist the New Quant? Stuart Kozola MathWorks

Introduction to Data Mining

Webinar: PCAOB Inspections of Small Firm Broker-Dealer Auditors. January 15, 2015

Market Manual Changes for Settlement of Capacity Based DR. Demand Response Working Group August 6, 2014

Audit Analytics. --An innovative course at Rutgers. Qi Liu. Roman Chinchila


Performing Audit Procedures in Response to Assessed Risks and Evaluating the Audit Evidence Obtained

Prerequisites. Course Outline

SAP FINUG Teknologiaseminaari

Best Practices in International Logistics. How Top Companies Use Technology and Logistics Partners to Improve Performance

INTERNATIONAL STANDARD ON AUDITING 330 THE AUDITOR S RESPONSES TO ASSESSED RISKS CONTENTS

Business Intelligence. Data Mining and Optimization for Decision Making

INTERNATIONAL STANDARD ON AUDITING (UK AND IRELAND) 520 ANALYTICAL PROCEDURES CONTENTS

Predictive Modeling Techniques in Insurance

Data Mining Part 5. Prediction

CRG Academy Course Descriptions. Corporate Renaissance Group 6 Antares Drive, Phase 1, Suite 200 Ottawa, ON K2E 8A9

EVERYTHING THAT MATTERS IN ADVANCED ANALYTICS

Machine Learning with MATLAB David Willingham Application Engineer

Business Process Services. White Paper. Improving Agility in Accounts Receivables with Statistical Prediction and Modeling

U S I N G D A T A A N A L Y S I S T O M E E T T H E R E Q U I R E M E N T S O F R I S K B A S E D A U D I T I N G S T A N D A R D S

System Behavior Analysis by Machine Learning

Analytical Procedures

Prof. Dr. Nick Gehrke Alexander Rühle

Introduction to Data Mining

Data Mining for Customer Service Support. Senioritis Seminar Presentation Megan Boice Jay Carter Nick Linke KC Tobin

Database Marketing, Business Intelligence and Knowledge Discovery

Azure Machine Learning, SQL Data Mining and R

CoolaData Predictive Analytics

Imputing Values to Missing Data

Introduction to Data Mining and Machine Learning Techniques. Iza Moise, Evangelos Pournaras, Dirk Helbing

An Introduction to. Metrics. used during. Software Development

Audit Quality Thematic Review

Expert Systems in Fraud Detection: Expert Knowledge Elicitations in a Procurement Card Context

Maximierung des Geschäftserfolgs durch SAP Predictive Analytics. Andreas Forster, May 2014

A Hybrid Modeling Platform to meet Basel II Requirements in Banking Jeffery Morrision, SunTrust Bank, Inc.

How To Audit A Financial Statement

Anomaly detection. Problem motivation. Machine Learning

A STUDY ON DATA MINING INVESTIGATING ITS METHODS, APPROACHES AND APPLICATIONS

Predictive Analytics Techniques: What to Use For Your Big Data. March 26, 2014 Fern Halper, PhD

COPYRIGHTED MATERIAL. Contents. List of Figures. Acknowledgments

Transforming big data into supply chain analytics

R s and Predictive Modeling Boot Camp Nov. 8-9, Session #1: Predictive Modeling: An Overview Syed Muzayan Mehmud, ASA, FCA, MAAA

How to use Big Data in Industry 4.0 implementations. LAURI ILISON, PhD Head of Big Data and Machine Learning

THE ABC S OF DATA ANALYTICS

Comparison of ISA 330 with AS-402 Objectives and Requirements Only

A Case of Study on Hadoop Benchmark Behavior Modeling Using ALOJA-ML

Data Isn't Everything

Chapter 12 Discovering New Knowledge Data Mining

Stages of the Audit Process

INTERNATIONAL STANDARD ON AUDITING 540 AUDIT OF ACCOUNTING ESTIMATES CONTENTS

A Regression Approach for Forecasting Vendor Revenue in Telecommunication Industries

Beyond Traditional Management Reporting IBM Corporation

Accounts receivable/credit & collections audit program

Data Science & Big Data Practice

Meet The Data 19 juni 2014 AccountancyExpo. Drs Pieter de Kok RA

Business Analytics and Data Mining for CRM Business Analytics and Data Mining for CRM: Jumpstart workshop

2. A typical business process

Analytical Procedures

SELDOC Headquarters, East Dulwich

Getting insights about life cycle cost drivers: an approach based on big data inspired statistical modelling

Dan French Founder & CEO, Consider Solutions

Why is Internal Audit so Hard?

Business Intelligence Solutions. Cognos BI 8. by Adis Terzić

THE AUDITOR S RESPONSES TO ASSESSED RISKS

Audit Quality Thematic Review

SAS Fraud Framework for Banking

Thompson Publishing Group, Inc. Audio Conference June 6, 2006 Troubleshooting Subrecipient Monitoring: Review of Pass-Through Entity Responsibilities

Contents of This Paper

Application of Predictive Analytics for Better Alignment of Business and IT

DEMYSTIFYING BIG DATA. What it is, what it isn t, and what it can do for you.

INTERNATIONAL STANDARD ON AUDITING 320 AUDIT MATERIALITY CONTENTS

SCAD V: Parts 2 and 3 - Assessing Control Risk

not possible or was possible at a high cost for collecting the data.

Maschinelles Lernen mit MATLAB

Auditing Standard ASA 330 The Auditor's Responses to Assessed Risks

Data Mining - Evaluation of Classifiers

Information and Decision Sciences (IDS)

FINANCIAL ADMINISTRATION MANUAL

International Journal of Advanced Engineering Research and Applications (IJAERA) ISSN: Vol. 1, Issue 6, October Big Data and Hadoop

Civica Health & Social Care

Customer Classification And Prediction Based On Data Mining Technique

Loss Prevention Data Mining Using big data, predictive and prescriptive analytics to enpower loss prevention

Proposed Auditing Standard: Existence and Valuation of Inventory (Re-issuance of AUS 506)

Transcription:

Audit Data Analytics Bob Dohrer, IAASB Member and Working Group Chair Miklos Vasarhelyi Phillip McCollough IAASB Meeting September 2015 Agenda Item 6-A Page 1

Audit Data Analytics Agenda Introductory remarks Illustrations o Revenue Three Way Match o Revenue Segregation of Duties o Predictive Analytics o Clustering What could the future hold? What can others do in this area? Page 2

Audit Data Analytics Technique, not a Tool Risk Assessment Internal Control Evaluation Substantive Analytical Procedures Substantive Procedures (tests of detail) Audit Data Analytics Page 3

Illustration 1 Revenue Three Way Match Procedure: For every invoice, shipping document and sales order received from customers, compare the invoiced customer, quantity, and unit price to the quantity shipped per the shipping documents and the quantity and unit price reflected in the sales order received from the customer. Objective: Obtaining audit evidence over the existence and accuracy of revenue. (ISA 500 paragraphs 6 and 9). Prior year approach: Tests of internal controls over the revenue process, substantive analytical procedures and tests of details (sampling). Page 4

Illustration 1 Revenue Three Way Match (cont.) Entity ABC has revenue of 125 million generated by 725,000 transactions. The three way match procedure is executed with the following results: Amount ( 000) % Note: Materiality for the audit of the financial statements as a whole is 1,000,000. Number of Transactions % No differences 119,750 95.8 691,000 95.3 Outliers: Quantity differences 3,125 2.5 16,700 2.3 Pricing differences 2,125 1.7 17,300 2.4 Page 5

Illustration 1 Revenue Three Way Match (cont.) Some of the Challenges that Arise Is the three way match procedure data analytics? Or is it automation? How do the ISA s currently address 100% testing? (ISA 500 paragraph A53) Is risk assessment relevant (beyond determining to execute the procedure)? Is this risk assessment or a substantive procedure (or both)? What impact does the ability to measure results so precisely have? Why does the procedure produce audit evidence? What level of evidence is needed on outliers? How does the knowledge gained from performing procedure in year 1 affect year 2? What about the integrity of the data used in the procedure? Are process level controls relevant? Auditor verification / evaluation of the technology / routine used to execute the procedure? Page 6

Illustration 2 Revenue Segregation of Duties Procedure: For every sales transaction, evaluation of any segregation of duties conflicts relative to customer master file maintenance, sales order processing, sales invoicing, sales returns / credit notes, and applying cash collections. Objective: Obtaining audit evidence over the effectiveness of internal control over sales processing. 1 (ISA 330 paragraphs 8 9 and ISA 315 paragraph 12) 1 This is one of the tests of control over sales processing. Page 7

Illustration 2 Revenue Segregation of Duties (cont.) For entity ABC, an analysis of segregation of duties was executed with the following findings: Number of users Amount ( 000) Number of transactions Population of sales 542 125,000 725,000 Instances in which same individual created sales invoice, sales return or credit note and applied cash Instances in which same individual executed sales order processing, dispatched goods (delivery document) and applied cash Note: Materiality for the audit of the financial statements as a whole is 1,000,000. 7 620 3,934 96 7,692 46,903 Page 8

Illustration 2 Revenue Segregation of Duties (cont.) Some of the Challenges that Arise Is the segregation of duties procedure data analytics? Is risk assessment relevant (beyond determining to execute the procedure)? Is the segregation of duties a risk assessment procedure or a test of control? What impact does the ability to measure results so precisely have? Should the conclusion on internal controls be bi-furcated (i.e., transactions with segregation of duties conflicts, transactions without segregation of duties conflicts)? What level of evidence is needed on outliers? How does the knowledge gained from performing procedure in year 1 affect year 2? What about the integrity of the data used in the procedure? Auditor verification / evaluation of the technology / routine used to execute the procedure? Page 9

Illustration 3 Predictive Analytics What is predictive analytics? Time series extrapolation and cross-sectional analytics can be used to predict results substantive analytical procedure (ISA 520 paragraphs 5 and A4 A16). If proposed results fall within acceptable variance, assess whether further year-end procedures are necessary. In addition to substantive analytical procedures and analytical procedures presently performed at the planning and completion phases of an audit, predictive analytics are increasingly being performed as a substantive analytical procedure. Example: Big Data as Audit Evidence: Utilizing Weather, Geolocational and other Indicators across outlets of a major retailer, Alexander Kogan and Kyunghee Yoon Page 10

Illustration 3 Predictive Analytic (cont.) Data and Model Description Objective: Predict revenue at the store level (approximately 2,000 stores) for a publicly held retail company using internal company data and nontraditional data (e.g., weather). Forecasting daily store level sales (one step ahead forecasting). Multivariate regression model with / without the peer store indicator and weather indicators. AR(1)+ +AR(7) with / without the peer store indicator and weather indicators. Page 11

Illustration 3 Predictive Analytic (cont.) Preliminary Results Expectations derived from store level disaggregated data are more accurate than ones from entity level aggregated data. Peer stores have predictive powers for sales. Data not traditionally used in substantive analytical procedures (e.g., social media, weather, geolocation) are often effective in predicting sales. Page 12

Illustration 3 Predictive Analytic (cont.) Clustering Using Store Sales by Peer Group Page 13

Illustration 3 Predictive Analytic (cont.) Some of the Challenges that Arise Are predictive analytics relevant to audit data analytics? Are predictive analytics a risk assessment procedure, substantive procedure, or both? What procedures are necessary to validate non-traditional external data (e.g., social media, weather, traffic patterns)? Is it appropriate to reduce or eliminate other substantive tests with predictive analytics? (ISA 330 paragraph 7) Should there be guidance of an acceptable level of variance? Should there be an experimentation period to create guidance for predictive analytics? What predictive methodologies would be acceptable? Page 14

Illustration 4 Clustering Multidimensional clustering is a powerful tool to detect groups of similar events and identify outliers Audit Sampling (ISA 530, paragraph 6) Can be used in any set of data examination procedures (preferably with a reduced set of data). Looking for anomalous clusters and outliers from the clusters. Statistically complex. Multidimensional Clustering for audit fault detection in an insurance and credit card settings and super-app Sutapat Thiprungsri, Miklos A. Vasarhelyi, and Paul Byrnes Page 15

Illustration 4 Clustering (cont.) Page 16

Illustration 4 Clustering (cont.) The need for a Super-application to facilitate auditor statistical decisions Normalizes all data to be clustered. Creates normalized principal components from the normalized data. Automatically selects the necessary normalized principal components for use in clustering and outlier detection. Compares a variety of algorithms on selected set of normalized principal components. Determines the preferred model based upon the silhouette coefficient metric. This model is then used for final clustering and outlier detection procedures. Produces relevant information and outputs throughout the process, thereby facilitating additional, deeper analyses. Page 17

Illustration 4 Clustering (cont.) Some of the Challenges that Arise What audit evidence does clustering create, if any, for populations that have tight clustering vs. outliers to the cluster? Is clustering a risk assessment procedure, substantive procedure, or both? What follow-up on outliers is necessary? Is it appropriate to reduce other substantive tests with clustering? Should there be guidance of acceptable level of variance from cluster geometric center? If auditors (or others) are developing applications for use in the audit, what quality control should be performed to verify that the application is fit for purpose, technically sound etc.? Should there be an experimentation period to create guidance for clustering? What clustering methodologies would be acceptable? Page 18

What could the future hold? Machine Learning / Cognitive Computing As an alternative to statistical and optimization methods, human decision models can be built either by capturing human behavior and models (expert / cognitive computing) or by using machine learning methods to develop these rules. Typically supervised machine learning methods such as decision trees, logistic regression, and vector machines will be used. Possibility that in the future, the volume of data may exceed human capacity to audit that data. Expert Knowledge Elicitations in a Procurement Card Context at PG Deniz Applebauum and Abdullah Al-Awadhi Credit card default and lawsuit prediction at Itau-Unibanco Miklos A. Vasarhelyi, Eduardo Miyaki, Qi Liu, Jun Dai, Fei Qi Huan, Pei Li, and Basma Modarrat Page 19

What could the future hold? (cont.) Some of the Challenges that Arise Machine Learning / Cognitive Computing Are machine learning techniques relevant to audit data analytics? Is it reasonable to reduce or eliminate other substantive tests with machine learning? What impact might this reduction have? Level of guidance needed on methods? Should there be a new method of separate review of the technologies used if these methods are used? Should there be an experimentation period to create guidance for machine learning? What predictive methodologies would be acceptable? Page 20

What could the future hold? (cont.) Others Continuous Control Monitoring formalization of controls evaluation Continuity Equations structural modeling in continuous auditing Process Mining in Auditing audit Logs in Enterprise Resource Planning systems Exceptional exceptions choosing exceptions from huge data Evidence from Big Data electronic logs everywhere Audit Data Standard normalizing data to facilitate analytic applications Visualization for continuous assurance monitoring creating auditor dashboards Page 21

What can others do in this area? Where in the audit of historical financial statements are these methods to be used? How to create an experimentation period where supervised analytics projects are performed in real engagements? How to deal with the economic limitations of using data analytic methods in audits? Page 22