Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
A statistician’s view on the EU sampling guidelines Paul van Batenburg CB Conference, June 1st, 2010, Budapest © 2010 Deloitte Touche Tohmatsu A statistician’s view on the EU sampling guidelines • The situation: 1. PA executes on-the-spot checks on samples from claims. These risk based and random samples are performed for different support schemes 2. CB investigates whether the PA sample checks can be relied upon by: a) Performing a MUS sample on the population of items that were randomly selected by the PA b) Choosing a sample size using sheet A1 to compare the PA’s audit values with the CB audit values c) Evaluating by using sheet A2 confronting the most likely absolute error rate with a 2% materiality threshold • Questions I would like discuss (in reverse order): 1. Is this the right sample from the right population? 2. Is this the right sample size? 3. Is this the right evaluation? • If one or more answers turn out to be negative, a new question comes up: ‒ Is there an alternative method to recommend? • I believe so, and will briefly introduce: ‒ AOQL ‒ Regression estimates 2 © 2010 Deloitte Touche Tohmatsu Presenter •Paul van Batenburg •Statistician, not an auditor •Deloitte Netherlands •“Effective and efficient audit by using statistical methods and techniques” •Relevant assignments: NL Ministry of Economic Affairs, NL Ministry of Agriculture (PA’s and CB), EU DG Budget, Regione Puglia •www.steekproeven.eu (goto: CB Conference for Excel sheet and this presentation) 3 © 2010 Deloitte Touche Tohmatsu Question 3: evaluation •Do the percentages that make up (j) have the same denominator? •MUS samples are not equipped to evaluate understatements; taking their absolute values does not solve that •The sample was designed for a 95% maximum error to meet 2% materiality, so why confront the most likely error with this threshold? 4 © 2010 Deloitte Touche Tohmatsu Question 2: sample size •The audit risk model was designed to economize on substantive testing in situations where controls were in place to prevent errors. A low required confidence from substantive testing combined with a high expected error rate is inconsistent with the rationale behind the ARM •The form of the ARM used is additive: total confidence level = confidence level from control testing plus confidence level from substantive testing •The AICPA audit risk model (as used by audit firms globally) is multiplicative: •audit risk = control risk * detection risk, or total confidence level = 1-(1-confidence level from control testing)*(1- confidence level from substantive testing) •I personally believe both models are flawed, but the AICPA model is better since it cannot yield a required confidence level of 0%, conform ISA 330 par.20 and A 42 •Error rates mentioned are percentages of materiality and not percentages of the population •Sample sizes should have been rounded up, see www.steekproeven.eu for an Excel sheet 5 © 2010 Deloitte Touche Tohmatsu Question 1: population and sample (in bold: what I read from the guidance) •If the CB’s sample is meant to confirm the accuracy of the PA’s statistics The sample should be selected from all risk based and random samples by the PA The sample should be a line item sample The PA’s audit value should be compared to the CB’s audit value A materiality level should be chosen instead of the 2% that is meant for financial audits •If the CB’s sample is meant to confirm with 95% assurance that the error rate in the payments is below 2% The sample may be selected from the PA random sample items (not MUS, not the risk based items) The CB sample should be MUS The PA’s recorded value should be compared to the CB’s audit values The (95% reliable maximum) error rate should be compared to 2% materiality In my view, 2 approaches are mixed: conclusion and comparison do not match population, sample selection and materiality definition 6 © 2010 Deloitte Touche Tohmatsu On understatements •A MUS sample of 1 from this population of 300 in 3 transactions based on recorded values gives probabilities of selection of 1/3 for each item •If no errors had been made, the 2nd transaction would have had a smaller probability: MUS samples are an excellent tool to audit on overstatements as the error increases the chance of being caught •And the 3rd would have had a larger probability: MUS samples are not equipped to audit on understatements •Either use MUS from the difference between a maximum value and the recorded value (completeness of VAT invoiced) or use a Line Item Sample •Taking absolute error amounts yields an incorrect population audit value: understated amounts are no part of the population so it does not contain understatement errors 7 © 2010 Deloitte Touche Tohmatsu Alternative: relying on CB sample •The horizontal axis shows the PA’s audit values and the vertical axis shows the CB’s audit values in the CB’s sample from all PA sample items •The statistical technique of regression yields a line that best expresses the relation to predict CB audit values from PA audit values for all other PA sample items (Cochran, 1977) •If only small differences (+ and - !) are found, the correlation will be high and the CB can rely on the PA’s sample checks • When large differences occur, the PA may not have followed the audit protocol and the CB should consider rejecting the PA sample •The decision to accept or reject the PA sample needs not to be taken for the sample as a whole but can be applied in batches of, say, 10% of the total PA sample based on AOQL (Dodge and Romig, 1933, 1957) •References: • Vera Raats, Monotone missing data and repeated controls of fallible auditors, Ph.D. thesis, Tilburg University, 2004 8 © 2010 Deloitte Touche Tohmatsu Questions? 9 © 2010 Deloitte Touche Tohmatsu