One important aspect of IT audits is sampling and sampling methodologies. It is important to understand the different methodologies an auditor could use and when to use which one. The choice of methodology also affects the interpretation of the results. For example, if the auditor discovers one or two errors in the sample, what does that mean? It could be that the methodology chosen has an error rate that allows two errors in that particular sample (which means there is no need to expand the sample), or it could be that the methodology chosen allows no errors at all (which means there is trouble of some sort, even if it is just a larger sample and more work).
Many auditors rely on one of the standard audit procedure support systems that provides standardized forms for performing substantive tests and includes charts for determining sample size. It is tempting to rely totally on the packaged sample information or form (i.e., pull form, find sample size in chart, pull sample), rather than to go through a rigorous process to decide which sampling method applies and what the sample size and potential deviations mean to the audit. Also, according to some experts, the trend today is to use less-rigorous, nonstatistical sampling to reduce cost, and there is a risk that such an approach may be substantially less capable of detecting a material error than a statistical approach, such as probability-proportional-to-size (PPS) sampling. The downside of this rigorous statistical approach is the complexity of statistical sampling concepts and process (if done by hand). However, there are a number of tools, such as Excel worksheets and plug-ins, to facilitate the process.
Therefore, this article will attempt to summarize the four most common statistical methods used in audit, and provide some guidance in applying those methods.
There are four basic sampling methodologies:
- Attribute sampling—This type of sampling enables the auditor to estimate the rate of occurrence of certain characteristics of the population (e.g., deviations from performance of a control). It is most often used in performing tests of controls. A deviation would be the failure of a control to function properly (i.e., an error).
- Discovery sampling—This type of sampling is designed to locate a small number of deviations or exceptions in the population. It is most often used to detect a fraudulent transaction. If there is one deviation (i.e., one fraudulent transaction) in the sample, the auditor must examine the population. A deviation in discovery sampling, however, is not the same as a deviation in other sampling methods. In the former, it refers to fraud; in the latter, it refers to an error. Discovery sampling is used primarily to detect critical deviations. Because they are considered critical, the discovery of a single deviation (e.g., fraud) is intolerable. Consequently, if a critical deviation is discovered, the auditor may abandon the sampling procedures and investigate the population, rather than relying on the sample. For fraud detection, a fraudulent transaction or event would be considered critical. If using discovery sampling to detect fraud, and the auditor uncovers a simple US $300 transposition error in a transaction, that “error” would not be considered critical.
- Classical variables sampling (CVS)—This method is used to provide auditors with an estimate of a numerical quantity, such as the balance of an account. It is primarily used by auditors to perform substantive tests. It includes mean-per-unit estimation, ratio estimation and difference estimation. For example, this method would be used to confirm accounts receivable.
- Probability-proportional-to-size sampling—This method develops an estimate of the total monetary amount of misstatement in a population. PPS uses dollar-unit sampling or monetary-unit sampling (MUS). Other methods are based on instances or occurrences, but this method is based on monetary values, where higher monetary value transactions have a higher likelihood of being chosen in a sample—thus the name PPS. MUS includes:
a. A tolerable misstatement amount (the total misstatement the auditor will allow in the population)
b. Acceptable risk of incorrect acceptance (risk that the sample does not support the conclusion about not being materially misstated, i.e., a false-positive; generally 5 percent or 10 percent)
c. Acceptable risk of incorrect rejection (opposite of b; sample shows material misstatement in population when it is not materially misstated, i.e., a false-negative)
d. Assumption of average percent of misstatement (for items misstated, the assumed average size of each misstatement compared to the recorded amount)
MUS is often used in statistical examinations where the purpose is fraud detection.
The American Institute of Certified Public Accountants (AICPA) Statistical Sampling Subcommittee prepared an audit guide in 1983, titled Audit Sampling, that describes PPS. The audit guide lists several advantages of PPS over CVS.
Choice of Sampling Methodology
The choice of a method depends on the primary purpose of the sample and substantive test. If the auditor needs to perform a test of control, the best choice is attribute sampling, generally speaking. If the purpose of the audit procedure is to detect fraud, then discovery sampling is the best choice, but MUS is a good choice, too. If the purpose is to look for material misstatements in an account balance or class of transactions, CVS is a good choice. But, CVS does tend to require larger samples than other methods and is, therefore, costly. PPS requires smaller samples. PPS is designed to be especially effective in the audit of accounts receivable and inventory, with a few exceptions, and thus is usually a better choice than classical variables for account balances such as these. However, PPS is prone to trigger falsepositives, and the auditor must be aware of this possibility.
It is possible to use a different method from that generally chosen, if there is an extenuating circumstance or objective. Obviously, discovery sampling has a more stringent requirement regarding deviations or exceptions, so is usually the prime choice for fraud detection.
In discovery sampling, a key point is what is meant by “critical deviation.” In particular, the standardized audit methodologies indicate that if the auditor detects a fraudulent transaction, such as an invoice from a shell (fictitious) vendor, that transaction is considered a critical deviation. An identified deviation (or anomaly), therefore, can be classified into two categories:
- Those that are clearly fraudulent or highly suspicious of fraud
- Those that are clearly errors
According to the discovery sampling methodology, if a fraudulent deviation (i.e., a critical deviation) is detected, then the review of the sample should be stopped and the entire population should be reviewed (this method is sometimes referred to as stopand- go sampling). The theory behind discovery sampling is that the goal is zero critical deviations. As defined, that means zero fraud. Because the purpose is to have zero tolerance for fraud, the sample sizes tend to be larger than other sampling methodologies and, obviously, have a significantly smaller allowance for deviations. However, that does not mean that, if a deviation that is the result of error is found, the auditor must stop and review the population. In fact, the language of authoritative sources says the auditor may decide to review the population, not that the auditor must do so.
Example of Application
What does it mean when a deviation occurs in the sample? The following is an illustration of what would happen if two different sampling techniques were used to examine a common population for the purpose of fraud detection. The set of circumstances for the illustration is as follows:
- The population is 10,000 transactions.
- The objective is the effectiveness of antifraud controls.
- The IT auditor chose discovery sampling.
- A sample size of 483 was taken, based on discovery sampling table.
- Two errors were discovered but neither had any fraudulent implications.
According to the discovery sampling rules, the two occurrences were (minor) errors and, therefore, there were no critical deviations. The conclusion is that the auditor could rely upon the sample in assessing the likelihood of fraud, and there is a 95 percent probability that no critical deviation exists in this population.
If the auditor had used attribute sampling, because the auditor was testing controls, the process and sample size would have been different. If a 1 percent expected deviation rate is assumed (typical rate), with a 7 percent tolerable deviation rate and 95 percent confidence interval, the AICPA chart shows a sample size of 66 (notice how much smaller the sample size is for attribute sampling than for discovery sampling), with one allowable actual deviation. The 7 percent is the top end of the low level of assessed control risk (2-7 percent), and within the moderate control risk (6-12 percent). If none or one deviation was found in a sample of 66, then according to attribute sampling, the assessed level of control risk would not be too low, and the controls are as effective as assessed. If more than one deviation occurs in a sample of 66, the interpretation is that actual control risk is higher than assessed.
Classical variables sampling is not applicable and is based on monetary amount, or number of occurrences. PPS is subject to monetary amounts and it is unknown what the exact sample would have been determined using PPS.
According to Practitioner’s Guide to Audit Sampling, there are several practical advantages for auditors who use statistical sampling: less likelihood of over- or under-auditing, more objective and defensible audit work, better work paper documentation, and greater confidence in the audit opinion.
Therefore, it is important to understand and properly apply sampling techniques. This article attempts to discuss the basics of the four common statistical sampling methods used in IT audit (and internal and financial audit as well). Auditors need to take the time to conduct an informed and rigorous thought process when choosing a statistical method and to achieve the appropriate interpretation of the results, if there are any deviations or exceptions in the sample. A thorough approach to sampling will generally lead to many advantages for the IT auditor, including efficiency and effectiveness of the audit.
- Guy, Dan M.; D.R. Carmichael; O. Ray Whittingham, Practitioner’s Guide to Audit Sampling, John Wiley & Sons, 1998
- Wampler, Bruce; Michelle McEacharn; “MUS Using Excel,” CPA Journal Online, May 2005, www.nysscpa.org/cpajournal/2005/505/essentials/p36.htm
- New York State Society of CPAs, “Software to Download,” The CPA Journal, www.cpajournal.com/down.htm
- AICPA, Audit Guide, Audit Sampling
- Yancey, Will; Comprehensive list of references and links related to “Sampling for Financial and Internal Audits,” www.willyancey.com/sampling-financial.htm
Tommie W. Singleton, Ph.D., CISA, CITP , CMA , CPA
is an associate professor of information systems (IS) at the University of Alabama at Birmingham (USA), a Marshall IS Scholar and a director of the Forensic Accounting Program. Prior to obtaining his doctorate in accountancy from the University of Mississippi (USA) in 1995, Singleton was president of a small, valueadded dealer of accounting IS using microcomputers. Singleton is also a scholar-in-residence for IT audit and forensic accounting at Carr Riggs Ingram, a large regional public accounting firm in the southeastern US. In 1999, the Alabama Society of CPAs awarded Singleton the 1998-1999 Innovative User of Technology Award. Singleton is the ISACA academic advocate at the University of Alabama at Birmingham. His publications on fraud, IT/IS, IT auditing and IT governance have appeared in numerous publications, including the ISACA Journal.
ISACA Journal, formerly Information Systems Control Journal, is published by ISACA, a nonprofit organization created for the public in 1969. Membership in the association, a voluntary organization serving IT governance professionals, entitles one to receive an annual subscription to the ISACA Journal.
Opinions expressed in the ISACA Journal represent the views of the authors and advertisers. They may differ from policies and official statements of ISACA and/or the IT Governance Institute and their committees, and from opinions endorsed by authors, employers or the editors of this Journal. ISACA Journal does not attest to the originality of authors’ content.
Instructors are permitted to photocopy isolated articles for noncommercial classroom use without fee. For other copying, reprint or republication, permission must be obtained in writing from the association. Where necessary, permission is granted by the copyright owners for those registered with the Copyright Clearance Center (CCC), 27 Congress St., Salem, Mass. 01970, to photocopy articles owned by ISACA, for a flat fee of US $2.50 per article plus 25¢ per page. Send payment to the CCC stating the ISSN (1526-7407), date, volume, and first and last page number of each article. Copying for other than personal use or internal reference, or of articles or columns not owned by the association without express permission of the association or the copyright owner is expressly prohibited.
US: one year (6 issues) $75.00
All international orders: one year (6 issues) $90.00
Remittance must be made in US funds.