Random Sampling for Audits and Surveys

The sampling frame is the whole problem

The frame is the list you sample from, and it is only ever a proxy for the population you care about. Sampling invoices from the accounting system tells you about invoices in the accounting system, not about transactions — and the transactions that were never entered are precisely the ones an audit exists to find.

The practical discipline is to write down the population you care about and the frame you actually have, side by side, and describe the gap explicitly. Sometimes the gap is negligible; sometimes it swallows the entire finding.

Coverage error is the name for items missing from the frame, and it cannot be corrected by sampling harder. No sample size, no randomisation quality and no statistical adjustment recovers something that was never in the list.

• Write down the target population and the available frame separately.
• Describe the gap between them explicitly, before sampling.
• Coverage error is not fixable by a larger sample.
• Ask what kind of item would be systematically missing — that is usually the finding.

Choosing a size

For a proportion estimate, the useful rule of thumb is that the margin of error is roughly one divided by the square root of the sample size. A sample of 100 gives about ten percentage points; 400 gives about five; 1,000 gives about three.

That relationship is the source of the most useful counter-intuitive fact in sampling: the population size barely matters. A sample of 1,000 gives roughly the same precision whether the population is 50,000 or 50 million, because the precision depends on the sample size rather than the fraction — until the sample becomes a large fraction of the population, at which point it gets better.

For detecting a rare problem rather than estimating a proportion, the calculation is different and more useful. To have a 95% chance of catching at least one instance of a problem affecting 1% of items, you need about 300 items. For a problem affecting 0.1%, about 3,000.

Stratifying when subgroups matter

A simple random sample treats every item identically, which means a small but important subgroup can be missed entirely. Twenty items sampled from a thousand, of which thirty are high-value transactions, will contain zero high-value items about half the time.

The fix is stratification: divide the population into groups and sample within each. That guarantees representation and usually reduces variance as well, because items within a stratum tend to be more similar to each other than to the population as a whole.

The cost is that stratification requires knowing the strata in advance, and requires combining the results correctly afterwards — a stratified sample where each stratum was sampled at a different rate cannot simply be pooled.

The mistakes that invalidate a sample

Replacing an inconvenient selection is the most damaging and the most common. If a selected item is hard to obtain and you substitute the next one, the sample is no longer random, and the substituted items are systematically different — hard-to-obtain items are hard to obtain for a reason.

Sampling from a sorted list with a fixed interval is a subtler version. Systematic sampling is a legitimate method, and it fails badly when the list has a periodicity matching the interval — sampling every seventh item from a daily log gives you the same weekday every time.

Choosing the sample size after seeing early results is the third. Stopping once you have found enough problems, or continuing until you find one, both invalidate any statement about rates.

• Do not substitute a replacement for a hard-to-obtain item.
• Beware systematic sampling on a list with hidden periodicity.
• Fix the sample size before you look at any results.
• Record the method, including the seed if you used one.

Making it reproducible

An audit sample should be reproducible by someone else, which is the same requirement as a verifiable prize draw and has the same solution: record the frame as it stood, and record the seed if you used a seeded draw.

That combination lets a reviewer confirm the sample was not selected to produce a particular result — which is not usually the suspicion, and is exactly the kind of thing that is impossible to rebut afterwards if you did not record it.

It also lets you re-run the same sample later to check whether the items have changed, which is a genuinely useful capability for a repeated audit and is unavailable if the sample was drawn unpredictably.

Frequently asked questions

Tools that use this

Related guides