Drawing a Sample for a Quality Check
The sampling frame is where quality checks go wrong, not the random draw.
A quality-check sample almost always fails at the frame rather than the draw: the items that most need checking are the ones that never made it into the list you sampled from, and no amount of randomness recovers them.
The theory of sampling frames and sample sizes has its own reference page. This is the operational half: what to do on a Tuesday when somebody has asked you to check a batch and you have two hours.
Almost everything that goes wrong here happens in the twenty minutes before the draw, not in the draw.
Get the list out of the system that owns it
Export it rather than reading it off a screen, and export it before you decide anything about size. A list you can paste is a list you can count, timestamp and keep; a list you are scrolling through is a list you will re-derive differently next month.
Note where the export came from and what it excluded by default. Most systems apply a filter you did not ask for — closed items, archived records, anything flagged as an exception — and the excluded set is disproportionately where problems live.
Keep the export. Six months later the question will not be what you found, it will be what you looked at.
Draw, then stop touching it
Paste the export, set the size, draw once, and copy the result somewhere it cannot be regenerated. The failure this prevents is the quiet second draw — you closed the tab, you re-ran it, and the sample you are now checking is not the sample you recorded.
Number the drawn items in the order they came out and work them in that order. It sounds fussy and it removes a real bias: working the easy ones first means the hard ones get checked when you are tired, or not at all.
If you are drawing several samples across a period, use the seed generator instead. A recorded seed lets you re-draw the identical sample later, which turns two independent snapshots into a comparison of the same items over time.
The record to keep
- Export source and timestamp, plus what the export filtered out by default.
- Population size, sample size, and the seed if you used one.
- The drawn items, in draw order, before any checking started.
- Which items were unobtainable, counted separately from those that failed.
The item you cannot get hold of
This is the moment the sample is most often destroyed, and it never feels like destruction. The record is archived, the unit shipped last week, the person who has it is on leave — so you take the next one.
Do not. Whatever made an item hard to reach is plausibly related to whatever you are checking for, so the substitute is systematically different from the thing it replaced and the difference runs in the worst possible direction.
Record it as unobtainable and report the count separately. A sample of thirty with four unobtainable is a weaker finding stated honestly; a sample of thirty with four substitutions is a stronger-looking finding that is wrong, and nothing downstream will catch it.
Reporting it so it survives a question
Lead with the denominator. "Four of thirty" carries information; "four issues found" does not, and the second phrasing is what invites somebody to ask whether you looked at four items or four hundred.
State the frame in the same sentence as the finding. A defect rate is about the population you sampled from, and the whole point of writing down what the export excluded is that this sentence can be accurate.
And resist extrapolating past what the size supports. A sample of thirty from three thousand tells you about common problems and says nothing at all about anything affecting one item in a hundred, which is worth saying explicitly before somebody else assumes otherwise.
Frequently asked questions
What should I export, and when?
The full list, before deciding anything about sample size, with a note of what the export filtered out by default. Keep it — the later question is what you looked at.
Why draw only once?
Because a quiet second draw after closing the tab means the sample you checked is not the sample you recorded, and nothing downstream will notice.
Why work the items in draw order?
Because working the easy ones first means the hard ones get checked when you are tired, or not at all.
What if I cannot obtain an item?
Record it as unobtainable and count it separately. Substituting the next item biases the sample in the worst possible direction.
Why is substitution so damaging?
Whatever made an item hard to reach is plausibly related to what you are checking for, so the replacement is systematically unlike what it replaced.
How should I report the result?
Lead with the denominator and state the frame in the same sentence. 'Four of thirty' carries information; 'four issues found' does not.
Should I use a seed?
If you will repeat the check, yes. A recorded seed re-draws the identical sample later, which compares the same items over time.
How far can I extrapolate?
Not past what the size supports. Thirty of three thousand speaks to common problems and says nothing about a one-in-a-hundred rate.
Tools for this job
Random Sample From a List
Draw a fixed-size sample without replacement — for audits, spot checks, and surveys.
Random Subset Picker
Include each entry independently at probability p — the sample size is a random variable.
Random Number Generator
Any range, any count, unique or repeating — true cryptographic randomness.
Background reading
Random Sampling for Audits and Surveys
Sampling frames, sample size, and the mistake that invalidates the whole exercise.
Sampling With and Without Replacement
The single decision that changes every number downstream of your draw.
The Law of Large Numbers, Practically
What 'it evens out' actually promises, on what timescale, and what it never promises.