A sample is only as good as the list it was drawn from. If your list of production changes leaves out the changes that skipped review, the auditor can sample from it all day and never find a problem. Auditors know this, so before they pick a single item, they test the list.
Teams tend to treat the population export as administration: pull a CSV, send it, wait for the sample. Auditors treat it as evidence that itself needs evidence. When the two views collide, fieldwork stalls on a spreadsheet nobody thought was important.
This article covers what auditors mean by completeness and accuracy, the methods they accept as proof, where populations usually fail, and how to have yours ready before the request arrives.
Why the population is tested before the sample
Auditors call a report or export that you generate for them information produced by the entity, usually shortened to IPE. The term comes from financial auditing, and SOC 2 auditors apply the same discipline. Because you produced it, they cannot assume it is reliable. They have to test it.
The test has two halves:
Completeness. Every item that belongs in the population for the period is there. No changes, hires, terminations, or incidents are missing.
Accuracy. The fields the auditor will rely on are correct: dates, approvers, status, the person or system involved.
Once the population is accepted, the auditor selects a sample from it. How that selection works is covered in how auditors sample controls. This article is about the step before.
How formally this is done varies. SOC 2 firms often document completeness and accuracy testing for each population explicitly. ISO 27001 certification auditors also sample, but they tend to be less formal about the source list. Ask your auditor what they expect, early.
What auditors accept as proof
There is no single method. Most auditors use one or more of the following, and the choice depends on how the report was produced.
Watching the report run
The most common approach is observation. On a screen share, you open the source system, set the parameters, and run the export while the auditor watches. They note the date range, the filters, and the record count. You send the file, and they confirm the count matches.
This works well for standard reports from a system of record. It removes the question of whether anyone edited the list between pulling it and sending it.
Inspecting the query logic
If the population comes from a custom SQL query or a script, watching it run is not enough. The query could be wrong. The auditor will want to read the logic and understand every filter.
A clause such as status = 'merged' quietly excludes changes that were pushed directly. A join on a team table drops records from people who have since left. These are honest mistakes, and they are exactly what the inspection is designed to catch.
Many auditors treat a vendor's standard, unmodified report as more reliable than a query you wrote. That is not a rule, but it is a reason to prefer built-in reports where one exists.
Reconciling to an independent source
The strongest proof of completeness is agreement between two systems that do not depend on each other. If the HR system shows a number of terminations in the period, the identity provider should show matching deactivations. If the deploy log shows a set of releases, each should trace to a merged pull request or a change ticket.
Some auditors also use a rollforward: the accounts at the start of the period, plus additions, minus removals, should equal the accounts at the end. A mismatch means something was created or deleted outside the process you described.
Sequence and gap checks
Where items carry sequential identifiers, such as ticket numbers, the auditor may look for gaps. A missing number is not a failure in itself. It needs an explanation: the ticket was deleted, moved to another project, or never created.
Matching each population to a cross-check
The table below shows common populations with a system of record and an independent source to reconcile against. Your systems will differ, but the pairing principle holds.
| Population | Pull it from | Reconcile it against |
|---|---|---|
| New hires | HR system | Account creations in the identity provider |
| Terminations | HR system | Deactivations in the identity provider |
| Production changes | Deploy pipeline log | Merged pull requests or change tickets |
| Security incidents | Incident tracker | Escalations from alerting and on-call tools |
| Vendors | Vendor inventory | Accounts payable or expense records |
| Accounts in a system | That system's user list | Group assignments in the identity provider |
The vendor row surprises people. Payment records are often the only place a tool bought on a company card shows up, which makes them the honest test of whether your vendor inventory is complete.
Where populations usually fail
The same handful of problems account for most rejected populations.
Assembled by hand. A list built in a spreadsheet from several sources, with rows added or removed, cannot be tested against the system. Send the raw export. Keep your working copy separate.
The wrong date field. Filtering changes by creation date when the control applies at deploy date gives you a different population. So does ignoring time zones at the period boundary.
Silent filters. Only tickets in a "Done" state. Only one repository. Only the main change project, when emergency changes live in another.
Pulled too late. Many systems drop deleted users from their lists. A user export pulled after the period ends may no longer show the people who left, which is the population the auditor cares about most. As access reviews that hold up puts it, evidence is a claim about a moment.
More than one source. "All production changes" may include application deploys, infrastructure as code, database migrations, and configuration changes made in a console. If those flow through different systems, each is part of the population.
Preparing populations before fieldwork
You can do most of this work weeks before the auditor asks. The request itself usually arrives on the PBC list, covered in the PBC list: handling auditor requests.
List every population your controls imply. Any control the auditor will test by sampling needs one.
Name the system of record and the exact report or query that produces the list. Save the query, not a description of it.
Write down the boundaries. The period, what counts as one item, and anything excluded with the reason.
Pick a reconciling source for each population, using the table above as a starting point.
Do a dry run. Pull the population, reconcile it, and investigate every difference before the auditor sees it.
Preserve raw exports. Record when each was run, by whom, and with what parameters. A system-generated file with that context is stronger than a screenshot, for reasons covered in screenshots vs system-generated evidence.
Agree the method with your auditor. Some will want to observe every pull. Others will accept a query review and reconciliation. Knowing which saves a round of requests.
The question to ask about every population
For each list, ask: is there any way this event can happen without appearing in this system?
Break-glass accounts. Changes made directly in a cloud console. Database edits run from a laptop. Contractors who never pass through the HR system. These side doors are where an experienced auditor looks, because they are where populations are incomplete in practice.
You will not always be able to close them, and proving a negative is hard. What you can do is either bring the side door into the population, or show a detective control that would catch its use, such as an alert on console changes that someone reviews. If neither exists, tell your auditor before they find it. That conversation goes much better in week one.
How TrustLynce helps
TrustLynce collects evidence on a schedule from cloud, identity, HR, and code systems, timestamped and attributed, so populations exist as system exports from the moment they apply. The auditor portal keeps population requests, the files sent, and any follow-up questions in one place.