06. Audit bias and plan sensitivity analyses
Lesson
LEARNING OBJECTIVES
Identify how a specific bias could enter a study, separate prevention from analysis-stage remedies, and preplan checks that test important assumptions.
USE A BIAS REGISTER
For each study stage, record the possible mechanism, affected population or comparison, likely information needed, prevention measure, diagnostic check and remaining limitation. Avoid generic entries such as “bias will be minimized.” A register is a reasoning tool, not an unweighted quality score.
Consider frame coverage and recruitment; contact attempts and response; exposure ascertainment; outcome detection; follow-up and missingness; analysis decisions; and selective reporting. Assess whether a problem could differ between exposure or outcome groups. The direction of bias may be uncertain, so do not label every error “toward the null.”
WORKED FICTIONAL AUDIT
A household study recruits only during weekday mornings. Working caregivers are frequently absent. If working patterns relate to both water access and reported childhood symptoms, the observed association may differ from the intended population association.
Prevention could include planned repeat contacts at different times and accurate eligibility/response records. A process check compares contact and response patterns across prespecified areas or household characteristics. Weighting may be considered under documented assumptions and adequate information, but cannot restore information from a completely unobserved group automatically.
A second problem is differential symptom recall. Use the same wording, recall period and interviewer training across groups. Document the outcome's limitations and consider an independent validation subset where feasible and appropriate. Increasing the sample size does not fix systematic recall error.
SENSITIVITY PLANNING
A sensitivity analysis asks whether conclusions depend on a consequential analytical choice or assumption. Examples include a prespecified alternative outcome definition, examining results with and without a disputed record after source verification, or evaluating plausible missing-outcome scenarios.
For unmeasured confounding or misclassification, quantitative bias analysis requires defensible inputs and suitable methods. Do not invent accuracy parameters or add arbitrary constants simply to obtain a reassuring estimate. Specialist input may be needed.
YOUR TASK
Write five bias-register entries for your study. For two, specify a prevention action before recruitment and an analysis-stage check. Distinguish what a check can reveal from what it cannot establish.
For one missing-outcome problem, list the information you would collect about losses and propose a sensitivity question. State in advance which result would lead you to weaken your conclusion or revise the design.
SELF-CHECK
1. Does a low p-value rule out bias? No; statistical testing does not remove systematic errors.
2. Can response weighting guarantee representativeness? No; it depends on measured information, assumptions and coverage.
3. Should you retain only sensitivity analyses that agree with the main finding? No; report the planned checks and consequential disagreements transparently.
PORTFOLIO OUTPUT
A practical bias register, two prevention procedures and a short sensitivity-analysis plan linked to explicit assumptions.
FURTHER READING
CDC Field Epidemiology Manual: Collecting Data and Analyzing and Interpreting Data.
https://www.cdc.gov/field-epi-manual/php/chapters/collecting-data.html
https://www.cdc.gov/field-epi-manual/php/chapters/analyze-interpret-data.html
Identify how a specific bias could enter a study, separate prevention from analysis-stage remedies, and preplan checks that test important assumptions.
USE A BIAS REGISTER
For each study stage, record the possible mechanism, affected population or comparison, likely information needed, prevention measure, diagnostic check and remaining limitation. Avoid generic entries such as “bias will be minimized.” A register is a reasoning tool, not an unweighted quality score.
Consider frame coverage and recruitment; contact attempts and response; exposure ascertainment; outcome detection; follow-up and missingness; analysis decisions; and selective reporting. Assess whether a problem could differ between exposure or outcome groups. The direction of bias may be uncertain, so do not label every error “toward the null.”
WORKED FICTIONAL AUDIT
A household study recruits only during weekday mornings. Working caregivers are frequently absent. If working patterns relate to both water access and reported childhood symptoms, the observed association may differ from the intended population association.
Prevention could include planned repeat contacts at different times and accurate eligibility/response records. A process check compares contact and response patterns across prespecified areas or household characteristics. Weighting may be considered under documented assumptions and adequate information, but cannot restore information from a completely unobserved group automatically.
A second problem is differential symptom recall. Use the same wording, recall period and interviewer training across groups. Document the outcome's limitations and consider an independent validation subset where feasible and appropriate. Increasing the sample size does not fix systematic recall error.
SENSITIVITY PLANNING
A sensitivity analysis asks whether conclusions depend on a consequential analytical choice or assumption. Examples include a prespecified alternative outcome definition, examining results with and without a disputed record after source verification, or evaluating plausible missing-outcome scenarios.
For unmeasured confounding or misclassification, quantitative bias analysis requires defensible inputs and suitable methods. Do not invent accuracy parameters or add arbitrary constants simply to obtain a reassuring estimate. Specialist input may be needed.
YOUR TASK
Write five bias-register entries for your study. For two, specify a prevention action before recruitment and an analysis-stage check. Distinguish what a check can reveal from what it cannot establish.
For one missing-outcome problem, list the information you would collect about losses and propose a sensitivity question. State in advance which result would lead you to weaken your conclusion or revise the design.
SELF-CHECK
1. Does a low p-value rule out bias? No; statistical testing does not remove systematic errors.
2. Can response weighting guarantee representativeness? No; it depends on measured information, assumptions and coverage.
3. Should you retain only sensitivity analyses that agree with the main finding? No; report the planned checks and consequential disagreements transparently.
PORTFOLIO OUTPUT
A practical bias register, two prevention procedures and a short sensitivity-analysis plan linked to explicit assumptions.
FURTHER READING
CDC Field Epidemiology Manual: Collecting Data and Analyzing and Interpreting Data.
https://www.cdc.gov/field-epi-manual/php/chapters/collecting-data.html
https://www.cdc.gov/field-epi-manual/php/chapters/analyze-interpret-data.html
My study notes
Stored only in this browser. Do not enter patient information.Saved in this browser. No account needed.