07. Maintain secure data and a traceable quality workflow
Lesson
LEARNING OBJECTIVES
Describe data handling from collection to deletion, distinguish source data from analysis copies, and implement quality checks that preserve an audit trail.
PLAN THE DATA LIFECYCLE
Document what is collected, why each identifier is necessary, how data move between devices or paper forms and storage, who has access, how backups are restored, how long records are retained and who authorizes deletion or sharing. Match procedures to the project's approved requirements and actual technical resources.
Use study IDs in analysis files where appropriate and restrict any linkage key separately. Pseudonymized records may still be identifiable. Limit permissions by role; do not use a shared public folder or an unprotected email attachment as a default data-transfer method. Protect paper materials as well as digital files.
PRESERVE THE SOURCE
Keep an unchanged authorized source copy. Work on documented derivatives rather than overwriting original values. Maintain a codebook, processing instructions or scripts, software versions and a change log. Versioned files should have clear ownership and a naming convention that avoids confusion over “final-final” copies.
A correction log records the record ID, variable, original value, proposed value, evidence, reason, authorized person and date. Record IDs in logs must themselves be protected. A real data audit trail belongs in restricted project storage, not in a public LMS exercise.
WORKED FICTIONAL CHECKS
An age of 999 violates the dictionary's plausible range, but do not replace it with a guessed value. Check permitted source information, document the finding and apply the declared unresolved-value rule if it cannot be resolved.
Two rows with the same study ID might be an accidental duplicate or valid repeated visits. Compare dates, design and collection history before deletion.
A quantity recorded in litres in one batch and millilitres in another requires a verified unit conversion and documented provenance, not an arbitrary decision to remove the larger values.
BUILD THREE KINDS OF CHECKS
Range checks identify values outside allowed bounds. Consistency checks compare logically related variables. Process checks monitor missingness, timing, contact outcomes, duplicate submissions and deviations. Distinguish hard constraints from warnings that require review of a possible genuine unusual observation.
Daily monitoring should help improve procedures, not reward interviewers for generating implausibly complete or uniform data. Prespecify how queries are raised, resolved and approved, and how unresolved issues enter analysis.
YOUR TASK
Draft a data-flow description and an access-role table. Write one range rule, one consistency rule and one process indicator. Create three fictional correction-log entries, including an unresolved record and a duplicate investigation.
Describe a backup-restoration check and a release checklist for an analysis dataset that includes a codebook, missing-value rules and retained design variables.
SELF-CHECK AND FEEDBACK
Does an outlier prove an entry error? No. Is a backup useful if nobody can restore it? Not reliably; test the process. Can “we removed bad data” substitute for a log? No; decisions must be traceable and justified.
PORTFOLIO OUTPUT
A data-management plan, quality rules, correction log and analysis-release checklist.
FURTHER READING
CDC Field Epidemiology Manual: Collecting Data.
https://www.cdc.gov/field-epi-manual/php/chapters/collecting-data.html
Describe data handling from collection to deletion, distinguish source data from analysis copies, and implement quality checks that preserve an audit trail.
PLAN THE DATA LIFECYCLE
Document what is collected, why each identifier is necessary, how data move between devices or paper forms and storage, who has access, how backups are restored, how long records are retained and who authorizes deletion or sharing. Match procedures to the project's approved requirements and actual technical resources.
Use study IDs in analysis files where appropriate and restrict any linkage key separately. Pseudonymized records may still be identifiable. Limit permissions by role; do not use a shared public folder or an unprotected email attachment as a default data-transfer method. Protect paper materials as well as digital files.
PRESERVE THE SOURCE
Keep an unchanged authorized source copy. Work on documented derivatives rather than overwriting original values. Maintain a codebook, processing instructions or scripts, software versions and a change log. Versioned files should have clear ownership and a naming convention that avoids confusion over “final-final” copies.
A correction log records the record ID, variable, original value, proposed value, evidence, reason, authorized person and date. Record IDs in logs must themselves be protected. A real data audit trail belongs in restricted project storage, not in a public LMS exercise.
WORKED FICTIONAL CHECKS
An age of 999 violates the dictionary's plausible range, but do not replace it with a guessed value. Check permitted source information, document the finding and apply the declared unresolved-value rule if it cannot be resolved.
Two rows with the same study ID might be an accidental duplicate or valid repeated visits. Compare dates, design and collection history before deletion.
A quantity recorded in litres in one batch and millilitres in another requires a verified unit conversion and documented provenance, not an arbitrary decision to remove the larger values.
BUILD THREE KINDS OF CHECKS
Range checks identify values outside allowed bounds. Consistency checks compare logically related variables. Process checks monitor missingness, timing, contact outcomes, duplicate submissions and deviations. Distinguish hard constraints from warnings that require review of a possible genuine unusual observation.
Daily monitoring should help improve procedures, not reward interviewers for generating implausibly complete or uniform data. Prespecify how queries are raised, resolved and approved, and how unresolved issues enter analysis.
YOUR TASK
Draft a data-flow description and an access-role table. Write one range rule, one consistency rule and one process indicator. Create three fictional correction-log entries, including an unresolved record and a duplicate investigation.
Describe a backup-restoration check and a release checklist for an analysis dataset that includes a codebook, missing-value rules and retained design variables.
SELF-CHECK AND FEEDBACK
Does an outlier prove an entry error? No. Is a backup useful if nobody can restore it? Not reliably; test the process. Can “we removed bad data” substitute for a log? No; decisions must be traceable and justified.
PORTFOLIO OUTPUT
A data-management plan, quality rules, correction log and analysis-release checklist.
FURTHER READING
CDC Field Epidemiology Manual: Collecting Data.
https://www.cdc.gov/field-epi-manual/php/chapters/collecting-data.html
My study notes
Stored only in this browser. Do not enter patient information.Saved in this browser. No account needed.