03. Appraise diagnostic accuracy and intended-use evidence
Lesson
LEARNING OBJECTIVES
Assess the design behind an accuracy table, distinguish sensitivity/specificity from predictive values, and identify applicability and verification problems.
BEGIN WITH INTENDED USE
Ask what the index test is meant to do: screening, triage, diagnosis or another role. Define the population, setting, threshold and clinical pathway before judging whether the evidence fits. A high accuracy estimate from a selected sample does not establish usefulness in every setting.
In a diagnostic 2 × 2 table, sensitivity conditions on reference-standard disease presence and specificity on its absence. Predictive values condition on the observed test result. Predictive values depend on disease frequency in the population assessed; sensitivity and specificity may also vary with the spectrum and measurement context.
APPRAISE RECRUITMENT AND VERIFICATION
Were participants recruited in a way that reflects the intended population? Were relevant disease stages, alternative conditions and difficult-to-interpret results included? Comparing very severe cases with unusually healthy volunteers can produce a misleading picture of real-world performance.
What reference standard was used, and how credible is it? Did all participants receive it, or did verification depend on the index result? Were interpreters aware of other results? Were the tests performed close enough in time for the comparison to be meaningful? An imperfect reference may misclassify status and complicate interpretation.
WORKED FICTIONAL APPRAISAL
A study advertises 98% sensitivity after testing previously diagnosed severe cases. Controls were healthy staff volunteers. The index threshold was chosen after trying many cutoffs in the same dataset; indeterminate results were excluded without counts.
The headline does not reveal performance in the intended screening population. Recruitment spectrum, threshold selection, missing/indeterminate handling and reference procedures need scrutiny. Separate what is reported from what is assumed, and ask how many participants were affected by exclusions.
RESULTS NEED MORE THAN A HEADLINE
Extract the full 2 × 2 counts, denominators, threshold, uncertainty and missingness. If thresholds were developed and evaluated in the same data, assess optimism and the need for suitable validation. Accuracy alone does not establish benefits, harms, feasibility, costs or improved patient outcomes.
Use STARD to assess reporting completeness for diagnostic studies; use an appropriate appraisal framework and reasoning for validity. Do not calculate a generic checklist percentage and call it an accuracy-quality score.
YOUR TASK
Select an accessible diagnostic study or use the fictional case. Write an appraisal under intended use, selection, index test, reference standard, verification, flow/timing, missing results and applicability. For each judgment record evidence and uncertainty.
SELF-CHECK AND FEEDBACK
Does 98% sensitivity establish that 98% of positive results are true cases? No; that confuses sensitivity and positive predictive value. Does excluding indeterminate results always preserve validity? No; report their number, reasons and analytical handling. Can an accuracy result alone show improved health outcomes? No.
PORTFOLIO OUTPUT
A diagnostic evidence-extraction sheet and a balanced applicability judgment.
FURTHER READING
FDA: Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests.
https://www.fda.gov/regulatory-information/search-fda-guidance-documents/statistical-guidance-reporting-results-studies-evaluating-diagnostic-tests-guidance-industry-and-fda
EQUATOR reporting guidelines, including STARD.
https://www.equator-network.org/reporting-guidelines/
Assess the design behind an accuracy table, distinguish sensitivity/specificity from predictive values, and identify applicability and verification problems.
BEGIN WITH INTENDED USE
Ask what the index test is meant to do: screening, triage, diagnosis or another role. Define the population, setting, threshold and clinical pathway before judging whether the evidence fits. A high accuracy estimate from a selected sample does not establish usefulness in every setting.
In a diagnostic 2 × 2 table, sensitivity conditions on reference-standard disease presence and specificity on its absence. Predictive values condition on the observed test result. Predictive values depend on disease frequency in the population assessed; sensitivity and specificity may also vary with the spectrum and measurement context.
APPRAISE RECRUITMENT AND VERIFICATION
Were participants recruited in a way that reflects the intended population? Were relevant disease stages, alternative conditions and difficult-to-interpret results included? Comparing very severe cases with unusually healthy volunteers can produce a misleading picture of real-world performance.
What reference standard was used, and how credible is it? Did all participants receive it, or did verification depend on the index result? Were interpreters aware of other results? Were the tests performed close enough in time for the comparison to be meaningful? An imperfect reference may misclassify status and complicate interpretation.
WORKED FICTIONAL APPRAISAL
A study advertises 98% sensitivity after testing previously diagnosed severe cases. Controls were healthy staff volunteers. The index threshold was chosen after trying many cutoffs in the same dataset; indeterminate results were excluded without counts.
The headline does not reveal performance in the intended screening population. Recruitment spectrum, threshold selection, missing/indeterminate handling and reference procedures need scrutiny. Separate what is reported from what is assumed, and ask how many participants were affected by exclusions.
RESULTS NEED MORE THAN A HEADLINE
Extract the full 2 × 2 counts, denominators, threshold, uncertainty and missingness. If thresholds were developed and evaluated in the same data, assess optimism and the need for suitable validation. Accuracy alone does not establish benefits, harms, feasibility, costs or improved patient outcomes.
Use STARD to assess reporting completeness for diagnostic studies; use an appropriate appraisal framework and reasoning for validity. Do not calculate a generic checklist percentage and call it an accuracy-quality score.
YOUR TASK
Select an accessible diagnostic study or use the fictional case. Write an appraisal under intended use, selection, index test, reference standard, verification, flow/timing, missing results and applicability. For each judgment record evidence and uncertainty.
SELF-CHECK AND FEEDBACK
Does 98% sensitivity establish that 98% of positive results are true cases? No; that confuses sensitivity and positive predictive value. Does excluding indeterminate results always preserve validity? No; report their number, reasons and analytical handling. Can an accuracy result alone show improved health outcomes? No.
PORTFOLIO OUTPUT
A diagnostic evidence-extraction sheet and a balanced applicability judgment.
FURTHER READING
FDA: Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests.
https://www.fda.gov/regulatory-information/search-fda-guidance-documents/statistical-guidance-reporting-results-studies-evaluating-diagnostic-tests-guidance-industry-and-fda
EQUATOR reporting guidelines, including STARD.
https://www.equator-network.org/reporting-guidelines/
My study notes
Stored only in this browser. Do not enter patient information.Saved in this browser. No account needed.