02. Appraisal clinic: trace the evidence behind a claim
Lesson
LEARNING OBJECTIVES
Support appraisal judgments with reported evidence, distinguish missing reporting from demonstrated flaws, and evaluate whether analysis and conclusions match the design.
USE AN EVIDENCE-TO-JUDGMENT RECORD
For each appraisal domain, record what the report says, where it is stated, why it matters, your interpretation and unresolved questions. Mark information as reported, not reported or unclear. Do not invent methods to fill gaps, and do not assume every unreported procedure was never performed.
Assess participant selection, exposure/outcome measurement, timing, confounding, missingness, analysis, selective reporting and applicability. Ask whether clustering, matching or repeated observations are respected, and whether adjusted effects target the same question as crude results.
WORKED FICTIONAL PAPER
A report compares service use among 300 people reached through a clinic messaging group. It states that long travel time causes low service use. Travel time and use were self-reported in the same online questionnaire. Only people with working phone access were invited. Income and health needs were not measured; 25% of invitees completed the survey, and outcome missingness among respondents is unclear.
Evidence-linked appraisal: recruitment limits coverage and may create selection differences; simultaneous self-report leaves temporal and measurement questions; important common causes may be absent; and unclear response and item-missingness processes limit interpretation. The paper describes an association in a selected respondent group and does not, as presented, establish a causal effect.
A useful critique also notes what the paper contributes: an initial view of respondents' experiences and hypotheses for better-designed work. Critical appraisal is not an exercise in dismissing every imperfect study.
APPLICABILITY IS A SEPARATE QUESTION
Even a credible estimate may transfer poorly when populations, implementation, resources, outcome definitions or baseline frequency differ. Specify what matches your setting, what differs and what further information would be needed. Do not equate “conducted nearby” with automatically applicable.
YOUR TASK
Use Research lab → Critical appraisal for one paper you can lawfully access. Make at least six domain judgments, each linked to a section, table or statement. Include two strengths, two consequential limitations and two unresolved questions.
https://aces-af.org/learn/index.php?r=tool&tool=appraisal
Write a balanced 150-word conclusion: what the study estimates, how credible it appears under stated assumptions, how relevant it is and what it cannot establish. Select a suitable reporting guideline to identify missing information; do not turn its checklist into a total quality score.
SELF-CHECK AND FEEDBACK
Does “unclear allocation method” prove deliberate manipulation? No; describe the gap and its implications. Can regression repair all recruitment problems? No. Is an imperfect study necessarily useless? No; judge its contribution and limitations in relation to a specific question.
PORTFOLIO OUTPUT
An evidence-linked appraisal, a balanced conclusion and a list of questions for clarification or future research.
FURTHER READING
EQUATOR: reporting guidelines and their purpose.
https://www.equator-network.org/reporting-guidelines/strobe/
CDC Field Epidemiology Manual.
https://www.cdc.gov/field-epi-manual/
Support appraisal judgments with reported evidence, distinguish missing reporting from demonstrated flaws, and evaluate whether analysis and conclusions match the design.
USE AN EVIDENCE-TO-JUDGMENT RECORD
For each appraisal domain, record what the report says, where it is stated, why it matters, your interpretation and unresolved questions. Mark information as reported, not reported or unclear. Do not invent methods to fill gaps, and do not assume every unreported procedure was never performed.
Assess participant selection, exposure/outcome measurement, timing, confounding, missingness, analysis, selective reporting and applicability. Ask whether clustering, matching or repeated observations are respected, and whether adjusted effects target the same question as crude results.
WORKED FICTIONAL PAPER
A report compares service use among 300 people reached through a clinic messaging group. It states that long travel time causes low service use. Travel time and use were self-reported in the same online questionnaire. Only people with working phone access were invited. Income and health needs were not measured; 25% of invitees completed the survey, and outcome missingness among respondents is unclear.
Evidence-linked appraisal: recruitment limits coverage and may create selection differences; simultaneous self-report leaves temporal and measurement questions; important common causes may be absent; and unclear response and item-missingness processes limit interpretation. The paper describes an association in a selected respondent group and does not, as presented, establish a causal effect.
A useful critique also notes what the paper contributes: an initial view of respondents' experiences and hypotheses for better-designed work. Critical appraisal is not an exercise in dismissing every imperfect study.
APPLICABILITY IS A SEPARATE QUESTION
Even a credible estimate may transfer poorly when populations, implementation, resources, outcome definitions or baseline frequency differ. Specify what matches your setting, what differs and what further information would be needed. Do not equate “conducted nearby” with automatically applicable.
YOUR TASK
Use Research lab → Critical appraisal for one paper you can lawfully access. Make at least six domain judgments, each linked to a section, table or statement. Include two strengths, two consequential limitations and two unresolved questions.
https://aces-af.org/learn/index.php?r=tool&tool=appraisal
Write a balanced 150-word conclusion: what the study estimates, how credible it appears under stated assumptions, how relevant it is and what it cannot establish. Select a suitable reporting guideline to identify missing information; do not turn its checklist into a total quality score.
SELF-CHECK AND FEEDBACK
Does “unclear allocation method” prove deliberate manipulation? No; describe the gap and its implications. Can regression repair all recruitment problems? No. Is an imperfect study necessarily useless? No; judge its contribution and limitations in relation to a specific question.
PORTFOLIO OUTPUT
An evidence-linked appraisal, a balanced conclusion and a list of questions for clarification or future research.
FURTHER READING
EQUATOR: reporting guidelines and their purpose.
https://www.equator-network.org/reporting-guidelines/strobe/
CDC Field Epidemiology Manual.
https://www.cdc.gov/field-epi-manual/
My study notes
Stored only in this browser. Do not enter patient information.Saved in this browser. No account needed.