Methods

Reliability

Reliability concerns consistency under specified measurement conditions. Relevant conditions include repeated occasions, multiple items, and different observers. Reliability evidence belongs to scores obtained in a particular setting and population; a familiar instrument name is not a permanent guarantee of dependable measurement.

How much would scores vary because of the measurement process rather than the differences of interest?

How it works

  1. Identify the consistency question: stability over time, coherence among relevant items, or agreement between observers. Choose the comparison conditions to reflect the intended use, including whether genuine change should occur between occasions.
  2. Collect an appropriate reliability sample and preserve administration conditions. For observer ratings, have raters score independently before discussing disagreements; retain those original ratings when estimating agreement.
  3. Report the coefficient or agreement measure, its uncertainty, the sample, and the conditions studied. Inspect patterns of disagreement and revise unclear instructions, then reassess rather than declaring success from one threshold.

Evidence: (Reliability and Validity of Measurement, n.d.); (COSMIN Manual, Version 2.0, n.d.)

Coding interruptions in a group task

Teaching scenario

Synthetic example: Two observers independently code interruptions in recordings of a collaborative puzzle. Before coding, they must decide whether overlapping supportive speech counts as an interruption and whether several words in one turn form one event. They keep their first ratings, compare disagreements, and revise the manual for a later pilot. If both observers consistently code the same event incorrectly, agreement alone will not reveal that conceptual problem. This exercise generates a plan for examining consistency, not a reported reliability coefficient or a claim that the coding scheme has been validated.

This example is constructed for learning. It does not report an observed research result.

What can you conclude?

Helps judge whether scores can support the intended comparisons with acceptable consistency.

Evidence: (Reliability and Validity of Measurement, n.d.); (COSMIN Manual, Version 2.0, n.d.)

Limits and common mistakes

  • Consistency does not establish validity: scores can repeat a systematic mistake.
  • An unstable score may reflect genuine change if the construct or testing context changes; the interval and conditions matter.

A mistake to catch

Using internal consistency as a substitute for every kind of reliability, or calling a high coefficient proof of validity.

Evidence: (Reliability and Validity of Measurement, n.d.); (COSMIN Manual, Version 2.0, n.d.)

Sources and inspection notes

Links checked October 21, 2026. Inspection scope is stated for each source; an access date is not a publication date.

  1. COSMIN manual, version 2.0. (n.d.). https://www.cosmin.nl/wp-content/uploads/COSMIN-manual-V2_final.pdf

    Printed pages 12–14, taxonomy and Table 1.1 Official methods manual for patient-reported outcome measures. Its scope is PROMs; broader psychology applications here are explicit teaching adaptations. Opened and checked 21 October 2026.

  2. Reliability and validity of measurement. (n.d.). https://opentextbc.ca/researchmethods/chapter/reliability-and-validity-of-measurement/

    Test-Retest Reliability; Internal Consistency; Interrater Reliability; Content and Criterion Validity Original teaching chapter. Use distinctions, not its universal-looking coefficient cutoffs or oversimplified alpha explanation. Opened and checked 21 October 2026.

Keep reading

Spielman, R. M., Jenkins, W. J., & Lovett, M. D. (2020). Psychology 2e. OpenStax. https://openstax.org/books/psychology-2e/pages/2-3-analyzing-findings

Nearest related OpenStax Psychology 2e reading. This is a course connection, not evidence for every specialist detail above.

Specialist reading: Reliability and validity of measurement. (n.d.). https://opentextbc.ca/researchmethods/chapter/reliability-and-validity-of-measurement/

Related methods

Try the methods · Read a study · Explore psychological measures