Furr, & Bacharach (2014) – Psychometrics An introduction
Chapter 7 – The Importance of Reliability
Applied behavioural practice: evaluation of an individual’s test score
- True score of individual on psychological construct can never be completely known
use test scores to estimate true scores. Reliability of test scores can be used to
calculate info that helps to evaluate the quality of test scores.
- Sources of info that can help us evaluate test scores are point estimate + confidence
interval.
- Point estimates of true scores: specific value that’s interpreted as a ‘best estimate’ of
individual’s standing on a particular psychological attribute.
o 2 kinds:
Based solely on individual’s observed test score: point estimate is the
observed score with test taken at given point in time is the single
best estimate of quantity of underlying psychological attribute at the
moment the individual took the test.
Adjusted true score estimate: takes measurement error into account.
Individual’s test score may be inflated/deflated because of momentary
factors (e.g., fatigue = error) test score at point is artificially
higher/lower compared with score when doing test 2nd time/occasion.
Both observed test scores are point estimates of true score. First
estimate can be used to estimate results of 2nd test/occasion.
o Regression to the mean: likelihood that on 2nd testing an individual’s score is
likely to be closer to group mean than was his/her 1st score (below mean =
scoring higher, above mean = scoring lower). Based on CTT + random
measurement error: is random + likely to affect all test scores.
o Adjusted true score estimate reflects discrepancy in individual’s observed
scores that’s likely to arise between 2 testing occasions.
Size + direction of discrepancy will be a function of 3 factors:
Reliability of test scores.
Size of difference between individual’s original observed test
score + mean of test scores.
Direction of difference between original score + mean of test
scores.
𝑋𝑡̂𝑖 = 𝑋𝑒𝑠𝑡 = 𝑋̅ + 𝑅𝑥𝑥 (𝑋𝑜𝑖 − 𝑋̅ )
𝑋𝑒𝑠𝑡 = adjusted true score estimate.
𝑋̅ = test score mean.
𝑅𝑥𝑥 = reliability of the test.
𝑋𝑜𝑖 = individual’s observed score.
When 𝑅𝑥𝑥 = 0, 𝑋𝑡̂𝑖 = 𝑋̅ 𝑋𝑜𝑖 = 𝑋𝑒𝑖 , therefore no reliable
individual differences in true score.
When 𝑅𝑥𝑥 = 1, 𝑋𝑡̂𝑖 = 𝑋𝑜𝑖 No measurement error, therefore
observed score is equal to true score.
Estimated true score is closer to mean than the initial observed score,
because adjusted true score attempts to account for occurrence of
regression to the mean.
1
Chapter 7 – The Importance of Reliability
Applied behavioural practice: evaluation of an individual’s test score
- True score of individual on psychological construct can never be completely known
use test scores to estimate true scores. Reliability of test scores can be used to
calculate info that helps to evaluate the quality of test scores.
- Sources of info that can help us evaluate test scores are point estimate + confidence
interval.
- Point estimates of true scores: specific value that’s interpreted as a ‘best estimate’ of
individual’s standing on a particular psychological attribute.
o 2 kinds:
Based solely on individual’s observed test score: point estimate is the
observed score with test taken at given point in time is the single
best estimate of quantity of underlying psychological attribute at the
moment the individual took the test.
Adjusted true score estimate: takes measurement error into account.
Individual’s test score may be inflated/deflated because of momentary
factors (e.g., fatigue = error) test score at point is artificially
higher/lower compared with score when doing test 2nd time/occasion.
Both observed test scores are point estimates of true score. First
estimate can be used to estimate results of 2nd test/occasion.
o Regression to the mean: likelihood that on 2nd testing an individual’s score is
likely to be closer to group mean than was his/her 1st score (below mean =
scoring higher, above mean = scoring lower). Based on CTT + random
measurement error: is random + likely to affect all test scores.
o Adjusted true score estimate reflects discrepancy in individual’s observed
scores that’s likely to arise between 2 testing occasions.
Size + direction of discrepancy will be a function of 3 factors:
Reliability of test scores.
Size of difference between individual’s original observed test
score + mean of test scores.
Direction of difference between original score + mean of test
scores.
𝑋𝑡̂𝑖 = 𝑋𝑒𝑠𝑡 = 𝑋̅ + 𝑅𝑥𝑥 (𝑋𝑜𝑖 − 𝑋̅ )
𝑋𝑒𝑠𝑡 = adjusted true score estimate.
𝑋̅ = test score mean.
𝑅𝑥𝑥 = reliability of the test.
𝑋𝑜𝑖 = individual’s observed score.
When 𝑅𝑥𝑥 = 0, 𝑋𝑡̂𝑖 = 𝑋̅ 𝑋𝑜𝑖 = 𝑋𝑒𝑖 , therefore no reliable
individual differences in true score.
When 𝑅𝑥𝑥 = 1, 𝑋𝑡̂𝑖 = 𝑋𝑜𝑖 No measurement error, therefore
observed score is equal to true score.
Estimated true score is closer to mean than the initial observed score,
because adjusted true score attempts to account for occurrence of
regression to the mean.
1
