Psychological
Assessment
Summaries
October / November
, THEME 1: PCYCHOMETRIC THEORY
Psychometric properties
The purpose of testing needs to be specified before assessment battery can be assembled. A measure is
evaluated in terms of these characteristics before it’s included in the battery:
✓ What attribute, construct or characteristic does it measure?
✓ Is it appropriate for the group, individual or organisation?
✓ Is the measure psychologically sound? (validity, reliability & eq.)
Norm-referenced vs. criterion-referenced
Norm-referenced: individual’s score on measure is interpreted by comparing score to performance to
similar people (a relevant standardisation sample or norm group) so that individual’s position relative to
normative sample can be determined
Criterion-referenced: individual’s score interpreted by comparing performance to external criterion
(examination / codes). Expectancy table is used to set cut-off score
CHAPTER 3 (3.6)
NORMS
Standard normal distribution
✓ bell shaped curve with mean=0 & SD=1
✓ raw scores have no meaning, so to make the interpretation more meaningful, these raw scores are
converted to normal (standardised) score through statistical transformation
DEFINITION: A measurement against which an individual’s raw score is evaluated so that the individual’s
position relative to that of the normative sample can be determined.
Establishing norm groups
✓ two subgroups: applicant pool & incumbent population
✓ choice of norm group must be representative of both & appropriate for position for which assessment is
conducted
EXAMPLE: pilot selection test battery - norm group selected from both pool of applicants (net yet
competent) & population of licenced pilots (already competent). Norm group is now representative of both
by similar proportions of the biographic data of these 2 groups. Entire test battery then completed by norm
group and standardised / normal scores are calculated for each of the tests in the battery. New applicants’
test scores can now be compared to these established standardised / normal scores.
Co-norming of measures
✓ when 2 or more related but different measures are administered and standardised as a unit on the same
norm group
✓ address potential issues like test-order effects, learning and memory effects, gender, age and education
effects (these issues can’t be addressed if different norm groups are used - so the problem where these
effects are ascribed to sample differences can now be overcome by using a common sample group)
,Types of test norms (a norm score tell you and individual’s standing in relation to norm group)
1. Developmental scales
Rationale: certain human characteristics increase with increases in age and experience
✓ Mental age scales: basal age is computed (highest age at and below which measure was passed).
Child’s mental age on measure = basal age + additional months of credit earned at higher age levels
✓ Grade equivalents: example school setting: pupil’s grade value is described as equivalent to 7th grade
performance in math, 8th grade performance in spelling etc.
2. Percentiles
✓ percentile score = percentage of people in a normative sample who fall below a given raw score
(Pete has percentile score of 70: 70% of normative population scored lower than Pete OR Pete
scored higher than 70% of normative population)
✓ used for individual test performance
✓ 50th percentile = median
✓ disadvantages: unequal scale units, ordinal level measurements, non-comparable
3. Standard scores
✓ z-scores
• expresses an individual’s distance from the mean in terms of the standard deviation units
• raw score=mean is equal to z=0
• Positive z-scores = above average & negative z-scores = below average
• advantages: they represent interval scales & can be statistically manipulated
• disadvantage: half the z-scores in distribution have negative values & range of scores is
limited (-3 to =3)
✓ Linearly transformed standard scores
to eliminate disadvantages of z-scores, this transformation can be done:
• multiply z by constant (to compensate for limited range) & add a constant (to eliminate
negative scores)
• disadvantage: scores are less evident (“understood”) than ordinary z-scores & they are
statistically more complex and less useful
✓ Normalised standard scores
• standard scores that have been transformed to fit a normal distribution
• this is done if there’s reason to assume that the attribute (intelligence) is normally distributed
in the population
• advantage: the distribution of test scores corresponds to the normal distribution
• disadvantage: same as z-scores, and NSS change the form of the original distribution of raw
scores
• McCall’s T-score: mean=50 & SD=10 (to eliminate negative values)
• Stanine scale: range from 1 (low) to 9 (high), mean of 5 & DS of 1.96
▪ advantages: scale units are equal, reflect person’s position, performance in rank order
is evident, comparable across groups, allow statistical manipulation
▪ disadvantages: only 9 scale units, so they are approximate
• Sten scales: same as stanine, only this one has 10 scales (mean=5.5 & DS=2, same advantages
& disadvantages as stanine)
, 4. Deviation IQ scales
✓ normalised standard score with mean=100 & SD=15
✓ used by well-known intelligence measures
✓ advantages: easily comprehensible & interpretable; suitable for age levels +18
✓ disadvantages: not directly comparable with transformed standard scores, because the SD’s differ (p.
42 & 43 table)
Interrelationship of test norms (p. 43 table)
✓ choice of norm score = personal preference
✓ standard score and deviation IQ scores have replaced other types of norm scores
Setting standards and cut-off scores
✓ instead of comparing to norm group, compare to external criteria [you need 70% to pass Health
Board Exams - so 60% may be better than classmate who got 50% (norm referenced comparison),
but you still failed because you are below the cut-off score
✓ necessary where large numbers of people are addressed (pass / fail, job applicants, to treat or not to
treat, etc.)
✓ one way of setting cut-off scores: expectancy table (p.44 table)
✓ advantage: provide easily understandable way of interpreting relationship between test scores and
probable levels of success on criterion
✓ disadvantages / considerations:
▪ correlation data can be unstable from sample to sample (large sampling sizes need to
be used)
▪ magnitude of correlation between scores (predictor) & criterion
▪ criterion changes over time (e.g. job requirements)
▪ as assessment practitioners are encouraged to base decision on more than one test
score, attempts should be made to have a band of cut-off scores rather than a single
score
▪ other assessment info should be considered alongside cut-off score
▪ if cut-off score is determined for whole group, one or more subgroups could be
discriminated against
CHAPTER 4
RELIABILITY
The reliability of a measure refers to the consistency with which it measures what it measures. However,
consistency always implies a certain amount of error in measurement (random error and systematic error).
A person’s performance in 1 administration of a measure doesn’t reflect with complete accuracy the true
amount of the trait he / she possesses.
Numerical expression for reliability: reliability coefficient = correlation coefficient
Types of reliability
Assessment
Summaries
October / November
, THEME 1: PCYCHOMETRIC THEORY
Psychometric properties
The purpose of testing needs to be specified before assessment battery can be assembled. A measure is
evaluated in terms of these characteristics before it’s included in the battery:
✓ What attribute, construct or characteristic does it measure?
✓ Is it appropriate for the group, individual or organisation?
✓ Is the measure psychologically sound? (validity, reliability & eq.)
Norm-referenced vs. criterion-referenced
Norm-referenced: individual’s score on measure is interpreted by comparing score to performance to
similar people (a relevant standardisation sample or norm group) so that individual’s position relative to
normative sample can be determined
Criterion-referenced: individual’s score interpreted by comparing performance to external criterion
(examination / codes). Expectancy table is used to set cut-off score
CHAPTER 3 (3.6)
NORMS
Standard normal distribution
✓ bell shaped curve with mean=0 & SD=1
✓ raw scores have no meaning, so to make the interpretation more meaningful, these raw scores are
converted to normal (standardised) score through statistical transformation
DEFINITION: A measurement against which an individual’s raw score is evaluated so that the individual’s
position relative to that of the normative sample can be determined.
Establishing norm groups
✓ two subgroups: applicant pool & incumbent population
✓ choice of norm group must be representative of both & appropriate for position for which assessment is
conducted
EXAMPLE: pilot selection test battery - norm group selected from both pool of applicants (net yet
competent) & population of licenced pilots (already competent). Norm group is now representative of both
by similar proportions of the biographic data of these 2 groups. Entire test battery then completed by norm
group and standardised / normal scores are calculated for each of the tests in the battery. New applicants’
test scores can now be compared to these established standardised / normal scores.
Co-norming of measures
✓ when 2 or more related but different measures are administered and standardised as a unit on the same
norm group
✓ address potential issues like test-order effects, learning and memory effects, gender, age and education
effects (these issues can’t be addressed if different norm groups are used - so the problem where these
effects are ascribed to sample differences can now be overcome by using a common sample group)
,Types of test norms (a norm score tell you and individual’s standing in relation to norm group)
1. Developmental scales
Rationale: certain human characteristics increase with increases in age and experience
✓ Mental age scales: basal age is computed (highest age at and below which measure was passed).
Child’s mental age on measure = basal age + additional months of credit earned at higher age levels
✓ Grade equivalents: example school setting: pupil’s grade value is described as equivalent to 7th grade
performance in math, 8th grade performance in spelling etc.
2. Percentiles
✓ percentile score = percentage of people in a normative sample who fall below a given raw score
(Pete has percentile score of 70: 70% of normative population scored lower than Pete OR Pete
scored higher than 70% of normative population)
✓ used for individual test performance
✓ 50th percentile = median
✓ disadvantages: unequal scale units, ordinal level measurements, non-comparable
3. Standard scores
✓ z-scores
• expresses an individual’s distance from the mean in terms of the standard deviation units
• raw score=mean is equal to z=0
• Positive z-scores = above average & negative z-scores = below average
• advantages: they represent interval scales & can be statistically manipulated
• disadvantage: half the z-scores in distribution have negative values & range of scores is
limited (-3 to =3)
✓ Linearly transformed standard scores
to eliminate disadvantages of z-scores, this transformation can be done:
• multiply z by constant (to compensate for limited range) & add a constant (to eliminate
negative scores)
• disadvantage: scores are less evident (“understood”) than ordinary z-scores & they are
statistically more complex and less useful
✓ Normalised standard scores
• standard scores that have been transformed to fit a normal distribution
• this is done if there’s reason to assume that the attribute (intelligence) is normally distributed
in the population
• advantage: the distribution of test scores corresponds to the normal distribution
• disadvantage: same as z-scores, and NSS change the form of the original distribution of raw
scores
• McCall’s T-score: mean=50 & SD=10 (to eliminate negative values)
• Stanine scale: range from 1 (low) to 9 (high), mean of 5 & DS of 1.96
▪ advantages: scale units are equal, reflect person’s position, performance in rank order
is evident, comparable across groups, allow statistical manipulation
▪ disadvantages: only 9 scale units, so they are approximate
• Sten scales: same as stanine, only this one has 10 scales (mean=5.5 & DS=2, same advantages
& disadvantages as stanine)
, 4. Deviation IQ scales
✓ normalised standard score with mean=100 & SD=15
✓ used by well-known intelligence measures
✓ advantages: easily comprehensible & interpretable; suitable for age levels +18
✓ disadvantages: not directly comparable with transformed standard scores, because the SD’s differ (p.
42 & 43 table)
Interrelationship of test norms (p. 43 table)
✓ choice of norm score = personal preference
✓ standard score and deviation IQ scores have replaced other types of norm scores
Setting standards and cut-off scores
✓ instead of comparing to norm group, compare to external criteria [you need 70% to pass Health
Board Exams - so 60% may be better than classmate who got 50% (norm referenced comparison),
but you still failed because you are below the cut-off score
✓ necessary where large numbers of people are addressed (pass / fail, job applicants, to treat or not to
treat, etc.)
✓ one way of setting cut-off scores: expectancy table (p.44 table)
✓ advantage: provide easily understandable way of interpreting relationship between test scores and
probable levels of success on criterion
✓ disadvantages / considerations:
▪ correlation data can be unstable from sample to sample (large sampling sizes need to
be used)
▪ magnitude of correlation between scores (predictor) & criterion
▪ criterion changes over time (e.g. job requirements)
▪ as assessment practitioners are encouraged to base decision on more than one test
score, attempts should be made to have a band of cut-off scores rather than a single
score
▪ other assessment info should be considered alongside cut-off score
▪ if cut-off score is determined for whole group, one or more subgroups could be
discriminated against
CHAPTER 4
RELIABILITY
The reliability of a measure refers to the consistency with which it measures what it measures. However,
consistency always implies a certain amount of error in measurement (random error and systematic error).
A person’s performance in 1 administration of a measure doesn’t reflect with complete accuracy the true
amount of the trait he / she possesses.
Numerical expression for reliability: reliability coefficient = correlation coefficient
Types of reliability
