TEST DEVELOPMENT L.L Thurstone — very influential in the development of sound
an umbrella term for all that goes into the process of creating scaling methods.
a test. Types of scales — Scales are instruments to measure some trait,
Process of Test Development state or ability. May be categorized in many ways (e.g.
1. test conceptualization – multidimensional, unidemensional, etc.).
Conceptualizing a test Age-based scale — If the test taker’s test performance as a
2. test construction – tests are function of age is critical interest. (e.g. RPM)
drafted, Grade-based scale — If the test taker’s test performance as
stage of process of test development a function of grade is critical interest.
that entails writing test items (or re- Stanine Scale — if all raw scores on test are to be
writing or revising existing items) as transformed into scores that can range from 1 to 9
well as formatting items, scoring Describing Scales: unidimensional/multidimensional,
rules, and otherwise designing and comparative/ categorical
building a test.
3. test tryout – test is tried on a group,
administered to a representative
sample of test takers under Test Construction - Scaling Methods
conditions that stimulate the Numbers can be assigned to responses to calculate test scores
conditions that the final version of the test will be using a number of methods.
administered. a) Rating Scales — a
4. item analysis – using statistical analysis to assists in making grouping of words,
judgment about which items are good, needs to be revised or statements, or symbols on
discarded. Analysis involves item reliability, item validity, item which judgments of the
discrimination, or item difficulty. strength of a particular
5. test revision - At some point, the test developer will either trait, attitude, or emotion
finalize the form of the test or go back to the proverbial are indicated by the test
drawing board. taker. (e.g MDBS-R)
Action taken to modify a test’s content or format for the can be used to record
purpose of improving the tests’ effectiveness as a tool of judgments of oneself,
measurement. others, experiences, or objects, and they can take several
forms
Test Conceptualization Summative Scale — term for summing the ratings across all
The beginnings of any published test can probably be traced the items resulting to the final test score which is an Ordinal
to thoughts — self-talk, in behavioral terms Level Data.
The impetus for developing a new test is some thought that differs in the number of dimensions underlying the ratings
“there ought to be a test designed to measure [fill in the being made: unidimensional — only one dimension is
blank] in [such and such] way…” presumed to underlie the ratings/ multidimensional — more
The stimulus could be almost anything; knowledge of than one dimension is thought to guide the test taker’s
psychometric problems with other tests, a new social responses.
phenomenon, or any number of things. b) Likert scale — is used extensively in psychology, usually to
There may be a need to assess mastery in an emerging scale attitudes.
occupation. relatively easy to
Some preliminary questions construct
What is the test What special training will Each item presents
designed to measure? be required of test users the test taker with
What is the objective of for administering or five alternative responses (sometimes seven), usually on an
the test? interpreting the agree–disagree or approve–disapprove continuum
Is there a need for this test? usually reliable, which may account for their widespread
test? What types of responses popularity
Who will use this test? will be required of test c) Method of Paired Comparisons – Test-takers must choose
Who will take this test? takers? between two
What content will the Who benefits from an alternatives/pair of
test cover? administration of this stimuli (images,
How will the test be test? objects, statements)
administered? Is there any potential for according to some
What is the ideal format harm as the result of an rule.
of the test? administration of this For each pair of options, test takers receive a higher
Should more than one test? score for selecting the option deemed more justifiable by
form of the test be How will meaning be the majority of a group of judges.
developed? attributed to scores on The test score would reflect the number of times the
this test? choices of a test taker agreed with those of the judges.
d) Comparative scaling — Entails judgments of a stimulus in
Item Development in Norm Referenced and Criterion- comparison with every other stimulus on the scale.
Referenced Test EXAMPLE: A version of the MDBS-R that employs comparative
scaling might feature 30 items, each printed on a separate
a) Norm-reference test — the performance of each examinee
index card. Testtakers would be asked to sort the cards from
is interpreted in reference to relevant standardization sample
most justifiable to least justifiable)
(Pertersen, Kolen & Hoover, 1989).
e) Categorical scaling — Stimuli are placed into one of two or
Generally, a good item on a norm-referenced achievement
more alternative categories.
test is an item for which high scorers on the test respond
EXAMPLE: In our running MDBS-R example, testtakers might
correctly; Low scorers respond incorrectly.
be given 30 index cards, on each of which is printed one of
b) Criterion-reference test — the objective is to determine
the 30 items. Testtakers would be asked to sort the cards into
where the examinee stands with respect to very tightly
three piles: those behaviors that are never justified, those
defined educational objectives (Berk, 1984). What matters in
that are sometimes justified, and those that are always
criterion-reference test is whether the examinees meet an
justified
appropriate, specified criterion – for example 95% accuracy.
f) Guttman scale — Items range sequentially from weaker to
Ideally, each item on a criterion-oriented test addresses the
stronger expressions of the attitude, belief, or feeling being
issue of whether the respondent has met certain criteria.
measured.
Commonly employed in licensing contexts, be it a license to
All respondents
practice medicine or to drive a car.
who agree with
Development of a criterion-referenced test may entail
the stronger
exploratory work with at least two groups of test takers: one
statements of the
group known to have mastered the knowledge or skill being
attitude will also
measured and another group known not to have mastered it.
agree with milder
statements.
Test Conceptualization - Pilot Work
The method of equal-appearing intervals can be used to
Pilot work, pilot study, pilot research — Test items may be obtain data that are interval in nature
pilot studied (or piloted) to evaluate whether they should be scalogram analysis — an item-analysis procedure and
included in the final form of the instrument. approach to test development that involves a graphic
The preliminary research surrounding the creation of a mapping of a test taker’s responses (another term used for
prototype of the test. Guttman Scale)
A necessity when constructing tests or other measuring g) Method of Equal-Appearing Intervals — used to obtain
instruments for publication and wide distribution. data that are interval in nature.
The test developer typically attempts to determine how best This is an example of a scaling method of the direct
to measure a targeted construct. The process may entail estimation variety. In contrast to other methods that involve
literature reviews and experimentation as well as creation, indirect estimation (there is no need to transform the test
revision, and deletion of preliminary test items. taker’s responses into some other scale.)
an umbrella term for all that goes into the process of creating scaling methods.
a test. Types of scales — Scales are instruments to measure some trait,
Process of Test Development state or ability. May be categorized in many ways (e.g.
1. test conceptualization – multidimensional, unidemensional, etc.).
Conceptualizing a test Age-based scale — If the test taker’s test performance as a
2. test construction – tests are function of age is critical interest. (e.g. RPM)
drafted, Grade-based scale — If the test taker’s test performance as
stage of process of test development a function of grade is critical interest.
that entails writing test items (or re- Stanine Scale — if all raw scores on test are to be
writing or revising existing items) as transformed into scores that can range from 1 to 9
well as formatting items, scoring Describing Scales: unidimensional/multidimensional,
rules, and otherwise designing and comparative/ categorical
building a test.
3. test tryout – test is tried on a group,
administered to a representative
sample of test takers under Test Construction - Scaling Methods
conditions that stimulate the Numbers can be assigned to responses to calculate test scores
conditions that the final version of the test will be using a number of methods.
administered. a) Rating Scales — a
4. item analysis – using statistical analysis to assists in making grouping of words,
judgment about which items are good, needs to be revised or statements, or symbols on
discarded. Analysis involves item reliability, item validity, item which judgments of the
discrimination, or item difficulty. strength of a particular
5. test revision - At some point, the test developer will either trait, attitude, or emotion
finalize the form of the test or go back to the proverbial are indicated by the test
drawing board. taker. (e.g MDBS-R)
Action taken to modify a test’s content or format for the can be used to record
purpose of improving the tests’ effectiveness as a tool of judgments of oneself,
measurement. others, experiences, or objects, and they can take several
forms
Test Conceptualization Summative Scale — term for summing the ratings across all
The beginnings of any published test can probably be traced the items resulting to the final test score which is an Ordinal
to thoughts — self-talk, in behavioral terms Level Data.
The impetus for developing a new test is some thought that differs in the number of dimensions underlying the ratings
“there ought to be a test designed to measure [fill in the being made: unidimensional — only one dimension is
blank] in [such and such] way…” presumed to underlie the ratings/ multidimensional — more
The stimulus could be almost anything; knowledge of than one dimension is thought to guide the test taker’s
psychometric problems with other tests, a new social responses.
phenomenon, or any number of things. b) Likert scale — is used extensively in psychology, usually to
There may be a need to assess mastery in an emerging scale attitudes.
occupation. relatively easy to
Some preliminary questions construct
What is the test What special training will Each item presents
designed to measure? be required of test users the test taker with
What is the objective of for administering or five alternative responses (sometimes seven), usually on an
the test? interpreting the agree–disagree or approve–disapprove continuum
Is there a need for this test? usually reliable, which may account for their widespread
test? What types of responses popularity
Who will use this test? will be required of test c) Method of Paired Comparisons – Test-takers must choose
Who will take this test? takers? between two
What content will the Who benefits from an alternatives/pair of
test cover? administration of this stimuli (images,
How will the test be test? objects, statements)
administered? Is there any potential for according to some
What is the ideal format harm as the result of an rule.
of the test? administration of this For each pair of options, test takers receive a higher
Should more than one test? score for selecting the option deemed more justifiable by
form of the test be How will meaning be the majority of a group of judges.
developed? attributed to scores on The test score would reflect the number of times the
this test? choices of a test taker agreed with those of the judges.
d) Comparative scaling — Entails judgments of a stimulus in
Item Development in Norm Referenced and Criterion- comparison with every other stimulus on the scale.
Referenced Test EXAMPLE: A version of the MDBS-R that employs comparative
scaling might feature 30 items, each printed on a separate
a) Norm-reference test — the performance of each examinee
index card. Testtakers would be asked to sort the cards from
is interpreted in reference to relevant standardization sample
most justifiable to least justifiable)
(Pertersen, Kolen & Hoover, 1989).
e) Categorical scaling — Stimuli are placed into one of two or
Generally, a good item on a norm-referenced achievement
more alternative categories.
test is an item for which high scorers on the test respond
EXAMPLE: In our running MDBS-R example, testtakers might
correctly; Low scorers respond incorrectly.
be given 30 index cards, on each of which is printed one of
b) Criterion-reference test — the objective is to determine
the 30 items. Testtakers would be asked to sort the cards into
where the examinee stands with respect to very tightly
three piles: those behaviors that are never justified, those
defined educational objectives (Berk, 1984). What matters in
that are sometimes justified, and those that are always
criterion-reference test is whether the examinees meet an
justified
appropriate, specified criterion – for example 95% accuracy.
f) Guttman scale — Items range sequentially from weaker to
Ideally, each item on a criterion-oriented test addresses the
stronger expressions of the attitude, belief, or feeling being
issue of whether the respondent has met certain criteria.
measured.
Commonly employed in licensing contexts, be it a license to
All respondents
practice medicine or to drive a car.
who agree with
Development of a criterion-referenced test may entail
the stronger
exploratory work with at least two groups of test takers: one
statements of the
group known to have mastered the knowledge or skill being
attitude will also
measured and another group known not to have mastered it.
agree with milder
statements.
Test Conceptualization - Pilot Work
The method of equal-appearing intervals can be used to
Pilot work, pilot study, pilot research — Test items may be obtain data that are interval in nature
pilot studied (or piloted) to evaluate whether they should be scalogram analysis — an item-analysis procedure and
included in the final form of the instrument. approach to test development that involves a graphic
The preliminary research surrounding the creation of a mapping of a test taker’s responses (another term used for
prototype of the test. Guttman Scale)
A necessity when constructing tests or other measuring g) Method of Equal-Appearing Intervals — used to obtain
instruments for publication and wide distribution. data that are interval in nature.
The test developer typically attempts to determine how best This is an example of a scaling method of the direct
to measure a targeted construct. The process may entail estimation variety. In contrast to other methods that involve
literature reviews and experimentation as well as creation, indirect estimation (there is no need to transform the test
revision, and deletion of preliminary test items. taker’s responses into some other scale.)
