Page 1 of 37
, Page 2 of 37
BASIC MEASUREMENT AND QUESTIONNAIRE DESIGN
Keywords
Alternate-forms reliability: a measure of equivalence. Measures consistency of results of a
questionnaire over different forms, obtain an estimate of the alternate-forms reliability of each form
of the questionnaire. The closer the correlation coefficient is to 1, the greater the extent to which
the forms are equivalent and thus measure the same attribute
Anonymity: NOT the same as confidentiality. State of being anonymous; having no known name or
identity or known source
Attitude scales: group of items designed to reflect different attitudes toward the topic or question; a
technique for placing people on a continuum in relation to each other. Likert- semantic differential
scales
Bi-polar: opposites
Correlation coefficient: a measure of the interdependence of two variables that ranges in value from
-1 to +1
Coefficient of correlation: perfect negative -1, perfect positive +1; absence of correlation is 0
Concurrent validity: a type of criterion-related validity - obtaining measures on the criterion at
approximately the same time as the scores on the questionnaire. The extent to which scores
accurately estimates an individual’s present position on the relevant criterion is then determined
Construct: an unobservable quality that forms part of a theory designed to explain observable
behaviour eg anxiety, intelligence, stress, independence etc.
Construct validity: focuses on WHAT is being measured. It is concerned with the nature of reality &
the nature of properties being measured; estimation of the extent to which subjects posses the
characteristics presumed to be reflected by a particular scale or test.
Content validity: to what degree the content of items in the measuring scale corresponds with the
content of the domain being measured.
Convergent validity: a type of construct validity. Two questionnaires measure a construct that is
theoretically related and there’s a significant correlation between the scores on these questionnaires
Criterion: tests & questionnaires are often used to estimate an individual’s position or performance
on some outcome measure. This outcome measure is referred to as the criterion
Criterion validity: collective term used for predictive and concurrent validity
Discriminate validity: type of construct validity. Two questionnaires measure constructs that are
theoretically unrelated, there would NOT be a hight correlation between the scores on
questionnaires measuring these construct.
, Page 3 of 37
Domain = describes and classifies essentials and relevant elements; gives an overview of the
problem being investigated in the questionnaire; lists the principles in preparing the questionnaire
The content domain: all possible tasks, behaviours, attitudes etc implied by the purpose or
related to the questions you ask
Face validity: the degree to which items in a questionnaire appear to be relevant to what is being
measured. It is the subjective evaluation by people who are not necessarily experts – if respondents
do not regard items as relevant (insufficient face validity), they might be less motivated and unwilling
to cooperate
Funnel approach: start with broad questions then narrow it down to more specific toward the end
Item analysis: a set of procedures to select the best or the worst items in a measuring instrument
Item bias statistics: less commonly used form of item analysis; helps test constructors to identify
items that perform differently for different groups (are biased) eg. Race or gender
Item difficulty: or item facility or item variance is a commonly used form of item analysis; in an ideal
questionnaire at leas half the people gets each of the items correct
Item discrimination: a commonly used form of item analysis; the ability of an item to discriminate
between respondents according to whatever the measuring instrument is measuring
Item format: kinds of items used in the questionnaire or the rating scales, eg open or closed
questions
Halo effect: the tendency for respondent not to evaluate each item individually but for their
responses to be influenced by their general feeling of like or dislike
Likert scale: attitude scale - summated scale in which subjects indicate extent to which he or she
agrees or disagrees with statements.
Multi-item scale: group of items dealing with the same topic, each item requires a rated response
Predictive validity: a type of criterion-related validity - the measures on the criterion are obtained in
the future, usually months or even years after scores on the questionnaire is obtained. It is then
determined to which extent the scores on the questionnaire accurately predict an individual’s scores
on the relevant criterion; used to predict some future performance of individuals
Privacy: NOT the same as anonymity: quality of being secluded from the presence of others; the
condition of being concealed or hidden
Purpose of identifying content domain: gives indication of the content domain; provides a basis for
determining whether the questionnaire is properly within the scope
QWAN: Quality without a name. The quality everyone strives for but no one can claim. It’s a sort of
average of all the ratings
, Page 4 of 37
Rating scales: used to measure non-factual topics eg beliefs, opinions, attitudes & values.
Relative position: compared to other people
Reliability: when identical investigations are repeated, similar research results are obtained; degree
to which a scale yields consistent results or scores.
Reliability coefficient: the closer the correlation coefficient is to 1, the more reliable it is
Scope: range of perceptions, area covered by given activity
Semantic differential: attitude scale – measures attitudes. Seven point rating scale: each end are
defined by opposing adjectives or adverbs (items are bipolar)
Specification document for a questionnaire: a list of what the questionnaire should contain in terms
of type of items, number of items, layout … in order for the questionnaire to do what it’s supposed
to do.
Split-half reliability: measure of equivalence and internal consistency; the questionnaire is divided
into 2 parts in a manner that they may be regarded as 2 parallel halves of the questionnaire. Each
person has a total score on the one half and a total score on the 2nd half: 2 sets of scores are then
correlated
Test-retest reliability: to establish the stability or consistency of scores over time; administer the
same questionnaire to the same group of people on different occasions; correlate the 2 sets of
scores & correlation coefficient to establish the test-retest reliability. Ideally the correlation
coefficient should be close to 1
The target population: those who complete the questionnaire
Uni-dimensional: all the items measure the same dimension or topic (Likert scale)
Validity: methodological requirement for research methods as well as research results.
STEPS IN DESIGNING OF A QUESTIONNAIRE:
, Page 2 of 37
BASIC MEASUREMENT AND QUESTIONNAIRE DESIGN
Keywords
Alternate-forms reliability: a measure of equivalence. Measures consistency of results of a
questionnaire over different forms, obtain an estimate of the alternate-forms reliability of each form
of the questionnaire. The closer the correlation coefficient is to 1, the greater the extent to which
the forms are equivalent and thus measure the same attribute
Anonymity: NOT the same as confidentiality. State of being anonymous; having no known name or
identity or known source
Attitude scales: group of items designed to reflect different attitudes toward the topic or question; a
technique for placing people on a continuum in relation to each other. Likert- semantic differential
scales
Bi-polar: opposites
Correlation coefficient: a measure of the interdependence of two variables that ranges in value from
-1 to +1
Coefficient of correlation: perfect negative -1, perfect positive +1; absence of correlation is 0
Concurrent validity: a type of criterion-related validity - obtaining measures on the criterion at
approximately the same time as the scores on the questionnaire. The extent to which scores
accurately estimates an individual’s present position on the relevant criterion is then determined
Construct: an unobservable quality that forms part of a theory designed to explain observable
behaviour eg anxiety, intelligence, stress, independence etc.
Construct validity: focuses on WHAT is being measured. It is concerned with the nature of reality &
the nature of properties being measured; estimation of the extent to which subjects posses the
characteristics presumed to be reflected by a particular scale or test.
Content validity: to what degree the content of items in the measuring scale corresponds with the
content of the domain being measured.
Convergent validity: a type of construct validity. Two questionnaires measure a construct that is
theoretically related and there’s a significant correlation between the scores on these questionnaires
Criterion: tests & questionnaires are often used to estimate an individual’s position or performance
on some outcome measure. This outcome measure is referred to as the criterion
Criterion validity: collective term used for predictive and concurrent validity
Discriminate validity: type of construct validity. Two questionnaires measure constructs that are
theoretically unrelated, there would NOT be a hight correlation between the scores on
questionnaires measuring these construct.
, Page 3 of 37
Domain = describes and classifies essentials and relevant elements; gives an overview of the
problem being investigated in the questionnaire; lists the principles in preparing the questionnaire
The content domain: all possible tasks, behaviours, attitudes etc implied by the purpose or
related to the questions you ask
Face validity: the degree to which items in a questionnaire appear to be relevant to what is being
measured. It is the subjective evaluation by people who are not necessarily experts – if respondents
do not regard items as relevant (insufficient face validity), they might be less motivated and unwilling
to cooperate
Funnel approach: start with broad questions then narrow it down to more specific toward the end
Item analysis: a set of procedures to select the best or the worst items in a measuring instrument
Item bias statistics: less commonly used form of item analysis; helps test constructors to identify
items that perform differently for different groups (are biased) eg. Race or gender
Item difficulty: or item facility or item variance is a commonly used form of item analysis; in an ideal
questionnaire at leas half the people gets each of the items correct
Item discrimination: a commonly used form of item analysis; the ability of an item to discriminate
between respondents according to whatever the measuring instrument is measuring
Item format: kinds of items used in the questionnaire or the rating scales, eg open or closed
questions
Halo effect: the tendency for respondent not to evaluate each item individually but for their
responses to be influenced by their general feeling of like or dislike
Likert scale: attitude scale - summated scale in which subjects indicate extent to which he or she
agrees or disagrees with statements.
Multi-item scale: group of items dealing with the same topic, each item requires a rated response
Predictive validity: a type of criterion-related validity - the measures on the criterion are obtained in
the future, usually months or even years after scores on the questionnaire is obtained. It is then
determined to which extent the scores on the questionnaire accurately predict an individual’s scores
on the relevant criterion; used to predict some future performance of individuals
Privacy: NOT the same as anonymity: quality of being secluded from the presence of others; the
condition of being concealed or hidden
Purpose of identifying content domain: gives indication of the content domain; provides a basis for
determining whether the questionnaire is properly within the scope
QWAN: Quality without a name. The quality everyone strives for but no one can claim. It’s a sort of
average of all the ratings
, Page 4 of 37
Rating scales: used to measure non-factual topics eg beliefs, opinions, attitudes & values.
Relative position: compared to other people
Reliability: when identical investigations are repeated, similar research results are obtained; degree
to which a scale yields consistent results or scores.
Reliability coefficient: the closer the correlation coefficient is to 1, the more reliable it is
Scope: range of perceptions, area covered by given activity
Semantic differential: attitude scale – measures attitudes. Seven point rating scale: each end are
defined by opposing adjectives or adverbs (items are bipolar)
Specification document for a questionnaire: a list of what the questionnaire should contain in terms
of type of items, number of items, layout … in order for the questionnaire to do what it’s supposed
to do.
Split-half reliability: measure of equivalence and internal consistency; the questionnaire is divided
into 2 parts in a manner that they may be regarded as 2 parallel halves of the questionnaire. Each
person has a total score on the one half and a total score on the 2nd half: 2 sets of scores are then
correlated
Test-retest reliability: to establish the stability or consistency of scores over time; administer the
same questionnaire to the same group of people on different occasions; correlate the 2 sets of
scores & correlation coefficient to establish the test-retest reliability. Ideally the correlation
coefficient should be close to 1
The target population: those who complete the questionnaire
Uni-dimensional: all the items measure the same dimension or topic (Likert scale)
Validity: methodological requirement for research methods as well as research results.
STEPS IN DESIGNING OF A QUESTIONNAIRE:
