Design & Analysis Q - Summary Exam
Week 1 - Introduction to surveys
Krosnick & Presser (2010, pp. 263-264)
To minimise response errors, questionnaires should be crafted in accordance with best practices
● Experience and common lore
● Methodological research
Open versus closed questions (relevant to three types of measurements)
1) Asking for choices among nominal categories
E.g. What is the most important problem facing the country?
2) Ascertaining numeric quantities
E.g. How many hours did you watch television?
3) Testen factual knowledge
E.g. Who is Joseph Biden?
Social desirability response bias
= Een sociaal wenselijk antwoord geven
Recall bias
= Wanneer mensen zich dingen anders herinneren dan daadwerkelijk is gebeurd
Conventional wisdom
Advice
1) Use simple, familiar words
2) Use simple syntax
3) Avoid words with ambiguous meanings
4) Strive for wording that is specific and concrete
5) Make response options exhaustive and mutually exclusive
6) Avoid leading or loaded questions that push respondents towards an answer
7) Ask about one thing at the time (avoid double-barreled questions)
8) Avoid questions with single or double negations
How to optimise question order
1) Early questions should be easy and pleasant to answer
2) Questions at the beginning should address the topic of the survey
3) Questions on the same topic should be grouped together and should proceed from
general to specific
4) Sensitive questions should be placed at the end
5) Include filter questions (om te voorkomen dat respondenten vragen krijgen die niet van
toepassing zijn)
,Blog on biased survey questions
A biased survey is one that encompasses errors caused by the design of the survey and its
questions. This can lead to response bias and a higher drop-out rate.
Ten examples of survey bias
1) The leading question
= A question that leads respondents to give the ‘correct’ answer
2) The loaded question
= You force people into answering the question in a particular way
3) The double-barreled question
= Two questions at once
4) The absolute question
= Yes or no questions, or with the answer options ‘always’, ‘never’, etc.
5) The unclear question
= Make your question as easy as possible to answer
6) The multiple answer question
= Multiple questions
7) Prefer not to answer
= ‘Prefer not to answer’ answer option
8) Include all possible answers
= If you are unsure of all the options, you can add ‘other’
9) Use accurate scales
= Add answer options from bad to excellent
10) Survey structure
= Study and test out the survey to root out poor structure → ask more personal questions
at the end to avoid drop-outs
Video on latent variables
Latent variables versus manifest variables
Latent variable
= IV, DV, the construct
● You cannot see/taste/smell it
● It is almost hypothetical
● E.g. satisfaction, motivation, etc.
Manifest variable
= Items on a survey (scale)
● A measurement to make the latent variable visible
● It is real and measurable
● E.g. the scales to measure satisfaction, motivation, etc.
Connection between latent variables and manifest variables
The items of the manifest variable (the scale) are correlated because they all measure the same
construct (the latent variables)??
,Lecture 1 - Introduction to surveys
How to design a survey?
1) What do you want to measure?
2) From theory to questions and answers
3) Phrase specific items
4) Develop the survey
5) Pretest the survey
6) Run
Step 1: What do you want to measure?
→ Check the exact research objectives
(onderzoeksdoelen) with your supervisors (Toepoel)
→ Then, specification of your ‘rough’ assignment
Manifest variables: can be directly observed
● E.g., height, hair color, etc.
Latent variables: can only be observed indirectly
● E.g., wealth, intelligence
● E.g., attitudes as political efficacy, being introvert, comprehensibility, perceived physical
attractiveness, etc.
Latent variables
Multiple questions, because:
● The concepts are multi-faceted (veelzijdig)
● When you ask multiple questions about the same construct you will at the very least be
able to establish that you have measured one underlying thing
● You can detect/decrease the influence of unsystematic errors (people providing the wrong
answer)
Self-report measures of a latent construct (e..g, depression)
● Measurements that represent a set of indicators of the latent construct
● If you score high on Y, then this should reflect in A, B, C, …
Example of a scale
Is the following scale an okay scale to measure how much people like to eat vegetables?
→ Please indicate to what extent you agree with the following items: (1 = completely disagree, 7 =
completely agree)
● I love eating bananas
, ● I really like strawberries
● I find apples delicious things to eat
● I enjoy eating a good orange
→ This scale does not measure what we want to measure, so is not valid
Step 2: From theory to questions and answers
1) Option 1: existing scales
E.g., need for cognition, privacy concerns
→ Advantage: validated
→ Disadvantage: language/translations + not all scientists are survey methodologists
2) Option 2: non-existing scales
When there are scientific scales you can use, do it! When there is no scale available, do
the following things:
● Internal method (inductive)
Many items are used and through statistical grouping techniquest it is decided
after the fact which ones were relevant
→ Achteraf besluiten welke items relevant waren
● Facet method (deductive)
Instrument should fully represent each dimension of the construct that is
intended to be measured
→ Elke dimensie van construct volledig vertegenwoordigen, dus alles meten
Facets of Teacher Quality
What could be facets of teacher quality?
● Capacity to explain
○ Gives many examples
○ Uses a good pace when talking
○ Etc.
● Approachability (benaderbaarheid)
○ Is open to questions
○ Responds to email quickly
○ Etc.
● Etc.
Uni- versus multidimensional scales (unidimensional scales are recommended)
Unidimensional scale
Multidimensional scale
Week 1 - Introduction to surveys
Krosnick & Presser (2010, pp. 263-264)
To minimise response errors, questionnaires should be crafted in accordance with best practices
● Experience and common lore
● Methodological research
Open versus closed questions (relevant to three types of measurements)
1) Asking for choices among nominal categories
E.g. What is the most important problem facing the country?
2) Ascertaining numeric quantities
E.g. How many hours did you watch television?
3) Testen factual knowledge
E.g. Who is Joseph Biden?
Social desirability response bias
= Een sociaal wenselijk antwoord geven
Recall bias
= Wanneer mensen zich dingen anders herinneren dan daadwerkelijk is gebeurd
Conventional wisdom
Advice
1) Use simple, familiar words
2) Use simple syntax
3) Avoid words with ambiguous meanings
4) Strive for wording that is specific and concrete
5) Make response options exhaustive and mutually exclusive
6) Avoid leading or loaded questions that push respondents towards an answer
7) Ask about one thing at the time (avoid double-barreled questions)
8) Avoid questions with single or double negations
How to optimise question order
1) Early questions should be easy and pleasant to answer
2) Questions at the beginning should address the topic of the survey
3) Questions on the same topic should be grouped together and should proceed from
general to specific
4) Sensitive questions should be placed at the end
5) Include filter questions (om te voorkomen dat respondenten vragen krijgen die niet van
toepassing zijn)
,Blog on biased survey questions
A biased survey is one that encompasses errors caused by the design of the survey and its
questions. This can lead to response bias and a higher drop-out rate.
Ten examples of survey bias
1) The leading question
= A question that leads respondents to give the ‘correct’ answer
2) The loaded question
= You force people into answering the question in a particular way
3) The double-barreled question
= Two questions at once
4) The absolute question
= Yes or no questions, or with the answer options ‘always’, ‘never’, etc.
5) The unclear question
= Make your question as easy as possible to answer
6) The multiple answer question
= Multiple questions
7) Prefer not to answer
= ‘Prefer not to answer’ answer option
8) Include all possible answers
= If you are unsure of all the options, you can add ‘other’
9) Use accurate scales
= Add answer options from bad to excellent
10) Survey structure
= Study and test out the survey to root out poor structure → ask more personal questions
at the end to avoid drop-outs
Video on latent variables
Latent variables versus manifest variables
Latent variable
= IV, DV, the construct
● You cannot see/taste/smell it
● It is almost hypothetical
● E.g. satisfaction, motivation, etc.
Manifest variable
= Items on a survey (scale)
● A measurement to make the latent variable visible
● It is real and measurable
● E.g. the scales to measure satisfaction, motivation, etc.
Connection between latent variables and manifest variables
The items of the manifest variable (the scale) are correlated because they all measure the same
construct (the latent variables)??
,Lecture 1 - Introduction to surveys
How to design a survey?
1) What do you want to measure?
2) From theory to questions and answers
3) Phrase specific items
4) Develop the survey
5) Pretest the survey
6) Run
Step 1: What do you want to measure?
→ Check the exact research objectives
(onderzoeksdoelen) with your supervisors (Toepoel)
→ Then, specification of your ‘rough’ assignment
Manifest variables: can be directly observed
● E.g., height, hair color, etc.
Latent variables: can only be observed indirectly
● E.g., wealth, intelligence
● E.g., attitudes as political efficacy, being introvert, comprehensibility, perceived physical
attractiveness, etc.
Latent variables
Multiple questions, because:
● The concepts are multi-faceted (veelzijdig)
● When you ask multiple questions about the same construct you will at the very least be
able to establish that you have measured one underlying thing
● You can detect/decrease the influence of unsystematic errors (people providing the wrong
answer)
Self-report measures of a latent construct (e..g, depression)
● Measurements that represent a set of indicators of the latent construct
● If you score high on Y, then this should reflect in A, B, C, …
Example of a scale
Is the following scale an okay scale to measure how much people like to eat vegetables?
→ Please indicate to what extent you agree with the following items: (1 = completely disagree, 7 =
completely agree)
● I love eating bananas
, ● I really like strawberries
● I find apples delicious things to eat
● I enjoy eating a good orange
→ This scale does not measure what we want to measure, so is not valid
Step 2: From theory to questions and answers
1) Option 1: existing scales
E.g., need for cognition, privacy concerns
→ Advantage: validated
→ Disadvantage: language/translations + not all scientists are survey methodologists
2) Option 2: non-existing scales
When there are scientific scales you can use, do it! When there is no scale available, do
the following things:
● Internal method (inductive)
Many items are used and through statistical grouping techniquest it is decided
after the fact which ones were relevant
→ Achteraf besluiten welke items relevant waren
● Facet method (deductive)
Instrument should fully represent each dimension of the construct that is
intended to be measured
→ Elke dimensie van construct volledig vertegenwoordigen, dus alles meten
Facets of Teacher Quality
What could be facets of teacher quality?
● Capacity to explain
○ Gives many examples
○ Uses a good pace when talking
○ Etc.
● Approachability (benaderbaarheid)
○ Is open to questions
○ Responds to email quickly
○ Etc.
● Etc.
Uni- versus multidimensional scales (unidimensional scales are recommended)
Unidimensional scale
Multidimensional scale
