Design and Analysis
Summary Loes van de Pas
How to design a survey:
1. What do you want to measure?
2. Constructs: from theory to questions.
3. Phrase specific items.
4. Develop the survey.
5. Pre-test the survey.
6. Run the survey.
7. Factor Analysis: test the results.
, 1. What to measure
Manifest variables
= Variables that can be directly observed (e.g. hair colour).
Latent variables
= Variables that can be indirectly observed (e.g. intelligence).
Multiple questions are used to measure this.
Validity
= The scale should measure what we want to measure.
Divergent validity
= Whether different constructs really measure different things.
Reliability
= A respondent should give the same answer to a question at any time.
2. Constructs: from theory to questions
Constructs
= Every reality that can be thought of as an abstract concept.
There are two ways of generating questions:
- Use existing scales
Advantage: the scales are validated.
Disadvantage: the scale might not fit the data or might be outdated.
- No scale available
Internal method (inductive)
= Many items are used and through statistical grouping techniques it is decided which ones
were relevant.
Facet method (deductive)
= Instrument fully represent each dimension of the construct measured.
Uni-dimensional scales
(e.g. height: you are either short or tall linear).
Multi-dimensional scales
(e.g. intelligence: you can be good at quantitatives and bad at verbal)
3. Phrase specific items
Conventional wisdom:
- Simple/ familiar words
- Simple syntax
- Avoid ambiguous meanings
- Specific/ concrete wordings
- Make response options mutually exclusive
- Avoid leading/ loaded questions
- Ask about one thing at a time
- Avoid single or double negotiation
Mutually exclusive and exhaustive
= Provide all respondents with one options that answers the question.
2
Summary Loes van de Pas
How to design a survey:
1. What do you want to measure?
2. Constructs: from theory to questions.
3. Phrase specific items.
4. Develop the survey.
5. Pre-test the survey.
6. Run the survey.
7. Factor Analysis: test the results.
, 1. What to measure
Manifest variables
= Variables that can be directly observed (e.g. hair colour).
Latent variables
= Variables that can be indirectly observed (e.g. intelligence).
Multiple questions are used to measure this.
Validity
= The scale should measure what we want to measure.
Divergent validity
= Whether different constructs really measure different things.
Reliability
= A respondent should give the same answer to a question at any time.
2. Constructs: from theory to questions
Constructs
= Every reality that can be thought of as an abstract concept.
There are two ways of generating questions:
- Use existing scales
Advantage: the scales are validated.
Disadvantage: the scale might not fit the data or might be outdated.
- No scale available
Internal method (inductive)
= Many items are used and through statistical grouping techniques it is decided which ones
were relevant.
Facet method (deductive)
= Instrument fully represent each dimension of the construct measured.
Uni-dimensional scales
(e.g. height: you are either short or tall linear).
Multi-dimensional scales
(e.g. intelligence: you can be good at quantitatives and bad at verbal)
3. Phrase specific items
Conventional wisdom:
- Simple/ familiar words
- Simple syntax
- Avoid ambiguous meanings
- Specific/ concrete wordings
- Make response options mutually exclusive
- Avoid leading/ loaded questions
- Ask about one thing at a time
- Avoid single or double negotiation
Mutually exclusive and exhaustive
= Provide all respondents with one options that answers the question.
2
