Research skill – Design & Analysis of Questionnaires (blue = literature)
Lecture 1.1 – Course introduction
Examples
- Likert-scales of 7 options: label highest and lowest option
- How to make it not leading: … to which you agree or disagree
- Sometimes all options labeled
- Perceived severity scale
- Should not back people into a corner
- Yes or no option but maybe add “I don’t know”
Weekly schedule
- Every Friday (Monday latest), video lectures will come online for the week after
- Every week:
- Watch lectures
- Read literature
- If scheduled, do exercises
- Every Thursday, live Q&A session on campus
- Every Tuesday before the live Q&A 12:00: post questions on discussion board
- No Q&A session when there are no questions
Assessment
- A1 (with peer feedback) and A2 (for a grade) = 50%
- Exam = 50%
Assignment A (50%)
A1
- 17 sept 17:00
- Choose topic on Canvas (no limit on amount of students)
- Develop survey offline: on Word
- Static and not interactive yet
- Explain why survey looks the way it does (order/logic)
- Provide feedback on survey of two peers
- 24 sept 17:00
A2
- 8 oct 17:00
- Improve survey based on peer feedback & general feedback (Q&A on 30 sept)
- Create survey in Qualtrics
- Share A2 via Qualtrics and Canvas (download it into a static format (pdf/word))
- Pretest your survey twice (cognitive interview & informal debriefing)
- Improve survey again based on this
- Explain why final survey looks the way it does
- Using course literature (& other sources)
1
,Exam (50%)
1. Knowledge part
- Questions about Qualtrics
- Knowledge questions about survey design based on literature
- Why should/should not be designed a certain way
2. SPSS part
- Analysis of data file (factor analysis)
- Online with proctorio or on campus? Don’t know yet
- Both require sufficient grade (allebei voldoende: 5.5)
- Both have a resit
Lecture 1.2 – Introduction to questionnaires
How to design a survey?
1. What do you want to measure?
- What constructs are you interest in?
- Quality of teaching/convey information?
- Check exact research objectives with supervisor
- Specification of “rough” assignment
Variables
- Manifest variables: can be directly observed
- Height, hair colour, age
- Measure with one item
- Latent variables: can only be observed indirectly
- Wealth, intelligence
- Attitudes as political efficacy, being introvert, comprehensibility, perceived physical
attractiveness
- Need multiple items
- Multi-faceted: multiple views or dimensions
- You can see if you actually measure the underlying thing
- You can detect/decrease influence of unsystematic errors (people providing
wrong answer)
- Self-report measures of latent construct (depression)
- Indicators: if you score high on Y, it should be indicated in A, B, C, …
- Validity issue: does not measure what we want to measure
- Can still be reliable but not valid
2. From theory to questions and answers?
- Look at prior research
- Don’t need to make items yourself necessarily
- Option 1: look at existing scales
- e.g. need for cognition, privacy concerns
2
, - Advantage: they are validated
- Disadvantages:
- Most developed in English language
- Translating problems, literally translating is not always good
translating EN-NL & NL-EN and check if it’s good now
- Not all scientists are survey methodologists
- Maybe loaded questions for example
Items and translations
- I find satisfaction in deliberating hard and for long hours double barreled (hard but not
long)
- Iets langduring en nauwgezet afwegen geeft me voldoening
- The notion of thinking abstractly is appealing to me complexly formulated
- Abstract denken is een bezigheid die mij aanspreekt
- I usually end up deliberating about issues even when they do not affect me personally
complexly formulated
- Gewoonlijk denk ik uitgebreid na over zaken, zelfs wanneer ze mij niet persoonlijk
aangaan
When there is no scale available
- Developing your own times
1. Internal method (inductive)
- Many items are used and more statistical approach to group items which are more
relevant to your construct
- Through statistical test, 3/5 dimensions were not strongly representing your construct
- Missing out on subgroups that do score high to those dimensions
Can lead to wrong conclusion
2. Facet method (deductive)
- Instrument should fully represent each dimension of the construct that is intended to
be measured
- Preferred
- Look at which indicators/dimensions form your construct
- Extraversion facets: enthusiasm, sociability, energy mode, taking charge, trust of
others, tact
3
Lecture 1.1 – Course introduction
Examples
- Likert-scales of 7 options: label highest and lowest option
- How to make it not leading: … to which you agree or disagree
- Sometimes all options labeled
- Perceived severity scale
- Should not back people into a corner
- Yes or no option but maybe add “I don’t know”
Weekly schedule
- Every Friday (Monday latest), video lectures will come online for the week after
- Every week:
- Watch lectures
- Read literature
- If scheduled, do exercises
- Every Thursday, live Q&A session on campus
- Every Tuesday before the live Q&A 12:00: post questions on discussion board
- No Q&A session when there are no questions
Assessment
- A1 (with peer feedback) and A2 (for a grade) = 50%
- Exam = 50%
Assignment A (50%)
A1
- 17 sept 17:00
- Choose topic on Canvas (no limit on amount of students)
- Develop survey offline: on Word
- Static and not interactive yet
- Explain why survey looks the way it does (order/logic)
- Provide feedback on survey of two peers
- 24 sept 17:00
A2
- 8 oct 17:00
- Improve survey based on peer feedback & general feedback (Q&A on 30 sept)
- Create survey in Qualtrics
- Share A2 via Qualtrics and Canvas (download it into a static format (pdf/word))
- Pretest your survey twice (cognitive interview & informal debriefing)
- Improve survey again based on this
- Explain why final survey looks the way it does
- Using course literature (& other sources)
1
,Exam (50%)
1. Knowledge part
- Questions about Qualtrics
- Knowledge questions about survey design based on literature
- Why should/should not be designed a certain way
2. SPSS part
- Analysis of data file (factor analysis)
- Online with proctorio or on campus? Don’t know yet
- Both require sufficient grade (allebei voldoende: 5.5)
- Both have a resit
Lecture 1.2 – Introduction to questionnaires
How to design a survey?
1. What do you want to measure?
- What constructs are you interest in?
- Quality of teaching/convey information?
- Check exact research objectives with supervisor
- Specification of “rough” assignment
Variables
- Manifest variables: can be directly observed
- Height, hair colour, age
- Measure with one item
- Latent variables: can only be observed indirectly
- Wealth, intelligence
- Attitudes as political efficacy, being introvert, comprehensibility, perceived physical
attractiveness
- Need multiple items
- Multi-faceted: multiple views or dimensions
- You can see if you actually measure the underlying thing
- You can detect/decrease influence of unsystematic errors (people providing
wrong answer)
- Self-report measures of latent construct (depression)
- Indicators: if you score high on Y, it should be indicated in A, B, C, …
- Validity issue: does not measure what we want to measure
- Can still be reliable but not valid
2. From theory to questions and answers?
- Look at prior research
- Don’t need to make items yourself necessarily
- Option 1: look at existing scales
- e.g. need for cognition, privacy concerns
2
, - Advantage: they are validated
- Disadvantages:
- Most developed in English language
- Translating problems, literally translating is not always good
translating EN-NL & NL-EN and check if it’s good now
- Not all scientists are survey methodologists
- Maybe loaded questions for example
Items and translations
- I find satisfaction in deliberating hard and for long hours double barreled (hard but not
long)
- Iets langduring en nauwgezet afwegen geeft me voldoening
- The notion of thinking abstractly is appealing to me complexly formulated
- Abstract denken is een bezigheid die mij aanspreekt
- I usually end up deliberating about issues even when they do not affect me personally
complexly formulated
- Gewoonlijk denk ik uitgebreid na over zaken, zelfs wanneer ze mij niet persoonlijk
aangaan
When there is no scale available
- Developing your own times
1. Internal method (inductive)
- Many items are used and more statistical approach to group items which are more
relevant to your construct
- Through statistical test, 3/5 dimensions were not strongly representing your construct
- Missing out on subgroups that do score high to those dimensions
Can lead to wrong conclusion
2. Facet method (deductive)
- Instrument should fully represent each dimension of the construct that is intended to
be measured
- Preferred
- Look at which indicators/dimensions form your construct
- Extraversion facets: enthusiasm, sociability, energy mode, taking charge, trust of
others, tact
3
