Module 2.1 Science and Research
Science à Approach that involves the understanding, prediction, and control of some
phenomenon of interest.
Hypothesis à Prediction about the relationship(s) among variables of interest.
Scientific methods:
1. Marked by a logical approach to investigation
- A theory
- A hypothesis
- A basic curiosity
2. Science depends on data
- Intended to be relevant to the theory, hypothesis, or curiosity that
precipitated the investigation
3. Must be communicable, open, and public
- Journals, reports, and books
- Anyone can join the debate
4. Science doesn’t set out to prove theories or hypotheses but to disprove them.
5. Disinterestedness – Characteristic of scientists, who should be objective and
uninfluenced by biases or prejudices when conducting research.
- Researchers are expected to be dispassionate about the results they
expect that research to yield.
The role of science in society à Trustworthy science should be spread by trustworthy
scientists.
Expert witness à Witness in a lawsuit who can voice opinions about organizational
practices.
Good theories display these characteristics:
- oTer novel insights - are interesting
- are focused - are relevant to important topics
- provide explanations - are practical
Research design à Provides the overall structure of architecture for the research
study; allows investigators to conduct scientific research on a phenomenon of interest.
Spector (2001)
- Experimental
- Quasi-experimental
- Non-experimental (the most important one in IO Psychology)
,Experimental design à Participants are randomly assigned to diTerent conditions.
- “Golden standard”
- Random assignments
Quasi-experimental design à Participants are assigned to diTerent conditions but
random assignment to conditions is not possible
Non-experimental design à Does not include a “treatment” or assignment to diTerent
conditions.
- Survey (field research)
Observational design à Researcher observes employee behavior and systematically
records what is observed.
Survey design àResearch strategy in which participants are asked to complete a
questionnaire or survey
Data Collection
- Quantitative: numerical
- Qualitative: descriptive
- Introspection: The participant was also the experimenter, recording his
experiences in completing an experimental task. This is considered very
subjective compared to modern standards.
- Triangulation: apply various methods toward the same hypothesis
Generality – The degree to which the results of one study can be applied to various
groups, tasks, times, and organizations.
Job Analysis à Process to determine the important tasks of a job and the human
attributes necessary to successfully perform those tasks.
Statistical control - Using statistical techniques to control for the influence
of certain variables
Experimental control - Possible confounding influences that might make
results less reliable or harder to interpret are eliminated
Module 2.2 Data Analysis
Descriptive statistics à Statistics that summarize, organize, and describe a sample of
data
,Three measures or characteristics can be used to describe any score distribution
1. The measure of central tendency (the center of distribution)
2. Variability (the extent to which scores in a distribution vary)
3. Skew (the extent to which scores in a distribution are lopsided or tend to fall; if
most data are on the left of the mean: positive skewness)
Inferential statistics à Statistics used to aid the researcher in testing hypotheses and
making inferences from sample data to a larger sample or population.
Statistical Significance (denoted by 𝛼)
“𝛼” is the probability of the study rejecting the null hypothesis (indicates that there is no
diTerence between), given that the hypothesis is true.
p-value: the probability of obtaining a result at least as extreme, given that the
hypothesis is true.
𝑝 ≤ 𝛼 (𝑐𝑜𝑚𝑚𝑜𝑛𝑙𝑦 𝑝 < 0.05)
Statistical power à The likelihood of finding a statistically significant diTerence when
a true diTerence exists.
Correlation coeKicient à Most used statistic or measure of association; varies
between -1&+1; assess the relationship between 2 variables, giving the magnitude and
direction.
Scatterplot à Graph used to plot the scatter of scores on two variables.
Regression line à Straight line that best “fits” the scatterplot and describes the
correlation.
Multiple correlation coeKicient à Between several variables on the one hand and a
single variable on the other hand.
e.g. Cognitive ability
Personality Vs. Job performance
Experience
Big data à Using large data sets to examine relationships among variables and to make
organizational decisions based on such data.
Meta-analysis à Combining and analyzing the results from many studies to draw a
general conclusion about relationships among variables.
Statistical artifact à An inference that results from bias in the collection or
manipulation of data.
, Micro, Macro, Meso-research
The study of individual The study of collective The study of the interaction of
behavior behavior individual and collective behavior
Module 2.3 Interpretation through Reliability and Validity
Reliability = consistency/stability of a measure.
Validity = the accuracy of inferences based on test or performance data/whether it
represents the intention.
Test-retest reliability à Reliability check by finding the correlation between the two
times of measurements.
Equivalent forms’ reliability à Comparing the results from the sample of individuals
on two diTerent forms of the same test.
Internal consistency à Compare several items from the same construct.
A test is reasonably reliable if the reliability is between .70 to .80 or higher (until 1.00).
Traditional designs for demonstrating validity content-, criterion-, construct-related.
- Intended to answer whether better performance on the test or predictor is
associated with better performance on the job.
Content-related validity à The contents of the selection procedure = a sample of
important work behavior and activities and/or workers KSAOs (knowledge, skills,
abilities, or other characteristics) defined by the job analysis.
Criterion-related validity à Correlating a test score with a performance measure -
validity coeTicient is created.
Predictive validity design - there's a time lag between the collection of the test
data and the criterion data.
Concurrent validity design - no time lag.
Construct validity à Investigators gathered evidence to support decisions or
inferences about psychological constructs.
Science à Approach that involves the understanding, prediction, and control of some
phenomenon of interest.
Hypothesis à Prediction about the relationship(s) among variables of interest.
Scientific methods:
1. Marked by a logical approach to investigation
- A theory
- A hypothesis
- A basic curiosity
2. Science depends on data
- Intended to be relevant to the theory, hypothesis, or curiosity that
precipitated the investigation
3. Must be communicable, open, and public
- Journals, reports, and books
- Anyone can join the debate
4. Science doesn’t set out to prove theories or hypotheses but to disprove them.
5. Disinterestedness – Characteristic of scientists, who should be objective and
uninfluenced by biases or prejudices when conducting research.
- Researchers are expected to be dispassionate about the results they
expect that research to yield.
The role of science in society à Trustworthy science should be spread by trustworthy
scientists.
Expert witness à Witness in a lawsuit who can voice opinions about organizational
practices.
Good theories display these characteristics:
- oTer novel insights - are interesting
- are focused - are relevant to important topics
- provide explanations - are practical
Research design à Provides the overall structure of architecture for the research
study; allows investigators to conduct scientific research on a phenomenon of interest.
Spector (2001)
- Experimental
- Quasi-experimental
- Non-experimental (the most important one in IO Psychology)
,Experimental design à Participants are randomly assigned to diTerent conditions.
- “Golden standard”
- Random assignments
Quasi-experimental design à Participants are assigned to diTerent conditions but
random assignment to conditions is not possible
Non-experimental design à Does not include a “treatment” or assignment to diTerent
conditions.
- Survey (field research)
Observational design à Researcher observes employee behavior and systematically
records what is observed.
Survey design àResearch strategy in which participants are asked to complete a
questionnaire or survey
Data Collection
- Quantitative: numerical
- Qualitative: descriptive
- Introspection: The participant was also the experimenter, recording his
experiences in completing an experimental task. This is considered very
subjective compared to modern standards.
- Triangulation: apply various methods toward the same hypothesis
Generality – The degree to which the results of one study can be applied to various
groups, tasks, times, and organizations.
Job Analysis à Process to determine the important tasks of a job and the human
attributes necessary to successfully perform those tasks.
Statistical control - Using statistical techniques to control for the influence
of certain variables
Experimental control - Possible confounding influences that might make
results less reliable or harder to interpret are eliminated
Module 2.2 Data Analysis
Descriptive statistics à Statistics that summarize, organize, and describe a sample of
data
,Three measures or characteristics can be used to describe any score distribution
1. The measure of central tendency (the center of distribution)
2. Variability (the extent to which scores in a distribution vary)
3. Skew (the extent to which scores in a distribution are lopsided or tend to fall; if
most data are on the left of the mean: positive skewness)
Inferential statistics à Statistics used to aid the researcher in testing hypotheses and
making inferences from sample data to a larger sample or population.
Statistical Significance (denoted by 𝛼)
“𝛼” is the probability of the study rejecting the null hypothesis (indicates that there is no
diTerence between), given that the hypothesis is true.
p-value: the probability of obtaining a result at least as extreme, given that the
hypothesis is true.
𝑝 ≤ 𝛼 (𝑐𝑜𝑚𝑚𝑜𝑛𝑙𝑦 𝑝 < 0.05)
Statistical power à The likelihood of finding a statistically significant diTerence when
a true diTerence exists.
Correlation coeKicient à Most used statistic or measure of association; varies
between -1&+1; assess the relationship between 2 variables, giving the magnitude and
direction.
Scatterplot à Graph used to plot the scatter of scores on two variables.
Regression line à Straight line that best “fits” the scatterplot and describes the
correlation.
Multiple correlation coeKicient à Between several variables on the one hand and a
single variable on the other hand.
e.g. Cognitive ability
Personality Vs. Job performance
Experience
Big data à Using large data sets to examine relationships among variables and to make
organizational decisions based on such data.
Meta-analysis à Combining and analyzing the results from many studies to draw a
general conclusion about relationships among variables.
Statistical artifact à An inference that results from bias in the collection or
manipulation of data.
, Micro, Macro, Meso-research
The study of individual The study of collective The study of the interaction of
behavior behavior individual and collective behavior
Module 2.3 Interpretation through Reliability and Validity
Reliability = consistency/stability of a measure.
Validity = the accuracy of inferences based on test or performance data/whether it
represents the intention.
Test-retest reliability à Reliability check by finding the correlation between the two
times of measurements.
Equivalent forms’ reliability à Comparing the results from the sample of individuals
on two diTerent forms of the same test.
Internal consistency à Compare several items from the same construct.
A test is reasonably reliable if the reliability is between .70 to .80 or higher (until 1.00).
Traditional designs for demonstrating validity content-, criterion-, construct-related.
- Intended to answer whether better performance on the test or predictor is
associated with better performance on the job.
Content-related validity à The contents of the selection procedure = a sample of
important work behavior and activities and/or workers KSAOs (knowledge, skills,
abilities, or other characteristics) defined by the job analysis.
Criterion-related validity à Correlating a test score with a performance measure -
validity coeTicient is created.
Predictive validity design - there's a time lag between the collection of the test
data and the criterion data.
Concurrent validity design - no time lag.
Construct validity à Investigators gathered evidence to support decisions or
inferences about psychological constructs.
