IOP3701_ SUPPLEMENTARY STUDY MATERIAL FOR IOP3701(Questions and answers)

IOP3701_ SUPPLEMENTARY STUDY MATERIAL FOR IOP3701(Questions and answers). Industrial Psychological Testing And Assessment. TECHNICAL AND METHODOLOGICAL PRINCIPLES STUDY UNIT 3 – BASIC CONCEPTS, RELIABILITY AND VALIDITY Learning outcomes: 1. explain why statistical knowledge is necessary for the study of testing and assessment 2. discuss why it is necessary to use norm scores in the interpretation of test data and not raw scores 3. explain how to determine whether a test is reliable 4. explain how to determine whether a test is valid Measurement – the assignment of numbers to characteristics, or the transformation of qualities of attributes into numbers. Properties of measurement scales: o Magnitude: A scale has the property of magnitude if we can say that one attribute is more than, less than, or equal to another attribute.  Height has the property of magnitude – we can say that one person is taller or shorter than another o Equal intervals: A scale possesses equal intervals if there is a uniform difference between all points on that scale.  Length for example – difference between 8 and 10 on a ruler is the same as the difference between 10 and 12. In both cases the difference is exactly 2 cm o Absolute zero: This is obtained when there is absolutely nothing of the attribute being measured.  E.g. length – 0cm means no distance  Wind velocity – reading of zero means no wind blowing  Therefore both these have the property of absolute zero  Types of measurement scales: o Nominal scales: This is often used for the purpose of classification – that is to categorise items.  Not scaled along a dimension and do not have any of the properties of measurement scales  For nominal variables the number attached to a specific category has no meaning  E.g. gender: 1 = male and 2 = female o Ordinal scales: These scales order people, objects or events along some continuum.  They have the property of magnitude only.  E.g. in individual sport events – the positions are 1st, 2nd, and so on.  IQ tests – is another example. o Interval scales: This scale has the property of magnitude and equal intervals.  This means that the size of the difference between values can be interpreted.  E.g. temperature – magnitude: 30C is warmer than 25C  Equal intervals: the difference between 4C and 10C, the same as the difference between 30C and 36C o Ratio scales: Measurement scales that have all three properties – magnitude, equal intervals, and absolute zero.  The ratios are meaningful  E.g. speed – the point where there is no speed at all is 0 km/hour.  120 km/h is twice the speed of 60 km/h and 55 km/h is half the speed of 110km/h Measurement scales and their properties: Page 48 of 179 Downloaded by: Portialepesa | Distribution of this document is illegal S - The Marketplace to Buy and Sell your Study Material  Reliability – consistency with which it measures whatever it measures. (Reliability can also be defined as the ratio of true score variance to observed score variance.  Reliability of a test is expressed by means of the reliability coefficient, which is a correlation coefficient. Generally, o Standardised tests should have reliabilities in the 0.80’s or 0.90s o Reliability coefficients should be 0.85 or higher for decisions about individuals, while only 0.65 or higher for decisions about groups o Standardised personality and interest questionnaires should have reliability coefficients of 0.80 to 0.85, while those of aptitude tests should be 0.90 or higher  Types of reliability: o Test-retest reliability  Administer the measure twice to the same group of test-takers  Reliability coefficient in this case is simply the correlation between the scores obtained on the first and second application of the measure.  This coefficient is also called coefficient of stability.  Drawback: the testing circumstances may be different for both the test-taker and the physical environment which may contribute to systematic error variance.  For most types of measures, this technique is not appropriate for computing reliability o Alternate-form reliability  2 equivalent forms of the same measure are administered to the same group on 2 different occasions.  The correlation obtained between the 2 sets of scores represents the reliability coefficient.  The 2 measures should be truly equivalent, i.e. they should have the same number of items; the scoring procedure should be exactly the same; they must be uniform in respect of content, representativeness, and item difficulty level; and the test mean and variances should be more or less the same. o Split-half reliability  Obtained by splitting the measure into 2 equivalent halves and computing the correlation coefficient between these 2 sets of scores. This coefficient is also called a coefficient of internal consistency.  The corrected reliability coefficient is calculated by means of the Spearman-Brown formula. o Intern-item reliability  Coefficient of internal consistency, based on the consistency of responses to all items in the measure.  Obtained using the Kuder-Richardson method. o Inter- and intra-scorer reliability  Inter-scorer reliability can be determined by having all the test-takers’ test protocols scored by 2 assessment practitioners. The correlation coefficient between these 2 sets of scores reflects the inter-scorer reliability coefficient.  Inter-scorer reliability thus refers to the consistency of ratings between raters.  Intra-scorer reliability coefficient refers to the consistency of ratings for a single rater.  Factors affecting reliability: o Systematic respondent errors:  Non-response errors / self-selection bias – these occur when respondents do not fully complete their tests or assessments.  Response bias – occurs when respondents decide to systematically respond in a set or fixed manner to the item or questing, thereby presenting a skewed picture. Forms of response bias:  Extremity bias – type of bias that occurs when a respondent responds either very positively or very negatively to a particular question. Opposed to extremity bias is centrality or neutrality bias where a person constantly opts for the neutral or central response options.  Stringency or leniency bias – type of bias frequently encountered when raters or assessors are used to generate scores and are either very strict or very lenient. Page 49 of 179 Downloaded by: Portialepesa | Distribution of this document is illegal S - The Marketplace to Buy and Sell your Study Material  Acquiescence bias – occurs when a respondent is in agreement with all statements or questions he is asked about. No clear preferences or dislikes.  Halo effect – occurs where respondents are systematically influenced by favourable or unfavourable attributes of the object that they rate.  Social desirability bias – similar response bias where respondent reacts in a manner which is socially desirable or acceptable.  Positive falsifications – occurs when respondents or testees purposefully misrepresent facts or deliberately provide factually incorrect responses.  Unconscious misrepresentation – unlike falsification, misrepresentation is not done on purpose o When a measure is speeded:  Difference between speed and power measures:  A speed test is timed – the items are all relatively easy but there are many of them. So many that most testees will never finish the test.  Power tests refer to tests that aren’t normally timed and the questions increase in difficulty. This is assessing mastery of a skill.  Test-retest and equivalent-form reliability are appropriate for speeded measures.  Split-half techniques may be used if the split is according to time rather than items. o Variability in individual scores – Any correlation is affected by the range of individual differences in the group. o Ability level – variability and ability level of samples should also be considered.  It is desirable to compute reliability coefficients separately for homogeneous subgroups, such as gender, age, and occupation, instead of for the total heterogeneous sample. o Admin error – variations in admin procedures normally occur when non-standardised assessment practices are followed:  Variations in instructions – instructions to respondents are inconsistent or not provided in a standardised manner. Test manuals are important.  Variations in assessment conditions – should ensure that tests are conducted under standard classroom conditions.  Variations in the interpretation of instructions – this may result in variable assessment outcomes  Variations in scoring or ratings – there should be clear instructions on how responses on the instrument should be scored or rated.  An alternative way of expressing test reliability is through the standard error of measurement (SEM): o SEM can be used to interpret individual test scores in terms of the reasonable limits within which they are likely to vary as a function of measurement error. o SEM is independent of the variability of the group on which it was computed. o SEM can be computed from the reliability coefficient of a test o Reliability is related to error measurement in that smaller errors in measurement mean higher reliability  In mastery testing or criterion-referenced assessment, there is little variability of scores among testees. Mastery measures try to differentiate between people who have mastered certain skills and knowledge for a specific job or training programme and those who have not. The usual correlation procedures for determining reliability are inappropriate.  The validity of a measure concerns what the test measures and how well is does so.  Types of validity or validation procedures: o Content-description procedures  There are 2 important aspects when considering content validity of a measure:  Face validity – does not refer to what the test measures, but to what it appears to measure.  Content validity – involves determining whether the content of the measure covers a representative sample of the behaviour domain / aspect to be measured.  Non-statistical type of validity and refers to a specific procedure in constructing a psychological measure  A frequently used procedure to ensure high content validity is the use of a panel of subject expers to evaluate the items during the test construction phase.  Content validity is especially relevant for evaluating achievement, educational, and occupational measures.  This type of validity is not the most appropriate for personality and aptitude measures (criterionpredictive is more suitable for that) o Criterion-prediction procedures  Criterion-prediction validity is a quantitative procedure which can be defined as the calculation of a correlation coefficient between a predictor, or more than one predictor, and a criterion. Page 50 of 179 Downloaded by: Portialepesa | Distribution of this document is illegal S - The Marketplace to Buy and Sell your Study Material  Two different types of criterion-related validity can be identified and the distinction between these types is based on the purpose for which the measure is used:  Concurrent validity: involves the accuracy with which a measure can IDENTIFY or DIAGNOSE the current behaviour or status regarding specific skills or characteristics of an individual  Predictive validity: accuracy with which a measure can PREDICT the future behaviour or category status of an individual  Criterion Contamination – this is the effect of any factor or variable on a criterion such that the criterion is no longer a valid measure. The halo error is when you make judgements based on a general impression of a person or a very prominent aspect only.  Criterion must thus be free from any kind of bias  Most commonly used criterion measures:  Academic achievement - Most FREQUENTLY used for validation of intelligence, multiple aptitude and personality measures  Performance in specialized training - Criterion frequently used for specific aptitude measures  Actual job performance - Most APPROPRIATE criterion measure of the validity of intelligence , special aptitude and personality measures  Contrast groups - Sometimes used when validating personality measures  Psychiatric diagnoses - This based on prolonged observation and case history may be used as evidence of test validity for personality measures  Ratings - By teachers, lecturers, instructors, supervisors and co-workers are commonly used as criteria. Ratings are suitable for almost every kind of measure.  Meta-analysis – method of reviewing research literature. It is a statistical integration and analysis of previous findings on a specific topic