IOP3701_ EXAM PACK.

IOP3701_ EXAM PACK. IOP3701 - Industrial Psychological Testing And Assessment. Motivate the use of intra scorer and inters corer types of reliability as opposed to other types of reliability (5) Inter-scorer (rater) Inter-scorer (rater) reliability is used to measure the degree to which different raters/observers give consistent approximate scores/measures of the same observable fact. Examiner inconsistency is a possible source of error variance. Inter-scorer (or inter-rater) reliability can be determined by having all the test-takers' test procedures scored by two assessment practitioners. The correlation coefficient between these two sets of scores reveals the inter-scorer reliability coefficient. Advantages of inter scorer validity  Imposes level of objectivity.  Provides validation of evaluation results.  Certify skills competency  Assign the appropriate “grade”  Increase confidence that people are following the guidelines in like manner. Intra-scorer (rater) The intra-scorer reliability coefficient refers to the consistency of ratings for a single rater. Repeated ratings or scores by the same rater would give an indication of the degree of error variance between such ratings for that particular rater. Advantages of intra-scorer validity  Improve single marker consistency in evaluating candidates’ performance.  Eliminating external factors that can compromise marker’s evaluation.  Increase the confidence in marker’s ability.  Increase the marker’s competence in repeated occurrence. 2. Discuss the importance of item difficulty and discrimination indices in the analysis of items during the test development process.  Determining item difficulty – p value  This is proportion/percentage of individuals who answer correctly  Number of people who answered correctly divided by number of people who took the measure  The higher correct responses the easier the item  The smaller the percentages of correct responses, the more difficult  No info about characteristics of item  P value provides uniform measure of difficulty & is used when selecting final items for measure  Determining discriminating power  One of purpose of item analysis is to discover which items best measures the construct or content domain that the measure aim to assess  Good items consistently measure the same aspect that the total test is measuring. Individuals do well in answering a good item correctly, whilst those who do poorly would answer a good item incorrectly  The discriminating power of an item can be determined by discriminating index & item total correlation  A positive D value indicates an item that discriminates btw the extreme groups and a negative D value indicates an item with poor discriminating power 3.. Test results represent only one source of information and this information is predominantly approximate in nature. Explain this statement (5) Important issues  Test results represent only one source of information in the assessment process: - One 1 source of info in the assessment process - Assessment measures are objective measures scientific and unbiased thus practitioner begin 2 value them above their own option/judgment - Thus obscuring the view because they are one sided - Thus subjectivity should be included  Need to recognize the approximate nature of assessment results - Results can have errors of measure during: admin, scoring and interpreting thus uncertainty - Process has errors give uncertain results - Individual is influenced by social, economic, educational and cultural background thereby giving a distorted picture. Not just in the field of psychology, teachers, social workers, doctor etc 4. Discuss the planning involved for group assessments to minimise cheating (5)  Seating arrangement –leave enough space between test takers  Prepare multiple forms of the measure and distribute different forms to adjacent test takers.  Use multiple answer sheets with different layouts  Train assessment assistants who will be roving around during test administration. 5. The Human Resources manager in your company has been advocating for greater use of computerised adaptive testing (CAT). She requested you to provide a written document which you present a balanced view of the CAT as opposed to the standard testing procedures. In your discussion, also highlight the advantages and disadvantages of CAT (10) Advantages of computer-based testing:  Standardisation of assessment instructions  Eliminates bias practitioners  There is a reduction in the amount of time needed for the assessment  More information obtained on test-takers and instant scoring that allows for prompt feedback  Graphic abilities measure spatial and perceptual abilities  Useful to test-takers who have physical and neurological disabilities  Can be individually tailored – minimizes cheating  More control for practitioner  Fewer practitioners and assistants are needed during the administration  Decrease in errors that arise from inaccurate scoring  Computerised testing increases test security as test materials cannot be removed from the test room easily Disadvantages of computer-based testing:  Copyright violations when measures are made available on the internet  Lack of security when measures are made available on the internet  Problems of confidentiality  Computer-generated assessment reports still require clinical judgement as far as interpretation is concerned  Computerised scoring routines may have errors or may be poorly validated  Computerised testing involves high costs in item development 6. Discuss the guidelines of the EEA in terms of provisions it makes for fairness and bias (10) The employment equity act was passed in an attempt to regulate activities in the work context. Psychological testing and other similar assessments are mentioned specifically in the Act. The Employment Equity Act states that the purpose of the act is to achieve equity in the workplace by: a) promoting equal opportunity and fair treatment in employment through the elimination of unfair discrimination; and b) implementing affirmative action measures to redress the disadvantages in employment experienced by designated groups With regard to psychological testing and other similar assessments, the EEA 55 of 1998 states that: Psychological testing and other similar assessments of an employee are prohibited unless the test or assessment being used: a) has been scientifically shown to be valid and reliable; b) can be applied fairly to all employees; and c) is not biased against any employee or group. o Construct-identification procedures  Construct validity – the extent to which it measures the theoretical construct or trait it is supposed to measure.  E.g. of constructs or traits are: Intelligence, verbal ability, neuroticism, anxiety, spatial perception, eye-hand coordination, introversion-extroversion.  Statistical methods to ascertain whether the measure actually measures what it is supposed to be measuring:  Correlation with other tests:  High correlation between new measure and similar earlier measure of the same name indicates that the new measure assesses approximately the same construct or area of behaviour.  The measure should only be moderately high otherwise the measure is just a duplication of the old one. 7. Discuss the impact of social desirability on personality testing and the measures devised to overcome this problem. (10)  Social desirability is a problem in structured personality measures  It may be obvious form the content of the item/questions what the assessment practitioner is attempting to assess. For example in a job interview ,one might be requested to answer Yes/No to the following item :- “I tell lies sometimes”  One might feel pressurised to answer NO as this will create a negative perception to the interview panel.  One is therefore forced to answer in a socially desirable manner.  If one continues to answer in that format, a set of socially desirable answers will be discernible/ evident.  This can be due to both self-deception and other-deception. Social desirability can affect the validity of findings, but procedures such as the use of forced-choice items and the use of proxy subjects can be effective in preventing or reducing social desirability bias 8. The test scores can be interpreted in terms of expected criterion performance, thus the use of ct off scores. Discuss what the setting of standards and cut off scores entail and their importance to the screening and selection process. Also discuss the implications of cut off scores to redress past imbalances and to provide equal employment opportunities. (10) Refer to page 42  The term cut-off score refers to the lowest possible score on an exam, standardized test, highstakes test, or other form of assessment that a test taker must earn to either “pass” or be considered “proficient.”  In some cases, tests may have multiple cut-off scores representing tiered levels of proficiency, such as basic, proficient, or advanced. In education, cut-off scores may also be applied in certification and licensing exams that are used to determine whether educators and other school staff are professionally “qualified.”  Cut-off scores are determined though a process generally called standard setting (for criterionreferenced tests) or norming (for norm-referenced tests).  In a typical standard-setting process, a test developer will form a standard-setting panel by recruiting a group of experts,  the process typically entails reviewing test items (questions, problems, tasks), determining levels of difficulty for each item, and using a statistical process, based on collective opinion, to establish a cut-off score or a set of cut scores corresponding to different levels of “proficiency” (e.g., basic, proficient, or advanced).  Cut-off scores necessarily rely more heavily on individual professional judgment.  Cut-off scores may become an extension of ongoing debates related to high-stakes testing—i.e., the use of tests to make important decisions about interview candidates  In addition, cut-off scores—and the bar for proficiency they represent—can diverge significantly from system to system, state to state, test to test, 9. Discuss the construct validity identification procedures (10) Construct-identification procedures  The construct validity of a measure is the extent to which it measures the theoretical construct or trait that it is supposed to measure.  Examples of constructs are: intelligence, verbal ability, spatial perception, eye-hand coordination, and introversion-extroversion.  Statistical measures to ascertain whether the measure actually measures what it is supposed to measure:  Correlation with other tests: A high correlation between a new measure and a similar earlier measure of the SAME construct indicates that the new measure assesses approximately the same construct (or area of behaviour).  Factorial validity: statistical technique for analysing the interrelationship of variables. The aim is to determine the underlying structure or dimensions of a set of variables. The factorial validity of a measure refers to the underlying dimensions (factors) tapped by the measure, as determined by the process of factor analysis.  Convergent and discriminate validity: A measure demonstrates this when it correlates highly with other variables with which it should theoretically correlate, and correlates minimally with variables from which it should differ.  Incremental validity: A measure displays this when it explains numerically additional variance compared to a set of other measures when predicting a dependent variable.  Differential validity: A measure possesses differential validity if it succeeds in differentiating or distinguishing between characteristics of individuals, groups or organisations. 10. The use of psychological testing is governed by the strict ethics codes and prescribed procedures. Tests need to be treated with circumspection and responsibility, otherwise things could go wrong. Briefly discuss the consequences of the following a) The unauthorised use of tests by unqualified persons(4) b) An unprepared test administrator(3) c) A lack of rapport and poor test taking orientation (3)