Industrial Psychological Testing and Assessment iop3701201_2018

QUESTION 1 The Employment Equity Act (EEA) gives clear guidelines on the use of psychological assessments tools. Discuss. In your discussion, also make an argument for or against the adequacy of the Act to address concerns of psychological assessments. (10) Only three provisions that relate to reliability and validity, as well as fairness and bias, are mentioned in the study material. See pages 21 and 22 of Foxcroft and Roodt (2009) or page 22 of Foxcroft and Roodt (2013) for the exact wording used. For full marks, you had to discuss these provisions in relation to how they enhance fair and equitable testing in the South African context. Guidelines on the use of psychological tests: Psychological testing and other similar assessments of an employee are prohibited unless the test or assessment being used (a) has been scientifically shown to be valid and reliable; (b) can be applied fairly to all employees; and (c) is not biased against any employee or group. (1 mark if guidelines were provided – (a), (b) and (c)). An amendment to the Act in 2014 resulted in a fourth provision, which states that the test or assessment has to be “certified by the HPCSA established by section 2 of the Health Professions Act, 1974 (Act no 56 of 1974), or any other body which may be authorised by law to certify those tests or assessments". However, reference had to be made to the press release of the Association of Test Publishers (ATP) where the amendment of section 8 of the EEA (section “d”) was declared null and void. The EEA section 8 (a), (b) and (c) are still applicable. (1 mark if the student referred to the null and void as per the ATP.)  In the first instance, validity and reliability has to be scientifically investigated. This would mean that proper procedures of test development and test validation have to be followed. Validity concerns whether the test measures what it is supposed to measure and how well it does so; while reliability refers to the consistency with which it measures whatever it measures. (2 marks)  Secondly, assessment tools need to be fairly applied to all employees. This means that unfair practices or the use of tools that could constitute some unfair assessment, are prohibited. To ensure testing is done for optimal benefit, there should be thorough job analysis that can be used to justify and evaluate minimum requirements. Fairness is a value judgment and can differ from situation to situation. (1 mark) IOP3701/201/2/2018 4  Thirdly, care should be taken not to include any measures that show bias against any subgroup or individual. Many of the measures used have not been investigated for bias, nor have they been cross-culturally validated. Taking into account the different language and culture groups in South Africa, it means bias studies would need to be conducted for all subgroups, failing which each assessment would have to indicate for which subgroups it can be used. This is a field (bias) that will receive a lot of research attention in the future and the use of IRT and other procedures to ensure appropriate assessment tools in cross-cultural (language) contexts is considered very important for test development in South Africa in particular. (1 mark for bias and 1 mark if discussed in a South-African context) Students needed to argue for or against the adequacy of the Act to address concerns. The following are the arguments for:  The act serves as a deterrent against the improper use of assessment measures and discriminatory assessment practices. (1 mark)  It has as its aim the proper use of well-developed and scientifically justifiable measurement tools. People in industry, as well as test developers, welcome these guidelines, because they promote good practice in the field of psychological assessment. (1 mark)  Practitioners are able to check the available measures in terms of their quality, appropriateness of norms, availability of different language versions, etc. (1 mark) QUESTION 2 Discuss the relationship between reliability and validity – in particular whether one can be a prerequisite for the other. (5) Chapter 4 and 5 of Foxcroft and Roodt (2009 & 2013) are important for understanding these concepts, and study unit 3 provides clear guidelines to the chapters. Reliability refers to the consistency of a measure, while validity indicates whether the test measures what it is supposed to measure and how well it does so. (2 marks) As you would have gathered, both reliability and validity are psychometric properties used to judge the quality of a test. The validity of a measure is directly proportional to its reliability. The validity of a measure is therefore limited in relation to the reliability of a measure. According to the Employment Equity Act, the requirement for any measure is that it should be valid and reliable. (1 mark) For a measure to be valid, it would of necessity also have to be reliable. However, the opposite is not true, in that just because a measure has been shown to be reliable, it cannot be assumed that it is also valid. Reliability does not imply validity. (2 marks) QUESTION 3 Differentiate between inter-item consistency and inter-scorer (rater) reliability. (5) These are both types of reliability discussed in sections 4.2.2.4 and 4.2.2.5 of Foxcroft and Roodt (2009; 2013). Inter-item Inter-scorer Consistency of ratings of a single rater. Consistency of ratings between more than one rater. Based on consistency of responses to all items Based on consistency of ratings between raters (scorers) IOP3701/201/2/2018 5 Inter-item Inter-scorer Applicable to measures with highly standardised procedures for administration and scoring Applicable to measures with limited standardisation of administration and scoring procedures – thus examiner variance can be a source of error Applicable to items that have a multiplechoice format Applicable to open-ended questions, projective techniques, individual intelligence tests Repeated ratings or scores by the same rater – indication of the degree of error of variance between such ratings for the particular rater Determined by having all the test takers’ test protocols scored/checked by two (or more) assessment practitioners QUESTION 4 Describe what norms are and why they are used in psychological assessment. Why is it necessary to use norm scores and not raw scores in the interpretation of psychological test results? (5) The discussion of norms appears in sections 3.5.1 and 3.5.2 of Foxcroft and Roodt (2009) or sections 3.6.1 and 3.6.2 of Foxcroft and Roodt (2013). A norm is a measurement against which the individual’s raw score is evaluated so that the individual’s position relative to that of the normative sample can be determined. (2 marks) The raw scores which test takers obtain on psychological measures have little or no meaning on their own. In order to make the interpretation more meaningful, these raw scores are converted to normal scores through statistical transformation. (1 mark) By comparing an individual’s test score to that of a similar group of people (norm group) the individual’s score can be more meaningfully interpreted. It is important to determine if the norm group or standardised sample is representative of the candidates that you are testing. 1 mark) Norms give meaning to test scores – they make it possible to make comparisons between individuals (and groups). (1 mark) QUESTION 5 Discuss the responsibilities of organisations in terms of fair assessment practices. (10) The organisation has the responsibility to ensure that:  it has an assessment policy in place that reflects fair, ethical practices  it employs assessment practitioners who are competent, and who have been appropriately trained  valid assessment measures are used for appropriate purposes  assessment results are used in a non-discriminatory manner  it has support mechanisms in place to assist assessment practitioners to build a research database that can be used to establish the fairness and efficacy of the measures used and the decisions made  it regularly monitors the extent to which its assessment policy is being put into effect on the ground and revises it where necessary IOP3701/201/2/2018 6 QUESTION 6 The future of psychological assessment is certain to include greater use of computers. In view of this statement, discuss five advantages and five disadvantages that are related to the use of computers. (10) Refer to 14.3 in Foxcroft and Roodt (2009, 2013). Computers have been used in test development for a long time, as well as in item analysis and other statistical procedures. With the increasing impact of computers on every facet of life, computers are used now from test construction to administration, scoring, reporting and interpretation. There are testing standards that provide guidelines for computer-based testing. Contributions / Advantages (any 5 points, 1 mark each)  Speed  Accuracy in scoring  Computerised adaptive techniques and multimedia interactive computer technologies possible  New ways of presenting stimulus information using computer technology  Possibility to integrate data from many different sources and integrating large amounts of data  Computer-generated reports save practitioners’ time  Increases the standardisation of administration  It provides some additional security, as tests cannot be removed from the room  Fewer test administrators needed and this results in tests being more cost-effective  Assessments can be done globally, without the need to travel Challenges / Disadvantages (any 5 points, 1 mark each)  Copyright violation when measures are made available on the internet  Lack of security when measures are available on the internet  Problems with confidentiality  Computer-generated assessment reports still require the skills of a qualified practitioner for interpretation  High cost of item development, as larger item pool is required  Vital qualitative information may not be accessed (e.g. test-taking behaviour and problemsolving strategies)  Anxiety caused by human–computer interface issues may have a negative impact on performance  Lack of computer sophistication (literacy) – some practice time may have to be arranged to familiarise persons with the keyboard  Performance – e.g. disrupted/slow connections, downloading frustrations  Security – e.g. authenticity of the test taker’s identity; data generated from an internet-delivered test is stored on a central server, which allows for greater levels of security compared to data stored in filing cabinets by various assessment practitioners.  Fairness – e.g. digital divide: those who have vs. those who do not have access to computer technology and the internet QUESTION 7 In understanding assessment results, it is always important to consider the context in which the results were obtained. The social context is seen as one of the most difficult yet most important contexts. Discuss how language can affect the performance of assessment measures. (5)