IOP_3701_EXAM_PACK.

MAY/JUNE 2016 1. Motivate the use of intra scorer and inters corer types of reliability as opposed to other types of reliability (5) Inter-scorer (rater) Inter-scorer (rater) reliability is used to measure the degree to which different raters/observers give consistent approximate scores/measures of the same observable fact. Examiner inconsistency is a possible source of error variance. Inter-scorer (or inter-rater) reliability can be determined by having all the test-takers' test procedures scored by two assessment practitioners. The correlation coefficient between these two sets of scores reveals the inter-scorer reliability coefficient. Advantages of inter scorer validity Imposes level of objectivity. Provides validation of evaluation results. Certify skills competency Assign the appropriate “grade” Increase confidence that people are following the guidelines in like manner. Intra-scorer (rater) The intra-scorer reliability coefficient refers to the consistency of ratings for a single rater. Repeated ratings or scores by the same rater would give an indication of the degree of error variance between such ratings for that particular rater. Advantages of intra-scorer validity Improve single marker consistency in evaluating candidates’ performance. Eliminating external factors that can compromise marker’s evaluation. Increase the confidence in marker’s ability. Increase the marker’s competence in repeated occurrence. 2. Discuss the importance of item difficulty and discrimination indices in the analysis of items during the test development process. Determining item difficulty – p value This is proportion/percentage of individuals who answer correctly Number of people who answered correctly divided by number of people who took the measure The higher correct responses the easier the item The smaller the percentages of correct responses, the more difficult No info about characteristics of item P value provides uniform measure of difficulty & is used when selecting final items for measure Determining discriminating power One of purpose of item analysis is to discover which items best measures the construct or content domain that the measure aim to assess Good items consistently measure the same aspect that the total test is measuring. Individuals do well in answering a good item correctly, whilst those who do poorly would answer a good item incorrectly The discriminating power of an item can be determined by discriminating index & item total correlation A positive D value indicates an item that discriminates btw the extreme groups and a negative D value indicates an item with poor discriminating power