IOP 3701 EXAM PACK-2017
PREPARED BY CHIPO MAREGA
MAY/JUNE 2016
1. Motivate the use of intra scorer and inters corer types of reliability as opposed to other
types of reliability (5)
Inter-scorer (rater)
Inter-scorer (rater) reliability is used to measure the degree to which different raters/observers
give consistent approximate scores/measures of the same observable fact. Examiner
inconsistency is a possible source of error variance. Inter-scorer (or inter-rater) reliability can be
determined by having all the test-takers' test procedures scored by two assessment practitioners.
The correlation coefficient between these two sets of scores reveals the inter-scorer reliability
coefficient.
Advantages of inter scorer validity
Imposes level of objectivity.
Provides validation of evaluation results.
Certify skills competency
Assign the appropriate “grade”
Increase confidence that people are following the guidelines in like manner.
Intra-scorer (rater)
The intra-scorer reliability coefficient refers to the consistency of ratings for a single rater.
Repeated ratings or scores by the same rater would give an indication of the degree of error
variance between such ratings for that particular rater.
Advantages of intra-scorer validity
Improve single marker consistency in evaluating candidates’ performance.
Eliminating external factors that can compromise marker’s evaluation.
Increase the confidence in marker’s ability.
Increase the marker’s competence in repeated occurrence.
, 2. Discuss the importance of item difficulty and discrimination indices in the analysis of
items during the test development process.
Determining item difficulty – p value
This is proportion/percentage of individuals who answer correctly
Number of people who answered correctly divided by number of people who took the
measure
The higher correct responses the easier the item
The smaller the percentages of correct responses, the more difficult
No info about characteristics of item
P value provides uniform measure of difficulty & is used when selecting final items for
measure
Determining discriminating power
One of purpose of item analysis is to discover which items best measures the construct or
content domain that the measure aim to assess
Good items consistently measure the same aspect that the total test is measuring. Individuals
do well in answering a good item correctly, whilst those who do poorly would answer a good
item incorrectly
The discriminating power of an item can be determined by discriminating index & item total
correlation
A positive D value indicates an item that discriminates btw the extreme groups and a negative
D value indicates an item with poor discriminating power
3.. Test results represent only one source of information and this information is predominantly
approximate in nature. Explain this statement (5)
Important issues
Test results represent only one source of information in the assessment process:
- One 1 source of info in the assessment process
- Assessment measures are objective measures scientific and unbiased thus practitioner
begin 2 value them above their own option/judgment
- Thus obscuring the view because they are one sided
- Thus subjectivity should be included
Need to recognize the approximate nature of assessment results
, - Results can have errors of measure during: admin, scoring and interpreting thus uncertainty
- Process has errors give uncertain results
- Individual is influenced by social, economic, educational and cultural background thereby giving
a distorted picture.
Not just in the field of psychology, teachers, social workers, doctor etc
4. Discuss the planning involved for group assessments to minimise cheating (5)
Seating arrangement –leave enough space between test takers
Prepare multiple forms of the measure and distribute different forms to adjacent test takers.
Use multiple answer sheets with different layouts
Train assessment assistants who will be roving around during test administration.
5. The Human Resources manager in your company has been advocating for greater use of
computerised adaptive testing (CAT). She requested you to provide a written document which you
present a balanced view of the CAT as opposed to the standard testing procedures. In your
discussion, also highlight the advantages and disadvantages of CAT (10)
Advantages of computer-based testing:
Standardisation of assessment instructions
Eliminates bias practitioners
There is a reduction in the amount of time needed for the assessment
More information obtained on test-takers and instant scoring that allows for prompt feedback
Graphic abilities measure spatial and perceptual abilities
Useful to test-takers who have physical and neurological disabilities
Can be individually tailored – minimizes cheating
More control for practitioner
Fewer practitioners and assistants are needed during the administration
Decrease in errors that arise from inaccurate scoring
Computerised testing increases test security as test materials cannot be removed from the test room
easily
Disadvantages of computer-based testing:
Copyright violations when measures are made available on the internet
Lack of security when measures are made available on the internet
Problems of confidentiality
Computer-generated assessment reports still require clinical judgement as far as interpretation is
concerned
Computerised scoring routines may have errors or may be poorly validated
Computerised testing involves high costs in item development
6. Discuss the guidelines of the EEA in terms of provisions it makes for fairness and bias (10)
PREPARED BY CHIPO MAREGA
MAY/JUNE 2016
1. Motivate the use of intra scorer and inters corer types of reliability as opposed to other
types of reliability (5)
Inter-scorer (rater)
Inter-scorer (rater) reliability is used to measure the degree to which different raters/observers
give consistent approximate scores/measures of the same observable fact. Examiner
inconsistency is a possible source of error variance. Inter-scorer (or inter-rater) reliability can be
determined by having all the test-takers' test procedures scored by two assessment practitioners.
The correlation coefficient between these two sets of scores reveals the inter-scorer reliability
coefficient.
Advantages of inter scorer validity
Imposes level of objectivity.
Provides validation of evaluation results.
Certify skills competency
Assign the appropriate “grade”
Increase confidence that people are following the guidelines in like manner.
Intra-scorer (rater)
The intra-scorer reliability coefficient refers to the consistency of ratings for a single rater.
Repeated ratings or scores by the same rater would give an indication of the degree of error
variance between such ratings for that particular rater.
Advantages of intra-scorer validity
Improve single marker consistency in evaluating candidates’ performance.
Eliminating external factors that can compromise marker’s evaluation.
Increase the confidence in marker’s ability.
Increase the marker’s competence in repeated occurrence.
, 2. Discuss the importance of item difficulty and discrimination indices in the analysis of
items during the test development process.
Determining item difficulty – p value
This is proportion/percentage of individuals who answer correctly
Number of people who answered correctly divided by number of people who took the
measure
The higher correct responses the easier the item
The smaller the percentages of correct responses, the more difficult
No info about characteristics of item
P value provides uniform measure of difficulty & is used when selecting final items for
measure
Determining discriminating power
One of purpose of item analysis is to discover which items best measures the construct or
content domain that the measure aim to assess
Good items consistently measure the same aspect that the total test is measuring. Individuals
do well in answering a good item correctly, whilst those who do poorly would answer a good
item incorrectly
The discriminating power of an item can be determined by discriminating index & item total
correlation
A positive D value indicates an item that discriminates btw the extreme groups and a negative
D value indicates an item with poor discriminating power
3.. Test results represent only one source of information and this information is predominantly
approximate in nature. Explain this statement (5)
Important issues
Test results represent only one source of information in the assessment process:
- One 1 source of info in the assessment process
- Assessment measures are objective measures scientific and unbiased thus practitioner
begin 2 value them above their own option/judgment
- Thus obscuring the view because they are one sided
- Thus subjectivity should be included
Need to recognize the approximate nature of assessment results
, - Results can have errors of measure during: admin, scoring and interpreting thus uncertainty
- Process has errors give uncertain results
- Individual is influenced by social, economic, educational and cultural background thereby giving
a distorted picture.
Not just in the field of psychology, teachers, social workers, doctor etc
4. Discuss the planning involved for group assessments to minimise cheating (5)
Seating arrangement –leave enough space between test takers
Prepare multiple forms of the measure and distribute different forms to adjacent test takers.
Use multiple answer sheets with different layouts
Train assessment assistants who will be roving around during test administration.
5. The Human Resources manager in your company has been advocating for greater use of
computerised adaptive testing (CAT). She requested you to provide a written document which you
present a balanced view of the CAT as opposed to the standard testing procedures. In your
discussion, also highlight the advantages and disadvantages of CAT (10)
Advantages of computer-based testing:
Standardisation of assessment instructions
Eliminates bias practitioners
There is a reduction in the amount of time needed for the assessment
More information obtained on test-takers and instant scoring that allows for prompt feedback
Graphic abilities measure spatial and perceptual abilities
Useful to test-takers who have physical and neurological disabilities
Can be individually tailored – minimizes cheating
More control for practitioner
Fewer practitioners and assistants are needed during the administration
Decrease in errors that arise from inaccurate scoring
Computerised testing increases test security as test materials cannot be removed from the test room
easily
Disadvantages of computer-based testing:
Copyright violations when measures are made available on the internet
Lack of security when measures are made available on the internet
Problems of confidentiality
Computer-generated assessment reports still require clinical judgement as far as interpretation is
concerned
Computerised scoring routines may have errors or may be poorly validated
Computerised testing involves high costs in item development
6. Discuss the guidelines of the EEA in terms of provisions it makes for fairness and bias (10)
