IOP 3701 EXAM PACK-2017
PREPARED BY CHIPO MAREGA
MAY/JUNE 2016
1. Motivate the use of intra scorer and inters corer types of reliability as opposed to other types
of reliability (5)
Inter-scorer (rater)
Inter-scorer (rater) reliability is used to measure the degree to which different raters/observers give
consistent approximate scores/measures of the same observable fact. Examiner inconsistency is
a possible source of error variance. Inter-scorer (or inter-rater) reliability can be determined by
having all the test-takers' test procedures scored by two assessment practitioners. The correlation
coefficient between these two sets of scores reveals the inter-scorer reliability coefficient.
Advantages of inter scorer validity
➢ Imposes level of objectivity.
➢ Provides validation of evaluation results.
➢ Certify skills competency
➢ Assign the appropriate “grade”
➢ Increase confidence that people are following the guidelines in like manner.
Intra-scorer (rater)
The intra-scorer reliability coefficient refers to the consistency of ratings for a single rater. Repeated
ratings or scores by the same rater would give an indication of the degree of error variance between
such ratings for that particular rater.
Advantages of intra-scorer validity
➢ Improve single marker consistency in evaluating candidates’ performance.
➢ Eliminating external factors that can compromise marker’s evaluation.
➢ Increase the confidence in marker’s ability.
➢ Increase the marker’s competence in repeated occurrence.
, 2. Discuss the importance of item difficulty and discrimination indices in the analysis of items
during the test development process.
❖ Determining item difficulty – p value
✓ This is proportion/percentage of individuals who answer correctly
✓ Number of people who answered correctly divided by number of people who took the measure
✓ The higher correct responses the easier the item
✓ The smaller the percentages of correct responses, the more difficult
✓ No info about characteristics of item
✓ P value provides uniform measure of difficulty & is used when selecting final items for measure
❖ Determining discriminating power
✓ One of purpose of item analysis is to discover which items best measures the construct or
content domain that the measure aim to assess
✓ Good items consistently measure the same aspect that the total test is measuring. Individuals
do well in answering a good item correctly, whilst those who do poorly would answer a good
item incorrectly
✓ The discriminating power of an item can be determined by discriminating index & item total
correlation
✓ A positive D value indicates an item that discriminates btw the extreme groups and a negative
D value indicates an item with poor discriminating power
3.. Test results represent only one source of information and this information is predominantly
approximate in nature. Explain this statement (5)
Important issues
• Test results represent only one source of information in the assessment process:
- One 1 source of info in the assessment process
- Assessment measures are objective measures scientific and unbiased thus practitioner begin
2 value them above their own option/judgment
- Thus obscuring the view because they are one sided
- Thus subjectivity should be included
• Need to recognize the approximate nature of assessment results
- Results can have errors of measure during: admin, scoring and interpreting thus uncertainty
- Process has errors give uncertain results
, - Individual is influenced by social, economic, educational and cultural background thereby giving a
distorted picture.
Not just in the field of psychology, teachers, social workers, doctor etc
4. Discuss the planning involved for group assessments to minimise cheating (5)
➢ Seating arrangement –leave enough space between test takers
➢ Prepare multiple forms of the measure and distribute different forms to adjacent test takers.
➢ Use multiple answer sheets with different layouts
➢ Train assessment assistants who will be roving around during test administration.
5. The Human Resources manager in your company has been advocating for greater use of
computerised adaptive testing (CAT). She requested you to provide a written document which you
present a balanced view of the CAT as opposed to the standard testing procedures. In your
discussion, also highlight the advantages and disadvantages of CAT (10)
Advantages of computer-based testing:
• Standardisation of assessment instructions
• Eliminates bias practitioners
• There is a reduction in the amount of time needed for the assessment
• More information obtained on test-takers and instant scoring that allows for prompt feedback
• Graphic abilities measure spatial and perceptual abilities
• Useful to test-takers who have physical and neurological disabilities
• Can be individually tailored – minimizes cheating
• More control for practitioner
• Fewer practitioners and assistants are needed during the administration
• Decrease in errors that arise from inaccurate scoring
• Computerised testing increases test security as test materials cannot be removed from the test room
easily
Disadvantages of computer-based testing:
• Copyright violations when measures are made available on the internet
• Lack of security when measures are made available on the internet
• Problems of confidentiality
PREPARED BY CHIPO MAREGA
MAY/JUNE 2016
1. Motivate the use of intra scorer and inters corer types of reliability as opposed to other types
of reliability (5)
Inter-scorer (rater)
Inter-scorer (rater) reliability is used to measure the degree to which different raters/observers give
consistent approximate scores/measures of the same observable fact. Examiner inconsistency is
a possible source of error variance. Inter-scorer (or inter-rater) reliability can be determined by
having all the test-takers' test procedures scored by two assessment practitioners. The correlation
coefficient between these two sets of scores reveals the inter-scorer reliability coefficient.
Advantages of inter scorer validity
➢ Imposes level of objectivity.
➢ Provides validation of evaluation results.
➢ Certify skills competency
➢ Assign the appropriate “grade”
➢ Increase confidence that people are following the guidelines in like manner.
Intra-scorer (rater)
The intra-scorer reliability coefficient refers to the consistency of ratings for a single rater. Repeated
ratings or scores by the same rater would give an indication of the degree of error variance between
such ratings for that particular rater.
Advantages of intra-scorer validity
➢ Improve single marker consistency in evaluating candidates’ performance.
➢ Eliminating external factors that can compromise marker’s evaluation.
➢ Increase the confidence in marker’s ability.
➢ Increase the marker’s competence in repeated occurrence.
, 2. Discuss the importance of item difficulty and discrimination indices in the analysis of items
during the test development process.
❖ Determining item difficulty – p value
✓ This is proportion/percentage of individuals who answer correctly
✓ Number of people who answered correctly divided by number of people who took the measure
✓ The higher correct responses the easier the item
✓ The smaller the percentages of correct responses, the more difficult
✓ No info about characteristics of item
✓ P value provides uniform measure of difficulty & is used when selecting final items for measure
❖ Determining discriminating power
✓ One of purpose of item analysis is to discover which items best measures the construct or
content domain that the measure aim to assess
✓ Good items consistently measure the same aspect that the total test is measuring. Individuals
do well in answering a good item correctly, whilst those who do poorly would answer a good
item incorrectly
✓ The discriminating power of an item can be determined by discriminating index & item total
correlation
✓ A positive D value indicates an item that discriminates btw the extreme groups and a negative
D value indicates an item with poor discriminating power
3.. Test results represent only one source of information and this information is predominantly
approximate in nature. Explain this statement (5)
Important issues
• Test results represent only one source of information in the assessment process:
- One 1 source of info in the assessment process
- Assessment measures are objective measures scientific and unbiased thus practitioner begin
2 value them above their own option/judgment
- Thus obscuring the view because they are one sided
- Thus subjectivity should be included
• Need to recognize the approximate nature of assessment results
- Results can have errors of measure during: admin, scoring and interpreting thus uncertainty
- Process has errors give uncertain results
, - Individual is influenced by social, economic, educational and cultural background thereby giving a
distorted picture.
Not just in the field of psychology, teachers, social workers, doctor etc
4. Discuss the planning involved for group assessments to minimise cheating (5)
➢ Seating arrangement –leave enough space between test takers
➢ Prepare multiple forms of the measure and distribute different forms to adjacent test takers.
➢ Use multiple answer sheets with different layouts
➢ Train assessment assistants who will be roving around during test administration.
5. The Human Resources manager in your company has been advocating for greater use of
computerised adaptive testing (CAT). She requested you to provide a written document which you
present a balanced view of the CAT as opposed to the standard testing procedures. In your
discussion, also highlight the advantages and disadvantages of CAT (10)
Advantages of computer-based testing:
• Standardisation of assessment instructions
• Eliminates bias practitioners
• There is a reduction in the amount of time needed for the assessment
• More information obtained on test-takers and instant scoring that allows for prompt feedback
• Graphic abilities measure spatial and perceptual abilities
• Useful to test-takers who have physical and neurological disabilities
• Can be individually tailored – minimizes cheating
• More control for practitioner
• Fewer practitioners and assistants are needed during the administration
• Decrease in errors that arise from inaccurate scoring
• Computerised testing increases test security as test materials cannot be removed from the test room
easily
Disadvantages of computer-based testing:
• Copyright violations when measures are made available on the internet
• Lack of security when measures are made available on the internet
• Problems of confidentiality
