EDUC 240- LESSONS 11-12-13-14-15
Lesson 11: Training
Evaluation What is Training Evaluation?
▪ Training Evaluation: The process to assess the value (the worthiness,
the effectiveness) of training programs to employees and to organizations.
• Involves the analysis of data collected from trainees, supervisors, and through
other stakeholders (trainees) or forms of measurement.
• Objective and subjective information is gathered at different points: before,
during, and after training.
Why Conduct Training Evaluations?
▪ Assist managers in identifying what, and who, should be trained (or needs additional
training).
▪ Determine cost-benefits of a program.
▪ Determine if training program has achieved expected results (?)
▪ Diagnose strengths and weaknesses of a program and pinpoint needed improvements.
• Go back to the design phase of ADDIE and re-evaluate to see if the
improvements worked.
▪ Justify and reinforce the value of training.
Barriers to Training Evaluation
Barriers fall into two categories:
1. Pragmatic (can be time consuming)
- Requires time (loss of production) to organize the collection of the data and
specialized knowledge to properly analyze and interpret it.
2. Political (can be costly, and hard to justify to management, lack of allocating
resources)
- Pressure from administration to justify the costs, potential to reveal ineffectiveness of
training, conflict of interest (may require added resources to get external audit).
Watch this Video
Clip Dr. Robert Haccoun
suggests that “not training” is sometimes the best solution to a
problem.
,Dr. Haccoun, is training the solution to all the
problems? Dr. Haccoun: Yes, to everything. Dr. Devey:
There you go, thank you.
Dr. Haccoun: I think that we have to understand that there are many reasons why people's
performance are on the par. Certainly motivation is one of them, just initial and general
ability, confidence. A large number of issues can be supervisory and help. Lots of things
really do not need to be handled by training but what I think is really important in this matter
is evaluation, is assessing and analyzing why do we have a problem. That is really what is
important and then training becomes one of the possible solutions that we have to this
problem.
I would like to give you an example of how training evaluation, in this particular case of a
problem, turned out to produce something that was quite unexpected but very successful as it
turns out. This particular case is described in the evaluation chapter.
It took place at Bell Canada where they were introducing a new system of electronic
telephones that were more complex; it was as few years back. The way they normally would
have introduced these business telephones, the way they would normally do is by having
employees or people of the company who had bought the system, they would be provided
training by Bell Canada employees, by Bell Canada service advisors. Well, they did that
for this system and it was a complete failure.
I did seven different studies using different combinations of instructors and systems to see
if we can improve the effectiveness of training and we consistently failed. There were no
better ways we could use, with kind of experienced or naive trainers etc...
But we did notice one thing which was that the features that people used a lot they seemed
to know much better than those they rarely used. From this we developed a simple idea –
why don't we simply develop a booklet where the book would explain each of the
features of the telephone and provide the training not about how to use the system but
the training was how to use the booklet. This took half an hour to do. The company, we
tested this and based on the results of that, we found that it was extremely effective. All of a
sudden people were using their telephone instead of calling Bell to say that their system did
not work. They were actually using it properly. As a result of this, this booklet reduced the
training costs by several million dollars a year. They translated it into seventeen different
languages and is now (it was not at the time) packaged in every single telephone that they
sold worldwide.
What started off as a simple training problem was solved using a non-training or
nonconventional training approach and it had direct impact on marketing. Sometimes not
training is actually the best thing to do.
,Types of Training Evaluations
***Evaluations may be distinguished from each other in two fundamental ways:
1. The data gathered and analyzed:
1. Trainee perceptions, learning and behaviour at the conclusion of training.
2. Assessing psychological forces that operate during training.
3. Information about the work environment.
• Transfer climate and learning culture
b. The purpose of the evaluation:
Types of Training Evaluations
Formative evaluations: Summative evaluations:
Provide data about various aspects of a Provide data about the worthiness or
training program. effectiveness of a training program.
▪ Tend to focus on the content ▪ Based on evidence (data) gathered
and how it is presented with the goal at the conclusion of the training
of improving it. intervention (can also be carried out at
mid-point).
▪ Based on feedback received during
the design and development of the • Tend to focus on
learning experience. financial benefits to organization
(return on investment).
• Can be carried out on a
regular basis (i.e. a progress report) ▪ Of greater interest to managers and
or at the time of a given incident or administrators.
problem.
▪ Of greater interest to instructors
and instructional designers.
Descriptive evaluations: Casual evaluations:
Provide information that describes the Provide information to determine if
trainee once he or she has completed a training caused the post-training
training program. behaviours.
Four Models of Training
Evaluation Kirkpatrick’s
Hierarchical Model:
▪ Donald Kirkpatrick first published his ideas about the evaluation of learning in 1959
in a series of articles he wrote for Training and Development. He later published them in
a 1975 book titled Evaluating Training Programs.
▪ "The Four Levels of Training Evaluation" remains to this day the seminal model
used in the training industry.
, The Four Levels of Training Evaluation:
▪ Trainee perceptions of the training program and/or
Level 1: specific aspects of the course/experience.
Reactions ▪ Methods: the “smiley” sheets , post-training surveys.(used in
lancome)
▪ Trainee acquisition (gained knowledge) of the program
Level 2: material (procedural and declarative knowledge).
Learning ▪ Methods: before and after assessments. (what has the trainee
acquired and learnt during the training)
▪ Extent to which the new knowledge and skills are applied to the
Level 3: job (the extent to which the trainee transfers knowledge and skills
Behaviour acquired in the training).
s ▪ Methods: and interviews (over time), self-reports, supervisor
reports, production records, and evaluation records.
▪ The effect on the business or the work environment by the
trainee. Was the training helpful or DERTERMINANT of the
Level 4:
results the trainee has portrayed?
Results
▪ Methods: Expenses, revenues, sales, and other existing business
measures.
Kirkpatrick’s Hierarchical Model (Cont’d)
▪ Each increase in the level of evaluation also increases the complexity of the
measurements and the costs associated with it. The higher in levels you go, the
more complex the measurement becomes. We need to look beyond a level 1!
• ****Very few of the organizations surveyed carried out high-level
evaluations. Most use Kirkpatrick level one, but only to improve the overall efforts, not
necessarily to improve performance (Bloom, 2003).
▪ In a study by the Conference Board of Canada (Hughes & Grant, 2007) about 75%
of businesses evaluate at Level 1 (trainee reactions), 33% evaluate learning (Level 2),
15% assess changes in behaviour (Level 3), and very few look at results (Level 4).
• Overall evaluation has been declining, likely due to barriers to evaluation.
Kirkpatrick’s Four Levels – Common Critiques
Kirkpatrick’s guidelines are not without critics.
▪ Doubt about the validity.
• No evidence of correlation between the levels.
▪ Insufficiently diagnostic.
Lesson 11: Training
Evaluation What is Training Evaluation?
▪ Training Evaluation: The process to assess the value (the worthiness,
the effectiveness) of training programs to employees and to organizations.
• Involves the analysis of data collected from trainees, supervisors, and through
other stakeholders (trainees) or forms of measurement.
• Objective and subjective information is gathered at different points: before,
during, and after training.
Why Conduct Training Evaluations?
▪ Assist managers in identifying what, and who, should be trained (or needs additional
training).
▪ Determine cost-benefits of a program.
▪ Determine if training program has achieved expected results (?)
▪ Diagnose strengths and weaknesses of a program and pinpoint needed improvements.
• Go back to the design phase of ADDIE and re-evaluate to see if the
improvements worked.
▪ Justify and reinforce the value of training.
Barriers to Training Evaluation
Barriers fall into two categories:
1. Pragmatic (can be time consuming)
- Requires time (loss of production) to organize the collection of the data and
specialized knowledge to properly analyze and interpret it.
2. Political (can be costly, and hard to justify to management, lack of allocating
resources)
- Pressure from administration to justify the costs, potential to reveal ineffectiveness of
training, conflict of interest (may require added resources to get external audit).
Watch this Video
Clip Dr. Robert Haccoun
suggests that “not training” is sometimes the best solution to a
problem.
,Dr. Haccoun, is training the solution to all the
problems? Dr. Haccoun: Yes, to everything. Dr. Devey:
There you go, thank you.
Dr. Haccoun: I think that we have to understand that there are many reasons why people's
performance are on the par. Certainly motivation is one of them, just initial and general
ability, confidence. A large number of issues can be supervisory and help. Lots of things
really do not need to be handled by training but what I think is really important in this matter
is evaluation, is assessing and analyzing why do we have a problem. That is really what is
important and then training becomes one of the possible solutions that we have to this
problem.
I would like to give you an example of how training evaluation, in this particular case of a
problem, turned out to produce something that was quite unexpected but very successful as it
turns out. This particular case is described in the evaluation chapter.
It took place at Bell Canada where they were introducing a new system of electronic
telephones that were more complex; it was as few years back. The way they normally would
have introduced these business telephones, the way they would normally do is by having
employees or people of the company who had bought the system, they would be provided
training by Bell Canada employees, by Bell Canada service advisors. Well, they did that
for this system and it was a complete failure.
I did seven different studies using different combinations of instructors and systems to see
if we can improve the effectiveness of training and we consistently failed. There were no
better ways we could use, with kind of experienced or naive trainers etc...
But we did notice one thing which was that the features that people used a lot they seemed
to know much better than those they rarely used. From this we developed a simple idea –
why don't we simply develop a booklet where the book would explain each of the
features of the telephone and provide the training not about how to use the system but
the training was how to use the booklet. This took half an hour to do. The company, we
tested this and based on the results of that, we found that it was extremely effective. All of a
sudden people were using their telephone instead of calling Bell to say that their system did
not work. They were actually using it properly. As a result of this, this booklet reduced the
training costs by several million dollars a year. They translated it into seventeen different
languages and is now (it was not at the time) packaged in every single telephone that they
sold worldwide.
What started off as a simple training problem was solved using a non-training or
nonconventional training approach and it had direct impact on marketing. Sometimes not
training is actually the best thing to do.
,Types of Training Evaluations
***Evaluations may be distinguished from each other in two fundamental ways:
1. The data gathered and analyzed:
1. Trainee perceptions, learning and behaviour at the conclusion of training.
2. Assessing psychological forces that operate during training.
3. Information about the work environment.
• Transfer climate and learning culture
b. The purpose of the evaluation:
Types of Training Evaluations
Formative evaluations: Summative evaluations:
Provide data about various aspects of a Provide data about the worthiness or
training program. effectiveness of a training program.
▪ Tend to focus on the content ▪ Based on evidence (data) gathered
and how it is presented with the goal at the conclusion of the training
of improving it. intervention (can also be carried out at
mid-point).
▪ Based on feedback received during
the design and development of the • Tend to focus on
learning experience. financial benefits to organization
(return on investment).
• Can be carried out on a
regular basis (i.e. a progress report) ▪ Of greater interest to managers and
or at the time of a given incident or administrators.
problem.
▪ Of greater interest to instructors
and instructional designers.
Descriptive evaluations: Casual evaluations:
Provide information that describes the Provide information to determine if
trainee once he or she has completed a training caused the post-training
training program. behaviours.
Four Models of Training
Evaluation Kirkpatrick’s
Hierarchical Model:
▪ Donald Kirkpatrick first published his ideas about the evaluation of learning in 1959
in a series of articles he wrote for Training and Development. He later published them in
a 1975 book titled Evaluating Training Programs.
▪ "The Four Levels of Training Evaluation" remains to this day the seminal model
used in the training industry.
, The Four Levels of Training Evaluation:
▪ Trainee perceptions of the training program and/or
Level 1: specific aspects of the course/experience.
Reactions ▪ Methods: the “smiley” sheets , post-training surveys.(used in
lancome)
▪ Trainee acquisition (gained knowledge) of the program
Level 2: material (procedural and declarative knowledge).
Learning ▪ Methods: before and after assessments. (what has the trainee
acquired and learnt during the training)
▪ Extent to which the new knowledge and skills are applied to the
Level 3: job (the extent to which the trainee transfers knowledge and skills
Behaviour acquired in the training).
s ▪ Methods: and interviews (over time), self-reports, supervisor
reports, production records, and evaluation records.
▪ The effect on the business or the work environment by the
trainee. Was the training helpful or DERTERMINANT of the
Level 4:
results the trainee has portrayed?
Results
▪ Methods: Expenses, revenues, sales, and other existing business
measures.
Kirkpatrick’s Hierarchical Model (Cont’d)
▪ Each increase in the level of evaluation also increases the complexity of the
measurements and the costs associated with it. The higher in levels you go, the
more complex the measurement becomes. We need to look beyond a level 1!
• ****Very few of the organizations surveyed carried out high-level
evaluations. Most use Kirkpatrick level one, but only to improve the overall efforts, not
necessarily to improve performance (Bloom, 2003).
▪ In a study by the Conference Board of Canada (Hughes & Grant, 2007) about 75%
of businesses evaluate at Level 1 (trainee reactions), 33% evaluate learning (Level 2),
15% assess changes in behaviour (Level 3), and very few look at results (Level 4).
• Overall evaluation has been declining, likely due to barriers to evaluation.
Kirkpatrick’s Four Levels – Common Critiques
Kirkpatrick’s guidelines are not without critics.
▪ Doubt about the validity.
• No evidence of correlation between the levels.
▪ Insufficiently diagnostic.
