Effective Training: Strategies,

Systems and Practices, 3rd Edition

P. Nick Blanchard and James W. Thacker

Evaluation of Training

Chapter Nine
Purpose of Training Evaluation
 Determine if accomplished objectives
 Identify strengths and weaknesses
 Cost-benefit analysis
 Who should participate and who
benefited most
 Determine if program appropriate
 Database for decision making
Chapter 8 2
Evaluation Phase
Input Process Output
Process
Evaluation Measures
Evaluation Strategy
Objectives and
Design
Outcome
Organizational Measures
Constraints - Reaction
- Learning
Design Issues - Behavior
- Results

Chapter 8 3
Kirkpatrick’s Levels of Criteria
 Reaction – did trainees like the program
 Learning – demonstration of learning at
end of program
 Behavior – actual transfer to job
 Results – impact on bottom line
including efficiency, productivity, cost,
etc.

Chapter 8 4
Data Collection Methods
 Interview
 Questionnaire
 Direct observation
 Tests and simulations
 Archival performance data
 Types of data – individual, group,
system wide
Chapter 8 5
 Potential Questions to Be Addressed
in a Process Analysis   
(Before Training) – Part 1 of 2
Were needs diagnosed correctly?

  • What data sources were used?

  • Was a knowledge/skill deficiency identified?
  • Were trainees assessed to determine their prerequisite KSAs?

Were needs translated into training objectives?

  • Were all objectives identified?

  • Were the objectives written in a clear, appropriate manner?

Was an evaluation system designed to measure objectives?

Chapter 8 6
 Potential Questions to Be Addressed
in a Process Analysis 
(Before Training) – Part 2 of 2
Was the training program designed to meet all the training
objectives?
  • Was previous learning that supports or inhibits training identified?
  • Were individual differences assessed/factored into training design?
  • Was trainee motivation to learn assessed and addressed?
  • Were processes built into training to facilitate recall and transfer?
  • Were steps included to call attention to key learning events?
  • Were aids for symbolic coding and cognitive organization designed?
• Were opportunities included for symbolic and behavioral practice?
  • Were steps included to ensure transfer of learning to the job?

Are the training methods appropriate for the learning objectives?

Chapter 8 7
 Potential Questions to Be Addressed
in a Process Analysis
(During Training)
Was there a match between trainer, training techniques, and
training/learning objectives?
  • Were lecture portions of the training effective?
Was involvement encouraged/solicited?
  Were questions used effectively?
 • Did the trainer appropriately conduct the various training methodologies
(case study, role play, etc.)?
Were they explained well?
      Did the trainer use the allotted time for activities?       
Was enough time allotted?
      Did trainees follow instructions
Was there effective debriefing following exercises?
  • Did the trainer follow the training design and lesson plans?
Was enough time given for each of the requirements?
Was time allowed for questions?
Chapter 8 8
 Who Is Interested in the Process Data

Training Department

  Trainer: Yes, to determine what does and does not work well.
 Other trainers: To the extent that process generalizes to other
training.
  Training manager: Only if training is not successful, or if there is a
problem with a particular trainer.

Customers
Trainees: No
Trainees’ supervisor: No
Upper management: No

Chapter 8 9
 Reaction Questionnaire for the
Trainer – Part 1 of 2
Please circle the number that reflects the degree to which you agree or disagree with the
following statements.
1= Strongly disagree, 2= Disagree, 3= Neither agree nor disagree 4= Agree 5 =Strongly agree

1. The trainer did a good job of stating the objectives

at the beginning of training ……….………..
…………………..1 2 3 4 5

2. The trainer made good use of visual aids (easel, white

board) when making the presentations
……………....………1 2 3 4 5

3. The trainer was good at keeping everyone interested

in the topics ……………………………………………..…….. 1 2 3 4 5

4. The trainer encouraged questions and participation

from trainees …………………………….………………1
Chapter 8 2 3 4 510
 Reaction Questionnaire for the
Trainer – Part 2 of 2

5. The trainer made sure everyone understood the concepts

before moving on to the next topic…………………………..1 2 3 4 5

6. The trainer summarized important concepts before

moving to the next module……………………………………1 2 3 4 5

7. Overall, how would you rate this trainer (check one)

____1. Poor; I would not recommend this trainer to others.
____2. Adequate; I would recommend this trainer only if no others were available.
____3. Average
____4. Good; I would recommend this trainer above most others.
____5. Excellent; this trainer is among the very best I’ve ever worked with.

Chapter 8 11
 Steps to Consider in Developing a
Reaction Questionnaire – Part 1 of 2

1.  Determine what you want to find out (consider training objectives).

2.  Develop a written set of questions to obtain the information.

3.  Develop a scale to quantify respondents’ data.

4.  Make forms anonymous so participants will feel free to respond honestly.

5.  Ask for information useful in determining differences in reactions by

subgroups (e.g., young vs. old; minority vs. non-minority). This could
be valuable in determining effectiveness of training for different
cultures, for example, which may be lost in an overall assessment.

Chapter 8 12
 Steps to Consider in Developing a
Reaction Questionnaire – Part 2 of 2

6. Allow space for “Additional Comments” in order to allow participants the

opportunity to mention things you might not have considered.

a. If right after training, have someone other than the instructor

administer and pick up the information.

b. If some time later, develop a mechanism for obtaining a high response

rate (e.g., encourage the supervisor to allow trainees to complete
the questionnaire on company time).

Chapter 8 13
Course Reaction Questionnaire
 Develop a questionnaire to measure
student reactions to COBA courses
 About 10 questions
 What would you want to know to help
you decide what courses and instructors
you would want to take?

Chapter 8 14
 Procedures for Developing a Multiple
Choice Test
1. Examine objectives to determine the content area you wish to test.

2. Write the questions in a clear manner. Shorter is better.

3. Choose alternatives to the correct response from typical errors

made during training. Make alternatives realistic.

4. Avoid making the correct response always longer than incorrect

responses.

5. Four alternatives are usually enough.

6. Pretest items. Ask for feedback on clarity. Note questions that most
get wrong.

Chapter 8 15
 Example of an Attitudinal Measure

Attitudes Toward Empowerment

Please indicate the degree to which you agree or disagree with the following statements.
1 = Strongly disagree 2 = Disagree 3 = Neither agree nor disagree
4 = Agree 5 = Strongly agree
1. Empowering employees is just another way to get more
work done with fewer people. (reverse scored)…………..1 2 3 4 5
2. Empowering of employees allows everyone to
contribute their ideas to the betterment of the company….1 2 3 4 5
3. The empowerment program has improved relationship
with my supervisor………………………………………………1 2 3 4 5
4. Empowerment has brought more meaning to my life at
this company……………………………………………………..1 2 3 4 5
5. Empowerment interventions should be introduced in other
plants in this company…………………………………………..1 2 3 4 5
6. The empowerment process has been a positive influence in
labor-management relations…………………………………….1 2 3 4 5

Chapter 8 16
 Guidelines for Writing Effective
Questionnaires – Part 1 of 4
1.  Write simply and clearly, and make the meaning obvious.

Bad:  To what extent do supervisors provide information regarding the

quality of performance of people at your level?
Good: How often does your boss give you feedback on your job?

2.  Ask one question at a time.

Bad:  Both the organization’s goals and my role within the
organization are clear.
Good: The organization’s goals are clear.
My role within the organization is clear.

Chapter 8 17
 Guidelines for Writing Effective
Questionnaires – Part 2 of 4
3.  Provide discrete response options.

Bad:  During the past three months how often did you receive
feedback on your work?

1 2 3 4 5
rarely occasionally frequently

Good:  During the past three months how often did you receive
feedback on your work?

1 2 3 4
5
not once 1–3 times about once 2-4 times once a
a month a Chapter
week 8 a week day or more
18
 Guidelines for Writing Effective
Questionnaires – Part 3 of 4
4.  Limit the number of response options.
Bad:  What percent of the time are you sure of what your
compensation will be?
1 2 3 4 5 6 7 8 9 10
0–10% 11–20% 21–30% 31–40% 41–50% 51–60% 61–70% 71–80% 81–90% 91–100%

Good:  What percent of the time are you sure of what your
compensation will be?
1 2 3 4 5
0–20% 21–40% 41–60% 61–80% 81–100%

Chapter 8 19
 Guidelines for Writing Effective
Questionnaires – Part 4 of 4

5.  Match the response mode to the question.

Bad:  To what extent are you satisfied with your job?
1 2 3 4 5
strongly disagree agree strongly
disagree agree

Good:  To what extent are you satisfied with your job?

1 2 3 4 5
not at all a little bit some quite a lot very much

Chapter 8 20
 Example of 360-Degree Feedback
Results
Skills being Very Low Average High Very
Assessed Low High
Listening Skills S P, B E
Managing Conflict S P, B E
Organizing S, P B, E
Written Communication S, P B E
S = Subordinates
P = Peers
B = Supervisor (Boss)
E = Employee being assessed

Chapter 8 21
 Scripted Situation Item for Evaluation
of a School Superintendent
The following is a scenario about a school superintendent. Read the scenario
and place an X next to the behavior you believe your superintendent would
follow.
The administrator receives a letter from a parent objecting to the content of
the science section on reproduction. The parent strongly objects to his
daughter having exposure to such materials and demands something be
done. The effective administrator would most likely: (check one)
____  Ask the teacher to provide handouts, materials, and curriculum
content for review.
____  Check the science curriculum for the board-approved approach to
reproduction, and compare board guidelines to course content.
____  Ask the head of the science department for his or her opinion about
the teacher’s lesson plan.
____  Check to see if the parent has made similar complaints in the past.
Chapter 8 22
 Types of Outcomes and Examples of
Factors Affecting those Outcomes
External environment of the organization:
Organizational Economy, regulations, suppliers, etc
Results Internal environment of the organization:
Policies, procedures, systems
Employee performance, KSAs, and needs

Job Behavior Transfer of training

Motivational forces in the job setting
Opportunity to applying training on the job
KSAs

Learning Trainee readiness for the course

Trainee motivation to learn
Design, materials, and content
Trainer(s) behaviors
Reactions Perceived match between trainee
Exceptions and what training provided
Chapter 8 23
 Cost Savings for Grievance Reduction
Training
Pretraining Post training
Management time
(for those going to
10 hrs. X 63 grievances = 630 hrs. 10 hrs. X 8 grievances = 80 hrs.
3rd step) 10 hours
per grievance
Union Reps time
(paid by 7.5 X 63 grievances = 472 ½ hrs 7.5 X 8 grievances = 60 hrs.
management) 7.5
hrs per grievance
Total Cost
Management time 630 hrs X $50 per hr. = $31,500.00 80 hrs. X $50 per hr. = $4,000.00
Union reps time 472 ½ hrs X $25 per hr.= $11,812.50 60 hrs. X $25 per hr. = $1,500.00
Total $43,312.50 $5,500.00
Reduction in cost of grievances going to the third step 43,312.50 – 5,500.00 = $37,812.50
Cost of training $32,310.00

Cost saving for the first year $ 5,502.50

Chapter 8 24
 Calculation of the Utility of the
Grievance Training – Part 1 of 2

Formula: U = (N)(T)(DT)(SDY) - C

N = 30
T = 1 year (This is probably an overly conservative
estimate)
DT = .2 DT = Xt – Xu
SD (r yy)

Xt = average job performance of the trained supervisors

Xu = average job performance of the untrained supervisors
SD =standard deviation of job performance for the untrained
supervisors
r yy = reliability of job performance measure

Chapter 8 25
 Who Is Interested in the Outcome
Data
Outcome Data
Reaction Learning Behavior Results
Training Department
Trainer Yes Yes Yes No
Other Trainers Perhaps Perhaps Perhaps No
Training Manager Yes Yes Yes Yes
Customers
Trainees Yes Yes Yes Perhaps
Trainees’ Supervisor Not really Only if no Yes Yes
transfer
Upper Management No No Perhaps Yes

Chapter 8 26
 Reaction to Training – Part 1 of 2

Answer the following questions about the training in Active Listening skills using the
scale below:
1= Strongly disagree 2= Disagree 3= Neither agree nor disagree 4= Agree
5=Strongly agree
  
1. The training met the stated objectives 1 2 3 4 5
 
2. The information provided was enough so I
understood the concepts being taught 1 2 3 4 5
 
3. The practice sessions provided were sufficient 1 2 3 4 5
to give me an idea of how to perform the skill
 
4. The feedback provided was useful in helping 1 2 3 4 5
me understand how to improve. 

Chapter 8 27
 Reaction to Training – Part 2 of 2
5. The training session kept my interest throughout 1 2 3 4 5

6. The pace of the Active Listening session was

1. Way too fast
2.   A bit fast
3.   Just right
4.   A bit slow
5. Way too slow
 
7. What did you like best about this part of the training
  
8. What would you have changed
 
Additional Comments:
  
Note: A similar scale would be used for each of the other components of training
that were taught.
Chapter 8 28
 Calculation of the Utility of the
Grievance Training – Part 2 of 2
DT is a measure of the improvement (in standard deviation units) in
performance that trained supervisors will exhibit.

SDY = $14,000 This is based on: .40 X $35,000 = $14,000

The above assumes average salary of 35,000 dollars. The .40 comes
from the 40% rule, which is a calculation based on 40% of the average
salary of trainees.

So based on the above information, the utility of the training based on

this formula is:

30 X 1 X .2 X 14,000 - 32,020 = $51,980

Chapter 8 29
 Paper & Pencil Test for Evaluation of
Learning
Evaluation of learning
 
There is no specific time limit on this test, but most should be able to finish in
about one hour. Answers to the questions should be written in the
booklet provided.
 
Please read each question carefully as some of the questions have more
than one part to them.

1. List four types of active listening and provide an example for each.

2. List the steps in the Conflict resolution model. After each step, provide a
relevant example of a phrase that would represent that step.
3. And so forth…..
 

Chapter 8 30
 Training Evaluation Documents for
Active Listening – Part 1 of 8
Scenarios for evaluating learning of Active Listening Skills
Part 1 Initiators Role
 
Role Scenarios to Test Trainees (The initiator to be played by the same actor for
all trainees)
 
Test Scenario 1
You have just been asked by your supervisor
(the trainee) to serve on the same committee
again. You are angry that they always ask you.
You Start. Say Angrily:
"OH NO, YOU DON'T. I'VE BEEN ON THAT
COMMITTEE THREE YEARS IN A ROW AND
IT TAKES UP TOO MUCH TIME".
 

Chapter 8 31
 Training Evaluation Documents for
Active Listening – Part 2 of 8
Part 1 Initiators Role
 
Test Scenario 2
Your supervisor has just talked to you about
why you don’t follow procedures.
You say angrily: "WHY ARE YOU PICKING ON ME ALL THE
TIME. I'M NOT THE ONLY ONE WHO
DOESN'T FOLLOW THESE STUPID
PROCEDURES".
  
You have just been asked by a your supervisor
second time today whether you will be attending
the weekly meeting.
You say Angrily: "I ALREADY TOLD YOU, I CAN'T ATTEND
THE WEEKLY MEETING BECAUSE I HAVE
TO COMPLETE THE STAFF REPORTS FOR
And so forth (for a total of 5) TOMORROW".
 
Chapter 8 32
 Training Evaluation Documents for
Active Listening – Part 3 of 8

Scenarios for evaluating learning of Active Listening Skills

Part 2
Trainees instructions
 
This test will require you to respond to 5 different short scenarios where you
are a supervisor, and you say something to a subordinate that elicits an angry
response. You will be expected to respond using the skills of active listening.
Below is a description of each of the scenarios and what you have initially said
to the subordinate. When you are ready for each of the scenarios to begin nod
your head to the initiator. At that time the initiator will say something. You need
to respond to the comment, and when complete, explain to the evaluator the
rationale for your response.
 

Chapter 8 33
 Training Evaluation Documents for
Active Listening – Part 4 of 8
Scenario 1
You have asked a subordinate to continue working on a particular committee
for another year.
Listen then respond using active listening. Nod your head when ready….
 
Scenario 2
You have just talked to a subordinate regarding the importance of following
procedures.
Listen then respond using active listening. Nod your head when ready….
 
Scenario 3
Today is the day of your weekly meeting. You asked your subordinate if
he/she would be attending the meeting, they said no. It is now time for the
meeting and you call once more to check if they can make the meeting.

Listen then respond using active listening. Nod your head when ready….
Chapter 8 34
 Training Evaluation Documents for
Active Listening – Part 5 of 8

Part 3: General guidelines for evaluating active listening

scenarios

Trainee fails the scenario if his/her response is focused on the issue instead of
reflecting back what the initiator says.
 
Passing responses reflect back what the person is saying without being a
mimic of what was said.

Examples of  acceptable and poor responses are given in the following slides.

Chapter 8 35
 Training Evaluation Documents for
Active Listening – Part 6 of 8
Scenario 1

The supervisor (trainee being tested) asked the subordinate to

continue working on a particular committee for another year. Listen to
response and grade according to guidelines
 
Unacceptable response: “I am willing to talk about reducing
the work you have to do if you will
be on it”

Acceptable response "YOU DON'T WANT TO BE ON THAT

COMMITTEE AGAIN BECAUSE IT INTERFERES
WITH YOUR WORK AND YOU HAVE DONE
YOUR SHARE."

Chapter 8 36
 Training Evaluation Documents for
Active Listening – Part 7 of 8

Scenario 2

The supervisor (trainee being tested) has just talked to a subordinate

regarding the importance of following procedures. Listen to response
and grade according to guidelines.

Unacceptable response: “You are not the only one I have

talked to about this”

Acceptable response "YOU BELIEVE THAT YOU'RE THE ONLY

ONE THAT I AM SINGLING OUT FOR NOT
FOLLOWING PROCEDURES".  

Chapter 8 37
 Training Evaluation Documents for
Active Listening – Part 8 of 8
Scenario 3

The supervisor (trainee being tested) called first thing in the AM, and
asks the subordinate if he/she would be attending the weekly meeting
and the subordinate said no they were busy. The supervisor has just
called again at meeting time to check if they could make the meeting.
Listen and grade.

Unacceptable response: “The meeting will only be

an hour”.

Acceptable response: "YOU'RE NOT ABLE TO ATTEND THE

MEETING BECAUSE YOU ARE FINISHING
STAFF REPORTS THAT ARE DUE
TOMORROW".
Chapter 8 38
 Multiple Techniques
39

Reactive to Evaluation
External
Sources of Invalidity – Part 1 of 2

Selection
-

Testing
Interaction of Selection and Maturation, etc.
Mortality

-
Selection

-
Regression

Chapter 8
Instrumentation

-
Testing
Internal

Maturation

-
History

-
Posttest Only (no control group)
 Sources of Invalidity – Part 2 of 2

Note: In the tables, a minus indicates a definite weakness, a plus

indicates that the factor is controlled, a question mark indicates a
possible source of concern, and a blank indicates that the factor is
not relevant.

It is with extreme reluctance that these summary tables are

presented because they are apt to be “too helpful” and to be
depended upon in place of the more complex analysis. No + or -
indicator should be respected unless the reader comprehends why it
is placed there. In particular, it is against the spirit of this
presentation to create fears of, or confidence in, specific designs
without an clear understanding of its strengths and weaknesses for
a particular purpose.
Chapter 8 40
Research Design
 Internal Validity
 Did a change occur?
 Was the change a result of the training?

 External Validity
 Will the change occur in other situations
with different trainees?

Chapter 8 41
Threats to Internal Validity
 History – events occurring during
training
 Maturation – natural improvements with
development
 Testing – effects on pre-tests on
changes
 Instrumentation – different measures at
different point in time

Chapter 8 42
Threats to Internal Validity
(continued)
 Statistical regression – select trainees
measured at extremes in abilities/KSAs
and regress to mean
 Reactive effects of research situation –
motivation (Hawthorne effect)
 Multiple treatment effects – previous
training

Chapter 8 43
Experimental Designs
 Control group – random assignment to
treatment and control groups so
trainees have similar characteristics
 Two-group posttest only (used most
often)
 Two-group pretest/posttest
 Four group design – control for effects
of pretest and prior knowledge

Chapter 8 44
Non-experimental Designs
 Case study – intensive, descriptive
study with after only measures
 One group pretest/posttest design

Chapter 8 45
Quasi-experimental Designs
 Nonequivalent control group design –
analyze for equivalence or use multiple
regression to control for demographic
factors
 Time series – establish base line, then
training, then series of measures to
determine if change has occurred

Chapter 8 46
Ethical Issues Concerning
Evaluation Research
 Confidentiality
 Informed consent
 Withholding training – can provide later
 Use of deception
 Pressure to produce positive results

Chapter 8 47
Assessing the Impact of HRD
Programs
 Cost-benefit – monetary costs in
relation to nonmonetary benefits
 Cost-effectiveness – monetary costs in
relation to monetary benefits
 Return on investment = results/costs

Chapter 8 48