Donald
L Kirkpatrick's training evaluation model - the four levels of learning
evaluation
(Fonte
)
Donald L Kirkpatrick
first published his ideas way back in 1959, in a series of articles in the
US Training and Development Journal. the articles were subsequently included
in Kirkpatrick's book Evaluating Training Programs (1975), published by
the American Society for Training and Development (ASTD), with whom Kirkpatrick
still maintains (as at 2005) close connections, having previously served
as president. Donald Kirkpatrick has written several other significant books
about training and evaluation, and has consulted with some of the world's
largest corporations.
Kirkpatrick's book Evaluating
Training Programs defined his originally published ideas of 1959, thereby
further increasing awareness of them, so that his theory has now become
arguably the most widely used and popular model for the evaluation of training
and learning. Kirkpatrick's four-level model is now considered an industry
standard across the HR and training communities. The four levels of training
evaluation model was later redefined and updated in Kirkpatrick's 1998 book,
called 'Evaluating Training Programs: The Four Levels'.
The four levels of Kirkpatrick's
evaluation model essentially measure:
reaction of student
- what they thought and felt about the training
learning - the
resulting increase in knowledge or capability
behaviour - extent
of behaviour and capability improvement and implementation/application
results - the
effects on the business or environment resulting from the trainee's performance
All these measures are
recommended for full and meaningful evaluation of learning in organizations,
although their application broadly increases in complexity, and usually
cost, through the levels from level 1-4.
kirkpatrick's
four levels of training evaluation
This
grid illustrates the basic Kirkpatrick structure at a glance. The second
grid, beneath this one, is the same thing with more detail.
level
evaluation
type (what is measured)
evaluation
description and characteristics
examples
of evaluation tools and methods
relevance
and practicability
1
reaction
reaction
evaluation is how the delegates felt about the training
or learning experience
eg., 'happy
sheets', feedback forms
also verbal
reaction, post-training surveys or questionnaires
quick and very
easy to obtain
not expensive
to gather or to analyse
2
learning
learning
evaluation is the measurement of the increase in knowledge
- before and after
typically assessments
or tests before and after the training
interview or
observation can also be used
relatively
simple to set up; clear-cut for quantifiable skills
less easy for
complex learning
3
behaviour
behaviour
evaluation is the extent of applied learning back on
the job - implementation
observation
and interview over time are required to assess change, relevance
of change, and sustainability of change
measurement
of behaviour change typically requires cooperation and skill of
line-managers
4
results
results
evaluation is the effect on the business or environment
by the trainee
measures are
already in place via normal management systems and reporting - the
challenge is to relate to the trainee
individually
not difficult; unlike whole organisation
process must
attributing clear accountabilities
kirkpatrick's
four levels of training evaluation in detail
This
grid illustrates the Kirkpatrick's structure detail, and particularly the
modern-day interpretation of the Kirkpatrick learning evaluation model,
usage, implications, and examples of tools and methods. This diagram is
the same format as the one above but with more detail and explanation:
level
evaluation
type (what is measured)
evaluation
description and characteristics
examples
of evaluation tools and methods
relevance
and practicability
1
reaction
reaction
evaluation is how the delegates felt, and their personal
reactions to the training or learning experience, for example:
did the trainees
like and enjoy the training?
did they consider
the training relevant?
was it a good
use of their time?
did they like
the venue, the style, timing, domestics, etc?
level of participation
ease and comfort
of experience
level of effort
required to make the most of the learning
perceived practicability
and potential for applying the learning
typically 'happy
sheets'
feedback forms
based on subjective personal reaction to the training experience
verbal reaction
which can be noted and analysed
post-training
surveys or questionnaires
online evaluation
or grading by delegates
subsequent
verbal or written reports given by delegates to managers back at
their jobs
can be done
immediately the training ends
very easy to
obtain reaction feedback
feedback is
not expensive to gather or to analyse for groups
important to
know that people were not upset or disappointed
important that
people give a positive impression when relating their experience
to others who might be deciding whether to experience same
2
learning
learning
evaluation is the measurement of the increase in knowledge
or intellectual capability from before to after the learning
experience:
did the trainees
learn what what intended to be taught?
did the trainee
experience what was intended for them to experience?
what is the
extent of advancement or change in the trainees after the training,
in the direction or area that was intended?
typically assessments
or tests before and after the training
interview or
observation can be used before and after although this is time-consuming
and can be inconsistent
methods of
assessment need to be closely related to the aims of the learning
measurement
and analysis is possible and easy on a group scale
reliable, clear
scoring and measurements need to be established, so as to limit
the risk of inconsistent assessment
hard-copy,
electronic, online or interview style assessments are all possible
relatively
simple to set up, but more investment and thought required than
reaction evaluation
highly relevant
and clear-cut for certain training such as quantifiable or technical
skills
less easy for
more complex learning such as attitudinal development, which is
famously difficult to assess
cost escalates
if systems are poorly designed, which increases work required to
measure and analyse
3
behaviour
behaviour
evaluation is the extent to which the trainees applied the
learning and changed their behaviour, and this can be
immediately and several months after the training, depending on
the situation:
did the trainees
put their learning into effect when back on the job?
were the relevant
skills and knowledge used
was there noticeable
and measurable change in the activity and performance of the trainees
when back in their roles?
was the change
in behaviour and new level of knowledge sustained?
would the trainee
be able to transfer their learning to another person?
is the trainee
aware of their change in behaviour, knowledge, skill level?
observation
and interview over time are required to assess change, relevance
of change, and sustainability of change
arbitrary snapshot
assessments are not reliable because people change in different
ways at different times
assessments
need to be subtle and ongoing, and then transferred to a suitable
analysis tool
assessments
need to be designed to reduce subjective judgement of the observer
or interviewer, which is a variable factor that can affect reliability
and consistency of measurements
the opinion
of the trainee, which is a relevant indicator, is also subjective
and unreliable, and so needs to be measured in a consistent defined
way
360-degree
feedback is useful method and need not be used before training,
because respondents can make a judgement as to change after training,
and this can be analysed for groups of respondents and trainees
assessments
can be designed around relevant performance scenarios, and specific
key performance indicators or criteria
online and
electronic assessments are more difficult to incorporate - assessments
tend to be more successful when integrated within existing management
and coaching protocols
self-assessment
can be useful, using carefully designed criteria and measurements
measurement
of behaviour change is less easy to quantify and interpret than
reaction and learning evaluation
simple quick
response systems unlikely to be adequate
cooperation
and skill of observers, typically line-managers, are important factors,
and difficult to control
management
and analysis of ongoing subtle assessments are difficult, and virtually
impossible without a well-designed system from the beginning
evaluation
of implementation and application is an extremely important assessment
- there is little point in a good reaction and good increase in
capability if nothing changes back in the job, therefore evaluation
in this area is vital, albeit challenging
behaviour change
evaluation is possible given good support and involvement from line
managers or trainees, so it is helpful to involve them from the
start, and to identify benefits for them, which links to the level
4 evaluation below
4
results
results
evaluation is the effect on the business or environment
resulting from the improved performance of the trainee - it is the
acid test
measures would
typically be business or organisational key performance indicators,
such as:
volumes, values,
percentages, timescales, return on investment, and other quantifiable
aspects of organisational performance, for instance; numbers of
complaints, staff turnover, attrition, failures, wastage, non-compliance,
quality ratings, achievement of standards and accreditations, growth,
retention, etc.
it is possible
that many of these measures are already in place via normal management
systems and reporting
the challenge
is to identify which and how relate to to the trainee's input and
influence
therefore it
is important to identify and agree accountability and relevance
with the trainee at the start of the training, so they understand
what is to be measured
this process
overlays normal good management practice - it simply needs linking
to the training input
failure to
link to training input type and timing will greatly reduce the ease
by which results can be attributed to the training
for senior
people particularly, annual appraisals and ongoing agreement of
key business objectives are integral to measuring business results
derived from training
individually,
results evaluation is not particularly difficult; across an entire
organisation it becomes very much more challenging, not least because
of the reliance on line-management, and the frequency and scale
of changing structures, responsibilities and roles, which complicates
the process of attributing clear accountability
also, external
factors greatly affect organisational and business performance,
which cloud the true cause of good or poor results
Since Kirkpatrick established
his original model, other theorists (for example Jack Phillips), and indeed
Kirkpatrick himself, have referred to a possible fifth level, namely ROI
(Return On Investment). In my view ROI can easily be included in Kirkpatrick's
original fourth level 'Results'. The inclusion and relevance of a fifth
level is therefore arguably only relevant if the assessment of Return On
Investment might otherwise be ignored or forgotten when referring simply
to the 'Results' level.
Learning evaluation
is a widely researched area. This is understandable since the subject is
fundamental to the existence and performance of education around the world,
not least universities, which of course contain most of the researchers
and writers.
While Kirkpatrick's
model is not the only one of its type, for most industrial and commercial
applications it suffices; indeed most organisations would be absolutely
thrilled if their training and learning evaluation, and thereby their ongoing
people-development, were planned and managed according to Kirkpatrick's
model.
For reference, should
you be keen to look at more ideas, there are many to choose from...
Jack Phillips' Five
Level ROI Model
Daniel Stufflebeam's
CIPP Model (Context, Input, Process, Product)
If you
are responsible for HR functions and services to internal and/or external
customers, you might find it useful to go beyond Kirkpatrick's evaluation
of training and learning, and to evaluate also satisfaction among
staff/customers with HR department's overall performance. The parameters
for such an evaluation ultimately depend on what your HR function is responsible
for - in other words, evaluate according to expectations.
Like anything
else, evaluating customer satisfaction must first begin with a clear appreciation
of (internal) customers' expectations. Expectations - agreed, stated, published
or otherwise - provide the basis for evaluating all types of customer satisfaction.
If people
have expectations which go beyond HR department's stated and actual responsibilities,
then the matter must be pursued because it will almost certainly offer an
opportunity to add value to HR's activities, and to add value and competitive
advantage to your organisation as a whole. In this fast changing world,
HR is increasingly the department which is most likely to see and respond
to new opportunities for the support and development of the your people
- so respond, understand, and do what you can to meet new demands when you
see them.
If you
are keen to know how well HR department is meeting people's expectations,
a questionnaire, and/or some group discussions will shed light on the situation.
Here are
some example questions. Effectively you should be asking people to say how
well HR or HRD department has done the following:
helped me to identify,
understand, identify and prioritise my personal
development needs and wishes, in terms of: skills,
knowledge, experience and attitude (or personal
well-being, or emotional maturity, or mood, or mind-set, or any other
suitable term meaning mental approach, which people will respond to)
helped me to understand
my own preferred learning style and learning methods for
acquiring new skills, knowledge and attitudinal
capabilities
helped me to identify
and obtaineffective learning and development that
suits my preferred style and circumstances
helped me to measure
my development, and for the measurement to be clear to my boss
and others in the organisation who should know about my capabilities
provided tools
and systems to encourage and facilitate my personal development
and particularly
helped to optimise the relationship between me and my
boss relating to assisting my own personal development and well-being
provided a working
environment that protects me from discrimination and harassment
of any sort
provided the opportunity
for me to voice my grievances if I have any, (in private, to
a suitably trained person in the company whom I trust) and then if I
so wish for proper consideration and response to be given to them by
the company
provided the opportunity
for me to receive counselling and advice in the event that I
need private and supportive help of this type, again from a suitably
trained person in the company whom I trust
ensured that disciplinary
processes are clear and fair, and include the right of appeal
ensured that recruitment
and promotion of staff are managed fairly and transparently
ensuring that systems
and activities exist to keep all staff informed of company plans,
performance, etc., (as normally included in a Team Briefing system)
(if you dare...)
ensuring that people are paid and rewarded fairly in relation
to other company employees, and separately, paid and rewarded fairly
when compared to market norms (your CEO will not like this question,
but if you have a problem in this area it's best to know about it...)
(and for managers)
helped me to ensure the development needs of my staff
are identified and supported
This is
not an exhaustive list - just some examples. Many of the examples contain
elements which should under typical large company circumstances be broken
down to create more and smaller questions about more specific aspects of
HR support and services.
If you
work in HR, or run an HR department, and consider that some of these issues
and expectations fall outside your remit, then consider who else is responsible
for them.
I repeat,
in this fast changing world, HR is increasingly the department which is
most likely to see and respond to new opportunities for the support and
development of the your people - so respond, understand, and do what you
can to meet new demands when you see them. In doing so you will add value
to your people and your organisation - and your department.