1. The Kirkpatrick’s Four Level Approach:
In order to classify areas of evaluation, the first one would be Kirkpatrick Four Levels of Evaluation. Donald Kirkpatrick, Professor Emeritus at the University of Wisconsin and past president of the American Society for Training and Development (ASTD), first published his Four Level Training Evaluation Model in 1959, in the US Training and Development Journal.
Kirkpatrick model is now nearly 59 years old. Its elegant simplicity has caused it to be the most widely used methods of evaluation training programs. Almost organizations that conduct evaluations use the Kirkpatrick model.
Level 4: Results Did the change in behavior positively affect the organization?
Level 3: Behavior Did the participants change their behavior based on what was learned in the program?
Level 2: Learning What skills, knowledge, or attitudes have change? By how much?
Level 1: Reaction Were the participants pleased? What do they plan to do with what they learned?
The first level of evaluation (the reaction level) can provide invaluable data on problems that have arisen during the training program and, sometimes, an insight into the causes if the program is less than fully effective. The reaction and learning levels are relatively easy to organize but they do not provide any significant indicators of the final test of a training program. He defined Knowledge as “I know it”, Skill as “I can do it right now” and Attitude as “I believe this will be worthwhile to do on the job” (Kirpatric P, 2017).
Accordingly, Kirkpatrick inserts two further levels. The performance level tries to measure job performance through a range of evaluation tools over a period of time. Closely allied to this is the results level that seeks to measure the effect that the training program on the overall performance of the organization. The power of the Kirkpatrick model, therefore, lies in its potential as a diagnostic tool in monitoring progress in overall reform objectives.
Critical Appreciation of Model Kirkpatrick's Four-Level Training Evaluation Model is popular and widely used; there are a number of considerations that need to be taken into account when using the model. One issue is that it can be time-consuming and expensive to use levels 3 or 4 of the model, so it's not practical for all organizations and situations. This is especially the case for organizations that don't have a dedicated training or human resource department. The model also assumes that each level's importance is greater than the last level, and that all levels are linked. For instance, it implies that Reaction is less important, ultimately, than Results, and that reactions must be positive for learning to take place. In practice, this may not be the case. Most importantly, organizations change in many ways, and behaviors and results change depending on these, as well as on training. For example, measurable improvements in areas like retention and productivity could result from the arrival of a new Team Lead or from a new computer system, rather than from training. Kirkpatrick's model evaluate training in a "scientific" way, however, so many variables can be changing in dynamic organizations that analysis at level 4 can be limited in usefulness.
2. Kaufman’s Five Level of Evaluation approach:
It is named as “Kirkpatrick Plus”. Some researchers, recognizing some shortcomings of Kirkpatrick’s four level approaches, have attempted to modify and add to this basic framework. It is articulated by Kaufman, Keller, and Watkins in 1995, this evaluation framework connects performance to expectations.
5. Societal Outcomes Societal and client responsiveness, consequences and payoffs.
4. Organizational Output Organizational contributions and payoffs.
3. Application Individual and small group (product) utilization within the organization
2. Acquisition Individual and small group mastery and competency
1b. Reaction Methods’, means’ and processes’ acceptability and efficiency
1a. Enabling Availability and quality of human, financial, and physical resources input
Level 1: Resources and processes; Level 1 is divided into two levels, 1a and 1b.
Level 1a focuses the evaluation lens on inputs, such as the availability and quality of materials needed to support a learning effort.
Level 1b considers processes. What’s their quality? Are they efficient? Are learners satisfied with them?
Compared to Kirkpatrick’s Level 1 (Reaction), Kaufman’s Level 1 focuses not only on learner satisfaction, but on the organizational factors that can impact learner satisfaction.
Level 2: Acquisition; this level is focused on individual and small group payoffs, called “micro” benefits.
Are the objectives or desired outcomes of the learning intervention met?
It’s pretty analogous to Kirkpatrick’s Level 2 evaluation (Learning), but Kaufman notes that the learning intervention may not necessarily be training.
Level 3: Application; it is a micro analysis, examining individual and small group impacts. The relevant inquiry here is whether newly acquired knowledge and skills are being applied on the job.
Level 3 also is quite similar to Kirkpatrick’s Level 3 Behavior/ Performance). Level 4: Organizational payoffs; here, the analysis examines macro benefits. What are the benefits from an organizational standpoint?
Level 4 is analogous to Kirkpatrick’s Level 4 (Results).
Level 5: Societal contributions; Kaufman considers this a mega analysis. How is the organization contributing to its clients and society? Is it responsive to client/societal needs? This moves evaluation beyond the organization, and examines the extent to which the performance improvement program has enhanced society and environment surrounding the organization (Kaufman, Keller & Walkins, 1995). Comparison with Kirkpatrick Model The “Kirkpatrick Plus” framework doesn’t stray that far from Kirkpatrick’s Evaluation Model. While measuring organizational payoff’s an important part of an evaluation, it will be not be judicious to consider only effects of a learning intervention from all the other variables that impact ROI. The organization’s commitment to success, by providing necessary resources, processes, and other supports should be subject to as much scrutiny as the learner’s performance. Issues of health, continued profits, pollution, safety, and well-being are central. The basis for mega level concerns is an ideal vision, which is a measurable statement of the kind of world required for the health, safety, and well being of sustainable world. Level 5 has no analog in Kirkpatrick’s Evaluation Model.
3. CIRO (Context, Input, Reaction, Outcome) Approach:
The CIRO four level approaches was developed by Warr, Bird and Rackham (1970).Adopting the CIRO approach to evaluation gives Organization a model to follow when conducting training evaluation.
1. C- Context or environment within which the training took place
2. I -Inputs to the training event
3. R- Reactions to the training event
4. O- Outcomes As the name suggests, a context evaluation seeks to measure the context within which a training program takes place. It scrutinizes the way performance needs were identified, learning objectives were established, and the way the objectives link to and support the necessary competencies.
Training objectives may be at three levels: The ultimate objective: The particular organizational deficiency that the training program will eliminate. The intermediate objectives: The changes to the employees work behaviors necessary if the ultimate objective is to be achieved. The immediate objectives: The new knowledge, skills or attitudes that employees need to acquire in order to change their behavior and so achieve the intermediate objectives. In addition, it ought to consider how these components of the program reflect the culture and structure of the organization. Input evaluation tries to measure a number of inputs to a Training program, with a view to assisting managers in the process of identifying those which will be most cost-effective.
Accuracy of the inputs is crucial to the success of the training initiative (Warr, Bird & Rackham, 1970). If, for example, the wrong types of learners were chosen to attend a Webinar based program, this would be a waste of time and money for the organization. As in the Kirkpatrick model, the reaction evaluation tries to measure how the trainee officers reacted to the program. Against what was intended by the program, this type of evaluation draws on the subjective opinions of participants about the Training and how it might be improved. Finally, the outcome evaluation should measure the training outcomes against the benchmark of the programs ‘objectives (Janakiram, 2011). Comparison with Kirkpatrick Model The key difference in CIRO and Kirkpatrick’s models is that CIRO focuses on measurements taken before and after the training has been carried out. One criticism of this model is that it does not take into account behavior. It is, therefore, more suited to management focused training programs rather than those designed for people at lower levels in the organization.
The four levels of outcome evaluation that has strong parallels with the Kirkpatrick model:
1. the learning outcomes of trainees i.e. changes in their knowledge and skills
2. the outcomes in the workplace i.e. changes in actual job performance,
3. outcomes for the relevant areas of the organization i.e. departments or specialist units, and finally,
4. the outcomes for the organization as a whole. In addition to evaluating the context, inputs, reactions and outcomes to training and development, Organization must continuously measure the costs.
A cost/benefit analysis is usually conducted prior to committing to any training initiatives. Costs must be monitored to ensure that they don't scale over budget.
4. Phillip’s Five Level Return on Investment Approach:
The return on investment model is based on the Kirkpatrick’s evaluation model. Phillips added an additional step of ROI, which provides a monetary valuation of the training impact. Return on Investment (ROI) is a measure of the monetary benefits obtained by an organization over a specified time period in return for a given investment in a training program. It assumes conversion of qualitative issue into quantitative measures. ROI can be used both to justify a planned investment and to evaluate the extent to which desired return was achieved (Phillips Jack, 1997
1. Reaction & Planned Action Measures participant’s reaction to the program and out- lines specific plans for implementation
2. Learning Measures skills, knowledge, or attitude changes
3. Job Applications Measure change in behavior on the job and specific application of the training material
4. Business Results Measures business impact of the program
5. Return on Investment Measures the monetary value of the results and cost for the program, usually expressed as a percentage Critical Appreciation of the Model ROI Model cannot measure all the aspects of training success: whether the learners liked the training or not, the numbers of learners participating in the training, the extents to which learner’s personal objectives were accomplished.
ROI adds the fifth level to the Kirkpatrick for some reason. There are some pros and cons of calculating ROI of a training program. The costs of training are known and expressed in monetary terms, but the benefits are often soft, subjective, and difficult to quantify and convert into the monetary terms. Costs are known up front, before training, but benefits may accrue slowly over time. But on the other hand, course objectives and content will become more lean, relevant, and behavioral with focus on monetary results rather than on the acquisition of information. And by calculating ROI on the courses where it is possible, it is more apt to be trusted on the ones cannot evaluate at four levels.
francis1897 answered the question on August 25, 2022 at 07:55