This is the fifth blog in a series on training evaluation. It explores 2 stages of the Phillips ROI Methodology – data collection and data analysis.
This blog is the 5th in the series on training evaluation. Part 1, Part 2, and Part 3 of this series dealt with the Kirkpatrick Model of evaluating a training program. In Part 4 of the series, we saw what the Phillips model added to the Kirkpatrick model, and how the model encourages measuring data at every level. Each level provides valuable information and by measuring this data, we can identify the most successful levels and those that fail to provide the desired results.
In the previous blog, we briefly reviewed its 5 levels and what each level is intended to measure. Let’s go a little deeper now, shall we?
The Phillips ROI Methodology: Data Collection & Analysis
Data Collection:
- During the program, data is collected on:
- Reaction (level 1)
- Learning (level 2)
- After the program, data is collected on:
- Application (level 3)
- Impact (level 4)
Data Analysis:
- Isolating the effects of the training program
- Converting data to monetary value
- Identifying intangible benefits
- Calculating ROI (level 5)
The 5 Level Phillips ROI Methodology
The Phillips ROI Methodology is a simple and proven process that can be used to generate reliable and accurate data – especially useful to demonstrate the value of the program to sponsors and stakeholders. Here’s a brief recap of the 5 levels and what they help measure.
Explore how to build a successful business case for eLearning.
|
Level |
Measures/Captures |
|
| 1 | Reaction and Planned Action | Measures learners’ perception of the relevance of the training program. Captures planned actions (the learner’s intent to put into use the knowledge gained during the training program). |
| 2 | Learning | Measures competence levels of learner: changes in knowledge, skills, and attitudes during the program. |
| 3 | Application and Implementation | Measures on-the-job application and implementation of the learning:
|
| 4 | Business Impact | Measures the impact of the training on the business on (among others):
Analyzes the role of other factors (a new product, seasonal demand, etc.). |
| 5 | Return on Investment | Compares the monetary benefits of the program to its costs. |
Though the Phillips ROI Methodology includes evaluation planning, data collection, data analysis, and reporting, we’ll be looking only at data collection and data analysis in this blog.
Data Collection
Data needs to be collected during and after the program:
During the program, data is collected on:
- Reaction (level 1) – through questionnaires or surveys
- Learning (level 2) – through tests or assessments/quizzes before and after the training
After the program, data is collected over a period of time on:
- Application (level 3) – through on-the-job observations and interviews
- Impact (level 4) – through sales figures, changes in quality of work, time spent
Data Analysis
Data analysis includes:
- Isolating the effects of the program
- Converting data to monetary value
- Identifying intangible benefits
- Calculating ROI (level 5)
Isolate the Impact of Training
This is a very critical step that shows that training program is related to the improvement in a business metric. Changes or improvements in business could occur due to reasons other than the training program itself.
The only way to know what caused the change is by using isolation techniques (control groups, inputs from experts and customers, estimates from managers and participants, forecasting methods).
Convert Data to Monetary Value
Before we can calculate the ROI, all the data collected must be given a monetary value so that it can be compared with program costs. Each organization has its own standard unit of measurement for such data. Employees’ wages against time saved to do a job is an example of how some organizations convert data into monetary terms.
Identify Intangible Benefits
There will be some data that can’t be given a monetary value. These are known as the intangible benefits and include increased job satisfaction, improved self-confidence, leadership, etc. These intangibles are very important and must be considered as well. The best way to measure them is by using validated tests.
Calculate ROI
Return on Investment (ROI) is the ultimate measure of the program’s success. At this level, it is essential to isolate the benefits of training and eliminating other factors (such as work environment, motivation, trends) that may have contributed to the impact on the organization.
|
ROI (%) = Total Program Benefits (–) Total Program Costs × 100 Total Program Costs |
Final Thoughts
So, is it essential to calculate the ROI of every training program? The short answer is “No”. The ROI model is not meant for all training programs because knowing the ROI of a training program doesn’t automatically provide a lot of practical benefits to a business. It’s definitely nice to know but is not critical. After all, you can know from Level 4 (Impact) itself whether the program was a success or not. There’s no need to spend more time and money to calculate Level 5, the ROI.
The important thing to remember is that all programs need to be evaluated, if not for ROI, at least at the other levels – all of them should be evaluated at Level 1, around 90% of them at Level 2, around 30% at Level 3, and 10-20% at Level 4.
But if you want to prove to your stakeholders how your training programs are helping increase the organization’s profits, the Philips ROI model is the best way to do that. Though it does take a lot of practice to master the process and it is challenging to eliminate non-training-related factors, the Phillips ROI Model can help you determine the effectiveness of your training program and whether it is helping your organization meet its business goals.
For more information or a consultation session on the ROI methodology, visit the ROI Institute.
