The Phillips ROI Methodology – Measuring Data at All Levels – Part 5

In this post we will look at how ROI is measured at the final level in Phillips ROI methodology.

Published on Updated on January 27 2020

Access the guide on ROI and eLearning - Myths and Realities.

Download Now

This blog is the 5th part of the Kirkpatrick series that I have been writing about over the last few weeks. Part 1, Part 2, and Part 3 of this series dealt with the Kirkpatrick Model of evaluating a training program.

In the fourth part of the series, we saw how the Phillips ROI MethodologyTM suggests that measurements be taken at every level. Each level provides valuable information and by measuring this data, it is possible to identify the most successful levels and those levels that fail to provide the desired results.

Here’s a look at what can be measured at each level and how they can be measured:

Level 1: Reaction and Planned Action

What can be measured:

  • The learner’s perception of the training program’s relevance.
  • The learner’s intent to put into use all the knowledge gained during the training program.

Methods to measure: This is done by verbally asking learners what they thought of the course, conducting surveys on their reaction to the course. After the program, questionnaires and surveys may be used to measure this data.

Level 2: Learning

What can be measured:

  • The amount of knowledge obtained from the training program.
  • A change in the learner’s attitude.
  • Competence levels of the learner.

Methods to measure: Tests and assessments are conducted before the training as well as after the training to measure the amount of knowledge gained through the training program. These tests and assessments can be carried out online or via the traditional method; whichever way they are conducted, they must be related to and based on the training program.

Level 3: Application and Implementation

What can be measured:

  • The degree to which the learner applies the knowledge gained, to the job.
  • Changes in the learner’s behaviour.
  • The degree to which processes and procedures are followed and executed.

Methods to measure: Measurements must be ongoing and taken over a long period of time through
on-the-job observations and interviews. An analysis tool must be used to measure outcomes. Focus groups can be used to assess the application of knowledge on the job. Action plans can be used to encourage learners to apply their knowledge – their progress and impact on work is measured.

Level 4: Business Impact

What can be measured:

Improvements in:

  • Output
  • Quality
  • Costs
  • Time
  • Customer satisfaction
  • Job satisfaction
  • Work habits
  • Innovation

Methods to measure: A lot of sources such as performance records and operational data will provide valid business impact measurements at this level – sales figures, changes in the quality of work, and the amount of time saved to name a few.

But let’s be realistic – changes or improvements in business would occur due to several reasons other than the training program itself. The only way to know what causes the change is by using isolation techniques such as control groups, inputs from experts and customers, estimates from managers and participants, and even various forecasting methods.

All data collected at this level must be converted to a monetary value so that it can be compared with program costs. Each organization has its own standard unit of measurement for such data. Employees’ wages against time saved to do a job is an example of an alternative way some organizations convert data into monetary terms.

There will, however, be some data that cannot be tagged with a monetary value. These are known as the intangible benefits and are most often seen in soft skill training programs such as communication skills, interpersonal skills, and management development programs. These intangible benefits include improved customer relations, reduction in complaints, employee grievances, and conflicts. These intangibles are very important and must be mentioned separately and taken into account as well.

Level 5: ROI

What can be measured:

Return on investments.

Methods to measure: Several ROI calculating tools are available.

The formula to measure ROI is:

As you can see, the most important parts of the Phillips ROI MethodologyTM are collecting and analyzing data, and we have seen how this can be done successfully. My final blog in this series will look at how organizations have used this to their advantage.

ROI and E-learning - Myths and Realities
Copy link