The Google Plus logo

Get Data Smart: How to Measure Effectiveness Using Learner Engagement Metrics

User-generated video is a great way to encourage peer-to-peer knowledge sharing, but it can also be an unexpectedly powerful tool to help you measure the business impact of learning. This blog post explores how, with a little planning, you can link learner engagement in content generated by colleagues to an improvement in their critical performance behaviors—providing you with the data you need to demonstrate the value of social learning.

We have always known that the majority of learning is an iterative dynamic process that takes place mostly in the workplace and is enabled by practice and constant interaction with other people.

By the very nature of its informality, it is essentially a one-to-one process that impacts the individual but not the workgroup as a whole.

The evidence of its effectiveness is vital because it can be used to persuade your organization to invest in the emerging tools—and behavior change—that will enable the wisdom of one person to be shared not just with the individual alongside them, but with the whole organization.

In a recent webinar, our team of learning consultants explored with the senior data analyst from the InterContinental Hotels & Resorts Group’s (IHG) how they are measuring the impact of learning.

leo learning - using learner engagement metrics to measure business impact of learning

IHG, one the largest global luxury hotel brands, manages more than 5,500 hotels in 100 countries. The company’s strategic objective is to double the pace of growth by 2020 and with 400,000 staff, this will put enormous pressure on the global L&D department to keep knowledge and skills up to date.

To ensure IHG get the best value from the investment they’re making in training, it was decided to introduce a comprehensive impact measurement strategy. They sought LEO Learning’s help to put the strategy in place; this included co-designing and delivering a learning program for IHG’s 100 strong global Learning & Development team, to help them understand how to measure impact and how to use the tools and techniques required to gather the critical data.

How Do You Measure Learner Engagement?

Measuring learner engagement is crucial when it comes to looking for correlations between learners who are invested in their training, and their subsequent impact on business performance.

But how do you actually measure something that is arguably subjective? For most L&D departments, the focus is generally on measuring attendance and satisfaction scores. This is often dismissed as ‘happy sheets’. In fact ‘Level 1’ data can be incredibly useful as we move towards learning architectures that combine instructor-led training and informal peer-to-peer learning.

Examples of IHG’s Learner Engagement Metrics

One of the stories from IHG bears this out. You will hear in the webinar that IHG already had a small learning analytics team in place. They had been focusing on measuring the impact of one of the key programs: a learning and development course for the hotel group’s general managers.

The design of the solution was ground-breaking because it used a learning experience platform that had recently been installed. This enabled the use of video-based peer-to-peer knowledge sharing. General managers were encouraged to record on camera the initiatives they had put in place that made a difference to guest satisfaction ratings and revenue.

The platform also delivered videos that had been produced by the training team and subject matter experts. The analysts looked at the number of views each video had garnered and it emerged that the most popular were those recorded and shared by the general managers themselves.

This insight emerged from a comprehensive set of data that included learner information such as:

  • Learner registration/attendance
  • Learner progress
  • Learner satisfaction
  • Self-assessment (pre- and post-workshop)
  • Training credits

leo learning - leaners looking engaged during training

In addition, they also tracked engagement in the social elements of the program and responses to the user-generated content. Measures included:

  • Likes
  • Comments
  • Shares

By combining this, it allowed for a thorough bank of data to be built, providing a rounded picture of the degree to which the general managers engaged with the content.

The graphs showing which ‘learning nuggets’ the GMs had viewed and the spikes in interest in their own videos quickly became evident. The data was fed back to the general managers to encourage them to tell stories of what has worked well to improve customer satisfaction.

They also collected the critical business performance data for the hotels where the general managers worked and this included revenue and guest satisfaction.

Relating the Learning to Business Impact

With the data assembled in Excel spreadsheets, they then looked for correlations. The results were fascinating. As described earlier, it showed that the general manager videos were the most popular. It also revealed clearly that the improvement in revenue and guest satisfaction was greatest for those general managers who engaged most with the program.

The analysis did not claim that the L&D program was the only cause of the improvement. It was also noted that those general managers who didn’t engage with the training at all also showed a small improvement in business KPIs, so there were clearly other factors at play that were driving the improvement.

However, they could say with confidence that there was a correlation between the program and improved performance – and one of the key ingredients was the learning content provided by the general managers themselves.

Embracing the Power of Peer-To-Peer Engagement

As learning platforms become more user-focused, the introduction of Learning Experience Platforms such as LTG’s Instilled, is enabling a growing emphasis on user interaction.

This means that user-generated content can be uploaded, viewed and discussed between peers. Functionality such as comments, shares and likes encourages users to interact using techniques that are familiar from social media platforms such as Facebook and YouTube.

Interaction between peers can prove to be a powerful engagement tool—it offers learners the benefits of social learning, such as questioning, benchmarking and learning from colleagues.

An additional factor that can make user-generated content more engaging is the feeling of empowerment that comes with being able to upload unique content. It’s a chance for individuals to express opinions on topics in which they have expertise.

Peer-to-peer interactions can be engineered into the design phase of any learning program by creating authentic, meaningful scenarios and tasks that drive conversation between groups of learners.

measuring learner engagement metrics using data shown on an ipad

Success Breeds Confidence

Although the measurement activity at that time within IHG was focused on just a few programs, the data they collected gave people confidence that there would be great value in scaling up the activity.

The impact measurement team, with LEO Learning’s assistance, has now standardized the processes and tools, and is embedding measurement in the day-to-day task of designing and delivering great learning.

In the future, they will be using data to refine the design and, in turn, improving the contribution the L&D team can make towards IHG’s target of doubling growth by next year.

Watch our webinar to discover the five critical breakthroughs that helped InterContinental Hotels & Resorts Group (IHG) demonstrate the value of their new global training scheme.

Watch the Webinar