Exclusive to EX Space members

A critique of the employee engagement survey approach

Emma Bridger

Minutes
30 Mar 2023
Employee Experience
Employee Engagement
Measurement
Business case

You must be logged in to read this content - log in, or sign up to essentials for free now

Over the last two decades, engagement surveys have become common place inside many large organisations worldwide – to the point where, for many, this tool is today their primary method for capturing views and insights from employees.

Is the tail wagging the dog?

There is no doubt that data and insight regarding employee engagement within your organisation is a critical part of the engagement and EX strategic planning process. It’s vital to understand how your employees are feeling, how they rate their experience, how engaged (or disengaged) they are, and what action needs to be taken to improve engagement. You also need to be able to track and monitor action taken to understand what works and inform your plan. However, over the years the employee engagement survey has taken centre stage for many organisations to the point where it has become their overriding focus – a classic case of tail wagging dog!
It’s important to remember that the employee engagement survey is a means to an end, not the end itself. If used wisely, the survey process and the insight provided can be invaluable, but often an over-reliance on the survey process results in a transactional approach to employee engagement. This involves putting the survey process at the heart of engagement activity, which is a reactive way of approaching engagement and can feel like a tick-box exercise; once the action planning has finished, the company forgets about engagement until it’s time for the next survey.
Companies have become over-reliant on the survey because it provides a false sense of security, giving the illusion that something is being done about engagement. Improving engagement can feel like an overwhelming task, but undertaking a survey gives confidence that something is happening to tackle engagement.

The problem with data

Survey feedback always involves providing the results in the format of charts, data and engagement indexes. This can feel like a comfortable and familiar place for many senior leaders who operate in a world of management reports and numbers. The survey takes something which is predominantly about feelings, emotions and behaviours and then uses the language of data and numbers to communicate findings. This is part of the reason why companies have become so reliant on the survey, and why we still see transactional engagement as the predominant approach in organisations today.
 
So, whilst data and insight are important to improve engagement, there are a number of problems which you need to be mindful of when using an employee engagement survey.

1. Definition problems

Employee engagement is very personal, and while there are common themes in the different definitions, what really defines and drives engagement can differ from company to company. Each engagement survey provider will explain why their model is the right model and back this up, however there is no substitute for undertaking qualitative research to understand:

  • How your organisation wishes to define engagement
  • what drives engagement for your employees


Undertaking qualitative research to answer these questions will then allow you to build a bespoke measurement tool which will be valid for your organisation, that is it will measure what it is supposed to measure.

2. Obsession with response rates

Many companies still seem to be obsessed with improving their response rates to the annual survey, to such an extent that this can often be seen as a key component of their engagement strategy:

"To improve survey response rates by x%."

This could be because it offers a useful diversion from the real challenge of actually improving engagement. That is, if we focus on improving response rates, and we have success in this area, then it feels like where we’re making progress, right?

Wrong.

Common sense would say that higher response rates are better; the more people filling out the survey, the more engaged they must be. When you look at best-practice social science research, what we actually need is a representative sample of employees to ensure that the findings observed can be generalised to the wider population. Therefore, as long as we sample in the correct way, and the sample is large enough, this is enough to give us what we need.

The question of sample size has long been debated within social science, but there are some helpful sample size calculators you can find on the internet which take account of:

  • Your total population size i.e. number of employees 
  • The margin of error you’re willing to tolerate, usually between 3% and 6%
  • They then provide a calculation for how many responses you need to have either 90%, 95% or 99% confidence level

To illustrate:

  • Population size: 10,000 employees
  • Margin of error: 3%
  • Sample size:
  • 90% confidence = 703 responses
  • 95% confidence = 965 responses
  • 99% confidence = 1556 responses
    • You’ll see that for this example you’re looking at a 10% - 15% sample size for the results to have a low margin of error and high confidence level…. This is nowhere near the 80% plus response rates companies go after.

    The same tools also provide a calculator to work out how many surveys you’ll need to send out to generate the required response rate.

    To illustrate:

    • If we are going for 95% confidence we need 965 responses back
    • Our predicted response rate is 35%
    • Therefore we will need to send out 2754 surveys to ensure we reach the necessary response rate
    Finally the tool then calculates the accuracy of your response rates.

    To illustrate:

    • Population size is 10000
    • We actually had back 2460 completed responses from the 2754 surveys we sent out
    • Therefore we will need to send out 2754 surveys to ensure we reach the necessary response rate


    Which gives us:
    • An error level of 1.4% at 90% confidence
    • An error level of 1.7% at 95% confidence
    • An error level of 2.3% at 99% confidence
    You can see from the example above that with a population of 10,000 employees, an approximate response rate of 25% comes with a very low error rate and a high confidence level. And yet many CEOs would be disappointed with a 25% response rate. Again, this comes back to establishing the purpose of the survey and understanding what a higher response rate will actually provide.

    Sampling

    Finally, a quick word on sampling. In order to be able to generalise any findings to the wider population, a probability sample must be taken. A probability sample means that everyone within the population (i.e. your employee base) has an equal chance of being included in the sample. The easiest way to achieve this is to send the survey to everyone, or you can select a sample of employees to send the survey to, as detailed by the worked example above; you just need to ensure probability sampling methodologies are used so that you can generalise the results once they in.

    3. Mind your language

    One of the challenges of using a survey is that of the semantics and language involved. For example, surveys will often ask questions about senior leadership; depending on where employees sit in the hierarchy, this will mean different things to different people. Establishing questions with as little ambiguity as possible is a real challenge. Often differences in interpretation become clear in the survey follow-up process. For example, when running focus groups to delve deeper into issues raised within the survey, it becomes clear that employees have interpreted certain questions in very different ways.

    4. Obsession with benchmarking

    In addition to obsessing about response rates, senior leaders are also obsessed with external benchmarking. Making the Sunday Times Best Companies list or the top companies in the Great Place To Work list are often set as employee engagement objectives. Whilst this motivation does provide a burning platform to enable a focus of resources on engagement, very often making the list eclipses the purpose of engagement itself, to improve the business and create a competitive advantage. The objectives and plan are at risk of being short-term when the focus is to benchmark. Making it onto such a list is a benefit of having a focused engagement strategy and plan, rather than the plan itself.

    5. Deficit approach

    Finally, one of the major problems with placing the survey process at the heart of your engagement strategy and plan is that it encourages a deficit approach to engagement. Action planning often focuses on fixing those areas in which the business performed poorly in, rather than learning from the insight about the organisation’s areas of strength. When focus groups are run after the survey process, there are rarely questions asked about what we are doing well in and how we get more of this. Very often the post-survey discussion and action planning is quite a depressing process, looking at the bottom scores and coming up with ways to improve them. This in turn results in an engagement survey process that is actually quite disengaging. Even when companies are performing really well, when it comes to engagement, the action planning process will often still focus on those areas that could be better.

    Conclusion

    There’s no doubt that engagement surveys are an incredibly useful tool, but only if used in the right way. The tips shared here will enable you to overcome the common problems with survey use and ensure you can use the data to improve how it feels to work in your organisation.

    Related Resources