Why employee surveys, like political polls, are misleading

When results of the 2020 elections began to come in, a familiar question resurfaced: Why did so many polls get their predictions so wrong? The question has sparked all sorts of analysis, with Stephen Engelberg, editor-in-chief of the nonprofit news agency ProPublica, describing a “big failing.”

It isn’t just news media and political groups that need to take note of the mismatch between polled behavior and actual behavior. This moment should also serve as a wake-up call to businesses. The systems our society uses to measure how people feel simply aren’t, at some level, working. That includes the surveys organizations use to try to gauge the sentiment of their workforces.

Through my work helping businesses of all kinds, from tech companies to the U.S. military, design successful workplace cultures, I run into this problem all the time. The internal data businesses collect gives them what they believe is a clear impression of whether their employees are feeling engaged and satisfied. But when leaders look at external websites such as Glassdoor, where current and former employees anonymously rate the organizational culture, they see an entirely different picture. I’ve seen this huge disconnect occur in both directions. External websites sometimes suggest employees are less engaged than internal surveys indicate, and sometimes they signal the opposite.

Some executives are quick to assume that external websites must have skewed data, creating a wrong impression, because the people who submit information might not represent a valid sample of the employee base. But through my work, I speak with current and former employees to get a full picture. And very often I find that it is the internal data that is skewed.

There are several reasons for this. Some of them echo the problems that plague political polls. The sample could be unrepresentative. For example, it’s extremely difficult to know whether the people who fill out internal surveys are in fact typical of the organization as a whole. As Rajeev Peshawaria, author of the book Open Source Leadership, explained in an interview with the Society for Human Resources Management, some employees are so busy that they don’t take time to fill out these surveys. Some shun such surveys, or hold back from sharing their true feelings when they do complete them, because they don’t trust that their answers will really be anonymous. The people who do fill out these surveys are, in fact, a self-selecting, non-representative sample.

In Harvard Business Review, Wharton professor Peter Cappelli and postdoctorate researcher Liat Eldor point to another problem: It isn’t always clear how each respondent interprets a question. “One problem that companies often stumble over when using engagement measures is that different definitions of the term abound,” they write. When asked how “engaged” someone feels with their work on a scale from 1 to 10, someone overwhelmed and working around the clock might feel all too engaged and desperately in need of a break, and give 10 as their answer. Someone who feels they’ve struck a relatively good work–life balance might say 5, believing that’s an ideal amount of engagement — not too much, not too little. Meanwhile, executives may believe that a high engagement score indicates a happy and satisfied workforce.

Of course, steps can be taken to improve these surveys. Surveys can be designed to offer specific, clear definitions of terms. They can leave room for extensive qualitative data in which people explain in their own words why they answered the way they did. And detailed information can be included to clarify how anonymity will be ensured.

But even with these changes made, problems will remain. Humans are much more complex than these surveys account for. Data provides clues — not insight. And that means businesses should rely less on data when it comes to understanding employees and predicting their behavior. Here are three empirical approaches leaders aiming to assess the sentiment of their employees can take.

Data provides clues — not insight. And that means businesses should rely less on data when it comes to understanding employees and predicting their behavior.

Count on culture. Workplace culture is a collection of beliefs, mindsets, mental models, principles, world views, and attitudes that inform behaviors and drive operational outcomes. In a study conducted by Gapingvoid, we found that businesses that create cultures that are simultaneously focused on customers and employees outperform the competition. In these businesses, employees are encouraged to openly discuss problems and challenge prevailing ideas. They have what Harvard Business School professor Amy Edmondson calls “psychological safety,” which she described in strategy+business as “the belief that the environment is safe for interpersonal risk taking.”

Ask less, observe more. To determine how your employees are feeling and what challenges they’re facing, look around and take note of what you see. Are people collaborating and interacting well? Are they treating one another with respect? Is everyone getting a chance to speak, and are they being heard? Encourage your staff to be on the lookout for these kinds of things. People respond to what they observe. The more they see an engaged, psychologically safe environment and act with vigilance to maintain one, the more the organization will change for the better.

Avoid the “pseudo voice” conundrum. When employees point to problems, either by mentioning something to a superior or filling out a qualitative section of a survey, there’s too often a tendency to let these problems slip through the cracks. Researchers Gerdien de Vries and Bart W. Terwel of Leiden University and Karen A. Jehn of Melbourne University call this “pseudo voice” — giving employees “the illusion of having participative influence.” Unfortunately, they report, the pseudo voice “is common even at companies that say they are committed to giving employees a chance to contribute their ideas.” To engage employees, take all feedback seriously. It also helps to have someone double-check to make sure managers haven’t mistakenly missed any crucial feedback from their teams.

Most important, it’s time for a new mindset. Businesses should focus more on building toward a better future and less on assessing where things are. Making this shift means embracing the unknown and accepting complexity. David Snowden, founder and director of the Centre for Applied Complexity at the University of Wales, calls for an end to the “entrainment of thinking.” As Mark Buchanan reported in s+b, Snowden says organizations should “dip people into chaos on a regular basis.” (Snowden collaborates with my team at Gapingvoid.)

Putting less stake in the kind of data businesses have been counting on for years may feel chaotic. But that’s what it takes to move forward, change our paradigms, and get a clearer sense of how our workforces — just like our voters — are really feeling.