Are Surveys Evil?

By Theresa M. Welbourne, Ph.D.

How many times have you been annoyed by a survey? The survey taker calls you on the weekend or in the evening, promises it will take only a few minutes of your time, and 10 minutes later you curse yourself for agreeing to stay on the line. Or you clicked on a link to take an on-line survey, and the form will not let you proceed to the next page until you complete ALL of the questions (but there are simply some questions you don’t want to answer).  Lastly, you agreed to complete the survey, but you keep seeing the same questions over and over again. Sure the questions are slightly different, but you ask yourself, “Do they think I’m an idiot?”

These stories are very common. I hear from many employees who are sick of their employee surveys and I often talk to people who feel incredibly insulted and angry by many of the surveys they receive. Given this reaction, I decided to run a survey on surveys in the October Leadership Pulse™. The Leadership Pulse is a study that I have been doing with a large sample of global leaders. I reach out to them every two months with a short set of questions; they answer, and I get results back to them within weeks.

I also used October as the time to do this project; thus, the influence of Halloween may have made everyone a bit more open to the subject. I say this because I used the word “evil” twice in this particular project.

Participants were asked the degree to which they thought the annual employee survey and customer surveys were “evil.” I followed up with a few other questions about surveys; these were more traditional in nature. Evil was defined as follows (from the dictionary):   Something evil = “a situation that is very unpleasant, harmful, or morally wrong”

Most of the 307 respondents did not take the extreme stand that surveys were evil. The table below reports the percentage of people who agreed with the two statements (using a 1 to 5 scale, where 1=strongly disagree, and 5=strongly agree; respondents who agreed answered 4 or 5 on the scale).

However, delving into the details, I found some interesting variability in the degree to which subgroups were more or less favorable about these types of surveys. When it comes to employee surveys, the group that is most likely to think they are evil has a job function listed within research and development (not information technology or IT), with 23% agreeing. Other groups with relatively high agreement scores are: engineering at 22%, finance and accounting at 20%, marketing at 19%, and manufacturing at 17%.

When it comes to customer surveys, in the subgroups with higher agreement on surveys being evil the scores are similar: finance and accounting came in at 20%; manufacturing 17%, marketing 13%, engineering 11%, and research and development (not IT) at 8%.

You may wonder why anyone would think a survey is evil. Below are some of the prevalent comments that explain these attitudes:

• “They are poorly worded, do not address the real issues, fail to be acted upon constructively, and are typically used to manipulate employees.”
• “People have learned that surveys can be manipulated so the importance of surveys has been minimized.”
• “Auto dealership surveys tell you they have to have an "excellent" response to all questions. This is intentionally skewing the data.”
• “Most of the time the information goes into a black hole or is used to 'beat people up' for not making the right scores. More often the focus is on fixing the numbers instead of understanding what is being said. The last one we did the CEO did nothing with the information.”
• “Surveys do not lead to improvements. It seems more like a 'check the box' exercise.”
• “The problem with most surveys is the lack of feedback and action after the data is collected and analyzed. Most leave you with a sense of 'Why did I bother?'"

Clearly, survey taking wasn’t a positive experience for these people. However, even with those negative experiences these individuals were willing to take time to share their opinions and experiences in this particular survey. Even though they have had negative experiences in the past, they seem hopeful enough that something more positive may come out of surveys, thus they decided to participate in this particular project.

Other dimensions of surveys

In addition to the “evil survey” questions, I asked a few more traditional questions about surveys. The overall responses are in the table below:

As you review these additional scores, it becomes clear that although no one really considers surveys “evil,” they don’t think very highly of them either. And as you dig into the data, you quickly see that the most favorable comments come from the people in jobs most likely to do surveys.

From individuals in marketing (those who do customer surveys):

• “When a company conducts a survey, I feel it is making the necessary steps to want to improve as a business."
• "I have participated in my own companies' surveys and as a customer of another company. I feel providing feedback is critical to improving the customer experience”

From individuals in human resource (those who conduct employee surveys):

• “I have written them, and have participated in them. I believe, if done well, they can be a valuable tool in information gathering and continuous improvement.”
• “If surveys contain relevant content and are used, they work great.”
• “Surveys are a valuable tool for leadership to assess the attitude of the work force.”

After reading all the comments, my conclusion is that the positive comments are not very positive. In general, the managers included in this study are incredibly ambivalent about surveys. They don’t’ really “hate” them (although I did not ask that question specifically), but they don’t like them either. So, what’s going on?

Lack of viable alternatives?

Keep in mind that the sample studied here is managers; they need data to do their jobs well. Maybe they simply cannot imagine a viable alternative to the survey. This makes sense to me because I see many companies continue to do surveys and spend lots of money even though they admit they get no or close to no value out of the process. Perhaps it’s the hope that they could be something more that keeps everyone from turning away from the survey, or maybe it’s our own personal need to have a voice that keeps us hoping that the data will go somewhere and result in some action.

I think it’s time to suggest a real alternative to surveys, or at least it’s time to define what a survey is and is not. I would suggest that surveys and dialogue tools are two points on a continuum describing stakeholder relationship management tools. Just like in the CRM and ERM worlds, some tools developed in this space collect data only, while others are being modified to put the real “relationship” back into the systems.

Surveys:

• One-way communication (you ask questions of the survey respondent)
• Focused on getting a score
• Primarily uses quantitative data
• Interested in benchmarking (which means question wording cannot be customized)
• No guarantee that the survey taker’s concerns will be met
• There may be no action whatsoever; company may be content with score or only want the score
• Focused on survey developer’s agenda

Dialogue tools:

• Two-way and interactive communications
• Focused on getting deep understanding of a situation and then engaging the other party in a dialogue based on an initial conversation or collection of data
• Score is only used to start the dialogue
• Not focused on benchmarking; questions are customized
• Extensive use of qualitative data
• Guarantee you will listen and drill down to take action, but no guarantee that all comments and dialogue will result in action
• Focused on a business agenda and a desire to understand using the language of the population being studied

What types of dialogue tools exist today?

The use of dialogue tools has existed for some time. Focus groups use data to engage people in dialogue about a number of subjects. The goal is to truly understand a customer or employee experience by letting him or her speak up.

Today, technology has allowed a number of organizations to create sophisticated dialogue tools that replicate and enhance the focus group approach. Blogs and on-line collaboration tools all fall into this domain.

Data and Dialogue Leadership™ tools: the next frontier

I am convinced that merging data collection tools (survey technology) with rigorous on-line dialogue tools can be the next frontier for both doing research, for knowledge management, and for high involvement or high performance organizations.

The Leadership Pulse study grew out of my interest in changing how I did academic research. For the last three years, a group of leaders has been answering what I call short Pulse Dialogues™ every two months. I change the subject, collect trend data, and then try very hard to share the data as soon as I can. There are three ways respondents can receive feedback. First, they have access to their own personal reports; this shows them their own score versus their industry and the overall sample. On the personal report there is another opportunity for them to engage in follow-up dialogue. Second, they can view a summary report (presentation), and, lastly, they have complete access to the overall results (on-line reports).

My goal is to create a learning team that shares stories and opinions and also provides some data for limited benchmarking. We have an agreement that I keep the dialogues short, and I agree to get the results back to everyone within weeks. This deal has worked well.

Are surveys evil? If so, could data and dialogue tools also be evil?

Surveys are evil when we lie to employees or customers about their use. They also may be evil if we don’t communicate what we plan to do with the survey data, and respondents are basing their decision to participate on an implied trust or hope that something will happen.

The key to preventing “evil” survey syndrome is full disclosure. Be honest with employees and customers. These stakeholders will give you data for a score only. Generally, they want you to do more than obtain a score; employees and customers both want to have a voice. Data and dialogue tools hold great promise in being able to deliver on improving the number one asset any company has—its relationships with stakeholders. But data and dialogue tools can fall into the same trap that surveys fall into, unless you are completely honest and open and show that your goal is true dialogue.

1. The Leadership Pulse sample consists of leaders who have opted into the study. About 40% of the sample are C-level executives, and about 75% director level and above. About 4,000 individuals are part of the core group.
2. SD = standard deviation.

If you would like to join the Leadership Pulse study, you can sign up at www.umbs.leadership.eepulse.com/signup.html

About the Author(s)

Theresa M. Welbourne, Ph.D., is the founder, president, and CEO of eePulse, Inc. (www.eePulse.com) and is an adjunct professor of executive education at the University of Michigan Business School.