<< Return

An Alternative to Averages - 11/8/2011 -

Here’s why and how to ‘de-average’ your survey satisfaction ratings.

By Larry J. Seibert, Ph.D.

How many times have you conducted a satisfaction survey of your membership only to find that the average ratings are all pretty much the same? Why is it that the average satisfaction ratings on your website, blog, magazine, journal, etc. don’t seem to change from year to year, no matter what you do differently? With no distinguishable differences in average satisfaction, it is difficult to use this information to provide meaningful feedback.

The reality is that there may be significant differences in individual ratings given by your members, but those differences are masked as soon as you calculate the average. In other words, because an average is a measure of the middle of a set of numbers, the high and low responses offset each other.

Consider the "big picture”. As Table 1 illustrates, the distribution of members’ responses can vary significantly for different areas of the association and still produce the same average satisfaction rating. Relying on a central number that neutralizes the extremes can be misleading. This brings to mind the college professor who suggested that using an average is like sticking your left foot in a bucket of ice water and your right foot in a bucket of boiling water and declaring that "on average” your feet are at room temperature.

Even though the annual conference and the website have the same average satisfaction rating in Table 1, it would be a mistake to believe that members have the same opinion of both. These respondents are evenly split between being "delighted” and being "disappointed” with the annual conference, while all the respondents find the website’s performance "acceptable”.

Table 1

Response

Response Value

Member Benefits

Annual Conference

Member Services

Website

Excellent

5

20

50

Very good

4

20

10

Good

3

20

80

100

Fair

2

20

10

Poor

1

20

50

Total Responses

100

100

100

100

Average Rating

3.0

3.0

3.0

3.0

In spite of this basic flaw, averages are still widely used because they are easy to calculate and easy to understand—but there is an alternative. When analyzing opinions and attitudes, researchers use an alternative statistic that is just as easy to calculate as averages and gives better information. This statistic is called the top-two rating. A top-two rating is the percentage of respondents who gave one of the two best possible responses to a rating question. Notice in Table 2 that the responses that produced identical average ratings produce different top-two ratings.

Table 2

Response

Response Value

Member Benefits

Annual Conference

Member Services

Website

Excellent

5

20

50

Very good

4

20

10

Good

3

20

80

100

Fair

2

20

10

Poor

1

20

50

Total Responses

100

100

100

100

Average Rating

3.0

3.0

3.0

3.0

Top 2 Rating

40%

50%

10%

0%

When analyzing responses to satisfaction rating questions, top-two ratings are superior to average ratings for several reasons. First, top-two ratings have the sensitivity to identify differences in response distributions, as shown in Table 2.

Second, to change members’ behavior (e.g. sign up for the e-newsletter, read the blog, attend the annual conference, enroll in continuing education programs), it is necessary for them to be motivated. And for them to be motivated, they must first have a strong positive opinion. Only members who give a top-two response have a strong positive opinion. For this reason alone, it is more important to know how many members’ responses are in the top two, than it is to determine what the average or middle rating is.

A third reason to use top-two ratings is that when there are differences between the top-two ratings from one year to the next, that change is meaningful. Changes in average ratings can sometimes provide a false sense of accomplishment. Table 3 illustrates this point.

In this example, the average satisfaction rating for this activity improved from year one to year two, while the top-two score remained unchanged. Notice that the only change was that 50 percent of respondents changed their opinions from poor to fair.

Relying only on average satisfaction ratings would lead to the conclusion that the performance of this publication has improved, even though there has been no change in the percentage of respondents who have a strong positive opinion of it.

Table 3

Response

Response Value

Year 1

Year 2

Excellent

5

50

50

Very good

4

Good

3

Fair

2

50

Poor

1

50

Total Responses

100

100

Average Rating

3.0

3.5

Top 2 Rating

50%

50%

Moving from average satisfaction ratings to top-two ratings requires no change in how surveys are administered; the only change is the way in which the responses are analyzed and reported. Top-two ratings are as easy to calculate as average ratings, and top-two ratings provide a more accurate reflection of your members’ opinions of the content and experiences you provide them.

Larry J. Seibert, Ph.D. is the president/CEO of Association Metrics.


 

© Copyright 2017, Association Media and Publishing. All rights reserved.