By Robert J. Holland
British poet Edward Hodnett wasn't thinking about communication measurement when he wrote these words, but they are relevant nonetheless: "If you do not ask the right questions, you do not get the right answers. A question asked in the right way often points to its own answer.”
I've seen communication surveys yield data with minimal value, and they usually have one thing in common: They didn't ask the right questions.
Knowing the right questions to ask is not always easy. Asking those questions the right way can be even more difficult. The effort required, however, is always worthwhile because good data can be a powerful tool for improving how you service your association with content.
Here are some tips for knowing the right questions to ask members and advertisers on your next communication survey:
Begin with the End in Mind
It's important to be clear about how you will use the data you collect from a survey. What's the purpose of asking each question? Could the resulting data be acted upon? Ask only about things you intend to address. If you don't foresee using the information in some way, don't ask for it.
For example, a popular question on readership surveys goes something like this: "How much of the association's magazine do you read? All / Most / Some / None.”
One problem with this question is that it would yield data of limited value. What if you learned that 60 percent of your members read "most” of the publication? Does reading "most” of the publication help them in their work? And what parts of the publication do they read? Even if you knew what parts they read, how would that help you improve the parts they don't read?
A better question would be: "From the list below, select the publication features you find most relevant to your everyday work.” A list would help you pinpoint what people read, and the phrasing of the question asks them to rate the relevance of the features in the specific context of their everyday work.
Outcome-Oriented and Output-OrientedQuestions
Outcome-oriented questions address the impact of your association's communications and the difference they make in the achievement of your association's goals. On the other hand, output-oriented questions deal with how communication vehicles work and their tactical effectiveness.
What is your association attempting to measure? It could be one or the other, or both—but be clear about it up front.
Output-oriented questions provide information than can help you improve the mechanics of communication. For example, asking for the reasons why a potential advertiser does not advertise might reveal that the advertiser has never received a sales call from your association or that the cost of advertising is too high.
Outcome-oriented questions are the most powerful because they measure the business impact of communication. For example, if one of the goals for an association publication is to help members do their jobs more efficiently, you might ask members to rate their agreement with this statement: "The association's monthly magazine helps me do my job better.”
Keep the Big Three in Mind
When it comes to measuring outcomes, it all comes down to three things: knowledge, attitudes, and behaviors. Therefore, when deciding what to ask your members or advertisers in a survey, think about organizing questions around what people know, how they feel about what they know, and what they do as a result.
An example of a knowledge-focused question is asking members to rate their agreement with this statement: "As a result of the webcast, I have a better working knowledge of how to improve my website's ranking on Google.”
Attitude questions ask people how they feel about an issue as a result of communication about it: "After reading the current issue of the association's publication, I feel confident that the association can achieve its goals for our industry.”
Behavior questions are a bit trickier because they must hone in on what members do or might do as a result of a communication effort: "After reading this article, I understand the three things I must do to help improve vendor relationships.”
Take your time determining the right questions to ask and then refine the questions so they yield the data you need. After all, as Hodnett also pointed out, "Asking questions is the ABC of diagnosis.”
How to Ask
Now that you're clear on the kinds of questions to ask, how can you be sure you're asking them in the right way? Here are some quick guidelines:
- Make sure questions are clearly written so they are easily understood by the members and/or advertisers taking the survey. Test them first on a handful of your association colleagues.
- Define terms. What do you mean when you ask about the "quality” of communication?
- Ask about one thing at a time. For example, don't ask readers if a story helped them understand the association's mission and values.
- Be specific. Avoid asking questions that are too broad, such as: "How do you feel about the publication overall?”
- Bring some focus to open-ended questions. Don't ask, "Do you have anything to add?” Instead, ask: "What three improvements to our association's publication would you most like to see?”
- Brevity is a virtue on surveys. Ask only the most important questions and ask them succinctly.
Robert J. Holland, ABC is principal of Holland Communication Solutions, LLC.