DETERMINING WHAT MEMBERS WANT: HOW TO DEVELOP A MEMBER SURVEY THAT ACTUALLY GENERATES THE INFORMATION YOU NEED.

Transcription

1 DETERMINING WHAT MEMBERS WANT: HOW TO DEVELOP A MEMBER SURVEY THAT ACTUALLY GENERATES THE INFORMATION YOU NEED. Private clubs are interesting institutions. Where else are an institution s customers also its owners? This dichotomy represents a unique challenge for club managers meeting the needs of one customer/owner may come at the expense of another customer/owner. Every club manager has experienced the vocal member, or worse, vocal board member, who has a well-formed (even if uninformed) opinion on a particular issue or topic. But a sample size of one, or two or three, no matter how vocal, generally does not represent the views of the majority of the membership. And, reacting to the demands of a minority may negatively impact the majority. When faced with a particularly vocal member, the most valuable resource for any club manager is factual data. After all nothing, will stop an opinion in its tracks like a number. But factual data about the opinions of a membership is often difficult to obtain. Unlike pro-shop transaction data or account billing data, you probably don t have a database that you can access that will answer questions like, What percentage of the membership is satisfied with the current food and beverage operation? or What do most members perceive to be the short and long term priorities of the club? Soliciting input from individuals in the men s grill or the locker room may provide some insight, but chances are the individuals who are most likely to readily express an opinion are, again, not representative of the membership as a whole, but rather part of the vocal minority. Often, the only way to gather the data necessary to truly ascertain the extent of a problem voiced by a vocal minority or to assess the level of satisfaction with the club s performance in a particular area, is through a survey. Customer surveys are commonly used in private industry as a tool to measure customer satisfaction. Survey results are then used to develop programs with the intention of increasing the satisfaction of customers, which theoretically then leads to increased product/service usage and greater revenues. Customer surveys should also be common practice in private clubs for the same reasons. However, in the absence of a dedicated marketing research department, club managers are often left to their own devices to develop and execute a member survey. As any club manager who has done this can tell you, developing a survey that actually generates usable and actionable data is more difficult than it might appear to be at first glance.

2 A survey needs to be viewed as a tool used to gather data used for decisionmaking a survey or the results of a survey, does not represent an end in itself. After all, its not the answers to the surveys that are truly of interest, but rather how these answers provide insight into the larger issue or question that necessitated the survey in the first place. In an effort to develop a survey that generates useful and actionable results, consider the following six steps: STEP 1: DETERMINE YOUR OBJECTIVE This may seem obvious. However, surprisingly, many clubs spend the time, resources and funds to send out a survey to their membership without a clear statement of what they are trying to accomplish. As a result, the response data is at best meaningless, and at worst misleading. As a rule, survey objectives need to be deep and narrow, rather than broad and shallow. For example, consider a club that is realizing an unsustainable operating loss. Club management may decide to solicit the input of the membership to determine why members are not using certain billable services (e.g., dining, catering, private functions, tennis/golf/swimming lessons, bringing guests to the club, etc). This type of survey objective is too broad and shallow it will cover a lot of ground, but very little, if any of the results will be specific enough to be actionable. With a modicum of analysis, club management (or a designated committee) could probably narrow down the losses to identify the primary source or sources. For example, if the food and beverage operation is identified as a source of operating losses, a highly specific survey focused on the level of importance and the associated level of satisfaction that members have with specific areas within the food and beverage operation could then be developed. This type of objective would be considered to be deep and narrow the results should be specific enough to generate an action plan. STEP 2: BRAINSTORM A LIST OF QUESTIONS Once a survey objective has been determined (preferably one that is narrow and deep), develop an unedited list of all of the questions that you would like to see answered after the survey is completed. At this stage, don t worry about wording or length of question, just brainstorm a list. Strive to generate 20 questions. The only guideline should be that every question listed should address your objective. This step is best done with a designated committee everyone will have different perspectives on the issue in question. And these different perspectives will ultimately make for a more thorough survey (as well as create leadership buy-in into the process).

3 In addition to the questions developed to specifically address the survey objective, identify demographic questions that will be needed to segment the results to make them more actionable. For example, do you want to understand responses by age? gender? years of membership? category of membership? total annual spend? STEP 3: REFINE THE QUESTIONS Once the list of questions (including demographic questions) has been developed, the wording of the questions needs to be refined to prevent the three most common mistakes in survey development Ambiguity, Bias and Complexity. The first mistake that people make when developing surveys involves the issue of ambiguity. For example, consider the following question taken from an actual member survey: How much red meat do you typically eat? This question is highly ambiguous and most likely generated highly misleading results. What is the time frame in question? per day? per week? And how is red meat being defined? Is pork included? And depending upon the answers provided, options such as a lot or not very much are clearly open to interpretation. Consider another question from the same member survey: How would you rate the quality of the food and service in the dining room? Again, the answers to this question could be highly misleading. Is the respondent supposed to address the food or the service? Ambiguity can be avoided by: 1. Developing wording that is clear and concise; 2. Providing definitions where there is any room for interpretation; 3. Isolating one issue within a question. Another common mistake, and the one that is arguably the most difficult to identify, is bias. Generally, a member survey is necessitated because of an issue or opportunity facing the club. Although difficult, survey questions must be worded in such a way to prevent any pre-conceived ideas regarding potential

4 solutions from biasing the respondent. Consider the following question taken from a member survey the author was asked to review: We need to make some decisions about our Clubhouse. The Clubhouse itself is too small and requires substantial physical improvement, and it s been a long time since we undertook a major re-decorating project. Do you favor: a. Remodeling the current Clubhouse b. Building a new Clubhouse c. Doing nothing The wording of the question is clearly biased in favor of doing something (options a or b), and the usage of words like too small and requires substantial physical improvement, would likely have driven respondents to option b. The committee members who developed this survey question were either consciously or sub-consciously steering respondents towards a particular response option. To ensure accurate and actionable response data, bias needs to be omitted. This survey question was reworded as follows: Considering the current clubhouse, which of the following statements most closely reflects your views? a. The current clubhouse should remain the same b. The current clubhouse should be remodeled (size will remain the same) c. The current clubhouse should be remodeled and expanded d. The Club needs a new clubhouse (current clubhouse torn down) The revised survey question, as well as the response options, uses neutral words and avoids biasing the respondent towards a particular option. The final mistake that is commonly made in developing survey questions involves the issue of complexity. Specifically, questions should not be so complex or outside of the typical members knowledge base as to be unanswerable. Consider the following example take from another member survey the author was asked to review: Last year, the food and beverage operation lost $85,000. Do you consider this to be acceptable? a. Yes b. No Most likely, members are professionally engaged in for-profit enterprises, where operating losses are generally not acceptable. However, in a club environment, certainly one with a large membership and/or a multi-million dollar food and beverage operation, an $85,000 loss may be perfectly acceptable. This is not an

5 appropriate question for a member survey, because most members would not have a proper perspective from which to evaluate this issue. This issue would be more appropriately addressed through a dedicated committee rather than through a member survey. STEP 4: FORMAT THE SURVEY The manner in which your member survey is formatted is almost as important as the wording of the questions. The three most common issues associated with the way a survey is formatted involve: 1. Length 2. Structure 3. Placement of Demographic Questions Length Although your survey will probably get more attention than the average piece of mail received by your members, if the survey appears to be daunting or too long, your members are less likely to take the time to complete it, resulting in disappointing response rates (more on response rates below). On the other hand, if your survey does not incorporate all of the questions necessary to gather the information needed, the time and effort used to create and mail the survey will have been wasted. As a rule, surveys should be about two pages in length (plus a cover letter). However, there are a few tricks of the trade, using different layouts and structures, that can maximize the information gathered while minimizing the space required. Structure Where possible, grids, tables and matrices should be used. Consider the following example A large club is evaluating its food and beverage operation. The general manager specifically wants to understand why a member would choose (or not) to eat at the club versus a restaurant. He develops the following questions: 1. When choosing a restaurant, how important is the quality of the food? Very important Important Unimportant 2. When choosing a restaurant, how important is the variety of menu selection? Very important Important

6 Unimportant 3. When choosing a restaurant, how important is atmosphere? Very important Important Unimportant These questions appear to be aligned with a narrow and deep objective, are objectively worded and the answers should provide actionable information. However, they are structured in such a way to take up an enormous amount of space. Consider this alternative structure: 1. Please indicate how important each of the following factors are when choosing a restaurant: Not Important Very Important Quality of food Variety of menu selection Atmosphere Clearly, this alternative structure asks the same questions, but in a more condensed layout. And, because this layout takes up less space, it will appear to be less time consuming for the respondent. Placement of Demographic Questions Most surveys include questions that ask for basic demographic/club usage information. These questions enable important post hoc segmentation analysis of the final results. For example, after a survey on food and beverage satisfaction is completed, you may find that the 75% of all members are very satisfied with the current operations indicating, perhaps, that no action should be taken. However, lets say that you included a question asking how much the member spends on food and beverage each month. Rather than simply accept the club average of 75%, now you can segment satisfaction based upon usage. You may find that 95% of members that spend less than $100 a month are very satisfied with the current operations, but only 40% of members that spend more than $500 (your best customers) are very satisfied with the current operations indicating that action probably should be taken. So, given the importance of asking basic demographic and usage questions for the purposes of post-hoc segmentation analysis, the question becomes, should these questions come at the beginning or the end of the survey?

7 Generally, questions regarding age, income, membership type, number of children, facility usage, etc., should be asked in the beginning of the survey rather than at the end. The rationale for this is two fold. First, these questions can serve as warm up questions for your respondents. These questions are generally easy for the respondents to answer and do not require much thought. It also helps to focus their thinking on their club experiences, and away from their bills, work, dinner, kids, etc. Second, there is a higher probability that these questions will not be answered if they are left to the end after respondents have taken the time and intellectual energy to complete a survey asking for their opinions across a variety of topics, they may begin to experience survey fatigue and not bother to provide answers to questions at the end that they may perceive to be less relevant. STEP 5: PRETEST PRETEST PRETEST No matter how carefully you word the questions to eliminate ambiguity, bias and complexity, no matter how clear the instructions, chances are that someone will misunderstand something in your survey. Misunderstandings result in misleading results and lower response rates. To minimize misunderstandings, be sure to pretest your survey. Although pretesting is very simple to do, and always improves the final survey, very few clubs that undertake surveys incorporate pretesting into their schedule. Pretesting simply means that you take some small sub-sample (10-15 is appropriate) of members and ask them to complete the survey. Generally this should be done in their own homes to simulate as close as possible the survey experience of the larger membership. Have these members write comments in the margins about words or concepts that were unclear. The feedback from the pretest is then used to further refine and improve the survey. STEP 6: MAXIMIZE THE RESPONSE RATE The number one issue that club managers raise when considering a survey is What can I do to maximize my response rate? Lets begin by setting expectations. The average response rate for mailed customer surveys in private industry is approximately 20%. Although no published statistics are available for private clubs, the author s experience has been that response rates generally range from 40-75% depending upon the size of the club and the survey topic. Response rates in this range will generally ensure statistical validity. There are three best practices that should be followed to maximize response rate:

8 Include a personalized cover letter. In this cover letter, you should include information regarding the survey objectives, how the information will be used, a few words about the importance of the member s input, and your phone number if the member has any questions. This letter should be addressed to the member, using their name, and not be addressed as Dear Member Include a postage-paid return envelope. This makes the process of returning the survey almost effortless making your respondents search for stamps, envelopes and addresses will always depress response rates. Include a return by date. This simple request will accomplish two things. First, members will be more likely to respond right away if they have a deadline. Otherwise, the survey may continually be moved to the bottom of the to do pile. Second, providing a date will minimize the time period over which you will receive responses you don t want to be receiving surveys three months later. Surveys represent an important (sometimes the only) tool through which to obtain factual data regarding the opinions of your club s membership. Although it is easy to react to the points raised by a vocal minority, you have a responsibility to the club s larger ownership/customer base to obtain factual data upon which to make important decisions regarding the future of your club. The six steps outlined here should provide you with a guide to develop and execute a successful survey that will generate meaningful results. Jennifer Lewis Priestley, Ph.D., is an Assistant Professor of Applied Mathematics at Kennesaw State University in Atlanta, GA. She works with club managers to help them get the most out of their data. jpriestl@kennesaw.edu.