OPEN TEXT VS. WORD CLOUDS. Your guide to making the most of your qualitative data

Transcription

1 OPEN TEXT VS. WORD CLOUDS Your guide to making the most of your qualitative data

2 THE TROUBLE WITH QUALITATIVE DATA Identifying patterns in open ended questions Qualitative data, which is collected via free-form survey questions like open text or essay boxes, can be a gold mine of information. It can also create a mountain of data that seems impossible to scale. While word clouds can show you an instant overview of common words and phrases from your responses, open text analysis will give you invaluable insight that automatic tools like word clouds just can t match. 2

3 AN ILLUSTRATIVE EXAMPLE: MOVIE SCENES Revealing the power of open text analysis To illustrate the difference, we surveyed our employees about their favorite movie scenes, then examined the data using both word clouds and open text analysis. As expected, we discovered that word clouds could give us a very basic overview of the responses, but that was it. The only way to see more significant trends was to perform open text analysis. Here s what we did, and how you can do the same thing with your qualitative data. 3

4 OUR MOVIE SURVEY We asked just two questions, both with the essay answer option: 1. What is your favorite scene from a movie? 2. Please describe the scene what happens? Who are the characters? How does it figure into the plot of the movie? How does it make you feel during and after watching it? The goal of the survey was to determine whether or not there were any commonalities among the answers. We were curious if violent scenes, romantic scenes, triumphant scenes, or something else were the ones that resonated with our colleagues. We distributed the survey via to all SurveyGizmo employees, and got a 34% response rate. While not statistically significant, the results are interesting as a snapshot of our office s movie preferences (and as an illustration of how to work with long form survey responses). 4

5 RESULTS VIA WORD CLOUD The word cloud of question #1 looked like this: Letting people write out their choices free form means there aren t many common words. But what we can see is that Harry and Potter are slightly larger (and therefore appear more often). We can use this in our analysis later. 5

6 RESULTS VIA WORD CLOUD The word cloud of question #2 looked like this: You can see that Eli, Sunday, and Plainview, all appear a disproportionate amount. What does that mean? 6

7 RESULTS VIA WORD CLOUD If we just trusted the word cloud results, we d figure that those three words were the most popular ones in our respondents descriptions. This could lead us to assume that that film was the most loved, or that movies with characters named Eli or Plainview resonate most strongly with our staff. Both these conclusions are false, but we had to dive into the individual results to find this out. It turns out one respondent typed in the full dialogue of their favorite scene from, There Will Be Blood, including the names of the characters who were speaking, Eli Sunday and Plainview. 7

8 RESULTS VIA WORD CLOUD This had a HUGE impact on our word cloud results, but we can now filter out that particular response and get an updated word cloud: Unfortunately now our word cloud isn t showing us much at all. Time for something more robust! 8

9 OPEN TEXT ANALYSIS In SurveyGizmo, you can easily use bucketing to quantify patterns in your qualitative questions. Word clouds can be a good first step in this process, because they can help you choose what buckets you want to create. After you collect responses, you go to the Reports option under the Results tab. Choose Open Text Analysis, then begin your analysis of each question you want to examine. In our example these are just essay boxes, but open text analysis also works on multiple choice questions with an Other please specify option, or any other question for which respondents can enter text. 9

10 OPEN TEXT ANALYSIS The next thing you need to do is select your Buckets, which are the categories into which you ll manually drop your responses. Below are the ones I chose for the movie answers, based in part on the word clouds we looked at earlier and partially on my own quick glance at the answers. 10

11 OPEN TEXT ANALYSIS Now the hard part: you read (or scan) each response and choose which bucket(s) it belongs in. This can be time consuming, but it s really the only accurate way to determine which categories your responses belong in. Here are two of my responses and the buckets I put them in: 11

12 WHY CHOOSE OPEN TEXT Reading all of the responses people type in to your open text questions can be time consuming, but the reporting power that it gives you makes it completely worth it. The word clouds we looked at earlier were sort of neat, but they didn t really give us insight into how many people preferred a particular kind of movie scene. However, after bucketing all the responses I was able to create these reports: 12

13 WHY CHOOSE OPEN TEXT Similarly interesting results come from Question 2, in which the respondents described their favorite scene: If we needed to, we could then do another round of bucketing on questions that fall into the fighting and death category to try and learn more about patterns within those responses. Open text analysis offers you practically unlimited ways to examine your qualitative data. 13

14 Open Text vs. Word Clouds WORD CLOUDS + TEXT ANALYSIS Open text analysis helps us create quantifiable segments of our qualitative data, but word clods can do a great job of helping us to figure out what those segments should be. If you need to choose just one method of reporting, open text analysis will give you much better data than a word cloud. But ideally you could create a quick word cloud and then use the patterns it reveals to create the buckets for your text analysis. It s the best of both worlds! Open Text vs. Word Clouds 2015 Widgix, dba SurveyGizmo 14

15 Ready to start experimenlng with word clouds and open text analysis yourself? Try SurveyGizmo free for 7 days: START A TRIAL 15