the magazine of the Marketing Research and Intelligence Association ONLINE RESEARCH Canadian Publications Mail Agreement #

Size: px
Start display at page:

Download "the magazine of the Marketing Research and Intelligence Association ONLINE RESEARCH Canadian Publications Mail Agreement #40033932"


1 the magazine of the Marketing Research and Intelligence Association vueaugust 2009 ONLINE RESEARCH Canadian Publications Mail Agreement # The Net Gain for the Research Industry: What Recent Election Polling Has Taught Us Can Social Values Address the Biases in Online Access Panels? How Can Market Researchers Embrace the I in MRIA? In Search of Readers: A Brave New World for Researchers?




5 AUGUST 2009 COMMENTARY 6 Editor s vue 8 President s Message 9 Executive Director s Message INDUSTRY NEWS 27 People & Companies in the News 28 Qualitative Research Registry (QRR) EDUCATION AND PROFESSIONAL DEVELOPMENT NEWS 29 Where is the Research on Research? COLUMNISTS 30 Client-Side Insights by Eleanor Austin 31 Qual Col by Pierre Bélisle 32 The Innovation Accelerator by Margaret Imai-Compton SPECIAL FEATURE 14 The Net Gain for the Research Industry: What Recent Election Polling Has Taught Us Measuring projected outcomes against actual results, election opinion polling has tested, enhanced and validated online methodologies as the equivalent of traditional RDD methodologies. by John Wright FEATURES 18 Can Social Values Address the Biases in Online Access Panels? Environics has developed a social values segmentation to ensure that online research panels address biases in Internet populations and approach representativeness. by Barry Watson 21 How Can Market Researchers Embrace the I in MRIA? Faced with gaps that traditional research cannot fill, we researchers need to embrace intelligence. Real examples lead to key learnings to guide us all. by David Lithwick 24 In Search of Readers: A Brave New World for Researchers? Parallel telephone and online surveys by NADbank show that web panels based on the general population cannot currently provide projectable, representative samples. by Anne Crassweller RETURN UNDELIVERABLE CANADIAN ADDRESSES TO The Marketing Research and Intelligence Association L Association de la recherche et de l intelligence marketing 2600 Skymark Avenue, Bldg 4, Unit 104, Mississauga, Ontario L4W 5B2 Canadian Publications Mail Agreement #

6 COMMENTARY Editor s vue Stephen Popiel, PhD Many years ago, I watched the Gallup Poll move from a faceto-face methodology to a telephone CATI method. The data came in, fingers were crossed, the resident expert (Clara Hatton, along with Lorne Bozinoff and Peter MacIntosh) came in and, after carefully studying the numbers, gave CATI the blessing. I have spent the better part of this decade watching our industry struggle with online data collection (and in some cases assist in the struggle). This issue of vue shows how far we have come in our struggle to embrace and understand online polling and research. In our feature article, John Wright throws down the gauntlet and gives us a challenge. He reminds us that we are a results-driven industry and that online election results tend to mirror telephone results with uncanny accuracy. John s article is very thought-provoking but, as is often the case when science is in the throes of a Kuhnian paradigm shift, there are other views. Through a very carefully crafted set of experiments, Anne Crassweller shows us that different online samples give different results. Barry Watson uses social values data to understand just who completes online surveys with some surprising results. Lastly, David Lithwick shows us all how we can begin to more deeply embrace the I in MRIA. We at vue do not suppose that we have an answer to the online issue. However, between the process that David gives us and the insights (intelligence) from John, Anne and Barry, you the readers should be well equipped to help our industry find an answer. And when you do find that answer, please let us know. Until next month. Il y a bien des années, j ai observé le sondage Gallup passer de la méthodologie des entrevues en personne à la méthode de l ITAO d entrevues téléphoniques. Les données sont entrées, les doigts étaient croisés, l experte-résidente (Clara Hatton, accompagnée de Lorne Bozinoff et de Peter MacIntosh) est arrivée et, après avoir examiné attentivement les chiffres, elle a donné sa bénédiction à l ITAO. J ai passé la plus grande partie de cette décennie à observer notre industrie se débattre avec la collecte des données en ligne (et j ai parfois participé moi-même à cet effort). Ce numéro de vue montre jusqu à quel point nous avons progressé dans nos efforts pour accepter et comprendre le sondage et la recherche en ligne. Dans notre article de fond, John Wright jette le gant et nous invite à relever le défi. Il nous rappelle que nous sommes une industrie axée sur les résultats et que les résultats en ligne des élections tendent à refléter ceux des entrevues téléphoniques avec une exactitude troublante. L article de John nous porte à réfléchir sérieusement mais, comme c est souvent le cas lorsqu une science traverse un changement de paradigme kuhnien, d autres points de vue existent. Par le biais d un ensemble d expériences soigneusement réunies, Anne Crassweller nous démontre que des échantillonnages en ligne différents produisent des résultas différents. Barry Watson utilise des données sur les valeurs sociales pour comprendre quelles sont au juste les personnes qui remplissent les sondages en ligne avec des résultats surprenants. Enfin, David Lithwick nous apprend tous comment nous pouvons commencer à nous engager plus profondément dans le «I» de l ARIM. vue ne prétend pas offrir une réponse à la question des sondages en ligne. Par contre, entre le processus que David nous propose et les perspectives (intelligence) de John, Anne et Barry, vous, nos lecteurs, devriez être équipés pour aider l industrie à trouver une réponse. Et quand vous la trouverez, je vous prie de nous en informer. Au mois prochain. Stephen Popiel, PhD Senior Vice-President, Synovate Motoresearch Editor-in-Chief, vue (416) ext. 147 Stephen Popiel, Ph.D. Vice-président principal, Synovate Motoresearch Rédacteur en chef, vue (416) , poste vue August 2009


8 COMMENTARY / COMMENTAIRE President s Message David W. Stark,CIPP, MRIA President An update on activities and issues affecting the profession / Mise à jour des activités et enjeux affectant notre profession Since the theme of this month s issue is online research, I think it would be fitting to comment on the federal government s antispam legislation, Bill C-27, known as the Electronic Commerce Protection Act (ECPA). Summaries of ECPA and another antispam bill, S-220, appeared within the pages of vue last month under the Perspectives masthead. Perspectives is MRIA s new quarterly bulletin about our government relations activities (I hope that you enjoyed reading the inaugural issue). The House of Commons Standing Committee on Industry, Natural Resources, Science and Technology held hearings on ECPA in June. MRIA prepared a brief and is one of several organizations that have expressed an interest in appearing before the committee. Unfortunately, when the House of Commons adjourned for the summer on June 23, the committee suspended its deliberations. We expect to have an opportunity to present our views when the committee resumes its work in late September. Of course, ECPA itself could be derailed if a non-confidence vote on other government business were to trigger an election. Your guess is as good as mine as to whether this will happen. Some pundits speculate that Canadians could be going to the polls this fall, while others believe the spring of 2010 is a more likely scenario if an election is going to occur at all within the next twelve months. In any event, if ECPA dies on the Order Paper owing to an election, I think that we can expect to see a similar anti-spam bill introduced in a new Parliament by whichever party is in power. I also believe that anti-spam legislation has the potential to receive all-party support. Anti-spam and do-not-call laws are popular with consumers. Indeed, the amendments to the Telecommunications Act in late 2005, which mandated that the CRTC create and operate the national Do Not Call List (DNCL), received the support of all parties in the House. We understand that ECPA is intended to apply to electronic messages that solicit, and that survey and market research electronic messages are not within its scope. MRIA representatives Puisque la recherche en ligne est le thème du numéro de ce mois-ci, je crois qu il est de mise de commenter le projet de loi anti-pourriel C-27 du gouvernement fédéral intitulé Loi sur la protection du commerce électronique (LPCE). Des résumés de la LPCE et d un autre projet de loi anti-pourriel, le S-220, ont été présentés dans le numéro de vue du mois dernier sous la rubrique Perspectives. Il s agit d un nouveau bulletin trimestriel au sujet de nos activités de relations gouvernementales (j espère que vous avez aimé lire le premier volet). Le Comité permanent de la Chambre des communes sur l'industrie, les ressources naturelles, les sciences et la technologie a tenu des audiences sur la LPCE en juin. L ARIM a préparé un mémoire et a fait partie des organisations qui ont exprimé leur désir de comparaître devant le comité. Malheureusement, lorsque la Chambre des communes a ajourné pour l été le 23 juin, le comité a suspendu ses délibérations. Nous prévoyons avoir l occasion de présenter nos points de vue lorsque le comité reprendra ses travaux à la fin de septembre. Il se peut évidemment que la LPCE elle-même soit abandonnée si un vote de non-confiance sur d autres activités du gouvernement déclenchait une élection. Vos prédictions sont aussi bonnes que les miennes concernant cette possibilité. Certains journalistes prédisent que les Canadiens pourraient être appelés aux urnes cet automne, alors que d autres croient qu il est plus probable que ce sera au printemps de 2010 si une élection a toutefois lieu au cours des douze prochains mois. Quoi qu il arrive, si la LPCE meurt au feuilleton à cause d une élection, je crois qu on peut s attendre à voir un projet de loi anti-pourriel similaire lors du nouveau Parlement, quel que soit le parti au pouvoir. Je crois également que la loi anti-pourriel a le potentiel d obtenir l appui de tous les partis. Les lois anti-pourriel et sur les numéros de télécommunication exclus sont populaires auprès des consommateurs. En effet, des amendements à la Loi sur les télécommunications à la fin de 2005, mandatant le CRTC de créer et d exploiter une liste nationale de numéros de télécommunication exclus (LNNTE), ont reçu l appui de tous les partis en Chambre. Nous comprenons que la LPCE est conçue pour s appliquer aux messages électroniques de sollicitation et que sa portée ne s étend pas aux courriels relatifs aux sondages et à la recherche marketing. Des représentants de l ARIM consulteront tout de même des membres du personnel continues on page 10 voir page 10 8 vue August 2009

9 COMMENTARY / COMMENTAIRE Executive Director s Message Brendan Wycks, MRIA Executive Director Marketing research results and the media Résultats de la recherche marketing et les médias FROM ONE STANDARD-SETTER TO ANOTHER, WORKING TO IMPROVE PRACTICES I am pleased to report that MRIA has recently established a collegial, cooperative relationship with The Canadian Press that promises to benefit MRIA members, as well as editors, reporters and freelancers who use the authoritative resource, The Canadian Press Stylebook. Before reporting on a recent MRIA meeting with CP, allow me to give you some background context. In the November 2007 issue of vue, David Stark contributed an informative article headlined Marketing Research Results and the Media: From Pitfalls to Better Practice. He offered suggestions on what MRIA could do to elevate the media s standards for reporting on public opinion polls and other types of research. David noted that The Canadian Press Stylebook, which is an authoritative media resource, contains sections on reporting on polls and statistics. The section on polls, though, has not been substantially revised in some years. For example, it includes the following statement: An essential qualification is called the margin of error. It needs to be spelled out in stories on polls and its significance explained. When this statement was written, most polls were derived from probability-based samples. Today, many surveys are based on convenience samples in which margins of sampling error are unknowable and cannot be calculated. In his article, David mentioned that MRIA had been undergoing a significant review of its ethical code of conduct. Indeed, the updated code was unanimously approved by the D UN NORMALISATEUR À UN AUTRE POUR AMÉLIORER LES PRATIQUES EXEMPLAIRES Il me fait plaisir de signaler que l ARIM a établi récemment une relation collégiale de collaboration avec la Presse canadienne (PC) qui promet de bénéficier aux membres de l ARIM, de même qu aux rédacteurs en chef, aux journalistes et aux pigistes qui utilisent le Canadian Press Stylebook, un guide de style de la PC qui fait autorité. Avant de vous offrir un compte rendu d une rencontre récente entre l ARIM et la PC, permettez-moi de vous offrir un peu de contexte. Dans le numéro de novembre 2007 de vue, David Stark a présenté un article instructif intitulé «Les résultats de la recherche marketing et les médias : des embûches aux pratiques exemplaires*». Il a offert trois suggestions sur ce que l ARIM pourrait faire pour hausser le niveau des normes des médias concernant les reportages sur les sondages d opinion publique et sur d autres recherches. David notait que le Canadian Press Stylebook, une ressource faisant autorité chez les membres des médias, contient des sections sur la couverture de sondages et de statistiques. Or, la section sur les sondages n a pas été révisée depuis plusieurs années. Par exemple, on y trouve : «Une qualification essentielle s appelle la marge d erreur. Elle doit être indiquée clairement dans les reportages sur les sondages et il faut que sa signification soit expliquée.» Lorsque cette déclaration a été rédigée, la plupart des sondages découlaient d échantillons probabilistes. Aujourd hui, un grand nombre de sondages se fondent sur des échantillons de convenance dont la marge d erreur est inconnue et ne peut être calculée. Dans cet article, David soulignait que l ARIM procédait à une révision en profondeur de ses normes d'éthique et de son code de déontologie. En effet, le nouveau Code de déontologie et règles de pratique de l ARIM a été approuvé par le conseil d administration de l ARIM vers la fin du même mois. Comme vous le savez, une des clauses du nouveau Code exige que les membres s abstiennent de continues on page 11 voir page 11 9 vue

10 COMMENTARY / COMMENTAIRE President s Message, cont d will nonetheless consult with Industry Canada members of staff to obtain clarifications on some technical points, and we look forward to appearing before legislators. We will talk about the societal benefit of research and express our support for the legislation on the understanding that ECPA does not apply to legitimate survey and market research electronic messages. Of course, ECPA would and should apply to illegitimate mugging, sugging and frugging electronic communications, since the real intent behind these so-called surveys is to solicit. Muggers and suggers use the information collected to pitch products or services to respondents, while fruggers make appeals for donations. ECPA would also impose tough penalties on fraudsters (if they can be caught and brought to justice) who conduct identity theft and other scams via electronic messages. We have all received phishing s in our inboxes, ostensibly from ebay, PayPal or other reputable companies whose logos and websites are spoofed by scam artists to try to dupe consumers into revealing their private data. A couple of years ago, I received a phishing cloaked as a survey invitation. It was purportedly from Bank of America, and the link led to a spoofed survey site where recipients were asked to submit their names, account information, and other sensitive personal data as part of the brief survey. I dub this type of scam tugging Theft Under the Guise of research. If ECPA is adopted and enforced, it will go a long way to reducing electronic messages that solicit or are fraudulent, so that legitimate survey and market research communications do not have to compete with unwanted spam. ECPA would enhance the privacy rights of Canadians. MRIA is a strong supporter and champion of consumer rights and privacy, as evidenced by our Charter of Respondent Rights, our Code of Conduct and Good Practice, and our support of PIPEDA, the national DNCL, and the CRTC s telemarketing rules. d Industrie Canada afin d obtenir des clarifications sur certains points techniques, et nous avons hâte de comparaître devant les législateurs. Nous parlerons des avantages sociaux de la recherche et nous exprimerons notre appui à la loi en autant que la LPCE ne s applique par aux courriels relatifs aux sondages et à la recherche marketing. Il est évident que la LPCE s appliquerait et devrait s appliquer aux communications électroniques illégales telles que le mugging, le sugging et le frugging, puisque le véritable but de ces soi-disant sondages consiste à faire de la sollicitation. Les praticiens du mugging et du sugging utilisent les renseignements recueillis pour promouvoir leurs produits et services auprès des répondants, alors que les praticiens du frugging sollicitent des dons. La LPCE imposerait également des peines sévères aux fraudeurs (si on peut les attraper et les poursuivre en justice) qui effectuent des vols d identité et d autres fraudes au moyen de messages électroniques. Nous avons tous reçu des courriels d hameçonnage dans nos corbeilles d'arrivées, prétendant provenir d ebay, de PayPal ou d autres entreprises réputées dont le logo et le site Web sont imités par les spécialistes de la fraude pour tenter de leurrer les consommateurs à révéler des données personnelles. Il y a deux ans, j ai reçu un courriel d hameçonnage déguisé en invitation à participer à un sondage. Il prétendait être envoyé par la Bank of America et le lien menait à un site de sondage frauduleux où on demandait aux destinataires de soumettre leur nom, les renseignements sur leur compte et d autres données personnelles de nature délicate dans le cadre de ce bref «sondage». J appelle ce genre de fraude du «tugging» (theft under the guise of research / vol déguisé en recherche). Si la LPCE est adoptée et mise en vigueur, elle aidera considérablement à réduire les messages électroniques de sollicitation ou frauduleux de manière à ce que les communications de sondage et de recherche marketing légitimes n aient pas à faire concurrence à du pourriel non désiré. La LPCE améliorerait les droits de protection de la vie privée des Canadiens. L ARIM est un défenseur vigoreux et un champion des droits des consommateurs et de la protection de la privée, comme en témoigne notre Charte des droits des répondants, notre Code de déontologie et règles de pratique, et notre appui à la LPRPDÉ, à la LNNTE et aux règles de télémarketing du CRTC. David W. Stark, CIPP Vice President, Public Affairs, TNS Canadian Facts Privacy Officer, North America, TNS (416) David W. Stark, CIPP Vice-président, communications externes, TNS Canadian Facts Agent responsable de la protection des renseignements personnels, Amérique du Nord, TNS (416) vue August 2009

11 COMMENTARY / COMMENTAIRE Executive Director s Message, cont d MRIA board of directors later that month. As you may know, one of the code s new clauses requires members to refrain from making statements about margins of sampling error on population estimates when probability samples were not used. Finally, David made the following assertion in his article: Because researchers have reported confidence levels for all types of samples, including those that are not based on probability theory, the media have come to expect them, and The Canadian Press Stylebook pretty much demands them. We need to do a better job of informing the media (and research practitioners) about when it is appropriate to report a margin of sampling error and when it is not. It is time for MRIA to reach out to The Canadian Press and offer to provide input on the polls section for future editions of its authoritative stylebook. MRIA has reached out to CP. As standard-setting organizations, we have established a work plan, that should pay off by giving greater clarity, understanding, and discernment capability to members of the media on what constitutes quality research and how to report on it appropriately greater recognition for MRIA members as suppliers of highquality, credible research, in whose work the Canadian Press and other media organizations can have confidence. On July 14, MRIA president David Stark, our public relations consultant Peter Rose, and I met with Patti Tasko, a senior supervising editor at Canadian Press and editor of The CP Stylebook, and Malcolm MacNeil, a senior desk editor at CP. The significant outcomes of that meeting include the following: CP welcomes MRIA s direct input into the revision of the stylebook sections on polls and statistics. The hard copy version is due for reprinting in 2010, while the online version can be updated as soon as changes are agreed upon. MRIA will provide draft revisions to these sections, and we ll work with CP in an iterative process to finalize the copy. faire des déclarations sur les marges d erreurs d échantillonnage d estimations démographiques lorsqu ils n utilisent pas des échantillons probabilistes. Enfin, David affirmait ce qui suit dans son article : Étant donné que les praticiens de la recherche ont signalé des niveaux de confiance pour tous les types d échantillonnage, y compris ceux qui n étaient pas fondés sur la théorie des probabilités, les médias en sont venus à s y attendre, et le Canadian Press Stylebook les exige pour ainsi dire. Nous devons mieux informer les médias (et les praticiens de la recherche) au sujet des cas où il est approprié d indiquer une marge d erreur d échantillonnage et ceux où ce n est pas approprié. Il est temps que l ARIM approche la Presse canadienne et offre de fournir des commentaires au sujet de la section sur les sondages pour la prochaine édition de ce guide de style qui fait autorité. L ARIM a approché la PC. En tant que deux organisations de normalisation, nous avons établi un plan de travail qui devrait être fructueux en apportant : plus de clarté, de compréhension et de discernement de la part des membres des médias concernant ce que constitue la recherche de qualité et la façon appropriée de la communiquer, et une plus grande reconnaissance des membres de l ARIM en tant que fournisseurs de recherches crédibles de grande qualité auxquelles la PC et les autres entreprises médiatiques peuvent avoir confiance. Le 14 juillet, le président de l ARIM David Stark, notre consultant en relations publiques Peter Rose et moi-même, nous avons rencontré Patti Tasko, une surveillante principale de la rédaction et rédactrice en chef du Stylebook de la PC, et Malcolm MacNeil, un des principaux chefs de l information chez la PC. Voici les résultats importants de cette rencontre : La PC apprécie la participation directe de l ARIM à la révision des sections du Stylebook sur les sondages et les statistiques. Une réimpression du guide est prévue en 2010, mais la version en ligne peut être mise à jour dès que les changements ont été acceptés. L ARIM fournira des ébauches de révisions de ces sections et nous travaillerons avec la PC dans le cadre d un processus itératif pour compléter ce contenu. La PC est consciente que la marge d erreur de recherches fondées sur des échantillons non probabilistes, de convenance, tels que les continues on page 12 voir page vue

12 COMMENTARY / COMMENTAIRE Executive Director s Message, cont d CP is well aware that a margin of sampling error should not be provided nor reported for research based on non-probability, convenience samples, such as surveys derived from online consumer panels. Recognizing that online panels are rapidly becoming the dominant research methodology in Canada as they already are in other developed countries, CP is keen to have MRIA s assistance in differentiating among the scores of public opinion research-based media releases it receives, separating the wheat from the chaff, and differentiating quality research that is worthy of CP s attention. CP would like to see MRIA devise an alternate mark of quality statement for convenience sample studies, and is willing to work with us to come up with something that meets the needs of both organizations. Some components of a new mark of quality statement for online panel studies that were touched on in our preliminary discussion are - panel size and recruitment methods - quality controls deployed in panel management - number of invitations sent and number of completed surveys achieved in the study - a statement asserting MRIA membership (Gold Seal Research Agency; Basic Research Agency; or Client-Side Researcher Corporate member) and adherence to its standards of practice and ethics. In this connection, it was agreed that MRIA would provide CP with a list of its Corporate members on a regular basis. Stay tuned for further developments, as this important new relationship for MRIA and the industry is built and strengthened in the months and years ahead. sondages découlant de panels de consommateurs en ligne, ne devrait être ni fournie, ni signalée. Reconnaissant que les panels en ligne sont en voie de devenir rapidement la principale méthodologie de recherche au Canada, comme c est déjà le cas dans d autres pays, la PC est heureuse d avoir l aide de l ARIM pour établir des distinctions entre les multitudes de communiqués entourant la recherche sur l opinion publique qu elle reçoit, afin de séparer les bons des mauvais, et pour reconnaître la recherche de qualité qui mérite l attention de la PC. La PC aimerait que l ARIM élabore un autre énoncé relatif au sceau de la qualité pour les études basées sur des échantillons de convenance, et elle est prête à travailler avec nous pour en arriver à une formulation qui satisfait aux besoins des deux organisations. Certaines des composantes de ce nouvel énoncé relatif au sceau de la qualité pour les panels d études en ligne qui ont été abordées au cours de notre discussion préliminaires sont - la taille d un panel et les méthodes de recrutement, - les contrôles de la qualité utilisés pour la gestion des panels, - le nombre d invitations envoyées et le nombre de questionnaires complétés dans le cadre d une étude, et - une déclaration confirmant l adhésion à l ARIM (société de recherche membre corporatif Sceau d or ou de base, ou praticien de la recherche côté client) et l observation de ses normes de pratiques et de son éthique. À cet égard, il a été convenu que l ARIM fournirait régulièrement à la PC une liste de ses membres corporatifs. Restez à l affût d autres développements, à mesure que cette importante nouvelle relation de l ARIM et de notre industrie grandit et se solidifie dans les mois et les années à venir. * Ce titre et la citation connexe sont des traductions du texte original. Brendan Wycks, BA, MBA, CAE Executive Director Marketing Research and Intelligence Association (905) ext Brendan Wycks, BA, MBA, CAE Directeur général L Association de la recherche et de l intelligence marketing (905) poste vue August 2009


14 FEATURE The Net Gain for the Research Industry: What Recent Election Polling Has Taught Us Measuring projected outcomes against actual results, election opinion polling has tested, enhanced and validated the online methodologies developed for market research. The findings are consistent and undeniable: in terms of predictive accuracy, online and traditional RDD methodologies are equivalent. John Wright 14 vue August 2009

15 FEATURE As a euphemism for opinion research, polling has not only branded a handful of companies that are almost daily in the public domain; it has also branded the industry. Whether you believe that this branding has been a good or a bad thing (some of the market research purists most certainly cringe), the fact is that it has made many in business as well as the general public comfortable with research and has given a legitimate, collective voice to common folk on matters of the moment and what s on their minds. And whether you agree or disagree with how a question is asked or on what finding the media have decided to hang their story when they report on it, we should remember a few things that we sometimes take for granted: The freedom to poll on issues of the day or on political party positioning is as rare in the world as democracy. Polling results in the public domain are open to immediate worldwide scrutiny. With rare exception, those who are in the public domain almost daily with polling results take it very seriously indeed; and they are careful to ensure that no current or prospective client will go somewhere else because of embarrassment (or worse) by association. GROWTH & SOPHISTICATION We ve witnessed massive growth and sophistication over just two decades. When I started at Angus Reid Group in 1989, the company already had an established track record. It had accurately called Mulroney s second majority in 1988 as well as a number of other contests. But what really put the company on the map was the 1990 poll that predicted the upset of Liberal premier David Peterson and the beginning of the NDP interregnum under Bob Rae. Since that time, Angus Reid has polled in six federal elections, twenty-odd provincial elections, the Meech Lake and Charlottetown Accords, and the 1995 Quebec referendum. Between then and now, Ipsos Reid (formed when Angus Reid Group was acquired by Ipsos) has progressed from a monthly telephone omnibus with a national sample of 1,500 and, at most, a weekly custom telephone survey with a sample of 3,300 before a national election to what exists today: rolling telephone samples and releases, continuous nightly Internet polling, instant post-debate polling with 2,512 Englishspeaking and 637 French-speaking Canadians, and an Internet panel exit poll of 39,000 voters who tell us not only how they voted but why they did and what they wanted from their new parliament. And the record: Rival polls jostle and preen over results that put all of them either in or just around the margin of error on the outcome. Occasionally, there has been controversy. In the Canadian federal election of 2004, a small percentage of voters in Ontario and Quebec changed their minds after the last polling just a few days before the actual ballot; and the outcome was far from the predictions. That discrepancy elicited withering denunciations of dunce pollsters and their chicanery. But there wasn t anything that was broken or in need of repair except the timing: when the final polls are taken and the results released. Now Canadian exit polls, carried out on the day of the election with massive online panels, make sure that changes in voters intentions are caught, shown and justified to the media right after the polls close. The discrepancy between projected and actual results in the Canadian federal election of 2004 elicited withering denunciations of dunce pollsters and their chicanery. If there was ever a time to concede that the polling apparatus was dead in the U.S., it would have been on that night in 2008 when Hillary Clinton took the state of New Hampshire for the Democrats and pulled off a huge no one saw it coming upset. It s evident that the ghosts of the 2004 Canadian election had simply wafted over the border for the evening. While this upset victory focused attention on the apparent fallibility of polling, it was a mere blemish on the plethora of polls carried out during the campaign. A review of all of the 2008 U.S. Democratic and Republican primaries shows an astonishing degree of accuracy between the projected polling outcomes and the actual voting outcomes. In fact, in the final outcome of the presidential election, Ipsos came within a percentage point of nailing the results, with 23 other firms in close contention. All of this may be justification for pundits, politicos and wonks to rejoice in their accuracy and their trade and live to practise it again another day. But that s not the point. The reality is that the tools, methodologies and practices of pure market research and its practitioners are put under the greatest 15 vue

16 FEATURE scrutiny and subjected to the volatility of the marketplace for a brief period of time and are then expected to get the answer within a point or two of a verifiable outcome. The point is this: it works. NET GAIN Election polling constitutes one of the few instances in which the machinery of market research is pitted against the measured output of the consumer citizen. Using market research to predict the outcome of consumer purchasing behaviour is tough. But if an election serves as a wind tunnel testing the industry and its products, market research continues, by all measures, to prove itself a vigorous and robust enterprise. The net gain of political polling is provable validity for the industry and for new product offerings for non-election polling application. So let s focus on some learnings from the 2008 craft. The 2008 Canadian and U.S. elections were watersheds in many ways. In North America, we witnessed many firsts one of them being the rise of new survey methodologies, alternatives to traditional random telephone sampling. These alternatives included online interviews and robo (automated) calls as well as telephone hybrid surveys (i.e., a mix of traditional and nontraditional methodologies). This is not to say that traditional telephone surveys will become passé. Instead, they will be used in conjunction with other methodologies. With this changing context in mind, Ipsos increasingly uses both telephone and non-telephone metho d - ologies (including the Internet) in our polling. And we have not arrived at this mix lightly. For the last three elections in Canada and in the most recent U.S. election, we ran parallel voting intention questions on our telephone and online polls. We have had the opportunity to look at the differences over time, understand those differences, and adjust for them. When it comes to producing our pre-election day polls, Ipsos always uses the traditional telephone methodology. Let s just say we re old fashioned and comfortable with it, and some in the media continue to be suspicious of anything less. But we believe based on scientific experimentation and our experience with market and public clients that telephone and scientifically produced online surveys (not the pop-up kind) are essentially equivalent. And we re clear on this point to clients in Canada who collectively spend $170 million a year for our market research wares and insights. In this respect, Ipsos market and opinion research increasingly uses the Ipsos online screened and validated panel, which, when combined with scientific sampling procedures, produces results that are equivalently representative of the Canadian public a practice mirrored in the U.S. We could choose from many publicly released and mirrored questions taken by online and RDD methodologies, but two will have to suffice, given the limited space we have here to make the point. 16 vue August 2009

17 FEATURE TABLE 1: DO YOU THINK THE UNITED STATES IS WINNING OR NOT WINNING THE WAR ON TERROR? Online Telephone Winning Not winning Don t Know N/A 5 The first is from parallel telephone and online surveys that asked respondents, Do you think the United States is winning or not winning the war on terror? The results, shown in Table 1, are essentially the same with online and telephone surveys. The only real difference is the Don t Know response, which was simply volunteered in the telephone sample but was deliberately eliminated in online in order to force a choice. But what about open-ended questions? Table 2 presents an example of an open-ended pre-code telephone and issue list for online. FIGURE 2: WHICH ONE OF THESE ISSUES WOULD YOU SAY IS MOST IMPORTANT WHEN THINKING ABOUT THE CURRENT PRESIDENTIAL ELECTION CAMPAIGN? Online (10/27-29) Telephone (10/23-27) Jobs and the economy National security Leadership 10 9 Healthcare 9 8 Family Values 4 7 Representing change 5 5 Foreign policy 3 5 Taxes 6 4 None 3 1 DK/NS - 2 While there is some fluctuation between online and telephone methodologies in each of these examples, the differences are within acceptable limits: R 2 =.93 for the full battery and R 2 =.59, with the potential outlier (jobs and the economy) removed. Moreover, the results mirror the findings from RDD. In addition to these examples, Ipsos conducted an experiment throughout the recent U.S. election campaign to determine whether online research could be utilized to make electoral college projections. The result: accurate identification of the winner in 47 of 48 states (no Alaska, D.C. or Hawaii completes). The only state our projection missed was Indiana. Obama won that state 50 49, and our projection had McCain winning Ipsos predicted an Obama victory with 353 electoral college votes; the actual result was 365 electoral college votes. LEARNINGS What have we learned over the past five years from our trials in the wind tunnel of election campaigns? First, online research is increasingly becoming the standard for market and opinion research. Its use in the marketplace is now approaching half of all of the research undertaken, and media political campaign opinion and issue polling is increasingly accurate and acceptable. Second, while market forces are actually driving the application of new techniques and methodologies, election opinion polling is providing valuable feedback by validating and enhancing the measurable performance of the tools and the systems rendered for market research clients. In the 2008 U.S. presidential campaign, of the 23 polling competitors, almost half used nontraditional RDD approaches, which helped many compare real results with projected outcomes and reflect on what worked, what didn t, and what might work better. To me it really is like watching the Grand Prix in a wind tunnel. Third, and this is the debate occupying many industry scions in North America, is the view that online polling is not random and representative of the population. Current AAPOR (American Association for Public Opinion Research) and MRIA guidelines for online research strongly recommend including a disclosure methodology statement which instructs users to state that the standard margin of error cannot be applied. The 2008 electoral cycle in North America will likely be viewed as a watershed year in which traditional methodologies made their last stand as the sole purveyors of electoral and research truth. The debate about the application of a margin of error may be short-lived, as the equivalencies between the two methodologies are constantly being validated. In short, on this measure, you can t argue that one is not like the other and requires different rules particularly when the results continually demonstrate otherwise. John Wright is senior vice-president and managing director of the Public Opinion Polling Division for Ipsos North America and is the lead media spokesperson for the company. 17 vue

18 SPECIAL FEATURE Can Social Values Address the Biases in Online Access Panels? To ensure that online research panels address biases in Internet populations and approach representativeness, Environics has developed a social values segmentation that can guide recruitment and manage strategies to attract underrepresented groups and weight overrepresented groups appropriately. Barry G. Watson, CMRP Ten years ago, the Internet was on its way to becoming mainstream, but it retained some of its reputation as a niche medium used disproportionately by the technologically inclined. Most people could see the value of , but spending a lot of time online chatting, dating, shopping remained risky or nerdy or both. Today, the niche status of the web is long gone. Many of us spend almost all our time connected in some way, whether in front of a screen or on a mobile device. It is remarkable how quickly the web moved from being a techie bastion to a mainstay of communication for almost every segment of affluent societies. But despite the rapid and nearly universal penetration of the web, certain biases do persist in the populations that use the Internet intensively. Over the past few years, our industry has been working hard to understand the biases and distortions that emerge in online research panels. Declining response rates in telephone surveys as well as general social and technological trends lead many to believe that the future of marketing research is predominantly online. 18 vue August 2009

19 SPECIAL FEATURE Although the Internet is already being used for a range of research applications and has the potential to do much more, researchers have not yet determined how the web can yield the same quality of general population sample that the telephone gave us in its heyday. There is some disagreement about the quality of the data we are able to gather online from opt-in access panels. Some studies suggest online results are highly accurate and representative; others find online data skewed. Most rigorous studies comparing the results of online and telephone surveys are mixed. Unfortunately, thus far it has been difficult to find patterns in the divergences between telephone and online data sets. Efforts to reconcile these differences started by weighting demographics, then by looking at product consumption variables, attitudes and psychographics. But these analytic efforts have mostly revolved around individual items measuring attitudes and psychographics. It may be worthwhile to try a more integrated approach. One framework that might inform such an approach is an Environics social values segmentation. Environics has been developing its social values framework for 25 years, and the model has been validated in a range of contexts. It has been successful in differentiating social groups and in measuring social change over time. It has also proved valuable in cross-cultural applications. This social values system is based on about a hundred values, most of which are composed of about three individual items. We at Environics plot the trends on a social values map that has become familiar not only to people in our industry but also to the general public, through the publication of our founding president Michael Adams s books, Sex in the Snow and Fire and Ice, among others. The data that inform this framework are generally taken from residential-based samples and are collected over an extended time frame generally six to eight weeks which allows us to use multiple contacts and maximize the sample quality. No methodology is perfect, but we believe these baseline studies are about as good as is practical for a commercial research firm. Environics has used its social values method to help clients in the pharmaceutical, automotive, financial and tourist industries, and we have also applied the framework to public affairsrelated projects. We have used our social values data to develop eight segments or milieus, each of which has a distinct and coherent values orientation. The milieus are primarily determined according to respondents scores on values along an authority-vs-individuality axis. Socioeconomic characteristics also help to constitute the segments. Environics has been developing, testing and applying the milieus over a period of years; the ones we are currently using are based on data collected over several years in the United States with a total sample of 8,000. Over the last two years, Environics has had occasion to profile data across a number of panels from a number of companies. Not surprisingly, there are significant differences from one company to the next, but a few key patterns do emerge. One of the most notable of these patterns is that the milieu we ve labeled liberal progressives is consistently overrepresented in online research panels. Liberal progressives are an educated, affluent milieu scoring low on authorityoriented values and high on individuality-oriented values. They tend to be strongly engaged with the world around them and interested in self-expression. It would be reasonable to hypothesize that these outspoken people would be eager to participate in something like a research panel that would allow them to express their views and describe their experiences. Our data bear out this hypothesis. While the highly engaged liberal progressives are overrepresented in online panels, a couple of relatively disengaged segments are consistently underrepresented. The milieus we call the mavericks and the disenfranchised tend not to connect with research panels. The values of these two segments, which are composed predominantly of young people, suggest outlooks that are less curious and less idealistic than the liberal progressives. People in the maverick and disenfranchised segments either have fewer opinions (some are extremely disengaged from the world around them) or have only minimal interest in self-expression at least through a medium like a research panel. It takes a carefully tailored effort to successfully recruit and motivate these groups. Ironically, these groups are very heavy Internet users and very big consumers. Missing these people leaves a real hole in the market information provided by panels. A third underrepresented segment is one we label modern Middle America. These people tend to be middle-class and generally mid-career, with kids at home. This is a segment under a lot of pressure; it has little time for activities that are not highly and obviously rewarding. People in the modern Middle America segment who do participate in online research are likely driven primarily by incentives. The type of incentive program appears to have an impact as well. In the worst case, this group can be distressingly similar to the professionals who plague qualitative recruiting. The patterns I describe here regarding the over- and underrepresentation of various segments are based on U.S. data, but our team has found similar patterns when we have compared online and telephone data in Germany, France and the U.K. 19 vue

20 SPECIAL FEATURE Environics USA Milieus In addition to ensuring that we are tapping the right populations, another concern market researchers have encountered in online research is compliance. In a phone interview, the interviewer has some control over the pace and focus of the interview but what does it mean when a respondent, for instance, races through an online survey? Researchers have tended to be suspicious of interviewees who respond to invitations immediately and those who complete questionnaires too quickly. When we look at these behaviours through a values segmentation lens, however, we find that the segments responding to invitations most rapidly are, in fact, the segments most committed to fastidious rule-following: the sovereigns and the traditionalists. These milieus, which are more traditional, are the groups least likely to be pulling pranks or going through the motions just to receive an incentive. More likely, they feel a strong responsibility to respond to communications quickly; they are likely to believe that they signed up for the panel and now have a very serious duty to follow through promptly and conscientiously. Those who complete online surveys most rapidly are more likely to be the younger milieus raised on Nintendo games. With finely honed mouse skills and great claims on multi-tasking, they may legitimately be faster at online surveys. What do these insights from values research tell us about the path forward for online research? First, at the very least we know that values have a role to play in helping researchers develop quality metrics. These segmentations offer us helpful information about the biases that exist in online samples and a bias is only a problem if we don't know what it is. Second, we know that we have work to do in continually testing and evaluating our online research to ensure that it is as rigorous as possible. Although the Internet offers some economies of time and cost, the old fast-cheap-good rule still applies: your work can have any two of these characteristics, but not all three at once. There is no free lunch, even online: in order to achieve robust results in any research project, careful sampling and operating procedures are vital. Finally, the more our industry experiments, the more it becomes clear that there is no universal solution. Some online research methods work well for some applications; others achieve less reliable results. A horses-for-courses approach seems to be the best we can do for now. It goes without saying that online research is here to stay. The question is not whether to adopt it or reject it, but how to make it as sound and rigorous as possible. We market researchers have a role to play in educating our clients about the possibilities and challenges of online research. When the Internet was first becoming mainstream, there was a lot of lofty talk across a range of fields about how fast, cheap, democratic and easy everything would become online. In practice, the Internet is a hugely important medium but one with its own quirks and problems. Our industry will continue to study and adapt to these and hopefully continue to share our insights as time passes and as our tools evolve. In the absence of representative sampling, we are still missing some way to ensure that access panels approach representativeness. A battery of social values measures can fill that gap. The understanding of panel behaviour achieved through social values can guide recruitment and management strategies to attract underrepresented groups and allow overrepresented groups to be weighted appropriately. They can also form one of the metrics panels needed to allow researchers to judge the quality of the panels they are buying. Barry G. Watson, PhD, CMRP, is president and CEO of Environics Research and a past president of MRIA. This article is based on a presentation made at Net Gain 3.0 in January vue August 2009

Advanced Techniques for Work Search

Advanced Techniques for Work Search Advanced Techniques for Work Search Target your work search Develop your network Market yourself effectively Stand out in an interview Government Career Practitioners This workbook was developed to help

More information

Bridging the Gap between Public Officials and the Public

Bridging the Gap between Public Officials and the Public Bridging the Gap between Public Officials and the Public A REPORT OF THE DELIBERATIVE DEMOCRACY CONSORTIUM 2011 Tina Nabatchi Syracuse University Cynthia Farrar Yale University This work is the result

More information

Communications Toolkit

Communications Toolkit Communications Toolkit Part 1 of 3 This online download includes toolkit copy without the FAQ and Sample sections. Principles & Practices for Nonprofit Excellence in Colorado second edition TABLE OF CONTENTS

More information

Climate Surveys: Useful Tools to Help Colleges and Universities in Their Efforts to Reduce and Prevent Sexual Assault

Climate Surveys: Useful Tools to Help Colleges and Universities in Their Efforts to Reduce and Prevent Sexual Assault Climate Surveys: Useful Tools to Help Colleges and Universities in Their Efforts to Reduce and Prevent Sexual Assault Why are we releasing information about climate surveys? Sexual assault is a significant

More information



More information


EXPLORING IT GOVERNANCE AND STRATEGIC CONTROL OF WEB SITE CONTENT: A FIELD STUDY. Sylvie Héroux, Ph.D., M. Sc., CA* and. Anne Fortin, Ph.D. Cahier de recherche 2010-03 EXPLORING IT GOVERNANCE AND STRATEGIC CONTROL OF WEB SITE CONTENT: A FIELD STUDY Sylvie Héroux, Ph.D., M. Sc., CA* and Anne Fortin, Ph.D. Professors Accounting department École

More information

Knowledge ManageMent toolkit

Knowledge ManageMent toolkit Knowledge Management Toolkit This Knowledge Management Toolkit has been created by the SDC Knowledge and Learning Processes Division in cooperation with Agridea Lindau. This toolkit is available in English

More information

Digital Inside Out Creating a digital-first Britain

Digital Inside Out Creating a digital-first Britain Digital Inside Out Creating a digital-first Britain 1 Foreword When Tim Berners-Lee invented the World Wide Web in 1989, I wonder if he realised the immeasurable impact it would have in such a short space

More information


LEADBUYING 101 PRESENTED BY LEADBUYING 101 PRESENTED BY 1 2 INTRODUCTION The act of buying leads is a fairly easy task. A few quick phone calls could easily result in hundreds of leads being delivered to you on a daily basis. However,

More information

Business plan for the mobile application 'Whizzbit'

Business plan for the mobile application 'Whizzbit' Business plan for the mobile application 'Whizzbit' Tom Leleu Promotoren: prof. ir. Ludo Theunissen, dhr. Pascal Vande Velde Masterproef ingediend tot het behalen van de academische graad van Master in

More information

Lessons Learned from 197 Metrics, 150 Studies, and 12 Essays: A Field Guide to Digital Metrics

Lessons Learned from 197 Metrics, 150 Studies, and 12 Essays: A Field Guide to Digital Metrics Journal of Advertising Research Vol. 54, No. 1, 2014 Lessons Learned from 197 Metrics, 150 Studies, and 12 Essays: A Field Guide to Digital Metrics Stephen D. Rappaport

More information

The Past, Present and Future of Information Management Report

The Past, Present and Future of Information Management Report White Paper The Past, Present and Future of Information Management Report From a physical to digital information world how the data revolution is driving competitive advantage Business Information Solutions

More information

Substance Abuse. Implementing PACT360 in Your Community

Substance Abuse. Implementing PACT360 in Your Community How to Build BuildEffective EffectiveCommunity How to Community to Partnerships Partnerships to Prevent Teen Prevent Teen Substance Abuse Substance Abuse Implementing PACT360 in Your Community How to Build

More information

Making Smart IT Choices

Making Smart IT Choices Making Smart IT Choices Understanding Value and Risk in Government IT Investments Sharon S. Dawes Theresa A. Pardo Stephanie Simon Anthony M. Cresswell Mark F. LaVigne David F. Andersen Peter A. Bloniarz

More information

A claims handler s guide

A claims handler s guide A claims handler s guide Fraud Investigation A practical guide to the key issues and current law Autumn 2009 Fraud Investigation OVERVIEW In 2003, we published the first edition of Fraud Investigation

More information

Get the Right People:

Get the Right People: WHITEPAPER Get the Right People: 9 Critical Design Questions for Securing and Keeping the Best Hires Steven Hunt & Susan Van Klink Get the Right People: 9 Critical Design Questions for Securing and Keeping

More information

MONEY MORE FOR MORE GOOD. Bob Ottenhoff & Greg Ulrich

MONEY MORE FOR MORE GOOD. Bob Ottenhoff & Greg Ulrich MORE MONEY FOR MORE GOOD Bob Ottenhoff & Greg Ulrich Acknowledgements & Copyrights We would like to thank the following people for their invaluable contribution to the creation of this guide: Hope Neighbor,

More information

Business Analytics in Retail

Business Analytics in Retail Compliments of 2nd IBM Limited Edition Business Analytics in Retail Learn to: Put knowledge into action to drive higher sales Use advanced analytics for better response Tailor consumer shopping experiences

More information

Privacy, Technology and Criminal Justice Information

Privacy, Technology and Criminal Justice Information U.S. Department of Justice Office of Justice Programs Bureau of Justice Statistics Privacy, Technology and Criminal Justice Information Public Attitudes Toward Uses of Criminal History Information Summary

More information



More information



More information

Municipal Officials Views About Public Engagement

Municipal Officials Views About Public Engagement MAKING LOCAL DEMOCRACY WORK Municipal Officials Views About Public Engagement Research report By William Barnes and Bonnie Mann Copyright 2010 National League of Cities Washington, D.C. 20004 MAKING LOCAL

More information

Trends in Higher Education Marketing, Recruitment, and Technology

Trends in Higher Education Marketing, Recruitment, and Technology Trends in Higher Education Marketing, Recruitment, and Technology March 2014 In the following report, Hanover Research explores a variety of trends that have developed within higher education marketing,

More information

How To Write a Good PRD. Martin Cagan Silicon Valley Product Group

How To Write a Good PRD. Martin Cagan Silicon Valley Product Group How To Write a Good PRD Martin Cagan Silicon Valley Product Group HOW TO WRITE A GOOD PRD Martin Cagan, Silicon Valley Product Group OVERVIEW The PRD describes the product your company will build. It

More information

1 The Data Revolution and Economic Analysis

1 The Data Revolution and Economic Analysis 1 The Data Revolution and Economic Analysis Liran Einav, Stanford University and NBER Jonathan Levin, Stanford University and NBER Executive Summary Many believe that big data will transform business,

More information

A Simpler Plan for Start-ups

A Simpler Plan for Start-ups A Simpler Plan for Start-ups Business advisors, experienced entrepreneurs, bankers, and investors generally agree that you should develop a business plan before you start a business. A plan can help you

More information

Watching the middlemen. Brokerage services for micro-business energy consumers Nigel Cornwall and Robert Buckley, Cornwall Energy

Watching the middlemen. Brokerage services for micro-business energy consumers Nigel Cornwall and Robert Buckley, Cornwall Energy Watching the middlemen Brokerage services for micro-business energy consumers Nigel Cornwall and Robert Buckley, About Consumer Focus Consumer Focus is the statutory consumer champion for England, Wales,

More information

Dialogue by Design. A Handbook of Public & Stakeholder Engagement

Dialogue by Design. A Handbook of Public & Stakeholder Engagement Dialogue by Design A Handbook of Public & Stakeholder Engagement A publication by Dialogue by Design to accompany - Dialogue Designer the online engagement design system This handbook has been prepared

More information

Contingent Commitments Bringing Part-Time Faculty Into Focus

Contingent Commitments Bringing Part-Time Faculty Into Focus Contingent Commitments Bringing Part-Time Faculty Into Focus A SPECIAL REPORT from the Center for Community College Student Engagement Acknowledgments Our gratitude is unconditional. For their incalculable

More information


OECD DEVELOPMENT CENTRE OECD DEVELOPMENT CENTRE Working Paper No. 285 THE EMERGING MIDDLE CLASS IN DEVELOPING COUNTRIES by Homi Kharas Research area: Global Development Outlook January 2010 The Emerging Middle Class in Developing

More information