Measurement and Result Variations in Research with Data Collection through Online and Printed Questionnaire

Transcription

1 PMKT Revista Brasileira de Pesquisas de Marketing, Opinião e Mídia ISSN: (Impressa) ISSN: (On-line) Editor: Fauze Najib Mattar Sistema de avaliação: Triple Blind Review Idiomas: Português e Inglês Publicação: ABEP Associação Brasileira de Empresas de Pesquisa Measurement and Result Variations in Research with Data Collection through Online and Printed Questionnaire Variações de Mensuração e Resultado em Pesquisas com Coleta de Dados por Questionário On-line e Impresso Submission: Apr./29/ Approval: Oct./16/2015 Diana Lúcia Teixeira de Carvalho Doctorate student in Business Management at Federal University of Paraíba UFPB, with bachelor degree in Social Communication, major in Advertising, at Institute of Superior Education of Paraíba IESP. Member and researcher of the Center of Marketing and quantitative methods MEQAD. dinda_carvalho@hotmail.com Professional address: Universidade Federal da Paraíba - Campus I - Centro de Ciências Sociais Aplicadas, Programa de Pós-Graduação em Administração - Jardim Cidade Universitária, João Pessoa/Paraíba Brasil. Francisco José da Costa Doctorate in Business Administration from Fundação Getúlio Vargas School of Business Administration of São Paulo/FGV-SP, and a master degree in Business Administration from the State University of Ceará UECE. Bachelor s degree on Business Administration and his bachelor s degree on Statistics at the Federal University of Paraíba UFPB. Professor at the Department of Administration in the Federal University of Paraíba, and leader of the MEQAD. franzecosta@gmail.com Josemar Jeremias Bandeira de Souza Bachelor in Business Administration and Master in Philosophy in the Federal University of Paraíba UFPB. Professor of the Department of Applied Social Sciences in the in the same university. Member and researcher of MEQAD. josemarbandeira@msn.com

2 ABSTRACT This study examines variations and similarities among research results with data collected from online tools and paper and pencil instruments. The research is aligned with a number of efforts that have been developed internationally, and aims to bring a Brazilian contribution to the methodological discussion of management research. Besides the theoretical discussion, we applied 218 questionnaires, of which 107 on-line and 111 paper and pencil, in samples of a same group of a population of a church s members. The procedures were compared based on the analysis of measurement scales, descriptive measures, measures of association and prediction. The results showed that the online collection generates better data regarding measurement quality, and showed also some variation for results in different forms of collection for the descriptive, association and prediction measures. KEYWORDS: Data collection on-line, data collection paper and pencil, measurement, research. RESUMO Esse estudo analisa variações e convergências entre resultados de pesquisas com dados coletados com instrumentos on-line e impressos. A pesquisa está alinhada com uma série de esforços já desenvolvidas internacionalmente e se propõe a ser uma contribuição brasileira para o debate metodológico da pesquisa em Marketing e Administração. Além da discussão teórica, foram aplicados 218 questionários, dos quais 107 foram coletados on-line e 111 impressos, em amostras de um mesmo grupo de uma população de fiéis de uma igreja. Os procedimentos de comparação foram baseados nas escalas de mensuração, nas medidas descritivas, nas medidas de associação e de previsão. Os resultados sinalizaram que a coleta on-line gera dados de melhor qualidade de mensuração, além de apresentar variações de resultados nos diferentes meios de coleta para medidas descritivas, de associação e de predição. PALAVRAS CHAVE: Coleta de dados on-line, coleta de dados impressa, mensuração, pesquisa. 17, p , outubro,

3 1 INTRODUCTION Since the massification of internet use, especially from the mid-1990s, the use of online forms has become a recurring practice to carry out academic and market research (ALUJA et al., 2007; CAMPOS et al., 2011). Despite the massification of internet use being a world phenomenon, in Brazil, the number of users has increased at an average rate of 16% per year (IBGE, 2012), forming a distinctive communication profile that allows the generation of research interest information both directly (with forms containing questions and answers) and indirect (through tracking information about behaviors of users in virtual environments according to research interest). The use of the internet as a resource for gathering data has been part of academic and market practices more intensely from the 2000s, and there is a tendency for growth in the coming decades. Calliyeris and Las Casas (2012) emphasize the fact that the survey conducted through the internet is due to the democratization of cyberspace, presenting low cost of application, time saving, reach of large or geographically distant populations, automatic creation of the database, reduction of problems associated with the researcher (such as unavailability or typographical errors), besides the complete anonymity of respondents. These advantages are clearly leveraged a new trend of formatting research tools. On the other hand, the widespread use of internet as a data collection method raises questions about the validity of this type of research. In fact, if it may be argued that the data collection through the internet is effective in the contact with the populations studied (and here there are signs of both quantitative studies and substantial arguments that indicate the feasibility of application of it online), there is research evidence and contrary arguments, such as selection bias (those who have internet access are the only ones who can answer), the possibility that the respondent is not the subject of interest, among others. The most important reference point of this issue is the dimension of the outcome of interest. Indeed, if there is no evidence of variations when these are not necessary, then, it is the researcher s responsibility to decide which method will be used. If there is more consistency in data from one context or another, the researcher will certainly opt for the most consistent procedures. The construction of a body of knowledge on these issues has been slowly conducted in the field of methodology studies in academic and market research. Thus, studies in this direction are required, especially in the case of Brazil, where there are still very few studies according to an exploratory research. Having said that, it is believed that the development of analyzes with that focus brings a methodological contribution to quantitative research in Marketing and Administration, in line with a research tradition which is commonly internationally developed (regarding the assessment of conditions in the search results derived from the type of scale, the number of points and items, context of collection, etc.). Given this understanding, this study carried out a comparative analysis of data results collected in conventional quantitative research procedures (with definition of sample, measurement scales and structured questionnaire application). In other words, the objective of this article was to comparatively analyze variations in research results with data collected through the Internet and the direct application with printed questionnaires by assessing whether there is a result variation and possible superiority of one context over the other. This analysis was undertaken by means of a discussion of the literature and an empirical incursion. 17, p , outubro,

4 This work is developed in three stages. First, a literature review on the subject to an analysis of what has already been published and is considered more relevant on the topic and for the basis of the empirical study is performed. Then, the methodological decisions implemented in the empirical work are presented. In the subsequent section, the main results of the fieldwork and the analysis of variations and convergences observed are given. Finally, in the final considerations, there is a discussion of the results and their theoretical and practical consequences. 2 THEORETICAL REFERENCE In this section, the theoretical bases that guide the work are shown. The first part presents a discussion on the interference of environments and contexts when applying research tools in their results. In the second part, there is a summary of the existing literature on the verification of equivalence between data collection being done online and on paper. 2.1 INTERFERENCE OF THE ENVIRONMENT AND CONTEXT OF DATA GATHERING IN RESEARCH RESULTS In 2013, Brazil had 50% of its population as internet users (IBGE, 2014). This number, along with the cost advantages and speed described by Bethlehem (2010), is quite attractive for professionals and students who want to use the internet as a means of application of scientific and market research tools (CALLIYERIS; LAS CASAS, 2012). Discussions about this scenario have already been undertaken since the final years of the 1990s by analyzing the interference of the environments and contexts of application of research tools on the results. An example of the first analysis is based on the work of Couper (2000), who brought to discussion the problem of coverage, that is, the difficulty that the research conducted over the internet would have to focus on the necessary target audience for the truth of data collected, with the risk of computing the position of respondents who are not part of the population being surveyed. In a more recent research on the effects of the instrument in the quality of the results of online surveys, Vallejo et al. (2007) compared two research instruments, noting that, although the results suggest that "both online questionnaires were very similar in their printed versions" (in the original, authors call the printed version of 'paper and pencil ; we chose the word 'printed'), a type of questionnaire (SCL- 90-R Symptom Check List-90-Revised) presented higher reliability values than the other one (GHQ- 28 General Health Questionnaire). In their conclusions they suggest that although there is a need for further research that could confirm or not their work, there seems to be research tools that have better performance being done online than in other formats in terms of psychometric characteristics. Another important question concerns the cultural interference associated with the use of contexts. In this sense, Aluja et al. (2007) and Kongsted et al. (2007) discuss the problems of adaptation with technology that may vary either according to the location where the research population is, or just with the respondent's level of familiarity with the internet and its interfaces. In this discussion, several demographic factors can be considered to investigate whether there are elements arising from cultural or geographical features, which may interfere on the use of the internet and hence on the quality of the responses provided in online questionnaires. When concluding that the format of scale presentation influences on the quality of the data collected, 17, p , outubro,

5 Derham (2011) pointed to the resulting problem of choosing the type of scale used in the survey. Although the author has not tested online forms, considerations on the differences in the use of scales can give evidence that the way the research instrument is applied can interfere considerably with the quality of the results obtained by the researchers. In short, the changes may result from the context, type of metrics, cultural aspects, etc. The result of variations can achieve both measures (with variations of position measurements derived from contextual factors) or even in terms of psychometric properties of scales (this case generates risk for research as the psychometric weakness may indicate inconsistency results). Given this diversity issues (environmental influences, context and format in which the instruments are applied and the possible impact on the results), it is possible to see that the subject of the equivalence of results obtained through the instruments in online and printed formats requires investigations to better understand the scope, applicability and results of these types of questionnaires and also signal referential of results interpretation to academic or professionals users. 2.2 ONLINE OR PRINTED DATA COLLECTION Much research has pointed to the existence of similarity in performance between online and printed data collection formats (RITTER et al., 2004; VALLEJO et al., 2007; SPEK et al., 2008), indicating that the differences in the results obtained in both are insignificant. In the literature review of Aluja et al. (2007), the authors state that various studies have a tendency to identify that the results do not differ significantly among them. However, different nuances can interfere with descriptive measurements and with the mensuration (validity, reliability and factor structure) of data collected online, compared with collection made face-to-face (CAMPOS et al., 2011). In terms of data analysis, there is the study of Ployhart et al. (2003), in which it was found that the online context favors the psychometric structure and measures of association more than the printed context, when equal scales are applied in both formats. In contrast, the work of Ward et al. (2012) pointed to differences in the measurements results of variables and constructs average, what signals the idea that, depending on the context, the data may reveal different results of position measurements. The study of Kongsved et al. (2007) indicated that the online version of the questionnaire is superior with regard to the integrity of the data, despite it points out the fact that the population needs to become more familiar with the internet before the first option of the researcher is the application online. The observations of the work of these authors open a gap for new research to try to understand whether this familiarity already exists in a sufficient manner, so that there is a possibility of considering an application of online research as efficient as the printed version. Attention is drawn to the fact that the familiarity to which the work of Kongsved et al. (2007) refers to is the result of a series of social circumstances such as literacy of the population studied, access to technology and demographic factors (age, gender, socioeconomic position, among others). As commented about the study of Ployhart et al. (2003), the question of similarity among the validity, reliability and factorial structure of the applied measurement instruments online in relation to the printed format is still controversial. There is, therefore, differences between the findings of various studies, some claiming that online applications are equivalent to the printed applications (BRESSANI; DOWNS, 2002; HEWSON; CHARLTON, 2005; MEYERSON; TRYON, 2003), and others stating exactly the opposite, in other words, that there is no equivalence (BOOTH-KEWLEY 17, p , outubro,

6 et al., 2007;. WHITENER, 1995). It is worth mentioning the appointment of Whitener (1995), for whom the research results may be affected by how the research instrument is applied, with emphasis also to the social environment in which the respondent is. Similarly to this perspective, the research conducted by Campos et al. (2011) and Aluja et al. (2007), although having a tendency to full equivalence among samples of online and printed questionnaires, point, in accordance with Vallejo et al. (2007), to the need to verify the nuances coming from the statistical methods used to analyze the collected data. In other words, these authors raise the question about the possibility of different methods better absorbing the different application formats in relation to data collection. While acknowledging the limitations and diversity of approaches to check psychometric consistency (COSTA, 2011), it is understood that the equivalence of psychometric results is requirement for comparability of results in the contexts of analysis. Obviously, if the measure results are not close, data collected in the environment for which there was greater consistency in the results of factor analysis, internal consistency analysis and other measures of validity and reliability are safer. On the idea of effects caused by the social environment of response, Booth-Kewley et al. (2007) inserted elements of psychological order related to the respondent, and their research suggested that the answer issued through online forms leads the respondent to feel greater security and impersonality, and give answers that are more real. In their work, the authors observed that university students were more sincere when answering about their behavior in relation to sexual life and consumption of alcohol when responding electronic forms. Overall, the investigated literature shows that there are some important factors to consider to decide the format (online or printed) that will be used for applying a research tool: As for the target group it is appropriate to analyze the coverage capacity (COUPER, 2000) that each format has to reach this group of people and pay attention to cultural aspects, commented by Aluja et al. (2007) and Kongsved et al. (2007), that enable or hinder the online filling of a research instrument; As for the content it is necessary to verify if the questions may cause some kind of embarrassment to the respondent. The online format is the most appropriate for the questionnaires with more intimate questions (BOOTH-KEWLEY et al., 2007); The methods of collection and evaluation Campos et al. (2011), Aluja et al. (2007) and Whitener (1995) warn of the interference of the methods of analysis and research on the results of studies showing that some methods are better for data resulting from online application, while others are more suitable for data obtained through printed instruments. This research aims to make an assessment of the results derived from questionnaire applications in these two contexts, as shown below. 3 RESEARCH METHODS In this section, we describe the implementation of research fieldwork. For designing the empirical research, this work is based mainly on research from Mulyanegara (2011) and Casidy (2013), which analyze if the orientation of the organizational brand of a church affects the perception of the benefits associated with the church and the extent of participation of its members in religious activities. These two studies were used for presenting very detailed scales, what facilitated the achievement of desired 17, p , outubro,

7 results. Thus, the following constructs were selected: 'image perception' with six items regarding the perception of the extent to which an organization is committed to the generation of a positive image; 'perceived benefit', which relates to the benefits expected when participating in activities related to the church and it is formed by the dimensions of spiritual benefits (related to prayer and meditation activities, reaching a personal relationship with God), social (through interaction with other members/ participants) and about the purpose of life (religious faith giving meaning and purpose of life), each with three items; and 'participation in the church', also with three items, and related to the extent of participation of the believers in the activities in relation to the church (MULYANEGARA, 2011; Casidy, 2013). In the questionnaire, the items of image and spiritual benefit and life purpose were presented in format of statements to assess the level of agreement on the Likert scale with 10 points, 1-10; items of social benefits were presented as incomplete sentences, with completion by the intensity in a 10-point scale ranging from (1) terrible to (10) excellent; items related to participation were also presented as incomplete sentences, with completion by the intensity indicated in a 10-point scale ranging from (1) bad to (10) excellent. The items used were translated from the original articles into Portuguese and can be checked in item 4 of this article. In addition, the instrument was completed by multiple-choice questions relating to aspects of frequency in the church, computational competence, as well as issues related to socioeconomic and demographic profile of respondents, which included gender, age, marital status, education level and family income. A questionnaire on paper was prepared and, following the same format, the questionnaire was prepared on the internet by the Google Docs platform. Both questionnaires allowed the option of not answering some of the questions, so that in both the Google Docs platform and the printed format was possible to answer the questions in the sequence the respondents wanted. Such organization was intentional so that there was possibility of comparison also between both formats in terms of response problems (missing values and outliers). The printed instrument was applied in an active evangelical church in the capital of a state in northeastern Brazil. Then, in the same institution and from the same group, were collected the addresses of people who had not responded to the printed questionnaire to answer the online instrument that was subsequently sent by . From these groups, 218 questionnaires were obtained, which were tabulated in the SPSS software, and 217 questionnaires were valid for analyzing after preliminary processing of data (one was excluded by excessive incomplete answers), of which 107 were from the online collection and 110 coming from the printed collection. These approximate numbers ensure suitable conditions for analysis and comparison of results. In both cases, the sample was characterized as non-probabilistic by convenience. Regarding the profile of the sample, the results indicate an equivalence between the genders, in which 50.7% were 'feminine' (37.4% in the online collection and 62.2% in printed collection) and 49.3% were 'masculine' (62.6% in the online collection and 37.8% in printed collection). The marital status 'married' was 51.6% (65.4% in the online collection and 37.8% in printed collection), and the educational level of 'higher education' (complete or incomplete) was 51 2% (54.2% in the online collection and 47.7% in printed collection). Family income 'above 8, reais' (29% of the total, 17, p , outubro,

8 30.5% in the online collection and 27% in printed collection) and age 'between 31 and 40 years' (33.2% of the total, 43.9% in the online collection and 23.4% in printed collection) also had higher frequency among the answer choices. In general, variations were observed among the characteristics of the samples in relation to the means of collection, what may affect the results and point to a comparative weakness of the research. Moreover, it is also possible to take as an equivalence reference of sample the fact that people invited were from the same context to issue opinion on variables that they all access equally (church services). For purposes of analysis, considering the interest of examining divergences and convergences in results of contexts, we decided to use as many comparisons of alternatives as applicable, and that they have sense of interpretation and recurrence of use. That decision was justified on the assumption that, to ensure a result with greater certainty, as many observations as appropriate should be made (HAIG, 2005). That is, the intention was to announce results from differences or similarities after thorough verification. Based on this construction, the analysis of the measurement scales was conducted initially, indicating the examination of the factors and scores obtained in each construct separately, besides the internal consistency of results. In this evaluation, three contexts were considered: general, online and printed. Based on the figures in the general psychometric analysis, the variables were added as a way to generate a single measurement for each construct, so that the measures of the average of descriptive statistics, median, standard deviation, skewness and kurtosis could be verified. After these procedures, there was an investigation of the association among aggregate constructs through an associative analysis of the correlations of Pearson and Spearman. Finally, a predictive analysis of data by performing a multiple linear regression to assess the prediction of the construct participation in the church through the constructs perceived benefits was developed. All procedures are based on specialized literature (CONOVER, 1999; COSTA, 2011; HAIR et al., 2009; LATTIN; CARROL; GREEN, 2011) and, in each case, the analysis sought to emphasize convergences and divergences in relation to the two contexts of collection. 4 ANALYSIS OF RESULTS In this section, the results of statistical procedures are presented in the following sequence: first, the analysis of measurement items; then, the descriptive measures of aggregate variables (mean, median, standard deviation, skewness and kurtosis); third, the analysis of influence (analysis of parametric and non-parametric correlation); and subsequently, a predictive analysis using linear regression. In the end, these results are discussed. In a preliminary check associated with the organization of the mass of data before any operation, a verification of missing values and outliers (univariate) was made. In terms of missing values, in the online context response failures were observed in 11.21% of the respondents (who left at least one variable unanswered), while in the context of printed questionnaire this percentage was 38.73%. Concerning the outliers, it was found that responses from 24.37% of the online context had at least a tag that configured extreme value in quantitative variables, whereas the percentage of respondents of the printed context showed the same kind of response problem was 12.8%. This result indicates that in the online context there are fewer respondents who leave unanswered questions, but these 17, p , outubro,

9 respondents generate higher volume of outliers. In other words, for these verifications, it is possible to signal that respondents have different behaviors when answering the printed or online questionnaires. 4.1 MEASURAMENT ANALYSIS According to Campos et al. (2011), certain nuances of quantitative research may cause the quality measurement (factorial structure, consistency and reliability) of data collected online present differences compared to the collection with printed instrument. In line with this, the research that Vallejo et al. (2007) showed specifically that the reliability (measured by Cronbach's alpha) of the samples with printed collection has lower coefficients than samples collected in the online context. Thus, in this first stage of data analysis those comparisons were done and, when there were differences, the potential specificities associated with the collection of contexts were analyzed. Based on the data collected, and considering that the items in the original scales would have, hypothetically, reflective relationship with the latent constructs, conventional procedures of psychometric consistency analysis indicated in Costa (2011) were followed. Thereby, exploratory factor analysis was carried out through the method of principal components and the internal consistency of the constructs (for reliability of this statement) was verified. Table 1 shows the factor scores (required above 0.40 when there is good adhesion between item and factor) and the Cronbach's alpha (required above 0.60 when the number of items has measurement reliability) for the constructs considering the answers in general and by context (online and printed). TABLE 1 Factor analysis. CONSTRUCT IMAGE PERCEPTION (KMO=0,738, Bartlett: ²=183,46, 15 df, p<0,001) FACTOR SCORES ITEMS GLOBAL ( =0,682) ONLINE ( =0,754) PRINTED ( =0,580) I consider that the communication efforts of my church disseminate 0,734 0,818 0,637 consistent messages about its image in the community I m sure that church members have unique values that are 0,671 0,751 0,558 transparent to society I believe that my church has a strong spiritual appeal within the 0,763 0,727 0,811 community For me, the promotional materials of my church create an image 0,713 0,750 0,674 that is well understood by its members. CONSTRUCT SPIRITUAL BENEFIT (KMO=0,652, Bartlett: ²=237,82, 3 df, p<0,001) FACTOR SCORES ITEMS GLOBAL ( 0,794) ONLINE ( =0,886) PRINTED ( =0,566) I believe that my church is able to help me live up to my beliefs 0,756 0,840 0,664 I m sure that my participation in my church deepens my spirituality 0,875 0,924 0,749 For me, my church has great importance in the development of the 0,904 0,940 0,822 spiritual meaning of my life CONSTRUCT SOCIAL BENEFIT (KMO=0,622, Bartlett: ²=140,82, 3 df, p<0,001) FACTOR SCORES ITEMS GLOBAL ONLINE PRINTED By participating of the church, the opportunities I have to meet new people through my current friends are... 0,804 0,747 0,854 17, p , outubro,

10 For me, by going to church I can develop social relationships that 0,798 0,806 0,788 are... I believe that the network of friends I have made, by attending my 0,847 0,850 0,856 church, is... CONSTRUCT LIFE PURPOSE BENEFIT (KMO=0,666, Bartlett: ²=184,36, 3 df, p<0,001) FACTOR SCORES ITEMS GLOBAL ( 0,769) ONLINE ( =0,886) PRINTED 0,566) I m sure that my participation in my church makes me feel good 0,779 0,789 0,771 about myself I believe my church gives meaning to my life 0,832 0,901 0,604 I believe that my church is able to help me achieve the fullness in 0,881 0,909 0,843 life CONSTRUCT PARTICIPATION IN THE CHURCH (KMO=0,692, Bartlett: ²=206,67, 3 df, p<0,001) FACTOR SCORES ITEMS GLOBAL,794) ONLINE 0,856) PRINTED ( =0,727) My attendance in celebrations can be considered... 0,798 0,838 0,763 My participation in fundraising events for my church can be 0,872 0,909 0,831 considered... For me, my participation in the social activities of the church is... 0,865 0,906 0,832 Source: the authors. Among the constructs, only the 'image perception', in a first factorial extraction, presented two factors, having two items of this construct excluded ( I believe that people attend my church because of its reputation and I think that my church is well known among communities on its surroundings ). About the factor scores, it was observed that the values exceed the parameter defined in the three extractions (in the construct image perception ) with the lowest score of 0.558, which shows that the items of each construct have an acceptable factor structure for each extraction. But, overall, the changes are evident in each extraction, indicating that in the sample with online data collected, the factor structure is better than the data sample collected from printed questionnaires, except for two variables of the construct social benefit. In terms of internal consistency, the variations were differentiated, so that in certain extractions, alpha values that attest lack of internal consistency were obtained (constructs image perception, spiritual benefit and benefits of life purpose, but only in the sample collected with printed questionnaire). Generally, the alpha coefficients for the online environment have higher levels than the printed one, besides the construct social benefit. These results show that, in terms of psychometric consistency of scales, only the construct social benefit had better factor structure on the printed context of the questionnaire, whereas in other constructs it is clear that the psychometric structure is better at collection made through the internet. This result converges with the findings of Vallejo et al. (2007) and enables the affirmation that, if the items are actually consistent indicators of (alleged) underlying constructs, a plausible hypothesis is that the answer through the internet is more careful than the printed answer. Possibly, it helps to understand why the answers of the construct associated with social benefits are better measured in this context (printed), since the answers are given when socialization is occurring and not in an isolated context of response on the internet. As a first conclusion of the study, it can be said that there is evidence that in the measured constructs, the answers given through the internet are closer to the psychometric consistency level expected on 17, p , outubro,

11 the scales used here. In other words, the scores of the respondents to the constructs are more consistent and reliable in the sample originating from the collection via the internet than in the sample whose collection was printed and following the traditional way. 4.2 ANALYSIS OF DECRIPTIVE MEASURES From the psychometric analysis of the data, the variables were added as a way to generate a single measure to the construct. As aggregation method, the average of the scores of each respondent was used in each construct. In possession of the aggregate results, the descriptive measures of position (mean and median), dispersion (standard deviation) and general format (skewness and kurtosis) and context of data collection were verified as shown in Table 2. TABLE 2 Descriptive measures. MEASURE MEAN MEDIAN STANDARD DEVIATION SKEWNESS KURTOSIS IMAGE PERCEPTION General 8,34 8,6 1,26-1,05 1,04 Online 8,21 8,6 1,36-1,15 1,13 Printed 8,47 8,6 1,14-0,77 0,18 SPIRITUAL BENEFIT General 8,91 9,3 1,65-2,68 8,94 Online 8,77 9,3 1,84-2,49 6,51 Printed 9,03 9,7 1,44-2,88 13,23 SOCIAL BENEFIT General 7,25 7,7 2,15-0,95 0,57 Online 7,61 8,0 1,84-0,96 0,72 Printed 6,91 7,3 2,37-0,81 0,12 LIFE PURPOSE BENEFIT General 8,71 9,3 1,72-2,38 7,53 Online 8,39 9,3 2,05-2,22 5,68 Printed 9,01 9,3 1,25-1,6 2,59 CHURCH PARTICIPATION General 6,34 6,7 2,34-0,63-0,19 Online 6,48 7,0 2,42-0,23-0,25 Printed 6,21 6,7 2,27-0,59 0,04 Source: the authors. According to the results, it was observed that the constructs image perception, spiritual benefit and life purpose benefit showed high mean and median values (close to the maximum point on the scale) with a low standard deviation (below 2.0). This occurred both in general values and in the values for context, although the results of the printed collection have generated means that were a little higher than the means of the online data collection. The constructs social benefit and participation in the church obtained mean and median values slightly lower than that of other constructs, with a greater dispersion. In terms of context, both constructs showed mean and medians higher in the data from the online collection than in the printed data collection. 17, p , outubro,

12 To interpret the values of skewness and kurtosis, the reference used was that the SPSS values of these measures must be between -1 and +1 for signaling of normality of the variable in the population that originated the samples. Thus, it was indicated that only the constructs social benefit and participation in the church have values in the normal range, from the general measures and measures by context. The other constructs presented measures of format outside the normal range, also in the general answers and context. The difference is that while the construct spiritual benefit indicate measures (in absolute value) higher for the printed context, the construct benefit of life purpose had higher scores in the online context. The results solely by the measures of effect sizes suggest that the context of data collection seems to influence on the behavior of the variables, although it is not something very pronounced. In other words, the context of collection seems to have a marginal influence on measures of variables. Therefore, there was an attempt to determine whether there were significant differences specifically in the position measurements of the two contexts of collection and, when there was, it was examined whether there were or not results indicating greater efficiency of one context compared to the other. In a similar investigation, Vallejo et al. (2007) applied the t test to examine the mean of four different samples (two collected in an online context and two through printed forms), and the result found indicated no statistically significant difference among the means. Thus, to have a greater certainty as to the nature of this difference, considering the possibility of being a difference resulting from random factors and not from an external source of variation (which would be the collection context), as an additional manner, a hypothesis test for the means (parametric) and posts (nonparametric) comparison between the online and printed contexts was held through the technical analysis of variance (ANOVA) and of Wilcoxon-Mann-Whitney. The results are shown in Table 3. TABLE 3 Measurements of hypothesis tests. CONSTRUCT TEST F (MEANS) MANN-WHITNEY TEST ESTATISTICS P-VALUE ESTATISTICS P-VALUE Image perception 2,403 0, ,000 0,228 Spiritual benefit 1,337 0, ,500 0,246 Social benefit 5,811 0, ,500 0,049 Life purpose benefit 7,118 0, ,500 0,017 Participation in the church 0,743 0, ,500 0,292 Source: the authors. According to the results in Table 3, it is observed that the constructs having different means by application context are social benefit and benefit of life purpose, in both tests (a p-value smaller than 0.05 as reference to deny the null hypothesis of 'no difference' between the contexts is considered). This suggests that the perception of these constructs is shown depending on the context, since, given the design of the research, there is some homogeneity among responders in this study and there is consistency in measurements. As a second conclusion of the study, and taking into account the measurements and the results of the tests of significance difference, it is possible to understand that, in terms of descriptive measures, there is rather a derived variation of context collection, although in some constructs, these differences are more pronounced, and may come from random factors and not from the data collection context. 17, p , outubro,

13 4.3 ASSOCIATION ANALYSIS In order to overcome the methodological difficulties highlighted by Campos et al. (2011), Aluja et al. (2007) and Whitener (1995), and as a way to increase the number of methods used in this work, a verification of the association among the aggregate constructs was performed through the extraction of Pearson and Spearman correlation measures, whose results are shown in Table 4 (values not highlighted with an asterisk are statistically non-zero p<0.05). The correlation measurements measure the strength and direction of the relationship among variables, that is, they measure if the variables analyzed are correlated with each other and if this association is positive or negative. If there is no effect of context collection on the level of association of the variables, it is expected that the correlation pairs have similar behavior in terms of measure and sign. TABLE 4 Correlation measurements. IMAGE PERCEPTION SPIRITUAL BENEFIT SOCIAL BENEFIT LIFE PURPOSE CONSTRUCT MEASURE BENEFIT GEN. OL. PR. GEN. OL. PR. GEN. OL. PR. GEN. OL. PR. Spiritual Pearson 0,46 0,53 0,34 benefit Spearman 0,50 0,56 0,45 Social Pearson 0,29 0,35 0,28 0,28 0,51 0,12* benefit Spearman 0,34 0,41 0,33 0,29 0,48 0,15* Life purpose Pearson 0,57 0,63 0,45 0,75 0,88 0,51 0,24 0,40 0,17* benefit Spearman 0,53 0,54 0,52 0,78 0,83 0,73 0,25 0,39 0,18* Church Pearson 0,17 0,27 0,07* 0,38 0,45 0,31 0,41 0,50 0,34 0,31 0,46 0,11* participation Spearman 0,17 0,29 0,10* 0,29 0,37 0,24 0,42 0,52 0,33 0,25 0,42 0,10* Source: the authors. From the results, it is clear that participation in the church seems to have little influence on people's perception regarding the image of the church, given that the correlation between these two constructs is very low, both in general and in the online context (lower than 0.3). In the printed context this relationship is statistically null (p> 0.05), which indicates that even if they attend to church, this does not influence the perception of the believers about their image and their communication efforts. Participation in the church had higher correlation with the constructs for the spiritual, social and life purpose benefits, although these correlations are also low (around 0.3 and 0.4), especially in the general set of data and those arising from the context of printed collection. In fact, the correlation between church attendance and the benefits of life purpose is statistically null (p> 0.05) in the printed context. Another interesting result is about the correlation between the perception of image and the benefit of life purpose, which can be considered moderate (above 0.5), especially regarding the online context. By this statement, it is believed that the promotional messages of the church, which define its perceived image, should appeal to issues related to the benefits of life purpose that the church can offer to its believers. In an overall assessment, it has been signaled that the highest correlations are in the data that had online collection, noting that the few statistically null correlations were all associated with the data collected from the printed method. This allows reaching a third conclusion of this study, namely, that the association between constructs, for parametric investigation (linear) or nonparametric, depends on the context collection, which evidenced that the printed collection context depreciates the 17, p , outubro,

14 correlation measures, some of which are expected according to specialized literature. Obviously, considering what has been pointed out in items 1 and 2 of this article, the results of the online collection seem safer, given that the psychometric structure of the measurement scales in this context was superior. 4.4 PREDICTIVE ANALYSIS In this fourth stage, the potential factors of influence of each of the constructs of benefits on participation are analyzed by checking if there is a difference of predictive capacity in each context. Based on the reports of the specialized literature (MULYANEGARA, 2011; CASIDY, 2013), it was expected that the levels of perception of the three benefits would influence on the level of church participation in a positive way (that is, the greater the perceived benefits, the higher the level of participation). The method of predictive analysis occurred through a multiple linear regression with estimation through linear standard modeling. Based on this understanding, the social, spiritual and life purpose constructs were used as independent variables to predict participation in the church (the dependent variable of the model). The results are shown in Table 5 (considering the objectives of the work, burdening the text with the information about behavior of the error in attending the classic assumptions of modeling applied here was avoided, which are linear relation, randomness, homoscedasticity and normality of errors). TABLE 5 Results of multiple linear regression. PREDICTIVE GENERAL (R²=0,250) ONLINE (R²=0,332) PRINTED (R²=0,204) VARIABLES β t p-value β t p-value β t p-value Spiritual benefit 0,276 3,004 0,003-0,053-0,288 0,774 0,331 3,274 0,001 Social benefit 0,331 5,337 0,000 0,389 4,131 0,000 0,323 3,669 0,000 Life purpose benefit 0,021 0,230 0,818 0,348 2,012 0,047-0,106-1,041 0,300 Source: the authors. From what can be seen in Table 5, the models show significant influence of social benefits (in general, online and print contexts) and spiritual (in general and printed contexts) in predicting participation in the church. The influence of life purpose benefit is configured as statistically null or very low (only emerged in the online context). In general, the dependent variable has only between 20% and 35% of its variance explained by the independent variables. The results clearly show that the level of explanation dependent of the context of collection is low. This elicits the fourth conclusion of this research: the potential benefits of influencing factors perceived by believers of a church on their participation in religious activities emerge depending on the context in which the believer provide data on their perceptions. Overall, the participation is better explained by the perceived benefits in the investigation of the virtual context. Considering that there is a greater consistency in the measurements of the online environment, this certainly seems to be the level closest to real of the degree of explanation of the response variable. 4.5 DISCUSSION OF RESULTS Resuming in this section the results presented above, initially it is noted that, in terms of psychometric characteristics (and assuming, according to studies where scales were extracted, that these properly 17, p , outubro,

15 measure the constructs), the online environment generated more consistent data in terms of measures (first conclusion). This result is in congruence with the study of Ployhart et al. (2003), which achieved greater consistency of action in the online than in printed context (the authors used measures of awareness, socialization and emotional stability). In that same study, the association measures of the online environment also proved to be higher than in the printed context (PLOYHART et al., 2003), what converges with the results of this research, since the correlations among constructs of the printed context were lower than expected (third conclusion). According to the results of measurements of aggregates constructs, and taking into account the results of difference of significance tests, it was realized that there were constructs in which major differences emerged, indicating that, depending on the context of response, perception about these constructs changes, though not very intensely (second conclusion). Differences in the results of measurements coming from printed online contexts are also found in the study by Ward et al. (2012) for the constructs of leisure activities, with greater measures in the printed context. The study by Aluja et al. (2007) likewise saw differences in average for online and printed formats in most of the used scales. Finally, regarding the predictive analysis, it was noted that there is a disagreement about the benefits that influence the participation of respondents in the church (fourth submission), given that, for online respondents, the spiritual benefits and life purpose are those which lead them to join the church; but for respondents of the printed collection, the spiritual and social benefits are set as conditions for their participation. Thus, it is understood that the response context influences the perception of what benefits have more impact on the participation of these believers in their church. The predictive outcome of the printed context can be compared with the study by Mulyanegara (2011), which was only applied through a printed questionnaire and scores signaled that the spiritual and social benefits are perceived as leading to greater participation of the believers in church activities, as it was shown in this study; however, such a comparison cannot be done for the online context, given that the studies on the relationship between perceived image and perceived benefits of a church, which motivated this research (MULYANEGARA, 2011; CASIDY, 2013) were not applied in an online context. In the results presented, it was initially possible to infer that the application of a research context can influence on the psychometric, descriptive and association measures, regardless of the answers coming from the same group of people (which ensures consistency between the two response groups). Indeed, Whitener (1995) points out the possibility that the results of a research are influenced by the context of application of the instrument and the social environment of the respondent. Thus, secondly, it was possible to assume that the online context, as it demonstrates better measurement, appears to be a format that can encourage more careful responses, as if the respondents were free of external pressures (as the presence of the questionnaire applicator, or the time available to answer questions, for example) that might interfere when filling in the questionnaire in printed form. However, it is obvious that this result is not universal, although it is a strong signal for surveys with similar designs to this. 5 CONCLUSION In this study, the changes in quantitative basis of research results were analyzed in order to compare variations and similarities of results with data collected through online instruments on the internet 17, p , outubro,

16 and printed instrument by using the traditional method. With the review of the literature and the empirical study conducted at the end of the survey, it was possible to achieve the set objective in full. It is noteworthy that the instruments (which were applied in a population belonging to the same group) presented result variations, especially in the psychometric structure of the scales used, in statistics of association and predictive analysis. According to results presented in section 4, there were signs in the scales that measurement is more consistent in the application through the internet; moreover, the association measures of aggregate variables, the associations (correlations) statistically null when observed, occurred on data collected with a printed instrument. Specific variations were observed in the descriptive measures, which suggested that the context does not interfere more intensively in these measures. As a result, the analysis of prediction performed by regression (section 4.4) still observed variations of the forecast of perceived benefits of the participation of the believers in the church. Although the quantitative study of Marketing and Administration does not ensure certainty as to the comparative quality of data collection in relation to the context, the strongest signal is that the online collection generates better data in relation to the printed collection. Possibly this is due to the obligation that subjects feel in the data collection which is carried out face-to-face, to give an answer to those who are in front of them waiting for the questionnaire, which does not occur in the online response. If this alleged pressure for a response is real, then it is possible that this creates a greater number of careless answers given only by the obligation to deliver the questionnaire completed in full. These findings have practical implications related mainly to data collection actions on research of interest to academics and Marketing managers. In fact, if it is recognized that there variations of result in the context of collection, then the results analyzed in aggregate need to be confronted with the results of the different contexts that generated the data. In case of variation, it is necessary that the researchers endeavor to analyze the reasons for such variation, or analyze which results should equally refer to the explanation of their research problem or decision. According to what was signaled here, the evidence is that the data collected online are more consistent, which also suggests that in the practical world, using this context (internet) to conduct empirical research with similar design and public to this research is advisable. 6 RESEARCH LIMITATIONS AND SUGGESTIONS FOR NEW RESEARCH It is also necessary to recognize that this research has some weaknesses. It is understood that the main limitation is the fact that the results are derived from a source with very specific characteristics (followers of an evangelical church analyzing its service). This can obviously generate a response bias, especially because the invitation to answer this online application was sent by by a minister of the church to which respondent groups belong to. Another aspect that may have weakened the research relates to the socioeconomic and demographic characteristics of the samples. Although it has been taken as equivalent reference of the sample the fact that all participants are from the same group (which means it can be considered as they are able to evaluate in an equivalent way the constructs measured in this study), the ideal situation for comparison was one in which the other characteristics were also more or less equal. This indicates, therefore, a need for further studies to achieve better control of these variables so as to enable a comparison closer to the ideal. 17, p , outubro,