REVIEWING THREE DECADES WORTH OF STATISTICAL ADVANCEMENTS IN INDUSTRIAL-ORGANIZATIONAL PSYCHOLOGICAL RESEARCH

Transcription

1 1 REVIEWING THREE DECADES WORTH OF STATISTICAL ADVANCEMENTS IN INDUSTRIAL-ORGANIZATIONAL PSYCHOLOGICAL RESEARCH Nicholas Wrobel Faculty Sponsor: Kanako Taku Department of Psychology, Oakland University Abstract The purpose of this study was to examine statistical improvements and professional experience in the past 30 year in Industrial-Organizational Psychology using the type of statistical analysis conducted. Eighty-five articles were collected using the years 1988, 1999, and 2008 from The Journal of Applied Psychology, with each different statistical analysis coded as a nominal 1 or 0. Current research was hypothesized to contain more complex statistical analyses than previous decades; also professional experience should be related to the complexity of the analysis conducted. The first hypothesis was supported with an upward trend in SEM use from 1988 (16.28%) to 2008 (56.52%). The second hypothesis was partially supported, a t-test demonstrated the groups using (n = 7) and not using (n = 24) SEM in 1988 significantly differed t(24.50) = 2.14, p <.05. Applications for improving research in the field are discussed along with an array of other statistical trends. Introduction The subfield of Industrial-Organizational Psychology (I-O Psychology) is relatively new to the field of psychology, it takes psychological principles and applies them to the workplace. In this review, the focus was on the statistical analyses in a randomly selected number of articles using the Journal of Applied Psychology (JAP). The statistical analyses from each article were

2 2 analyzed to look for trends of statistical improvements over thirty years using the years 1988, 1998, and The majority of previous research focused on analyzing the content in each area, rather than the statistical methods applied to each of the studies (Cascio & Aguinis, 2008). Previous research also focused on the different subfields of I-O psychology, rather than I-O Psychology as a whole (Aguinis, Pierce, Bosco, & Muslin, 2009). Furthermore, the validity of previous research on this topic is questionable due to the lack of random sampling critical to scientific research (Aguinis et al., 2009; Austin, Scherbaum, & Mahlman, 2002; Cascio & Aguinis, 2008; Duriau, Reger, & Pfarrer, 2007). Austin et al. (2002) conducted a large study analyzing research articles (n = 609) looking at trends and characteristics of empirical research in JAP. The researcher noted significant upward trends on the percentage of use for the statistic Confirmatory Factor Analysis (CFI) and Regression from the years 1980 to CFI had a percentage of use index (PUI) of 1.2% in 1980, 6.6% in 1990, and 16.4% in Regression had a PUI of 14.5% in 1980, 33.8% in 1990, and 46.3% in The researcher also noted a downward trend of the statistic ANOVA with a PUI index in 1980 at 51.2%, in 1990 at 33.8%, and 2000 at 28.4%. Austin et al. (2002) demonstrated that the field of I-O Psychology has changed drastically over the years in terms of measurement style and technique. This research leads to the first hypothesis in the current study; current research will contain more complex statistical analyses than previous research. The current study proposes that the results will follow along the lines of previous research in the field (Austin et al., 2002; Cascio & Agunis, 2008; Duriau et al., 2007). An example of a complex statistical analysis would be a CFA, because of its ability to simultaneously sort out the effects of a number of variables, and it requires statistical software to

3 3 produce. This study is specifically looking for an upward trend of statistical testing with the most basic analyses being in the earliest year (1988), and the most complex being in the latest year (2008). The next part of the study involves the role of research experience with the frequency of a statistical analysis conducted. The current study proposed that researchers who have longer experience will be more likely to use simpler statistical analyses. Many researchers may not have had the benefit of learning advanced statistical software because by the time these new complex statistical software programs were readily available many researchers were already conducting studies for a number of years, and possibly they were satisfied with their prior statistical knowledge. This logic leads to the proposal of the second hypothesis; researcher s length of experience should be related to the type of statistical analysis conducted. The current study created a variable termed professional experience, which represented the amount of experience publishing scholarly articles the author had. With this variable professional experience the current study would be able to tell if the number of years the researcher was publishing articles is related to what type of statistical formula they used. Method Sample Articles were selected from JAP using the database PsychInfo by using SPSS statistical software to randomly select 28% of journal articles in each year 2008 (31 of 110), 1998 (23 of 82), and 1988 (31 of 82). JAP was selected because it was the only journal in the field of I-O Psychology that met the following requirements: (a) the journal contained articles that had statistical analyses, (b) the journal contained articles dating back to the year Many of the

4 4 articles from our sample contained multiple experiments so the 85 articles in the study produced 115 usable studies, in 2008 (n = 46), in 1998 (n = 26), and in 1988 (n = 43). Another article was only used in the total analysis if: (a) the analysis was in the context of another experiment in the same study, (b) the experimenter was testing an analysis with a different sample. Pilot studies to the experiment were not included as another usable study. Procedure The content of each journal was analyzed for the statistical content and if there was a specific analysis in each study it was given a nominal 1 in an excel chart containing the 9 grouping variables discussed later, but if a statistical analysis was not applied a nominal 0 was given to that specific grouping variable. After the data was analyzed from each article the excel chart was copied to SPSS statistical software 15.0 to be analyzed. To assess the variable professional experience the researcher used the database PsychInfo to find the earliest published work of each author from each article. Then the year the article was published was subtracted by the author s first publication in PsychInfo and this number was put into an excel chart. The current study counted a dissertation, book, or research study as an author s first published work. The excel chart was then copied into SPSS statistical software Next, the current study looked at ways which the two groups of either using an analysis or not using an analysis would differ on the variable SEM, because this variable was predicted to show strong change throughout the years based off previous research (Austin et al., 2002). Measures All the different statistics in each article were grouped into 9 categories. One-way ANOVA any univariate one-way analysis fit into this category. Factorial ANOVA - any study with two independent variables with two or more conditions each measuring one dependent

5 5 variable (e.g., 2 2 design, 3 3 design, 5 7 design, etc.). MANOVA/MANCOVA - this included any multivariate analysis or multivariate analysis of covariance or one-way MANOVA/MANCOVA indication. t-test - included any single sample t-test, dependent t-test, or independent group t-test. Chi-square - any Chi-square goodness of fit analyses and Chisquare tests for independence without specifying a fit index. Regression - included multiple regression, basic regression, hierarchical regression modeling, hierarchical linear modeling, and path analyses with squares. Correlation - included zero-order correlations, partial correlations, intercorrelations, and basic correlations. Exploratory factor analysis - any factor analysis or multiple leveled factor analyses, and exploratory factor analyses. Structural Equation Modeling (SEM) - which included confirmatory factor analysis. Results In order to test the first hypothesis a series of chi-square analyses were conducted for the 9 variables: One-way ANOVA, Factorial ANOVA, MANOVA/MANCOVA, t-test, Chi-square, Regression, Correlation, Exploratory factor analysis, and Structural Equation Modeling. For all the variables their use and not use is displayed in Table 1, along with significant standardized residuals. The variable SEM was statistically significant χ² = (2, n = 115) = 17.82, p <.001. SEM use in the year 2008 was more than three times more prominent than in the year 1988 (Table 1). The variable Correlation yielded significant results χ² = (2, n = 115) = 10.28, p <.05. Also using the same formula the variable MANOVA / MANCOVA yielded significant results χ² = (2, n = 115) = 7.87, p <.05. The variable regression also showed significant results χ² = (2, n = 115) = 28.69, p <.001, with significant standardized residuals in a number of years (Table 1).

6 6 Table 1 Frequency of Statistic Use and Standardized Residuals Stat use χ² sig level n = 115 n = 46 n = 26 n = 43 SEM ª ª p < t-test n.s Correlation p < ª Exploratory n.s ANOVA n.s One-way n.s MANOVA ª p < Regression ª ª p < ª ª Chi-square n.s Note. 1 Statistical formula was used, 0 Statistical formula was not used ª indicates significant standardized residual Based on the independent group t-test using the years 1998 and 1988, the group using SEM (n = 12) had a mean of 5.16 (SD = 6.87), and the group not using SEM (n = 42) had a mean of 10.5 (SD = 10.57) the two groups significantly differed, t(27.53) = 2.08, p <.05 with equal

7 7 variances not assumed. Next, a independent group t-test was conducting using only the year 1988 and the group containing SEM (n = 7) had a mean of 3.0 (SD = 4.2), the group not using SEM (n = 24) had a mean of 8.5 (SD = 9.93), the two groups significantly differed from one another, t(24.50) = 2.14, p <.05. Discussion The first hypothesis was supported; a significantly higher amount of SEM formulas was demonstrated in the year 2008 than in the year These results concur with a number of different studies that are based off similar hypotheses (Austin et al., 2002; Cascio & Aguinis, 2008; Duriau et al., 2007). One explanation for these results is that the field of I-O psychology is moving toward using more complex statistical analyses as a whole, because of the availability of different statistical software programs such as LISREAL and AMOS. Also the variable MANOVA/MANCOVA was more significant in the year 1998 which partially supports our prediction of an upward trend in research, because MANOVA/MANCOVA is a more complicated statistic than an ANOVA which is used commonly in 1988, and SEM is more complicated than the MANOVA/MANCOVA variable giving us that upward trend as demonstrated in previous research (Austin et al., 2002). The variable correlation was also significant in 2008 most likely due to the multitude of intercorrelations generated from a Confirmatory Factor Analysis (CFA) statistic. A CFA makes up a large part of the SEM grouping variable. Also the variable regression was significant which could also be explained again by availability of advanced statistical software, because many of regression analyses are now hierarchical and need statistical software to produce.

8 8 The second hypothesis was partially supported by the results. Taken as a whole all the years together did not yield significant results, but when the years were separated (1998 and 1988 together and 1998 alone) significant results were demonstrated in terms of using and not using SEM. One explanation for the significant results obtained is that due to the inability of many of these statistical software programs more than 10 years ago many of the researchers with high research experience may have stuck to the things they know best and not have used these advanced statistical procedures. Since there was no significant differences demonstrated as a whole there is always the possibility that there is no significant differences between researcher s experience and the type of statistical analysis conducted; possibly researchers with many years of previous experience have adjusted to using new advanced statistics. Another possibility is the existence of many other factors including: author, sample, and research purpose characteristics could be the moderating factor of the significant relationship observed in the current study. Further research will be needed to demonstrate this hypothesis by sorting out the factors of: coauthors, number of co-authors, number of studies in each article, etc., to truly sort the effects of researcher s experience relating to the complexity of the statistical analysis. A limitation of this study is the limited years selected and the upward trend in significance testing in psychological journals as a whole. Much previous research analyzed every single year (Austin et al., 2002; Cascio & Aguinis, 2008; Duriau et al., 2007), but due to time and resource constraints the current study was only able to analyze three years individually. Although the current research cannot account for each decade as a whole, significant differences were still found between decades, so a sample from each decade individually seems to have given an overall impression of the decade as a whole. Furthermore, the results in the current study are very similar to much previous research on this topic (Austin et al., 2002; Cascio &

9 9 Aguinis, 2008; Duriau et al., 2007). A study done by Hubbard, Parsa, and Luthy (1997) demonstrated that significance testing in psychological research had an upward trend from only 17% of articles, by the 1950 s 75% of articles, and the 1990 s 94% of the articles. If this trend demonstrated by Hubbard et al. (1997) were to continue this would account for why we seen an upward trend in research for our grouping variables in 2008, it would simply be due to more articles containing significance testing procedures, which would lead to a greater probability of seeing the different statistical procedures in the current study found as a significant trend. These results lead to the conclusions of where research is going in the field of Industrial- Organizational psychology and what statistical procedures will be effective in the future. Training of these more advanced statistical procedures must be a staple of graduate education in order for future research to produce effective results. Also one could argue that the statistical analysis depends on the experiment, but as an example with 56.52% of the articles in 2008 containing SEM, it appears that many authors are now setting up experiments to use SEM and not that SEM is just the best fit for the research on testing a particular subject. Lastly, another argument could be the compliers of the Journal of Applied Psychology could now only be accepting journals with these complex types of analyses, this argument only further perpetuates that these complex types of formulas are deemed as effective, because journal editors are including so many of these types of formulas.

10 10 References Aguinis, H., Pierce, C. A., Bosco, F. A., & Muslin, I. S. (2009). First decade of organizational research methods: Trends in design, measurement, and data-analysis topics. Organizational Research Methods, 12, Austin, J. T., Scherbaum, C. A., & Mahlman, R. A. (2002). History of research methods in industrial and organizational psychology: Measurement, design, analysis. Malden, MA: Blackwell Publishing. Cascio, W. F., & Aguinis, H. (2008). Research in industrial and organizational psychology from 1963 to 2007: Changes, choices, and trends. Journal of Applied Psychology. 93, Duriau, V. J., Reger, R. K., & Pfarrer, M. D. (2007). A Content Analysis of the Content Analysis Literature in Organization Studies: Research Themes, Data Sources, and Methodological Refinements.. Organizational Research Methods. 10, Hubbard, R., Parsa, R. A., & Luthy, M. R. (1997). The spread of statistical significance testing in psychology: The case of the Journal of Applied Psychology, Theory & Psychology. 7,