Measuring Vitality using FACT Items: Creation of the FACT vitality scale Linda Bobbitt, ELCA 8/4/15

Transcription

1 Measuring Vitality using FACT Items: Creation of the FACT vitality scale Linda Bobbitt, ELCA 8/4/15 In 2011, the ELCA developed the 29 item Congregational Vitality Survey (CVS) to measure vitality as an outcome distinct from process oriented indicators (Bobbitt 2014). Process oriented indicators include what congregations do and how they do them (e.g. programs and practices), while outcomes focus on the qualities of the congregation itself. The CVS described vitality in terms of the congregation s connections with God, each other and the world. These three areas are key to understanding the broader concept of vitality because they are mandated in scripture through the great commandment and the great commission. The ELCA hoped to use the FACT over-sample to reveal overall vitality in its congregations and see how that vitality related to process indicators like type of worship, use of technology, young adult ministry practices, etc. While the CVS intentionally included several items from the FACT survey and the 2015 FACT survey added some new items from the CVS, not all items are the same and some were worded differently enough to potentially change their meaning. These differences necessitate the creation of a new vitality scale using the FACT survey items. A scale is preferable to a single item because it is more reliable, controlling for individual or contextual differences in how respondents answer individual questions. Using a scale also allows for the larger conceptual construct of vitality emerge as a pattern among indicators in ways that individual indicators themselves cannot. To create a vitality scale within FACT, we used the same standards that were used in creating the CVS, reliability over r=.8 and factor analysis with no more than 3 factors (ideally one factor). For validation, we chose to compare the resulting scale with the FACT item Future (Which (description) best describes your sense of this congregation s future?). The item reflects perceived vitality and sustainability. It is categorical because the response items mix the expectation of continuing the current level of vitality or expecting to decline. Though sustainability is a different construct than connections with God, each other and the world, we should still expect congregations with higher scale scores to predict futures that are more optimistic than those with lower scale scores. Method: A statistically representative oversample of ELCA congregations was surveyed for FACT 2015 with 573 congregations participating. This data was used to develop the FACT Vitality Scale. It should be noted that the FACT survey is a key informant survey while the CVS is most often used as an every-member survey. First, every item describing to how a congregation is connected with God, each other or the world was identified. Particular attention was paid to items that are represented in the CVS short scales (Bobbitt 2015), but all items with response options that could be considered scales were included for the initial analysis. Internal consistency and factor analysis were used to see how many concepts were represented. Items that hurt the scale or loaded primarily on a secondary factor were excluded in an iterative fashion until the resulting reliability was acceptable (r>.8) and the factor analysis included no more than 3 factors

2 (similar to the original CVS study). Ideally the resulting tool would resemble the shorter scale identified in subsequent research with internal consistency around.9 and one factor. Table 1 shows all the items on the FACT survey that appeared to measure the congregation s connections with God, each other and/or the world. Columns show whether they are also represented on the CVS with the same wording or different wording. Items that are also included on the 15 item short scale are indicated as well as the original subscale they came from or best relate to (G=God, E=each other, W=world). Table 1: FACT questions similar to CVS items. Same as CVS long Different from CVS On CVS short scale How well do the following describe your congregation s largest regular weekend worship service? (response items not at all- A lot) Reverent G Filled with a sense of God s presence X G Thought provoking G Nurturing of people s faith X X G Innovative G Inspirational G Joyful X G Our congregation (agree or disagree) Has a clear mission and purpose X X G Is quite different from other congregations in W our community Is good at incorporating newcomers into the congregation X X E Is spiritually vital and alive X G Is working for social justice X X W Is caring and supportive of members who E have health, financial or personal needs Is willing to change to meet new challenges X X G How much does your congregation emphasize the following (not at all, a little, some, quite a bit, a lot) Personal prayer, meditation, scripture study, X G devotions or other spiritual practices Talking with friends or members of the X X W congregation about one s faith Fasting G Keeping the Sabbath day holy G Parents talking to children about faith G Living out one s faith in all aspects of one s daily life- work, family, civic engagement etc. X X W Scale

3 Results Twenty Items were initially identified (Table 1). Those items were heavily weighted toward the God scale. Prior research showed that God scale items were the most closely correlated with individual respondents connections with God, each other and the world in every-member surveys. That suggests that over representation of that scale may be alright as long as other subscales contribute meaningfully. Cronbach s Alpha on these 20 standardized items was r=.873. Factor analysis without rotation revealed 4 factors with 31% of the variance loading onto the first component, 8.9% on the second, 8% on the third and 5.8% on the fourth. Table 2 shows factor loadings: Table 2: Initial factor analysis Component Matrix a Component Reverent filled with a sense of God's presence Thought-provoking Nurturing of people's faith Innovative Inspirational Joyful has a clear mission and purpose is quite different from other congregations in our community is good at incorporating newcomers into the congregation is spiritually vital and alive is working for social justice is caring and supportive of members who have health, financial or personal needs is willing to change to meet new challenges Personal prayer, meditation, Scripture study, devotions or other spiritual practices

4 Talking with friends and other members of the congregation about one's faith Fasting Keeping the Sabbath day holy Parents talking with children about faith Living out one's faith in all aspects of one's daily life - work, family, civic, engagement, etc Extraction Method: Principal Component Analysis. a. 4 components extracted. Factors that did not load primarily on the first component were removed one by one and the internal consistency and factor analysis was repeated until the smallest number survived while maintaining items in each of the three scales, internal consistency r>.8 and no more than 3 factors. After several iterations, 15 items remained with internal consistency of r=.876 and 3 factors with all items loading the most on the first factor. Unfortunately the scale was still heavily weighted toward the God scale with 10 God oriented items, 2 Each Other and 3 World. The world and each other factors had much lower factor loadings than God items confirming that the first factor is primarily about connecting with God (spirituality). In an effort to balance the scale, God items with lower factor ratings were removed. The resulting scale had 12 items: 4 relating to God in worship, 2 others from the CVS God scale (mission/purpose and willing to change), 2 each other and 3 World items. Table 3 shows the factor loadings and table 4 shows the explained variance. All but one item loaded primarily on to the first factor (God and overall vitality) while the other two factors appear to focus on either the internal (each other) or external (world) items. This scale as an internal consistency of r=.861. The each other scale item relating to caring and supporting members who need help, had higher factor loading on the third component. CVS research suggests that this is because there are many congregations where members care for each other, but which are not otherwise vital (spiritual or missional). However it has also been demonstrated that congregations that do not care for each other are not vital. In summary, caring for each other is always associated with vital congregations but not all congregations who care for each other are vital. For this reason, and the fact that it was the only each other item relating to internal congregational life, the item was kept in the scale. The scale could be consolidated further by eliminating some of the God items associated with Worship. However the items were not highly correlated (r= ) suggesting that each area of worship describes something different from the others. For that reason all 4 remaining worship items were maintained.

5 Table 3: Final Scale Factors Component Matrix a Component filled with a sense of God's presence Nurturing of people's faith Inspirational Joyful has a clear mission and purpose is good at incorporating newcomers into the congregation is spiritually vital and alive is working for social justice is caring and supportive of members who have health, financial or personal needs is willing to change to meet new challenges Talking with friends and other members of the congregation about one's faith Living out one's faith in all aspects of one's daily life - work, family, civic, engagement, etc Extraction Method: Principal Component Analysis. a. 3 components extracted.

6 Table 4: total explained variance Total Variance Explained Initial Eigenvalues Extraction Sums of Squared Loadings Component Total % of Variance Cumulative % Total % of Variance Cumulative % Extraction Method: Principal Component Analysis. Validating the scale To validate this tool, a new item called FACTCVS was created with these 12 items by averaging them for each congregation. Figure 1 is a histogram of how congregations scored on the new scale. Scores ranged from 1.83 to 5.0. The mean was The bottom 25% was at 3.6 or lower and the top 75% scored 4.2 or higher. Congregations were divided into quartiles and assigned a letter A (top 78%), B (50-74%), C (25-49%) and D (25% and below). Figure 2 shows the distribution of ELCA congregations. Cut off scores were as follows: A= >4.21, B= , C= , D=<3.5.

7 Figure 1: Histograms of FACT scale and FACT quartiles It should be noted that the scores associated with these letters are similar to the scores identified when creating the short CVS scale.(bobbitt 2015) In that study adjoining categories overlapped when compared to an external vitality rating, but letters that were not adjoining did not overlap in vitality (e.g. A overlapped with B but not C or D). This study will look to see if a similar pattern exists with the FACT scale. The Future item on the FACT survey had responses ranging from not sure this congregation will survive much longer to We are thriving and this should continue. Chi squared analysis showed that the quartiles differed significantly from each other (X 2 (15, N = 555) = , p=.000) with A congregations (coded as 4) more likely to say they were okay or thriving, rarely thinking they would not survive, while D congregations more likely to say they may not survive, were struggling or doing okay, but unlikely to say they would continue to do okay or that they were thriving. While most C congregations felt like they were doing okay and expected to remain that way, few said they were thriving. Figure 2 illustrates the differences. An ANOVA further confirmed the differences among quartiles (F (3/551) = , p =.000) and a Tukey post hoc test revealed that the differences were between D, C and A, B. A and B were not significantly different in how they answered the question about the future.

8 Figure 2 FACT Quartiles X Anticipated Future Conclusion The consistency of the FACT scales with the Future item suggest its construct validity. The scale appears to measure key concepts associated with vitality. This scale should be tested with the larger FACT dataset and with other denominational oversamples to confirm its psychometric qualities. The SPSS syntax for creating the FACTCVS is included in Appendix A. If it is found consistently reliable and valid, it may be useful in analysis of other data present in the FACT surveys. Bobbitt, Linda. Creating Shorter Scales to Measure Congregational Vitality. [In English]. Review of Religious Research (2015/07/ ): 1-2. Bobbitt, Linda D. Measuring Congregational Vitality: Phase 2 Development of an Outcome Measurement Tool. Review of Religious Research 56, no. 3 (2014):

9 Appendix A Testing Reliability SPSS Syntax RELIABILITY /VARIABLES= Worship5presence Worship5nurture Worship5inspiration Worship5joyful Identity1Mission Identity1incorp Identity1vital Identity1justice Identity1caring Identity1change Prog5witness Prog5living /SCALE('2NDSCALE') ALL /MODEL=ALPHA /STATISTICS=DESCRIPTIVE SCALE CORR /SUMMARY=TOTAL MEANS. FACTOR /VARIABLES Worship5presence Worship5nurture Worship5inspiration Worship5joyful Identity1Mission Identity1incorp Identity1vital Identity1justice Identity1caring Identity1change Prog5witness Prog5living /MISSING LISTWISE /ANALYSIS Worship5presence Worship5nurture Worship5inspiration Worship5joyful Identity1Mission Identity1incorp Identity1vital Identity1justice Identity1caring Identity1change Prog5witness Prog5living /PRINT INITIAL EXTRACTION /CRITERIA MINEIGEN(1) ITERATE(25) /EXTRACTION PC /ROTATION NOROTATE. Creating the FACTCVS item. Compare this item with Future for validation. COMPUTE FACTCVS = MEAN(Worship5presence,Worship5nurture,Worship5inspiration,Worship5joyful,Identity1Mission,Identity1incorp, Identity1vital,Identity1justice,Identity1caring,Identity1change, Prog5witness,Prog5living). EXECUTE. ELCA Variables (The ELCA used its own variable names.) COMPUTE FACTCVS = MEAN(godthere,Nurture,Inspire,Joyful,purpose,incorp,vital,justice,caremem,change,TalkFath,LiveFaith). EXECUTE. RELIABILITY /VARIABLES= godthere Nurture Inspire Joyful purpose incorp vital justice caremem change TalkFath LiveFaith /SCALE('2NDSCALE') ALL /MODEL=ALPHA /STATISTICS=DESCRIPTIVE SCALE CORR /SUMMARY=TOTAL MEANS. FACTOR /VARIABLES godthere Nurture Inspire Joyful purpose incorp vital justice caremem change TalkFath LiveFaith /MISSING LISTWISE /ANALYSIS godthere Nurture Inspire Joyful purpose incorp vital justice caremem change TalkFath LiveFaith /PRINT INITIAL EXTRACTION /CRITERIA MINEIGEN(1) ITERATE(25) /EXTRACTION PC /ROTATION NOROTATE.