Scalelet Reflection on Student Learning in the US

Transcription

1 The Dependability of NSSE Scalelets for College- and Department-Level Assessment Author(s): Gary R. Pike Reviewed work(s): Source: Research in Higher Education, Vol. 47, No. 2 (Mar., 2006), pp Published by: Springer Stable URL: Accessed: 15/03/ :35 Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at. JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.. Springer is collaborating with JSTOR to digitize, preserve and extend access to Research in Higher Education.

2 Research in Higher Education, Vol. 47, No. 2, March 2006 ( 2006) DOI: /sl THE DEPENDABILITY OF NSSE SCALELETS FOR COLLEGE- AND DEPARTMENT-LEVEL ASSESSMENT* Gary R. Pike*** College and university administrators, as well as faculty members, are more likely to take responsibility for student learning if they believe that the assessment data represent their students and suggest specific actions for improvement. This study examined whether it is feasible to develop scalelets (i.e., focused measures, usually consisting of four or five items) that provide dependable metrics for assessing student learning at the college or department level. A generalizability analysis of 12 scalelets from the National Survey of Student Engagement (NSSE) indicated that the scalelets provided dependable measures of educational effectiveness with respondents. KEY WORDS: assessment; surveys; generalizability theory. Assessment has become a fact of life in American higher education. Responding to external demands for accountability and institutional needs for improvement, virtually all colleges and universities are assessing the contributions of their curricular and co-curricular programs to student learning and development. A survey by the National Center for Postsecondary Improvement (NCPI) revealed that 96% of nearly 1400 responding institutions had implemented some form of assessment (Peterson, Einarson, Augustine, and Vaughan, 1999). For many institutions, surveys are an important component in assessment efforts. Peterson *SAIR 2004 Best Paper. Presented at the annual meeting of the Association for Institutional Research, Sandiego, CA, March Director of Institutional Research, Mississippi State University, Mississippi State, USA. Address correspondence to: Gary R. Pike, Director of Institutional Research, Mississippi State University, 269A Allen Hall, P.O. Box EY, Mississippi State, MS , USA. gpike@ir.msstate.edu /06/ / Springer Science* Business Media, Inc.

3 178 PIKE and his colleagues (1999) found that three-quarters of the institutions responding to their questionnaire made use of surveys in assessment. Banta (1993) and Muffo and Bunda (1993) noted that surveys are a particularly appropriate assessment tool because they focus on what students do in college. Likewise, Ewell, Jones, and their colleagues concluded that indirect measures, such as surveys, are among the best available assessment methods and could form the basis for a national assessment effort (National Center for Higher Education Management Systems, 1994). Despite the fact that assessment is widespread, examples of effective use of assessment information are relatively rare (Banta, 2002; Ewell, 2002). Peterson and Vaughan (2002) noted that very few institutions report that assessment information plays an important role in decision making, and Pike (2002, p. 147) concluded "there is no greater problem in assessment than our inability to influence decision making with assessment results." The NCPI survey found that survey data were no more likely to be used in decision making than other types of assessment data (Peterson et al., 1999). A major barrier to the use of survey results is the fact that the data are not disaggregated - either in terms of groups of students or survey items. Examples of effective use of survey data generally show that the data have been disaggregated at the college or department level and are based on small numbers of highly related questions that suggest specific improvement actions (Kuh, Gonyea, and Rodriguez, 2002; Pike and Kuh, 2005). Disaggregation of results frequently requires that a survey be administered to a very large number of students in order to produce dependable measures (Kuh et al., 2002). This is a requirement that institutions may not be able to meet for many of their programs. The present research proposes that scalelets be used in survey-based assessment research. These scalelets are analogous to testlets used in computerized adaptive testing. Wainer and Kiely (1987) advocated that testlets (i.e., highly interrelated groups of questions) be used in computerized adaptive testing to overcome problems of non-independence and context effects. Scalelets, clusters of highly related survey questions representing the experiences of a group of students, may allow assessment researchers to disaggregate results without large-scale administration of a survey. The research question underlying this study is whether scalelets yield dependable scores based on relatively small numbers of respondents. The findings of this study are important because they are a first step in evaluating a new assessment approach. The research is also important because it reminds higher-education professionals that groups, not individuals, are the appropriate units of analysis in many assessment studies.

4 THE DEPENDABILITY OF NSSE SCALELETS 1 79 RELATED LITERATURE Institutions involved in assessing the college experiences of their students utilize a variety of survey instruments (Pike, 1999). Commercially available surveys include the College Student Experiences Questionnaire (CSEQ) (Pace and Kuh, 1998), the Cooperative Institutional Research Program's College Student Survey (Higher Education Research Institute, 2005), the Noel-Levitz Student Satisfaction Inventory (Schreiner and Juillerat, 1994), and the National Survey of Student Engagement's (2004) College Student Report. In addition, many colleges and universities have developed their own specialized surveys to assess specific aspects of the student experience. The concept of scalelets can be applied to virtually any survey. For the purposes of this research, analyses focus on a single instrument - the National Survey of Student Engagement's College Student Report. The National Survey of Student Engagement Developed as an alternative to reputation- and resource-based rankings, the National Survey of Student Engagement (NSSE) is designed to measure the extent to which students are engaged in educationally purposeful activities that contribute to their learning and development during college (Kuh et al., 2001). Over the past five years, the NSSE survey has been administered to students attending more than 850 colleges and universities (Indiana University Center for Postsecondary Research, 2004). Half a century of research has shown that student engagement is an important factor in learning and development (Astin, 1977, 1993; Feldman and Newcomb, 1969; Pascarella and Terenzini, 1991, 2005), and several recent studies have found that measures of student engagement, similar to those used by NSSE, are significantly related to students' learning and development (Astin, 1993; Kuh, Pace, and Vesper, 1997; Pike, 1995; Pike, Kuh, and Gonyea, 2003). The framework underpinning NSSE is drawn from Pace's (1984) concept of quality of student effort, Chickering and Gamson's (1987) "Seven Principles of Good Practice in Undergraduate Education," and Astin's (1984) theory of student involvement (Kuh et al., 2001). NSSE's survey, The College Student Report, asks students about their engagement in activities that reflect good practice in undergraduate education. Student self reports are widely used in research on college effects, and the validity and credibility of these data have been extensively studied (Baird, 1976; Berdie, 1971; Pace, 1985; Pike, 1995; Pohlmann and Beggs, 1974). Research has shown that self reports are likely to be valid under five conditions:

5 180 PIKE 1. the information requested is known to the respondents; 2. the questions are phrased clearly and unambiguously; 3. the questions refer to recent activities; 4. the respondents think the question merit a serious and thoughtful response; and 5. answering the questions does not threaten, embarrass, or violate the privacy of the respondent or encourage the respondent to respond in socially desirable ways (Kuh et al., 2001). Empirical evidence indicates that The College Student Report satisfies these conditions (Kuh, 2001b). In addition, research has found that test-retest reliability is quite high (0.83), and mode of administration effects for paper and Web surveys are quite small (Kuh et al., 2001). Moreover, follow-up studies found few meaningful differences between respondents and nonrespondents in terms of their levels of engagement (Kuh, 2001b). Five institutional benchmarks have been developed using items from the survey: (1) Level of Academic Challenge; (2) Active and Collaborative Learning; (3) Student Interaction with Faculty Members; (4) Enriching Educational Experiences; and (5) Supportive Campus Environment. These benchmarks can serve as proxy measures to identify opportunities for improving undergraduate education (Kuh, 2001a). Although NSSE benchmarks can be used to identify institutional strengths and opportunities for improvement, not all institutions report that they have used the survey results for decision making. Institutions that do report making use of the NSSE data generally indicate that they have disaggregated results to the program level (El-Khawas, 2003; Hughes and Pace, 2003; Indiana University Center for Postsecondary Research, 2001, 2002, 2003; Kezar, 2002, 2003) and/or have used subsets of questions from the NSSE benchmarks to identify focused strategies for improvement (El-Khawas, 2003; Indiana University Center for Postsecondary Research, 2002). As with all surveys, disaggregation of NSSE results requires that institutions have ample numbers of respondents in order to be confident of the reliability of the results (Indiana University Center for Postsecondary Research, 2001). The Rationale for Scalelets The term scalelet, as used in this study, is loosely drawn from the concept of testlets proposed by Wainer and Kiely (1987, p. 190): "A testlet is a group of items related to a single content area that is developed as a unit and contains a fixed number of predetermined paths that

6 THE DEPENDABILITY OF NSSE SCALELETS 181 an examinee may follow [in a computerized adaptive test]." The use of testlets in computerized adaptive testing allows developers to construct testing units that contain more than one item and helps reduce problems associated with context and order effects (Wainer et al., 1990). The result is test scores with greater dependability and less error than scores based on individual items (Thissen and Mislevy, 1990). A scalelet consists of a limited number of survey questions that provide a measure of a specific aspect of the educational experiences of a group of students. Three elements of this definition deserve elaboration. First, a scalelet consists of a limited number of survey questions. As a general rule, it is not possible to make dependable generalizations about a construct, such as involvement in co-curricular activities, based on a single question, for example the number of organizations to which a student belongs. The richness of the constructs used in student-outcomes research and assessment create a situation in which generalizations based on a single item must be limited to that item. In the preceding example, generalizations should only be made about the number of organizations to which students belong. Although the ideal is to base generalizations and decisions about students' educational experiences on all possible questions about those experiences, the practice of survey research requires that assessment professionals base their conclusions on students' responses to a sample of questions. The second element in the definition of scalelets - that questions relate to a specific aspect of the educational experience - allows relatively small samples of items to be included in a scalelet. In effect, there is a continuum ranging from very broad generalizations, based on large numbers of survey questions, to specific conclusions about an item, based on responses to that item. Scalelets are intended to strike a balance between the breadth of generalizations and the number of questions in a survey. Another reason for focusing on specific aspects of students' educational experiences is that a focused approach is more likely to result in the use of assessment data. As previously noted, feedback from focus groups revealed that institutions were more likely to use NSSE results for improvement when the data identified specific actions that should be taken (Indiana University Center for Postsecondary Research, 2002). The third element in the definition of scalelets - that the measures represent the educational experiences of groups of students - is based on the understanding that both formative and summative assessment for accountability or quality improvement require data about groups, not individual students (Ewell, 1991; Pike, 1994). The co-curricular experiences of a student are certainly important and interesting, but data about a single student does not provide sufficient information about the

7 182 PIKE co-curriculum at an institution. Evaluations of the co-curriculum should be based on information about the experiences of all students or on the experiences of a sample of students that can be generalized to the population. A fundamental question underlying the use of scalelets in assessment research is how large the sample of students needs to be in order to generalize findings to the population of interest. This question is addressed in the present research. Evaluating the Dependability of Scalelets The use of scalelets potentially requires that researchers and assessment professionals make several different generalizations from samples to populations. Evaluating the quality and effectiveness of a program requires that an assessor generalize from a sample of questions about the program to the universe of all possible questions about the program. Likewise, as assessor may need to make generalizations about all of the students in a program based on a sample of students in that program. Generalizability theory, developed by Cronbach, Gleser, Nanda, and Rajaratnam (1972), provides a mechanism for researchers, assessment professionals, and policy makers to identify the limits of the inferences they can draw from their samples (Shavelson and Webb, 1991). Consequently, generalizability theory formed the conceptual basis for evaluating the dependability of NSSE scalelets. The first step in a generalizability analysis is identifying the object of measurement - that aspect of the measurement situation about which generalizations are to be made - and the facets of measurement - those aspects of the measurement situation that contribute to error (Brennan, 1983). Unlike classical reliability analyses, a generalizability analysis can take into account multiple sources of error (Pike, 1994). In evaluating the dependability of NSSE scalelets, the object of measurement would be a program because the reason for using scalelets is to make judgments about a program. The program could be an educational intervention, such as a diversity-awareness course or a learning community. It could also be an academic unit, or even a college or university. There are two possible sources of error affecting the dependability of generalizations based on NSSE scalelets. First, the sampling of items that are used to represent the construct underlying the scalelet may contribute to error. Second, the sampling of students in the program may contribute to error. It is important to realize that items and students are potential sources of error. For example, an assessor may only be interested in questions included in a scalelet, not in some underlying construct. In that case there is no sampling of items, and items are not a

8 THE DEPENDABILITY OF NSSE SCALELETS 1 83 facet of measurement. Similarly, a researcher may have obtained scores for all of the students in a program. In that case, there is no sampling and no source of error for students. For this study, it is assumed that the assessment design requires generalizing over samples of items and students, although the situation in which a researcher is only concerned with responses to the items included in a scalelet will be considered. In generalizability theory, a distinction is drawn between G (generalizability) and D (decision) studies. Brennan (1983) explains that G studies are designed to represent the universes of admissible observations and to provide estimates of the variance components for those universes. D studies incorporate the specifics of an assessment design (e.g., the number of items in a scalelet and the number of students sampled) in evaluating the dependability of a measure. D studies allow researchers to evaluate the effects of changing the conditions of a facet of measurement (e.g., increasing the number of participants in a study) on the dependability of the measures. The practical applications of G and D studies can best be understood within the context of an evaluation of the dependability of NSSE scalelets. RESEARCH METHODS Conceptual Model Table 1 depicts the conceptual model used to evaluate the dependability of the NSSE scalelets. The model presumes that samples of students, within programs, answer items on the NSSE survey and that these items comprise a scalelet. In the figure, program units are represented by the symbol U, and individual units are identified by the subscripts 1,2,..., w,..., j. Students, nested within units, are represented by the symbol S, and individual students are identified by the subscripts 1, 2,..., s,..., n. Items from the NSSE survey are represented by the symbol /, with individual items identified by the subscripts 1, 2,...,/,...,. A student's score on an item is symbolized by X. Thus, the response of student s, from unit w, to question / is represented by Xusi. Data Source The data for this study came from the 2001 administration of The College Student Report. The initial sample consisted of 177,103 first-year students and seniors who were randomly selected from the populations of 321 participating colleges and universities. Students at 261 institutions had the option of responding either via a paper-and-pencil questionnaire or via the Web. Sixty schools opted for web-only

9 184 PIKE TABLE 1. Conceptual Model of a Two-Facet Design h h /, h U\ S\ X\w X\n X\u X\\k»$2 -^121 -^122 X\2i X\2k Ss X\s\ X\S2... X\si... X\sk U2 Si X2\\ X2\2.- X2n... X2ik $2 ^221 ^222 ^22/. ^22A: $s Xis\ Xisi... Xisi... Xisk Sn Xin\ Xini... X^j... X^k Uu S\ Xu\\ Xu\2... Xu\i... Xu\k $2 %u2\ %u22 Xu2i %u2k»j s A. us i A us2... A usj... A usfc ^n Xun\ Xun2... Xunj... Xunk Uj S\ Xjii Xj\2... Xjij... Xjlk S2 Xfi\ Xj Xfn... Xpk Ss Xjs\ Xjs2... Xjsi... XjSk Sn Xjn\ Xjn2... Xjni... Xjnk administration of the survey. Table 2 displays the characteristics of the NSSE 2001 institutions and a national profile of all four-year colleges and universities. The data show that the NSSE institutions were very similar to the national profile in terms of geographic region and urban-rural location. Baccalaureate-General colleges were underrepresented among the NSSE participants, whereas Doctoral/Research- Extensive and Baccalaureate Liberal Arts institutions were overrepresented among the NSSE participants. Only seniors were included in this study for two reasons. First, they have had a wider range of college experiences and arguably can provide more informed reports about a variety of activities. Second, the experiences of first-year students and seniors differ substantially in terms of curriculum (i.e., coursework for first-year students emphasizes general

10 THE DEPENDABILITY OF NSSE SCALELETS 185 TABLE 2. Characteristics of the Institutions Participating in NSSE 2001 Institutional Characteristics NSSE 2001 (%) National (%) Carnegie classification Doctoral/Research - Extensive Doctoral/Research - Intensive 10 8 Master's I & II Baccalaureate-Liberal Arts Baccalaureate-General Sector Public four-year Private four-year Region Far West 9 10 Great Lakes Mideast New England 9 9 Plains 8 11 Rocky mountains 3 3 Southeast Southwest 9 7 Location Large city Mid-size city Urban fringe of large city Urban fringe of small city 7 8 Large town 5 4 Small town Rural 5 6 education, whereas seniors' coursework is concentrated in the major) and out-of-class experiences (i.e., first-year students spend more time in formal extracurricular activities, whereas seniors may have studied abroad, done internships, and so forth) (Pascarella and Terenzini, 1991). Including both first-year and senior students in the G study would have introduced another source of error into the analyses. The overall average unadjusted institutional response rate for 2001 seniors was 41.8%. Response rates ranged from 9.1% to 69.7%. About 69% of the seniors completed the paper version of the survey, and 31% completed the survey via the Web. Table 3 displays the characteristics of NSSE 2001 senior respondents in comparison to the characteristics of all seniors at the participating institutions. The results presented in Table 3 indicate that women were overrepresented among the respondents, as

11 186 PIKE TABLE 3. Characteristics of the NSSE 2001 Senior Respondents Respondent Characteristics NSSE Respondents (%) NSSE 2001 Schools (%) Gender Men Women Race/Ethnicity African- American/Black American-Indian/Alaska Native Asian/Pacific islander Caucasian/White (Not Hispanic) Hispanic Other Multiple race/ethnicity 4.8 NA International Enrollment status Full-time student Part-time student were Caucasians and full-time students. However, the observed differences were relatively small. For this research, 50 randomly selected seniors each from 50 randomly selected institutions were included in the G study. Measures of Student Engagement Fifty questions from The College Student Report were used to create 12 engagement scalelets. A list of the items comprising the scalelets is available from the author. Item scores for a scalelet were placed on the same metric, and the content of the scalelets paralleled the content of the NSSE benchmarks. Most of the items comprising the Course Challenge, Writing, and Higher Order Thinking Skills scalelets were drawn from the Level of Academic Challenge benchmark. Items included in the Active Learning and Collaborative Learning scalelets were taken from the Active and Collaborative Learning benchmark, and the items included in the Course Interaction and Out-of-Class Interaction scalelets were taken from the Student Interaction with Faculty Members benchmark. Many of the items in the Varied Experiences and Information Technology scalelets were drawn from the Enriching Educational Experiences benchmark. The items included in the Support for Student Success and Interpersonal Environment scalelets came from the Supportive Campus Environment benchmark. The Diversity scalelet included items

12 THE DEPENDABILITY OF NSSE SCALELETS 1 87 from the Active and Collaborative Learning and Enriching Educational Experiences benchmarks. Data Analysis The data analysis was carried out in two phases corresponding to G and D studies. In the first phase, the NSSE data were analyzed to identify the variance components that were used in the second (D study) phase of the analysis. Variance components were calculated using the BMDP-8 V program for the analysis of mixed models with equal cell sizes (Dixon, 1992). Five variance components were calculated for each scalelet: units ([/), students within units (S\ U), items (/), the interaction between units and items ( //), and the interaction between students and items within units (SI\U). In the second phase of the data analysis, variance components were entered into formulas for calculating group mean generalizability coefficients. Two formulas, described by Kane, Gilmore, and Crooks (1976), were used. The first formula evaluated the generalizability of a group mean when the objective is to generalize to the universes of all possible items and all possible students. This is equivalent to making generalizations about the higher-order constructs represented by scalelets. The second formula evaluated the generalizability of a group mean when the objective is to draw conclusions about the items comprising a scalelet, rather than a higher-order construct. For the decision phase of the study, the effect of changes in the number of students sampled was a key concern. Sample sizes of 25, 50, 100, and 200 were used in the evaluation. RESULTS Table 4 presents the variance components from the first phase of the data analysis. These variance components have little intrinsic meaning, except as they are used to calculate generalizability coefficients. They are provided to allow researchers to evaluate other scenarios, including varying the number of items in the scalelets. Generalizability coefficients, calculated using the two formulas described previously, are presented in Table 5. An examination of the results presented in the first subtable of Table 5 reveals that only four, or perhaps five, of the NSSE scalelets could produce dependable (i.e., Cp2 > 0.70) group means that could be generalized to all possible items and students based on a sample of 50 students. The Course Interaction, Out-of-Class Interaction, Varied Experiences, and Support for Student Success scalelets all produced generalizability

13 188 PIKE TABLE 4. Variance Components from the Generalizability Study Variance Component NSSEScalelet U S\U I UI SI\U Course challenge Writing Higher-order thinking skills Active learning Collaborative learning Course interaction Out-of-class interaction Varied experiences Information technology Diversity Support for student success Interpersonal environment U, variance component for academic units; S\ U, variance component for students nested within academic units; /, variance component for items; //, variance component for the college-byitem interaction; SI\U, variance component for the student-by-item interaction nested within academic units. coefficients greater than 0.70 for a sample size of 50 students. (The Higher Order Thinking Skills scalelet produced a group mean generalizability coefficient of 0.69 when the group mean was based on 50 students.) Several of the scalelets (i.e., Writing, Collaborative Learning, Information Technology, and Interpersonal Environment) would not provide dependable generalizations with samples of 200 students. The second subtable displays the generalizability coefficients for the scenario where the objective is to make generalizations to the universe of students, but not the universe of items. In this scenario, the assessment researcher can only draw conclusions about the items included in the scalelets, not higher-order constructs. Given that limitation, all of the scalelets yield acceptable generalizability coefficients (i.e., Cp2 > 0.70) for samples of 50 students. Four of the scalelets, Active Learning, Out-of-Class Interaction, Varied Experiences, and Support for Student Success, produced acceptable generalizability coefficients with samples of 25 students. Limitations Whereas results for 2001 are generally consistent with the results reported by NSSE across the first few years of surveys, only one year of

14 THE DEPENDABILITY OF NSSE SCALELETS 1 89 TABLE 5. Generalizabilitv Coefficients from the Decision Study Number of Students NSSEScalelet Generalize to universe Course challenge Writing Higher-order thinking skills Active learning Collaborative learning Course interaction Out-of-class interaction Varied experiences Information technology Diversity Support for student success Interpersonal environment Generalize to scalelet Course challenge Writing Higher-order thinking skills Active learning Collaborative learning Course interaction Out-of-class interaction Varied experiences Information technology Diversity Support for student success Interpersonal environment data, and a subsample of institutions, was analyzed in this study. If institutions participating in other years were included, the results might differ in unknown ways. Also, the generalizability results were derived from the responses of seniors. If a similar analysis was done using firstyear students, different results might have emerged. Furthermore, the data for the generalizability analyses used institutions as the object of measurement, not programs within institutions. If this research had used program-level data, different results might have been obtained. It is also important to understand that dependability (or reliability) is a necessary, but not sufficient, condition for concluding that a scalelet is a valid measure of some aspect of student engagement (Messick, 1989).

15 190 PIKE Additional research, focusing on the construct validity of scalelets, is required before concluding that they should be used in assessment research. Despite these limitations, the findings of this study have important implications for both theory and practice. DISCUSSION The central finding of this research is that the NSSE scalelets produce highly dependable group means based on relatively small samples of students when the assessment researcher is interested in making judgments about the mean (or sum) of items comprising the scalelet. As a result, the NSSE scalelets can provide a mechanism for disaggregating NSSE results to the college or department level without resorting to the expensive practice of over sampling large numbers of students. Scalelets may also be used by institutional and system-level decision makers to provide focused measures of students' experiences at an institution. In addition to increasing the likelihood that campus leaders will act on NSSE results, disaggregation can, potentially, identify problem areas within an institution. For example, it is unlikely that a low score on the Level of Academic Challenge benchmark indicates that all programs at the university are characterized by low levels of academic challenge. It is far more likely that some programs at the institution are very challenging, whereas other programs are not as challenging. If scores are available for the colleges, and some departments, at an institution, the chances of identifying program strengths and opportunities for improvement will increase substantially. The dependability of the NSSE scalelets also allows colleges and universities to develop more focused profiles of levels of engagement within the institution. Once again, a relatively low score on the Level of Academic Challenge benchmark probably does not indicate that an institution, or a college or department, is low in all areas of engagement represented by the benchmark. A low score for academic challenge may indicate a lower emphasis on writing or less emphasis on higher-order thinking in courses. Even more important, disaggregation of the benchmarks may help uncover opportunities for improvement. Average scores on the Level of Academic Challenge benchmark may mask the fact that there is a great deal of emphasis on higher-order thinking in courses, but that students are not asked to write many papers. Recent research by Pike and Kuh (2005) found that the scalelets were very useful in discriminating among institutions based on the ways in which they engaged students. As a general rule, institutions were not uniformly high, or uniformly low, on all of the scalelets comprising a benchmark.

16 THE DEPENDABILITY OF NSSE SCALELETS 191 Some institutions were characterized by high levels of course challenge, whereas other institutions were characterized by the emphasis placed on higher-order thinking. Given the austere fiscal environment faced by most colleges and universities, campus leaders are not likely to have sufficient resources to undertake broad, sweeping changes in response to NSSE results. The most intelligent use of limited resources is to make strategic investments in those areas where the needs and potential benefits are greatest. By providing results that are focused on specific programs and specific types of engagement, scalelets can be an effective tool for improving engagement in those areas measured by The College Student Report. When the goal is to make judgments about broad, overarching constructs that go beyond the specific aspects of student engagement measured by NSSE, even means based on very large numbers of respondents may not yield generalizable results. This finding is disappointing, particularly if one wishes to use the NSSE scales as a means of representing higher-order constructs, rather than clusters of engaging activities. Although disappointing, the results are not surprising. The questions included in The College Student Report, and used in this study, were not selected to represent random samples from larger domains. They were initially selected for the survey because of their face validity and because they were related to good educational practices. The creation of scalelets was an afterthought. Given the absence of an intentional sampling frame, it is not surprising that the items included in the scalelets represented rather poor samples of particular constructs. Rather than indicating a limited role for scalelets, the results of this study may suggest a new approach to survey construction. In advancing the concept of testlets, Wainer and Kiely (1987) argued that testlets represent the basic building blocks of computerized adaptive tests. Individual test questions are important insofar as they contribute to the development of cohesive testlets. A similar approach could be applied to survey development and construction. Development would begin with the identification of the constructs to be assessed by the survey. The definitions of those constructs would serve as frameworks for the seal-. elets in a survey. A sample of items would then be generated and evaluated to identify four or five questions that would yield dependable generalizations about the construct. The final step in survey development would be combining the various scalelets, and their associated items, with requisite demographic and biographic questions. It is important to understand that scalelets are not a panacea. Scalelets can provide highly dependable and focused measures of the college student experience, but they cannot overcome problems created by

17 192 PIKE poorly constructed surveys or samples of respondents that are not representative of the population. These are problems of internal and external validity that can render assessment data useless. Undoubtedly, one reason the NSSE scalelets yielded dependable group means is that a substantial amount of work was put into ensuring that the items provided accurate and appropriate measures of the student experience. Institutional researchers and assessment professionals would be wise to model their survey-development efforts after the procedures used by NSSE staff. The fact that the respondents in this study were random samples of all respondents at the institution may also have contributed to the dependability of the NSSE scalelets. Generalizability theory assumes that the data represent the responses of a random sample and departures from random sampling can adversely affect generalizability (Cronbach et al., 1972). Given problems of non-response in survey research, it is unlikely that the assumption of a strictly random sample can be met in most assessment research. Fortunately, generalizability theory is reasonably robust with respect to violations of the randomsampling assumption. Nevertheless, too little attention is paid to issues of non-response in outcomes assessment. CONCLUSION At a minimum, the results of this study indicate that scalelets can be a useful tool for motivating administrators, faculty, and student affairs professionals to take actions to improve the educational experiences of "their students," and for identifying strategic improvement initiatives that can lead to significant changes in engagement and undergraduate education. However, the potential contributions of scalelets extend beyond being a tool for interpreting NSSE results. Scalelets suggest the possibility of a new approach to survey development and administration that would more effectively represent the important constructs underlying theories and models of student learning and development. REFERENCES Astin, A. W. (1977). Four Critical Years, Jossey-Bass, San Francisco. Astin, A. W. (1984). Student involvement: A developmental theory for higher education. Journal of College Student Personnel 25: Astin, A. W. (1993). What Matters in College? Four Critical Years Revisited, Jossey-Bass, San Francisco.

18 THE DEPENDABILITY OF NSSE SCALELETS 1 93 Baird, L. L. (1976). Using Self-reports to Predict Student Performance, The College Board, New York. Banta, T. W. (1993). Summary and conclusion: Are we making a difference? In: Banta, T. (ed.), Making a Difference: Outcomes of a Decade of Assessment in Higher Education, Jossey- Bass, San Francisco, pp Banta, T. W. (2002). A call for transformation. In: Banta, T. (ed.), Building a Scholarship of Assessment, Jossey-Bass, San Francisco, pp Berdie, R. (1971). Self-claimed and tested knowledge. Educational and Psychological Measurement 31: Brennan, R. L. (1983). Elements of Generalizability Theory, ACT Publications, Iowa City, I A. Chickering, A. W., and Gamson, Z. F. (1987). Seven principles for good practice in undergraduate education. AAHE Bulletin 7(3): 3-7. Cronbach, L. J., Gleser, G. C, Nanda, H., and Rajaratnam, N. (1972). The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles, Wiley, New York. Dixon, W. J. (1992). BMDP Statistical Software Manual, University of California Press, Berkeley, CA. El-Khawas, E. (2003). Using NSSE data for assessment and institutional improvement. Documenting Effective Educational Practices: National Roundtable Series (Issue 5). Indiana University Center for Postsecondary Research, Bloomington, IN. Ewell, P. T. (1991). To capture the ineffable: New forms of assessment in higher education. In: Grant, G. (ed.), Review of Research in Education, (Vol. 17), American Educational Research Association, Washington, D.C., pp Ewell, P. T. (2002). An emerging scholarship: A brief history of assessment. In: Banta, T. (ed.), Building a Scholarship of Assessment, Jossey-Bass, San Francisco, pp Feldman, K. A., and Newcomb, T. M. (1969). The Impact of College on Students, Jossey-Bass, San Francisco. Higher Education Research Institute. College Student Survey. University of California Los Angeles, Los Angeles, CA. Hughes, R., and Pace, C. R. (2003). Using NSSE to study student retention and withdrawal. Assessment Update: Progress, Trends, and Practices in Higher Education 15(4): 1-2, 15. Indiana University Center for Postsecondary Research (2001). Improving the College Experience: National Benchmarks of Effective Educational Practice, Author, Bloomington, IN. Indiana University Center for Postsecondary Research (2002). From Promise to Progress: How Colleges and Universities are using Student Engagement Results to Improve Collegiate Quality, Author, Bloomington, IN. Indiana University Center for Postsecondary Research (2003). Converting Data into Action: Expanding the Boundaries of Institutional Improvement, Author, Bloomington, IN. Indiana University Center for Postsecondary Research (2004). Student Engagement: Pathways to Collegiate Success, Author, Bloomington, IN. Kane, M. T., Gillmore, G. M., and Crooks, T. J. (1976). Student evaluations of teaching: The generalizability of class means. Journal of Educational Measurement 13: Kezar, A. (2002). Faculty developers using theo National Survey of Student Engagement (NSSE) to be change agents. Documenting Effective Educational Practices: National Roundtable Series (Issue 1). Indiana University Center for Postsecondary Research, Bloomington, IN. Kezar, A. (2003). Student affairs administrators: building collaborations with students and academic affairs for institutional improvement. Documenting Effective Educational Practices: National Roundtable Series (Issue 3). Indiana University Center for Postsecondary Research, Bloomington, IN.

19 194 PIKE Kuh, G. D. (2001a). Assessing what really matters to student learning: Inside the National Survey of Student Engagement. Change 33(3): 10-17, 66. Kuh, G. D. (2001b). The National Survey of Student Engagement: Conceptual Framework and Overview of Psychometric Properties, Indiana University Center for Postsecondary Research, Bloomington, IN. Kuh, G. D., Gonyea, R. M., and Rodriguez, D. P. (2002). The scholarly assessment of student development. In: Banta, T. (ed.), Building a Scholarship of Assessment, Jossey-Bass, San Francisco, pp Kuh, G. D., Hayek, J. C, Carini, R. M., Ouimet, J. A., Gonyea, R. M., and Kennedy, J. (2001). NSSE Technical and Norms Report, Indiana University Center for Postsecondary Research, Bloomington, IN. Kuh, G. D., Pace, C. R., and Vesper, N. (1997). The development of process indicators to estimate student gains associated with good practices in undergraduate education. Research in Higher Education 38: Messick, S. (1989). Validity. In: Linn, R. (ed.), Educational Measurement, (3rd Ed.), Macmillan, New York, pp Muffo, J. A., and Bunda, M. A. (1993). Attitude and opinion data. In: Banta, T. (ed.), Making a Difference: Outcomes of a Decade of Assessment in Higher Education, Jossey-Bass, San Francisco, pp National Center for Higher Education Management Systems (1994). A Preliminary Study of the Feasibility and Utility for National Policy of Instructional "Good Practice" Indicators in Undergraduate Education, U. S. Department of Education, Washington, D.C.NCES National Survey of Student Engagement. (2004). College Student Report, Indiana University, Bloomington, IN. Pace, C. R. (1984). Measuring the Quality of College Student Experiences, Center for the Study of Evaluation, University of California Los Angeles, Los Angeles. Pace, C. R. (1985). The Credibility of Student Self-reports, Center for the Study of Evaluation, University of California Los Angeles, Los Angeles. Pace, C. R., and Kuh, G. D. (1998). College Student Experiences Questionnaire, (4th Ed.), Indiana University, Bloomington, IN. Pascarella, E. T., and Terenzini, P. T. (1991). How College Affects Students: Findings and Insights from Twenty Years of Research, Jossey-Bass, San Francisco. Pascarella, E. T., and Terenzini, P. T. (2005). How College Affects Students: A Third Decade of Research, (Vol. 2), Jossey-Bass, San Francisco. Peterson, M. W., Einarson, M. K., Augustine, C. H., and Vaughan, D. S. (1999). Institutional Support for Student Assessment: Methodology and Results of a National Survey, National Center for Postsecondary Improvement, Stanford, CA. Peterson, M. W., and Vaughan, D. S. (2002). Promoting academic improvement: Organizational and administrative dynamics that support student assessment. In: Banta, T. (ed.), Building a Scholarship of Assessment, Jossey-Bass, San Francisco, pp Pike, G. R. (1994). Applications of generalizability theory in higher education assessment research. In: Smart, J. (ed.), Higher Education: Handbook of Theory and Research, (Vol. X), Agathon, New York, pp Pike, G. R. (1995). The relationship between self reports of college experiences and achievement test scores. Research in Higher Education 36: Pike, G. R. (1999). Assessment measures. In: Banta, T. W. (ed.), Assessment Update: The First Ten Years, National Center for Higher Education Management Systems, Boulder, CO, pp Pike, G. R. (2002). Measurement issues in outcomes assessment. In: Banta, T. (ed.), Building a Scholarship of Assessment, Jossey-Bass, San Francisco, pp

20 THE DEPENDABILITY OF NSSE SCALELETS 1 95 Pike, G. R., and Kuh, G. D. (2005). A typology of student engagement for American colleges and universities. Research in Higher Education 46: Pike, G. R., Kuh, G. D., and Gonyea, R. M. (2003). The relationship between institutional mission and students' involvement and educational outcomes. Research in Higher Education 44: Pohlmann, J., and Beggs, D. (1974). A study of the validity of self- reported measures of academic growth. Journal of Educational Measurement 11: Schreiner, L. A., and Juillerat, S. L. (1994). Student Satisfaction Inventory: 4-Year College and University Version, USA Group Noel-Levitz, Iowa City, IA. Shavelson, R. J., and Webb, N. M. (1991). Generalizability Theory: A Primer, Sage, Newbury Park, CA. Thissen, D., and Mislevy, R. L. (1990). Testing algorithms. In: Wainer, J. (ed.), Computerized Adaptive Testing: A Primer, Lawrence Erlbaum, Hillsdale, NJ, pp Wainer, H., and Kiely, G. L. (1987). Item clusters and computer adaptive testing: A case tor testlets. Journal of Educational Measurement 24: Wainer, H., Dorans, N. J., Green, B. F., Mislevy, R. J., Steinberg, L., and Thissen, D. (1990). Future challenges. In: Wainer, H. (ed.), Computerized Adaptive Testing: A Primer, Lawrence Erlbaum, Hillsdale, NJ, pp Received June 6, 2005.