The impact of 360-degree feedback on management skill development

Transcription

1 See discussions, stats, and author profiles for this publication at: The impact of 360-degree feedback on management skill development ARTICLE in HUMAN RESOURCE MANAGEMENT NOVEMBER 2006 Impact Factor: 1.52 DOI: /hrm CITATIONS 132 READS 1,558 3 AUTHORS, INCLUDING: Joy Hazucha Korn Ferry 10 PUBLICATIONS 222 CITATIONS SEE PROFILE Sarah A Hezlett University of Minnesota Twin C 23 PUBLICATIONS 1,174 CITATIONS SEE PROFILE Available from: Joy Hazucha Retrieved on: 17 November 2015

2 The Impact of 360-Degree Feedback on Management Skills Development Joy Fisher Hazucha, Sarah A. Hezlett, - and Robert J. Schneider This study investigated changes in the skill levels of managers which occurred two years following the managers' participation in 360-degree feedback. The major objective was to investigate how skill development, development efforts, and environmental support for development are related. The Management Skills Profile (MSP) was used to measure skills and to give feedback. Of the original 198 managers who received feedback, 48 accepted the opportunity to complete the instrument again two years later. At that time, they also completed a questionnaire surveying what development activities they had completed and the extent to which they received support for development from their supervisor and the organization. The first hypothesis was that, for the group, (a) participants' skills would increase following feedback, and (b) self-other agreement would be greater at Time 2 than at Time I. This was supported. The second hypothesis was that management skills would be related to later advancement. This also was supported. The third hypothesis was that skill development would be related to both development tforts and environmental support, and that some development activities would be more efecfive than others. Hypothesis four was that those who put more effort into development (a) would have received at Time 1 ratings which were more favorable and more congruent with their self-ratings, and (b) would have received more environmental support. Hypotheses three and four were only partially supported. The results have implications for what managers can do to develop their skills and how others can support this development by John Wiley & Sons, Inc. INTRODUCTION Over the past 15 years, the number of publicly available 360-degree feedback instruments has increased dramatically (Van Velsor & Leslie, 1991). Such instruments provide managers with feedback about their skills from multiple sources: typically their supervisors, peers, subordinates, and themselves. The popularity of these instruments is easy to understand, given the multiple benefits conferred to their users. First, feedback recipients receive valuable information which they may use to identify and address their shortcomings as well as to recognize and Human Resource Management, Summer/Fall 1993, Vol. 32, Numbers 2 & 3, Pp by John Wiley & Sons, Inc. CCC /93/

3 build upon their strengths. One reason this feedback is valuable is that it comes from multiple perspectives: not just from the supervisor, as in traditional performance appraisal, but also from peers and subordinates who work closely with the manager and whose opinions are important. As Borman (1991) has pointed out, each may provide relevant but slightly different information. In addition, the use of 360-degree feedback instruments gives co-workers the opportunity to praise or criticize their fellow employees anonymously. Finally, 360-degree instruments are relatively inexpensive, in terms of time, effort, and money. Despite their increasing popularity, relatively little is known about the impact of 360-degree feedback instruments on managers and organizations. Most of the pertinent research has focused primarily on reliability-and to some extent on validity. These instruments seem to have good internal consistency and interrater reliability, and they are related to other important criteria, such as potential, past and future success, and career jeopardy (see Van Velsor & Leslie, 1991, for a summary). But the question of whether using a 360-degree feedback instrument improves a manager's skills remains unanswered. In addition, little is known about the processes which make training and development methods, including 360-degree feedback instruments, more or less effective. There has, therefore, been a recurring call to shift away from the evaluation of specific training programs and toward the development of an understanding of why and when training is effective (Campbell, 1988; Tannenbaum & Yukl, 1992). Recent work in the domain of behavior change provides a theoretical framework for understanding how the use of a 360-degree feedback instrument may produce changes in management skills. Drawing on existing training methods, training models, and theories of behavior change, Hellervik, Hazucha, and Schneider (1992) proposed a framework that outlines critical steps in the process of behavior change. These include analyzing or assessing needs, assigning behavioral standards, formulating and protecting intentions to reach these standards, learning and expressing new behaviors in a change environment, and generalizing and maintaining new or changed behaviors in the day-to-day environment. The behavior change framework proposed by Hellervik et al. (1992) notes that at each step in the behavior change process, key requirements must be met in order to maximize the likelihood that behavior change will occur. For example, the needs analysis must be accurate and credible, the assigned behavioral standards should be specific and achievable, and the work environment should facilitate the emergence and maintenance of new behaviors. Previous research on the Management Skills Profile (MSP)(Sevy, Olson, McGuire, Fraser, & Paajanen, 1985) suggests that it fulfills many of the requirements for change associated with several steps in the behavior change process. First, the key requirements for the assessmentheeds analysis step are accuracy and credibility. Evidence for the MSP's accu- 326 I Human Resource Management, Summer & Fall 1993

4 racy includes fairly high internal consistency and interrater reliability coefficients. In addition, ratings are related to other concurrent and future criteria, including potential, past success, later success, and career jeopardy (summarized in Van Velsor & Leslie, 1991). The credibility of the feedback comes from its multiplicity of sources, all co-workers who were chosen as raters by the manager receiving the feedback. A needs assessment which comes from only one person can be dismissed as coming from a "flaky psychologist," a "biased boss," or a "disgruntled subordinate." However, if seven co-workers report that a person does not "treat others with respect," it is difficult to avoid taking their perceptions to heart. In addition, the needs assessment is geared to the individual receiving the feedback, not to a group of people. Thus, it is an individualized needs analysis which is both accurate and credible. The MSP also appears to meet the requirements for change in the second step of an effective behavior change intervention. In the Hellervik et al. (1992) framework, the assignment of behavioral standards step requires goals which are attainable and specific. The MSP feedback report includes a development planning guide containing one to two pages of development suggestions for each of the nine items which are most in need of development. These suggestions should help feedback recipients make the link between their own needs and the actions they should take to develop these skills. In addition, the development planning guide gives people suggestions as to how to interpret their results and how to put together a development plan. This is the initial phase in the third step in the behavior change process, intention formulation and protection. The development suggestions are mostly for activities that can be performed on the job. Because behavior change takes place on the job, the final stage in the behavior change process-transferring the new behaviors from the change environment to the everyday environment-is not an issue. The major objective of this study is to better understand 360-degree feedback as a management development intervention. Key questions include: How much do managers' skills develop after feedback? How are initial feedback, development efforts, and environmental support related to skills development? Which development activities are most effective? What affects the level of effort managers put into their development? The answers to these questions have implications for how managers can develop their own skills and also for what human resource professionals can do to facilitate this development. This study has important features which differentiate it from other research on management development and transfer of training. Burke and Day (1986) found that short time frames and reliance on self-report measures typified management development research. Baldwin and Ford (1988) found that even research on transfer of training typically has been based on short-term, single-source, perceptual data. By contrast, this study (a) spans two years, (b) investigates the relationships between Hazucha, Hezlett, & Schneider: Impact of Feedback on Development / 327

5 self-reported development and other-reported change, (c) includes measures of on-the-job behavior, and (d) includes multiple measures of effort and change by multiple observers. Key pieces of the behavior change model described by Hellervik et al. (1992) include the effort put into development and the support for behavior change and for transfer to the environment. Unfortunately, Baldwin and Ford (1988) noted that operationalizing such variables has lagged behind anecdotal evidence of their importance. As a consequence, there are no established measures of these constructs and little empirical research upon which to draw. Therefore, most of the hypotheses will be based on the behavior change model proposed by Hellervik et al. (1992), which in turn was based on theoretical and empirical work on behavior change. In addition, we had to develop measures of the pertinent constructs. HYPOTHESES The first hypothesis is that 360-degree feedback is useful for management development. Participants skills at follow-up (Time 2) should be stronger than they were initially (Time 1). In addition, participants selfperceptions should be more similar to others perceptions at Time 2, as a result of the participants having received feedback. Therefore, the first hypothesis is: Hypothesis la: As a group, there will be significant improvement in skills from Time 1 to Time 2, as measured by self and others ratings. Hypothesis lb: The agreement between self and other ratings will be higher at Time 2 than at Time 1. A key assumption underlying the use of 360-degree feedback instruments is that they provide feedback about areas that are important to managers careers. If the management skills measured by the MSP are not important, any effort spent developing them, or studying them, is wasted. Therefore, the second hypothesis is a test of this assumption. Hypothesis 2: Managers whose skills are rated more highly (at Time 1) will receive larger pay increases and will be more likely to receive promotions between Time 1 and Time 2. It is likely that some participants skills will develop more than those of others. It is reasonable to expect that individuals who put more effort into development activities are more likely to improve their skills than are those who do not engage in development activities. We propose: Hypothesis 3a: Managers who put more effort into their own skill development will be more likely to improve their management skills. 328 I Human Resource Management, Summer & Fall 1993

6 We also expect that not all development activities will be equally beneficial to development. Specifically, we expect that those activities which involve more steps in the behavior change process proposed by Hellervik et al. (1992) will be more strongly related to skill development than those which involve fewer steps. Hypothesis 3b: Activities which require follow-through will be more effective than those which involve merely understanding the results and formulating intentions. Baldwin and Ford (1988) propose that an environment that supports development is critical to maintaining a changed behavior in the work environment. Therefore, we propose that: Hypothesis 3c: Managers who receive more support from their supervisors will be more likely to improve their management skills. Hypothesis 3d: Managers who receive more support from their organizations will be more likely to improve their management skills. The previous hypotheses concern the development of management skills after feedback. An assumption made by management development practitioners is that development efforts are related to skill development. The next hypotheses focus on the antecedents. of these development efforts. As discussed previously, feedback from the MSP tends to be both accurate and credible. However, there are additional reasons why trainees may react negatively to a skills assessment and may, therefore, be less likely to take the key steps to change their behaviors. Therefore, variables that may influence managers to accept or reject feedback should be related to the completion of developmental activities. DeGregorio (1991) and Jones and Whitmore (1992) found that participants who received more favorable ratings were more likely to accept them. In addition, DeGregorio found that participants were more likely to accept feedback that was less discrepant from their own ratings, regardless of its favorability. Therefore, we propose: Hypothesis 4a: Managers who receive more favorable ratings at Time 1 will be more likely to complete development activities. Hypothesis 4b: Managers whose feedback from others is closer to their self-ratings will be more likely to complete development activities. Two studies have found that perceived support for development may influence participation in self-development activities. Jones and Whitmore (1992) found that perceived support was related to acceptance of feedback and perceived effectiveness of development activities (but not Hazucha, Hezlett, & Schneider: Impact of Feedback on Development I 329

7 to number of development activities.) Noe and Wilk (1991) found that attitudes and perceptions of support influenced employees participation in development activities, but the results were not consistent across organizations. Hypothesis 4c: Managers who receive more support for development from their supervisors and their organizations will be more likely to complete development activities. METHOD Design Between 1987 and 1988 (Time l), 198 managers and their co-workers completed the Management Skills Profile (MSP). Two years later (Time 2), all were given the opportunity to retake the MSP and to complete the Management Development Questionnaire (MDQ), a questionnaire which asked about effort, change, development activities, and support for development. At Time 2 the MSP and the MDQ were also completed by the managers supervisors, peers, and subordinates. Participants The participating managers were employed by a large midwestem utility company. The 48 managers who took the MSP at both times and the MDQ at Time 2 comprise the main sample (the retakers). The group of 150 managers who took the MSP at Time 1, but who chose not to retake the MSP (the non-retakes) were compared with the retakers to assess the retakers representativeness. Measures The Managemenf Skills Profile: The MSP is an instrument used to encourage and facilitate management skill development by providing feedback about managerial proficiency on 19 job-related dimensions. The dimensions and their definitions are shown in Table I. Responses to the 122 items are collected from the manager himiherself, the supervisor(s), peers, and subordinates. Each item on the MSP is a description of positive managerial behavior on which the manager may be rated from 1 (not at all) to 5 (to a very great extent). A Does not apply response option may be selected. Specific psychometric information about the MSP is summarized in Van Velsor and Leslie (1991). The MSP feedback report shows the average data within each per- 330 I Human Resource Management, Summer & Fall 1993

8 Table I. Management Skills Profile: Dimensions and Definitions. Administrative Planning Organizing Personal Organization & Time Mgmt. Communication Informing Oral Communication Listening Written Communication Cognitive Problem Analysis & Decision-Ma king Financial & Quantitative Interpersonal Human Relations Conflict Management Leadership Leadership Style & Influence Setting goals and developing strategies and schedules for meeting those goals; anticipating obstacles and defining alternative strategies. Scheduling and coordinating work of others; setting priorities; establishing efficient work procedures to meet objectives. Allocating one s own time efficiently; arranging informa tion systematically and processing paperwork and other information effectively without getting bogged down in detail. Letting people know of decisions, changes, and other relevant information on a timely basis. Speaking effectively one-to-one and in groups; making effective presentations. Demonstrating attention to, and conveying understanding of, the comments or questions of others. Writing clearly and effectively; using appropriate style, grammar, and tone in informal and formal business communications. Identifying problems; recognizing symptoms, causes, and alternative solutions; making timely, sound decisions even under conditions of risk and uncertainty. Drawing accurate conclusions from financial and numerical material and applying financial principles and numerical techniques to management problems. Developing and maintaining smooth, cooperative working relationships with peers, subordinates, and superiors; showing awareness of, and consideration for, the opinions and feelings of others. Bringing conflict or dissent into the open and using it productively to enhance the quality of decisions, arriving at constructive solutions while maintaining positive working relationships. Taking charge and initiating actions; directing the activities of individuals and groups toward the accomplishment of meaningful goals and commanding the attention and respect of others. (continued) Hazucha, Hezlett, & Schneider: Impact of Feedback on Development I 331

9 Table I. (Continued) Motivating Others Delegating & Controlling Coaching & Developing Other Personal Motivation Personal Adaptability Occupational & Technical Knowledge Results Orientation Creating an environment in Ghich subordinates and others are rewarded for accomplishment of group and individual goals. Clearly assigning responsibilities and tasks to others and establishing effective controls; ensuring that employees have the necessary resources and authority; and monitoring progress and exercising control. Evaluating employees, providing performance feedback, and facilitating professional growth. Displaying a high energy level, working long and hard to get things done, and seeking increased responsibility on the job. Responding appropriately and competently to the demands of work challenges when confronted with changes, ambiguity, and adversity, or other pressures. Applying the knowledge and skills needed to do the job, including technical competence in one's own field and familiarity with policies and practices of the organization and the industry. Producing high-quality work, managing productively, getting results. spective: self, supervisor, peer, and subordinate. The three "other" perspectives are then averaged with each perspective receiving equal weight. In other words, the supervisor is given the same weight (1/3) as the peer average (113) and the subordinate average (1/3). In this study, we focus on the average of the ratings on the 19 dimensions: self and average other, measured at Time 1 and Time 2. Two additional variables reflect the change from Time 1 to Time 2. These are the residuals of the correlations (a) between the ratings from others at Time 1 and Time 2, and (b) between the self-ratings at Time 1 and Time 2. These residuals1 can be interpreted as Time 2 skill level, taking Time 1 skill level into account. Self-other congruence at Time 1 was measured by computing the absolute difference between self- and other ratings. The Managemmf Development Questionnai~e (MDQ): The MDQ was created specifically for this study and is based on interviews with consulting psychologists about behaviors which operationalized the constructs of interest. As shown in Table 11, the participants were asked about their advancement, development activities, supervisor support, and organizational support. Partidpants and their co-workers also rated both the degree of effort put into development and the change in skill for each of the 19 MSP dimensions. Raters were asked for both ratings because 332 / Human Resource Management, Summer & Fall 1993

10 Table 11. Structure of Managment Development Questionnaire (MDQ). Self Other Advancement Rater type Effort Change Management development activities MSP results Training Other activities Supervisor support Organizational support Section One: Background Information Changed positions since last MSP? (no, promotion, demotion, lateral move) Salary? What Percent Perspective (supervisor, peer, subordinate) Section Two: Effort and Change Ratings In the last two years, how much effort have you invested into each of the following areas? In the last two years, how much have you changed in each of the following areas? In the last two years, how much effort have you seen this person invest in each of the following areas? In the last two years, how much has this person changed in each of the following areas? Section Three: Management Development Climate After you received your MSP report, which of the following steps did you take? Which training programs have you attended in the last two years? What other management development activities have you experienced? How often does your boss take the following steps to help you develop? How well does each of the following statements describe your organization? people distinguish between these two in conversation. (e.g., He really tried, but he has not changed ), and we did not want raters to be forced to merge the two concepts in their responses. MDQ change and effort ratings were treated in the same way the MSP ratings were treated. First, ratings were averaged within perspective (supervisor, peer, subordinate). Next, the three perspective averages were averaged into a composite. A factor analysis of the effort and Hazucha, Hezlett, & Schneider: Impact of Feedback on Development / 333

11 change ratings showed that four factors accounted for much of the variance: other effort, other change, self-effort, self-change. Therefore, averages were computed for each of these resulting in four grand averages: self-effort, other effort, self-change, other change. Effort ratings. On a scale that ranges from 0 (no effort) to 4 (phenomenal effort), the average effort rating from others was 2.21 (SD = 0.70), and the corresponding rating from the managers themselves was 2.32 (SD = 0.40). Both of these means fall slightly above the scale point 2 which is described as "moderate effort" (3 is "significant effort"). The standard deviation of 0.70 for the "other" ratings indicates substantial variability: 95% of the managers fall between 0.92 (slight effort) and 3.72 (phenomenal effort). Change ratings. On a scale from -1 (skill has deteriorated) to 4 (dramatic improvement), the grand means for other ratings and self-ratings of degree of change were almost identical: 1.58 and 1.57, respectively. These averages fall slightly closer to the scale point 2 "noticeable improvement" than to 1 "slight improvement" across dimensions and individuals. Again, the standard.deviation of 0.77 for others indicates that the +/-2 SD interval covers a wide range: 0.04 (no noticeable change) to 3.12 (significant improvement). The two types of change measures-change ratings and residuals from the MSP-measure different information. The change ratings capture raters' perceptions of the degree of change and require that raters have known the ratee from Time 1 to Time 2. By contrast, the residuals compare two "snapshots" taken at different points in time. For various reasons (Beutler & Crago, 1983), these two types of change measures may yield different results, and using both will help us to better understand change and its measurement. The relationships among the different indicators of effort and change vaned considerably, as can be seen in Table 111. Measures based on others' ratings should be more reliable than those based on selfratings, because they are aggregated across people. In addition, the residuals are more reliable than the ratings of change because they are based on more observations: 122 behaviors at each of two points in time, rather than 19 dimensions at one point.2 Therefore, we expected (a) others' ratings to have higher correlations than self ratings, and (b) residuals to have higher correlations than change ratings. In addition, we expected within-perspective (i.e., self with self, other with other) ratings to intercorrelate more highly than (self with other) measures from different perspectives. Consistent with our expectations, correlations which used only variables based on others' ratings are as large or larger than those involving variables based on self-ratings. In addition, measures of skill change based on ratings from the same source tended to be more highly related. However, this pattern is not perfect. Residual change scores calculated from self-ratings correlated more highly with residual change scores from 334 f Human Resource Management, Summer & Fall 1993

12 . w 0 ul Table 111. Correlations among Skill, Support, Effort, Change, Advancement, and Demographic Measures Time 1 Skills 1 Time 1 MSP: Self 2 Time 1 MSP: Other 3 Avg. Abs. Diff. Effort and Support 4 Avg. spv support 5 Avg. org. support 6 Total dev. activities 7 Avg. effort: Self 8 Avg. effort: Other Time 2 Skills 9 Time2MSP: Self 10 Time 2 MSP: Other Change 11 Avg. chg rating: Self 12 Avg. chg rating: Other 13 Avg. MSP resid: Self 14 Avg. MSP resid: Other Advancement 15 % Salary change 16 Promotionn Demographics 17 Sexb 18 Age 07-23' ''' 08 30" "' -43"' ** "' ' 17-19' ' ' ()4'*' 26" 07 44"* 28'* ** 24' '- 36"' 03 26" 27'. 26" '' " " *** 38*** 70." '" 29" " 10 49"' '' " 32" 25" 93"' 29t' "' " 24' 26*' 88**' '" 29' " "' 39"' -20' "' 24' " -23' -.27** N = 'p <.lo, "p <.05, "'p < = Not promoted, 1 = promoted bmale = 1. female = 2.

13 other ratings (r =.29, p <.05) than they did with self-ratings of effort (I =.09, ns). Both self and other raters saw effort and change as strongly related, but they did distinguish between them, as shown by the correlations between ratings of effort and ratings of change. Within rater type (self or other), the change rating is more strongly related to the effort rating (r =.70, p <.01 for both self and other) than it is to the residual (self r = -24, p <.lo; other r =.51, p <.01). The self-rating of effort correlates only r =.15 (ns) with others ratings of effort. However, the self residual (a measure of change) correlates r =.27 and.29 (both p <.05) with the other measures of change (the rating and other residual, respectively). This pattern of relationships suggests good evidence of construct validiky, but also suggests that there is not a one-to-one correspondence between actual changes in skill development and perceptions of such change. When change has occurred, the self or others may not notice it. On the other hand, change may be reported but may not be apparent from comparing Time 1 and Time 2 ratings. Both actual change and peoples perceptions of it are important to peoples careers. It is important, therefore, to study both change and perceptions of change and to understand their commonalities and their distinctions. Dmeloprnent activities. There were three broad groups of development activities: MSP action steps, training programs, and other development steps. The MSP action steps were taken from the MSP development planning guide, which gave people advice about what steps to take to develop. These steps included reviewing results and preparing a development plan. The training programs listed were those offered by the participants organization, some of which were developed based on the group s MSP results. The other development steps included coaching, job rotation, task forces, and other steps that people can take to develop their skills on the job. Managers who accepted the opportunity to receive feedback at Time 2 actively pursued career development: 14 of the 25 development activities were engaged in by more than half the participants. The most frequent MSP development activities involved reviewing results and suggestions and preparing a development plan. MSP development activities undertaken by fewer than a third of the managers included reviewing development plans and progress quarterly, obtaining input into development plans from co-workers, and obtaining input into development plans from subordinates. Training programs were fairly well attended. Roughly two-thirds attended the performance management training program, and one-half attended the program on problem-solving. Forty-two percent attended other external programs. Of the other management development activities, roughly twothirds of the managers participated in task forces or special assignments. About one-half read books or listened to tapes about management, and 336 I Human Resource Management, Summer & Fall 1993

14 roughly 40% received coaching and feedback. Job rotation was the least frequently used development activity (12.5%). An exploratory factor analysis was conducted on the 25 management development activities to determine how the activities might be clustered.3 Seven composites were formed by counting the number of activities from that group in which participants engaged. These composites were 1. development planning 2. review progress 3. general review of results 4. on-the-job assignments 5. involve supervisor and set target dates 6. job rotation, coaching, and feedback 7. training programs Table IV shows the means, standard deviations, and intercorrelations of the composites. The internal consistency reliabilities of the composites, also reported in Table IV, are somewhat higher than expected, given the small number of items per scale. As shown in Table IV, intercorrelations among these composites ranged from.59 to The (2) review progress composite had the most consistent relationships with other activities: (4) on-the-job assignments (r =.32, p <.05), (5) involving supervisor, setting target dates (I =.34, p <.05), and (6) job rotation and coaching (Y =.32, p <.05). Each of these four included relatively long-term actions and several involved the supervisor. Training programs (7) was related to a general review of the MSP results (3) (r =.26, p <.05). These were the most traditional and least involving development activities. Supervisor supporf. On a scale of 0 (never) to 5 (whenever possible), the grand average supervisor support rating was 2.55 (SD =.86). This indicates that the typical supervisor acted to help managers develop between the two scale points occasionally (2) and moderately often (3). The actions supervisors most often used to help managers to develop tended to indicate general supportiveness, such as includes suggestions for development in performance appraisals (Y = 3.37), provides encouragement and support (Y = 3.0), and wants me to get ahead (Y = 3.0). Respondents reported that supervisors did these moderately often or more. The least frequent activities were those which involved specific input into development: regularly reviews progress with me (F = 2.19), gives input to my development plan (X = 2.19), and recommends training programs that would be helpful (E = 2.0). These means indicate occasional support in these areas. The Cronbachs alpha for the supervisor support scale (13 items) is.94. Orgunizufional support. There were 17 organizational support items which were rated on a scale from 1 (strongly agree) to 5 (strongly disagree).4 Average organizational support was 3.44 (SD =.43) which was Hazucha, Hezlett, & Schneider: Impact of Feedback on Development / 337

15 W gr,. R 5 w Table IV. Development Activity Composites: Means, Standard Deviations, Reliabilities, and Intercorrelations. Intercorrela tions Composites Mean SD Development planning (. 68) (.83) 2. Review progress General review (-58) 4. On-the-job assignments (.62) 5. Involve spv, set target dates (.55) 6. Job rot., coaching & feedback (.45) 7. Training programs ( - 30) N = 48. Note. Underline indicates p <.05. Numbers in parentheses are reliabilities (Cronbach s alphas).

16 slightly above the neutral midpoint of the scale. The most favorable responses indicated beliefs that: training is not considered punishment (X = 4.17); managers can be developed (X = 3.89); and skill development is relevant to getting ahead (Z = 3.85). In addition, the organization was rated above neutral (3) on statements about the facilitation of management development through planning and programs. However, organizational support for management skill development appears to fall short of rewarding those involved in development activities. The average rating of the items Managers and superiors are evaluated and rewarded for training and development activities and Bosses are evaluated by their success in developing subordinates fell below the neutral point on the scale; they were more disagreed with than endorsed. The Cronbach s alpha for the organizational support scale (16 items) is.78. The correlation between the supervisor support and the organizational support composites is.64 (p <.001) which indicates a link between managers perceptions of the two sources of support. RESULTS Comparability of the Sample of Retakers with the OriginaI SampIe To check the comparability of the main sample with the original larger sample, we contrasted these two groups on (a) their average skill ratings (by others and self) at Time 1, and (b) the absolute difference between their self-ratings and other ratings. The effect sizes corresponding to the differences between retakers and non-retakers on the MSP were -.06, -.08, and -.08, respectively, based on the average self-rating, average other-rating, and the difference between self and other. None of these is significantly different from zero. The small magnitude of differences between retakers and non-retakers generates confidence for a generalization of the results obtained in the retaker sample.5 Impact of 360-Degree Feedback The first hypothesis was that participants skills would improve between initial feedback and follow-up. To test this hypothesis, we performed paired t-tests between (a) average other ratings at Times 1 and 2, and (b) self-ratings at both times. The results of the t-test support this hypothesis and are also consistent with the average ratings of change which indicate slight to noticeable improvement. Average MSP ratings, by self and by others, at Time 1 and Time 2, are shown in Table V. The effect size for the average rating by others is d =.29 (t = 1.94, p <.lo). Hazucha, Hezlett, & Schneider: Impact of Feedback on Development / 339

17 Table V. Time 1 and Time 2 Skills (Management Skills Profile). Means con Time 1 Time 2 (Tl-T2) d t N Average MSP rating: Other * Average MSP rating: Self *** * 48 p <.lo, p <.05, ***p <.01. The effect size for the average self-rating is slightly larger: d =.40 (t = 2.44, p <.05). These effect sizes are quite respectable relative to those found in the management development literature (e.g., Burke & Day, 1986). These results indicate that (a) the group improved over the twoyear period, the improvement was detectable by both the self and others using the MSP. The correlation between average other ratings at Time 1 and Time 2 is.48 ( p <.01). For self-ratings, the correlation between the averages is.38 (p <.05).6 Hypothesis lb was that the correlation between self and other ratings would increase from Time 1 to Time 2. To test this, we contrasted the self-other correlations obtained at each point in time. The self-other correlation at Time 1 was r =.07, and at Time 2 the correlation was.32. The z corresponding to the difference between these two correlations was 2.0 which is significant at p <.05. This hypothesis was also supported indicating that, as a group, participants self-ratings were more consistent with others ratings at Time 2 than they had been at Time 1. Relationship between Management Skills and Advancement The dependent measures for Hypothesis 2, that management skills are positively related to advancement, were promotion (0 = no, 1 = yes) and percent change in salary, both self-reported. The correlations in Table III show that management skills at Time 1 are related to advancement during the two-year period following feedback. The correlations are r = -19 between others ratings and promotion, and T =.23 between self-ratings and salary change (both p <.lo). This is true despite the fact that MSP results were not used to make administrative decisions and were not even available to the decision-makers. These results lend support to the hypothesis: Management skills are relevant to managers career outcomes. Time invested in developing such skills (and in studying them) is likely to be well spent. 340 I Human Resource Management, Summer & Fall 1993

18 Influences on Skill Development It is clear from the standard deviation of change that, while the group as a whole developed its skills, there were also large individual differences in how much people developed. The third hypothesis proposed that development efforts and environmental support for development would be related to how much people's skills improved. Development activities. Correlations between groups of development activities and all the measures of skill development are shown in Table VI. Correlations between development activities and others' change ratings tended to be nonsignificant or negative, but activities were related to the residual of others' skill ratings. This means that those who engaged in more development activities were rated higher at Time 2, but observers did not remember that this was a change from Time 1. Development activities were related to both self-ratings of change and to the self residual. Of the development activity composites, the total number of development activities and "review progress" were strongly related to three of the four measures of change (all but the change rating by others). General reviews of results was positively related to self-rated change, but negatively related to other-rated change. Managers themselves saw a relationship between "on-the-job assignments" and their degree of Table VI. Correlations among Development Adivities, Environmental Support, and Skill Change Other Self Change Residual Change Residual Rating Change Rating Change Development Activities All management dev. actions Development planning Review progress General review of results On-the-job assignments Involving spv, setting target dates Job rotation, coaching and feedback Training programs Environmental Support Average supervisor support Average organizational support " ** 0.37**" "'" ** 0.27** *** 0.22' 0.36*** 0.31** 0.36"" *** 0.36"*" 0.29** ** * * 0.27" 0.26"* N = *p <.lo, **p <.05, ***p <.01. Hazucha, Hezlett, & Schneider: Impact of Feedback on Development / 341

19 change. Attendance of training programs was related to both the self and other residuals. Three specific activities were related (at least p <.lo) to at least three of the four change measures: reviewed plans and progress at least quarterly, obtained input into plan from co-workers, and received coaching and feedback. It is interesting to note that these activities, which were most related to skill development, were among the least frequently performed: 31.3% of the participants reported doing each of the first two, and 39.6% received coaching and feedback. These results provide support for the hypothesis that those who engaged in more development activities would change their skills more than those who pursued fewer development activities. The results also support Hypothesis 3b, that activities which involved taking more steps in the behavior change process would be more strongly related to skill development. For example, reviewed results in detail involves gaining an understanding of the assessment (step 1 in the Hellervik et al., 1992 model), wrote down a development plan involved formulating an intention to change (step 2 in the model), but neither was as strongly related to change as reviewed plans and progress at least quarterly which is part of intention protection or follow-through (step 3). Environmental support. The relationships between support and change in management skills are also shown in Table VI. Support from the supervisor is positively related to self-reported skill development (change rating and residual) but not to change as reported by others. The item most strongly related to the residual of others ratings is provides coaching in specific areas (r =.24, p <.05). These results indicate only partial support for the hypothesis that support from the supervisor would be related to skill development. Participants who report more support also report more change in their skills, but their observers do not report a similar increase in skiils. Organizational support is significantly correlated with three measures of change: change ratings by self, change ratings by others, and the self residual. Several correlations were greater than r = 30, including: the content of training programs is consistent with organizational values (r =.30, with other change rating and self residual, p <.05) bosses are evaluated by their success in developing subordinates (T =.37 with self change rating, p <.01) - our business planning includes development planning (7 =.32 with self residual, p <.05) training is considered punishment (I =.31 with self residual, p <.05). These results generally support the hypothesis that organizational support is related to skill development. However, organizational support is not related to what may be the most important of the change measures, 342 I Human Resource Management, Summer & Fall 1993

20 the residual of the others ratings, which is a measure of how observers view participants actual skills. Characteristics of Those Pursuing Self-Development Initial skill ratings. We expected that those who initially received ratings which were (a) more favorable and (b) more congruent with their self-ratings would put more effort into development. Table I11 shows that this hypothesis was not supported. Those who received less favorable ratings from others at Time 1 reported putting more effort into development (r = -.27, p <.05). Others ratings were not related to the number of development activities or to the effort observed by others, and self-ratings at Time 1 were not related to any of the measures of effort. A second part of this hypothesis was that those who received ratings that were more congruent with their self-ratings would put more effort into development. This hypothesis was not supported; congruence was not related to any of the effort measures. Support from Supervisor. We expected that those who reported receiving more support from their supervisor and from the organization would put more effort into their own development. As shown in Table 111, support from supervisor is strongly related to average self-ratings of effort (r =.44, p <.01) and to total number of development activities (r =.26, p <.05), but the correlation with other-rated effort (r =.11) is non-significant. The four supervisor support items that are related (at p <.05) to two different measures of effort are: gives feedback on my performance gives input into my development plan encourages my motivation to advance provides coaching in specific areas These findings support the hypothesis regarding a relationship between supervisor support and development efforts from the participants points of view. It is interesting that those who receive more support from their supervisors report putting more effort into their own development and engaging in more development activities, but others do not seem to notice these development efforts. Orgunizufionul support. Table I11 also shows correlations between organizational support and the measures of development activity. Average organizational support is related to self-rated effort (r =.28, p <.05), but the relationships with other-rated effort and number of development activities are non-significant (r =.17 and.07, respectively). One specific item is related both to self- and-other-rated effort: We have a formal and Hazucha, Hezlett, & Schneider: Impact of Feedback on Development / 343

21 active career development program. This means that those who reported that the organization had a formal and active career development program reported putting more effort into their development but did not actually complete more development activities. These results provide partial support for the hypothesis that organizational support and development efforts would be related. It is unfortunate that those who perceive greater support from their organization report putting more effort into development, but they do not actually perform more development activities, and others do not report seeing more effort. It is also unfortunate that self-reported effort is not related to change in skills as reported by others, so increasing participants perceptions of their effort is not likely to be productive if the goal is to increase managers actual skills. Other Findings Several other correlations from Table 111 are noteworthy. This table shows correlations between overall variables: skills at Time 1 (self, other, and difference), supervisor and organizational support, development activities, effort ratings, skills at Time 2 (self and other), change measures, advancement, gender, and age. The order of the variables roughly corresponds to the sequence of events in time. Three of the.effort and support variables (support from supervisor, organizational support, and self-rated effort) are negatively related to Time 1 average other ratings (r s = -.37, -.43, -.27, p s <.01,.01,.05, respectively.) In other words, those who received lower ratings from others report receiving more support and exerting more effort than those who received higher ratings. Self-ratings of effort are related to supervisor and organizational support and also to number of development activities (r s =.44,.28,.33; p s <.01,.05,.05, respectively), but others ratings of effort are related to none of these. This means that those who reported receiving more support and engaging in more development activities reported exerting more effort, but others reports of effort were not related to either development activities or perceived support. Self-ratings at Time 2 are related to Time 1 self-ratings (r = -38, p <.05), and to supervisor support, organizational support, and development activities (r s =.27,.24,.28; p s <.05,.lo,.05, respectively). Time 2 other ratings are related to ratings from others at Time 1, other-rated effort, and number of development activities. In other words, Time 2 skill ratings are related to both initial skill levels and number of develop- ment activities. Different measures of change-collected from participants and observers, retrospective perceptions of change and residuals of the Time 1-Time 2 correlation-are related to different variables. Both self mea- 344 l Human Resource Management, Summer & Fall 1993

22 sures of change are related to number of development activities and to support from the supervisor and the organization. In addition, those who rated their initial skills higher reported more change. In contrast, other-rated change is related only to organizational support. The other residual is related to development activities. Advancement is related to initial skill ratings and the self residual but not to effort, support, or change variables. Correlations with gender indicate that women received higher ratings at Time 2 but not at Time 1, and therefore had larger residuals (i.e., they changed more from Time 1 to Time 2). However, they rated their own effort and change lower. In addition, they received larger increases in salary between Time 1 and Time 2. There were no gender differences in the correlates of skill development: others ratings of effort, development activities, or environmental support. There were also several age effects. First, the negative correlation between age and sex indicates that women were younger. Younger participants received higher ratings at Time 1, had larger salary increases, and were more likely to be promoted. Older participants reported putting more effort into development, and others shared this perception of greater effort. Despite the extra effort, older participants did not change more. DISCUSSION The results generally support hypothesis 1: 360-degree feedback is an effective management development tool. The effect sizes-29 according to others ratings and.40 according to self-ratings-are quite respectable compared with Burke and Day s (1986) meta-analysis of the effectiveness of management development. It is also encouraging that these results indicate that (a) change occurred on the others noticed it, and (c) broad, complex skills are what changed. This is good news concerning the impact of 360-degree feedback and its usefulness for managers who want to manage their own development. It is also good news for the literature on management development, as it indicates that behavior change occurs in field settings even when the measures are not selfreport perceptual measures. If the participants continue to develop to the same degree in subsequent years, the difference in their skills would grow from noticeable to clearly apparent. The increased agreement between self- and others ratings from Time 1 to Time 2 indicates an increase in participants understanding of how others viewed their skills after feedback. The fact that self-ratings at Time 2 were more consistent with others ratings collected at the same time than they were with others Time 1 ratings indicates that participants were not reporting merely what they remembered about their (earlier) ratings. The increased agreement may be due to ongoing Hazucha, Hezlett, & Schneider: Impact of Feedback on Development / 345