: Implications for Best Practices Kenneth M. Nowack, Ph.D. Consulting Tools Inc. knowack@consultingtools.com 310-450-8397 Paper presented at the 20 th Annual Conference of the Society for Industrial and Organizational Psychology, Los Angeles, March 2005
ABSTRACT Managers (67) and raters (127) participating in a corporate leadership program utilizing a 360 feedback instrument were asked about perceptions and reactions one-year later. Participants reported that both positive and negative feedback was largely expected and the feedback increased motivation to make behavior changes largely on development areas (82.1%) as opposed to strengths (18.9%). Implications for best practices and future research are discussed. 2
Evaluation of a 360 Feedback Program: Implications for Best Practices The use of multi-rater or 360-degree feedback, the process in which direct reports, peers, team members, and bosses provide anonymous feedback to managers for coaching, development and performance evaluation continues to grow in popularity (e.g., Nowack, 1999). Antinoni (1996) reported that an estimated 25% of all companies use some type of upward feedback and Atwater and Waldman (1998) suggest that 90% of Fortune 1000 firms use some type of multi-rater feedback system as part of appraisal or development systems. Increasingly, multi-rater feedback systems have proliferated and are being used for diverse purposes (e.g., executive coaching, performance evaluation, talent management/succession planning, and leadership development). The practical use of multi-rater feedback is often based on expert opinion, publisher s suggestions or human resources fads, rather than, on empirical research or evaluation. In fact, there is a paucity of well designed research and evaluation studies to actually guide practitioners in the effective administration, interpretation and use of multi-rater feedback systems. Extending the current research literature on multi-rater feedback systems can only improve the practical implementation for a wide variety of organizational, team and individual uses and interventions. Recent research on 360-degree feedback suggests that some feedback recipients may not be benefiting from the process while others demonstrate significant improvements in performance (Atwater, Waldman & Brett, 2002; Atwater, Roush & Fischthal, 1995; Reilly, Smither & Vasilopoulos, 1996). It is not uncommon for recipients to experience strong emotional reactions to 360 degree feedback (Illgen & 3
Davis, 2000; Kluger & De Nisi, 1998). Some research has suggested that individuals may even experience discouragement and frustration when 360 degree feedback is negative or not as positive as expected (Brett & Atwater, 2001). Furthermore, it is expected that the different rater groups will provide somewhat conflicting information and data in 360 feedback processes (e.g., Harris & Schaubroeck, 1998; Nowack, 1992; Greller and Herold, 1975). These meta-analytic studies have consistently shown weak agreement between self-ratings and ratings made by others. Kluger and DeNisi s (1996) meta-analysis on the impact of performance appraisal feedback on performance also raised the issue that performance improvement is not the only outcome that can be expected. In fact, they concluded that in one third of the cases feedback actually resulted in decreased performance. Atwater, Waldman, Atwater and Cartier (2000) reported improvement following an upward feedback intervention only resulted for 50% of the supervisors who received it. It seems that organizations using 360 degree feedback would want to ensure that employees have positive reactions to the process and that it results in performance enhancements back on the job. It is important to understand how both participants and raters view a corporate 360 degree feedback process and how to maximize its effectiveness to translate increased awareness into individual and team behavior change. This one year follow-up study investigates the reactions of participants and raters around a 360 degree feedback process utilized within a corporate leadership development program. 4
Method Sample A multi-rater feedback process was utilized within a structured corporate leadership development program. This 3-day program was targeted to high performance managers and utilized a validated 360 feedback instrument (Manager View 360; Nowack, 1997), a personality inventory (FIRO-B) and career assessment tool (Career Profile Inventory). Program participants were given the results of these assessment tools within the 3-day workshop design and also met privately with an external psychologist for and individual feedback meeting (2 hours) to discuss the 360 results following the workshop. Participants (n=67) were high potential managers within a large international newspaper company, were largely male (59.6%) and had fairly long tenure (21.3% worked for the company more than 3 years and less than five and 25.% were employed for more than five years). Feedback from the three assessment tools ware shared only with the program participant and no other information was shared with anyone else in the organization, including the participant s own manager. Participants were encouraged to complete an individual professional development plan following the confidential meeting with the external psychologist and to present this plan to their manager. A memo from Human Resources was sent to each senior manager of the participant in this program asking them to meet with their direct report to discuss the professional development plan that would come out of the 360 feedback report and other workshop assessment data. No other attempts were 5
made to ensure that the high potential participant and his manager actually met to share and discuss the possible implementation of the professional development plan. Procedure Administration of 360 evaluation surveys. Two separate 360 evaluation surveys (Participant Survey and Rater Survey) were sent by internal mail to each rater and their selected raters approximately one-year following participation in the corporate leadership training program by the Human Resource department and returned directly to the researcher by mail. A total of 47 participant surveys and 167 rater surveys were returned (67 participant and 402 rater evaluation surveys were sent out) for a response rate of 70.1% and 41.5%, respectively. Measures 360 feedback evaluation survey. Two separate evaluation surveys were created for the purposes of this study. The Participant survey contained 33 questions and two personal information items and the Rater survey contained13 questions and two personal information items. The Participant Survey contained separate questions on the 360 feedback content of the corporate leadership workshop, the instrument used, feedback results, feedback impact, follow up with their managers and others and developmental planning. The Rater Survey contained questions focusing on behavioral observations of behavior change and the impact of the 360 feedback process on individual and team relationships. Copies of the 360 evaluation surveys used in this evaluation study are available from the author upon request. 6
360 feedback assessment. The 360 feedback assessment used in this study was Manager View 360 (Nowack, 1997). This 360 instrument measures 20 distinct managerial competencies clustered into four major areas (Task/Leadership, Interpersonal, Communication and Problem Solving) and was derived by job analyses of supervisory and managerial positions in diverse industries. It has shown adequate psychometric properties (internal consistency reliabilities of the scales range from.71 to.90 and average test re-test reliability over a 3-month period across all 20 scales is.65) in prior research along with a factor structure to support the competency model on which it is based (Nowack, 1992; 1997a; 1997b). Manager View 360 provides a comprehensive summary feedback report to the manager comparing norm based graphical self to other perceptions, summary tables for each question broken down by rater category, a most- and least-frequently observed behavior section, an extensive set of competency based developmental suggestions and two open-ended questions (strengths and developmental areas). One unique feature of this 360 feedback instrument is the competency based developmental suggestion section providing current readings, external workshops, website resources and specific activities for each of the 20 competencies that are measured. Results Table 1 summarizes the key findings from the 360 Rater Survey. In general, raters tended to strongly support the 360 feedback process and believed that the process had integrity and confidentiality. Raters reported spending quality time to complete both the qualitative and quantitative sections of the 360 feedback instrument 7
and report perceptions that the instrument will increase trust, cooperation and communication between themselves and the recipient. The raters completing this survey also reported observing improvement in one or more specific management skill behaviors as a result of the recipients participation in the 360 degree feedback program (73.7%). The program participant s feedback is summarized in Tables 2, 3 and 4. In general, the participants reported that the 360 feedback results provided helpful insights (87.2% strongly agreed or agreed), were mostly expected (72.4% reported receiving results that they either somewhat expected or expected), had increased awareness of strengths and development areas (89.4%) and as a result were likely to make some changes in their management behavior (76.6%). Participants reported mixed perceptions about whether the qualitative or qualitative sections were most valuable (48.9% reported that the written section was the most useful). The emphasis of the developmental action plan based on the 360 feedback was largely on development needs (82.1%) versus strengths to build on (17.9%). Table 4 summarizes the specific developmental activities that employees targeted. The most frequently reported activities included seeking additional feedback on the job (63.8%), asking for an internal or external coach (51.1%), practicing a specific skill or technique (42.6%), reading books, journals or other references (36.2%) and participating in internal/external training programs (36.2%). 8
Discussion This study examined perceptions and consequences of 360 degree feedback in a corporate leadership development program with their feedback (69.6% reported that the results were expected to a high extent) and 76.6% reported they were likely to make a change in their management behavior on the job. This finding is important based on research by Atwater et al., (2000), Berhardin et al., (1993), Mauer et al., (2002) suggesting that receiving lower scores on employee development than expected resulted in leaders feeling less motivated than those who did not receive lower than expected scores. Managers with deflated motivations as a result of a 360 degree feedback process are typically less committed to the organization and future job performance (Meyer et al., 1989). Participants had somewhat mixed perceptions about the helpfulness of the different sections included in the 360 feedback report (written comments, graphs, numeric summary tables) that were most helpful. Participants reported that the written comments were the most useful part of the 360 summary feedback report (12.8% disagreed, 38.3% were undecided and 48.9% felt it was the most useful). The only other study to directly explore this question was reported by Atwater et al., (2004). Their findings suggested that individuals prefer numeric scores and normative feedback and will be less angry and discouraged and more motivated and inspired if they receive numbers and comparative information regarding their leadership behaviors. It appears that when participants can be provided feedback that includes both numeric and qualitative data this format is most likely to enhance self awareness and 9
motivation to change behavior. Interestingly, participants reported being very focused on improving skills that appeared to be weaknesses, rather than, leveraging his/her strengths. Because so many coaches and organizations typically emphasize one over the other, it is important to understand what recipients are actually focusing on. Additional research is needed to examine how different 360 feedback approaches (e.g., not having a follow up individual meeting to discuss the 360 results with one s boss or an internal/external consultant following the program) might influence participant and rater reactions and consequences (e.g., Waldman & Atwater, 1998). Until these empirical questions are answered, it is proposed that organizations undertaking a 360 degree process invest efforts in conducting more thorough evaluation studies in this area. Several limitations should be emphasized that can limit generalizability of this study. This evaluation study is limited by a relatively small sample size, use of one type of 360 feedback intervention process within a corporate workshop format and within one industry (communication). No data was available about the actual raters completing the rater surveys to help with the interpretation of these findings. Finally, no objective measures of performance, retention or success were available at the one-year follow up point. This evaluation study did employ a longitudinal perspective and utilized ratings from both the program participants and sampling of their managers, direct reports and peers. Despite these limitations, the results presented would still appear to be valuable for other organizations utilizing 360 feedback interventions. Future studies should attempt to replicate this type of longitudinal evaluation design with alternate 360 degree 10
feedback processes (e.g., with or without individual feedback meetings with an external psychologist), tools (assessments that only provide qualitative results) and when accountability for implementing a developmental action plan has been built into the program. This one-year survey follow up of program participants seems to provide some limited insights about ways to enhance and facilitate successful behavior change using 360 degree feedback interventions. Providing external feedback with an internal or external coach would appear to be helpful (65.8% reported the coach facilitated the interpretation of the feedback and translation into a development plan) as a service to include in any workshop based feedback intervention. Involving the participant s manager in the 360 feedback process would also appear to be important for organizations to build into the program. Approximately 70% of program participants in this study met with their manger within three months to discuss the results and finalize a developmental action plan. One consequence of this interaction is the possibility of enhancing communication and understanding between the participant and his/her own manager. Approximately 68% of all raters in this survey reported that the 360 feedback process led to increased trust, cooperation and communication between themselves and the recipients. This is important in light of previous meta-analytic studies showing have shown weak agreement between self-ratings and ratings made by others (e.g., Harris & Schaubroeck, 1998; Mabe & West, 1982) and the importance of self-other ratings to performance, compensation and organizational level (Ostroff, Atwater, and Feinberg, 2004). Finally, having a wide variety of developmental options and 11
opportunities to individuals utilizing 360 feedback processes would seem to facilitate maximum behavior change and improvement (Kanfer et al., 1989). In this study, seeking additional feedback, exploring the use of an internal/external coach and actually practicing specific management behaviors or techniques were the most widely reported developmental activities being engaged in one year following the 360 feedback process. The results of a recent meta-analysis of 24 longitudinal 360 feedback studies by Smither et al. (2005) suggested that improvement following feedback was significant but generally small. They suggest that it is unrealistic for practitioners to expect large performance improvement based on 360 feedback and that some recipients will be more likely to improve than others. Their research suggests that specific conditions ( best practices ) along with specific individual characteristics (e.g., motivation to want to change) will optimally result in transfer of awareness into lasting behavior change. A recent study by Bono and Colbert (2005) suggests that motivation to change behavior following 360 feedback is related to personality (core self-evaluations). Specifically, they found that individuals with high levels of core self-evaluations (those with high selfesteem, generalized self-efficacy, internal locus of control and low neuroticism) will be most motivated to change behavior when they receive discrepant feedback and those with low levels of core self-evaluations will be most motivated when others ratings are most similar to their own. These results suggest the potential value of coaching to assist individuals to understand their potentially complex feedback and to increase motivation to set developmental goals. 12
Based on this one year 360 feedback evaluation study, it is possible to make some suggestions for best practice to ensure that awareness from the multi-rater feedback process is converted into behavior change: 1. Hold participant s managers accountable for meeting with their direct reports to discuss and finalize a professional development action plan. 2. Seek senior management support to repeat a 360 degree feedback process in 12 to 14 months following the first to create a mechanism to track and monitor progress targeted on the professional development plan. 3. Attempt to focus the professional development plan on measurable behaviors and activities that enhance learning (e.g., special assignments, new tasks, etc.) 4. Utilize either an internal or external consultant to facilitate the interpretation of the 360 feedback report and minimize any negative reactions that might occur as a result of the feedback. 5. Utilize a 360 feedback process that allows for both quantitative results (e.g., numeric data in the forms of graphs, tables) and qualitative feedback (e.g., inclusion of open-ended questions). 6. Ensure that an adequate number and type of raters are invited to provide feedback to the program participant and that this final rater group is discussed with their manager and/or consultant to the program. 13
7. Some attempt is made by the organization to evaluate the reaction and impact of the 360 feedback intervention from multiple perspectives (e.g., program participants, managers and other raters). Despite a relatively small sample size and evaluation of a specific corporate leadership development program, this evaluation study was designed to measure perceptions, reactions and observations of both participants and raters over the course of one-year using a validated 360 feedback instrument. The results of this evaluation study support prior research and provide some implications for both best practices and future research. 14
Table 1 Rater Survey Results (% responding) RATER SURVEY ITEM Strongly Agree Agree Neither Agree Nor Disagree Disagree Strongly Disagree The 360 assessment and feedback process will increase trust, cooperation and communication between myself and my direct reports Overall, I spent quality time completing the written comments section of the 360 degree feedback instrument to ensure that my responses were thoughtful, complete, candid and accurate Overall, I spent quality time completing the behavioral rating section of the instrument to ensure that my responses were thoughtful, complete, candid and accurate I believe that the 360 reports were kept confidential I believe our organizational culture supports the 360 feedback assessment and feedback process 24.1 44.8 27.6 3.4 0.0 36.7 56.7 0.0 0.0 0.0 43.3 56.7 0.0 0.0 0.0 53.3 36.7 6.7 3.3 0.0 10.0 46.7 30.0 13.3 0.0 Yes No As a result of the 360 assessment and feedback process, have you been able to observe some improvement in one or more specific behaviors or management skills? As a result of the 360 feedback process, were specific developmental activities included as part of the annual performance evaluation plan for your direct reports? 73.7 26.3 60.0 40.0 15
Table 2 Participant Survey Results (% responding) RATER SURVEY ITEM Strongly Agree Agree Neither Agree Nor Disagree Disagree Strongly Disagree The 360 feedback process provided me with useful insights and feedback I can use in my current job. Overall, I have confidence in the honesty of the 360 feedback results. The 360 feedback ratings reflect an accurate assessment of important behaviors which impact my performance. I believe that the 360 reports were kept confidential The written comments in the 360 feedback were more helpful than the numeric data. Overall, the results of the 360 feedback process increased my awareness of developmental areas. As a result of the feedback I received on the 360 feedback instrument, I am likely to change my management behavior. The 360 program I participated in will increase the trust, cooperation and communication between myself and my boss. I believe that my boss supports the 360 workshop and feedback process. I believe our organizational culture supports the 360 workshop and feedback process. The one-to-one feedback meeting that I had with the outside consultant was helpful to clarify the 360 results and implement an action plan. 34.0 53.2 8.6 2.1 2.1 19.1 57.4 10.6 12.9 0.0 17 48.9 27.7 6.4 0.0 31.9 38.3 19.1 8.5 2.1 12.8 36.1 38.3 12.8 0.0 23.4 66.0 4.3 6.4 0.0 8.5 68.1 17.0 6.4 0.0 6.7 31.1 46.7 1.1 4.4 27.7 38.3 23.4 8.5 2.1 12.8 44.7 31.9 8.5 2.1 23.7 42.1 21.1 10.5 2.6 16
Yes No At the end of the program did you target specific managerial skills for improvement? 76.6 21.4 17
Table 3 Participant Survey Results (% responding) RATER SURVEY ITEM Very Unexpected Somewhat Unexpected Somewhat Expected Expected In general, the positive results I received from the others on the 360 feedback instrument were: In general, the negative results I received from the others on the 360 feedback instrument were: In general, the extent to which there was direct agreement among the ratings of my boss, direct reports and peers on the 360 feedback tools was: 2.1 25.5 51.1 21.3 4.3 36.2 46.8 12.8 Very High High Low Very Low 12.8 63.8 21.3 2.1 Compared to my boss, my ratings on the 360 feedback instrument were: Compared to my direct reports, my ratings on the 360 feedback instrument were: Compared to my peers, my ratings on the 360 feedback instrument were: Strengths to build on Higher Lower Same 19.2 40.4 40.4 19.6 30.4 50.0 21.3 21.3 57.4 Development Needs Current Focus Future Focus In general, the emphasis of my 360 feedback developmental action plan was: 17.9 82.1 0.0 0.0 Less than 1 Week 1 to 2 Weeks 3 or More Weeks Never Met The average length of time following the 360 workshop or feedback that I had a meeting with my boss to discuss my results and action plan was: The average length of time following the 360 workshop or feedback that I had a meeting with my direct reports to discuss my results and action plan was: 23.4 34.0 10.6 31.9 10.6 44.7 12.8 31.9 18
Table 4 Participant Survey Results (% responding) What following developmental activities have you already started or plan to immediately begin as a direct result of your participation in the 360 feedback program? Developmental Activity Sought additional feedback from others. 63.8 Asked for coaching form my boss. 51.1 Spent time practicing a new skill. 42.6 Read journals, books or magazines related to specific skills I want to develop. Participated in a committee or task force in a nonleadership role 36.2 36.2 Held a leadership role in a committee or task force. 29.8 Listened to an audiotape or book on tape. 19.1 Participated in community or non-work activity. 6.4 Enrolled in a formal university degree or certificate program 6.4 Took a university extension course or workshop 4.3 Maintained a journal or record of ideas, thoughts or plans 2.1 19
References Antonioni, D. (1996). Designing an effective 360-degree appraisal feedback process. Organizational Dynamics, 25, 24-38. Atwater, L.A., Roush, P., & Fischthal, A. (1995). The influence of upward feedback on self-and follower ratings of leadership. Personnel Psychology, 48, 35-60. Atwater, L.A., Waldman, D., Atwater, D., & Cartier (2000). An upward feedback field experiment. Supervisors cynicism, follow-up and commitment to subordinates. Personnel Psychology, 53, 275-297. Atwater, L.A., Waldman, D., Brett, J. F. (2002). Understanding and optimizing multi-source feedback. Human Resource Management Journal. 41, 193-208. Berhardin, H.J, Dahmus, S.A., and Redmon, G. (1993). Attitudes of first-line supervisors toward subordinate appraisals. Human Resource Management, 32, 315-324. Bono, J. and Colbert, A. (2005). Understanding responses to multi-source feedback: The role of core self-evaluations. Personnel Psychology, 58, 171-203. Brett, J., & Atwater, L. (2001). 360-degree feedback: Accuracy, reactions and perceptions of usefulness. Journal of Applied Psychology, 86, 930-942. DeNisi, A., & Kluger, A. (2000). Feedback effectiveness: Can 360-degree appraisals be improved? Academy of Management Executive, 14, 1, 129-139. Greller, M. M. & Herold, D. M. (1975). Sources of feedback: a preliminary investigation. Organizational Behavior and Human Performance, 13, 244-256. Harris, M., & Schaubroeck, J. (1988). A meta-analysis of self-supervisor, selfpeer and peer supervisor ratings. Personnel Psychology, 41, 43-62. Ilgen, D. & Davis, C. (2000). Bearing bad news: Reactions to negative performance feedback. Applied Psychology: An international Review, 49, 550-565. Kanfer, R. & Ackerman, P.L. (1989). Motivation and cognitive abilities: An integration / aptitude- treatment interaction approach to skill acquisition. Journal of Applied Psychology, 74, 657-690. Kluger, A.N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback theory. Psychological Bulletin, 119, 254-284. 20
Kluger, A. N. & De Nisi, A. (1998). Feedback interventions: Toward the understanding of a double-edged sword. Current Directions in Psychological Science. 7, 67-72. Mabe, P., West, S. (1982). Validity of self-evaluation of ability: A review and meta-analysis. Journal of Applied Psychology, 67, 280-296. Maurer, T. J., Barbeite, F.G., & Mitchell, D.R. (2002). Predictors of attitudes toward a 360-degree feedback system and involvment in post-feedback management development activity. Journal of Occupational and Organizational Psychology, 75, 87-108. Meyer, J.P., Paunonen, S.V., Gellaty, I.R., Goffin, R.D., and Jackson, D. (1989). Organizational commitment and job performance: It s the nature of the commitment that counts. Journal of Applied Psychology, 74, 152-156. Nowack, K. (1999). 360-Degree feedback. In DG Langdon, KS Whiteside, & MM McKenna (Eds.), Intervention: 50 Performance Technology Tools, San Francisco, Jossey-Bass, Inc., pp.34-46. Nowack, K. (1997a). Manager View/360. In Fleenor, J. & Leslie, J. (Eds.). Feedback to managers: A review and comparison of sixteen multi-rater feedback instruments (3 rd edition). Center for Creative Leadership, Greensboro, NC. Nowack, K.M. (1997b). Congruence between self-other ratings as a predictor of assessment center performance. Journal of Social Behavior and Personality, 12 (5), 145-166. Nowack, K. (1992). Self-assessment and rater-assessment as a dimension of management development. Human Resources Development Quarterly, 3, 141-155. Ostroff, C., Atwater, L., & Feinberg, B. (2004). Understanding self-other agreement: A look at rater and ratee characteristics, context and outcomes. Personnel Psychology, 57, 333-375. Reilly, R. R., Smither, J. W., & Vasilopoulos, N.L. (1996). A longitudinal study of upward feedback. Personnel Psychology, 49, 599-612. Smither, J., London, M. & Reilly, R. (2005). Does performance improve following multisource feedback? A theoretical model, meta-analysis, and review of empirical findings. Personnel Psychology, 58, 33-66. Waldman, D. A., & Atwater, L.E. (1998). The power of 360-degree feedback: How to leverage performance evaluations for top productivity. Houston, TX: Gulf. 21