Business Writing 101: The Evaluation Results of a Corporate Application

Transcription

1 Business Writing 101: The Evaluation Results of a Corporate Application Scott R. Frasard, Thomas L. Graves, and Tanya L. Harris Business writing is a much-studied area of corporate training and has the opportunity to enhance the organization s capabilities in a variety of ways. Previous research demonstrated post-training improvements in time spent on s, better understanding by readers, and decreased costs associated with s. This study evaluated a 1-hour business writing training workshop s impact on the organization using multiple perspectives, a sample writing analysis, and follow-up measurements to identify improvements. The purpose was to understand how the training changed the participants writing practices. Following the training, participants and their supervisors were pleased with the design and improved effectiveness resulted. Introduction Background Over the last ten years, many studies explored communications training effectiveness. Most of these studies focused on performance improvements with key themes related to: (a) decreased time spent on s (Burgess, Jackson, & Edwards, 2004; Huang, Lin, & Lin, 2011; Moore & Moore, 2004; Soucek & Mouser, 2010), (b) better use of subject lines (Burgess, Jackson, & Edwards, 2004; Jackson & Culjak, 2006), (c) decreased message misunderstandings (Aguilar-Roca, William, Warrior, & O Doud, 2009; Huang, Lin, & Lin, 2001; Jackson & Culjack, 2006; Moore & Moore, 2004), and (d) decreased costs associated with use (Burgess, Jackson, & Edwards, 2004; Jackson & Culjack, 2006). The training themes in these studies included: (a) use of subject lines, (b) time management when dealing with s, (c) formatting s, and (d) writing for clarity. While some researchers found training participants did not take the class and subsequent skill assessment seriously (Pittenger, Miller, & Allison, 2006), others found that participants experienced increased perceptions of ing abilities (Huang, Lin, & Lin, 2011), improved grammar, spelling and punctuation (Hof & Ashworth, 2010), and faster replies to s (Huang, Lin, & Lin, 2011; Moore & Moore, 2004; Soucek & Moser, 2010). Combined, these effects produced better-written s that were both clearer and more succinct, which reduced the overall financial impact on organizations who delivered such training. In this study, instructional designers created the Business Writing 101 Workshop to address a significant organizational business need the need to write clear and concise s to customers. Feedback from customers prompted senior leaders to seek out a learning solution in order to improve customer relations. The training development, delivery, and evaluation used these identified themes from the literature with the intent to achieve a positive return on investment for the organization. The training course was a one-hour, instructor-led course attended by participants at a time. Each participant received a learning journal to take notes as he or she followed the facilitator s PowerPoint presentation. The presentation focused on the importance of a well-written . Facilitators used dialogue and real-life examples to explore topics such as structure, sentence structure, using words for maximum effect, positive versus negative tone, active versus passive verbs, avoiding jargon use, and proper spelling, punctuation, and capitalization. Facilitators showed participants how a G.R.E.A.T.

2 (Goal, Relevant, Emotional tone, Action, and Time frame) format will improve an s effectiveness. At the conclusion of the workshop, each participant received a fictitious customer with instructions to respond to the customer via his or her supervisor for scoring. In total, 934 participants from four sites across the organization participated in the training. The participants included customer service representatives with their supervisors participating in an extended version of the training in order to learn about scoring s. Research Problem and Research Questions According to some customers, communications were sometimes inaccurate and ambiguous. Thus, the Business Writing 101 Workshop set out to produce higher quality s. Once conducted, the organization needed to know what, if any, quality improvements came from the training. The major purpose for this research was to identify how training changed writing practices by examining learner satisfaction, learner achievement, and training application. Phillip s (1997) five-level evaluation model served as the measurement framework to answer the following research questions: 1. How did participants react to the training? 2. What did participants learn from the training? 3. How did the training affect the organization? Methodology To determine the training s affect, the researchers employed several strategies to evaluate the program s effectiveness. For the first evaluation level (learner satisfaction), the researchers used the corporation s standard training satisfaction survey. This survey included five items, which measured learners perception of the course's design and ten items, which measured learners perception of how well the facilitator taught the course. All items measured perceptions on a four-point Likert scale from 4 = strongly agree to 1 = strongly disagree. For the second evaluation level (learner achievement), participants were tasked with responding to a fictitious communicating a customer concern. Each participant sent the response to his or her supervisor who scored each according to a scoring rubric. The rubric included ten key criteria for an effective with each criterion scored on a 4-point scale (0 = not included, 1 = included, but did not meet expectations, 2 = included and met expectations, and 3 = included and exceeded expectations). The ten areas scored were: (a) includes a subject line, (b) includes relevant information in the body, (c) uses an appropriate emotional tone, (d) includes specific action items, (e) indicates a specific time frame, (f) uses correct grammar, (g) uses correct punctuation, (h) spells words correctly, (i) uses complete sentences, and (j) statements are clear. The maximum possible score was 30 with 20 set as the minimum acceptable score. The supervisors sent the completed scoring rubrics for analysis to the measurement specialist evaluating the program. In one location, the researchers blinded supervisor-scored s and the workshop facilitator scored these s a second time to establish interrater reliability. Additionally, the researchers randomly selected archived s from before the training, blinded the s, and the same facilitator scored these to establish pre- and post-training measures. For the third evaluation level (transfer of training), the researchers used two strategies. The first strategy involved surveying participants about changes in their work practices related to

3 the training. This survey had nine items, which measured participants perceptions of how the training helped him or her on the job. Five items measured how frequently participants do specific actions with his or her s. Four items measured how frequently participants perform specific activities. Four items measured whether or not participants believe the training provided help. Finally, one item measured participants confidence level that the training resulted in workplace practice changes. The second strategy involved interviewing participants supervisors. The researchers conducted the interviews by phone and each lasted ten to fifteen minutes. The researchers asked eight questions of each supervisor regarding his or her participation in scoring the samples, perceptions of training affects, and confidence levels that the training resulted in workplace practice changes. Data analysis included descriptive statistics to describe learner performance at each of the four organization sites, specific results from participant self-reports of changes in workplace behaviors, confidence levels that training made positive gains, and themes in the data from the supervisor interviews. The researchers compared site-specific results along with evidence from pre- and post-training scores. Findings The findings from this evaluation centered on the first three levels of Phillip s (1997) five-level evaluation model. Specifically, the evaluation focused on learner satisfaction with the training in both the training design and facilitator delivery, learner achievement in writing abilities, training concept use in the workplace, and writing practice changes from supervisors and participants. The researchers conducted surveys and interviews to determine what, if anything changed in participants practices via semi-structured exploratory interviews with the supervisors and surveys with both supervisors and participants. Learner Satisfaction Participants rated two aspects of the training via the learner satisfaction evaluation: training design and facilitator performance. With a 57% (n=532) return rate, the results indicated that the participants were mostly satisfied with the training s design, with satisfaction scores ranging from 3.1 to 3.3 on the 1 4 Likert scale. The facilitator scores were higher, ranging from 3.5 to 3.9 on the same scale. Participant comments reflected the training s value as a good knowledge refresher, enjoyable to attend, and the facilitator presented the content very well. Conversely, participants noted that they needed to have a better understanding of why they had to participate in the training, some of the materials used needed further editing, the workshop s length was too short for the topic, and the topic was not one they needed as they already possessed the skills taught. Learner Achievement Following the workshop, each participant responded to a fictitious customer s and submit this to his or her supervisor. Only 31.9% (n=298) of all participants completed the posttraining writing assignment. Overall, participant had a mean score of 71.3% with a mode of 66.7% and a standard deviation of 4.5. Interestingly, the supervisors at the first training location had scoring rubrics with a scoring summary included. This summary indicated levels of proficiency based on total score, to include the minimum acceptable score. This produced an unintended bias in scoring in which supervisors assigned the exact minimum score to

4 approximately 24% (n=54) participants. Once identified, the researchers removed the score summary from the rubric and this pattern did not appear in subsequent workshop deliveries. Among the ten variables measured, supervisors scores indicated that participants performed the best with spelling and complete sentence use. When the facilitator scored the same s, inclusion of a subject line and using complete sentences were the top performance areas. On the other end of the spectrum, both the supervisors and the facilitator s scores indicated that participants were weakest at including specific action items and a specific time frame. Of the pretraining s scored by the facilitator, complete sentence use improved the most following training followed by use of appropriate emotional tone, indicating a specific time frame, correct grammar use, and using clear statements. Means (with standard deviations in parentheses) for facilitator-scored pre-training s 55.3% (3.98), 95% CI [51.3%, 59.3%]. Post-training facilitator-scored s means were 63.9% (4.15), 95% CI [61.3%, 66.1%] while supervisorsscored s means were 71.8% (3.97), 95% CI [69.5%, 74.1%]. Though there was a significant increase in facilitator-scored s post-training, there was insufficient evidence to indicate supervisors and the facilitator scored s similarly (r = , p<0.01), making before and after training effect inferences across the organization unreliable. Application of Learning During interviews, 71% of supervisors indicated that their participation in the training via scoring s proved beneficial based on increased interaction regarding communications with their direct reports. The supervisors also communicated that the training changed how people write s. Most prominently, supervisors noted that including Action Needed or Informational in the subject line improved customer responses at least 38% of the time and allowed for a convenient method for prioritization. The supervisors also indicated a noticeable improvement in spelling, grammar, punctuation, sentence structure, conciseness, and professionalism in reviewed s following the training. Most significantly, supervisors noticed a dramatically increased awareness of importance by seeing his or her direct report by asking others to proof read an before sending. This practice increased relevant information inclusion and message clarity while decreasing the need for multiple follow-up s. Training participants (29%) indicated that the workshop did have an impact on their writing abilities and daily practices. From the surveys, participants indicated (in descending order of prevalence): (a) s were structured better (32%), (b) s answered customers questions more accurately (31%), (c) s better communicated what the customer needed (28%), and (d) s were proof read prior to sending (27%). Participants indicated that they frequently included Action Needed and Informational in their subject lines 48% of the time and referred to a grammar/punctuation job aid provided during the training 29% of the time. Further, participants were 60% confident the training changed how they wrote s and this change contributed an added value to the customer (76%), was worth his/her time to participate (75%), was worth the organization s time to conduct (74%), and helped him or her to do their job better (71%).

5 Conclusions Summary of Findings By evaluating this training, five main conclusions were made: (a) the training s design was acceptable, but the materials used in the delivery needed some minor modifications for optimal utility, (b) participant satisfaction scores indicated the facilitator was mostly effective in delivering the training, (c) the training was well-received by participants and supervisors, (d) participants did learn new concepts, and (e) distinctive on-the-job changes took place, which could be attributed to the training. Limitations of the Research Some limitations of this study should be noted. First, the Business Writing 101 Workshop was one hour in length and was a lecture-based format. While the training design encouraged interaction, participants did not do any actual writing practice. Second, workshop participants were required to attend. This mandate may have influenced the motivation to take the follow-up writing assignment seriously or even submit the writing sample. Third, pre-training archives were not available for all participants to evaluate, which may have influenced the pre-training and post-training writing proficiency evaluation. Fourth, an unacceptable interrater reliability among scorers made some training affects impossible to determine. Fifth, the supervisors who scored direct reports s did not have his or her writing abilities evaluated before training took place. Thus, the supervisor s own writing skills may have influenced his or her ability to score direct reports s. Finally, the facilitator scored only a small sample of s a second time and compared to supervisors scores. This analysis involved only one audience segment, making generalizing results to all organization segments potentially flawed. Contributions, Implications, and Future Research This evaluation contributes to the business writing literature base another perspective on evaluating training in the corporate setting. It demonstrated positive gains while illustrating areas for improvement. Its practical contributions can be used by human resource professionals in a number of ways: (a) important areas to consider in new employee and current employee training, (b) resource considerations needed to conduct similar training (i.e., time, reviewers, subject matter experts), (c) new hire candidate screening considerations related to candidate reading and writing abilities, and (d) a potentially useful evaluation strategy to measure business writing training. Following this evaluation, the researchers will repeat the scoring at six and twelve months post-training to determine the training s affect on sustained writing quality. Additionally, future evaluations will measure training affects on various business performance metrics such as customer loyalty, time to complete customer transactions, and information accuracy. Further, the researchers will measure the training s return on investment to determine the training s dollar value. Others conducting similar training may benefit by researching a number of different aspects of business writing. These may include investigating why certain measurement criteria were problematic for the participants while other criteria were not, determining which business writing skills are influenced by training versus skills that are improved in other ways, and which organizational factors are influential in these areas.

6 References Aguilar-Roca, N., Williams, A., Warrior, R., & O Dowd, D. K. (2009). Two minute training in class significantly increases the use of professional formatting in student to faculty correspondence. International Journal for the Scholarship of Teaching and Learning, 3(1), Retrieved from /v3n1/articles/pdfs/article_odowd_et_al.pdf. Burgess, A., Jackson, T., & Edwards, J. (2004). training significantly reduces defects. International Journal of Information Management, 25(1), doi: /j.ijinfomgt Hof, J. & Ashworth, S. D. (2010). An indirect measure of writing proficiency: More than facevalidity. Proceedings of the 25 th Annual Conference of the Society for Industrial/Organizational Psychology. Retrieved from Testing-Assessment/Hof%20%26%20Ashworth% pdf. Huang, E. Y., Lin, S. W., & Lin, S. (2011). A quasi-experiment approach to study the effects of management training. Computers in Human Behavior, 27(1), doi: /j.chb Jackson, T. W., & Culjak, G. (2006). Can Seminar and Computer-Based Training Improve the Effectiveness of Electronic Mail Communication within the Workplace? Proceedings of the 17 th Annual Austrailasian Conference on Information Systems. Retrieved from Moore, R., & Moore, M. (2004). Customer inquiries and complaints: The impact of firm response time to communications. Marketing Management Journal, 14(2), Retrieved from /pdfviewer?vid=4&hid=123&sid=cef91f3a-8fe4-43d2-8b76-abc5bd643348%40 sessionmgr112. Phillips, J. J. (1997). Handbook of training evaluation and measurement methods (3 rd ed.). Woburn, MA: Butterworth-Heinemann. Pittenger, K. K. S., Miller, M. C., & Allison, J. (2006). Can we succeed in teaching business students to write effectively? Business Communications Quarterly, 69(3), doi: / Soucek, R., & Moser, K. (2010). Coping with information overload in communication: Evaluation of a training intervention. Computers in Human Behavior, 26(6), doi: /j.chb Scott R. Frasard, Ph.D., Sr. Specialist Measurement and Evaluation, 4200 Osuna Rd., Albuquerque, NM 87119, scott.frasard@cardinalhealth.com; Thomas L. Graves, M.A., Sr. Specialist Instructional Design, 7200 Cardinal Place West, Dublin, OH 43017, thomas.graves@cardinalhealth.com; Tanya L. Harris, MLIS, tharris_sc@hotmail.com. The researchers would like to acknowledge the support of Cardinal Health, Cardinal Health Service University, subject matter experts, instructional designers, supervisors, and program participants for this evaluation project and subsequent conference presentation. Presented at the Midwest Research-to-Practice Conference in Adult, Continuing, and Community Education, Lindenwood University, St. Louis, MO, September 21-23, 2011.

7