Evaluation of the Piccard Integrated Service Delivery Pilot

Evaluation of the Piccard Integrated Service Delivery Pilot Prepared by Wade Bannister, PHD Arizona State University For the Center for the Study of Social Policy Customer Satisfaction Project

Table of Contents Introduction... 1 Process Overview... 1 2 Methodology and Data Sources... 2 Findings... 3 Customer Service... 3 7 Efficiency of New Service Model... 8 10 Discussion... 10 11 Conclusion... 11

Introduction The overarching objective of the Piccard pilot was to create a customer focused integrated system to improve the customer s application process and access to services in the Montgomery County Department of Health and Human Services. Preliminary observations indicated that improvements to the system were needed to streamline and standardize intake, screening, application, eligibility determination, and case tracking for health and human services clients in Montgomery County, Maryland. The pilot project based in the Piccard Drive site sought to demonstrate improvements to the efficacy of health and human service delivery through improved customer service. This report seeks to quantify the effect of the customercentric methodology employed in the Piccard pilot by evaluating the customer satisfaction, business processes, and customer outcomes. Pilot Services began in January 2007 and operate Tuesday through Friday mornings at the Piccard site at 1301 Piccard Drive in Montgomery County. The major changes to the existing process consisted of the addition of a customer service team responsible for assisting the customer when they arrived and making reminder calls for future appointments as well as the addition of triage workers to meet with new customers to help ascertain the customer s needs and make referrals. In addition, customer service training was provided to all staff members to encourage customer centric attitudes, an on site social worker was added to address and respond to customer clinical and case management needs, and building renovations were conducted to accommodate the process changes. Process Overview Customers arriving at the Piccard site during pilot operations were met with a new process, as outlined below: Step 1: Customer service team Greet customer Identify reason for visit, and direct toward correct service path Provide waiting room support and assistance with questions and applications Support the reception staff and triage workers Step 2: Reception Sign customer into database Give customer application and List of Service Needs for self identification of needs Provide information about today s service Step 3: Triage Search CARES and CRS/AIF databases to determine if customer is an existing customer; enter customer information Face to face dialogue with customer to screen for needs using List of Service Needs Review application Explain eligibility process and screen for preliminary eligibility 1

Schedule appointments for appropriate programs Work with Direct Service Providers to coordinate service Provide internal/external referrals, service information, directions Forward applications as appropriate Step 4: Direct service delivery (Social Workers Income Support Workers) Interview for services Assess customer s needs Work in conjunction with triage staff to provide additional referrals Refer to Case Manager as indicated If eligible, begin processing program services or benefits Step 5: Follow up and case management Reminder calls one day prior to appointment Ongoing intensive case management for high end users, as needed Methodology and Data Sources A customer service survey was developed and provided to customers of the Health and Human Services Department s Piccard Drive site in order to assess their satisfaction with their visit. This survey included questions regarding their overall level of satisfaction, their satisfaction with particular aspects of the process, as well as questions relating to their perceived efficacy of the service they received. Surveys were administered to visitors at the pilot site, with different survey forms being provided to new customers and customers with appointments. For comparison purposes, surveys were also administered to customers visiting during non pilot hours and customers visiting the Silver Spring site which was not part of the pilot project. It was felt that these groups could be considered to have experienced similar non pilot services and were thus combined into a single comparison group. The number of respondents in each group is given in Table 1. Not all respondents provided answers to all questions, and item nonresponse can be noted in the tables for individual questions. Table 1. Survey response rates Customer Type Respondents Percent New Customer Non Pilot 68 12.19% Returning Customer Non Pilot 89 15.95% New Customer Pilot 203 36.38% Returning Customer Pilot 198 35.48% Total 558 100% Business process data were also made available, including arrival times as well as the time when triage and/or services began. These data were manipulated to facilitate calculation of waiting times. A needs screening worksheet was completed for each customer during the triage process which detailed both the services requested by the customer at arrival as well as the services for which they received referrals. Attendance at future appointments was also 2

tracked, along with notations of whether reminder calls were made to the customer for the appointment. The above data were collected, summarized and analyzed in order to provide an accurate and methodologically sound representation of the performance of the Piccard pilot. This evaluation focuses on measuring both the customer satisfaction and perception as well the delivery of services to the customer. Findings Customer Service One of the primary objectives of the Piccard pilot was to improve the overall level of customer service provided to each client. Therefore, this report first seeks to establish whether there is evidence of an improved sense of satisfaction among customers who visited during pilot operations as compared to those who did not. Table 2. Q4: How did you feel about your visit today? (1 = Very Satisfied, 5 = Dissatisfied) Customer Type n Mean Score Standard 95% Confidence Deviation Interval New Customer Non Pilot 61 2.13 1.24 (1.82, 2.44) New Customer Pilot 188 1.63 0.94 (1.49, 1.76) Returning Customer Non Pilot 81 1.89 1.14 (1.64, 2.14) Returning Customer Pilot 166 2.12 1.31 (1.92, 2.32) Table 2 provides evidence that new customers experiencing the pilot site are more satisfied with their visit than new customers visiting during non pilot operations. The mean satisfaction score for new customers experiencing non pilot operations is 2.13 while the mean score for new customers experiencing the pilot operations is 1.63. This score difference is statistically significant as indicated by the non overlapping 95 percent confidence intervals. However, Table 2 also suggests that there is no evidence suggesting that there is a significant difference in satisfaction between returning customers who visit the site during pilot operations and those who visit during non pilot operations. This is possibly because returning customers with appointments do not experience many of the pilot services such as triage and would therefore be less impacted by these changes. Figure 1 shows the relative frequency of the responses for the new customers, and from this it is evident that a substantially higher percentage of the pilot customers reported being very satisfied as compared to non pilot customers. Similarly, the percentage of pilot customers reporting being only somewhat satisfied, somewhat unsatisfied, or unsatisfied was substantially lower than non pilot customers. 3

Figure 1. Frequency of responses by new customers to Q4: How did you feel about your visit today? Additional measures of customer satisfaction were obtained from Question 6 ( was our waiting room clean and comfortable? ), Question 7 ( were you able to understand the signs and directions? ), and Question 10 ( how you felt about the receptionist who checked you in ). The results of these questions are displayed in Tables 3, 4, and 5, respectively. Table 3 indicates that new pilot customers report being slightly more satisfied with the waiting room than new non pilot customers, but this difference does not appear to be statistically significant. There is also very little difference in satisfaction among returning customers, with returning pilot customers reporting an average score of 1.55 while returning non pilot customers report an average score of 1.58. Table 3. Q6: Was our waiting room clean and comfortable (1 = Very Satisfied, 5 = Dissatisfied) Customer Type n Mean Score Standard Deviation 95% Confidence Interval New Customer Non Pilot 65 1.62 0.78 (1.42, 1.81) New Customer Pilot 192 1.41 0.59 (1.33, 1.49) Returning Customer Non Pilot 85 1.58 0.71 (1.42, 1.73) Returning Customer Pilot 185 1.55 0.67 (1.45, 1.64) Similarly, Table 4 shows that there were no statistically significant differences between pilot and non pilot customers with respect to satisfaction with their ability to understand the signs and directions. While new pilot customers report a mean satisfaction score of 1.53 as compared to a mean score of 1.72 for the new non pilot customers, this difference is not statistically significant. Interestingly, the returning non pilot customers report being slightly more satisfied with this aspect than returning pilot customers (1.55 as compared to 1.66), but this was again not a statistically significant difference. Because the signage and waiting room were present for all types of customers, it is not unexpected that no significant differences were measured. 4

Table 4. Q7: Were you able to understand the signs and directions (1 = Very Satisfied, 5 = Dissatisfied) Standard 95% Confidence Customer Type n Mean Score Deviation Interval New Customer Non Pilot 64 1.72 0.77 (1.53, 1.91) New Customer Pilot 191 1.53 0.79 (1.42, 1.65) Returning Customer Non Pilot 84 1.55 0.68 (1.40, 1.69) Returning Customer Pilot 184 1.66 0.85 (1.54, 1.79) Table 5, however, indicates that there were significant differences in satisfaction with the receptionist who checked them in. New pilot customers report a mean satisfaction score of 1.52 whereas new non pilot customers report a mean satisfaction score of just 2.02, suggesting that the new pilot customers were more satisfied with the receptionist than the non pilot customers. Further, the confidence intervals indicate that this difference is indeed statistically significant. Table 5. Q10: How you felt about the Receptionist who checked you in (1 = Very Satisfied, 5 = Dissatisfied) Customer Type n Mean Score Standard 95% Confidence Deviation Interval New Customer Non Pilot 64 2.02 1.13 (1.74, 2.29) New Customer Pilot 193 1.52 0.80 (1.41, 1.64) Returning Customer Non Pilot 85 1.76 1.10 (1.53, 2.00) Returning Customer Pilot 181 2.03 1.23 (1.85, 2.21) Figure 2 provides a comparison of the relative frequencies of the individual responses which demonstrates that a much higher percentage of the pilot customers feel very satisfied than non pilot customers. Returning pilot customers, however, do not indicate significant differences from returning non pilot customers, likely because returning customers encountered the same receptionist as did non pilot customers. Figure 2. Frequency of responses by new customers to Q10: How did you feel about the receptionist? 5

Effectiveness of new service model In addition to improving the overall level of customer satisfaction, the Piccard pilot also sought to improve the effectiveness of the intake, screening, and referral services being delivered. The effectiveness of this new model can be assessed via the survey in several ways, including Question 13 which asks whether they were able to get what they came for, Question 8 which inquires about whether the customer service team members in the lobby were able to help them, and Question 11b which asks whether the staff told the customer about other services they may be able to obtain. Table 6. Q13: Customers who were able to get what they came for Customer Type n Yes Percent 95% Confidence Interval New Customer Non Pilot 57 25 43.86% (27.09%, 61.44%) New Customer Pilot 156 87 55.77% (43.83%, 67.42%) Returning Customer Non Pilot 82 68 82.93% (61.96%, 100%) Returning Customer Pilot 168 103 61.31% (49.15%, 72.95%) The responses to question 13 are illustrated on Table 6, which shows that 56 percent of the new pilot customers were able to get what they came for as compared to just 44 percent of the non pilot customers. However, the 95 percent confidence intervals suggest that these differences are not statistically significant. While it is not reasonable to expect that customers will always get what they came for due to eligibility restrictions which are unrelated to the service delivery model, it is encouraging that the pilot customers reported a slightly higher rate. Table 7 demonstrates the efficacy of the customer service team members working in the lobby. This table illustrates that over 84 percent of the new pilot customers and over 78 percent of the returning pilot customers reported that the Customer Service team members in the lobby helped them. Excluding those customers who reported not needing help, over 94 percent of the new pilot customers and nearly 91 percent of the returning pilot customers who needed help were provided help by a Customer Service team member. Similarly, Table 8 shows that the triage staff was able to inform over 80 percent of the pilot customers about additional services that they may be able to obtain. Table 7. Q8: Did our Customer Service Team members in the lobby help you today? Customer Type n Yes Percent Did Not Percent Not Yes Need Help Needing Help New Customer Pilot 185 156 84.32% 23 12.43% Returning Customer Pilot 189 148 78.31% 26 13.76% 6

Table 8. Q11b: Did the staff tell you about other services that you may be able to obtain? Customer Type n Yes Percent Yes New Customer Pilot 174 140 80.5% The effect of staff members advising customers of potential additional services can be measured in greater detail by examining the List of Service Needs documents which compare the services requested by the customer to the services for which they end up being referred to. A summary of this analysis is provided in Table 9, where on average, 0.28 additional services were offered to each customer, implying that over one in four customers were provided with an additional service beyond what they requested originally. Table 9. Q: Difference in number of services requested and number received by pilot customers Services Additional Services Requested Total Services Total Visits Requested, Not Services Per Visit Received Rendered* Offered 258 3.60 0.10 0.28 3.77 *Customers did not meet the eligibility requirements for some services requested and the service requested was sometimes inappropriate for the customer. Efficiency of New Service Model One concern with the implementation of the additional customer service features in the Piccard Pilot was whether the new focus on customer service would substantially reduce the efficiency of the site in processing customers. Because additional steps were added to the process and an increased effort was made to make customers aware of other services they may qualify for, some decrease in efficiency was expected. Therefore, this evaluation seeks to quantify the decrease in efficiency in order to verify that it is still acceptable. In addition, this evaluation studies the improvement to customer service related process issues such as the number of customers who leave without receiving services. For comparisons to the pre pilot environment, data were available only for customers receiving Income Support (IS) services. Therefore, comparisons between pilot and pre pilot customers will utilize only these customers. Table 10 provides an overview for all customers regarding the number of visits, average wait times, and the number of customers who signed in but never received services or saw a triage worker. For triage visits, the wait time is calculated as the elapsed time between when the customer first signs in and the time that triage service begins. 7

Month of Visit Number of Visits Table 10. Triage Summary Statistics for all pilot customers Percent Left Average Wait Left Without Without Time Service Service Percent Not Appearing for Future Appointment March 219 0:31:00 10 4.57% 6.39% April 228 0:39:53 13 5.70% 8.77% May 252 0:34:13 25 9.92% 7.94% June 216 0:23:55 29 13.43% 11.57% July 213 0:25:18 2 0.94% 10.80% August 297 0:22:51 44 14.81% 10.10% Average 237.5 0:29:30 20.5 8.63% 9.26% The number of visits remained relatively constant from month to month, with an average of 237.5 visits per month. The average wait time for a triage customer was 29.5 minutes, and it is noteworthy that the wait times in the last three months are notably shorter than the wait times in the first three months, suggesting that the process improved in efficiency over time. An average of 20.5 visits per month (8.63% of all customer visits) resulted in the customer leaving before receiving services. Table 11 indicates that 450 out of 1,329 triage visits (33.9%) resulted in services being provided on the same day the customer initially presented to the triage worker. Additionally, 268 of these visits resulted in follow up services being provided within 30 days. Because all future appointments are scheduled within 30 days of their initial visit, one can consider just these 268 customers combined with the 450 customers receiving service on the same day as the set of 718 customers who needed a direct service appointment, and therefore 450 out of 718 customers (62.7%) who needed service appointments were able to receive them on the same day. Please note that the totals in Table 11 do not match the totals in Table 10 because only the customer s first triage visit in each month is included in this analysis. Month Table 11. Days until direct service for triage visits Direct Service 1 30 days Direct Service Day Same No Direct Service within 30 Days Total March 48 86 75 209 (percent) 22.97% 41.15% 35.89% April 32 72 117 221 (percent) 14.48% 32.58% 52.94% May 32 70 133 235 (percent) 13.62% 29.79% 56.60% June 49 82 76 207 (percent) 23.67% 39.61% 36.71% July 53 66 65 184 (percent) 28.80% 35.87% 35.33% August 54 74 145 273 (percent) 19.78% 27.11% 53.11% Total 268 450 611 1,329 8

Table 12. Pre and Post Pilot Income Supports Wait Time Comparison Type Client Type Visits Wait Time Left Without Percent Leaving Service w/out Service Pre Pilot Appointment 468 0:22:15 5 1.07% Walk in 189 0:22:05 5 2.65% Pilot Appointment 1266 0:25:00 6 0.47% Walk in 165 0:29:37 2 1.21% Table 13. Significance test for Pre and Post Pilot Income Supports Wait Time Comparison Difference between Visit Type n Pilot & Non Pilot t Statistic P value Wait Time Appointment 1,723 2:45 1.67.0946 Walk in 347 7:22 1.95.0518 For comparison to pre pilot operations, Table 12 illustrates the differences in wait times among customers needing Income Supports services between pilot and pre pilot customers. For the purposes of these analyses, wait time was calculated as the difference between either the appointment time or when the customer arrived (whichever was greater) and the time when service began. This table indicates that the wait time has increased somewhat for these customers, particularly the walk in customers whose wait time has increased from 22 minutes to almost 30 minutes. Such an increase was expected as these customers received additional intervention during the course of their visit. Table 13 provides statistical significance testing of the wait time increases, and based on these calculations it appears that neither walk in customers nor customers with appointments are facing statistically significant differences in wait time, as both p values are greater than 0.05. However, it should be noted that the p value for the walk in customers is quite close to this value indicating that some caution should be used in interpreting these results. Another important facet of the Piccard Pilot was the introduction of reminder calls to clients with future appointments. Selected customers with scheduled appointments were provided with reminder calls by a Customer Service team member. As shown in Table 14, these reminder calls appear to dramatically reduce the number of customers who do not appear for their appointment. Of the 465 customers who received a reminder call, only 17.6 percent failed to make their appointment, while 56.7 percent of the 927 clients who did not receive a reminder call failed to appear. Table 14. Impact of Reminder Calls on No Show Rates Reminder Call Clients No Show Rate 95% Confidence Interval No Reminder Call 927 56.74% (51.83%, 61.60%) Reminder Call 465 17.63% (13.99%, 21.82%) 9

Discussion Based on the above findings, it appears that the overall level of customer service has improved via the new processes which were implemented, particularly for new customers of the site. New customers experiencing pilot operations reported significantly higher levels of overall satisfaction than did new customers not experiencing the pilot operations. Additionally, the customer s perceptions about the receptionist who checked them in were significantly better for pilot customers than non pilot customers. This new service model also appears to be more effective than the previous model; the survey data show that over 80 percent of the pilot customers were told about additional services that they may be able to obtain, and empirical data indicates that over one in four customers end up being directed to at least one additional service. Also, over 90 percent of the customers who needed help reported feeling that the customer service team members were able to provide help. The implementation of the reminder calls for follow up visits appears to be an overwhelming success in reducing the number of customers who fail to appear for future appointments, as no show rates among those receiving reminder calls were reduced to less than 18 percent from nearly 57 percent among customers not receiving calls. Not surprisingly, the increased efficacy and customer service did incur a penalty in the efficiency of the pilot site. These analyses show that the average wait time increased several minutes for customers being served during pilot operations. These differences, while substantial, were not found to be statistically significant. Additionally, it is worth noting that the wait times were longest during the first several months of pilot operations and decreased as the pilot project continued, suggesting that the pilot staff were able to become more efficient as they gained more experience with the new model. Conclusion This evaluation demonstrates that the Piccard pilot has been successful in both increasing customer satisfaction as well as improving the service being delivered to their customers. Customers visiting the pilot site reported higher levels of satisfaction and were being directed to additional services that they had been previously unaware of. The pilot methodology also appears to substantially reduce the number of customers who fail to appear for future appointments. There is, however, a performance tradeoff associated with these improved outcomes as wait times increased by several minutes for all customers visiting during pilot operations. Some evidence exists which suggests that these increased wait times may diminish over time, but management should nonetheless carefully evaluate the potential impact of these changes. Given the demonstrated improvements to customer outcomes, however, it appears that this slight decrease in efficiency is likely outweighed by the improved level of service. 10