CTG.ORMA Center for Technology in Government University at Albany / SUNY

Transcription

1 Reviewing the Performance of ORMA's Voice Response System for Automated Business Permit Information: Integrating Technical, Cost-Based, and Customer-Oriented Evaluations of System Performance CTG.ORMA - 10 Center for Technology in Government University at Albany / SUNY ª 1995 Center for Technology in Government The Center grants permission to reprint this document provided that it is printed in its entirety

2 Executive Summary The Center for Technology in Government, located at the State University of New York at Albany, is a collaborative research center which pursues new ways of applying computing and communications technologies to the practical problems of information management and service delivery in the public sector. The Center conducts projects through partnerships between the University, drawing and faculty members, state, and local government agencies, and over two dozen corporate partners. The New York State Office for Regulatory and Management Assistance is a small, nontraditional agency with diverse program responsibilities. ORMA programs include providing permit information to new or expanding businesses or non-profit undertakings; reviewing State agency rules before they become effective; conducting a regulatory reform program to improve the existing regulations; providing a Master Application Procedure to clients with complex business undertakings and facilitating projects to encourage innovation and increase productivity within government. The Permit Assistance Program is the oldest, most mature and most widely recognized program of the Office. The foundation for the program can be found in Section 887 of the Executive Law which directs the Office to provide a free permit information, coordination and assistance service At the high point in staffing, ORMA had seven full time professional permit coordinators, but because of statewide fiscal problems over the past several years staffing levels have been reduced to a total of four full time professionals. Client inquiries have not stopped, however, but have continued to increase every year of operation. In order to transform the way its Permit Assistance Program operates and to allow the agency to redeploy limited resources to other program areas, ORMA proposed to investigate advanced interactive voice response and information technologies. ORMA expected that an interactive voice system would increase client contacts, provide faster response, offer greater availability of services, increase personnel productivity, improve client satisfaction, lower operating costs, provide a higher and more consistent quality of service, and allow the agency to move ahead with its increased management and regulatory assistance efforts. As a mechanism for investigating the feasibility and effectiveness of implementing an advanced interactive voice response system, ORMA submitted a project proposal to the Center For Technology Government. This project was subsequently selected as one of four to be conducted by the Center. The goal of the project was to enable the Office for Management and Regulatory Assistance to increase operational efficiency in its Permit Assistance Program, increase the quality, of services provided, expand the array of services available and enable management to redeploy resources which have been freed up through the use of advanced voice information and response technologies to other program areas within the agency. The project objectives are outlined below.

3 1. Report Purpose and Overview This report presents the results of the Center for Technology in Government's (CTG) formal efforts to evaluate a New York State Office for Regulatory and Management Assistance (ORMA) prototype voice response system for automated business permit information. Other CTG reports, memos, and working papers report details of the original proposal and the project deliverables. This report has four specific objectives: (1) To review the original research objectives of the ORMA project as it was proposed by ORMA; (2) To document how those original and rather narrowly focused research objectives were eventually expanded to include a broader set of questions; (3) To summarize a multi-method research approach that has been used by CTG to evaluate this entire project; and (4) To present answers to each of the research questions posed. These answers draw from threads of investigation taken from the several methods that make up the overall research approach. For more complete methodological or substantive details concerning each of the sub-studies referred to herein, interested readers are referred to the more complete texts of the research studies commissioned by CTG to support the ORMA project. These are listed in the References and cited in the text of this report. Original Objectives of the ORMA Project As originally proposed, the ORMA project had four key objectives: 1. Investigate the range of advanced voice information and response technologies available and how they can be integrated into the delivery of services to clients offered through the Permit Assistance Program of the agency. 2. Evaluate how well the integrated voice response solutions meet the needs of ORMA clients. 3. Design, develop and implement an integrated voice response solution which will enable the Permit Assistance Program staff to better focus their efforts, thus increasing personnel productivity, and will offer clients faster and more consistent responses to inquires and a greater availability of services 4. Enable clients to prepare their own business permit assistance information kits using their own telephone equipment as an input device. 2. Expanded Objectives of the ORMA Project The original project objectives focused sharply on the process of developing, installing, and evaluating a prototype automated business permits systems within ORMA. As this process progressed, it became increasingly clear that in addition to those questions associated with the prototype technology itself, a number of other issues centering on how this new system would fit into existing processes, procedures, and technologies within ORMA also needed to be addressed. In turn, ORMA's existing processes, procedures and technologies were seen as just 2

4 one part of the larger system that New York State utilizes in disseminating business permit information to its citizens. Ultimately, citizens who call ORMA wish to do business with other agencies. Their goal is to understand and deal with relevant local, state, and federal governmental entities and regulations in New York State. Hence, the scope of the evaluation effort associated with the project expanded to include issues and questions that were focused more broadly than those related to the prototype system alone (CTG.ORMA-006). As shown in Appendix A, the final and expanded list of project objectives occur at three levels--(1) evaluation of the prototype; (2) evaluation of how the prototype fits into a broader picture of service delivery within ORMA; and (3) issues centering on how New York State disseminates business permit information and assistance to its citizens. At each level. the project focused on technical objectives, on issues of relative efficiency (volume of calls, cost of services, savings due to the system, and so on) and issues relating to effectiveness (comparative citizen satisfaction with services, accuracy of information collected, and reliability of responses). In the following discussion the original objectives are indicated in italics. 3. CTG's Approach to Evaluating the ORMA Business Permits Information Dissemination System In order to attain this expanded set of project objectives, CTG used a multi-method approach to learning about the prototype and how it fit into a broader picture of ORMA and New York State processes. As described below, five separate threads of activity and research are woven together to create the conclusions of this report. The first major category of lessons was arrived at in a "learn by doing mode." In the process of completing the project, a number of issues were raised and resolved and the record of these issues and their resolution form a major source of insights and lessons for the project. CTG has installed a careful project archiving process and the systematic review of the project archives is the major source of learning for this thread. The remaining four research threads represent formal research efforts designed to investigate specific questions related to the expanded set of research objectives. All five research and evaluation threads are described below. Archival Analysis of Prototype Project Records To a large degree, lessons from the ORMA project have been "learned by doing." In the course of solving problems associated with building the prototype or interacting with ORMA's staff or customers, a number of lessons about voice response systems in general and ORMA's business permits application process, in particular, have been learned. CTG maintains an extensive archive on each of its projects. Major documents associated with each project are in a separately numbered working memo series. All of these working memos are available at the cost of reproduction and distribution to any interested party. In addition, the detailed working notes and sketches of the entire project are captured in a formal archive. One of the working 3

5 memos, CTG.ORMA-008, Index to the Full CTG Archive for the ORMA Project, is an index to CTG's complete archive for the ORMA project. Many of the insights and lessons reported below are documented in the archive. Benchmarking Literature Review Early on in the project process, CTG staff completed a literature search on telephone answering and voice activated systems to provide background to project managers and to examine a number of specific questions posed by ORMA. The results of this benchmarking literature are presented in Giguere (CTG.ORMA-004). Cost Performance Modeling Conference During October, 1994 a team of researchers from CTG met with senior ORMA staff responsible for the permit assistance program. At that time, substantial progress had already been made on creating a prototype system. The purpose of this meeting was to analyze the costs and benefits of providing business permit information through an automated system. This conference elicited from the management team the costs of the new system in terms of hardware, software, and data entry plus miscellaneous expenses such as postage and copying associated with mailing out packets and the cost of faxing packets to citizens. The conference also elicited the costs associated with providing this same business permit information through human operators. An important portion of this conference was spent estimating how long it would take to serve a citizen through an automated system versus a human operator. The cost performance modeling conference focused almost entirely on the provision of business permit information for the Top 5 and Top 20 most high volume business models measured in terms of average number of calls per year per business model. ORMA had previously identified these "Top 5" and "Top 20" business models as the highest volume and hence the best candidates for automation. The results of this modeling conference revealed that for the current volume of calls, hiring one more human operator would be a more cost effective way than purchasing and installing an automated system to handle the business permit inquiry process (costs were estimated over a five year time horizon). The conference also revealed that the full costs and benefits of the automation could not be fully evaluated by looking only at the "Top 5" or "Top 20" business models in isolation. The results of the conference indicated that the potential impact of adding additional ports to the existing telephone response system to handle screening of all calls needed to be examined. The cost and performance modeling conference proposed that a more complete simulation model of ORMA's automated permit assistance processes be constructed and analyzed. That model is discussed immediately below. The full report of the cost and performance modeling conference is presented in Andersen and Rohrbaugh (CTG.ORMA-005). Process Simulation Model of Automated Permit Assistance Procedures Between November of 1994 and February of 1995 CTG constructed and analyzed a system dynamics simulation model of the ORMA's complete permit assistance program. The model simulated incoming calls including call backs, pre-screening by the automated system, forwarding to an automated permit assistance program, forwarding to human operators, and 4

6 the client service activities of human operators. Within a simulated environment, this model explored the implications of hiring more operators, adding more telephone lines, and adding more ports to the system supporting the voice response and automation system. The model also explored the implications of adding these various types of capacity in the face of constant customer demand for information, or constant volume as well as a doubling and tripling of the base level of demand. The simulation is capable of analyzing operator utilization, client waiting times for services, total call volume, estimates of number of lost calls, plus a number of other variables relevant to the operation of the permit assistance program. The model relied on data from the prototype system as well as data from ORMA's present system. The simulation model confirmed the results of the cost performance modeling conference that under present levels of client demand, hiring one more operator would be more efficient than automating the permit assistance program. However, this analysis demonstrated that hiring an additional operator without expanding the base capacity of the basic phone system and computer system supporting those operators would lead to low marginal productivity of human operators and wasted resources. The model demonstrated that at higher volumes of demand, the automated permit assistance program could substitute for one or more human operators. In addition, the model demonstrated that the volume of inquiries being handled by the current ORMA system is probably limited by internal capacity constraints, rather than demand (i.e., ORMA appears to lack sufficient capacity to respond to existing incoming calls). A complete description of the assumptions, structure, data needs, and results of this simulation exercise is contained in Mojtahedzadeh and Andersen (CTG.ORMA ). Experimental Survey of ORMA's Clients Whereas the prototype development process focused primarily on technical issues and the cost performance and process simulation models focused primarily on measuring and analyzing system efficiency (volume, costs, capacity constraints and utilization), a final research thread concentrated on looking at the relative effectiveness of various approaches to disseminating business permit information. During January, February, and March of 1995, CTG conducted an experimental evaluation of how well ORMA's permit assistance program met the needs of clients seeking permit-related information. Under quasi-controlled circumstances approximately 60 citizens (most of them graduate students in business and public administration) were directed to find and report permit information necessary to start one of six types of businesses in a New York State setting. The six business types chosen were ones for which the existing prototype system could provide the necessary information. These sixty students were divided evenly into three groups--one group of twenty was directed to use ORMA's prototype system as developed and tested in the CTG project, one group was directed to call into ORMA's existing system (eventually connecting to human operators), and the final group was given no information about the existence of ORMA and was told to solve the problem of getting permit information using other existing information sources (e.g. the public library or the phone book). All participants in the experiment were asked to limit their search to two hours. Participants turned in an answer sheet that described what permits they needed to start their assigned business as well as what forms and fees would be required. In addition, all participants turned in a time log 5

7 demonstrating how they used their time and completed an extensive survey to find out their reactions to doing business with New York State through their experimental condition (knowing about the Prototype, knowing about ORMA, not being directly referred to any assistance). Participants were paid twenty dollars for completing the experiment. The experiment gathered information on how accurately and reliably information was being gathered through the three types of treatments (referred to Prototype, referred to ORMA, or control group). Overall, participant satisfaction with the information search process was elicited along with satisfaction and frustration measures for sub-components of the information search process (quality of phone lines and connections, quality of interaction with human operators, pattern of referrals to other agencies, and so on). Participants were given several opportunities to report their overall reactions to doing business with ORMA and with New York State, in general. Finally, participants were asked to rate their willingness to pay a fee for several enhanced levels of services that New York State or ORMA might make available to citizens who were planning to open a business in New York State (for example, how much would participants be willing to pay for a service that located all of the necessary forms and actually filled them out and returned them for signature after an initial interview). The complete details of the experiment and participant survey are contained in Andersen, Avery, Hyde, Kelly, and Kim (CTG.ORMA-009). 4. Results and Issues Arising from the Prototype and Its Evaluation Results from the above five types of research activities yielded answers to the expanded set of research questions posed by the overall ORMA project. Summary answers to the research questions are presented below. Readers who are interested in the details, including the full methodological details for each thread of investigation, are referred to the separate research reports available as project memos from CTG. 4.1 Level I: Evaluating the Feasibility of Automated Business Permits Prototype Systems Prototype Level Technical Issues. 1. Can the dissemination of complex business permit information be automated? The prototype has clearly demonstrated that information as complex as that for obtaining business permits clearly can be automated within the context of a voice response system and made available over the phone to clients. How well this technology is received by citizens as well as its costs and benefits are discussed below. 2. What is the range of advanced voice information and response technologies available? The prototype utilized speaker-independent voice recognition (SIVR) as an alternative input method for callers who did not indicate that they had a touch-tone telephone. The SIVR was capable of understanding the spoken digits Zero (it would also accept Oh ) through Nine, and the words Yes and No. An alternative form of voice recognition, speaker dependent voice recognition (SDVR), was investigated. In general, SDVR can recognize a 6

8 wider vocabulary than SIVR, but it needs to be trained to understand different voices. Because this was impractical for an agency which services many thousand individual callers, SIVR was chosen. Call routing is available on the Precision Systems, Inc. UniPort system. Through the use of call routing, a caller can indicate if he wishes to be transferred to a different agency without having to hang up and dial a different number. This capacity requires an additional component, known as a front end switch, to be added to the system. Due to the expense involved in obtaining a front end switch, call routing was not part of the prototype. The UniPort system is capable of using pre-recorded voice files for delivering information to callers. It is also capable of converting textual files (such as the ORMA permit assistance database) into spoken speech (text-to-speech). The prototype design relied upon pre-recorded voice files since text-to-speech technology does not yet produce a speech sounding as lifelike as pre-recorded voice files. However, when ORMA implements a complete system, text-tospeech may prove better suited so that no additional modifications would be needed if information in the permit assistance database changes. If pre-recorded voice files were used in the final system, the voice file would have to be modified to match the change in the information contained in the permit assistance database. 3. How can these advanced technologies be integrated into the delivery of services to clients offered through the Permit Assistance Program of ORMA? The prototype system implemented by the CTG project created a voice response system that could be easily integrated into the existing phone response system being used by ORMA. This enhanced prototype system contained voice response features, fax back features (not evaluated in the experiment), and an automated business permits information module for the "Top 5" business models. In order to fully utilize the capabilities of the new system, ORMA would have to install a total of 2 T-1 telephone lines. Each T-1 line is the equivalent of 24 normal telephone lines, and the UniPort system is equipped with 48 input ports. Using 2 T-1 lines would thereby fully utilize UniPort s input capacity and would provide ORMA with the capability of serving 48 separate callers simultaneously. The fax-back capacity of a final production system would provide immediate transmission to callers of the information already contained in ORMA s permit assistance database. Currently, the permit coordinators must manually retrieve the information from the database, assemble hard copies of all relevant material, and send it to the caller by mail. In the event that the caller did not have a fax machine, the system would also allow the caller to leave a name and address at the end of the inquiry process so that ORMA permit coordinators could send the material by mail. 4. How should ORMA's automated business permit system be linked to the agency's main data base of permit information? 7

9 ORMA had originally intended to create a link between the new voice system and the mainframe which stored its permit assistance database. This would have required some programming efforts on both the new system and the mainframe, together with a physical interface joining the two systems. This proved difficult since the mainframe was no longer supported by its producer. Late in the prototype process, it was discovered that there was no need to link the prototype system to the agency's main data base of permit information, since all of the relevant permit information could be contained within the computer used to handle phone traffic and run the automated permit assistance program. However, having this additional capacity available to ORMA would require an upgrade from its then existing equipment. Prototype Level Cost and Performance Issues (Efficiency). 5. What are the relative costs and benefits of automating complex business permit information? This turned out to be a rather complex question that was analyzed both in terms of the cost and performance modeling conference and the process simulation model. Both threads of investigation returned similar, but complementary insights. At the present volume of calls, automating the Top 5 or even the Top 20 business models appears not to be cost justified. Under present conditions, ORMA fields approximately 150 inquiries per day and only about 35% of these are business permit inquiries. In turn, only a fraction of the business permit inquiries are handled in the Top 20 (about 20% of all business permit inquiries) Hence at present call levels, automation of the business permit assistance function would handle only a handful of calls per day (8 to 10 per day on average) and it turns out to be more cost effective to add more phone lines and then to add a human operator who could answer all types of questions--not just the Top 20 business permit questions. 6. How do these costs and benefits change as the number of business models automated increases? ORMA provides business permit information for literally hundreds of different types of businesses and most of these business types have unique features and may only come up once or twice a month. An analysis of volume of calls by business types indicated that it would not be cost effective to consider automating more than the Top 20 most frequently inquired about types of businesses. The marginal cost of automating more business models is more or less fixed and the marginal benefits decrease because of declining frequency of calls beyond the Top How do these costs and benefits change as the volume of customers calling ORMA increases? The process simulation model explicitly analyzed how the volume of incoming calls affects the cost-effectiveness of an automated permit system. Due to the fact that a higher number of calls will generate more Top 20 inquiries, the system becomes more cost beneficial as the volume of calls increases. When the volume of calls doubles and when additional lines and 8

10 ports have been purchased, automating the Top 20 business permits has the same effect on overall system response as hiring an additional operator. The same effect holds for a tripling of volume. Of course, if call volume were to increase by a very large amount (say a factor of 5 or 10), automation of the Top 20 could trade off for more than one operator. These extremely high volume increases were not explicitly analyzed in the present simulation analysis, but the analysis could easily be extended to include increases of this scale. Prototype Level Customer Reaction Issues (Effectiveness). 8. Is the information being disseminated by an automated business permit system accurate and reliable? Logically, the accuracy of information dissemination divides into two parts--the accuracy and completeness of the information being given out by some source and the accuracy and completeness of the information as received by a client. This analysis must assume that the information encoded in the prototype is accurate and that the information base of ORMA operators is complete and accurate. Hence any inaccuracies are a combination of incomplete information being either sought or given and inaccuracies in how information was perceived and recorded by participants in the experiment. The best measure of the accuracy and completeness of information thus defined is the final score that participants received on their answer sheets. Recall that participants were asked to obtain complete information about the agencies with whom they would need to interact to start a business, forms that they would need to file, and fees that they would need to pay. Within the context of this experiment, the grading of participant answer sheets demonstrated that for all groups tested, the accuracy and completeness of information being collected was low. The average accuracy and completeness rate for all experimental participants was only 38%. This means that when the information search was complete, the average participant, including those who called the prototype or ORMA, wrote down less than half of the information that they should have collected. The highest score for all participants in the experiment was 74%, indicating that even the best performing participant missed roughly one quarter of the information that should have been collected. The large standard deviation in responses indicates that this rather low overall accuracy varied quite a bit from participant to participant. Several participants received a score of zero, indicating that after 2 hours of information searching, they were not able to come up with any correct information. Ironically, in light of the relatively low scores for information collection, 43% of all of the participants were either confident or highly confident that they had obtained all of the information that they needed to file for the permits related to their business problem. A statistically significant correlation was found between participants actual accuracy score and their confidence level in having obtained all of the necessary information for the experiment. 9

11 Table 1, below, presents the number of individuals in each group, the mean and standard deviation for percent correct on permit answers for each of the conditions in the experiment, as well as the percentage of participants in each group who indicated that they were confident or highly confident that they got all the needed information. Notice that in Table 1 a fourth condition has been created--"control Group with ORMA Contact." Of the 16 useable responses in the control group, 5 respondents found out about and called into ORMA. Their responses are broken out from the rest of the control group. Of those five who learned about ORMA during the course of the experiment, four indicated their source. Two people indicated that they were referred by the Small Business Development Center at the University at Albany. Another individual indicated that they obtained information about ORMA from the phonebook, while another learned about ORMA during a telephone conversation with the NYS Department of Commerce. Table 1 Accuracy and Reliability of Responses by Group (with Group Measures of Confidence in Responses) Standard Deviation of Mean Score Fraction Confident Got All Information Experimental Group Group Size Mean Score on Permit Responses ORMA Group Prototype Group Control with ORMA Group Control with no ORMA Group TOTAL Of the four groups shown in Table 1, the portion of the control group which never had contact with ORMA stands out as lower than the other three groups with respect to mean score. Appropriate statistical tests indicated that the completeness score for the control group was consistently and significantly lower than the other three groups which had contact with either ORMA or the prototype. This demonstrates that, based on this sample, interacting with ORMA does have a significant and positive effect on overall accuracy of information gathering. Additional questions remain as to why those participants who had contact with ORMA in one form or another, did so poorly in terms of accuracy and completeness and why there was such a high degree of variability in scores. An item-by-item analysis of the answers which were missed did not reveal any strong pattern for either the ORMA or Prototype group. Missed answers seemed to be spread randomly in the answer sheets. There is some evidence from the experiment that participants using the Prototype were having some difficulty capturing all of the necessary information as it came to them over the phone (the FAX back feature was not working during the test). On average, a participant spent 39 minutes interacting with the 10

12 prototype phone system. Of the participants which interacted with the prototype system, 61% indicted that they had to have options repeated several times while 89% indicated that they had to have options repeated at least once. With respect to those individuals who contacted ORMA s current system, the data also suggest that participants had some difficulty navigating their way through the system. For example, one third of the participants who reported having contact with the present automated screening system did not report having made contact with a human operator. It appears that these individuals hung up before they got to an operator believing that no further assistance was available to them. It is necessary to contact a human operator under the current system in order to obtain the permit information. The inability of these individuals to get through to the human operator provides some explanation as to why the scores were so low. The experiment provides no clear answers as to why the overall participant accuracy and completeness scores were so low. These results point to a clear need to understand, in more depth, the behaviors of clients who call into the systems and how information can be more effectively transferred to these clients. 9. How satisfied are customers who interact with an automated business permit system? How do these measures of customer satisfaction compare with other customers who interact with human operators? Client satisfaction with the prototype can be decomposed into several component parts. First, the level of client satisfaction with their overall experience calling into the prototype and this level of satisfaction relative to that of the participants who contacted human operators at ORMA must be considered. Second, it is necessary to examine how the levels of satisfaction vary by different components of the overall experience such as getting through on a phone line, being pre-screened by the automated voice response system, and finally receiving more complex permit information from the voice response system or from human operators. Finally, an analysis needs to be done to determine whether the survey results provide any suggestions as to why clients are more or less satisfied with various components of the service that they get from the prototype. Each of these components of satisfaction are discussed below. Table 2 presents a summary of level of satisfaction of participants who interacted with the prototype as compared to those who interacted with ORMA--both human operators at ORMA and the present automated call screening system at ORMA. Each cell reported in Table 2 presents the percentage of participants in that group who agreed with the survey item. For example, 81% of the participants who called into the prototype either agreed or strongly agreed that they were "overall satisfied with the interaction". Table 2: Comparison of Satisfaction Measures for Participants Calling into the Prototype versus those Calling into ORMA by Selected Components of ORMA Service Percentage of Total Agreeing with Satisfaction Statements 11

13 ORMA Human Operators Survey Item Prototype Overall Satisfaction with Interaction 81% 100% NA Easy to Get Information 53% 100% NA Enjoyed Interaction 41% 100% 11% ORMA Automated Call Screening As shown in Table 2, 81% of the prototype clients expressed overall satisfaction with their interaction. However, only 53% agreed that it was easy to get information from the prototype system. The data also indicates that 30% disagreed with the statement that it was easy to get the information from the prototype system (not shown in the table). As indicated, 41% enjoyed their interaction with the system. This contrasts with a 100% satisfaction measure for all clients who interacted with human operators at ORMA for all three components. However, when asked about the portion of their interaction that involved ORMA's automated call screening, only 11% of the ORMA clients who responded agreed that they enjoyed this portion of the overall interaction. Table 3 further examines these interactions by comparing client satisfaction with the 800 phone line system connected to the prototype in comparison to the 800 and 474 phone lines currently being used by ORMA. The 800 phone line associated with the prototype was a commercial service maintained by the vendor. Service was interrupted for one day of the prototype test, consequently 90% of the prototype participants agreed that they had no difficulty getting through and 94% were satisfied with the timeliness of their phone connection. These two figures contrast with 92% and 100% satisfaction with ORMA's 474 service but a much lower 44% and 38% satisfaction with ORMA's present 800 phone line system. This table indicates that performance of ORMA's 800 phone line service was detracting from customer satisfaction during the time of the experiment. Table 3 Comparison of Client Satisfaction with Various Types of Phone Line Connections Survey Item Prototype 800 Phone Lines ORMA 474 Phone Lines ORMA 800 Phone Lines No Difficulty with Phone Connection 90% 92% 44% Satisfied with Timeliness of Phone Connection 94% 100% 38% Got Through on First Call 89% 73% 17% Table 3 also provides some explanation as to why clients are relatively dissatisfied with phone connections at ORMA, particularly the 800 phone system. While 89% of the clients calling 12

14 the prototype got through the first time that they called, this percentage dropped to 73% for ORMA's 474 phone system and to 7% for ORMA's 800 phone system. As shown in Table 4, 70% of those individuals who accessed the prototype system either agreed or strongly agreed that the recorded information was useful and appropriate for their needs. The remaining 30% were either neutral or disagreed with the statement. Of those individuals who interacted with the prototype, 47% indicated that they agreed with the statement that the process of obtaining this type of information was too complex to be handled by an automated system. Sixty-five percent of those using the prototype indicated that they perceived the recorded information to be complete. Table 4 Participant Perceptions of Prototype System (Percent in Agreement With Statement) Agree or Strongly Agree Statement Disagree Neutral Recorded Information was Useful 12% 18% 70% Process too Complex for an Automated System 47% 23% 30% Recorded Information Complete 12% 23% 65% With respect to overall satisfaction with the prototype system, 89% of the participants calling the prototype had to have options repeated at least once in order to get the needed information. This data seems to indicate that obtaining all of the necessary information from an automated system may be difficult. In addition, the average client calling into the prototype spent 39 minutes getting information about business permits. This figure includes one or more call backs for most of those calling into the prototype. On the other hand, clients who called into ORMA reported that they spent on average 10 minutes getting permit information from human operators. In sum, clients who called into the prototype system expressed relatively high levels of satisfaction with their interactions in spite of the fact that they did not find it easy to get the complete amount of information and most of them did not enjoy the interaction. The data indicates that those participants who called ORMA and spoke with human operators enjoyed the interaction more and felt that it was easier to get information. These conclusions from the experiment interact in important ways with those results from the cost-performance model and the process simulation study. These studies indicate that for present call volumes, it is not more cost-effective to automate business permit information as opposed to hiring another operator and that automating business permit information does not significantly improve the overall performance of the system. Hence, at present call volumes, 13

15 automating business permits would not reduce costs (with respect to the alternative of hiring another operator), would not significantly improve overall system performance, and would lead to relatively lower levels of customer satisfaction. However, at higher caller volumes (the model examined volumes that are double and triple the present call volume), the automated permits system can reduce costs and improve overall system performance with respect to total volume of calls processed. These cost reductions and capacity improvements will become increasingly higher as the volume of incoming calls increases. At these higher levels of incoming calls, ORMA's management will have to carefully weigh these cost and overall capacity enhancements against a potential decline in customer satisfaction caused by an inability to directly contact a human operator. 10. What Suggestions Do Clients Have For Improving the Operation of the Automated Business Permits System? Overall, 18 experimental participants used the prototype automated business permits system and all of these participants were asked what they liked best about the system, what they liked least about the system, and what changes would have made the system easier to use. Responses to questions such as these are often anecdotal and can sometimes be contradictory, with one participant wanting faster responses with less repetition in the system and another participant wanting slower responses with answers repeated more frequently. However, answers to this type of question can also be quite helpful because they provide for a type of focused and detailed insight into how the overall system functions and how it might be improved. Table 5 presents a summary of what participants liked best and least about the prototype system. In general participants appreciated most that the system existed and that they could get quick and easy access to quite a bit of information with abundant referrals. On the other hand, they found some of the details of the prototype itself to be difficult to work with--the number of and hierarchy of options was sometimes difficult to work with and participants wanted more ability to move freely within the menu of available choices. Table 5 What Participants Like Most and Least About the Prototype System What did you like best about the system? What did you like least about the system? 14

16 -quick and easy access (5 persons) -organization of main menu (4 persons) -the idea itself of the system -lots of information -easy to understand what to do -clear voice, non annoying tone of speaker -referrals for information & accessing an operator -too long -it was hard to go back and forth between options (2 persons) -the repetitiveness -too fast -model was too flat, needed to be more hierarchical -amount of options -once in a menu it was hard to go back to the original fields without going back to the main menu -often there is not enough time to write down the information When asked for suggestions as to system improvements, participants quite logically suggested that many of the features that they liked least be fixed (for a full description of participant suggestions, see the appendix of CTG.ORMA-009). In particular, participants indicated three broad classes of suggestions. First, they wanted to have more control over how they navigated through the system by being able to "skip around" more, by being able to "cut off" lengthy responses that they were not interested in, and by having access to "some sort of an index system." Second, participants wanted to have repetitious information (for their purposes) eliminated so that they could get through the menu more quickly. Finally, participants wished that they could gain access to more information, usually visual information. One participant wanted to have a general pamphlet available that could be mailed out and other participants wanted to be mailed a hard copy of what was being said over the phone (recall that the fax back feature of the system was not operational during the experimental evaluation). Another participant wanted even more referrals and detailed information about permits and licenses. Several participants believed that they got themselves caught in repetitive loops of information that they identified as bugs in the prototype system. 4.2 Level II: Evaluating ORMA's Overall Approach to Disseminating Business Permit Information ORMA Level Technical Issues. 11. Can we develop a system that will enable clients to prepare their own business permit assistance information kits using their own telephone equipment as an input device? The prototype, as developed, contained a feature which would enable clients to prepare and receive by FAX, their own custom-tailored business permit assistance information kits. However, this particular feature was not operational during the time that the prototype was being evaluated. Therefore, the impact of this feature on client reaction to the system has not been assessed. 15

17 Had the FAX-back feature been operational during the evaluation period, callers would have been given the option of having relevant information faxed to them. The information faxed to the caller, would have contained a one page document for each type of business permit required. These documents represent the actual records currently contained in the ORMA permit assistance database. Additionally, the caller would have received a faxed memo indicating any necessary additional steps, such as how to obtain a Federal Employer Identification Number from the Internal Revenue Service, or how to file incorporation papers. The particular package of information received by the caller would have been assembled by the UniPort system in response to the caller s menu selections and verbal or touch-tone responses to inquiries from the system. 12. How does the automated business permits assistance system interact with other types of telecommunications capacity such as the number of incoming phone lines and available ports in a computer system to answer the phone? The process simulation study explicitly analyzed the effects of varying combinations of number of phone lines, computer port capacity, level of automated permits, and number of human operators (CTG.ORMA-007). A relatively complicated set of relationships exists between these various types of capacities. In consideration of hiring additional staff or purchasing additional equipment of any kind, ORMA must carefully analyze which of these types of capacity are slack and which are fully utilized. For example, in the base run of the simulation model which represents ORMA s current configuration of operators, phone lines, and computer ports, hiring an additional operator without expanding the number of additional phone lines or ports would be relatively inefficient. This is because a relatively higher rate of marginal productivity of operators could be achieved with the addition of computer ports which would support the call screening features of the automated system and therefore free operators to address the more complicated calls. At this base volume, operators could not effectively be traded off for automation of permit assistance for the Top 20 model. As discussed above, the Top 20 model represents those business permits which comprise 20% of the total business permit transactions that ORMA handles. However, at higher volumes of calls, an automated permit assistance system, similar to that of the prototype, could replace human operators if the appropriate number of lines and computer ports were in place. In short, the relationships between types of capacity are complex and need to be carefully analyzed prior to the acquisition of additional resources. These observations about the need to carefully analyze how various types of capacity interact are especially important if ORMA were to plan to expand or modify substantially its business permits operations. As volume grows, the system moves through differing modes of capacity constraint and experience managing the system in a low capacity mode will most likely not apply at a much higher volume of business. ORMA should determine whether a simulated version of an expanded service system would "fly on paper" before it attempts to change its actual pattern of operations. ORMA Level Cost and Performance Issues (Efficiency). 16

18 13. Can we design, develop and implement an integrated voice response solution which will enable the Permit Assistance Program staff to better focus their efforts, thereby increasing personnel productivity? As discussed immediately above, the automated permit assistance program will not effectively substitute for human operators at the present volume and at the present cost of operators and software and other support for the permit assistance program. However, at higher volumes of calls, these trade-offs do lead to important increases in personnel productivity. At the base call level, expanding the number of lines and ports on the answering system immediately increases the marginal productivity of human operators. The process simulation model also indicates that the present ORMA system is capacity constrained (CTG.ORMA-007). Hence, adding lines and ports (and eventually operators) will relieve pressure on capacity. This will make it easier for callers to get through which will lead to the higher call volumes which will cost justify further automation. While these effects of capacity constraints were discussed in the simulation model analysis, they were not formally analyzed as the time horizon of the formal model was one day. These reputational effects will most likely take place over a much longer time period such as 18 or 24 months. The model captured these possible effects by assuming, as scenarios, that calls had increased by a factor or 2 or 3 for various simulation runs. The implications of these relatively long run reputational dynamics are very important in the consideration of shifting the overall scope of ORMA s business permits and information operations. 14. Will the new system offer faster services In examining the impacts of the new system on the amount of time it takes clients to do business with New York State, it is useful to think in terms of two states of affairs. First, ORMA continues to do business basically the same way that it does now, giving out permit information and using technology to disseminate that information or refer clients to other state agencies. Second, ORMA could consider drastically changing the way New York State interacts with its business clients by re-engineering its business processes as well as those of other state agencies. In the first instance, time impacts from the new system would be of three types--(1) time spent on the phone interacting with ORMA; (2) time spent following up on referrals; and (3) time spent getting copies of forms. As reported above, the automated permits system will most likely increase the amount of time that a client spends on the phone with ORMA. As previously indicated, those participants who called ORMA spent approximately 10 minutes on average, interacting with human operators and the automated screening system. Data from the experiment indicated that those participants who interacted with the prototype spent approximately 39 minutes on the phone. Based on this data, the increase in time spent per customer phone transaction could be as high as 400% (CTG.ORMA-009). Given the fact that the prototype system did not have a call routing feature which would have routed callers to the phone number of the next agency that they need to interact with, the 17

19 potential savings from such a feature has not be ascertained. Sections reported below document the number of agencies to which clients were referred and participants estimates of the amount of time it would take to contact and obtain information from the referrals. Finally, the prototype system did have a FAX back feature built into it. As mentioned above, this feature was not operational during the prototype evaluation period. Consequently, no data exists to measure the potential impacts of this feature on client turn-around time. Additionally, given the truncated time frame in which the participants were asked to work, comparative data on time to obtain referral information across groups would most likely have been invalid. If ORMA were to consider offering new types of services to its clients, clients might be able to considerably reduce the amount of time and bother associated with dealing with state regulations and requirements. Since these new products and services were not implemented in the prototype system, it is not possible to determine the potential time savings which would accrue to clients as a result of these features. However, as discussed in sections below, a proxy for time savings was collected in the survey in terms of a willingness to pay for services. The modal participant indicated a willingness to pay between $21 and $100 for a single phone-based service which would provide all of the needed forms and permits from one call and between $101 and $200 for a service that would actually fill out the forms based on information obtained from the client during an interview (possibly over the phone) (CTG.ORMA-009). These results are more fully discussed below. 15. What are the trade-offs between creating and maintaining an automated business permit system and hiring more human operators? How do these trade-offs change as the volume of incoming calls changes? These important trade-offs are fairly complex and have been treated explicitly in the process simulation model (CTG.ORMA-007). A summary of those findings are also presented above. 16, What are the implications of having business permit information available on a 24 hour per day basis? This issue was not explicitly and completely analyzed within the context of the process simulation model(ctg.orma-007). However, the major effect of 24 hour availability would undoubtedly be to relieve some of the peak load on the overall ORMA system. In the short run, this would allow ORMA day-time operators to provide service to a larger number of callers. Additionally, customer satisfaction levels associated with an increased ability to get through on the phone, would be expected to increase. This would have the effect of increasing ORMA's overall quality and quantity of services. The discussion in the simulation model indicates that, over time, these longer term reputational dynamics would draw more callers to ORMA, eventually leading to a new strain on capacity except at a higher equilibrium level. ORMA Level Customer Reaction Issues (Effectiveness). 18

20 17. How well do integrated voice information and response solutions meet the needs of ORMA clients? This larger issue was broken down into a number of components including the, accuracy of information dissemination, reliability or consistency of information given out, and overall client satisfaction with the process and its various parts (such as quality of phone service and connection or quality of interactions with operators). These components of client need are discussed separately below (CTG.ORMA-009). 18. Will the new system offer more consistent responses? Because the new system utilizes a computer to respond to a logically fixed set of questions through a fixed menu structure, the system will generate consistent answers. Each time a client enters the same set of responses to the system prompts, the computer will respond with the same set of answers. It would be expected that the consistency in responses would be higher with the automated system than those from an operator, everything else held equal. However, as shown in Table 1, above, groups of participants who called into the system with a matched set of questions did not, in aggregate, score very well on completeness and accuracy. In addition, there appeared to be a high degree of variability in those scores The best measure of consistency of responses available from the experiment is the standard deviation in the graded responses from the four experimental groups presented in Table 1, above--the prototype group, the ORMA group, the control group who called into ORMA, and the control group that never contacted ORMA. Since each group of clients was given exactly the same set of cases to solve, any differences in the standard deviation of the responses should reflect relative differences in the consistency of the responses as they were heard, interpreted, and written down by participants. Table 1 shows that the standard deviation for the completeness and accuracy score for the Prototype group was 16. This was less than the standard deviation for the ORMA group (23) and the control group that called ORMA (21), but less than for the control group that did not call into ORMA (13). In sum, the completeness and accuracy scores for those participants accessing the prototype system were found to be slightly more consistent than those who did not access the prototype. However, these scores may depend as much on the way that human clients listen to, respond to, and interact with the information disseminating systems than with the internal consistency of the information disseminating systems themselves. This suggests that perhaps ORMA should focus less on issues of consistency in the information disseminated and more on facilitating the ease of interpretation of that information. The Fax-back option implemented in the prototype system (but not tested in the experiment) may address this issue. If the participants had received a hard copy of the information disseminated by the prototype, less information would have been lost through participant misinterpretation, misperception, or inability to note quickly enough the system responses as they navigated through the system. ORMA might also focus on the manner in which the information is communicated. To do this, ORMA might engage in detailed debriefing of selected clients to assess the quality of the information communication as perceived by clients. Since gathering this data would be time 19