UM USER SATISFACTION SURVEY Final Report. September 2, Prepared by. ers e-research & Solutions (Macau)

Transcription

1 UM USER SATISFACTION SURVEY 2011 Fial Report September 2, 2011 Prepared by ers e-research & Solutios (Macau) 1

2 UM User Satisfactio Survey 2011 A Collaboratio Work by Project Cosultat Dr. Agus Cheog ers e-research & Solutios (Macau) Project Leader Dr. Paul W. T. Poo (Uiversity Libraria) Facilitator Wiie Leug (Uiversity Library) & User Satisfactio Survey Workig Team 2011 Uiversity of Macau 2

3 Cotets Executive Summary... 4 Itroductio... 6 Methodology... 7 I. Data Collectio... 7 II. Samplig... 7 III. Questioaire... 9 IV. Scalig... 9 V. Costructio of Customer Satisfactio Idex

4 Executive Summary The overall Customer Satisfactio Idexes (CSI) are costructed based o the four survey data, which are 70.6%, 71.9%, 69.8%, 70.1% ad 70.3% i 2004, 2005, 2007, 2009 ad 2011 respectively, idicatig a small fluctuatig patter. Takig the CSI, overall satisfactio scores ad specific figures of some uits ito cosideratio i the last five year surveys, the satisfactio levels ted to gettig small icrease for admi staff ad a dowward tred for academic staff ad studets respectively i AAO is the most importat factor that cotributes to the CSI while CMO ad ICTO are the two least importat factors i this regard i the staff sample. I the studet sample, REG, Faculty Office ad SAS are the three most importat areas that cotribute to the CSI while library is the least importat factor. For staff, about 82% of them claim that services meet or exceed their expectatios i 2011, which is 1% poit lower tha that i Besides, 64% of academic staff ad 88% of admi staff claim that services meet or exceed their expectatios, ad their mea score (-0.06 ad 0.52 respectively) differece is The overall evaluatio of academic staff is which is lower tha exactly meet their expectatio (0), at the same time, that of admi staff is 0.52 idicatig a attitude betwee exactly meet their expectatio (0) ad slightly exceed their expectatios (1). Therefore, it shows that the overall expectatio of admi staff is higher tha that of academic staff. For studets, about 78% claim that services meet or exceed their expectatios i 2011 which is 3% poit higher tha that i It shows a little chage for the staff ad studet samples. 65% of the staff claim that they sometimes or always make recommedatio of admiistrative services to others while 31% of studets sometimes or always do so i There is a slight decrease (2% respectively) for the staff sample ad the studet sample from last year. Sevety-two percet of the staff claim that the overall performace is improvig which is 5% poit less tha that i While 40% of studets have the same opiio which is 6% poit lower tha that i Twety-oe percet of the staff ad thirty-five of the studets replied that they ecoutered a service problem i the past year. The problems maily happe i the areas of classroom facilities, computer etworkig, commuicatios, ad procedure for reimbursemet claims, whereas frotlie service, computer rooms/computers, library are the mai areas that studets ecouter problems. Services like Cleaig, Procuremet, Reimbursemet procedures, Computer support, ad Maiteace are the top five that are suggested be improved by staff, while Catee service, Computer room service, Library service, E-purse value addig, Cleaig ad Sports complex veue retal are the most frequetly metioed services that eed to be improved by studets. 4

5 Demographic characteristics like staff type correlate the overall satisfactio with the performace of FO, Library, HRO, service year correlate satisfactio with Library i staff sample. It is foud, i staff sample, IT support service for computig facilities i offices, Disbursemet by auto-pay service, Politeess ad friedliess of the Library staff, Maiteace techiques, Staff welfare applicatio ad processig, Media service, Semester class schedulig, have the most sigificat effect o the satisfactio with ICTO, FO, Library, CMO, HRO, IPR ad AAO respectively. While i studet sample, Suitability of class schedulig, Studet couselig service, Procedure for payig fees & charges, Supportig service i computer rooms, Hygiee of restig areas o campus, Politeess ad friedliess of the Library staff are the most importat factors to the satisfactio with REG, SAS, FO, ICTO, CMO ad Library respectively. 5

6 Itroductio The Uiversity of Macau coducted user satisfactio surveys every 2 years i order to collect opiios about the facilities ad services provided by various admiistrative uits from the etire Uiversity commuity. Idetifyig the problems, weakess, stregth ad importace i these services will help the Uiversity maagemet to set a directio for future developmet ad provide better services for the Uiversity commuity. The 2011 survey adopted the same approach as that used i 2004, 2005, 2007 ad The curret report icludes the costructio of a customer satisfactio idex (CSI) for each survey i order to compare the performace i geeral over times. The followig research questios were asked ad aswered so as to provide useful referece for decisio-makig by the uiversity maagemet. How much are the respodets satisfied with the overall performace by the admiistrative uits? How do the respodets rate the performace by each of the admiistrative uit? What are the cocers by the respodets? What are the users suggestios to or opiios about the services? How does the users satisfactio chage over times? What demographics correlate satisfactio? What are the importat factors that cotribute to overall satisfactio with admiistrative uits? The structure of this report is divided ito six parts: Executive Summary, Itroductio, Methodology, Survey Results, Coclusio ad Recommedatios, ad Appedices. A more detailed Literature Review o user satisfactio survey ca be foud i the 2004 report. 6

7 Methodology I. Data Collectio The 2011 survey adopted three kids of data collectio methods. For the staff sample, we maily used olie survey ad supplemeted by paper-pecil questioaire. For the studet sample, we cotacted them by ad telephoe. II. Samplig For obtaiig a represetative sample, we coducted a cesus-like samplig of the staff i which each member of our staff received a stadardized questioaire by olie, distributio ad ig; ad we used a radom samplig techique for drawig a sample for olie ad telephoe iterviews with all registered studets. Eleve UM studets were traied to iterview, to exercise supervisio, ad to perform data-iput tasks. The samplig results are listed as follows. 1. Staff Sample A total of 1018 staffs were iformed to complete the olie survey at the first stage (4 th April to 13 th May, 2011) ad to complete the ad paper-pecil surveys at the secod stage (29 th April to 18 th May, 2011). A total of 553 completed questioaires were retured, amog which 520 were from olie survey ad 33 from paper-pecil surveys, coutig a overall retur rate of 54.3% 1 which is little higher tha that of the 2009 survey (50.8%). The retur rate from the admiistratio uits is 69%, whereas the retur rate from the academic ad research uit is 34%. The samplig error is ±2.8% at the 95% cofidece level. 2. Studet Sample A total of 1200 studets were radomly selected from the total of 6939 active studets of the Uiversity, iterviewed by olie survey (4 th April to 6 th May, 2011) ad the Computer-Assisted Telephoe (19 th April to 21 st April, 2011). A total of 623 completed questioaires were collected, amog which 289 were from olie survey ad 334 from telephoe survey. A total of 800 studets were iformed to complete the olie survey at the first stage, at the ed a total of 298 studets respoded the questioaire ad 289 successfully completed it, coutig respose rate of 97% 2.The a total of 911 studets which icluded the first stage icomplete samples combied with aother 400 studets were iterviewed by Computer-Assisted Telephoe Iterviewig (CATI) system at the secod stage. At this stage, a total of 348 studets respoded the questioaire 1 retur rate=the umber of completed questioaire/ the total umber of iterviews; 2 Respose rate= the umber of completed questioaire/ the umber of respoded the questioaire. 7

8 while 563 were ot available to be iterviewed due to busy lie, ot beig at home ad other techical reasos. I the ed, 334 were successfully iterviewed through telephoe, coutig a respose rate of 96%. Evetually, a total of 646 studets were cotacted, 623 available questioaires were received, coutig a overall respose rate of 96.4% The samplig error is ±3.7% at the 95% cofidece level. 8

9 III. Questioaire The same questioaires were adopted as that of the year 2009 survey except for a few wordig chages ad addig ad deletig of some service items by some uits (Refer to details i the appedix IV) IV. Scalig The te-poit scale For the satisfactio ad performace ratig questio, we adopted the te-poit scale for several reasos. 1. The te-poit scale is preferred because it ca reflect icremetal chages over time whe used repeatedly, ad it ca reflect the extet of progress i reachig service targets (Hero & Whitma, 2001). 2. The te-poit scale is easily uderstood ad avoids a umeric midpoit while a 5-poit or 7-poit scale offers a midpoit which would allow the respodet to avoid aswerig the questio. 3. The 10-poit scale ca help to measure whether the user is more or less satisfied, i however small degree. The labels at each ed ca deote the extreme limits of dissatisfactio ad satisfactio, respectively. The followig illustratio shows the iterpretatio of such scalig ad the average scores from the sample. Questio: What is your overall level of satisfactio with all services provided by various admiistrative uits of UM? [1] [2 3 4] [5] [6] [7 8 9] [10] Lowest Highest Scores of 1 ad 10 are extreme; few people probably choose either of these scores. Scores of [5 6] idicate oly slight dissatisfactio or satisfactio; however, selectig the 5 or 6 has a icliatio i oe directio or the other. The [2 3 4] ad [7 8 9] rages idicate dissatisfactio ad satisfactio, respectively. Most people will respod i these rages. [7 8 9] groupig offers the respodet a way to fie-tue a o-extreme score. That is, a score of 7 idicates moderate satisfactio ad sigals that there is room for improvemet without expressig actual dissatisfactio. The same reaso applies to [2 3 4] groupig. A average score of at least 8 is very good, whereas people who score a 7 are idicatig that they are ot exactly dissatisfied, but that they are ear the lowest rage of satisfactio. Scores below 7 should be a cause of cocer, but of greatest ad most immediate cocer are those who score i the 1 to 4 rage. These 9

10 resposes are clearly sigalig certai dissatisfactio. Imagie that the lower the score, the louder the voice of dissatisfactio. Aother type of sigificat questios is the users expectatios score: Please idicate whether our services fall short of, exactly meet, or exceed your expectatios Somewhat Slightly Exactly Slightly Somewhat Fall Short Fall Short Meet Exceed Exceed of of Expectatio Expectatio Expectatio Expectatio Expectatio Completely Fall Short of Expectatio Completely Exceed Expectatio A score of 0 would mea that expectatios were exactly met othig more, othig less. Scores above 0 idicate that the service exceeds the users expectatios, while scores below 0 idicate that the users expectatios are ot beig met. The latter would imply that a problem or misuderstadig should be idetified ad corrected. A recommedatio questio was also used to tap whether the users would recommed the service to others usig a scale of 1=Never, 2=Seldom, 3=Sometimes ad 4=Always: How ofte do you praise/recommed UM s admiistrative services to others? V. Costructio of Customer Satisfactio Idex I customer satisfactio research, two approaches are commoly used for calculatig the customer satisfactio idex (CSI): stated- importace ad derived- importace approaches. The stated- importace approach uses both stated importace ad performace scores i costructig the CSI, while the derived-importace approach uses regressio aalysis to derive betas for calculatig CSI (Chu 2002; Hill, et al., 2003). Both approaches have their stregth ad weakess. Cosiderig the advatage of usig the shorte versio of questioaires, the stability of statistical measure of the impact of attributes o overall customer satisfactio, ad the superior power of predictio ad explaatio of the derived-importace approach to stated-importace approach (Chu 2002), we adopt the derived- importace approach i this project. As illustrated i Table 1 below, regressio aalysis is first ru o overall satisfactio that is depedet o the attributes, the specific admiistrative uits i our case, to produce the relative impacts of each attributes. The beta score of each attribute (colum 1) is listed i colum 2. Secod, a beta weight of each attribute is calculated by the beta score divided by the sum of all beta scores (colum 3). Third, a mea score is computed for each attribute from the respodets evaluatio score of the performace of that attribute (colum 4). Fourth, a satisfactio weight is calculated by multiplyig the beta weight with the mea score (colum 5). Summig up the figures i colum 6 produces a overall customer satisfactio idex (colum 6). 10

11 Table 1.1 A illustratio of derived-importace approach to CSI (modelig results) Attribute (1) importace score (beta) (2) Beta weight (%) (3) mea score of satisfactio (4) satisfactio weight (5) AAO HRO FO CMO PUB Library ICTO IPR Faculty Office CSI(6) Total (82.26%) The CSI score varies from 0 to 100 by trasformig the origial sum of the satisfactio weight which rages from 0 to 10. Because of the customer respose ragig from 0 to 10, a score of 80 roughly traslates ito to a average customer respose of 8. Such approach is more stable tha simply lookig at the resposes to a sigle overall satisfactio questio as a idex is less affected whe a customer misuderstads oe questio. The satisfactio weights i colum 5 tell each attribute s relative cotributio to the total satisfactio idex score. For example, AAO receives a satisfactio weight of , idicatig that it is the most importat area amog others that affects the chage of the satisfactio idex. The attribute carryig a high beta weight with a low mea score of satisfactio meas is the oe eeds to be addressed ad studied carefully. 11