Pathfinder Evaluation Report (Final)

Transcription

1 For CP Community Only Pathfinder Evaluation Report (Final) Issue 4.0 : Draft Date : 30 th June 2009 Reference: Pathfinder Evaluation Report (Final) Abstract This report summarises progress, conclusions and current focus of BTs Pathfinder 1 activities in South Wales and aims to provide a simple executive summary of more detailed information contained within the Learning Review Log (which can be found in the Consult21 Secure Members area under Pathfinder) If CPs are impacted by BT s 21CN migrations and want to work with BT during Pathfinder, then they can choose from the following options: 1. Join our Pathfinder Experts group to collaborate closely with BT as part of Pathfinder learning. The group meets via audio every Thursday (details on Consult21 meeting Planner). The Learning Log can be found in the Consult21 Secure Members area under Pathfinder or 2. Simply keep advised of progress, tools available to CPs and recommended CP actions, by joining our awareness and education programme. Contact Helen.nurse@bt.com if you would like join this group. CPs can find more info on Pathfinder 1 in Consult21 definition briefing C21-IM-013 Pathfinder Definition & Approach and on how the Pathfinder Learning Process operates in Consult21briefing C21-IM-011 Pathfinder Learning Review Process Important Note: This document is supplied to you in confidence as part of the Consult21 process. It should not be circulated outside the Consult21 Communications Provider community. Document History Issue Date Author Comments Issue 3 30/06/09 B.Corner / Updated following Final Review J Tazzioli Issue 2 30/04/09 B.Corner / J Tazzioli Issue 1 15/01/09 B.Corner / J Tazzioli Updated following Cycles 2 & 3 Interim Review Initial report following Cycle 1 Interim Review British Telecommunications plc 2009

2 Contents 1 EXECUTIVE SUMMARY Pathfinder 1 Summary Cycle Cycles 2 & Cycle Status: CP Customer Experience Dashboard Cycle 1 Customer Experience Dashboard (as agreed with CPs) Cycles 2 & 3 Customer Experience Dashboard (as agreed with CPs) Cycle 4 Customer Experience Dashboard (as agreed with CPs at final review on 10/06/09) SUMMARY OF EVALUATION Migrations Cycle Cycles 2 & Cycle Post Migration Experience Fault & Event Analysis Network Key Performance Indicators (KPIs) End User Survey CONCLUSIONS Final Conclusions from Pathfinder END 18 2of18

3 1 Executive Summary 1.1 Pathfinder 1 Summary Phase one of the Pathfinder project (Pathfinder 1) was a trial specifically designed to start slowly with small numbers of end users migrated on different sites. These sites were selected to represent a cross section of rural, suburban and city centre conurbations. In addition both AXE and System X technologies were selected. The objectives were jointly to prove the technology on a range of sites, to prove the techniques and methodologies of the migration processes and also to prove the collaborative approach agreed with engaged CPs in terms of the reports and support tools. In a migration of this size and complexity some problems are inevitable and the slow ramp up approach was designed to minimise the impact of these problems whilst at the same time allowing as many issues as possible to be exposed and rectified at an early stage. As well as the way Pathfinder 1 was designed, the supporting processes and procedures put in place in order to help identify and rectify problems was another area that was thoroughly tested. This was also developed and improved as the phase progressed. Some early life failures were observed in the early phases and in particular a NDT problem affected a significant proportion of the migrated customer base. Although this was quickly highlighted and rectified, it did expose a requirement for a cross Line Of Business (LOB) collaborative approach to solving some of the more unusual type of problems. This led to the creation of the Enhanced Trial Management Team (ETMT) which was developed through Phase 1 and will continue to be a key element for future phases. Towards the latter stages, where higher volumes and more business oriented sites were migrated, a number of Customer Premises Equipment (CPE) problems were exposed. Overall a significant amount of learning was gained from this phase through a range of issues which were observed and fixes applied. Full details are contained in the Master Review Log which is held in the secure members area on the Consult21 web site. 1.2 Cycle 4 Cycle 4 of the Pathfinder 1 project covered the final two units of migration, these were Roath (two concentrators on the same site) and Lllandaff, Whitchurch, and Llanishen (three concentrators on separate sites). This gave an opportunity to prove the higher volume and multi-site capabilities whilst at the same time testing some of the improvements and fixes put in place to address the issues observed in previous phases. In previous cycles a number of failures were observed following migration of a card payment (PDQ) machines. These failures became more prevalent towards the end of Cycle 3 as larger, more business centric sites were migrated. In addition there have been some delays seen in meeting a number of the quality gate timings, this seems to generally have been due to delays with the premigration data checks prior to Quality Gate 3. Once the physical activities had begun they were all completed in short order, but the speed of the physical activities was insufficient to pull back enough time to correct the earlier delays. 3of18

4 1.3 Cycles 2 & 3 Cycles two and three of the Pathfinder 1 project covered further voice migrations at the remaining planned AXE10 sites (Ynysybwl, Treharris, Ferndale, Tonyrefail and Cearphilly) plus the initial System X sites of Penarth and Cardiff S. The two key aspects of these cycles was to begin to prove volume capabilities at the remaining AXE10 sites and identify any specific System X related problems by the early introduction in the programme of the two small System X sites. The post migration findings from these migrations have again vindicated the approach as being the correct one with a slow ramp up proving the appropriate course of action. A new network fault (relating to VPN calls) and some process issues were identified in the Cardiff S migration and the multi site and large volume migrations on the AXE10 sites proved the robustness and repeatability of the overall migration plans. Overall the majority of new faults were quickly rectified once root cause had been established and the process problems which have lead to delays to on the night activities continue to be worked through. The network itself has remained very stable with only a limited number of failures occurring, mainly relating to line card problems. It is fair to say that each migration within these two cycles has brought out further learning opportunities and these have been captured via the Pathfinder Learning Review Process. Steps taken to prevent reoccurrence of previous problems have been very successful as there has been very little evidence of repeated faults. 1.4 Cycle 1 This report covers migration of end users in Wick, Bedlinog and Nelson (Cycle 1) as part of BT s Pathfinder preparations for national voice network migration. At this point, we are broadly where we thought we would be, part way towards confirming a number of key items which are required to successfully roll out 21CN nationally: Migrations between July to November 3296 customer lines were successfully migrated in Wick, Bedlinog & Nelson and continue to have service delivered via 21CN. We have not used fallback processes although we have a high degree of confidence in the effectiveness of these processes as a result of their earlier live use and their continual exercise in our FOA network In-life Customer Experience - Some early life failures have been identified and resolved which showed up in our close monitoring of customer reported faults. Following their resolution fault numbers are currently running at or below figures expected on BTs PSTN network. Although the End User survey recorded a slight drop in overall levels of satisfaction, these figures remain impressively high, highlighting a noticeable improvement in speech quality by some end users. Network Performance - Monitoring of tactical network key performance indicators (KPIs) has revealed no significant concerns. The 21CN network continues to perform well having now handled over one and a half million live calls. The key to success in Pathfinder is BTs and CPs ability to learn from activities undertaken by identifying and taking appropriate corrective actions across the range of activities required to fulfil and assure end user service in 21CN WLR voice. A key indicator of whether this is working is whether: problems occur which we fail to address we currently have no issues in this category. 4of18

5 that problems continue to re-occur (from the same root cause) after they are believed to have been resolved we currently have no issues in this category. CPs participating in this process have given positive feedback through Consult21 on the level of transparency within the review process and BT s responsiveness to issues raised. 1.5 Status: CP Customer Experience Dashboard The dashboard reflects our CP customers confidence to proceed to the Pathfinder migrations in Penarth based on the levels of issues experiences to date and BT s response to them from a technical and procedural perspective. This dashboard is reviewed fortnightly with CPs Cycle 1 Customer Experience Dashboard (as agreed with CPs) Experience Component Status Status/Key Issues Migration Upgrade Feature Harmonisation & expected changes Post Migration/In-life performance CPE compatibility Interoperability Issues Pre-migration experience Managing incidents The on the night migration process has completed without any incidents on each of the five migrations carried out in Cycle 1 (Wick, 1 Bedlinog 1, Wick Final, Bedlinog Final and Nelson). All completed within the agreed timeframe PEW process implemented for upgrades for CPs Although star services have been harmonised, e.g. call barring, these services are used by a relatively small subset of customers. Cycle 2 will see changes to call waiting tone which will have a bigger impact as they are expected to be noticed by most customers Live validation of fixes for NDT issues Although a small number of compatibility issues have occurred, these were not unexpected given the volume and variety of end user equipment. However, the green status reflects the confidence in the high level of testing carried in the labs prior to migration No broadband or LLU faults have been identified as related to the migration activity. In addition, there have been no faults raised associated with how the 21CN platform operates with the legacy platform or with calls between different CP s networks. Confirmation of operational processes for pre-migration scheduling CP visibility of notifications and levels at which incidents declared when volumes are small For the remaining components at Amber status, actions have been agreed and are being tracked at the regular weekly audios with participating CPs. Where the actions have been completed, we are waiting validation of the solution. 5of18

6 1.5.2 Cycles 2 & 3 Customer Experience Dashboard (as agreed with CPs) Experience Component Status Status/Key Issues Migration Upgrade Feature Harmonisation & expected changes Post Migration/In-life performance CPE compatibility Specifically for the issues experienced with the System X migrations in Penarth and Cardiff S. Looking to final migrations in Cycle 4 to demonstrate the ability to complete a migration within the agreed timeframe (i.e. all quality gates issues on time) and without impacting legacy lines. Note Green for AXE10 migrations. Awaiting completion of the planned upgrades following completion of Pathfinder 1 to validate the actions already implemented. No issues. Amber for Consumers due to secondary resilience problem at Tonyrefail. Need to demonstrate successful resolution (in-flight VAG rearrangement work) without significant impact. Due to known incompatibility with some PDQ machines. Need to understand root cause and implement a fix to return to Green. Mitigation in place for remaining migrations in Cycle 4 identify Directory Numbers associated with these payment machines and groom prior to migration. Note Green for Consumers. Interoperability Issues Due to problems with ISDN and legacy lines experienced on System X. Note Green for Consumers. Pre-migration experience Managing incidents Due to some minor issues with detailed operational process as well as Data Set 51. A concern was noted with regard to the length of time required for the issue of white labelled communications. However the status is Green. For the remaining components at Amber status, actions have been agreed and are being tracked at the regular weekly audios with participating CPs. Where the actions have been completed, we are waiting validation of the solution. 6of18

7 1.5.3 Cycle 4 Customer Experience Dashboard (as agreed with CPs at final review on 10/06/09) Experience Component Status Status/Key Issues Migration Upgrade Feature Harmonisation & expected changes Post Migration/In-life performance CPE compatibility Interoperability Issues Pre-migration experience Managing incidents Need to be able to demonstrate that the Quality Gate 4 midnight target can be achieved. Awaiting completion of the planned upgrades after Pathfinder 1 to validate the actions already implemented and prove the fixes included in the upgrades. A concern was noted regarding the need to complete Root Cause Analysis for the Direct Connect Issue. However, it was agreed that the status is now Green. No issues noted. Status is now Green as a fix has been identified and will be included in the upgrades. Previously the Amber status was due to ISDN problems which were attributed to data issues which have now been resolved. Agreed status to be Green. A concern was noted with regard to the minimum notice period for migrations and Data Set 51. However, it was agreed that this will be tracked ans an issue and that the status is Green. No issues noted. For the remaining components at Amber status, actions have been agreed and will be tracked on an Interim Review Log which will be reviewed at the regular weekly audios with participating CPs. Please note that these audios will continue even though the Pathfinder 1 migrations are complete and will also oversee the planned upgrade activity. It is also proposed that the learning process be continued to support the Pathfinder Volume POTS migrations. 2 Summary of Evaluation The three key areas of the overall evaluation, fault analysis, customer surveys and network key performance indicators (KPIs) have all provided valuable information. They have informed our learning and helped to identify a number of early life failures. These have been successfully resolved and actions taken to avoid reoccurrence. Overall the evaluation has proved to be largely positive and the issues that have arisen have been of the type that would be predicted in a delivery of this nature and complexity. The results obtained have generally confirmed that the areas under observation are the right ones to correctly inform 7of18

8 our learning, and some modification and improvement to the measurement and analysis techniques have been carried out as we have progressed through this cycle of the roll out. 2.1 Migrations Cycle 1 Migrations have gone well with 3296 lines now successfully migrated to 21CN and continue to have service successfully delivered via 21CN. There were six separate migrations in total during this cycle and all of the on the night activities required to transfer the customers were carried out successfully with no major incidents occurring. The average outage for incoming and outgoing calls was 90 seconds which gives us confidence we will be able to deliver migration outages within the times specified on our Customer Experience Impact matrix. Feedback from customers via survey results indicates that the impact of the on the night activities was virtually unnoticeable to the vast majority of end users. Migration processes and procedures have been found to be robust and thorough and have required only minor additions to include additional checks and balances. During Nelson, we have used our TCP shoe migration methodology which we will use for national migration. This has given us a high degree of confidence to complete migration of the remaining lines for Pathfinder Cycles 2 & 3 Cycles 2 and 3 saw a significant ramp up of both the number of migrations and end user volumes with a total of 47,334 lines successfully migrated to 21CN following completion of the Caerphilly transfer on 7 th April. This figure includes approximately 35,880 working customer lines. Cycle 2 started with the migration of the first System X exchange and the first remote unit at Penarth and the introduction of a second call server in Bristol. This was followed by successful migrations at Ynysybwl and Treharris before finishing with Cardiff S, the second and larger System X. The Ferndale and Tonyrefail migrations at the start of Cycle 3 saw the first time two concentrators were successfully migrated simultaneously. Cycle 3 ended with Caerphilly, which contained the largest batch sizes migrated so far. During these two cycles, the migration processes, tools and techniques have proved to be successful. The duration of physical migration has been consistently within target times and fully acceptable in terms of speed of execution. There are however, a number of areas where delays have been incurred resulting in late achievement of quality gate target times. The cause of the delays has not been consistent and has had various root causes. However at no time has a migration been at risk of non completion. Specifically, on System X migrations, there have been delays relating to data transfer and the supporting tools. Work is in progress to address these issues by examining the running order of activities and any enhancements which may be available to help systems run faster Cycle 4 Cycle 4 saw the addition of some multi-site units of transfer, initially with Roath D and E, representing two separate concentrators on the same site (approximately 20,672 lines). This was followed by Llandaff, Llanishen and Whitchurch (approximately 18,560 lines), three concentrators at three separate locations. Cycle 4 therefore increased both the volume and complexity simultaneously. Completion of these migrations 8of18

9 saw the final total number of lines successfully migrated rise to 85,582 which includes 64,023 working customer lines. There were no significant issues with the physical on the night activities and in fact on the Llandaff, Llanishen and Whitchurch migration which was probably the most complex, the actual outage times were reduced to less than 1 minute 30 seconds. However, some of the delays seen in previous System X migrations were repeated in this cycle. In particular, Quality Gate 3 was delayed by between 1 and 2 hours in both migrations. Root cause was due to the necessary checking of the data to ensure correct migration of all nominated lines. Work is ongoing to address this issue, however it is now felt that migrating a whole DLE will simplify the operation and result in less checks being required rather than more. BT will continue to work with CPs to demonstrate improvements in the capability in this area. 2.2 Post Migration Experience Fault & Event Analysis Cycle 1 From the first migration in Wick on 17 th July 2008, fault queues have been monitored regularly and any customer reported faults logged for analysis. No customer reported faults were made prior to September Two unexpected trends were identified which have been linked back to specific problems, a summary of these is shown in the table below. Number of faults Symptom Root cause Current situation >100 Reports of no dial tone, bell not ringing, unable to receive calls Resource allocation problem Fix in place and no further occurrences 6 No dial tone or locked lines. Configuration issue Fix waiting to be loaded The first problem was discovered following the Nelson migration and has proved to be volume related, the symptoms of this problem were a temporary loss of dial tone for some customers. The loss of dial tone could in some cases extend for several hours and led to a large number of customer reported faults. The first customer reported faults related to this problem began to appear on 11 th November and a trend was identified on 14 th November. Root cause was quickly established and a fix was supplied by the vendor and following test was loaded on 29 th November. In total over a hundred customer reported faults were generated as a result of this problem and these have been excluded from the overall analysis for purposes of clarity. Detailed investigation of the no dial tone (NDT) faults lead to a the discovery of a second trend of reported faults which showed similar symptoms the problem described above. In this case once the end user experienced the problem it remained until cleared by an engineering activity in the exchange. Faults of this type were much fewer than the previous NDT problem numbering only 6 in total. The more persistent nature of these faults helped in identifying the root cause and a fix has been supplied and is ready to load at an appropriate time. 9of18

10 Details of all faults received between 1 st September and 31 st December were collected and analysed on an ongoing basis in order to identify any further trends or 21CN related issues. Following the analysis it was found that the majority of faults received were attributable to the external network or other factors not related to 21CN, this is a normal situation and in line with expectations. Of those faults that were related to 21CN, two broad categories were identified; firstly there were a number of faults that were attributable to carrying out the work itself, this would typically be due to human error when carrying out the engineering activities involved in a project of this nature. The actual number falling into this category was very low and demonstrates a high degree of care in executing the tasks. The second broad category was faults that were directly attributable to the 21CN network and these can be split down further. Of the faults directly attributable to the 21CN network these can divided into; CPE related, hardware faults, and configuration related. The CPE related faults showed no significant trend as they affected a range of CPE all of which had previously been identified as at risk. Faults of this type were also low in number. The hardware faults of which there were only 2 are the type of fault that is currently seen in the legacy network and can be expected to appear occasionally, no specific trend was identified. Prior to Wick and Bedlinog final migrations we successfully completed network upgrades whilst it was supporting small volumes of live end users. This helped us refine still further our upgrade processes by identifying a requirement to include additional code to reset any new memory areas being brought into service. Spurious data which can exist in these new memory areas can cause problems such as locked lines. During the course of Cycle 1 three lines were regressed as a precautionary measure and to ensure a rapid return of service to the affected end users. One case is CPE related and further testing is required on that specific item, the other two have had root cause established and await validation of the fix. All regressed lines will be re-migrated under controlled conditions at a point yet to be decided. Category No. Details Total 47 Not related to 21CN 27 External, customer premises, customer equipment etc. Total faults related to 20 21CN Caused by 21CN work 7 Includes 3 CSS errors, 1 number table omission. Comms related 1 Customer confused on operation of OCB facility Genuine 21CN faults 6 4 CPE, 2 faulty port Early life failure (NDT) faults 6 Second trend identified and root cause established Cycles 2 & 3 Volumes correct as of 31 st December 2008 The same approach as above was applied to analyse the faults raised during Cycles 2 and 3. However, the genuine 21CN faults fell into two different broad categories: CPE issues, in particular payment machines and one System X related network issue (VPN calls failing). Category Cycle 2 Cycle 3 Details No. No. Total Not related to 21CN External, customer premises, customer equipment, customer misoperation, customer queries etc. Total faults related to 21CN of18

11 Caused by 21CN work Data errors, process errors etc Genuine 21CN faults key trends identified :- CPE, early life failure (VPN) Other 1 12 NDT, hanging devices *Note faults reported in the Cycle 2 and 3 migrated sites between 22/01/09 and 31/03/09 Volumes correct as of 31 st March 2009 Monitoring of any faults raised on the concentrators migrated during Cycle 1, i.e. Wick, Bedlinog and Nelson continued during Cycles 2 and 3. In these sites, the fault rate was extremely low, demonstrating the ongoing stable nature of the 21CN platform. For completeness the same analysis was applied to these faults and the results are recorded below:- Category Cycle 2 & 3 No. Details Total 82 Not related to 21CN 71 External, customer premises, customer equipment, customer misoperation, customer queries etc. Total faults related to 21CN 11 Caused by 21CN work 4 Data errors, process errors etc Genuine 21CN faults 1 2 key trends identified :- CPE, early life failure (VPN) Other 6 NDT, hanging devices *Note faults reported in the Cycle 1 migrated sites between 22/01/09 and 31/03/09 Volumes correct as of 31 st March Cycle 4 During Cycle 4 no new trends emerged as the platform fault levels continued to stabilise. At this point the full impact of the payment machine problem seen at a low level in Cycle 3 was experienced as the fault levels related to this issue peaked following the Caerphilly migration and to a similar extent, following Roath. Although attempts were made to identify and groom the potential problematic payment machines prior to the Roath migration, the process applied did not identify all the affected machines and they largely emerged as faults during Cycle 4. However, this process was further refined prior to the Llandaff, Llanishen and Whitchurch migration and a higher success rate was achieved which resulted in a significant decline in the number of faults in this category at this point. As this cycle concludes the evaluation for Pathfinder 1, a slightly different approach was adopted during the analysis of the faults. In the previous interim reviews, the analysis focussed on the faults raised on the sites migrated during the cycle(s) under review. However, to conclude the evaluation process, the analysis has focussed on all faults raised across the entire Pathfinder 1 footprint between 1 st January 2009 and 31 st May (Note: for analysis prior to this please refer to Section above). Therefore the figures in the table below represent the total number of faults reported across all Pathfinder 1 migrated sites, for each of the learning cycles between January 2009 and May 2009 and as such, the figures reported at the last review (see Section above) have been merged with those for Cycle 4. They do not represent the incremental change since the last review at the end of Cycle 3. 11of18

12 As Cycle 4 also saw the most significant increase in volumes, the number of end customers migrated has been included to add context to the fault numbers reported. Cycle 2, 3 & 4 Total Fault Analysis Category Cycle 2 Cycle 3 Cycle 4 Details (all faults) (all faults) (all faults) Customers on CN Total Faults Not related to 21CN External, customer premises, customer equipment, customer misoperation, customer queries etc. Total faults related to 21CN Caused by 21CN work Data errors, process errors etc Genuine 21CN faults key trends identified :- CPE, early life failure (VPN) Other NDT, hanging devices *Note faults reported across entire Pathfinder footprint between 01/01/09 and 31/05/09 Volumes correct as of 31 st May Enhanced Trial Monitoring Team (ETMT) As a result of fault analysis carried out during the first cycle it became apparent that a dedicated team was required to ensure rapid resolution of unusual customer faults relating to the migration activities. The ETMT was launched prior to the Treharris migration and brings key cross Line-Of-Business (LOB) operational teams together daily to bring focus to any unusual issues occurring in order to protect the customer experience. The ETMT also convenes at weekends for a light-touch check for any unusual activity. During Cycle 4, the ETMT continued to meet daily with full commitment from all operational teams involved to complement Business As Usual (BAU) processes to ensure that any negative impact on customer experience was kept to a minimum. In particular, with the payment machines problem, the ETMT brought the necessary cross LOB collaboration and focus to BAU processes to ensure rapid implementation of the agreed tactical solution i.e. regression, as soon as the problem was identified by the front line teams Strategic Enhanced Trial Monitoring Team Following a review of the valuable role provided by the ETMT during Pathfinder 1, it has been agreed that a Strategic ETMT will continue to convene during the period between Pathfinder 1 and Pathfinder Volume POTS. The forum will meet weekly to monitor the network for evidence of any unusual problems or potential trends in the Pathfinder 1 migrated sites. In addition, some of the ETMT governance team will meet daily to ensure the continued effective operation of the BAU processes and to support the weekly Strategic ETMT discussions. It is also proposed that the daily ETMT, as successfully operated during Pathfinder 1, will be reconvened to oversee the planned Call Server and MSAN upgrade activities later this year. This forum can 12of18

13 also be reconvened at any time on an ad-hoc basis should the Strategic ETMT governance team deem it necessary Network Key Performance Indicators (KPIs) Cycle 1 BT and CPs have agreed a set of Network Performance Indicators which have been reviewed weekly during Pathfinder activity to date. These measures are currently derived from test calls as live traffic volumes are too small until sufficient end users have been migrated. These are:- Mean Opinion Score Target 4 (99.9%ile) Consistently above 4 (MOS-LQO) 1 Round Trip Delay Target 70ms (99%ile) Not yet available see note below IP Packet Loss Ratio Target (99.9%ile) Consistently within threshold Jitter Difference Target (99.9%ile) Consistently within threshold Post Dialling Delay Target 600ms (99%ile) Consistently within threshold for 21CN to 20CN as this is currently the most common route 21CN to 21CN calls are much lower, typically in the order of 330ms Note: Round Trip Delay - figures have not been reported as a result of an outstanding issue with regard to the measurement accuracy. Unfortunately this issue has proved to be more complex than originally anticipated and a fix has not yet been supplied. However, as we progress into Pathfinder 1, we expect to be generating sufficient call volumes to make strategic measures viable. Round Trip Delay will be reported via the strategic route. Overall, the network has been performing within the agreed thresholds and no issues have been raised Cycles 2 & 3 No significant changes have been noted as the network remains stable and continues to perform within the agreed thresholds detailed above. On a small number of occasions (approximately 4% of all measurements) where results outside these threshold limits were recorded, investigations concluded that this was due to work taking place with the test equipment, resulting in invalid data being recorded. Note: Round Trip Delay the problems with the accuracy have proved to be complex to fix and a decision was made to focus on the strategic solution which does not involve the use of the measuring equipment causing the problems with the tactical solution. The strategic solution is in the final stages prior to completion. Round Trip Delay will therefore not be reported for the remainder of Pathfinder 1. 1 the international standard of speech quality, in this case MOS-LQO rated on a scale of 1-5 where a score above 4 is considered to be very good 13of18

14 Overall, the network has continued to perform within the agreed thresholds and no issues have been raised Cycle 4 The network continues to perform within the agreed thresholds with no change to report. As in Cycle 3, results outside these limits were recorded for approximately 4% of all measurements and were also largely due to problems with the test equipment. It is expected that the strategic solution will be used for Pathfinder Volume POTS but it is intended to have a period of overlap of approximately 1 month for comparison purposes. This will also see the introduction of the strategic Round Trip Delay measures. Overall, the network has continued to perform within the agreed thresholds and no issues have been raised End User Survey Cycle 1 As part of the review process end user surveys were undertaken in Wick, Bedlinog and Nelson both before and after migration to loosely compare satisfaction levels and identify any perceptual change between legacy and 21CN service. The pre and post migration surveys were undertaken with different end users after completion of the 7 day stability period. The opportunity was also taken to identify any red flag issues for further investigation and the participants were offered a call back from their CP to further discuss any issues. These surveys targeted SME and consumer end users of two large CPs, and will be repeated in Penarth and Cardiff S in Cycle 2. which will complement the largely rural/suburban footprint with a more urban one. The results of the before and after comparison of satisfaction levels gathered to date are shown in the following graph. Satisfied (%) 90% 89% 91% 92% 97% 90% 95% 90% Wick Bedlinog Nelson Total % Customers Satisfied Trends/Analysis A problem with the network became evident after the Nelson migration and the symptoms of this were that some customers experienced a temporary loss of dial tone causing them to report a fault. It is believed that other customers may have experienced the effect but did not report it as the fault could clear at random. This problem occurred during the period which the post migration surveys were conducted at Nelson, and 14of18

15 this had a significant impact on customer experience. Monitoring of all customer reported faults showed that more than 5% of the Nelson end users reported faults of this type during the time the issue was in evidence. Surveys completed at Nelson showed a statistically significant reduction (7%) in levels of end user satisfaction post 21CN migration. The majority of dissatisfied respondents did not give any verbatim comment relating to the reasons for this dissatisfaction, of those that did the verbatim comments mainly related to problems making or receiving calls or interference or crackling on the line. This result remains valid for Nelson end users even when the results are normalised to remove the impact of the known network problem causing a loss of dial tone. Wick & Bedlinog End Users did not receive a Switched-On door drop prior to 21CN migration, however in line with standard processes Nelson did. One possibility is that expectations have been set by contacting the end users to inform them of the migration, and they see no significant change when the changeover takes place. 10% of respondents indicated that they had reported a fault to their CP and in all cases these can be cross referred to our / CP s records. Analysis of these reported faults indicates that around two thirds of these related directly to the known network problem and one respondent reported problems with call barring which was later found to relate to a misunderstanding of the feature harmonisation. One respondent noted that They have taken some services away which could indicate that they were unaware of the features harmonisation which takes place due to the migration. 10% of customers who had noticed a change told us that they had experienced an improvement in voice quality in particular that the line now appears to be clearer. Overall levels of satisfaction, although showing a reduction, remain very high Cycles 2 & 3 During these cycles additional customer surveys were carried out at Penarth and Cardiff S, these two concentrators were included in order to capture any differences between migrations carried out at AXE10 sites and those at System X sites. Due to the very small customer base at Penarth it was not possible to carry out pre-migration surveys and only a limited number of post migration surveys were conducted in order to give an early warning of any issues that had not previously come to light. Results from these additional sites did not reveal any problems and were very similar to results obtained at previous locations. Again the overall level of satisfaction remained very high and these additional numbers helped to reinforce the overall view that attempts made to minimise impact to end users has been very successful. 15of18

16 Pre migration Satisfied (%) 94% 93% 93% Post Migration Penarth Cardiff S % Customers Satisfied Cycle 4 No further customer survey work was carried out in Cycle 4. This work completed in Cycle 2/3 with the post migration surveys in Cardiff. 3 Conclusions Fault Analysis Cycle 1 We found that in some cases the business as usual processes were designed to perform a function, e.g. repair a fault, rather than to provide the detailed information that we required to help us understand what was happening. In a very few cases the BAU processes failed as they were not designed to fully consider the impact of migration on the reported fault. In order to better capture the required data and to ensure customer experience was maintained, a Complex Faults Team was formed to which any fault could be referred for resolution or further investigation. Although only used occasionally, it is intended for this team to remain available throughout the remaining migrations of Pathfinder 1. During the 7 day stability period 16of18

17 following each migration, this team will meet daily to review any reported faults that have occurred. This will form part of the stability period process and will be managed by the Migration Control Centre (MCC). Fault Analysis Cycles 2 & 3 During these cycles the fault levels appear to be at or lower than would have been expected on 20C. In addition, there have been no unresolved complex fault trends nor have there been any repeats of the network problems experienced in the previous cycle. The fault trends that have been identified in these cycles have generally been quickly identified and resolved with the exception of the payment machine problem for which mitigation is currently being developed. The formation of the ETMT has contributed greatly to speeding up the resolution of faults trends identified. Fault Analysis Cycle 4 Following the focus provided by the ETMT in the previous cycle, Cycle 4 saw BAU fault processes maturing sufficiently to deal with the majority of faults raised. These remained at a relatively low level and no new trends were identified. The ETMT continued to take a watching brief, intervening only when customer experience was at risk. End User Survey Cycle 1 A more detailed report of the end user survey results received to date can be found in the Consult21 Secure Members area. It has not been possible to fully explain the reduction in overall satisfaction levels from the results returned. This is because of those respondents stating that they were overall dissatisfied or very dissatisfied, very few of them have supplied verbatim comments or stated that they have noticed any differences, had any problems or reported a fault. A specific question will be included in future surveys in order to qualify the reasons for any respondent stating they are dissatisfied or very dissatisfied. It is also planned to follow up any verbatim comments that are unclear or indicate some confusion in order to better understand the degree to which the information provided to end users has been understood and retained. End User Survey Cycles 2 & 3 There were no significant differences seen in the survey results on these cycles. End User Survey Cycle 4 There were no surveys carried out in Cycle 4. CPE Compatibility Cycle 1 Although a large amount of customer premises equipment has been tested for compatibility with 21CN, the CPE related faults that were recorded related to equipment which had not been included in testing to date. In each case arrangements have been made to obtain from the manufacture a duplicate piece of equipment, or the actual item displaying the problem in order for further tests to be carried out at the labs. This information will be used to update the detailed CPE testing database and inform our ability to judge the level of incompatibility problems that are likely to occur in national roll out. CPE Compatibility Cycles 2 & 3 During these cycles there have been very little CPE issues with the exception of the ongoing problem related to payment machines which has become more prevalent with rising volumes. The previous concerns regarding call display have not materialised and there has been no significant increase in the volume of 17of18

18 ATM issues. This has helped to increase confidence going forward that there will not be large volumes of customer complaints as the migration numbers ramp up through the remainder of the schedule. CPE Compatibility Cycle 4 The root cause of the payment machine problem was confirmed during this cycle and a decision taken to groom suspect lines prior to migration. This was only partially successful in the Roath migration as further refinement was required in order to fully identify all PDQ machines potentially at risk. For the final migration at Llandaff, Llanishen and Whitchurch all PDQ machines were successfully identified and groomed off. One new incompatibility was identified related to an older model of a branded Smart Phone. This has led to further investigation into this particular manufacturer s portfolio and investigations are ongoing in this area. 3.1 Final Conclusions from Pathfinder 1 The objectives of Pathfinder 1, as outlined in the Executive Summary, have been fully met in that all migrations were successfully completed as planned and the tools, techniques and processes as originally devised were proved to be effective. There were some changes made during the journey in order to improve focus in certain areas. These were captured during the learning process which itself evolved throughout this phase of the project and was also proved to be effective. The slow-start approach was universally agreed to be the correct one and whilst there were some minor disruptions to customer experience, the customer surveys conducted show that these were largely unnoticed which endorses the decision to carry out physical migration activities out-of-hours. Pathfinder 1 can be considered to be a great success largely attributed to the collaborative approach adopted by both BT and CPs, working in an open and honest manner, applying the learning captured to achieve a common goal. We have proved that we have both the technical and operational capabilities to carry out complex migrations which can be carried forward into larger scale transfers. END 18of18