The BBC s management of its Digital Media Initiative

Transcription

1 The BBC s management of its Digita Media Initiative Report by the Comptroer and Auditor Genera presented to the BBC Trust s Finance and Compiance Committee, 13 January 2011

2 Department for Cuture, Media and Sport The BBC s management of its Digita Media Initiative Report by the Comptroer and Auditor Genera presented to the BBC Trust s Finance and Compiance Committee Presented to Pariament by the Secretary of State for Cuture, Oympics, Media and Sport by Command of Her Majesty February 2011

3 BBC 2011 The text of this document may be reproduced free of charge in any format or medium providing that it is reproduced accuratey and not in a miseading context. The materia must be acknowedged as BBC copyright and the document tite specified. Where third party materia has been identified, permission from the respective copyright hoder must be sought.

4 The BBC s management of its Digita Media Initiative BBC Trust Response to the Nationa Audit Office Vaue for Money study The BBC s management of its Digita Media Initiative Background to this study As the BBC s governing body, the BBC Trust acts to get the best out of the BBC for icence fee payers. In part this means ensuring that the BBC continues to provide exceent vaue for money. We commission Vaue for Money investigations into specific areas of BBC activity to hep ensure that icence fee payers are getting the best possibe return on every pound of their icence fee. Each year the Trust commissions a series of Vaue for Money reviews after discussing its programme with the Comptroer and Auditor Genera the head of the Nationa Audit Office (NAO). The reviews are undertaken by the NAO or other externa agencies, reporting to the Trust. The Trust carefuy examines the findings from each study and requires a fu response and action pan from the BBC Executive expaining how it wi take forward the specific recommendations made. This study, commissioned by the Trust s Finance & Compiance Committee on behaf of the Trust and undertaken by the NAO, ooked at the BBC s management of its Digita Media Initiative (DMI). The DMI is a major technoogy project that is intended to deiver efficiencies whie aso improving creativity and productivity. As with a significant projects, the Trust is under an obigation to ensure that the project represents, and deivers, vaue for money. In view of the importance of this project to the BBC, and the probems that the Trust was aware of concerning its progress, the Trust was keen for this Vaue for Money study to take pace to identify any essons that coud be earned. The Trust thanks the NAO for undertaking this review and for the report s concusions and recommendations. Our views on the NAO s findings We have considered carefuy the NAO s report and we accept its key findings. The Trust considers that the DMI is a cutting edge project that wi improve the way the BBC operates, eading to greater efficiencies of time, efficiencies of cost and by providing scope to increase creativity. By its nature, this was aways a high risk project as there was no off the shef technoogy avaiabe. However, we remain convinced that with the Saford Quays and Broadcasting House deveopments coming on ine in 2012 it was right that we deveoped the DMI rather than insta soon-to-be-obsoete technoogy. The nonfinancia benefits of the DMI are significant, and the financia benefits of the DMI, whie important, are ony part of the picture.

5 The Trust agrees with the NAO that the first phase of the project did not proceed according to pan. The BBC awarded the DMI contract to Siemens, the BBC s technoogy framework suppier. Framework agreements are intended to provide efficiencies, in part by reducing procurement costs and times. However, with hindsight, the decision to award the contract to Siemens did not ead to the panned outcome and resuted in a deay of 21 months to the project and hence a deay in the improved working practices. It is difficut to concude whether a more detaied assessment at the time and/or a competitive procurement process woud have reduced the risk of this occurrence, as the inherent nature of IT projects means that they are compex and at high risk of cost overruns throughout both the pubic and private sectors. However, we accept that a detaied assessment of procurement decisions is a key area in project approva and we wi require the BBC management to take note of the NAO comments in this respect when considering future business cases. The Trust notes, however, that the contractua agreement with Siemens did protect the BBC from direct financia oss (athough not from oss due to deay). Since bringing the project in-house, the Trust has been satisfied with the progress of the project. We note the NAO s recognition that the BBC has now started to deiver the DMI system and that users have been positive about the eements deivered. In the context of the DMI being a compex and cutting edge IT project, the Trust considers this is something of which the BBC shoud be proud. However, there is sti some way to go to compete the project and we wi be continuing to monitor progress. Simiary, it is encouraging that the NAO has reported significant improvements in the way in which the BBC management has presented the more recent business case, particuary in respect of the anticipated benefits. We acknowedge the NAO s assistance in recommending improvements to the process of approva in its past studies for the BBC. The Trust wi now require the BBC Executive to provide a comprehensive action pan outining how it intends to impement the NAO s recommendations. Where impementation is aready in progress, we wi require fu detais on the actions being undertaken. We wi aso require the Executive to consider how these points may have reevance for other BBC projects. Specific Trust recommendation The NAO makes a specific recommendation to the Trust concerning the referra threshods. We consider that the referra threshods as currenty drafted are cear and have worked we to date, ensuring an appropriate baance such that the Trust is invoved in strategic rather than the more routine operationa decisions. However, we note the NAO s comments that these are narrow financia threshods. We agree that we shoud review these referra criteria in ight of the NAO s comments and consider expanding these to incude significant changes to the cost-benefit of a project. The Trust wi consider how best to impement this recommendation.

6 With specific regard to the decision to continue the project in-house in Juy 2009, the Trust is mindfu of the important division between the Executive s operationa responsibiities and the Trust s oversight responsibiities. The Executive Board and the Trust were kept informed of the status of the project through ora briefings, and both were satisfied that the BBC Finance Committee (which incudes the BBC Director-Genera, the BBC CFO and the BBC COO) was an appropriate forum in which to take the decision as to how to continue the DMI given that the overa financia cost to the BBC was not increasing and the aims of the project remained as previousy stated. However, the proposa to increase the scope of the project was righty brought before the Trust in mid Action pan The Trust has discussed the recommendations with the Executive and considered its response, which is pubished in this document. The Trust is satisfied that the Executive s proposed actions are an appropriate initia response to the NAO s findings, but wi require the Executive to provide a more detaied action pan for Trust review. Progress against this pan wi be reported to the Trust. BBC Trust January 2011

7 The BBC s management of its Digita Media Initiative BBC Executive Response to the Nationa Audit Office Vaue for Money study The BBC s management of its Digita Media Initiative The BBC Executive wecomes this study, in which the Nationa Audit Office (NAO) reviewed our management of the Digita Media Initiative (DMI). DMI is a strategic investment in infrastructure, peope and production processes that aims to support the reaisation of the BBC s creative vision in the most cost effective manner possibe. DMI wi seek to transform the way the BBC makes content for its audiences. It wi create a digita content production system that enabes easy manipuation of content, effective sharing across ocations, desk-top editing and access to the archive. The project covers hardware, software and running costs for eight years and repaces disparate egacy systems. It wi provide a foundation not ony for content fows across the BBC, but across the UK-wide sector as a whoe. The media industry is moving towards this way of working and the BBC needs to keep pace with current digita technoogy, so investment in technoogy of this type is a necessary step in the evoution of programme making. By investing in DMI, the BBC is making a significant commitment to strategic and operationa requirements, incuding: The BBC s strategy, Putting Quaity First (incuding re-investing savings in quaity content, deveoping partnerships and strengthening digita pubic space). A networked BBC buit around sustainabe centres of production across the UK. A pan-bbc digita archive. An end-to-end digita workfow and a soution for managing digita assets that woud aso hep estabish common standards in metadata in partnership with the industry. In February 2008, the BBC entered into an agreement with Siemens, its strategic technoogy partner, for the provision of deivery of the DMI programme. This agreement was a ca-off against the Technoogy Framework Contract (TFC) which was signed by the BBC and Siemens in September This is a 10-year outsourcing contract designed to deiver the BBC s technoogy requirements in an efficient and cost effective manner. The deveopment of digita production technoogy was envisaged to be a key part of this reationship when the BBC seected Siemens as the most suitabe strategic technoogy partner.

8 In summer 2009, after a number of deays to deivery of the programme, the BBC and Siemens reached a mutua agreement that aowed the BBC to take back ownership for the deivery of the DMI programme. Foowing evauation of the avaiabe options and further financia approva, the BBC is now deivering DMI in-house, using interna staff, contractors and a range of third party suppiers. The BBC Executive notes the recognition by the NAO that, despite the initia difficuties, the deivery of the programme is now we underway. A number of reeases have been successfuy deivered and initia feedback from users has been very positive. Response to the NAO s recommendations The BBC Executive has reviewed the NAO s recommendations, and notes that many are aready in progress or have aready been impemented in the atest phases of this project. Recommendation a The BBC did not have an up-to-date assessment of its contractor s capacity and capabiity to deiver the programme. The BBC assessed Siemens during a competitive procurement process in 2004 when it entered into a ten-and-a-haf year Technoogy Framework Agreement with Siemens as the BBC s strategic partner responsibe for its information systems. The BBC did not have to have a competitive procurement for the Digita Media Initiative as it coud appoint Siemens to deiver the programme in a straightforward and quick procurement under that Framework. However, to provide assurance the BBC is not making procurement decisions on sub-optima grounds, it shoud demonstrate in investment cases why its procurement route is ikey to offer the best vaue for money. The BBC fuy agrees with the need to ensure that its contractors are very thoroughy seected and assessed through a process which invoves open competitions when this is required or appropriate. The BBC Finance guideines set out that a procurements shoud foow best practice procedures to ensure vaue for money is achieved by seeking eary advice from our Procurement Department. Before its submission to the BBC Finance Committee, any investment case invoving procurement aspects needs to be approved by the Director of Procurement, who is responsibe for confirming, inter aia, that: The procurement (sourcing) strategy is appropriate for the particuar tender and fits to the overa outsourcing strategy (this incudes ensuring the appropriate evauation incuding seection and award methodoogy). The procurement governance mode and indicative timescaes are appropriate and reaistic for the services/goods being procured. The commercia dea ensures best vaue on a whoe ife basis.

9 When the BBC decided to use its ongterm strategic technoogy partner Siemens to deiver DMI it did so having taken fu account of the BBC financia guideines and procurement procedures. The principa reasons for seecting Siemens in February 2008 were: Siemens had been seected as the BBC s strategic partner to provide better vaue for money, and to better meet its future technoogy needs. In signing the DMI ca-off with them, the BBC compied with a recommendation made by the Committee of Pubic Accounts (PAC) that technoogy services and commodities avaiabe through the technoogy framework contract are procured through the contract where they are avaiabe and offer better vaue for money than procurement outside the contract. 1 The BBC ooked at other deivery partners and options but coud not identify any better deivery mode than using Siemens. The BBC estimated that procuring the programme as a ca-off contract within the TFC and thereby avoiding open procurement coud save the BBC six to nine months and costs of up to 3 miion. Externa consutants were asked to advise on whether the Siemens price was competitive. They concuded that open competition coud have generated a better price, but that any savings woud ikey be offset by the cost of having to integrate the new system with the Siemensrun BBC network, and deays in securing benefits. The BBC had fuy tested Siemens capabiity and capacity to deiver through assessments made as part of the TFC procurement. Siemens had a good understanding of BBC systems and experience of deivering projects with the BBC. The BBC had worked with Siemens (under the TFC) in defining the programme during 2007 and beieved Siemens had a good understanding of the BBC s requirements. In 2007 Siemens had deivered a project requiring a broady comparabe skiset to Pacific Quay, the BBC s new buiding in Scotand, incuding tapeess workfow, HD production and use of a digita ibrary. The BBC therefore considers that, based on information avaiabe at the time, its appointment of Siemens to deiver the DMI programme was a sound decision, and fuy compiant with procurement procedures in pace. However, when the BBC saw that the contract was not deivering, it took steps to rectify the situation as soon as it coud and minimise the oss of vaue. 1 cmseect/cmpubacc/1067/1067.pdf

10 Recommendation b Athough it took the Programme technoogy deveopment in-house, the BBC did not test whether that was the best option. To manage risks and maximise the cost-benefit of investments: the BBC shoud prompty re-submit for approva those approved programmes where there are significant changes in the deivery mode, risk profie or costbenefit projection; and the BBC Trust shoud adopt referra threshods based on the forecast costbenefit of investments rather than a narrow financia threshod. When the BBC reaised the deivery of DMI was not proceeding as panned it made a number of distinct decisions over time. The first one was to take contro of the programme and therefore responsibiity for future decisions regarding the deivery of the programme. In doing this it was then abe to assess in detai progress on the technica soution and estabish the compete financia situation. This in-depth understanding was necessary to keep a options open and, subsequenty, to compete a revised business case. The BBC then considered a avaiabe options incuding (1) continuing the programme in-house (2) procuring from another suppier or (3) stopping the programme atogether. Due to the time critica nature of deivery versus time that woud have been required to compete a fu EU procurement; the desire not to deay benefits further and deteriorate the business case; and the fact that through working with a range of suppiers, it beieved it had buit up the necessary expertise to compete the project; the BBC decided to take responsibiity for deivering the project in-house. Finay, the BBC decided the extent to which DMI shoud be roed-out across the BBC. It ooked at options from expanding the programme across a production areas to reducing it to support the new faciities in Saford ony or even stopping the programme atogether. The case was then thoroughy scrutinised and chaenged at the BBC Executive s Finance Committee and Executive Board before Trust submission. The case was strengthened in response to requests, in particuar around the non-financia and strategic benefits, as the Executive wanted to satisfy itsef on this point before submission to the Trust. The BBC recognises that this took time, in particuar to ensure the case was appropriate, which is recognised by the NAO. The Trust Protoco on oversight of the BBC aready sets out that the Executive Board must submit revised financia proposas to the BBC Trust for approva where: a b circumstances change significanty subsequent to the approva of a budget for a particuar period of activity; or the Executive Board wishes to vary approved budgets outside the imits set within the referra framework. In addition, the BBC s investment poicy states that business cases must return to the Finance Committee for additiona approva if the costs or benefits over the whoe ife of the project change by the ower of 1 miion or 10%.

11 It is for the BBC Trust to respond to the NAO s specific recommendation to amend the referras threshods, but the Executive woud have no issues with impementing it. Recommendation c Without a proper understanding of the approach being foowed by a contractor and the abiity to intervene, the BBC wi be unabe to act as an inteigent cient. The BBC shoud: commission independent technica assurance reports on system design when contracting-out software deveopment; estabish the minimum technica and management requirements for effective oversight of contracts on a contract-by-contract basis; and estabish how and when it wi be abe to intervene to secure deivery of outsourced contracts rather than waiting for either contract non-deivery or termination. The BBC agrees with this recommendation, and is aready impementing it for new contracts. In particuar, a investment cases submitted to the Finance Committee must demonstrate how the contract governance structure aows for effective contract management. Recommendation d The financia benefits of the Programme were initiay overstated. The BBC shoud continue to test the benefits projections with the rigour it showed in reviewing the revised investment cases for the Programme by: reducing budgets to refect projected benefits; and estabishing baseines against which it can demonstrate savings. The BBC agrees with this recommendation, notes recognition by the NAO of the rigour it appied to the revised June 2010 investment case, and wi continue to strive to deiver appropriate business cases. Recommendation e There wi be essons to be earnt from the initia contract for the Programme. Athough the BBC and Siemens had separate interna essons earning reviews they did not share their understanding of the programme in a no-bame environment, even after they had setted the contractua dispute, to generate an agreed and comprehensive record of essons earned. The BBC shoud invite Siemens to draw up a joint understanding of essons to be earnt, not east because they have common business interests in the form of the Technoogy Framework Contract which runs unti March The BBC agrees with this recommendation, which it has in part impemented. It wi continue to work with Siemens to ensure a essons earned from the DMI experience are fuy considered as part of the continuous review of services and methods of service deivery within the TFC. securing sign-up for benefits from those responsibe for deivering them;

12 Recommendation f The technoogy system supporting the Programme has so far been shown to be vaid but the BBC has not yet as at October 2010 put in pace to the eve required the fu range of processes and contros that shoud aow it to compete the deveopment of the technoogy to the panned time, budget and functionaity. Specificay, it shoud: for its technica soution, compete the independent technica assurance of the design to provide a framework against which it can assess interdependencies and the impact of change contro requests; for its technoogy panning, draw up more detaied team-based pans specifying resource requirements and responsibiities for each team; for its testing, document a testing strategy to embed the discipine of testing and increase the use of automated testing toos to improve efficiency and acceptabiity of new technoogy components; and for suppier management, ensure that the suppier management ead appointed in October 2010 sets out how he wi secure eary understanding of the deivery risks and potentia mitigation posed by third party suppiers. The BBC notes the NAO s finding that the technoogy system supporting the programme has been shown to be vaid. The BBC is we aware that there are sti many compex issues in the remaining deiverabes, but is encouraged by the fact that the atest functionaities have a been successfuy reeased to users. The BBC agrees with the specific recommendations made by the NAO, which wi yet strengthen its abiity to deiver the programme, and has aready taken steps to impement them, as foows: Technica soution: The programme has now put in pace a compete set of architecture documentation, which aows the programme to determine interdependencies between components and to assess the impact of any proposed changes. Technoogy panning: The programme is now using Reease Managers who are accountabe for the deivery of each reease, and deveopment of detaied endto-end pans with cear resource responsibiities. Progress towards these reeases is reviewed on a reguar basis in ine with NAO recommendations. Testing: The programme produced a fu testing strategy in December Suppier management: As recognised by the NAO, the BBC has aready impemented this recommendation. Whist the programme has aways had a commercia ead in pace, a suppier management ead has now been appointed. This function wi grow, with the recruitment of three members of the suppier management team as further third party suppiers contracts are signed. BBC Executive January 2011

13 British Broadcasting Corporation The BBC s management of its Digita Media Initiative Report by the Comptroer and Auditor Genera This report has been prepared at the request of the BBC Trust under cause 79(3) of the Broadcasting Agreement between the Secretary of State for Cuture, Media and Sport and the BBC dated Juy Amyas Morse Comptroer and Auditor Genera Nationa Audit Office 13 January 2011

14 The BBC s Digita Media Initiative is a transformation project which is designed to aow BBC staff and partners to deveop, create, share and manage video and audio content and programming on their desktop. Nationa Audit Office 2011 The text of this document may be reproduced free of charge in any format or medium providing that it is reproduced accuratey and not in a miseading context. The materia must be acknowedged as Nationa Audit Office copyright and the document tite specified. Where third party materia has been identified, permission from the respective copyright hoder must be sought.

15 Contents Summary 4 Part One What is the Digita Media Initiative? 11 Part Two The BBC s initia approach to deivering the Programme 15 Part Three The BBC s revised approach to deivering the Programme 21 Part Four The preparation and evauation of the BBC s investment cases for the Programme 29 Appendix One Methodoogy 33 The Nationa Audit Office study team consisted of: Chris Jamieson, Ashey McDouga and David Sewe, assisted by Rache Neathey, under the direction of Keith Hawkswe This report can be found on the Nationa Audit Office website at Photographs courtesy of xxxxxxxxxxxxxxxxxxxxxxxx For further information about the Nationa Audit Office pease contact: Nationa Audit Office Press Office Buckingham Paace Road Victoria London SW1W 9SP Te: Emai: [email protected] Website:

16 4 Summary The BBC s management of its Digita Media Initiative Summary 1 The BBC s Digita Media Initiative (the Programme) is a transformation project which is designed to aow BBC staff and partners to deveop, create, share and manage video and audio content and programming on their desktop. It aims to reduce the time and cost of accessing and editing digita content and to foster creativity. The estimated gross cost of deivery and impementation to the end of March 2017 is miion. 2 The Programme is supported by a technoogy system (the system) which has to bind together production (where users manage the recreation of media content and need the atest creative digita media toos) and archiving (where users manage data storage, indexing and retrieva and require more traditiona Information Technoogy and toos). Successfu deivery of the programme is critica to the BBC s vision of staff across the BBC being abe to create, share and access digita content to make better programmes more efficienty. 3 Our review was prepared under an agreement between the Secretary of State for Cuture, Media and Sport and the BBC. 1 When the BBC Trust approved the Programme in January 2008, the system supporting it was due for deivery by the contractor by the end of May In Juy 2009, the contract for the system was terminated by mutua agreement and responsibiity for deivering the Programme brought in-house. Given the Programme is one of the BBC s seven key cross-cutting programmes supporting the future direction of the BBC, we ooked to see how effectivey the BBC has been managing the Programme. 4 Our methods are at Appendix One. 5 It is important to understand how the deivery timetabe, costs and benefits have changed over time. The panned and atest positions for the Programme are at Figure 1. The 2008 and 2010 costs and benefits are not directy comparabe as the atest deivery pans invove much wider ro-out of the Programme across the BBC than the 2008 pans. A cost and benefit figures in this report are expressed in cash terms. 1 The Broadcasting Agreement (Juy 2006) between the Secretary of State for Cuture, Media and Sport and the BBC, cause 79(3).

17 The BBC s management of its Digita Media Initiative Summary 5 Figure 1 Deivery Pans for the Programme Panned timeine for when deivery of the Programme by Siemens was to be to 6 business units January 2008 (approved by the BBC Trust) Investment cost 81.7 miion Panned benefit 99.6 miion Juy 2009 Contract termination Revised panned timeine for deivery of the Programme by BBC to 13 business units June 2010 (approved by the BBC Trust) Investment cost miion Panned benefit 97.9 miion Investment cost miion September 2010 (atest position) Panned benefit 95.4 miion System deveopment Use of system by the BBC NOTE 1 A cost and benefi t fi gures are in cash terms. Source: Nationa Audit Office anaysis

18 6 Summary The BBC s management of its Digita Media Initiative Key findings The contracted-out stage of the Programme 6 The BBC did not run an open procurement competition before awarding to Siemens the contract to deiver the Programme. In February 2008, the BBC contracted with Siemens for the deveopment, deivery and operation of the system supporting the Programme at a cost of 79 miion to March As the BBC aready had a ten-year Technoogy Framework Contract with Siemens, competitivey procured in 2004, it did not need to go through a competitive process to appoint Siemens, athough it was free to do so. It reied instead on Siemens s knowedge of the Programme, the assessment of Siemens made during the 2004 competition and its view that after ooking at other deivery partners and options it had not identified a better contractor. As a resut the BBC did not have as strong assurance on price, quaity and capacity to deiver as a new and specific competition may have provided. 7 The BBC negotiated a fixed price contract with fixed deivery miestones with Siemens. This transferred the risk of cost escaation to Siemens. The contract provided financia protection for the BBC in the event of non-deivery against expicit deivery miestones, uness the BBC undermined this risk transfer by contributing to any deay. 8 When it became ikey that the Programme woud be deayed, the BBC worked with Siemens to get deivery back on track. However, the two parties did not reach agreement on the causes for the deay and the Programme never reached the User Acceptance Testing phase. Because of the contractua transfer of deivery and financia risk to Siemens, the BBC did not want to intervene in a way that woud undermine that transfer of risk. The BBC s approach, even when it was concerned about the deiverabiity of the Programme, couped with its incompete knowedge of the system design, meant it was not in a position to deveop a detaied recovery pan unti the contract was terminated. 9 The contract with Siemens was terminated by mutua agreement with effect from the end of Juy As part of a no-faut settement, the BBC reached financia arrangements with Siemens which aowed the BBC to aocate 27.5 miion to meet the increased cost of competing the deayed Programme. This was funded through what the BBC and Siemens agree shoud be efficiency savings of 15 miion and reduced service charges of 9.5 miion, both within the overa Technoogy Framework Contract, a transfer of Programme assets ( 2.2 miion) and a payment to the BBC of 0.8 miion. 10 At the time the contract was terminated, the BBC estimated the competion of the system woud be 21 months ater than originay panned. As a resut, it did not achieve 26 miion in benefits it expected from the Programme in the period to In response, the BBC, to meet its corporate savings targets, made compensating savings in and pans to do the same for

19 The BBC s management of its Digita Media Initiative Summary 7 The BBC s in-house deivery stage of the Programme 11 The BBC took responsibiity for deivery of the Programme in Juy 2009, accepting the risks of deivering the Programme, athough without testing the vaue for money of this approach. When the contract with Siemens was brought to an end, the Programme was in difficuty and behind schedue; the deivery method was changing fundamentay, the chaenge in terms of the BBC s in-house capabiity increased and the financia risks transferred to the BBC. 12 Despite the known difficuties, the BBC did not revisit the investment case at this point or test deivery options, such as finding a new contractor. It tod us this was argey because of the time a fu EU pubic procurement woud take and the potentia impact of further deay on other time-critica BBC projects. It considered that taking the programme in-house was the ony soution and was achievabe, athough recognised that it did not, at that time have a the in-house capacity and capabiity necessary to deiver the Programme. 13 The BBC s in-house deivery of the system has started we but the compex stages to foow wi be a severe test of its approach. The in-house team deivered the first two system components, on schedue, in February and June 2010, and eary users have been positive about the impact. In September 2010, because of deays in defining procurement requirements, the deivery pan was revised, with the resut that competion of the in-house deivery of the system woud be deayed by a further five months. Since then the BBC has deivered a further two major system components on time against this revised schedue. The deays mean there is no time contingency eft in the revised deivery schedue, athough there is sti 10 miion of financia contingency avaiabe to mitigate risks to deivery. 14 The technoogy soution for the Programme has so far proven to be vaid. However, at the time of our review in November 2010, the panning processes the BBC had put in pace around the deveopment and testing of the system were not then sufficienty rigorous to support the more compex integration of system eements as the Programme proceeds. The BBC has since made appointments and commissioned externa assurance that shoud strengthen its processes. The financia case for the Programme The financia case for the Programme has weakened over time. 15 The origina cost-benefit estimate in January 2008 was a projected net benefit of 17.9 miion by March These figures were based on impementing the system across six BBC business units. The atest forecast is of a net cost to the BBC of 38.2 miion by March 2017 for a wider ro-out to 13 business units. This net cost fas to 10.7 miion after taking account of the financia package agreed with Siemens.

20 8 Summary The BBC s management of its Digita Media Initiative 16 When approving the revised business case in June 2010 the BBC Trust gave weight to the strategic benefits of moving the BBC more fuy into digita technoogy and the non-financia benefits expected from the Programme, such as improved creativity and increased partnership working with other organisations and potentia pubic access to the BBC archives. The BBC has no direct contro over the deivery of those non-financia benefits which wi not be deivered soey by the BBC, but has aready signed memoranda of understanding with partners incuding the British Library to reaise these benefits. 17 There was a marked improvement in the June 2010 investment case approved by the BBC Trust compared with earier cases. Previous reports by the Nationa Audit Office and the Committee of Pubic Accounts have highighted weaknesses in the BBC s investment appraisas process. The fina case had, unike previous cases, fu cost and benefit comparisons for scope options, as we as for a stop option. The expanation in the fina investment case of how benefits woud be secured was an area of marked improvement, as a resut of a more rigorous chaenge by the BBC Finance Committee and the BBC Trust. However, in the context of a programme with deivery difficuties, and where the BBC was accepting the risks, the case woud have been strengthened by independent assurance on costs and system design. Concusion on vaue for money 18 This concusion on vaue for money ooks at the Programme in two phases: the period when the Programme was contracted-out unti the BBC brought it in-house; and how the BBC is managing the in-house deivery of the Programme. 19 The way in which the BBC appointed the contractor without a new competition and was then unabe to intervene effectivey in system deveopment without undermining its transfer of financia risk to the contractor was not an effective way of approaching the deivery of a compex programme. Whie the BBC s financia arrangements with the contractor shoud aow the BBC to compete the Programme, the deay of 21 months and the 26 miion in Programme benefits the BBC did not achieve in that period, and had to find esewhere in the BBC, meant that the eary stage of the Programme was not good vaue for money. 20 The Digita Media Initiative is a chaenging Programme, but the BBC has now started to deiver the system and users have been positive about the eements deivered. There is sti a considerabe way to go in the deveopment of a technicay compex system which requires the integration of a number of interdependent eements without any time contingency. In addition, the success of the Programme wi depend on take-up by users across the BBC and esewhere. It is therefore too eary to concude on the ikey vaue for money of the Programme.

21 The BBC s management of its Digita Media Initiative Summary 9 Recommendations a b The BBC did not have an up-to-date assessment of its contractor s capacity and capabiity to deiver the Programme. The BBC assessed Siemens during a competitive procurement process in 2004 when it entered into a ten-and-a-haf year Technoogy Framework Agreement with Siemens as the BBC s strategic partner responsibe for its information systems. The BBC did not have to have a competitive procurement for the Digita Media Initiative as it coud appoint Siemens to deiver the Programme in a straightforward and quick procurement under that Framework. However, to provide assurance the BBC is not making procurement decisions on sub-optima grounds, it shoud demonstrate in investment cases why its procurement route is ikey to offer the best vaue for money. Athough it took the Programme technoogy deveopment in-house, the BBC did not test whether that was the best option. To manage risks and maximise the cost-benefit of investments: the BBC shoud prompty re-submit for approva those approved programmes where there are significant changes in the deivery mode, risk profie or costbenefit projection; and the BBC Trust shoud adopt referra threshods based on the forecast costbenefit of investments rather than a narrow financia threshod. c Without a proper understanding of the approach being foowed by a contractor and the abiity to intervene, the BBC wi be unabe to act as an inteigent cient. The BBC shoud: commission independent technica assurance reports on system design when contracting-out software deveopment; estabish the minimum technica and management requirements for effective oversight of contracts on a contract-by-contract basis; and estabish how and when it wi be abe to intervene to secure deivery of outsourced contracts rather than waiting for either contract non-deivery or termination. d The financia benefits of the Programme were initiay overstated. The BBC shoud continue to test the benefits projections with the rigour it showed in reviewing the revised investment cases for the Programme by: securing sign-up for benefits from those responsibe for deivering them; reducing budgets to refect projected benefits; and estabishing baseines against which it can demonstrate savings.

22 10 Summary The BBC s management of its Digita Media Initiative e f There wi be essons to be earnt from the initia contract for the Programme. Athough the BBC and Siemens had separate interna essons earning reviews they did not share their understanding of the programme in a no-bame environment, even after they had setted the contractua dispute, to generate an agreed and comprehensive record of essons earned. The BBC shoud invite Siemens to draw up a joint understanding of essons to be earnt, not east because they have common business interests in the form of the Technoogy Framework Contract which runs unti March The technoogy system supporting the Programme has so far been shown to be vaid but the BBC has not yet as at October 2010 put in pace to the eve required the fu range of processes and contros that shoud aow it to compete the deveopment of the technoogy to the panned time, budget and functionaity. Specificay, it shoud: for its technica soution, compete the independent technica assurance of the design to provide a framework against which it can assess interdependencies and the impact of change contro requests; for its technoogy panning, draw up more detaied team-based pans specifying resource requirements and responsibiities for each team; for its testing, document a testing strategy to embed the discipine of testing and increase the use of automated testing toos to improve efficiency and acceptabiity of new technoogy components; and for suppier management, ensure that the suppier management ead appointed in October 2010 sets out how he wi secure eary understanding of the deivery risks and potentia mitigation posed by third party suppiers.

23 The BBC s management of its Digita Media Initiative Part One 11 Part One What is the Digita Media Initiative? 1.1 The Digita Media Initiative (the Programme ) is the means by which the BBC aims to achieve its strategic goa of moving to more efficient and effective digita creation, archiving and sharing of content. The Programme has two components: deveopment of the technoogy system. The aim is to ink new and existing BBC systems to aow for the efficient transfer and use of digita fies across the BBC using desktop computers; and ro-out of the system in user Divisions. Once the system has been deveoped, the aim is that it wi hep users make programmes and generate new outputs, such as interactive and onine content, in new and more efficient ways, and support the creative use of the BBC s new and archived materia (Figure 2 overeaf). 1.2 To users the archive wi offer searchabe content and everything from whoe programmes to sound cips and unedited materia. The digitisation of the historic archive, which requires additiona investment, is outside the scope of the Programme, and managed as a separate programme by the BBC. However, the Programme is an important enaber for the cost-effective digitisation of the archive. 1.3 Both the new system and changed ways of working to use the system are necessary to maximise the benefits of the Programme. 1.4 The BBC expects to reaise both financia and non-financia benefits from the impementation of the Programme. Financia benefits wi be in three areas: cost reduction reductions in operating costs, brought about by more efficient production and archiving processes; cost avoidance the avoidance of costs that woud have been incurred in the future, if the Programme were not to be impemented, for exampe, the costs of deveoping oca aternatives to the Programme across the BBC; and creative dividend the savings that coud be achieved by making use of materia in the Digita Archive rather than producing new content.

24 12 Part One The BBC s management of its Digita Media Initiative Figure 2 The eements of the Programme and the production process Capture Desktop production Muti-patform deivery Digita Archive Editing faciities Source: BBC 1.5 The Programme is one of seven BBC-wide strategic programmes, enabing and supporting deivery of key BBC priorities incuding: the increased avaiabiity of materia that can be used across the BBC s fu range of outets TV, onine, radio, interactive and ipayer, and greater accessibiity of digita content for audiences; the move of 2,400 staff to Saford. From May 2011, the BBC pans to start moving five of its divisions to Saford and the working practices there are buit on the avaiabiity of the Programme; and the BBC s efficiency programme. 1.6 When initiating the programme in 2007, the BBC judged that the quick deivery of the Programme was essentia. With the above dependencies a being time-sensitive, particuary the move to create a media hub in Saford, any deay in deivering the Programme woud have knock-on effects, principay through diminished or ost financia benefits and a faiure to move with audience expectations.

25 The BBC s management of its Digita Media Initiative Part One 13 Responsibiities within the BBC for approving and deivering the Programme 1.7 The Programme is co-sponsored by two BBC Divisions: Future Media and Technoogy, which is responsibe for deveoping the investment cases and securing investment approva from the BBC Executive and the BBC Trust; and BBC Vision on behaf of user Divisions, who are responsibe for depoyment of the system and for benefits management. The Programme team in the Future Media and Technoogy Division is aso responsibe for commissioning the technoogy system, roing it out to users, getting them trained and prepared, supporting and maintaining it as an operationa (business critica) service and securing the support of user divisions necessary for the benefits of the Programme to be deivered. Figure 3 sets out the key payers in reation to the Programme. Figure 3 Financia and organisationa responsibiities for the Programme BBC Trust Approves investment cases above 50m in ifetime costs BBC Executive Board Approves submissions to BBC Trust Finance Committee Evauates investment cases above 2m in ifetime costs BBC Vision (on behaf of User Divisions e.g. Vision, Audio and Music) Co-sponsor for Programme Appy the Programme and generate financia and non-financia benefits Future Media and Technoogy Division Co-sponsor for Programme Sponsors the investment case to the Finance Committee and responsibe for deivering the system. The Programme team is in this Division NOTE 1 This report refers in most paces to the BBC as incuding a the above eements (apart from the Trust) and ony exceptionay to the individua eements where such distinctions are necessary to understand actions and responsibiities. Source: Nationa Audit Office

26 14 Part One The BBC s management of its Digita Media Initiative What is this report about? 1.8 This report ooks at three aspects of the Programme: The BBC contracted with Siemens in February 2008 to deiver the Programme and to support the operation of the system across the BBC unti The contract was brought to an end in Juy 2009 (Part Two of the report). The BBC brought the deivery of the Programme in-house in Summer 2009 and is currenty impementing and roing-out eary reeases of the system. It has deivered some eements but the competion of the fu soution has recenty been deayed (Part Three). The strength of the approvas process and investment cases for the Programme (Part Four).

27 The BBC s management of its Digita Media Initiative Part Two 15 Part Two The BBC s initia approach to deivering the Programme 2.1 In January 2008, the BBC Trust approved the investment case, proposed by the Future Media and Technoogy Division, for proceeding with the deveopment of the Programme. The proposa was for deveoping and roing-out the Programme to 6 of the BBC s 22 business units at an estimated cost of 81.7 miion and projected benefits of 99.6 miion to March 2015 (Figure 4 overeaf). A cost and benefit figures in this report are expressed in cash terms. The Trust was provided with estimated fu ifetime costs ( 143 miion) and benefits ( 184 miion), covering the deveopment of the system and ro-out to the whoe BBC but these were provisiona estimates for information ony. The Trust was not asked to approve the additiona ro-out, as this was to be considered once the system deveopment and eary ro-out had been achieved. Seection of the contractor to deiver the Programme 2.2 Competitive tendering is good practice and is normay a ega requirement for a contract of this size ( 79 miion). Competition aows the purchaser to compare potentia suppiers prices and technica proposas, as we as their capabiity and capacity to deiver. After an open procurement competition, the BBC signed a ten-and-a-haf year Technoogy Framework Contract with Siemens in 2004 for the provision of technoogy services, which mentioned the Programme as a ikey future project. Athough this did not guarantee award of the contract for this particuar Programme to Siemens, it meant that under the framework contract the BBC was abe to award a ca-off contract to Siemens without competition.

28 16 Part Two The BBC s management of its Digita Media Initiative Figure 4 Origina panned timeine for deivery of the Programme January 2008 (approved by the BBC Trust) Investment cost 81.7 miion Panned benefit 99.6 miion February 2008 Contract et November 2008 First key deivery miestone May 2009 Second and fina key deivery miestone Juy 2009 Contract termination System deveopment Use of system by the BBC NOTE 1 A cost and benefi t fi gures are in cash terms. Source: Nationa Audit Office anaysis 2.3 The BBC decided that the potentia benefits of competition for the contract to deiver the Programme were not sufficient to outweigh the increased risk of deaying a key part of its corporate strategy and deivery of the benefits so did not run a new procurement competition for the Programme contract. The Programme team, on the basis of discussions with Siemens and other potentia suppiers, considered that Siemens had the abiity to deiver the contract to specification, cost and time and set this out as an expicit assumption in the investment case for the Programme. The principa reasons for seecting Siemens in February 2008 were: Siemens had a good understanding of BBC systems and experience of deivering projects with the BBC; the BBC had tested Siemens s capabiity and capacity to deiver through assessments made as part of the Technoogy Framework Contract procurement. Athough it had been four years since the Framework Contract was et, the BBC did not update its assessment when deciding to use Siemens for the Programme;

29 The BBC s management of its Digita Media Initiative Part Two 17 the BBC had worked with Siemens (under the Technoogy Framework Contract) in defining the Programme during 2007 and beieved Siemens had a good understanding of the BBC s requirements; the BBC considered that after ooking at other deivery partners and options it coud not identify any better deivery mode than using Siemens; and the BBC estimated that procuring the Programme as a ca-off contract within the Technoogy Framework Contract and thereby avoiding open procurement coud save the BBC six to nine months and costs of up to 3 miion. 2.4 Before submitting its investment case for the Programme to the BBC Trust, the BBC Executive asked externa consutants to advise it on whether the Siemens price was competitive. The consutants concuded that open competition coud have generated a better price, but that any savings (which were not quantified) woud ikey be offset by the cost of having to integrate the new system with the Siemens-run BBC network and deays in securing benefits. 2.5 Athough good practice and the BBC s own Investment Guidance highight the vaue of testing and comparing the cost and benefits of a range of options, in presenting the investment case for Trust approva in January 2008, the BBC did not offer any other options for deivering the Programme. The Trust therefore had the option of authorising the BBC to contract with Siemens or not undertaking the Programme. Without transparenty testing and expicity comparing the costs and benefits of other options for deivering the Programme, the Programme team was unabe to demonstrate to the Trust that the chosen deivery course optimised vaue for money. 2.6 In February 2008, the BBC awarded Siemens a 79 miion fixed price contract to design and deiver the system supporting the Programme by May 2009, and to support ro-out of the system unti March The baance of 2.7 miion in the approved budget was for BBC costs to support the Programme to the end of March Figure 5 sets out the eements of the Programme to be deivered by Siemens. Figure 5 The eements of the Programme to be deivered by the contractor Contract with Siemens to deiver Design of the system to meet contractor specifications Procurement and management of subcontractors Integration of the system with the BBC s IT system Support for the operation of the system unti March 2015 Source: Nationa Audit Office

30 18 Part Two The BBC s management of its Digita Media Initiative Management of the contract 2.7 The Programme is a BBC strategic priority and other key programmes are dependent on its timey deivery (paragraph 1.5), particuary the BBC s efficiency programme. 2.8 A key area for the contract was the specification of what the system shoud ook and fee ike to users. The contract incuded a 30 day eaboration phase to be competed in March The purpose of the eaboration phase was for Siemens and the BBC to agree the user expectations for the Programme, which Siemens woud then incorporate into the design phase. The eaboration phase was not competed unti June 2008 deaying the start of the design phase and putting timey deivery at risk. In fact the two parties did not reach agreement on the detaied design for the system and the Programme never reached the User Acceptance Testing phase. 2.9 The contract specified two cear key miestones for the deivery of the Programme technoogy (Miestone 1 in November 2008 and Miestone 2 in May 2009). The BBC transferred to Siemens the financia risk of deveoping the Programme within a fixedprice contract, whie retaining BBC sign-off of the outputs from the contract. Payments were dependent on deivery of the miestones From March 2008, one month into the contract, the BBC was aware that deivery of the first key miestone in November 2008 was ikey to be deayed by three months (athough it anticipated this was ony a temporary deay). The BBC and Siemens worked together to get deivery of the Programme back on track In managing the outsourced contract, the BBC s knowedge of the adequacy of Siemens s design and deveopment work, and therefore of the causes of any deay, was imited. The BBC did not have any independent technica assessment of the system, as woud be good practice for system design, and unti May 2009 did not seek access to the Siemens code supporting the system When it became ikey that the Programme woud be deayed, the BBC worked with Siemens to get deivery back on track. The BBC tod us that, in ine with its interna ega advice, it was keen not to interfere with or undermine the design and deivery of the system which was Siemens s responsibiity under the contract. As a resut of this approach and incompete knowedge of the system design, the BBC was not in a position to deveop a detaied recovery pan unti after the contract was terminated.

31 The BBC s management of its Digita Media Initiative Part Two 19 Handing of the contract termination 2.13 In the period February to May 2009, the BBC s expicit goa was sti to secure deivery of the system by Siemens, even though it woud be ater than contracted, whie estabishing, protecting and strengthening its ega position shoud the need to terminate the contract arise. The BBC and Siemens were in dispute over the reasons for the deay in deivering the Programme. To protect its position, the BBC: did not vary the key deivery miestones set out in the contract; and formay notified Siemens that time was of the essence and set a fina deadine for deivery In addition, the BBC began to assess its potentia capabiities to deiver the programme in-house, were that option to be chosen at a ater stage By June 2009, the BBC and Siemens had entered into without prejudice negotiations to resove the dispute arising from the ate deivery of the project. The BBC entered into negotiations with Siemens with a cear target, approved by the BBC Finance Committee, of the amount the BBC required from Siemens ( 25 miion in cash and 10 miion in non-cash benefits). It sought this sum to cover what it estimated to be the 20 miion cost of competing the system supporting the Programme and to provide compensation. The BBC tod us that whie it had taken externa advice on ikey costs to compete the system deveopment stage it had no documentation setting out independent assurance on the estimated cost to compete In September 2009, the BBC and Siemens entered into a no-faut settement terminating the contract by mutua agreement with effect from 31 Juy As part of the settement the parties agreed financia arrangements which aowed the BBC to aocate 27.5 miion to compete the Programme, comprising; what the BBC and Siemens consider to be efficiency savings for some contracted functions within the Technoogy Framework Contract (paragraph 2.2) eading to 15 miion in reduced service charges for the BBC over the period October 2009 to September 2013; both parties tod us that these savings were possibe as a resut of their contractua agreement to changes to how services were deivered which enabed Siemens to deiver the same eve of services in a different and more efficient way. The BBC is estabishing a baseine against which it can consider whether there is any diminution in the service received in these areas; additiona adjustments eading to a service charge reduction within the Technoogy Framework Contract of 9.5 miion from Siemens to the BBC over the period October 2009 to September 2013; a transfer of assets reated to the Programme vaued at 2.2 miion from Siemens to the BBC; and a cash payment from Siemens of 0.8 miion.

32 20 Part Two The BBC s management of its Digita Media Initiative 2.17 The BBC Finance Committee approved the settement vaue, athough ower than originay sought, as covering its expected costs to compete system deveopment for the Programme. The financia arrangement with Siemens did not however cover the 26 miion of benefits that the BBC estimated it had not achieved due to the deay in the Programme. To meet the corporate savings target the BBC covered the ost Programme savings for through increased savings from other areas within the BBC and pans to do the same to make good the shortfa in Programme-reated savings for There is no absoute measure of whether the financia arrangements agreed with Siemens represent vaue for money for the BBC. They were the outcome of a process of negotiation, and there is no way of knowing how successfu an aternative course of action woud have been. But the BBC did take ega advice on the potentia costs and risks of going through itigation, and reaching agreement meant the BBC coud proceed with the deveopment of the Programme without time-consuming and costy itigation which coud aso have adversey affected the overa reationship between Siemens and the BBC for the baance of the Technoogy Framework Contract (to March 2015) The settement was dependent on a confidentiaity cause. BBC management agreed to the confidentiaity cause in the settement agreement to secure a costeffective dea and the unencumbered deveopment of the Programme. HM Treasury guidance discourages confidentiaity causes where they might prevent, or seek to prevent, pubic accountabiity and scrutiny of the use of pubic money. The confidentiaity provisions expressy identify the Nationa Audit Office as a competent authority which the BBC tod us it incuded in order to protect pubic accountabiity. The provisions require the BBC to in good faith. use reasonabe endeavours. to secure the nondiscosure of the settement agreement by a third party With the prospect of itigation not removed unti the settement was agreed in September 2009, the BBC and Siemens did not share information with each other about where improvements in contracting, designing and deivering the programme coud have been made. Both organisations drew up interna essons earned documents from their own perspective. However, there was no joint esson earning even after the prospect of itigation had gone. As a resut, even after the prospect of itigation had gone, the two organisations have no record of any shared understanding of where and how the programme had not worked and how probems coud be avoided in future work The main service impact of the termination of the contract for the BBC was that by the Summer of 2009 it had effectivey used up at east 18 months of deivery time without securing a working system. The BBC had contracted with Siemens for the deivery of the technoogy for the Programme by May 2009 but the BBC estimated that in practice it woud be approximatey 21 months after that date before it woud have the functioning system fuy avaiabe.

33 The BBC s management of its Digita Media Initiative Part Three 21 Part Three The BBC s revised approach to deivering the Programme 3.1 In Juy 2009, the BBC decided to terminate the contract with Siemens and bring responsibiity for deivering the Programme in-house, taking over the software deveopment, contractor management, systems integration, system operation and maintenance roes previousy contracted to Siemens. Taking deivery of the Programme in-house was a significant step for the BBC as it now carried the financia risk of deveoping and deivering the system. Without fuy understanding, quantifying and mitigating those risks through a considered process at the time of this decision, the BBC coud have exposed itsef to further deay in achieving benefits and increased cost in deivering the Programme. The decision to bring the Programme in-house 3.2 Before bringing the Programme in-house the BBC needed to be sure of its own capacity and capabiity. A review in February 2009 (before the Siemens contract was terminated) concuded that taking deivery in-house was the highest risk option, given the BBC s capacity and capabiities at that time, and one for which the BBC was not adequatey prepared. The BBC did not re-run the capacity and capabiity exercise when deciding to take the Programme in-house five months ater. However, before taking the Programme in-house: the BBC had taken steps to strengthen its Programme team; the new Chief Technoogy Officer (appointed in Apri 2009) had identified the practica issues to be managed in deivering the Programme in-house; and the BBC s Future Media and Technoogy division prepared a contingency pan for taking deivery of the Programme in-house, but acknowedged at the time that the pan was prepared without a fu understanding of the technica and design issues that Siemens had encountered and that as it did not have the capacity or capabiity to deiver the system it woud have to recruit or use a number of third party suppiers to fi this gap.

34 22 Part Three The BBC s management of its Digita Media Initiative 3.3 Despite a significant departure from the costs and benefits in the origina investment case (paragraphs 3.5 to 3.9 beow), the deivery responsibiity and financia risks no onger being outsourced and the programme incurring a 21 month deay, the Future Media and Technoogy division, with the knowedge of the BBC Finance Committee, brought responsibiity for deivering the programme in-house without any forma re-approva or revision of the strength of the investment case. The BBC considered that taking the programme in-house was the ony soution. It took responsibiity for deivering the Programme in-house because: it aready had financia approva from when the Trust approved the investment case (to deiver the Programme through Siemens) in January 2008 (paragraph 2.1); it judged that deivery of the Programme was time critica and that a fu EU procurement woud take too ong. The BBC s most pressing driver was the overriding need to have Programme technoogy avaiabe for Saford by May 2011 (paragraph 1.5); it had confidence it coud do the deveopment work within the existing budget and the money provided by the agreement with Siemens (paragraph 2.16); and any further deay to the programme woud resut in ost or deayed benefits above the 26 miion aready identified (paragraph 2.17), woud ead to the oss of key project expertise, and woud make the ega transfer of contracts more difficut or impossibe. The changing timescae, costs and benefits January The origina approva for the Programme from the BBC Trust in January 2008 was for ro-out to 6 of the BBC s 22 business units at an estimated cost of 81.7 miion and projected benefits of 99.6 miion to March November By November 2009, when the Programme team submitted to the Finance Committee a revised investment case to refect the increased costs associated with taking responsibiity for deivering the Programme in-house, the estimated cost to compete the Programme was miion, 23.4 miion (29 per cent) above the origina 81.7 miion. The increased cost was because of the need to deveop the system for the Programme, athough the BBC expected the financia settement with Siemens woud more than cover the estimated increased cost to the BBC. The estimated financia benefit had faen from 99.6 miion to 74.1 miion because of the deay to the Programme. The November 2009 investment case was rejected by the BBC Finance Committee because the expanation of how benefits woud be secured was inadequate. The Finance Committee asked that a revised case be re-submitted.

35 The BBC s management of its Digita Media Initiative Part Three 23 June In June 2010, the BBC Trust approved a revised investment case, which was for ro-out of the Programme on a wider scae than originay approved in January The gross costs were miion ( miion net of the settement with Siemens) and the projected financia benefits were 97.9 miion to March 2017 (the investment case period was extended by two years to take account of the 21 month deay aready incurred on the Programme). The main reason for the increase in costs and benefits compared with November 2009 was that the scope of ro-out had widened from 6 to 13 of the BBC s 22 business units. 3.7 The principa forecast financia benefits from the Programme are in three areas: Cost reduction 51.1 miion in reductions in operating costs, brought about by more efficient production and archiving processes. Cost avoidance the avoidance of 29.8 miion in costs that woud have been incurred in the future, if the Programme were not to be impemented. Creative dividend 17 miion in savings by making use of materia in the Digita Archive rather than producing new content. 3.8 With the estimated cost being 35.7 miion higher than the estimated financia benefits ( 8.2 miion after taking into account the funds reeased after the agreement with Siemens), un-quantified non-financia benefits expected from the Programme were a crucia factor in the Trust s approva to continue. Maximising and securing the non financia benefits is therefore vita for the success of Programme in the ong term. The BBC identified the principa non-financia benefits as being: improving the quaity of content; supporting the BBC s out-of-london strategy by aowing digita content to be shared across the BBC and potentiay buiding partnerships with, for exampe, the independent programme-making sector; giving BBC staff access to a the digita materia in BBC archives; and working through Memorandums of Understanding with other organisations, such as the British Library, to deveop common access standards which woud eventuay aow the pubic to access content regardess of source, subject to rights cearance. The BBC@BL initiative is a project to provide access to BBC archive materia through the British Library s reading rooms, using the Programme s technoogy. 3.9 The BBC Trust chaenged these non-financia benefits but accepted that they did offer the prospect of creative benefits. It aso gave weight to the strategic benefits of moving the BBC more fuy into digita technoogy for managing and sharing content and these factors aowed it to approve continuation of the Programme. Deivery of some of the non-financia benefits remains uncertain and outside the BBC s contro where it is dependent on the actions or cooperation of other organisations athough the BBC is discussing its Programme with those partners.

36 24 Part Three The BBC s management of its Digita Media Initiative September By September 2010, the projected ifetime cost was unchanged at miion. However, 6 miion of the 16 miion contingency within that budget had been aocated and the system deveopment phase had sipped by a further five months. As a resut of the deay, the ikey financia benefits had faen by between 1 miion and 6 miion, with the most ikey impact being a reduction of 2.5 miion. Overa financia benefits woud therefore fa to 95.4 miion, 4.2 miion (4 per cent) ower than when the BBC Trust first approved the Programme in January 2008 (when ro-out was to be significanty more restricted to 6 business units rather than the current 13). So by September, the expected costs exceeded benefits by 38.2 miion Figure 6 sets out the BBC s projected timescaes, cost and benefits as at January 2008 when first approved by the BBC Trust, November 2009, when re submitted by the Programme team but rejected by the BBC Finance Committee, and June 2010 when re-approved by the Trust and in November Two reeases of system technoogy (Basic Archive and Basic Production Toos) were deivered on schedue in February and May 2010 respectivey. Eary users have been positive about the impact of having eements of the new system. By Juy 2010, the remaining reease dates were changing as the BBC carified deivery dates with suppiers, sought to reconcie procurement ead times with impementation pans and ooked to deiver the agreed minimum requirements to support operations at Saford In August 2010, the BBC drew up a revised deivery schedue to compete deivery by Juy 2011, five months ater than estimated when the BBC took the deveopment of the Programme in-house in Juy Based on the revised schedue, it has since deivered on time the next two reeases (Advanced Production Toos and Advanced Editing Toos) in September and December How the current approach is working to deiver the Programme and its benefits 3.14 In considering the essons from the Siemens phase of the Programme, the BBC concuded that an approach with big bang deiveries was not appropriate for this type of programme invoving the deveopment, procurement and integration of different technoogies. It therefore adopted a more iterative agie approach. This invoves breaking down deveopment into smaer steps, with quick feedback oops from users to improve the products being deveoped, aongside more structured miestone deiveries for the underying infrastructure supporting the Programme. Instead of the two arge big bang reeases the BBC commissioned from Siemens, the approach had 6 distinct stages for the reease of technoogy to BBC user Divisions over a period of 16 months to February 2011, graduay increasing the functionaity avaiabe to users as we as the number of users The agie approach adopted by the BBC does not end itsef to straightforward reporting against performance miestones. The approach encourages fexibiity in panning and deivery as feedback is received from users within an overa deivery timetabe. This is both a strength and a risk for the approach.

37 The BBC s management of its Digita Media Initiative Part Three 25 Figure 6 Panned timeine, costs and benefi ts for deivery of the Programme Panned timeine for deivery of the Programme to 6 business units January 2008 (approved by the BBC Trust) Investment cost 81.7 miion Panned benefit 99.6 miion November 2009 (rejected by the BBC Finance Committee) Investment cost miion Panned benefit 74.1 miion Panned timeine for deivery of the Programme to 13 business units June 2010 (approved by the BBC Trust) Investment cost miion Panned benefit 97.9 miion Investment cost miion September 2010 (atest position) Panned benefit 95.4 miion System deveopment Use of system by the BBC NOTE 1 A cost and benefi t fi gures are in cash terms. Source: Nationa Audit Office anaysis Chaenges the BBC faces in competing the deveopment of the Programme and securing the forecast benefits 3.16 It is industry good practice to obtain independent assurance reports to management and other interna stakehoders for technoogy programmes. The Programme team commissioned a progress review in January The BBC Trust ater (June 2010) commissioned the same consutants to review progress. The second report found cear evidence of progress in areas of governance and programme management but noted that for the benefits to be achieved there needed to be increased engagement with stakehoders; the iterative approach to deveopment needed more contros; deivery panning, forecasting and dependency management needed to improve; and the timing and content of the deivery reeases needed reconfirming.

38 26 Part Three The BBC s management of its Digita Media Initiative 3.17 We used these reports as background for our examination as at November 2010 and identified the foowing key chaenge areas that the Programme team wi have to manage carefuy as it continues in deivering the Programme soution: Technica Soution and Infrastructure. Panning. Suppier Management. Testing. Technica Soution and Infrastructure 3.18 The Programme team has to bind together what the BBC has historicay seen as very different business areas: Production, where users manage the creation of media content and need the atest creative digita media toos; and Archiving, where users manage data storage, indexing and retrieva and require more traditiona Information Technoogy and toos. Because digita content is invoved, both business areas need to manage and transfer significant data voumes using a mix of third party components and products over the BBC network There was rapid progress over the first part of The eary programme reeases have shown that so far the system fundamentay works, is feasibe and abe to meet the business requirement. However, the eary reeases are arguaby the most straightforward to deiver. As the Programme advances, the chaenge of integrating the different technoogy components and appying them in new user situations wi increase A major risk for the Programme is the number of inter-dependencies that exist between the products and components that make up the fu soution. Probems and deays caused in one area wi impact others with the potentia for a significant cumuative impact on the BBC as a whoe. An exampe of this is the five month Programme sippage the BBC recognised in September 2010 (paragraph 3.10), caused by a deay in specifying and thus procuring one component of the archiving and search soution Whie the existing design documentation has proved satisfactory for the eary product reeases, improved architecture documentation and change contro processes wi essen the risks of design errors as the Programme continues to be roed-out It is standard practice in technica design to seek an independent technica assessment. Such an assessment shoud provide assurance that the design is vaid, robust, and that the various aspects of the soution are both necessary and sufficient to meet requirements. There was no such independent technica assessment commissioned by the BBC during the first phase with Siemens, nor after the Programme was taken in house. A technica assessment has now been commissioned in September 2010 and the BBC expects the findings to be avaiabe before the end of the year.

39 The BBC s management of its Digita Media Initiative Part Three Dependence on the abiity of the BBC s information technoogy network to cope with the voume of materia generated by users of the Programme moving across the BBC s network has been fagged by the Programme team and externa consutants as being a major risk. However, the Programme team is increasingy addressing this, through the use of detaied capacity panning using oad simuations and has secured the agreement of the BBC s network suppier, Siemens, that the network wi support the demands paced upon it. Panning 3.24 Panning is critica to the timey and cost-effective deivery of any programme and aows measurement of the efficiency and effectiveness with which teams use avaiabe resources to achieve objectives. After a mid-deveopment review of its approach in mid-2010, the Programme team recognised that it woud incur a five month deay. The revised pan, if met, shoud deiver the minimum functionaity necessary for the move to Saford athough the panning process does not foow industry best practice for a programme of this size However, the revised pan gives no room for manoeuvre over reease content or dates. There is some financia contingency (paragraph 3.10) but athough the BBC has some scope to adjust suppier deiverabes and move functionaity between system reeases, there is no time contingency. Specific weaknesses in the current approach to panning are that the pan: does not set out team-based resources and usage information. As a resut, there is a risk that the Programme team of up to 180 peope may not be the right size and may not have the right baance of skis; does not map into the reease schedue and, combined with inconsistent use of terminoogy, this coud ead to confusion about reeases; and contains ony imited information on the activities to be carried out by some of the third party providers eading to a risk of ack of visibiity of progress. Suppier Management 3.26 Deveopment of the system reies heaviy on third party products, such as editing software and databases, but unti October 2010, the BBC did not have a fu-time Suppier Management Lead in post. The BBC is managing this dependency by securing access to eary versions of products. For exampe, one of the major Programme dependencies is the work being done by a suppier on the archiving function. The Programme team manage the risk of deay by having access to eary reeases from the suppier against which they are running tests on data transfer between systems. Nevertheess, the procurement and acceptance timetabe for this area is very tight with itte room for sippage.

40 28 Part Three The BBC s management of its Digita Media Initiative 3.27 There are suppier management risks that the Programme team needs to manage carefuy, namey: for a number of third party suppiers the Programme pan ony gives a date for agreeing a specification and a date for the fina technoogy reease from the suppier. Due to the high dependency on suppiers deivering the right product at the right time, coser invovement of the team with suppiers is required; and the European Commission procurement rues that the BBC must adhere to pose chaenges to meeting the requirements of an iterative deveopment process. Deays in ordering essentia components account for the buk of the five-month sippage during 2010 (paragraph 3.10). The BBC is managing procurement by running mini-competitions between existing BBC Framework Contractors ( 42 miion) and Office of Government Commerce Framework Contractors ( 51 miion), rather than a project-specific procurement competition. There are no open procurements in the current procurement strategy for the Programme. The impact procurement timescaes remains a risk for the Programme s deivery schedue. The BBC recognises that vaue for money may not be optimised by this approach, as shoud be the case through an open competition, but considers the benefits of quick procurement outweigh the risks to vaue for money. Testing 3.28 Thorough testing is centra to software deveopment. Without a structured approach to testing and the capabiity and capacity to pan and carry out such testing the Programme team risks deivering partiay tested components with a high expected rate of faiure. This woud add the additiona expense of fixing fauts at a ate stage in the process with serious damage to user confidence. The BBC has we documented processes for integration testing and user acceptance testing and has carried out such testing successfuy on the technoogy reeased so far The main risks that the current approach to testing has are that: there is no documented Testing Strategy defining the test poicy to be foowed, athough this is a priority for the recenty-appointed Test Manager. Having such a strategy heps define the approach to testing and makes it easier to enforce a test discipine Programme-wide. This is particuary important in the iterative deveopment process that the BBC has adopted, where testing has to be carried out reguary and repetitivey through each iteration. Inadequate or incompete testing coud ead to further deays; and whie manua testing has been adequate for the eary reeases, it wi become progressivey more difficut and time consuming as the Programme continues and inter-dependency between component parts of the system increases. Insufficient use has been made so far of the automated test toos that are avaiabe. Such test toos formaise the test process and automate difficut areas such as repetitive regression testing, which can then be competed as a matter of course.

41 The BBC s management of its Digita Media Initiative Part Four 29 Part Four The preparation and evauation of the BBC s investment cases for the Programme 4.1 Without rigorous processes for appraising proposed investment cases organisations do not test the vaue for money of proposed expenditure. The BBC has financia threshods, above which spending decisions must be approved by Divisiona Boards, the Finance Committee, the BBC Executive or the BBC Trust. The approvas processes are designed to ensure that proposas for significant spending are tested for costs, benefits, risks, dependencies, strategic fit and deiverabiity in advance of committing the BBC to a course of action. 4.2 In previous reports on the BBC the Committee of Pubic Accounts and the Nationa Audit Office have been critica of aspects of the BBC s approva processes. Figure 7 overeaf sets out the core eements we woud expect to see in an investment case and how we the Programme investment cases addressed these. It shows an improvement over time in most areas. Areas where there is evidence of improvement 4.3 In the period since the BBC decided to bring the Programme in-house, there has been evidence of greater interna chaenge of spending proposas. 4.4 In November 2009, the Future Media and Technoogy division sought the BBC Finance committee s approva for a budget increase to be funded from the settement with Siemens (paragraph 3.5). The Committee rejected the proposa: it was concerned that the Heads of Production in user Divisions were not sufficienty engaged with the Programme. It aso wanted a re-assessment and re-vaidation of the benefits, which had been simpy been roed forward from proposas approved by the Trust in January To avoid further deay to the Programme, the Committee agreed that work coud continue pending the review of panned benefits, but made cear that new costs incurred woud have to be approved by the Committee. 4.5 The BBC Executive approved a new investment case in Apri 2010, incuding wider ro-out across the BBC. As the revised case had a budget increase of more than 10 per cent, the case needed BBC Trust approva and was submitted to the Trust in Apri In considering the revised investment case, the Trust wanted greater carity on the deiverabiity of non-financia benefits and gave its approva in June 2010 (paragraph 3.8).

42 30 Part Four The BBC s management of its Digita Media Initiative Figure 7 Did the BBC foow good practice in investment appraisas? Core eements of an investment case Criteria for Good assessment Jan 2008 investment case approved by BBC Trust Nov 2009 investment case rejected by BBC Finance Committee June 2010 investment case approved by BBC Trust Option appraisa An assessment of the fu-ife costs An assessment of the deivery timetabe of the programme A benefits reaisation pan An assessment of risk to the programme A broad range of options are examined incuding a do nothing or do minimum option. Cost-benefit anaysis has been carried out for each option, incuding a net present vaue cacuation. A materia direct and indirect costs have been examined. Costs provided are based on appropriate evidence and reasonabe assumptions. A deivery timetabe is incuded ceary showing key deivery miestones. Timetabe is based on appropriate evidence and reasonabe assumptions. A materia direct and indirect benefits have been examined. Benefits provided are based on appropriate evidence and reasonabe assumptions. Description of when benefits wi be deivered with named individuas signed up to their deivery. A reevant risks were appropriatey discussed and presented. Risks have been aocated and mitigating contros identified. Identified risks have been quantified (ikeihood and impact have been assessed, incuding financia impact where possibe). Weak Weak Good Fair Fair Good Fair Fair Good Fair Weak 2 Good Fair Weak Good NOTES 1 The fu assessment range is: Weak, Fair, Good, Exceent. 2 The November 2009 investment case expicity recognised that its benefi ts reaisation pan was incompete and that further anaysis woud be required. Source: Nationa Audit Office assessment against industry good practice

43 The BBC s management of its Digita Media Initiative Part Four The 2010 investment case showed improvements in terms of setting out options: the origina January 2008 investment case set out three options: deveop a centraised Programme (the preferred option); depoy oca, site-specific soutions in key ocations; or, continue to aow the growth of mini, utra-oca Programme soutions (essentiay the do nothing/stop option). However, the costs, benefits and risk were presented ony for the preferred option; the November 2009 investment case did not expore a range of deivery options; and by Apri 2010, the scope of ro-out had widened and the investment case now incuded Saford in a range of depoyment options, incuding depoyment to Saford ony and a costed stop option. However, given the time-critica Saford dependency and the fact that the Programme had started to deiver, it is not cear that this was a reaistic option. 4.7 The articuation of intended benefits aso improved in the 2010 investment case approved by the BBC Trust. Athough the origina 2008 investment case had categorised and quantified the benefits and given a broad indication of when they woud be reaised, the benefits had not been assigned to BBC executives responsibe for deivering them and it was uncear how the benefits woud be measured. Athough the appraisa process does not in itsef guarantee that the benefits wi be deivered, the BBC made significant improvements in this area by June 2010 through: taking a systematic approach to revisiting the benefits; agreeing quantified targets with named benefit owners responsibe for deivering the benefits. Potentia savings which were not agreed by benefits owners, such as those projected for BBC WordWide, were excuded from the cost-benefit case. Agreed savings were to be removed from Divisiona budgets; re-vaidating the benefits with divisions and getting owners to sign up to deivering savings against a specific timeframe; and estabishing a minimum basis for how most of the benefits wi be tracked and measured. 4.8 The majority of financia benefits, 51.2 miion, wi be tracked via the BBC s current efficiency programme where Divisions are responsibe for deivering savings and detaied pans for each Division are currenty being estabished.

44 32 Part Four The BBC s management of its Digita Media Initiative Areas where the BBC shoud strengthen its approach 4.9 Where a programme does not deiver as panned, it is good practice to revisit the investment case to confirm continued business justification and acceptabiity of the risks. The risk of not carrying out an investment review or in taking an incrementa approach to continuing with deivery is that a programme becomes more and more embedded and aternative deivery options, incuding the stop option, are no onger reaistic When responsibiity for deivering the Programme was brought in-house in Juy 2009, the Programme was in difficuty and behind schedue, the deivery method had changed fundamentay, the chaenge in terms of the BBC s in-house capabiity increased and the financia risks transferred to the BBC. The Programme was taking on a very different shape from the one originay approved by the Trust. It woud have been prudent for the BBC and the BBC Trust to revisit the investment case at this decision point. The Trust was not formay asked to re-review the investment case for the Programme unti approva for additiona funds for wider ro-out of the Programme was sought in Apri 2010 (paragraph 4.5). In particuar: the BBC did not document the advice it received from consutants on the estimate of the cost to compete the Programme deveopment at the time it took the work in-house (paragraph 2.15); and the BBC did not commission an independent assurance review of the system design unti October 2010, over a year after it took the deveopment work in-house (paragraph 3.22) 4.11 When the BBC Finance Committee gave approva in November 2009 for the programme to continue pending re-assessment of the benefits, there was no requirement to seek the BBC Trust s approva because the additiona costs were to be met from the financia arrangements agreed with Siemens foowing the termination of the contract in Juy As additiona BBC money was not required, the narrow financia threshod for Trust re-approva was not triggered. The Trust was aware of the pan to use the settement money to fund the Programme, but the prudent course woud have been to seek the Trust s forma consideration and approva.

45 The BBC s management of its Digita Media Initiative Appendix One 33 Appendix One Methodoogy Method Review of the investment case for carrying out the Programme (2008) and the case for increasing the scope of the programme (2010) against recognised good practice. Review of the Programme documentation incuding: Purpose Assess the investment appraisa and decision-making process behind the Programme. Assess and understand the programme management of the Programme. Finance Committee minutes and reports to the Finance Committee; Steering Group minutes; Programme Management Office reports on progress; the contract between Siemens and the BBC for the Programme; and externa reviews of the programme commissioned by the BBC. Interviews with key stakehoders in the programme. Commissioned a review of the current position of the Programme and the risks it currenty faces. Assess and understand the programme management and investment appraisa process of the Programme. Assess and understand how the BBC has deveoped the technoogy for the Programme against industry good practice.