Case Study : Software Disasters -1

Transcription

1 Case Study : Software Disasters -1 WELL-KNOWN: DENVER INTERNATIONAL AIRPORT BAGGAGE HANDLING SYSTEM BY DHRUV P & UDIT P erairporttransportation.com/images/dia.jpg

2 Background City of Denver 1990 New state-of-the-art airport 140km2 area Capacity : > 50 million passengers annually Largest in the USA Critical component : Automated Baggage handling System Reduce aircraft turnaround time by 30 minutes TPF/wp-content/uploads/2008/12/denverbag5.jpg

3

4 Baggage handling system at a glance 88 airport gates in 3 concourses 17 miles of track and 5 miles of conveyor belts 3,100 standard carts oversized carts 14 million feet of wiring Network of more than 100 PC s to control flow of carts 5,000 electric motors 2,700 photo cells, 400 radio receivers and 59 laser arrays

5 Causes of project failure Became a public humiliation for everyone involved Airport opening delayed by 16months Cost the city $1.1M per day throughout the delay On the opening day, System was used in just 1 concourse by single airline and only for outbound flights All other baggage was handled using conveyer belts, manual tug and trolley system In 2005, United airlines abandoned the system

6 Chronological Events Time Nov, 1989 Oct, 1990 Feb, 1991 June, 1991 Summer, 1991 Event Work starts on the construction of the airport City of Denver engages Breier Neidle Patrone Associates to analyse feasibility of building an integrated baggage system. Reports advises that complexity makes the proposition unfeasible Continental Airlines signs on and plans on using Denver as a hub United Airlines signs on and plans on using Concourse A as a hub Airport s Project Management team recognizes that a baggage handling solution for the complete airport was required. Bids for an airport wide solution are requested

7 Chronological Events Time Fall, 1991 Early 1992 Apr, 1992 Event Of the 16 companies included in the bidding process only 3 respond and review of proposals indicate none could be ready in time for the Oct 1993 opening. The 3 bids are all rejected Denver Airport Project Management team approach BAE directly requesting a bid for the project Denver Airport contracts with BAE to expand the United Airlines baggage handling system into an integrated system handling all 3 concourses, all airlines, departing as well as arriving flights. In addition system is to handle transfer baggage automatically. Contract is hammered out in 3 intense working sessions

8 Chronological Events Time Aug, 1992 Sept, 1992 Oct, 1992 Event United Airlines changes their plans and cuts out plans for the system to transfer bags between aircraft. Resulting changes save $20m, but result in a major redesign of the United Airlines portion of the system. Change requests are raised to add automated handling of oversized baggage and for the creation of a dedicated ski equipment handling area Continental requests ski equipment handling facilities be added to their concourse as well Chief Airport Engineer, Walter Singer dies. Mr Singer had been one of the driving forces behind the creation of the automated baggage system

9 Chronological Events Time Jan, 1993 Event Change orders raised altering size of ski equipment claim area and adding maintenance tracks so carts could be serviced without having to be removed from the rails Feb, 1993 Target opening date shifted from 31 Oct 93 to 19 Dec 93 and soon thereafter to 9 Mar 94 Sep, st Oct, th Dec, 1993 Jan, 1994 Target opening date is shifted again, new target date is 15 May 1994 Original target for opening Second target for opening United Airlines requests further changes to the oversize baggage input area

10 Chronological Events Time 9th Mar, 1994 Mar, 1994 Apr, 1994 Apr, 1994 Event Third target for opening Problems establishing a clean electrical supply results in continual power outages that disrupt testing and development. Solution requires installation of industrial filters into the electrical system. Ordering and installation of the filters takes several months Airport authorities arrange a demonstration for the system for the media (without first informing BAE). Demonstration is a disaster as clothes are disgorged from crushed bags Denver Mayor cancels 15 May target date and announces an indefinite delay in opening

11 Chronological Events Time May, th May, 1994 May, 1994 Event Logplan Consulting engaged to evaluate the project Fourth target for opening BAE Systems denies system is malfunctioning. Instead they say many of the issues reported to date had been caused by the airport staff using the system incorrectly Aug, 1994 City of Denver starts fining BAE $12K per day for further delays 28th Feb, 1995 Aug, 2005 Actual opening In order to save costs the system is scrapped in favour of a fully manual system. Maintenance costs were running at $1M per month at the time.

12 Reasons of Failure : Strategical Changes United Airlines insisted automated high speed baggage system Denver agreed to build for all concourses Assumed : Individual airlines would make their own baggage handling mechanism Prior to this, every airline used to build their own baggage handling system Change : in Summer 1991, Project Management Team realised: in-order to build an integrated system throughout the airport, they need to take the responsibility back from the airlines. Decision was taken little more than 2 years prior to the opening date Schedule pressure on the team

13 Reasons of Failure : Strategical Changes Manual baggage system was rejected Size of airport : not feasible to handle baggage on time Labour intensive and expensive process Diesel powered tugs would travel through poorly ventilated tunnels Chock the tug driver Tugs would jam the tunnel when they pass each other or take turns

14 Reason of Failure : Yet, they proceed 3 Red Lights not to proceed : Yet, they proceed The 1990 Breier Neidle Patrone Associates report indicated the complexity was too high for the system to be built successfully Analysis of the three bids received indicated that none of the vendors could build the system in time for the Oct 1993 opening Experts from Munich airport advised that the much simpler Munich system had taken 2 full years to build and that it had run 24 / 7 for 6 months prior to opening to allow bugs to be ironed out

15 Reason of Failure : Yet, they proceed BAE was not initially chosen for bid 3 official bids were rejected Airport team contacted BAE BAE didn t have prior experience Yet they accepted it!!! Big opportunity to grow the business by securing other big contracts around the globe BAE started working on specification and prototype Prototype misled Said to have filled up sq ft warehouse Demo convinced Slinger that project was feasible BAE senior management team only communicating with Slinger Voices concerning the feasibility of the system was unheard

16 Reason of Failure : Yet, they proceed Slinger and BAE underestimated the complexity of the project They didn t take the expert advise Slinger, as a civil engineer might not understand the mathematical complexity of the problem Slinger, responsible for entire airport had other considerable amount of duties Pressure to move quickly would have caused them to put the reasonable thinking aside Belief : manual system won t work because of the airport size

17 Reason of Failure : Over Commitment BAE agreed to deliver the project within the fixed scope, schedule and budget Contractual conditions and scope was decided in just 3 sessions 3 sessions were clearly not enough to get the complete understanding of the scope and risk Airlines were major stakeholders Excluded from the key decisions discussions This is always a losing strategy When they were consulted, Asked for significant changes which caused a severe rework or negation of previous work

18 Reason of Failure : Over Commitment BAE claimed to deliver the project in 2 years time Really needed 3 to 4 years Tight schedule Team would have focused on Happy Path scenarios And less time thinking on making the system fault tolerant Full simulation of the system was avoided Full testing of the complete system was not completed Training schedule was cut down

19 Mitigation Plan Do not commit something you cannot deliver Consult with external experts such as Logplan as BAE didn t have prior experience Defining activities with the resources and duration in sequence could have given the appropriate schedule Testing Less coverage and less scenarios covered Major bugs cannot be identified

20 Reason of Failure : Ongoing Changes Project was already in trouble In original negotiations, BAE : no changes would be made Meeting stakeholder s need became too strong accepted the changes Changes: Add ski equipment racks Add maintenance tracks : Cart can be serviced without being removed from the tracks Changes to oversized baggage handling Acceptance of changes indicates No clear understanding of impact Or communication gap

21 Reason of Failure : Ongoing Changes Changes to requirements/design was carried out by DIA without consulting the airlines or BAE. The same situation occurred with the schedule. BAE failed to appreciate change impact this was due to the complexity of the system and its tightly coupled nature. As the project was slowly implemented, it swelled in complexity. When design changes were made, this only made things worse. Centralized system faults, which ran through many of the baggage system s subsystems, were discovered. To solve BAE decentralized all of the tracking and sorting computers.

22 Reason of Failure : Ongoing Changes By far, the biggest change was caused by United Airlines penny pinching. They removed a complete loop of track; having just one loop. Saved $20 million. Required a complicated redesign at a critical phase of the project. Other major changes included relocating the outside stations, Continental Airlines larger baggage link and another mezzanine baggage platform. Due to time constraints, many of the problems which required substantial redesign were covered over by quick fixes. No time was given to investigation of other solutions.

23 Mitigation Plan Communication should include all the team members and stakeholders Critical findings should be escalated To make the system more modular and less tightly coupled, Patchwork could have been avoided and development life cycle had followed to resolve them and SE rule had followed All stakeholders could have discussed and brainstormed at requirement gathering sessions

24 Reason of Failure : Incompetence with the physical building Design of building was initiated prior to the baggage automation Building designers made space for the baggage handling based on their prior experience Baggage system team was forced to work within the constraint of the physical structure of the airport Change of structure would cost : $100M Sharp turns : bags ejected from carts Speed of the carts was halved from 60car/min to 30 car/min Performance was degraded

25 Mitigation Plan Open communication strategy between DIA and airlines could have avoided the decision about who would design the baggage system Never make an assumption Decision and construction of the automated baggage system and building should have started at the same time. Building team could have consulted/work along side with baggage team for the baggage space related design work PMT failed to identify the interface(allowance of space) between baggage team and building construction team

26 Reason of Failure : Take Alternative In Apr, 1994, the Mayor of Denver recognized that project was on fire Logplan was consulted Recommended to take the alternative and build the manual system with trolley Additional cost of $51M Project was already 6 months behind the schedule

27 Mitigation Plan Such complex, big and risky project must have Frequent reviews Expert assessments

28 Reason of Failure : Incomplete Testing Computer simulation was avoided Line-balancing problem was detected 6 months after airport was opened Whole system was consisted of more than 100 queues Due to tight schedule, testing time was reduced Munich airport spent 2 years in testing BAE ignored this fact Test engineers unable to communicate on radio channel because of dead spot in radio transmission around the airport Shared large power supply caused the fuses to trip, stopping the motors Installed industrial power filters took several months

29 Mitigation Plan Running the full computer simulation could have avoided line balancing problem Simulation could have been used to see the redesign impacts Spending more time for testing Consider cases for the similar projects and learn from them such as Munich Take expert advise For issues such as radio channel communication and power fluctuation, Focus on risk management activity would have cover these scenarios

30 Reason of Failure : Reliability of the Components Barcode reader are not accurate every time Photocell quality and their adjustment caused the computers presume that there was a telecare jam Faulty latches dump luggage on the tracks or jammed against the side of the tunnel Airflow flipped light or empty bags out of their telecars Sharp turns high stress area tracks frequently broke

31 Mitigation Plan Should have investigated multiple redundancy Considered multiple failure scenarios

32 Reason of Failure : Politics Oct 1992, Chief airport engineer Walter Slinger died Gail Edmond became new project head Not given same power of autonomy BAE access and involvement restricted, resulting in difficulties

33 Mitigation Plan More efficient and expert management team should have been assigned from the beginning. Decision making should involve by all the management team members not just by a project head as in the case with Slinger

34 Reason of Failure : Failure Recovery BAE didn t have any backup system in case of failure Alternative tug and cart system implementation caused additional $51M There was no way of reliving strain from the system No alternate ways for baggage

35 Mitigation Plan Alternative system should exist in the case of automation failure It should have been defined at the requirement stage

36 Conclusion Never underestimate the mathematical complexity of the problem Spend time in planning activity Spend time on risk management and mitigation plan drawing activity Spend time on design of the system Do not give over commitments in the face of massive risks and uncertainty Make clear communications across the team and keep record of each communication Understand and consider the implications of the change requests If required, take them as separate iteration Include stakeholders in key decisions

37 WELL-KNOWN: VIRTUAL CASE FILE SYSTEM. Google Images

38 Virtual File System $ 378M UAC VCF FBI Information Technology Upgrade Trilogy Hardware Infrastructure Network Infrastructure

39 Requirement of the UAC Integrate five most legacy systems of the FBI Automated Case Support System. IntelPlus. Criminal Law Enforcement System. Integrated Intelligence Information Application. Telephone Application. Provide point and click web based interface. 09/2001

40 9/11 even affected FBI In midway of VCF, the 9/11 Commission revealed all the loopholes of the FBI hardware and software infrastructure. Report stated FBI s information system were woefully inadequate. There was no effective mechanism for capturing or sharing its institutional knowledge. In the face of intense public and congressional pressure, team of IT, accessed the web-interface developed by SAIC. Team realized that letting the agents to just have point and click wont do much help. 11/2001

41 Introduction VCF was supposed to automate the FBI s paper-based work environment, allowing the agents and intelligence analyst to share the vital investigative information. Ubiquitous nature of data. Access from anywhere. No existing commercial software package was available. Science Application International Corp (SIAC) was the company who TRIED to developed it by June /2000

42 FBI wanted Big (in middle of UAC) Pivot Entirely new centralized database. New Graphical User Interface. Now VCF would combine ACS with other two applications only The Telephone Application. Law Enforcement Application. Multi-media Capabilities. Allowing agents search across various investigation and would be accessible to both, the FBI agents as well as intelligence analysts. (Including more actors).

43 Joint Application Development. To formalize, requirement for VCF, SIAC embarked on a serious of JAD sessions. Agents and experts got to gather with a group of SIAC engineers to has out what functions the VCF would perform. JAD Session finally produced an exhaustive detailed requirement document (6 month long). SIAC agreed to deliver the initial version of the VCF in December 2003 instead of June 2004 i.e. within 22 months, using the flash cutover. 02/2002

44 What requirement specification said Matthew Patton was hired as an security engineer in the SAIC. After the failure, he was interviewed for the failure of VCF. Patton s description for the SRS, suggest that the project was going off the rails right from the beginning. Outcome of JAD => Tried to design the system, before even figuring out what they wanted the system to do The document violated first rule of the software planning: Keep it simple. SRS should describe what and not how. The SRS was of quality specifying details like there will be a page with a button saying on it. SRS had the description of the system layout, but devoid of what the system would do.

45 What requirement specification said (Cont d) JAD session resulted into a massy and overstuffed requirement document. Generally Customers say what the want and Consulting firm says how they will deliver. But one of the project manager in the SAIC, said that the failure to VCF was the culture of the FBI. The culture within the FBI was, We are going to tell you how to do it too. Overly Specific nature of the requirement focused developers on their tiny pieces of the puzzles. Patton added that, there were so many individual modules, that developers had no idea of how these pieces would fit with each other. Writing the code to develop applications, that were already present.

46 Before the start of VCF Matthew Patton, a real hero to capture the infeasibility of the VCF. Questioned the technical feasibility of the project. Aired the objection regarding the feasibility to the Web Discussion Board. Reward: A visit by two FBI agents concerning about disclosing the national security secrets. Result: SAIC and FBI agreed on deeply flawed 800-Page set of system requirement. 08/2002

47 Before the start of VCF (Cont d) Disagreements among the members for VCF Interview with the FBI agents regarding the information transfer from paper-based system to electronic-based system. Zalmai Azmi, raised the question on VCF over current system? Raises legal policy questions In an electronic world, nothing really is destroyed; it is always somewhere.. Concluded the reluctance among the users for the VCF.

48 Software Development In December 2002, the requirements were settled, and there was something to start on with SAIC adopted on a spiral development methodology, an iterative approach to developing the software. Build a particular function Validate by the agents if something is wrong sent an official request to the change control board. From December 2002 to 2003, 400 official notices were collected by the change control board. 12/2002

49 Software Development (Cont d) Comments on Validation by Agents Oh, we ve got to change this, That isn t what I meant to be. And that s how we started logging change request after change request after change request after change request.

50 One More Pivot For about 25% completion of the VCF, FBI wanted a page crumb capability added to all the screens. This functionality gives the user, a list of URLs identifying the path taken by the VCF to arrive at the current screen. One of the 8 development team, created the functionality as demanded and the design model and documentation were updated.

51 Audit Reports In September 2003, U.S. General Accounting Office released a report titled, FBI needs an Enterprise Architecture to guide its modernization Activities. Randolph C. Hite, added that, it was a classic case of not getting the requirement sufficiently defined in terms of completeness and correctness from the beginning. Randolph, advised to redesign the requirement that would be in compliance with what is already designed and produced. 09/2003

52 Late and Limited Blueprint Today many organizations rely on blueprint (enterprise architecture). To guide hardware and software investment decision. It describes an organization s mission and operations. How it uses the technology to accomplish its task. How the system is structured and designed to accomplish the task. According to Government Accountability Office, inspector general and NRC pointed out that FBI didn t had what it should have.

53 Reports says Incompetency with the real-world Environment 13,000 Computers could not run modern software. Most of the computers were just connected with the low speed INTRANET links of about 56 kbps. Network Components were not supporting the addressed protocols used in VCF. Conclusion Not supporting environment 09/2003

54 Unrealistic Contract FBI chose cost-plus-award fee contact This would include the cost of all labor not based on fixed requirements Additionally to include all the unforeseen cost. To encourage the developers, they used award fee contact.

55 The D-Day Comes On 13 th December, 2003 SAIC delivered the VCF to the FBI, only to have the declared DOA. FBI Rejected the VCF. 17 functional deficiencies. Big and small deficiencies Big: Not providing the ability to search for indiv iduals by specialties and job tittle. Small: Button on the GUI labeled as STATE should hav e been STATE/PROVINCE/TERRITORY. Arbitrator s findings (12 th march,2004) 59 issues and sub issues derived from 17 original deficiencies. 40 SAIC s error and 19 FBI s fault. 12/2003

56 Comments from SIAC Because of tight schedule: SAIC developers didn t ensure that all the programmers were making the change in the same way. This inconsistency occasionally meant that different modules of VCF handled data in the different way. Consequently, when one module needed to communicate with other, error sometimes occurred. Punaro said that this didn t compromised the system. The real killers were Significant Management turbulence. Ever-shifting nature of the requirement. Trail and error approach of the FBI Agents. We will know it when we will see it approach.

57 Time to heal Aerospace Corporation., California. Reviewed the VCF, in order to come up with what to do. SAIC to work on electronic workflow portion of VCF, and turn it into Initial operating Capability (IOC) ($ 16.4 million). Create a way to translate the output from the VCF System. Check out and improve the network performances. Develop training program. With new management in-place, 120 SAIC Engineers began work on IOC project in June Agreed on strict development schedule. Acceptance criteria. Series of control gates. 06/2004

58 Release date for IOC Completed before the schedule time Was an aid to the task of management. Did not improve the productivity of most users. (Internal FBI assessment). Fill the form electronically routed for approval To comply with paper based case management system print out, sign up and file the form. 01/2005

59 Factors Contributing to Failure Enterprise Architecture: Efforts and results in EA where late, limited and fall for short of what was required. How the interaction among the various applications needs to be automated? How the data among the application should flow? What about the security of data when it comes to various actors and applications? Justification for why 17 functional deficiencies and 4000 change requests during the development and lack of experts in enterprise architects.

60 Factors Contributing to Failure (Cont d) Transition Plan Tried to transfer from ACS (800 pound gorilla) to VCF in just one shot. Data migration was a major headache. No body knew how many interfaces where present and how they were defined. No Failure Recovery. No Real World Testing Mitigation Plan Phase roll out with Rapid Prototype Development approach. Smaller steps, more rapid done, rather than one big jump and make it all the way across Usability testing.

61 Factors Contributing to Failure (Cont d) Scope of the VCF Just concentrated on the end results. Failed to lookover the interoperability of the existing systems with VCF Made the scope of the VCF unrealistic and very wide. SIAC had an messy custom coded effort, trying to build all the functionalities, and expecting all to work flawless and perfect(developed by 8 teams). Mitigation Plan Commercial Software Integration.

62 Factors Contributing to Failure (Cont d) Project Management Some of them didn t had any prior experience in the project management. (As an when approach) Sherry Higgins: 29-year veteran of AT&T, previously handling the help desk at the technology Command and Control Center. Made Depew, as a VCF Project Manager. Concluded that FBI didn t had adequate human resources and skill base needed to deal with FBI s modernization project. All the well-wishers for the VCF were rewarded with resignation letter.

63 9/11 Attack Doubts on capabilities Improper start of the project Tensed Atmosphere VCF More Complex Poor EA Less time Flush Cutover Less experienced PM Failure

64