Requirements Gathering. Hazard Analysis. Configuration Management

Transcription

1 !" # $

2 Requirements Gathering Hazard Analysis Configuration Management

3 %&!'! () ) *! & ) )!!

4 !" +&,(),!) #,'! -) &() ) '. /!! ' 0),! &!

5 !

6 (). &.! %, & # " 1! 2!() 3 &45. 6! /!6 )! &! &!,6),!,. &())!! 7!,!,$

7 &,%, & Requirements Gathering Hazard Analysis Configuration Management

8 !, & 8) )! # () 9! :!!*. ;<!,

9 $%&

10 $%& Linear accelerator system built for cancer therapy Instrument was further advancement of earlier models (controlled entirely through software) 11 Units installed in US / Canada; hundreds of patients treated (thousands of treatments) Mechanism: radiation beam destroys cancer tissue Electron beam treats shallow tissue X-rays penetrate deeper, minimal damage to overlying area X-rays produced by hitting metal target with high-energy electrons Six overdose accidents (3 fatal): June 1985, July 1985, December 1985, March 1986, April 1986, January 1987 Overdoses (~100x intended dose, ~20x lethal wholebody dose) traced to two specific software errors

11 $%& Single electron gun produces both modes In x-ray mode, electron energy must be ~100x higher (target is a good attenuator) Low energy + target = underdose High energy + no target = huge overdose

12 $%& System: Cryptic error messages; errors and malfunctions common Operators could, and often did, override Treatment Pause Did not produce audit trail that could help diagnose problems Relied entirely on software removed electromechanical safety interlocks that were in previous models Quality Practices: Little documentation during development; no problem tracking after release Minimal unit and integration testing QA was primarily 2700 hours of use as integrated system Failed to look for root causes - seized on each issue as the problem Treated requirements / design / directed testing as troublesome afterthought

13 ' ( September 16, 2008: Automated External Defibrillator recalled Device was configured with the incorrect software, for semi-automatic instead of fully automatic use. When needed for a cardiac arrest emergency, device will require user to press the shock button instead of automatically delivering a shock (but the shock button is covered). August 2, 2008: MultiLeaf Collimator recalled Charged-particle radiation therapy system. Dose Calculation Error: Software anomaly may result in failure of a MLC leaf to reach planned position, potentially resulting in misadministration of dose to a patient. July 29, 2008: Cardiac Resynchronization Therapy Defibrillator (CRT-D) recalled Timing Feature Error: An implementation error allows the usage of the V-V timing feature, which is not approved for use in the U.S.

14 ' ( March 6, 2007: AEDs recalled Self-test software may allow a self-test to clear a previously detected low battery condition September 2006: Infusion pump Touch-sensitive keypad used to program the pump sometimes registers a number twice when it has been pressed only once ( key bounce ). Thus the pump would deliver more than the intended amount of medication, leading to over-infusion and serious harm or death to the patient. June 6, 2006: Ventilators recalled Older generation software - incorrect oxygen cell calibration (without compressed air supply) - can disable all alarms March 6, 2006: Dialysis device recalled Class I recall of dialysis device (11 injuries, 9 deaths): excessive fluid loss may result if caregiver overrides device's "incorrect weight change detected" alarm. (Device used for continuous solute and/or fluid removal in patients with acute renal failure.)

15 ' ( June 2005: Glucose Meter If dropped or operated in battery-low condition for extended time, meter may set readout to different units. March 15, 2005: Infusion pump fails if receives data Serial port on back allows nurse-call or hospital-info system to monitor pump. If external system sends data to pump, failure condition 16:336 may result; if during infusion, pump must be powered off and restarted. August 2004: Software card for programming infusion pumps Software Application Card was used with an external programming device to set up parameters on specific models of implantable infusion pumps. Users could mistakenly enter a periodic bolus interval into the minutes field, rather than the hours field, resulting in deaths and injuries due to drug overdose. Numerous adverse events occurred, including two patient deaths and seven serious injuries.

16 ' ( June 2001: Radiation treatment planning software Software used to calculate dose duration for radiation treatment of cancer would allow use of no more than four protective blocks (stated in the user guide). Physicians at Natl. Cancer Institute in Panama devised a way to "fool" the software into using five blocks, by entering data as if they were a single shape. If coordinates entered a specific way, the calculated dose would be as much as twice that intended. Users did not confirm calculated results; at least five patients died as a direct result of overexposure to radiation.

17 ) #*"'+, ) )=!*) % >)?,+@ 8

18 -. CDRH study: 2792 total medical device recalls, recalls (6% of total) software-related Of the 165: General Device Recalls Device Software Design SW Chg Control post-launch Mfg Process SW Design 133 (81%) Inadequate software design 32 (19%) Post-launch software change control

19 /0#*" D. Wallace & R. Kuhn (NIST, 1999): 383 SW-related device recalls Manufacturer recalls No deaths or serious injuries Data only from FDA records Could only classify fault type for 342

20 1 "

21 $- "23

22 1-

23 4 Practices: both for error prevention and error detection Complete specification of requirements Expertise in application domain Attention to details of current process Mental execution of trouble spots Change impact analysis Defect tracking and analysis Employing checklists, questions, methods to force symptoms to occur Software configuration management Traceability of all artifacts Traceability & CM of all changes

24 8&&, & 2 () " 7 7. () &)&

25 (). &.! %, & +)!, & & )!6) &! &)!)

26 # 8 Requirements Gathering Hazard Analysis Configuration Management

27 Have you ever tried driving to a new destination without a map? If you don t know where you re going, how will you know you got there? A sailor without a destination cannot hope for a favorable wind. - Leon Tec, M.D.

28 9 &,!# '! + #$ % # (! * " # )

29 /. 3 ;!() & )! = 3:)8 )45 End users / marketing find it difficult to express wants / needs Some (many?) requirements are assumed Relationships and ramifications can be tricky to imagine up front Language is often vague, ambiguous, or imprecise True needs vs. would-be-nice are difficult to distinguish Sometimes, requirements are too explicit, and limit choice of technology

30 ; < User requirements - description of the tasks a user must be able to perform by use of the product Functional requirements - specific behaviors the software needs to exhibit

31 =" Requirements Specifications

32 >8 +!;! 8; ) 4

33 Characteristics of an SRS: Correct (every item is one S/W shall meet) Unambiguous Complete Consistent (i.e. internally) Ranked for importance / stability Verifiable Modifiable (structured, not redundant) Traceable

34 B Tests for each Requirement: Complete (source, rationale, identifier, use case, type, fit criterion) Traceable Consistent (terminology used uniformly) Relevant (to product purpose) Unambiguous (measurable fit criterion) Viable (within product constraints!) Solution Neutral Also tested: gold plating and creep

35 $! Measurable: Quality measure to gauge any solution Coherent: Definition of every essential subject matter term Consistent: Every use a defined term consistent with definition Complete: Wide enough context of the requirements; included conscious, unconscious, undreamed of requirements Relevant: Every requirement relevant to this system Solution Neutral: No solutions posturing as requirements Traceable: Each requirement uniquely identifiable; Each requirement tagged to all parts of the system where used Ranked: Stakeholder value defined for each requirement Risk Assessed: Hazard level stated, were the software not to fulfill the requirement

36 +$/ $C Focus on the purpose, not on the implementation You don t need: A web interface Username / password login Radio button to select age group You DO need: Access for users outside the LAN Access only to authorized users Only one age group assigned each patient record

37 > 1 ),& /!,

38 0 Where to search for requirements? Clinicians (users) Other technical experts Design / Manufacturing Engineers Developers Quality specialists Regulatory experts Professional bodies (standards) Adjacent systems Management Inspectors

39 + C " Who are the actors? What is the system s boundary? OTS software: which features are actually used? Both normal and exception cases included? Everything from user / actor point of view? Do we understand the user s job?

40 +*" + () 2 & "!2!,6&)!&!6!,!)! A1! B! *

41 1 +) 2. The system shall store a list of all gauges on the calibration program 3. The system shall maintain information about each gauge 3.1. Each gauge shall have a unique identification 3.2. For each gauge, the system shall store its type, the property it measures, the units of measurement, and tolerances at up to five calibration points The system shall permit setting high and low tolerances independently of each other The system shall permit setting tolerances on only one side (high or low) 4. The system shall maintain a schedule of calibration due date for each gauge 5. The system shall provide means for a user to enter calibration data, and apply pass/fail criteria to the data entered.

42 1-9 User Security R_S1 The metrology lab needs to limit use of the calibration system to authorized users. R_S2 The metrology lab needs to restrict specific actions according to the employee's job function. Gauge Data R_D1 The metrology lab needs to maintain an up-to-date list of all gauges. R_D2 The metrology lab needs to store properties, calibration dates, and calibration data for each gauge. Gauge Operations R_O1 The metrology lab needs the ability to set and track status of each gauge (in lab, checked out, out for repair, out for calibration, inactive). R_O2 R_O3 R_O4 R_O5 The metrology lab needs to be able to calibrate any gauge which is considered available (i.e. in lab and Master Gauge calibration is in date.) The metrology lab needs to have the calibration routine set a calibration to PASS only if all measurements are within tolerance. The metrology lab needs to prevent any gauge from being checked out for use if a calibration has not been carried out and passed within the calibration period. The metrology lab needs to record a history of all calibrations and status changes for every gauge.

43 1 8 Instead of a series of shalls, describe functional behavior for a series of situations Much easier to link privileges, functions, prerequisites, error responses Training is worth the time/expense; Use Cases are not always intuitive to construct

44 1 8 Gauge Calibration ID CAL1 Description Perform routine gauge calibration Depend DAT4 Actor Calibrator Input Enter measurement at each defined calibration value. Record any special circumstances or observations. Note that system automatically records user name and date. Output If all measurements fall within defined tolerances: Display calibration succeeded message and print report Save measurements, date, user name Calculate next calibration date from interval and today s date Error If any measurements fall outside tolerances: Display calibration failed message, set gauge status = NOT CAL

45 .4 " %>)" *() = %>)+@=

46 (). &.! %, & +)!, & & )!6) &! &)!) 1! 2!() 3 &45 " " $

47 Requirements Gathering Hazard Analysis Configuration Management

48 . 0 ) 9. 2! & &! & = )), & )&, = ;! 2(). 6 6)@

49 .4 " ;() = ; ) :!=

50 ' C) D?!43 5 E1$F35%!7 "!:!&& G!.#9#9#+HFI #8EIJKE G! )!&! % :/!

51 Change is inevitable Lack of CM is frustrating, inefficient and may be harmful CM controls and coordinates work products of many different software engineers on common project Simultaneous Update Double Maintenance Shared Code and Work Products Common Code Versioning

52 2 Product attributes defined Product configuration documented Correlation with requirements Changes identified and evaluated for impact Change activity managed using a defined process Configuration information stored and organized for retrieval

53 # We don't lose" elements introduced to mitigate identified risks * ' Involves systematically managing and controlling changes to item baselines and software releases Ensure accurate and timely changes to configuration documentation Eliminate unnecessary change proliferation

54 4 #) &)!!)

55 .4 " 1! % )8&&). ; 8' *= %),8:&.

56 () ;! &4 3 )8:&5 ; ) 3)! EE=5

57 2

58 !. Version Control Basic requirement Ensures repeatability and traceability Configuration Support Mechanisms enable user to identify software system correctly Tracks relationships between items in configuration between components Library/Repository Captures SCM information and stores versions of items Mechanisms create audit trail of all SCM activities and database transactions

59 !. 5 =7 Reporting/Query Dependency Report Impact Report Build Report Change Status Report Difference Report History Report Access Control Report Conflict Detection Report Query capability supports configuration audits and collection of metrics

60 !. 5 =7 Security/Protection Allow user to define varying levels of access Provide protection from data corruption and loss Provide backup, archive, and restore capabilities Build Support Track each item and information about each item that comprise the build Implement own build facility Cross Development Builds Parallel Builds Distributed Builds

61 Baseline Specification or product formally reviewed and approved to serve as basis for future development Must be changed only through formal control procedures Configuration Control Board Group responsible for evaluating and approving proposed changes to configuration items Configuration Identification Selection and recording configuration items and characteristics for system Configuration Item Aggregation of hardware, software, or both treated as single entity and designated for configuration management

62 5 =7 Configuration Status Accounting Consists of recording and reporting of information needed to manage configuration effectively Patch Interim software fix pending final design fix in source code Release Formal notification and distribution of approved version Software Library Controlled collection of software and related documentation designed to aid in software development or maintenance Software Life Cycle Period of time begins when software product conceived and ends when software no longer available for use

63 -D Best practice is a four part number <major version>.<minor version>.<build number>.<revision> Major The internal version of the product that rarely changes Minor Usually an incremental release of the product Build Changes with every new build of the code Revision Can have several meanings such as bug number, predecessor build, service pack etc. Example: Version Major version for Win XP 2600 is the Release to Manufacturing build number 2180 is service pack 2

64 2 Process of creating application binaries for useable software release Releases can be intermediate for testing or final Inputs specified Structure of Product Items in Product Environment Items Good practices Build early and build often Automate the build process Check all files before build Increment build number every single time Don t use dates in the version numbers Don t use marketing labels for the versions

65 . Maintain continuous records of status of all baselined items Provides mechanism for disaster recovery Typical information required Time baselines were established When each item included in baseline When each change added to baseline Description of each configuration item Status and description of each software related change Documentation status for each baseline Changes planned for each future baseline

66 4 8$ Version Description Document (VDD) Release Notes Hardware/Software Compatibility Matrix 4 Problem capture and tracking Patches and interim updates Customer support

67 "" ;?, &&,&) ; &! ) )

68 (). &.! %, & +)!, & & )!6) &! &)!) 1! 2!() 3 &45. 6! /!6 )! & 3

69 +,. Requirements Gathering Hazard Analysis Configuration Management

70 +,. %!-./!!0 1 2!-!

71 ., Hazards identified through systematic methods (FMEA, FMECA, or FTA) Requirements: become more refined as design evolves Hazards: evaluate repeatedly throughout project Systematic analysis: best if we know the design Hazard mitigation: changing or adding to requirements

72 -.33", Don t necessarily need to know details of system For each hazard, ask why it would occur Include combination circumstances For each immediate cause, ask why again Trace all the way down until root causes are identified

73 #*"-. %3 4/3 5 3)0 65 "3#! " " 7!!!!! ;"$< 3 # # # 98 58#!3!! /.8! : 8! 7 / "! "!

74 -#.2"" Failure mode and effects analysis: requires some knowledge of the system s architecture For each component (object, function, subsystem), ask how it might fail and what the effect would be Generally, consider only single point failures (not combinations of failures) Each identified failure mode need to trace to a system-level hazard

75 #*"-#. Failure Mode Effect Causes S1 Mitigation S2 Sample ID / results array off by one Wrong results reported Inconsistent array logic; incorrect initialization 5 Optional operator approve results before saving 2 Initialization fails to warm up lamp Can t perform analyses Startup logic can be set to skip steps and left that way 4 Reset all startup parameters on initialization 1 Dilution factor associated with pipet tip, picked in advance Wrong result reported (dilution applied to wrong sample) Counting logic not rechecked when pickin-advance process introduced 5 (a) Track dilution and pipet tip separately; (b) show dilution with reported result 1 S1 = Severity rating before mitigation; S2 = severity rating after mitigation Severity (sample values only): 5 = critical; 1 = nuisance Note this analysis does not include two of the standard engineering estimates: occurrence (probability) or detection.

76 +,") * Direct failure Software flaw in normal, correct use of system causes or permits incorrect dosage or energy to be delivered to patient. Permitted misuse Software does not reject or prevent entry of data in a way that (a) is incorrect according to user instructions, and (b) can result in incorrect calculation or logic, and consequent life-threatening or damaging therapeutic action. User Complacency Although software or system clearly notes that users must verify results, common use leads to over-reliance on software output and failure to crosscheck calculations or results.

77 +,") * User Interface confusion Software instructions, prompts, input labels, or other information is frequently confusing or misleading, and can result in incorrect user actions with potentially harmful or fatal outcome. Security vulnerability Attack by malicious code causes device to transmit incorrect information, control therapy incorrectly, or cease operating. No examples in medicaldevice software known at this time, but experience in personal computers and "smart" cellular phones suggests this is a serious possibility.

78 # D 11.+! :, ) ) % ) ) 18,8,L; ;)!#&=L >) )!)A& B

79 ."" #8EIJKE7!, & & )!!6,!) 6!,() #+HFI7 ) &.#;H!!&!!)! &%06& EIJKE.##F7!, ; AB ;HEIJKE ) M &!6 )! 6,&,&EIJKE()!2#+HHE/E6*0EJJ

80 0)EFGHE 3 Sets forth plan: Identify hazardous situations and evaluate Implement control measures Check remaining risk for acceptability Track information post-launch Annexes (larger than standard itself) provide questions to ask, list of example hazard areas, methods of control, descriptions of analysis techniques

81 Compare the development lifecycle... Customer Needs Activities outside the scope of Customer Needs Satisfied SYSTEM development ACTIVITIES (including RISK MANAGEMENT) 7 Software RISK MANAGEMENT 5.1 SW Devel Planning 5.2 SW Rqmts Analysis 5.3 SW Architect design 5.4 SW Detailed design 5.5 SW Unit Implem & verif 5.6 SW Integrn, Int Tstg 5.7 SW System Testing 5.8 SW Release 8 Software CONFIGURATION MANAGEMENT 9 Software problem resolution

82 With the MAINTENANCE lifecycle - Maintenance Request Activities outside the scope of Request Satisfied SYSTEM maintenance ACTIVITIES (including RISK MANAGEMENT) 7 Software RISK MANAGEMENT 6.1 Estab SW Maint Plan 6.2 Prob & modificn analysis 5.3 SW Architect design 5.4 SW Detailed design 5.5 SW Unit Implem & verif 5.6 SW Integrn, Int Tstg 5.7 SW System Testing 5.8 SW Release 6.3 Modification Implementation 8 Software CONFIGURATION MANAGEMENT 9 Software problem resolution

83 EFGHE " * Topics include: Causal chains, first & last points of software control Direct and indirect causes (with tables of examples) Risk control measures for software-related hazards Architecture, design, coding approaches to minimize hazards Types of maintenance

84 +, Direct hazards are a natural outcome of the interactions between the design and the outside world Indirect hazards come about specifically because of how the item is designed!

85 +,) Collaborate with users and domain experts to identify them Create data and software with the capacity to inject hazards into the system (external simulation or comprehensive mock object) Coverage based both on external systems and on actions of users Do this instead of up-front Analysis Paralysis!

86 +, Unit test unit test unit test! Keep unit tests and acceptance tests up to date! Automate testing to keep frequency and coverage both high Refactor to keep design as simple as possible

87 (). &.! %, & +)!, & & )!6) &! &)!) 1! 2!() 3 &45. 6! /!6 )! &! &!,6),!, 1 =

88 . Requirements Gathering Hazard Analysis Configuration Management

89 > )8" ()+ +: 8!60!6!1 ))0

90 # Changes in Use Field Issues Enhancement Requests Defect Tracking CAPA Requirements Revision Design Updates Configuration Management Regression Testing Hazard Re-Analysis Impact Assessment Mini-SDLC

91 #4 +. & () !/)M () 3'=5.99!&, '! +@., &! &,

92 +,0 C) ) & )%)%,! /%0& )!)& )! * &!@! ) '& )&

93 >' 5E7 FDA: Design Control, Medical Devices (March 11, 1997) FDA: General Principles of Software Validation (Jan 11, 2002) FDA: Premarket Submissions, Software Contained in Medical Devices (May 11, 2005) FDA: Off-The-Shelf Software Use in Medical Devices (Sep 9, 1999) FDA: Cybersecurity for Networked Medical Devices, OTS Software (Jan 14, 2005) FDA draft guidance: Radio-Frequency Wireless Technology in Medical Devices (Jan 3, 2007)

94 >' 5%7 AAMI TIR32:2004 Medical device software risk management IEC 60812:2006 (2 nd ed) Analysis techniques for system reliability Procedure for failure mode and effects analysis (FMEA) IEC : 2005 (3 rd ed) Medical electrical equipment Part 1: General requirements for basic safety and essential performance ( Programmable Electrical Medical Systems is available standalone, but will not be in the future) IEC 62304:2006 Medical Device Software Software Life Cycle Processes ISO 13485:2003 (2 nd ed) Medical devices Quality management systems Requirements for regulatory purposes ISO 14971:2007 (2 nd ed) Medical devices Application of risk management to medical devices

95 - FDA, CDRH Software Related Recalls for Fiscal Years (May 1992). Institute of Electrical and Electronics Engineers, "Recommended Practice for Software Requirements Specifications" (IEEE ), 01-May Robertson, James and Suzanne, Mastering the Requirements Process, Harlow (England), Addison-Wesley / ACM Press, Schneider, Geri, and Jason P. Winters, Applying Use Cases: A Practical Guide, Harlow (England), Addison-Wesley / ACM Press, Therac-25 Investigation: Leveson, N.G., Safeware - System safety and Computers. 1 ed. 1995: Addison-Wesley Publishing Company Inc. Revised version at: Wallace, D., R. and D.R. Kuhn, Lessons from 342 Medical Device Failures. in 4th IEEE International Symposium on High-Assurance Systems Engineering Washington, D.C.: IEEE. (medical device software failure study) NIST version at:

96 : 6600 N($ $($ OO) 6,6.EIO JK/JHO/EEE 6:$%$ :!)! 6 EJJ 6%.H KE/JJ/OJK N $ 29$ $!" G : 6 M%,! 6 # $ EE,$6 $K %6JEE!$N $ 29$ $