True Root Cause and Corrective Action (RCCA) Training

Transcription

1 True Root Cause and Corrective Action (RCCA) Training

2 Objective: To provide a project management approach, tools, and tactics for Teams to conduct successful True root cause investigations and define and implement effective and accountable corrective action plans that prevent re-occurrence. Empower more people and functions to lead and/ or more effectively contribute to root cause and corrective action (RCCA) projects

3 Features: Utilizes three primary tools: Project Rolling Action Item List (RAIL) Issue Closure Plan (ICP) Root Cause Verification Action Plan (VAP) Not all tools referenced are always going to be required. There is flexibility depending on the complexity and scope of the problem. The Corrective Action Board (CAB), Integrated Corrective Action Team (ICAT), and the project champion will offer guidance. The ICP should be utilized for most issues/ projects and serve as a company record. Less complex problems can outline and execute the full closure plan including verification actions within the ICP itself. Simple issues can be worked and documented within a CAT project tracking spreadsheet Ex. PCAS RAIL The Method works even for the most complex problems. The 8D sequence of the steps is critical and should not be cut short or skipped. The time each step will take is again a variable depending on the complexity and scope of the problem.

4 Features (cont d): Contains notes (highlighted Blue) for key guidance and warnings relative to conducting a successful process and accomplishing the best team performance Clarifies terminology involved in the process Company record of accountability Willingly, by name documentation Names recorded at each separate progress point since the players may be different

5 Features (cont d): This training is based on the HBC modified 8-D problem solving steps but with additional instruction offered to increase the focus needed to assure that a team captures true root cause and confidently prevents re-occurrence. Therefore, Step 4 has been expanded into 3 sub steps. HBC Modified 8-D Process (steps 1 and 2 switched, Step 8 includes closure justification): 1. Construct the problem statement 2. Form a team 3. Implement interim containment 4. Determine the root cause (RC) 1. Brainstorm 2. Sort and Analyze 3. RC Verification Action Plan (VAP) 5. Develop corrective action (CA) 6. Validation of corrective action (Implement a permanent fix) 7. Prevent recurrence, preventive action (PA) 8. Project closure justification, recognize the team This training assumes that you have some familiarity with the 8-D problem solving process and the basic problem solving tools.

6 Reference the 8-D steps: 1. Construct the Problem Statement Take the initiative Go to the floor See and touch the part, physically examine and interview the situation first hand Talk to everyone who should know something 1. Get on the phone with suppliers if applicable a.) Make sure this is a non defensive discussion and doesn t involve liability Take written notes Consider the failure mode/ modes and aircraft operating conditions (consistent?) Utilize a minimum of three relevant people (you + 2 others) to confirm it is: Correct and focused (single problem) Short/ concise as possible Simple/ clear as possible States the problem/ effect/ fault: does not infer or declare the solution Typically includes who, what, when, where, impact = 1 or 2 sentences 1. Typically employ Quality Eng, Design Eng, Mfg Eng, Tool coordinators, Operators / Mechanics, Ops Team Lead, SCM (MPN, Buyer, Matl Handler) 2. Work hard at this, expect iterations a.) Note: a problem well stated is a problem half solved b.) Note: If you cannot say it simply, you do not understand the problem Albert Einstein c.) Einstein also said: If he had one hour to save the world he would spend 55 minutes defining the problem and only 5 minutes finding the solution.

7 1. Construct the Problem Statement (Cont d) Start the project RAIL right away (eventually the Project Leaders responsibility) First step in a written trackable plan Use for logging anything that needs follow up, no need to limit content Typical columns needed: - Item number (entry order) - Assigned to (responsible) - Priority number - ECD Pert Diagram - Date initiated - Date complete - Description - Comments Start the Issue Closure Plan (ICP) Project Leaders responsibility First step: Record the problem statement in the ICP A full written structured and trackable plan that works with the RAIL Show higher level root cause (RC), corrective action (CA), preventive action (PA) investigation and closure tasks descriptions with key indentured interdependent sub tasks Clearly reveals the critical path to closure (RAIL does not) Think Pert diagram for multi-dimensional interdependence of tasks and their sequence» Note: keep asking what s the next step? Make sure all ECDs have a basis/ plan, therefore a PCD = planned completion date (or promise)» PCDs are: written, scope of task analyzed, prioritized, resources committed» Make sure all interdependent handoffs are clear and agreed to (don t drop the baton) It s presentation ready to brief mgmt (retains continuity of actions, 3 panel charts do not) It s a company record of project the RAIL and the VAP are placed into as attachments 6

8 Form : Issue Closure Plan (ICP) Communications and Interim containment are 8D steps for emerging issues (normally not applicable for CAB selected projects) The key tasks and sub tasks required to reach closure should all be separately numbered according to their interdependent sequential flow (clearly reveals the critical path) ICP form available under Form tab at 05/5/11

9 True RCCA Training Class Form : Issue Closure Plan (ICP) Assignee Company ECD PCD Status As key as the Problem Statement The key tasks and sub tasks required to reach closure should all be separately numbered according to their interdependent sequential flow (clearly reveals the critical path) The ICP is the central hub for attached documents and the full project record 08/17/10

10 Form : Issue Closure Plan (ICP) Issue Title: Date Opened: Project Lead: Supplier Name & Supplier # Status Date: Buyer: Part # Corrective Action # (1) Problem Statement: Technical Significance: Potential Reduction in Component Reliability Significance to Production/Delivery: No Impact Significance to Field: No Impact Product Line: Assignee Company PCD ECD Status (1.1) Communications 1. Notify Product Line QA Lead 2. Notify Chief Engineer 3. Notify PC Administrator (Larry Moore s Group) C Montgomery HBC 4. Red Flag Notification Released (2) Form Team 1. Contact Supplier QA Manager: Name: Phone Number John Doe, Quality Engineer, 6-XXXX Jim Smith, Design Engineer, 6-YYYY (3) Interim Containment: 1. Determine suspect lot: 2. Stock: 3. WIP: ICP form available in both Excel and Word versions 08/17/10

11 Reference the 8-D steps: 2. Identify and Organize the Primary Team That Will Be Needed 3-5 people is best, cross functional, needed skill sets, diverse Pick a project leader, facilitator (RH man) & champion for large complex projects The Leader s role is to assure the team follows the process, gets everyone engaged, utilizes the tools, drives and maintains the plan, updates and communicates the project information, calls the meetings The facilitator helps the leader and the team utilize the tools and conduct more effective team meetings The champion s role is to offer guidance, helps get priority for closure tasks, removes constraints, be accountable with the team for results [also the CABs role] Approach as a Special Project that s above your day job not typically a full time assignment, Note: continuous improvement is part of everyone's job Everyone can either lead or support these Many simple projects can be done alone but use 2 others for 3 person confirmation points 3. Interim Containment This is not part of the True RC training class focus assumes that if the problem is a new emerging issue that containment/ remedial action are accomplished in parallel and recorded in the ICP

12 Reference the 8-D steps: 4. Find the Root Cause 4.1 Brainstorm the potential root causes and the escape point Assign someone to capture the thoughts (facilitator) Recommend using a white board, at end of meeting capture with digital camera Note: the best way to have a good idea (or the correct idea) is to have a lot of ideas Everyone still takes written notes:» For their personal understanding and active engagement» To get started on their action items prior to RAIL update and distribution» To capture anything the group dynamics may have missed» Assure everything gets captured on the RAIL at the end of the group s meeting Focus the discussion by using: System element level block diagrams Process flow diagrams 3D illustration (from A/C Maintenance Manual (AMM), Illustrated Parts Catalog (IPC)) Digital photographs Check Lesson s Learned databases for problem/ solution similarity Available data and trends

13 4.1 Brainstorm (Cont d) Death by 1000 questions (5+ whys / maybes = Socratic debate) Ask why did or how could that happen» Followed by maybe it s because.» Ask what s changed over time (especially recently)» Ask why doesn t it happen in another similar situation/ application» Ask what contributed, when did it occur, who was involved» Look for underlying causal chains or sequence of events leading to the problem (they may or may not be interdependent) Differentiate between symptoms, effects of, and actual root causes Symptoms are indicators or observations, they characterize what s changed» When and where it was first observed, how it was first identified, quantifies the size and trend of the problem (ex. the problem became more frequent when it got hot outside) Effects are something brought about by a cause, a description of a fault» Effect = Fault = Problem: the spar cap fractured (failure mode- vs bent, buckled, crippled)» Direct cause = Defect: because it developed a crack (failure mechanism- tensil crack growth)» Contributing causal chain: part was damaged during transportation (crack initiation), because handling procedures weren t followed, because new employees assigned to the task and not trained» True Root cause = supervisors not assuring new employees are trained on requirements» Other contributing cause: undetected during subsequent inspections Identify what you know (supported by data) versus opinion but capture/ record/ and explore everything.

14 4.1 Brainstorm (Cont d) Everyone participates, Repeat until no one can think of another why to ask Encourage free thinking, don t criticize Don t jump to conclusions too quickly this causes the process to stop too soon» Think intensely and at multiple levels (ref: the causal chain)» Peel the onion professionally and unemotionally challenge the basis» Permit no pride of authorship within the team» Note: most people can only support 1-2 layers of questions with their basis for root cause or a corrective action Don t try to implement solutions before the analysis is complete» Note: Our desire to act overpowers our need to understand Supplier Issues should start with a pre visit phone call» Includes the full team (include SMEs) and the suppliers counter parts» This is the best first step to capturing potential root causes and making any required on site visits more efficient Success test eventually can t tell who on the team came up with the original idea or the full string of the idea» Note: Individuals can t be perfect but teams can

15 4. Find the Root Cause (Cont d) 4.2 Sort and analyze the potential root causes Start by utilizing the simple project tools during the brainstorming Further focuses the team in a structured way to organize and clarify the interactions and interdependencies at different levels of the cause Simple project tools Affinity diagrams groups supporting ideas into categories and a priority structure Pareto reveals what causes or effects dominate and may lead to the priority of items to investigate Cause and effect (fishbone) diagrams collects potential root cause elements and sub elements» Typically: people, materials, machines/ equipment, methods, measurements, training, environment (noise, lighting, temp) Fault Tree- a vertically oriented fishbone Process flow diagram Sequence of events diagram Problem, Fault, or Un-desired Effect Complex project tools Histograms identifies patterns, nominal values, and data limits Relation diagrams identifies relationships between drivers and indicators Scatter diagram compare relationship between two variables Web diagram compare multiple variables to look for combination that could be the problem Trend charts FMEA DOE

16 4.2 Sort and analyze the potential root causes- (cont d) Simple project tools Fault, Problem Fault Trees (essentially a vertically oriented fishbone diagram) Design Fab / Assy Testing Quality Sys Ops Example fault tree elements / realms / branches: Design (from detail components, sub systems, system level schematic)» Relevant elements for investigation» Key characteristics» Dimensional/Tolerance analysis (includes: temperature, vibration, dynamic loads/ deflections.) Fabrication, Assembly, Test process and controls» Actual dimensions of relevant parts and tolerance trends (selective assembly practiced?)» Property inspection results of relevant parts-hardness, surface finish, conductivity, etc.» Process documentation and control» Contamination control Quality (variability reduction and Type Design compliance)» Non conformance condition history and trends of relevant parts» Delegation of inspection authority» Sampling rates Maintenance System operation (see example)

17 Fault Tree Example Generator main contacts remain closed (System Operation Branch Example) 1.0 Physical problems 2.0 Commanded close 1.1 Physically stuck 1.2 Contacts welded closed Damaged return spring Over current Short circuit FOE Aircraft harness Damaged parts Transient load during switching Close/ release timing Commands to relays provides clear sub level identification numbering for VAP Can be constructed in Excel Relay operation Two power sources linked Magnetized Over voltage External heat

18 4.2 Sort and analyze the potential root causes- (cont d) Complex project tools Failure Mode and Effects analysis (FMEA) Excel spreadsheet produced during development that determines up front preventative action plans or used during an investigation to determine priority of potential root cause items. Based on a calculated Risk Priority Numbers (RPN):» Significance of the effect (severity), likeliness of occurrence, and probability of detection.» RPN scoring systems can be unique for an individual project based on complexity of the problem (they can be simple) Design/Product FMEA: built from a top level system schematic ( what box talks to what other box/ components in the system? ) and/ or a detail part or assy level drawings» What design characteristics/functions could lead to a particular failure mode? Process FMEA: built from manufacturing process flow diagram» What process characteristics/functions could lead to a particular failure mode?

19 True RCCA Training class Example FMEAs entries for leaking landing gear investigation See Step 4.3: the VAP elements should be utilized ilo this section found in a typical FMEA Design FMEA Identifier Design Characteristic Item/Design Function/ Requirements Potential Failure Mode Potential Effect(s) of Failure S e v Potential Cause(s) / Mechanism(s) of Failure O c c Current Design/Process Controls D e t R P N Comments / Notes Recommended Actions Responsibility & Target Completion Date Actions Taken Dimensional Lower Bore Diam Oversize Lower Bearing Static Seal leakage due to insufficient seal squeeze 5 Poor machining practices 3 Air gauge inspection QNs for Oversize Surface Plating Sealing Surface Soft surface scores & flakes easily Leakage 5 Plating too thick 8 Dimensional Inspection High RPNs require more Mfg and QA controls Process FMEA Identifier Process Characteristic Item/Process Function/ Requirements Potential Failure Mode Potential Effect(s) of Failure S e v Potential Cause(s) / Mechanism(s) of Failure O c c Current Design/Process Controls D e t R P N Comments / Notes Recommended Actions Responsibility & Target Completion Date Actions Taken Seal physical charecteristics Elongation Local thinning, microcracks Leak path thru the seal 5 Non-homogenius material composition 1 Inspection at the supplier Seal storage Shelf life Shelf life exceeded Seal dried out, brittle- may crack 5 Poor inventory mgmt (FIFO) 1 Cure date label on seal bag /24/10

20 Example RPN scoring values Ranking Severity 1 No effect 2 Convenience item affected; next process customer not affected 3 Rework needed at the process station 4 Minor dissatisfaction to next process 5 Rework needs to be done off-station 6 Rework that can lead to scrap 7 Potential scrap to be decided by customer 8 Scrap with part affected without damage to machine and tooling 9 Scrap with machine, tooling and part affected 10 Hazardous. Affects safety of operator Ranking Detection/Prevention 1 Will prevent failure mode from happening with notification of prevention 2 Will prevent failure mode from happening without notification of prevention 3 Will prevent failure mode from happening most times 4 Will prevent failure mode from happening sometimes 5 Will not prevent failure mode from happening but will detect it with notification 6 Will always detect failure mode but without notification 7 Will detect failure mode most times 8 Will detect failure mode sometimes (On and Off) 9 Will not detect failure mode 10 No controls in place

21 Example RPN scoring values Ranking Possible Failure Rates Probability of Failure Occurrence 1 < = 1 in 1,500,000 Remote: Failure is unlikely. Never happened before 2 1 in 150,000 Happens rarely and may cause FM / effect rarely Low: Relatively Few Failures 3 1 in 15,000 Happens sometimes and may cause FM / effect rarely 4 1 in 2,000 Happens every time and may cause FM / effect rarely 5 1 in 400 Moderate: Occasional Failures Happens rarely and may cause FM / effect sometimes 6 1 in in 20 High: Repeated Failures Happens sometimes and may cause FM / effect sometimes Happens every time and may cause FM / effect sometimes 8 1 in 8 Happens rarely and will cause FM / effect every time 9 1 in 3 10 > = 1 in 2 Very High: Failure is almost inevitable Happens sometimes and will cause FM / effect every time Happens every time and will cause FM / effect every time

22 4.2 Sort and analyze the potential root causes- (cont d) Other potential actions from lessons learned on complex problems: Where do we have data that could help? How does teardown evidence and lab analysis information map to the Fault Tree elements? Create a chronological map of failure events to all manufactured units (serial numbers) and delivery dates for a correlation to when something may have changed. Capture change history of all Fault Tree realms and branches Design configuration, process, specs, inspection/test methods, personnel, tooling, location, suppliers, etc. Capture field service history Pareto list of all event S/N parts to the FT branches Look for combinations of FT branches/ variables as possible RC Review the LAI/ FAI and re-qualification process and reports for change of source or manufacturing location Construct a sub tier supplier map Recent interview results, recommendations Audit reports and findings Turnover rate, business level changes Training and certifications compliance controls

23 4. Find the Root Cause - (Cont d) Set Up the Root Cause Verification Action Plan (VAP) The VAP is a comprehensive Indentured spreadsheet used to manage the verification actions required to dismiss or confirm every potential RC at each level and record the results and conclusions. Built from and matches the fishbone or fault tree branches Columns: Indentured numbering system matching locator numbers set up for each branch of the fault tree Title of the Fault Tree element and sub elements RPNs to prioritize RC theory statement for each causal level» Every potential RC must have a written theory statement of how it could have caused the problem» Use the guidelines for good problem statements Verification action/ task description and applicability to confirm or dismiss Action assigned to ECD Status: A-active, C-complete Results: basis for closure and conclusion Potential RC conclusion: Yes (color cell and FT element red), No (green), Maybe (yellow)

24 Fault Tree Example Generator main contacts remain closed 1.0 Physical problems 2.0 Commanded close 1.1 Physically stuck 1.2 Contacts welded closed Damaged return spring Over current Short circuit FOE Aircraft harness Damaged parts Transient load during switching Close/ release timing Commands to relays Relay operation provides clear sub level identification numbering for VAP Two power sources linked Magnetized Over voltage External heat

25 Root Cause Verification Action Plan PCD (Promised Completion Date) Root Cause Verification Action Plan: "Generator main contacts remain closed" Fault Tree Locator Number Fault Tree Element Fault Tree Subelement Fault Tree Subelement Fault Tree Subelement Fault Tree Subelement RPN VAP Root Cause Theory Statement Verification Action/ Task Description Assigned to: ECD Status: A = Active C = Complete Results Potential RC Conclusion 1.0 Physical problems 1.1 Physically stuck Damaged return spring Leads to FOE No Damaged parts 1.2 Contacts welded closed Overcurrent Maybe Short circuit Aircraft harness Transient load during switching Close/ release timing Commands to relays Yes Relay operation Two power sources linked Magnitesed Over voltage External heat 2.0 Commanded close (A new sheet typically started for each major fault tree element) 08/24/10

26 Root Cause Verification Action Plan Root Cause Verification Action Plan: "Generator main contacts remain closed" Fault Tree Locator Number Fault Tree Element Fault Tree Subelement Fault Tree Subelement Fault Tree Subelement Fault Tree Subelement RPN 1.0 Physical Spar Cap problems fractured 1.1 Physically Developed stuck crack Damaged during return transportation spring FOE Damaged parts VAP Handling procedures not followed New employees not trained PCD Any level could be the root cause Root Cause Theory Statement Verification Action/ Task Description Assigned to: ECD Status: A = Active C = Complete Results Potential RC Conclusion No 08/24/10

27 4.3 Set up the Verification Action Plan (VAP) (cont d) - Separate section/ sheet of the spreadsheet for each fault tree major element may be needed - Report % complete based on number of FT elements closed - Determines what verification action required to dismiss or confirm every potential RC at each level Verification = proving beforehand that the planned action will do what is intended - Ex.) analysis, tests/ trials (objective evidence), comparisons with similar activities utilizing a subject matter expert (SME) - Without a completed verification action the potential RC can t be eliminated based on available data Validation = after CA implementation, checking that the action is achieving it s goal and without adding a new problem (proof that the change has worked) Note: nothing is particularly hard if you divide it into small jobs Ultimately verify by: demonstrating you can turn the problem on and off The selected RC also fully explains why the problem doesn t occur in a similar applications and scenarios A minimum of three relevant people to commit they concur the True RC is correct, comprehensive in terms of multiple levels of contributing causes, and the escape point is clear Reference them in the ICP for future info (willingly held accountable) Should also include the champion

28 Reference the 8-D steps: 5. Corrective Action (CA) plan definition and implementation Repeat the brainstorm/ death by 1000 questions Ask and challenge why would or how will this prevent re-occurrence Peel the onion 5+ layers deep From the escape point determine where the problem should have been detected and where the future controls should be placed within the process Consider implementation point and effects on the operation Do pro/ cons if there are CA alternatives Belts and suspenders or multi-phased implementation may be worth the investment Keep the customer in mind Do the right thing even if it s hard or takes a long time Add the action plan tasks to the ICP A minimum of three relevant people to commit they concur the CA plan is correct and comprehensive in terms of appropriate action assigned to all levels of contributing cause Reference them in the ICP for future info (willingly held accountable) Should also include the champion 6. Define what will constitute validation for the CA Ex.) an overall effectiveness metric for periodic review, follow up audits, periodic detail inspections or tests A recalculation of the original RPN to show significant improvement ( = 0, if re-occurrence eliminated) Add the action plan tasks, including tracking the implementation to the ICP A minimum of three relevant people to commit they concur the validation plan is correct and reference them in the ICP for future info (willingly held accountable) Should also include the champion

29 Reference the 8-D steps: 7. Preventative Action (PA) plan definition and implementation 1. Systemic analysis where else or what else could be impacted by this problem and RC? Think both other products and processes- assure they take action 2. Is there an underlying cultural or operational problem that needs addressed? How could this have been avoided? New communication or training required Change to policies, practices, procedures 3. What potential unintended effects could occur from implementing the CA? This requires repeat of the brainstorm process and a check of the lessons learned database for applicability What will be the verification action to assure they re avoided 4. Mistake proofing: ex. physical/ mechanical controls on tooling (see Juran training material) Add the action plan tasks to the ICP A minimum of three relevant people to commit they concur the PA plan is correct and comprehensive Reference them in the ICP for future info (willingly held accountable) Should also include the champion

30 Reference the 8-D steps: 8. Project Closed: The full team agrees that the predetermined closure criteria were satisfied For each step above (as recorded in the ICP) Typically status ICP as closed once CA and PA closure plan accepted and implemented Update the ICP once the validation step is complete (Leaders responsibility) Assured lessons learned are recorded and shared across the organization for continuous improvement Appropriate recognitions and celebrations followed through.