Human factors and Usability Testing for drug-device combination products

Transcription

1 A presentation by Greg Thay Human Factors Director, THAY Medical.

2 Introduction Greg Thay B.Eng. (Hons) Companies I have worked for.. Current role: Human factors THAY Medical Me! I live here! (England)

3 Presentation Scope Human factors and Usability Testing for drug-device combination products Human factors What is Human factors and Usability? Regulations applicable to DDCP s Abbreviation Drug-device combination product = D D C P Usability Testing Formative Human factors & Usability Testing Summary of Usability Testing aims Tips & Helpful hints The Future Changes in the regulations The future of combination products?

4 Human factors What is Human factors and Usability? Regulations applicable to DDCP s Usability Testing Human factors & Usability Testing Summary of Usability Testing aims Tips & Helpful hints The Future Changes in the regulations The future of combination products?

5 What is Human factors and Usability? Human factors is the DISCIPLINE OF APPLYING KNOWLEDGE OF HUMAN BEHAVIOUR, METHODS FOR ANALYSIS, TEST AND EVALUATION TECHNIQUES THAT LEAD TO OUTCOMES INCLUDING USABILITY AND EASE OF USE, AND SAFE OPERATION WITHIN THE CAPABILITIES AND LIMITS OF THE HUMAN OPERATOR. Often the term Usability is confused with the definition of Human factors, but you will note, it is used in the definition. To make this clear, Usability is the A QUALITY OF A SYSTEM THAT INCLUDES INTUITIVENESS OF INTERACTION, SUCCESSFUL USER PERFORMANCE, USER PREFERENCES, EASE OF HANDLING, EASE OF LEARNING AND SKILL ACQUISITION. A more friendly definition is found in the IEC standard

6 What is Human factors and Usability? IEC from 2007 originally, defines Usability as the CHARACTERISTIC OF THE USER INTERFACE THAT ESTABLISHES EFFECTIVENESS, EFFICIENCY, EASE OF USER LEARNING AND USER SATISFACTION. This I feel is a defined statement and leads you to look at specific areas for evaluation. One key way to view all of this is shown below: USER INTERACTION

7 What is Human factors and Usability? Key areas to consider when looking at DDCP human factors: Ergonomics Anthropometrics Biomechanics Clinical Practices (Healthcare) Economics Language Sociality Safety Compliance Cognition Note how Usability is a sub-set of human factors! Usability Emotion User Experience

8 What is Human factors and Usability? Key areas often researched, defined and used as part of the human factors engineering scope include: Ergonomics, Anthropometrics & Biomechanics since we all have differing abilities to use products. Clinical Practices variation in how drug device combination products are used globally. (Healthcare) Economics how the user purchases & receives the product varies country to country. Language not just the spoken or written, but also the context may differ by country/region and also by product (more important for user interfaces & digital ddcp s). Sociality The variability in how people are perceived using ddcp s in different countries. Safety How people understand clinical and use safety globally. Compliance How users remain compliant to the drug product and benefit from the clinical outcomes. Cognition Variation on how users comprehend and interpret use instructions, methods of use, symbols, clinical terminology and context. Usability the ability of the users to use the ddcp safely as intended. Emotion & User Experience Use emotion and the ability of the ddcp to give a positive user experience.

9 What is Human factors and Usability? Some examples of great human factors as determined by the UK based IEHF (Institute of Ergonomics & Human Factors) Winner 2012 Winner 2011 Winner 2010 Winner IEHF Winners 2010: 3M, Ergonomidesign, Linköping University and Royal Institute of Technology, Sweden. 2011: Bombardier Transportation, UK. 2012: Kinetic Concepts, Inc. (KCI), USA. 2013: Loughborough Design School, Loughborough University, UK.

10 What is Human factors and Usability? Some examples of not so great human factors! Confusing directions Opening packaging Mislabelling Understanding symbols We all find examples of products that haven t considered the user elements in their design. With the above examples there is often significant risk with the outcome of the wrong instruction or action.

11 What is Human factors and Usability? So how does human factors apply to drug device combination products? It can generate a set of design requirements and design considerations that designers/engineers can use to improve the potential usability of the DDCP. They can be used to define a design direction. A design requirement could be If at all possible, for an inhaler, avoid the use of the colour red as the primary branding. WHY? Because people associate the colour of red with blood. Many DDCP s do not interact with blood! Also, we often perceive the colour of red to be a warning colour or a stop instruction and so on.. x Always consider the impact of the decision on the user.

12 What is Human factors and Usability? A design consideration could be - If at all possible, design packaging to be within the abilities of the intended user. WHY? Packaging is often a barrier, and although often seen as a low level risk of harm to the user, it is vital for access to the DDCP, to the Instruction/User Manual(s) and Clinical information (PIL s). It is worth considering making the packaging usable, easy to open and using guidance from a regulatory standard often helps with designing it in the first place. Using standardised packaging methods also keep development simple and also helps out users since standard methods of packaging are widely used Tyvek pouches, zip based cases, suitcases and cardboard boxes etc.

13 What is Human factors and Usability? Quick case study packaging! Child Proof caps on pill containers. Safety element: protects child users from access to pill container contents drugs that could harm them. Fine for: Most users, who are adult, able bodies and have no dexterity limiting conditions. Not good for: 1) adults with dexterity limiting conditions such as arthritis. 2) adults with hand based injuries. 3) drugs that are required to be used by children! 4) Rescue medications where quick access is required. There are often human factors elements that are not always obvious.

14 Human factors What is Human factors and Usability? HF Regulations applicable to DDCP s Usability Testing Human factors & Usability Testing Summary of Usability Testing aims Tips & Helpful hints The Future Changes in the regulations The future of combination products?

15 Human factors regulations applicable to DDCP s. There are many harmonised and Consensus standards, regulations and guidelines available to assist with developing a DDCP from a human factors perspective. Below is a list of a few of the most commonly used: IEC/ISO AAMI He-75 UCM IEC ISO 9241 MIL STD 1472f

16 Human factors regulations applicable to DDCP s. And a few more less commonly used: ISO 26800: 2011 ISO 24501: 2010 ISO 28802: 2012 ISO 9921: 2003 ISO 11428: 1996 ISO : 2010 IEC :2015 Ergonomics -- General approach, principles and concepts Ergonomics -- Accessible design -- Sound pressure levels of auditory signals for consumer products Ergonomics of the physical environment -- Assessment of environments by means of an environmental survey involving physical measurements of the environment and subjective responses of people Ergonomics -- Assessment of speech communication Ergonomics -- Visual danger signals -- General requirements, design and testing Respiratory protective devices -- Human factors -- Part 2: Anthropometrics Medical electrical equipment -- Part 1-11: General requirements for basic safety and essential performance -- Collateral standard: Requirements for medical electrical equipment and medical electrical systems used in the home healthcare environments

17 Human factors regulations applicable to DDCP s. One of the most widely used standards used internationally is IEC/ISO The application of usability engineering to medical devices. This IEC & ISO level standard is Accepted by most notified/regulatory bodies as a standard for enhancing the usability of medical devices. Is subtly different than the methods mandated by the US FDA guidance in UCM A thorough method of focusing on the user of the medical device and includes specific detail on the theoretical and the practical sides of human factors and usability. It requests that an amount of user research is performed (scalable to the expected risk level of the medical device) to characterise the user(s), their abilities, their disabilities and their human factors. It links very tightly into ISO The application of risk management to medical devices. It details a logical method of risk management based upon use safety through user research, usability engineering (and design), verification and validation. Being updated this year (2015) and harmonised with the US FDA current guidance on human factors.

18 Human factors regulations applicable to DDCP s. Another widely used regulatory standard in the development of DDCP s is the US FDA guidance on human factors: UCM Applying human factors and usability engineering to optimise medical device design. It uses a nearly identical process of usability engineering as IEC/ISO mandating that user research is performed, that hazard analysis (risk assessment) is performed and that verification and validation of the use of the medical device is performed. Although it is classed as a guidance document, it is what the reviewers of the Human Factors Engineering File at the FDA use as the basis for the minimum requirements to show evidence of use safety. In other words, they prefer to see that the medical device has been developed to this guidance document. NOTE: FDA = US Food and Drug Administration

19 Human factors regulations applicable to DDCP s. The US based Food and Drug Administration A Frequently Asked Question: If I produce a human factors engineering file, who/where reviews it at the FDA? Since DDCP is a combination product and involves both a drug and a medical device, it will fall under the remit of the CDER function within the FDA. CDER = Centre for Drug Evaluation and Research. The HFE file will first go to the staff in CDER, who are knowledgeable on human factors engineering. In some cases, the HFE file may also be shared with the human factors team at the CDRH within the FDA. CDRH = Centre for Diseases and Radiological Health.

20 Human factors regulations applicable to DDCP s. WHY use these regulations at all? To make sure the safest DDCP is developed or maintained. To develop the DDCP to the industry accepted levels. To create a level of confidence that the DDCP is safe. To gain guidance on what technical requirements should be met (rather than finger in the air approaches!) To show evidence that safety has been incorporated in the product development. To assist with a smarter review of product safety when reviewing the Essential Requirements of a MDD. By basing the product development of the DDCP on internationally recognised regulations (standards, guidelines, industry best practices), there is a better chance the DDCP is safer and has incorporated some level of human factors in its design.

21 Human factors Usability Testing What is Human factors and Usability? Regulations applicable to DDCP s Human factors & Usability Testing Summary of Usability Testing aims Tips & Helpful hints The Future Changes in the regulations The future of combination products?

22 Methodology with DDCP s: Process Firstly, it is worthwhile referencing the US FDA s guidance on design control, and what it calls the Waterfall method. As you can see it is a pictorial illustration of the way to develop devices, and for today, DDCP s.

23 Methodology with DDCP s: To overlap with the process defined for usability engineering (as per IEC/ISO 62366, four key areas stand out: Concept Phase Design & Development Validation Validation

24 Methodology with DDCP s: When this model is overlapped with the model detailed in IEC/ISO 62366, we see equivalence and a subtle change in terminology. Concept = User Research and Design & Development = User Interface Development. Concept Design & Development Verification Validation User Research Application Specification Contextual Enquiry Task Analysis Comparative Usability Testing Proof of Concept Testing Ethnographic Research User Interface Development Usability Specification User Interface Requirements Paper Prototyping Formative HF studies Verification Formative HF studies Design Iteration User Interface Requirements Review Usability Specification Review Validation Summative HF studies HF & UE Report Usability Risk Assessment Risk Management Risk Review Acceptable use based Risk

25 Methodology with DDCP s: When to test? At what point is Human factors and Usability testing required? The answer is logical. When you have an unknown. For example: If you don t know who the intended users really are - Research them and find out. If you aren t sure whether the user can re-cap after use Test this and find out. If you don t have any data on the probability of a risk occurring in real life Test the device and find out. If you an unsure of the safety of the device Test and find out. All of the above create data. Data is evidence that can answer a question and clarify an unknown.

26 Methodology with DDCP s: User Research & User Interface Development In the first two stages of the product development, testing should be considered. WHY? When developing a new DDCP, it helps to understand what already exists and what is good and bad about the existing DDCP s. With many DDCP s the performance data is often kept strictly confidential. So, testing often performed as part of the User Research part of the Usability & Human factors engineering effort includes Comparative Usability Testing, Proof of Concept Testing and Paper Prototyping. Comparative Usability Testing taking existing products and evaluating their usability with real users in a simulated environment, comparing the results against each other (and possibly a paper prototype). Proof of Concept Testing taking a concept to real users (normally in sketch formats) and researching the human factors and usability surrounding the concept. Paper Prototyping as it states; paper prototypes of a concept design, which are then evaluated with real users in a simulated environment (can be combined with a Comparative Usability Test see above).

27 Methodology with DDCP s: User Interface Development & Verification In these two stages, two types of testing are often performed to ensure that use based risks and usability specifications are being developed in the product, and evidence gained from testing often shows this. The two types of testing are: Paper Prototyping (see previous slide) Formative Human factors & Usability studies taking a prototype and evaluating its usability with real users in a simulated environment. These are early stage formative human factors and usability studies using prototypes of the product and user interfaces (physical examples that the intended users can interact with). These may not be fully functioning, but may have some functionality!

28 Methodology with DDCP s: Verification By this stage, it would be hoped that the product had been evaluated possibly at least once with its intended users. Sometimes budgets do not allow this. So, it is at this point in the development, the product should be tested with real users, where the design has been 95-99% finalised, where Instruction/User Manuals have been drafted, Software has been bench-tested and that all the user interfaces are in a final state design-wise. A Formative Human factors & Usability study should be performed at least once. This type of study requires a fully functioning prototype if at all possible. You are evaluating the final design to gain confidence that the development has met all the User Research and User Interface Development specification and use based requirements. Sometimes, it take more than one round of testing to gain that confidence!

29 Methodology with DDCP s: Validation Following the design verification of the DDCP, and once the usability of the device has been found to be safe and acceptable, it may be prudent to validate the usability of the device. With this, you are upping the sample size to see what the effects of volume are with respect to use safety. Like a device, it is verified on a smaller sample size of product than when it is validated. The usability testing performed earlier in the development may have only been conducted on a small sample and as such, to ensure that the same use errors do not occur in the larger population (as can be expected post launch), the validation testing is recommended (and mandated if wishing to be compliant to the regulatory standards). VERIFICATION n = small VALIDATION n = larger 5 8 per user group per user group

30 Methodology with DDCP s: Validation Validation is specific in what it is requesting but is basically a stricter, equivalent test regime, larger in size than a Formative Human factors and Usability Study. The key differences are: Sample size - as previous slide. Relevance to real life - The testing should reflect real life as far as consistently possible. Moderator intervention - The moderator in the test sessions cannot assist the participant in any way unless there is a compromise in the safety of the participant. Devices Production ready devices should be used,ideally from a validation batch. No prototypes or models without 100% intended functionality can be used unlike Formative testing. The User Manuals, Instructions for Use, PIL s, safety information and all user documents, including training DVDs, video s and websites are ALL tested. Often they are tested separately to the device. The data has to prove acceptable use safety it is the final time in the development this can be done.

31 Methodology with DDCP s: Human factors and Usability Testing in more detail In more detail The Human factors & Usability study is Performed in a simulated or real (actual) use environment as defined by the User Research. Performed using external, real (intended) users as defined by the User Research. Performed using the final design. Performed in the most likely territory/country that the product will be launched into. Performed in verification with 5-10 study participants per distinct user group, or in validation. A user group is a collection of expected (intended) users that you expect to be using the product. Includes an evaluation of the use of the product, without assistance by the research team. Includes an evaluation of the Instruction/User Manuals, User Interfaces and Training methods & training materials.

32 Methodology with DDCP s: Human factors and Usability Testing in more detail For this section, we will use an example of a Drug Device Combination Product (DDCP) to give context. So let us consider an injection pen which delivers one dose of a drug weekly for people suffering from rheumatoid or psoriatic arthritis. A simple(ish) prospect to put this into context with. So we have 1) an injection pen 2) a weekly injection & 3) people with RA or PA.

33 Methodology with DDCP s: What to evaluate? So for our case of a pen injector, testing would evaluate 1) The use and safety of the pen injector system with people with RA or PA. Depending on whether they may use the pen injector differently, this may be two distinct user groups, or one user group split between patients with RA and patients and/or those with PA. Also the User Research should have picked up that with more severely affected patients, a care-giver or a Nurse may give the patient the injection instead. If this is the case include them in the formative study as a separate user group. 2) The pen injector User Manual and any instructional or training regimes & materials available. NOTE: All aspects of the user interface should be evaluated if possible. Evaluate how the intended users are likely to use the pen injector in real life; and evaluate the usability and safety of the pen injector. Safety is determined using the relevant Risk Assessment.

34 Methodology with DDCP s: What to evaluate? In addition, if there is novel, new or ground-breaking technology in the pen injector, then it is also worthwhile evaluating the impact of this technology. This can be separate to the system evaluation in side studies. 1) Symbology if a new symbol, colour or text label on a button is being proposed. 2) Use methods if a new use method is being proposed such as a different needle attachment process. 3) Packaging if a new type of (smart/intelligent) packaging is being proposed. 4) User Interfaces if a new level or type of user interface is being proposed such as a digital GUI. 5) Drug delivery if a new method of drug delivery is being proposed moving from oral spray to injection. Side studies on specific elements of the device being proposed/developed will generate data that shows the proposal (change) is or is not acceptable with the intended users, and is or is not safe.

35 Methodology with DDCP s: Study Documentation The documentation involved with performing a Formative Human factors & Usability Study would look something like: A Study Protocol and the Moderator Script, A copy of the Usability / Human factors Risk Assessment, A copy of the User Manual / Instructions for Use, A copy of the Recruitment (Screening) Document, Any Training Scripts, Guides or Protocols, An Informed Consent Form, An Audio & Video Release Form,, An Intellectual Property Waiver, A Confidentiality Agreement, An Adverse Incident Reporting Form, A Background Study Participant Questionnaire.

36 Methodology with DDCP s: Test Equipment The test equipment involved with performing a Formative Human factors & Usability Study would look something like: A quantity of the DDCP s (be they inhalers, pen injectors, oral sprays, infusion pumps etc.). Any set up, storage or disposal equipment typically used (sharps containers, storage cases etc.). Any personal protective equipment (hand cleaners, wipes, disposable gloves etc.). Any consumables required to use the DDCP (needles, cannula, extension lines etc.). Recording cameras, power cords, extension leads, SD cards, data storage units. A simulated body part (skin pad, training dummy, Sim-Man etc.). Placebo drug solutions/particles ideally NOT real drug. Towels, cloths etc. Body or skin preparation wipes, swabs, cleaners. Any DDCP packaging, Instructions, PIL s.

37 Methodology with DDCP s: Participants and Test Sessions The participants for the human factors and usability study should be recruited externally to the organisation. This will Reduce/eliminate any organisational bias the participants may have. Create independent data. Create data based upon real intended users. (Q. Are your employees the intended users???) Things not to do when recruiting participants: Use KOL s who are affiliated to the organisation (again bias minimisation). Use participants who do not fit the user profile as determined by the User Research. Use participants who regularly perform clinical or human factors and usability studies. Use participants who have performed in a study on the same project at an earlier stage. Use participants who have performed in a study recently (<6mths). Use participants who work for a rival organisation!

38 Methodology with DDCP s: Participants and Test Sessions A study is the term used for a group of test sessions. In a human factors and usability study, the test session is singular in a formative study and may be multiple in a validation study. The study should ideally have rehearsal a Pilot. This will reduce or eliminate any interruptions to the test session flow and show that the test method works and gains the expected data. The test sessions should read like a story. They have An introduction: This includes introducing the study, checking paperwork, putting the participant at ease. The beginning: This may include a training session, training decay time, or a read of the User Manual / IFU by the participant whatever is real life! The middle: The test session itself a run through of a series of tasks that the user would do with the device, and in validation an examination of the user documentation. The end: A wrap up, debrief, Q&A s, re-iteration of key study points, giving of honoraria and most importantly a thank-you!

39 Methodology with DDCP s: Post Study Once the test sessions are complete the study needs to be reported. The following may be completed: 1) The test sessions data would be collated in a manner that makes reviewing it easy. 2) The test sessions data is then analysed could the study participant use the DDCP as intended? 3) The Study Report is written. 4) The Usability & Human factors Risk Assessment is updated. At this formative stage, the data should be divided into groups use errors, successes with difficulty, near misses and successes. Also, use errors can be sub-divided in to device based errors or condition based errors (especially for patient users) since it may due to the patients condition that they were unable to use the device as intended. Note - it is not user errors, but use errors, since the context is that it is the fault of the device not the user at this stage of product development. You are attempting to design out (mitigate) use-based risk.

40 Human factors Usability Testing What is Human factors? Regulations applicable to DDCP s Human factors & Usability Testing Summary of Usability Testing aims Tips & Helpful hints The Future Changes in the regulations The future of combination products?

41 Methodology with DDCP s: Outcomes The outcome of a Human factors and Usability Study should be: 1. Evidence that the intended user(s) can use the device / system to an acceptable level (rarely can all users use it perfectly). 2. Areas for improvement of the design of the device are identified EXCEPT when validating the device. 3. A reduction of the use-based risk to an acceptable level by the end of development if not earlier. 4. Confidence that the design of the device is on the right development path.

42 Human factors What is Human factors? Regulations applicable to DDCP s Usability Testing Human factors & Usability Testing Summary of Usability Testing aims Helpful hints & tips The Future Changes in the regulations The future of combination products?

43 Methodology with DDCP s: Helpful hints & tips The test set up for a Human factors & Usability Study should Be consistent as far as possible. Be set up, assembled, started in the same way for each study participant. Use the same test equipment each time. Use traceable DDCP s, (placebo) drug containers & contents, packaging, PIL s etc. Be performed with one study participant at a time reflecting real life. Always start with a friendly introduction with the study participant(s). Do the paperwork first put the mind at ease of the study participant(s). Ensure the study participant is of a fit state to finish the test session continuingly evaluate this. If patients are involved, ensure they have opportunities to take medications or stick to routines. If HCP s are involved ensure they are told not to rush their test session. Always end positively with the study participant. It is a user experience for them!

44 Methodology with DDCP s: Helpful hints & tips A list of common formative HF/usability testing mistakes: 1. Helping the user (study participant) to get the tasks completed. 2. Not keeping track of time or talking too much. 3. Not taking it seriously enough or keeping it focused. 4. Not pre-testing the Study (Test) Protocol (the Pilot ). 5. Rushing the study participant(s). 6. Leading or biasing the study participant(s). 7. Not making the test sessions interactive and friendly. 8. Making the study participant(s) feel inferior. Helpful hint try and avoid doing the above. Note: The origin of this picture is unknown. The author claims no right on its use. At the end of the day, each test session is an experience for all involved!

45 Human factors What is Human factors? Regulations applicable to DDCP s Usability Testing Human factors & Usability Testing Summary of Usability Testing aims Helpful hints & tips The Future Changes in the regulations The future of combination products?

46 Methodology with DDCP s: Changes in the regulations This year (2015), there are plans to harmonise the two most frequently used regulatory standards the draft US FDA UCM Human factors guidance document (released in June 2011) and IEC/ISO 62366: The new standard will incorporate the best of both of these two standards and be split into two parts and will be called IEC and IEC TR Key changes are to Have a more thoroughly integrated standard alongside ISO (Risk Management of medical devices). Have defined, simpler, non-conflicting, terminology. Have examples with illustrations (this is the second part IEC TR ). In addition, the guidance document also frequently used when developing devices and DDCP s is the AAMI document He-75. Edition 2 is currently being worked upon.

47 Human factors What is Human factors? Regulations applicable to DDCP s Usability Testing Human factors & Usability Testing Summary of Usability Testing aims Helpful hints & tips The Future Changes in the regulations The future of combination products?

48 The future of combination products This page is a personal belief, based upon time developing medical devices in my career and the latest trends I see in running a consultancy focused on human factors engineering. 1. Consumer products are crossing over into healthcare and into how DDCP s are being controlled and monitored. This trend will continue. 2. With more and more money being spent on DDCP s, there will be ever increased choice for the purchaser to choose from. More similar devices are being designed. 3. The DDCP Intellectual Property space is now so tight, that innovation is being forced onto the application of existing technology, rather than focusing on easier, more innovative methods of drug delivery into the body. 4. DDCP s will not just be singular devices, they are becoming systems. 5. Money savvy economies will want low cost DDCP solutions, where as the effluent economies will want gold standard DDCP solutions.

49 THE END Any questions?