Trials Units Information Systems. System Standards. Data and Information Systems Project. The DIMS Project Team

Transcription

1 Trials Units Information Systems System s Data and Information Systems Project The DIMS Project Team

2 Trials Unit IS System s Contents Introduction...1 s and...2 Validation...5 s Implementation and Management...6 The Lists of s...7 Infrastructure Requirements...8 Procurement and Installation (Servers)... 8 Physical Access Control (Servers and Desktops)... 8 Management (Servers)... 9 Logical Access Control (Servers and Networks) Business Continuity...11 In House Software Development Validation (General)...14 User Training and Support IT Staff Competence...16 General QMS Integration...16 Functional Requirements...18 Setup (Clinical Data Systems) Change management (Clinical Data Systems) Validation (Clinical Data Systems) Randomisation Systems...20 Access Control (Specific Systems) Audit Trails Data Correction Systems Trial Administration Systems Data Coding Questionnaire Based Studies Importing, Uploading and Directly Amending Data Safety Data Management...28 Data Export and Reports (internal) Data Exports (External)...30 Generating Analysis Datasets International Capability...31 Long Term Curation...32

3 Trials Unit IS System s Introduction 1 This document represents one of the planned deliverables of the Data and Information Management Systems (DIMS) project, jointly sponsored by NIHR and UKCRC. It sets out to list the standards expected of IT systems in UK academic trials units, in terms of both their core functionality i.e. the collection, storage and manipulation of clinical and administrative data to support the scientific activity of the trials and in terms of the underlying infrastructure for instance the way in which systems are protected from failure, or validated as fit for purpose, or how users are properly prepared for their roles. 2 Establishing a consensus on indicators of good IS practice has always been one of the objectives of the DIMS project, because of the perceived lack of clarity in this area. The consultation exercise with the 40 registered academic trials units in the UK, also part of the DIMS project, only confirmed the need for this. Many units reported uncertainty about the functions and standards expected of Information Systems, especially in the light of MHRA inspection, and requested better guidance. 3 This document is an initial attempt to fill that perceived gap, but there is no claim that it represents the definitive catalogue of standards for clinical trials IT. The validity and practicality of the standards proposed will need evaluating by the trials units themselves and, if it is to be a useful guide to preparation for audit, with inspecting organisations like MHRA. Even once agreed, the resultant set of standards will need regular review and updating because expectations and requirements will evolve over time, for instance in response to changes in regulations, or technology, or to support other initiatives. 4 The scope of the standards is limited to the structure, functions and use of information systems and the value, accuracy and integrity of the data within them. The only exceptions are some proposals relating to the need for integration within an overarching quality management system (in other words these standards should not be perceived or managed in isolation from other quality management in units). A lot of related activities e.g. CRF design, data sampling for accuracy checks, SDV which are currently (largely) manual activities, even if they are supported by IT systems, are not covered. 5 This is not the first time that quality standards have been examined in the context of Trials Unit IT, and this document draws heavily from two key papers written last year: a) GCP compliant data management in multinational clinical trials. ECRIN 2, Deliverable D10, Version 1, 15 September 2008: Transnational Working Group on Data Management, chaired by Christian Ohmann, Heinrich Heine University, Düsseldorf, and b) Auditing IT NCRI Guidelines (2008), a document written under the auspices of the ACTRIS group, the forerunner of the UKCRN ISWG by Paul Mason, Birmingham University (with input from several other ACTRIS members Will Crocombe, Jim Charvill, Chris Morris, Ian Kennedy and James Batchelor) Page 1

4 Trials Unit IS System s 6 The next section describes how standards are conceptualised and categorised in this document, and this is followed by a brief discussion of one of the key standards issues: validation. A final introductory section discusses how standards might be best implemented, managed and reviewed. 7 The standards themselves are then given, as a series of lists, one for each major component or aspect of functioning. The first group, of 10 lists, relate to infrastructure (or non functional ) components; the second group, of 17 lists, to the functional parts of trials units systems. 8 Note there is no attempt to justify each statement in any detail, though some are commented. If a suitably categorised statement does not seem reasonable it may need clarification, or it may just need to be removed from the list. 9 This paper does not consider the possible consequences of non compliance in any detail. That is seen as an issue for those who might audit and / or accredit units, and those, including funders, who might make use of the judgements made. s and 10 There is an argument that says a standard is a standard, and it is either met or not. There is no room for half meeting a standard, and having important and less important standards just makes the system messy and hard to understand. 11 The argument is appealing logically but difficult practically. The fact is that some standards are more important than others failing to meet one might seriously jeopardise the security of data, failing to meet another may simply make a function more difficult or expensive to carry out than it should be. This has long been recognised by inspection teams, who differentiate critical findings from other problems. 12 In addition, if a trials unit is confronted with a long list of standards, all of which must be complied with in their entirety to achieve some level of accreditation, not only is the list potentially demotivating because it does not recognise the progress that may have been made, it can also make it difficult to know where to start to tackle the outstanding issues. 13 Trials units vary considerably in their size and maturity, and so some are much more likely to have reached certain levels of operation than others, indeed may be operating well above what is only just attainable for a newer unit. Most standards have to reflect what the bulk of units can realistically achieve, not the best practices of the best resourced. On the other hand those best practices are often worthy of emulation, and over time what started out as a premium practice restricted to a few units might, as its advantages became clearer and resources accumulate, become more attainable for, and more expected of, all units. The standard could rise to reflect this. 14 Similarly standards, if they come with sufficient notice and explanation, can be used as levers for change. At the moment it would be impractical, for instance, to insist on standards on the use of data standards, not least because the data standards themselves are not well defined. Once such standards exist however, units could be given notice that, in a specified time period, Page 2

5 Trials Unit IS System s a statement relating to (for instance) metadata exports would be included in the relevant list and units would be expected to comply with it 15 To reflect this, and to provide more flexibility in future discussions about standards, the statements listed in the following pages are divided into four types, and within each list are given the following order: a) s: These are important, often fairly basic, standards of operation that should be attainable and implemented by all trials units, however new and whatever their size. Failure to achieve these standards would be a critical finding and should result, at the least, in urgent corrective action. standards represent an absolute baseline for trials unit information systems. b) s: These are quality standards as that term is normally understood, i.e. there is an expectation that trials units will implement all the structures / processes described in them, assuming they are applicable to their operation, if they want their information systems to be considered as high quality. Failure to meet a relevant standard will therefore normally trigger remedial action. It is anticipated that demonstrating the listed standards would be at the core of audit and accreditation. c) s: These are made up of two sub groups. The first are standards that represent the next notch up of quality, and which may be present in some units but are not currently expected of all. Within a continuous process of review they may be regraded at some point into a normal standard. The second are standards that could support national initiatives but that are not practicable expectations at the moment, because the initiatives themselves need better definition and resourcing, and the tools and support that the trials units would need to implement the standards are not yet in evidence. Again, within a review process, expectations could change and units would then be supported and encouraged to work to these new standards to retain high quality status. d) : Some statements indicate good practice but are hard to justify as quality standards, because if they are not met there is no direct impact on the potential safety and accuracy of the data. They might affect the efficiency of the process, but if not met the impact is likely to be on the trials unit itself rather than its core functionality. For instance it is good (and normal) practice to have an air conditioned server room but assuming effective backups it is the trials unit that lives with the risk of servers overheating and going down, and has to suffer the consequences if this feature is not implemented, because the data can be restored when it is possible to do so. It would therefore be hard to insist that all units implement this feature as a necessary component of quality. 16 The last two groups overlap considerably and it is often a matter of opinion whether a particular statement belongs in the or the category, indeed the categorisation of all the standard statements is clearly a matter of opinion. If a standard Page 3

6 Trials Unit IS System s review process was set up it would make it easier to officially differentiate the statements into one group or another. Infrastructure s (80) Functional s (122) 17 As illustrated above, there are currently 202 standards listed, 80 in the infrastructure group, 122 in the functional set, a roughly 40 / 60 split. There is a marked difference in the relative proportions of different types of standards however, with 59% of the infrastructure standards proposed as current or critical standards, where as only 38% are so categorised in the functional group. 18 This may be an indication that infrastructure standards are easier to define (they can often be related to general IT standards) than those dealing with the core functionality of trials IT. It is also almost certainly a reflection of the fact that many aspects of trials IT, especially those Page 4

7 Trials Unit IS System s dealing with data standardisation and trial administration, are currently relatively underdeveloped and standards are correspondingly harder to assert. 19 The standard categorisations could also be used to provide an overall rating of the quality of information systems in trials units. One idea, used in other contexts, would be to award a gold / silver / bronze level of accreditation. For instance, a a) Gold rating might equate to all standards met plus a stipulated minimum number of the s / criteria. b) Silver would correspond to all the standards being met, including of course all the critical standards, and would therefore correspond to the normal expectation of high quality. c) Bronze would mean a number of standards were not met (excluding any critical standards and up to a stipulated maximum number) and that processes were in place to remedy the deficiencies. Validation 20 There has probably been more debate over the various aspects of validation than over any other part of implementing system standards. Assuring fitness for purpose, which is how validation is usually defined, is clearly a key goal of any system of standards, but the general approach here is to argue that validation must always be a deliberate, thoughtful and proportionate process, based upon a risk assessment, rather than a blind and blanket application of rules. 21 Apart from anything else academic units do not, in general, have the resources to indulge in full validation for every component i.e. drawing up detailed requirement specifications, validation and test plans, and then carrying out and documenting the test results (and then repeating the exercise after each major revision). This is not to deny that validation probably needs to be improved in many units, only to point out that funders are unlikely to resource validation at the level (for instance) generally found in the pharmaceutical industry. 22 Another difficulty with validation is that there is no simple way to know how much is required. One can generate tests against requirements, but even if those requirements are fully comprehensive it is impossible to test all possible input scenarios prior to deployment. Some strange combination of circumstances, or some completely unanticipated sequence of operations, six months into use, may cause a glitch to occur, though clearly validation is useful in reducing the probability of that happening. 23 Validation in use, is therefore an inevitable feature of the systems we use, as it is for all systems (which is why beta testing, in the real world by real users, is so important for companies like Microsoft). We should acknowledge that we are lucky in clinical trials in having a very high level of expertise amongst our users i.e. the trial managers and statisticians who can not only identify issues that arise but have the motivation to report them promptly. In other words we have the significant boon of users who care very much about data quality. Page 5

8 Trials Unit IS System s 24 It is also worth remembering that the final responsibility for validation has to belong to the system s users, for only they can sign off a system as being fit for purpose. For database design systems, whether built or bought, the users will often be the IT staff. For the clinical databases themselves, as well as administration systems, the users will be data management staff and ultimately the trial statisticians. 25 Two lists of standards for validation are given, one for general system validation and the other for the more specific testing of clinical databases. Statements about validation for randomisation and data export systems are included within the relevant lists. The requirement for a risk based approach is further discussed in the context of the general validation list, as well as the section on in house software development. 26 Validation prior to deployment is clearly more critical for systems involved with clinical data rather than administration. In any particular context, however, the planned validation programme will have to involve an assessment of both the consequences of errors emerging and the best use of time of the various staff groups involved. s Implementation and Management 27 s need regular review if they are to remain comprehensive and reflect current practice, let alone help to direct future developments. They also need support in the sense that, to be successfully implemented, they need to be disseminated and discussed, explained and exemplified. The users and wider trials community s views on the standards both general and specific also need to be collected and fed back into any review process. 28 There are various groups with a potential interest in system standards in trials units information systems. Most obviously there are the units themselves, particularly but not exclusively the IT staff within them. There are also inspecting / accrediting organisations such as MHRA and UKCRN, and the funders such as NIHR, MRC and CRUK. 29 Mechanisms need to be developed that allow all those groups to feel they have effective input into the standards setting and review process. It is suggested that the prime group involved in regular review should be a collection of trials unit staff, coming together as a working group under the auspices of the current IS and QA working groups, with the bulk of the membership from the former (e.g. 6 IS staff, 3 QA). MHRA Funders NIHR ISWG QAWG System s Review Group Page 6

9 Trials Unit IS System s 30 If there were also an individual or two in the NIHR whose remit included supporting and developing system standards (a much more efficient and safer method of providing the necessary support, rather than trying to use occasional input from ISWG members) then they too should be part of the review group, and indeed service it with feedback and other evidence about the current functioning of standards. 31 As illustrated in the diagram, the review group could both present to and receive from (directly or via the NIHR representatives) information and requests related to the use of system standards, from funders and inspecting organisations like MHRA. 32 In the meantime it is proposed that, as a first step, the standards in this document are used as the basis of a self assessment exercise by the 40 registered trials units. That would not only allow a more structured evaluation of current practice and infrastructure, it would also begin the process of evaluating and refining these proposals. The Lists of s 33 Some general points about the lists that follow are given below, though many are also prefaced with further clarifying or specifying notes. a) Scope: In general the title of the list will indicate the range of application of that list. In some cases further qualification is given to clarify the scope. b) Applicability: Some of the standards, e.g. those relating to erdc, will obviously only apply to those units actually using that technology. c) Responsibility: Some standards may be the responsibilities of the host organisation, especially its central IT services, rather than the trials unit itself. This applies in particular to aspects of server and security management, but it may also apply, for instance, to procurement and backup policies. In such cases it is considered the responsibility of the trials unit to be aware of the relevant policies and procedures, (and where they do not exist to help create them) and to collect and collate the necessary evidence, even if they are not responsible for generating it. d) Evidence: The suggestions and comments in the evidence column are just that suggestions. They may be the most obvious things to check but there is no claim that they are the only way to verify that a standard has been reached. In any particular circumstance other evidence may be more appropriate, as long as it is independently verifiable. e) Document Classification: Mention is made throughout the lists to SOPs ( Operating Procedures), policies and guidance. Because units organise their quality management systems differently, and one unit s SOP may be another unit s guidance note, these categories should not be taken literally they simply refer to the relevant part of the quality system s documentation. Page 7

10 Trials Unit IS System s Infrastructure Requirements Procurement and Installation (Servers) 34 Note: It is recognised that hardware procurement is very often constrained by the policies of the host organisation (e.g. a University may insist that all servers are purchased from a single supplier). Evidence Status Specification: Servers and similar equipment should be specified and selected according to the specific requirements of the trials unit and the functions being supported Procurement: Purchases should comply with policies stipulated by the host organisation or show evidence of appropriate selection between alternative suppliers Installation and Build: Detailed records of builds must be available, for maintenance and safe rebuilding Procurement Planning: There should be a defined retirement / replacement policy for servers, given expected lifetimes Warranties and Support: Sufficient support arrangement should be in place for the expected lifetime of the equipment The detailed purchase specification should be retained and be justifiable in terms of requirements (a formal, pre specified requirements listing is unnecessary). Purchase documentation and any quotes should be retained Documents describing the build and / or image processes and the decisions taken therein, specific to each server (re)build Relevant policies and purchase records Documentation relating to warranties and support agreements should be retained Physical Access Control (Servers and Desktops) Evidence Status Servers: Servers must be housed within a dedicated locked room with unescorted access limited to specified individuals Desktops: Desktops should be located in locked room(s) with restricted access (e.g. by key code, swipe card) Physical Inspection Policies relating to access to servers Physical Inspection Page 8

11 Trials Unit IS System s Access Control: Access to server room should be via a key pad (which allows regular code change) Physical Inspection Policies relating to access to servers Management (Servers) Evidence Status Power Supply: The power supply to servers should be secured, e.g. by a UPS unit or secondary generator Maintenance: Necessary patches and updates should be applied in a timely but safe manner to: f) The operating system g) Anti malware systems h) Backup systems i) Major apps (e.g. SQL Server, Citrix, Clinical DBMSs, etc.) Theft and Malicious Damage: The server room should not be visible externally (e.g. through a window) Hazard Control: The server room should be fitted with heat and smoke alarms, monitored 24/7 Theft and Malicious Damage: The server room/building should be alarmed with the alarm linked to a central response centre Controlled Environment: Servers should be housed in an air conditioned environment Hazard Control: The server room should be fitted with automatic fire response measures (e.g. inert gas) Hazard Control: Water ingress (e.g. from water pipe, external flooding) that could inundate power supply / servers should be very unlikely Physical Inspection Records of any incidents when power supply restoration required An SOP describing patch evaluation and testing before deployment. Evidence (e.g. logs, patch records) of managed but timely updates to the software listed, on any particular server Physical Inspection Physical Inspection Physical Inspection Physical Inspection Physical Inspection Physical Inspection Page 9

12 Trials Unit IS System s Hazard Control: The room should be inspected regularly for infestation by pests and appropriate action taken Response System: Alerts on server failure should be sent automatically to relevant personnel 24/7 Records of inspection and any subsequent actions Relevant policies in place and individuals identified Records of alerts Logical Access Control (Servers and Networks) 35 Note: The standards below apply to general aspects of access to servers and networks, e.g. initial network logon, access to file groups. Application specific and application mediated access is dealt with in the functional standards section. Evidence Status Admin role: Servers should be protected by a highly restricted administrator password (i.e. known to essential systems staff only) Access Rights: Users should only have the minimum necessary access to carry out their roles Firewalls: External firewalls should be in place and configured to block inappropriate access Testing: Regular penetration testing by host organisation should occur, at least 6 monthly, with appropriate responses Access Rights: Users roles and rights should be reviewed and amended regularly to reflect changes Access Rights: Network password management should be enforced on all users, including regular password change and password complexity Access Rights: Access control for any remote access (e.g. via Citrix) controlled to the minimum required Admin role: The administrator password should be changed regularly Documentation of groups, membership and roles on servers Documentation of groups, membership and roles and rights on servers and network Relevant policies in place Documentation of current firewall configuration Relevant policies in place Testing results, record of any subsequent actions required Relevant policies in place Documentation of groups, membership and roles on servers Relevant policies in place Records of password change Documentation of groups, membership and roles on servers Inspection of system settings that enforce change Page 10

13 Trials Unit IS System s Firewalls: Internal firewalls should be in place and correctly configured, e.g. blocking access to other departments, students Testing: Regular security testing should be carried out by a professional third party company Monitoring: Traffic activity should be monitored and hacking attempts identified and investigated. Access Rights: Users roles and rights change requests, and their execution, should be logged Access Rights: Login activity should be logged on a central server (e.g. using Active Directory). Access Control: Desktop logins should post a blank screen or screensaver after (max.) 10 minutes and require password re activation Relevant policies in place Documentation of current firewall configuration Relevant policies in place Testing results, record of any subsequent actions required Relevant policies in place Logs of monitoring History logs of changes and their actioning Logs of login activity Physical Inspection Business Continuity Evidence Status Business Continuity: A basic BC plan should be present, covering likely action in event of possible disasters Back up: Documents detailing backup policy, procedures, restores and testing must be in place Back up: Back ups must be taken at least nightly, using a managed, documented regime Back up: Tapes should be stored in a fire proof safe Recovery: Testing of full restore procedures should take place at least annually Back up: The back up regime should involve regular offsite storage of archive tapes (e.g. monthly) Relevant documents / guidance in place Relevant SOP and / or detailed guidance in place Back up logs Physical Inspection Documentation of restore procedures and incidents Relevant procedures; physical inspection Page 11

14 Trials Unit IS System s Business Continuity: A detailed plan should be present covering specifics of actions, contacts, suppliers, etc. Business Continuity: The detailed plan should be integrated with the host organisation s BC plan and appropriate access arranged Business Continuity: Regular review, should occur, at least annually, of the detailed BC plan Back up: Transaction log backups should take place through the working day, at least 2 hourly Back up: Server / DBA environment (groups, log ins, jobs etc.) should be captured and restorable Back up: Log shipping or mirroring procedure should be in place to a warm / hot failover system in a different location Recovery: If available, testing of full restore procedures from a warm / hot failover system should take place at least annually Relevant documents / guidance in place Relevant documents / guidance in place; details of storage location / access Document history Relevant procedures, Inspection Relevant procedures, scripts, files Relevant procedures, Inspection Log of shipping processes Documentation of restore procedures and incidents In House Software Development 36 Note: (on scope) This section refers to in house systems programming, for instance using Java or Microsoft.Net, NOT the development of databases for specific trials or the SQL scripting that might be used to extract data or run DBA jobs. 37 With commercial clinical DBMSs this distinction is fairly clear, but some units write their own clinical systems using programming tools such as.net. In those cases this section refers to the underlying systems, generally as used for several trials, and again excludes the trial specific databases, forms, questions, checks etc. In other words these standards apply to the system and its development seen as a piece of software, whilst the functional set up standards are focused on the system s fitness for purpose as a tool to support the trial. 38 Note: (on risk assessment) The element of risk associated with not meeting specified standards will vary according to the type of system being developed. In general, tighter controls and procedures will be needed for any systems dealing with clinical data compared with those dealing with administrative data. Errors that emerge in the latter can usually be corrected with no lasting ill effect, but errors that emerge from clinical data systems may have introduced Page 12

15 Trials Unit IS System s systematic bias into the data or (as with randomisation systems) compromised the entire trial design. 39 For that reason (as with the section on validation that follows) the initial standard makes reference to a risk based set of policies and procedures being drawn up and followed. The unit needs to be able to justify any distinctions made inspectors may not agree with the assessments made! but at least there is scope for sensible variation of procedures. 40 Note: (on prototypes) IT development in trials units, especially for pilot and administrative systems, may proceed by a rapid prototyping approach, with users providing immediate feedback and the systems changing rapidly with successive requests, often with the underlying data model being modified at the same time. In general trials units do not have the IT resources to fully document each iteration of a system in such a process, and the procedures relating to development and its documentation may need to reflect this. Evidence Status Procedures: SOPs / policies should exist on in house development, including risk assessment Specification: A functional spec. should be available, e.g. use cases, according to procedures Documentation: Adequate documentation should exist e.g. for components, architecture, functions, calls, etc. Source Control: Source code versions should be maintained (if only via standard backup) Source Control: Source code should be stored in a full source control system (e.g. Visual Source Safe) Change Control: Change control systems should be in evidence, according to procedures Documentation: A detailed specification should be available in UML Documentation: Code should be clear, commented extensively and appropriately Relevant documents / guidance Examples of such specification Examples of such documentation Inspection of systems Inspection of systems Logs of change requests and action taken Examples of Documentation Examples of Documentation Page 13

16 Trials Unit IS System s Testing: Detailed test plans, based on specification, should exist or can be generated for systems Programming: Agreed, documented methodology / architecture(s) should be used Programming: Conventions should be followed for coding, e.g. naming / casing, commenting, in line documentation etc. Programming: Common user derived modules and controls should be available, as a developer toolbox Programming: Regular code review should take place e.g. using peer review of code Validation (General) Examples of test plans Inspection of systems Inspection of systems, source code Inspection of systems, source code Relevant policies; interviewing staff 41 Note: (on scope) This section refers to validating both purchased and in house software systems, but NOT the databases developed for specific trials, which are dealt with under functional requirements in Validation (Clinical Data Systems). In general, and in contrast to the databases, the principal responsibility for validation in this context falls to the unit IT staff. 42 Note: (on risk assessment) The element of risk associated with errors and problems in different systems will vary according to the type of system being developed. In general, any systems dealing with clinical data will require more rigorous validation than those dealing with administrative data, because of the potentially more serious consequences of error. (Indeed the validation overhead that something like a randomisation system carries is a powerful reason for centralising the design and testing of such systems). 43 Validation policies and procedures must, therefore, be risk based. The same applies to system changes, patches and updates and consequent re validation. An assessment of the degree of change involved, and specifically what system components need to be re examined, should take place before a tailored re validation exercise is launched. Evidence Status Procedures: SOPs / policies should be in place on system validation, including risk assessment Planning: Validation plans should be drawn up for any particular system, listing extent / types of tests Relevant documents / guidance Validation and revalidation plans for each major system Page 14

17 Trials Unit IS System s Testing: Testing should be carried out and recorded, normally against requirements and / or validation plan. Requirements: Where relevant test plans should relate to previously defined requirements Recording sign off: Signed and dated records should attest to the validation of systems / versions Change control: Changes should result in review of the need for revalidation User Training and Support Examples of test documentation and results for particular systems; where relevant linked to use cases and other documents. Comparisons of test and requirement documents Copies of sign off records, from users or user representatives Record of validation reviews and any associated retests Evidence Status Policies: Policies / SOPs should be in place describing induction and training requirements / policies / procedures Core Training: Induction / Training and regular update must include a) GCP b) Data protection c) Record confidentiality Records: Records of Training should be in place for all staff, held centrally and / or by the staff themselves IS Training: Users of systems must be adequately trained (may not be formal training) IS Support: Mechanism(s) for requesting support should be in place IS Support: Help and related materials should be available for users IS Support: External users should also be supported where they exist Relevant documentation Relevant documentation and logging of attendance and regular update Inspection of records Training records; training materials; user interviews; Inspection of relevant system(s); user interviews; Inspection of material Inspection of relevant systems; user interviews; Page 15

18 Trials Unit IS System s IS Support: A formal mechanism for requesting support and logging requests / actions should exist IS Training: Training should occur on systems logically separate from production systems Inspection of relevant systems and logs / databases Inspection of relevant systems; interviews with staff IT Staff Competence Evidence Status Competence: Staff should be previously competent, trained or being trained to do the job(s) required of them. Records: Record of technical training / competence should be in place for IT staff Appraisal: Training plans should be linked to annual appraisal Accreditation: There should be an adequately resourced IT accreditation / training plan with timescales Inspection of staff employment histories and training records; IT staff interviews Training records Relevant policies; appraisal documents Relevant policies and documentation General QMS Integration 44 Note: An IT standards system cannot exist in isolation from an effective general Quality Management System (QMS) covering the whole unit. In the context of an overall audit most of these issues would probably be considered elsewhere, but if the focus of the inspection was only Information Systems then all of the standards below would need to be assessed. Note that all of them are seen as critical or necessary standards. Evidence Status QMS Docs: QMS Documentation should be up to date, wellorganised, indexed and readily available. QMS Procedures: Policies / procedures should be documented, reflect good practice and be used in practice. Inspection of QMS documents and access systems Relevant documents / proformas etc.; interviews with staff Page 16

19 Trials Unit IS System s Integration: IT s documents and systems should be integrated with the overall QMS. Training: Staff members should be trained on relevant policies and procedures. QA Review: Policies and procedures should be regularly reviewed. QA Processes: QA Procedures should exist covering the writing, review and approval of QA documents, e.g. SOPs Inspection of QMS documents and systems Induction / training records Documentary evidence of review, e.g. minutes of meetings Documentary evidence of procedures, e.g. minutes of meetings, document histories Page 17

20 Trials Unit IS System s Functional Requirements Setup (Clinical Data Systems) 45 Note: Though only a few current standards are identified the key role of database setup in potentially introducing greater uniformity in trial data is reflected in the number of potential standards listed. These could be augmented further in the future, for instance to reflect the use of a central, managed metadata repository (currently included only as an example of good practice, and then only with reference to a local repository). Evidence Status Procedures: SOP / guidance should exist covering development lifecycle including specification, development, testing, deployment Requirements: The initial requirements specification must be recorded, at the least at the level of annotated CRFs Environment: The development / test environment should be logically separated from the production environment Requirements: A detailed functional specification exists identifying each item (e.g. CRFs, fields, validation logic, conditional branching). Staff: Development should be overseen and signed off by a crossdisciplinary team (e.g. programmer, trial manager, statistician) Subject Identifiers: The subject identifiers should be collected (in new trials) as per national standards Subject Identifiers: The subject identifiers should be stored using an approved encrypted algorithm Key non clinical variables: A core set of (non clinical) variables should be collected and coded according to national standards Relevant documents in place Inspection of relevant documents Inspection of systems Inspection of relevant documents Relevant policies; inspection of documents; staff interviews Inspection of new databases Inspection of systems Inspection of new databases Page 18

21 Trials Unit IS System s Repository: There is use of a metadata repository that enables to be re used Procedures: Further quality documents covering good design practice, usability, local design conventions etc. should be available Requirements: Detailed functional specifications should be stored in a database, elements numbered, versioned, part of overall design system Data s: Functional spec. and eventual design (schedule and data items) should be available as XML files in defined schema(s). Inspection of repository; staff interviews Relevant documents in place Inspection of system and its outputs Generated XML files Change management (Clinical Data Systems) Evidence Status Procedures: SOPs and Policies for clinical DB change management should be in place Requests: Individual request should be justified, itemised and recorded Requests: Changes, implication and consequent further actions should be recorded for each major change Testing: Where retesting is identified as necessary it, and the test results, are recorded Versions: A detailed functional specification should be available for each version with details of changes highlighted Relevant documents Inspection of relevant systems, documents Inspection of relevant systems, documents Inspection of relevant test documents Specification for each version; can be linked to actual software version in use. Page 19

22 Trials Unit IS System s Validation (Clinical Data Systems) Evidence Status Procedures: SOPs and Policies for trial DB validation / sign off should be in place Testing: Testing should be carried out before deployment to live environment, against basic functional spec. Testing: Final sign off must be by the end users, who have ultimate responsibility for the testing Testing: Documented testing should be performed against detailed functional specification. Testing: Failures / issues from testing should be documented for attention of developers Testing Support: Systems should be able to generate test documents (e.g. lists of validation checks) Testing: A beta testing period with entry of real data is allowed for before final sign off Relevant documents Staff interviews; sign off documents Staff interviews; sign off documents Test documentation Test documentation Test documentation and inspection of systems Relevant policies and documents Randomisation Systems 46 Note (scope): These statements only apply to randomisations mediated by computer systems within the trials unit. Units who make use of external randomisation services should obtain the corresponding documents and evidence from the organisation running that service. 47 Note: The criticality of randomisation systems is reflected in the high proportion of critical standard and standard designations below. Evidence Status Procedures: SOPs and Policies for setting up randomisation in any particular trial should be in place Procedures: SOPs and policies for managing randomisation and drug allocation in blinded trials, whilst protecting the blinding, should exist Relevant documents Relevant documents Page 20

23 Trials Unit IS System s Randomisation Lists: Any lists generated by the randomisation systems must be stored securely in electronic form and hard copy. Minimisation records: Details of each allocation decision in a minimisation system must be stored as part of the audit trail. Failover to Manual: System(s) must be in place, supported by training, to deal with a loss of normal electronic randomisation Randomisation Lists: A full audit trail should exist for the generation and any alteration of randomisation lists Documentation: The underlying logic and operations of randomisation systems, including minimisation systems, must be fully documented Validation: List generation should be validated, including the randomness of the resultant lists. Validation: Minimisation systems should be validated with test data before deployment. Monitoring: The randomness of list generation or minimisation should be monitored in the context of any particular trial Access Control (Specific Systems) Inspection of stored lists; relevant policies The audit trail records Inspection of policies, systems; interviews with staff The audit trail records Relevant documents Validation records Validation records Mechanisms and results of monitoring 48 Note (scope): This section applies to access control within, and usually mediated by, specific applications i.e. after the user has gained initial access to the network. The precise way in which access is controlled may vary with the system involved. 49 The systems that require access control are clinical databases and any other systems that include patient identifiable data. In general, however, it is good practice to have the means to apply user / group / object based access control to any system. 50 Note (groups and roles): Using pre defined groups and roles is a very common and efficient method of managing access and is generally recommended. Groups and roles should not, however, be used in too broad a fashion and provide an individual with more access than they actually need for their own purposes. Page 21

24 Trials Unit IS System s Evidence Status Procedures: SOPs and Policies for access control to specific systems should be in place Mechanisms: Each system requiring access controls should have a mechanism to restrict access to that required for any individual Change Control: Changes to access should be requested and actioned according to defined procedures and by designated individuals Records: Records of access rights, when granted and by whom, should be available Mechanisms: Membership of appropriate groups and roles should be used to manage access Review of Access: Regular review of access should be carried out to ensure it accurately reflects needs, staff changes etc. Audit Trails Relevant documents Inspection and demonstration of systems Relevant policies, records and demonstration of process Inspection of records Inspection of group / role membership and mechanisms Relevant policies in place, inspection of records 51 Note (scope): The systems that require audit trails on data items are clinical databases and any other systems that will provide data for analysis. In general, however, it is good practice to have the means to apply audit trials in any system, even if they may not always be switched on in every case. 52 Note (reason for change): Opinions vary about the usefulness of recoding the reason for change in any audit system. Anecdotal evidence suggests that many operators find the request for this information so irritating that they simply put down the first option offered. Nor is it clear to what use the information is ever put. Reason for change is therefore not included in the list below. Evidence Status Scope of Trail: Systems requiring audit trails should log all data changes: the date time of the change, the person making the change and the old and new values Inspection of systems and audit trail records; demonstration of mechanism. Page 22

25 Trials Unit IS System s Permanence: The audit trail data can not itself be altered / deleted without formal and documented mechanisms Accessibility: The audit trail for any particular data item should be visible from the user interface Data Correction Systems Inspection and demonstration of systems Inspection and demonstration of systems 53 Note: This section covers the initial detection of possible data errors and the raising, sending and receipt of data queries. In erdc systems this is usually all tightly integrated with the data entry system; in paper based systems, though the process will normally start with the clinical database s validation logic, subsequent steps may be less tightly coupled and may involve several different systems 54 One specific data correction issue also included here, as a potential standard, is the logged removal of all of an individual s data on request. 55 Whether or not a trials unit is compliant with these standards will partly depend upon the clinical database system they have selected or built. Evidence Status Procedures: SOPs and Policies for data correction should be in place Problematic Data: Simple validation (e.g. range checks) should be available on data entry Problematic Data: Complex (e.g. cross form) validation should be available at data entry Problematic Data: There should be the ability to execute validation checks via batch process, and to identify new warnings Query Tracking: Responses should be recorded when returned, identified when outstanding and resent as necessary Removal of an Individual s Data:: Systems should be able to support the logged removal of all of an individual s data (bar trial identifier) Relevant documents, records N.B. There is no expectation that data is removed from backups Page 23

26 Trials Unit IS System s QCing of data: Systems should be able to support quality checks of data by generating specified data in formats that match input format (e.g. that mimic CRFs) Query Generation: Query generation, content and selection should be under the ultimate control of users and not fully automated Reference Queries: Systems should be able to generate reference queries (i.e. information giving, not expecting a reply) not linked to specific data items (paper based systems only) Trial Administration Systems 56 Note: These systems cover a wide range of functions, from contact management and tracking data receipt, through monitoring accrual and centre research governance and agreements, to supporting drug distribution and payments. The development of systems to support these functions varies greatly between different units (though it is hoped this will improve in the future) and for that reason almost all the standards below are given as potential standards. Evidence Status Data Receipt: Systems should be able to identify / log CRFs received, with date stamp, (in paper based trials prior to actual data entry) Data Receipt: Systems should identify and report on missing late CRFs / data Data Receipt: Systems should be able to easily truncate and / or amend schedules to maintain accuracy in identifying outstanding data. Contacts Database: There should be a single, integrated contacts database in the unit (including relevant organisational data) Logging systems (automatic with erdc systems) Page 24

27 Trials Unit IS System s Contacts Database: Names and addresses information should be stored in e GIF format Contacts Database: People identifiers should be able to be mapped to NIHR identifiers Contacts Database: Organisation / location identifiers should be able to map to NHS equivalents, for NHS based locations Contacts Database: The contacts management system should be integrated with other trial administration systems Accrual: Systems should be able to provide up to date accrual data, by centre and by trial Accrual: Systems should be able to generate subject level accrual records in the format required for reporting to NIHR Data Coding 57 Note: Because of the relatively undeveloped usage of formal coding systems, all of the standards below are categorised as potential standards. Evidence Status Procedures: SOPs and Policies related to coding should be in place (to promote consistency and proper use of versions) Coding Systems: Coding should, as far as possible, use named standard systems for particular types of data Coding Systems: Coding should, as far as possible, use consistent systems across different trials Staff: Coding / categorisation should only be carried out by personnel trained on the relevant systems Relevant documents Relevant documents and policies (Systems including but not limited to MedDRA, SNOMED CT, ICD, LOINC) Relevant documents and policies Staff interviews Page 25