Notes
Article history
The research reported in this issue of the journal was funded by PGfAR as project number RP-PG-0108-10049. The contractual start date was in September 2014. The final report began editorial review in April 2015 and was accepted for publication in November 2015. As the funder, the PGfAR programme agreed the research questions and study designs in advance with the investigators. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The PGfAR editors and production house have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the final report document. However, they do not accept liability for damages or losses arising from material published in this report.
Declared competing interests of authors
Kim Cocks declares money for employment outside the submitted work from University of York and Adelphi Values.
Permissions
Copyright statement
© Queen’s Printer and Controller of HMSO 2016. This work was produced by Wright et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.
Chapter 1 Introduction
Patient involvement has become a universal mantra in health services. However, busy health professionals frequently struggle to achieve real involvement in practice. Although intentions are well-meaning, the methods used – patient representatives, patient surveys and patient interviews – are often tokenistic, unreliable and unsustainable. This programme of research emerged from a common chorus of requests from the regional clinical networks of the Yorkshire Quality and Safety Research group: how can we involve patients in improving patient safety? The paradox for the development of a programme centred on patient involvement was that, when we consulted with our patient and public panels, there was very little understanding about patient safety and general surprise that health services were unsafe.
The programme of research was developed to address the deficit of evidence about patient involvement in research and to design, develop and evaluate innovative approaches to engage patients in preventing patient safety incidents (PSIs) and protecting themselves from unintended harm in collaboration with health professionals. Our aim was to develop feasible and effective methods that would allow practitioners to embed the patient voice in routine care.
Background
Estimates suggest that that approximately 5–10% of hospitalised patients in high-income countries experience harm and that about one-third of harmful events are preventable. 1–4 In the NHS this translates to between 300,000 and 1.4 million adverse events each year, with estimated costs of £2B a year for the extra time that patients have to spend in hospital, £1B for associated infections and > £400M in clinical negligence claims. It is against this background that patient safety has become a major priority for the NHS.
A PSI has been defined as any unintended or unexpected incident that could have led to or did lead to harm to one or more patients. 5 This includes a spectrum of events from near misses through to PSIs causing severe harm or death. Strategies to improve safety and reduce PSIs have focused on changing systems of care and professional behaviour. However, there has been a growing interest in involving patients in safety initiatives. The World Health Organization’s (WHO) World Alliance for Patient Safety (WAPS) has made the mobilisation and empowerment of patients one of its six action areas that will be taken forward in its Patients for Patient Safety programme. 5 The approach advances the development and use of interventions to promote and support patients’ (and their representatives’) roles in securing their own safety in health-care contexts. Patients are in a unique position to contribute to both learning about health-care safety and improvements to the safety of health-care systems by feeding information about safety issues that they have identified or experienced into local and national safety reporting systems. Support for patient involvement in safety-orientated activities also reflects the broad policy aim for people to be more involved in their care in general. 6
Involving patients in the safety of their care is a relatively new focus for health services and researchers, with calls to focus on the patient role in safety surfacing only around a decade ago. 7 The need to engage patients in their safety has rapidly gained traction, however, with a range of research and policy activity now focused on this area. Patients are being invited to engage in safety initiatives across a variety of health-care services and settings, from hospital-aqcuired infections in acute hospitals through to service development activity and, increasingly, primary care. There has also been an increased spotlight on patient involvement in safety from policy makers, with the challenge from recent major national reports from Francis,8 Keogh9 and Berwick10 for NHS trusts to reduce patient harm and strengthen the patient voice. Indeed, these reports specifically refer to the need to elicit, and respond to, the concerns of patients. However, the methods to achieve this systematically, using robustly developed and evidence-based tools, have hitherto been unavailable to health services.
Despite the international emphasis on patient involvement in safety there is a dearth of research evidence on its acceptability to patients and, as yet, no robust evidence that such involvement leads to improvements in safety. Prior to conducting this programme of research, reviews of the literature undertaken by the research team11 highlighted the scarcity and poor quality of studies evaluating the effectiveness of patient involvement strategies. The evidence that did exist indicated that patients are willing and able to participate in error prevention strategies12 and have the potential to improve safety. 7,11,13,14 However, there was clearly a need to understand further the ways in which safety improvement can be enhanced through patient involvement and the benefits (and disbenefits) to patients of taking a more informed and active role in the safety of their care. Early work carried out by the team prior to this programme of research suggested that patient involvement in safety initiatives may have unwanted effects (e.g. patient perceptions of safety; loss of trust; delegation of responsibility for safety; social inequalities in health-care experiences and outcomes). 13
A criticism of the extant literature on patient involvement has been that it lacks a coherent theoretical or conceptual framework15 (see Chapter 4). This is problematic for the field as utilising theory in the development and testing of interventions facilitates the articulation of underlying mechanisms through which intervention activities are hypothesised to lead to change. 16 The overarching theoretical framework for this programme of work was a systems approach to patient safety, perhaps most typified by the organisational accident model. 17,18 This theory suggests that patient harm results from the interaction of a wide range of contributing factors, from ‘upstream’ issues such as management decisions and policy through to ‘downstream’ local working conditions and, ultimately, active failures at the point level of the individual practitioner. This programme of research aimed to design and test patient-centred interventions that run across the range of contributing factors to patient harm. Figure 1 summarises this approach with reference to the organisational accident model. The individual workstreams are described in more detail in the following section but broadly this programme aimed to provide interventions for (1) engaging with patients in the assessment of ‘upstream’ contributory factors (termed here ‘assessing risk’); (2) engaging with patients in the identification of harm (‘reporting incidents’); (3) facilitating the direct engagement of patients in reducing risk and ameliorating harm; and (4) engaging with patients to improve the patient safety education and training of health-care professionals.
An applied health research programme on patient involvement in patient safety
A variety of roles exist through which patients can potentially contribute to enhancing the safety of care in the NHS. Four key priority areas were identified for the focus of this research programme:
-
assessing risk
-
reporting incidents
-
direct engagement in preventing errors
-
education and training.
The aim of the programme was to undertake high-quality research in each of these areas to provide clear guidance to the NHS on how patient involvement in enhancing the safety of their care can be efficiently and effectively achieved.
-
Assessing risk. Health organisations have been slow in responding to the need for changes in ‘systems’ of care. 17,18 Reason19,20 argues that organisational failures are easier to diagnose and manage proactively than individual errors. Based on these ideas, proactive risk measurement tools have been developed in high-risk industries and in the NHS to monitor an organisation’s ‘safety health’. 21–23 However, no such measurement tools have been developed for customers or patients.
Patients are well placed to observe the organisation of their care and the practices around them and could provide useful information if tools could be designed to make it easy for them to provide this information. Scales measuring patients’ perceptions of health care are available24–26 but have been criticised for being subjective and unreliable and having little validity. 27,28 There is a need for reliable and valid tools that allow patients the opportunity to provide feedback on the safety of their care environment to inform local and organisational changes to improve patient safety.
-
Learning from error is a key element of patient safety. 29 Current methods of learning focus on reporting, audit and case note review with the limitations of incomplete reporting (particularly errors/near misses) and failure to identify underlying causes of errors. 30–33
Patients can report PSIs that are undetected by other mechanisms34–37 but are rarely given the opportunity. Patients want to ensure that lessons are learnt from PSIs to prevent harm to others and, unlike staff, are not constrainted by a blame or organisational culture. 38 We addressed gaps in the evidence by developing informative patient reporting systems and evaluating their impact on organisational learning.
Studies have demonstrated the feasibility and value of patient reporting34–36 but no study to date has attempted to evaluate the most effective method of patient reporting or attempted to collect patient reports of events that did not result in harm (e.g. near misses). Finally, no study has systematically linked reporting of PSIs to mainstream quality improvement mechanisms.
-
Direct engagement. Patients are potential contributors to improving safety – as one of the barriers to harm – but they are only one part of the safety management system. There is a need to understand how patients can best be involved and what the impact is on safety of care and other factors such as patients’ trust in, and experience of, care. Despite the wide support and clear rationale for patient engagement in their own safer care, there is no robust evidence that such involvement leads to improvements in safety and little evidence on the acceptability to patients or staff.
The provision of safety-related advice is the most common method used by health-care providers to promote patients’ contributions to their own safety. Examples include the SPEAK UP39 campaign in the USA and Please Ask40 in the UK, as well as the National Patient Safety Agency’s (NPSA) cleanyourhands campaign. 41 However, there is scant evidence on the effectiveness of such interventions. In addition, potential unintentional adverse consequences (such as erosion of trust) and their acceptability to patients have rarely been considered. 13,42,43 This programme developed and tested theoretically grounded interventions to enhance patient engagement in improving their safety in high-risk areas of clinical practice.
-
Education and training of staff is a priority for promoting patient safety culture in health care. There is evidence that education, and in particular experiential learning, can be effective in changing attitudes and behaviours of health professionals as well as act as an important lever for improving patient outcomes. 44,45 Patients can make valuable contributions to teaching clinicians with benefits for both the learners and the patients. 46,47 Learning from patients encourages reflection, insight and a user perspective on improving health services. Personal narratives can aid understanding about patient experiences, develop professionalism and promote illness scripting to support clinical decision making. 48,49 This programme addressed the lack of evidence in this field and evaluated the impact of a generalisable patient-centred safety training session. However, there is a notable gap in the evidence for effective educational interventions promoting knowledge, skills and attitudes to patient safety. 50
Programme management
The programme benefited from the experience of a well-established team – the Yorkshire Quality and Safety Research (YQSR) group [see www.bradfordresearch.nhs.uk/research-teams/quality-and-safety-research-team (accessed 29 May 2016)]. The YQSR group was set up in 2008 to provide a platform for clinicians, patients, academics and senior managers to implement applied health research in the field of quality and safety improvement. This programme grant was aligned to the structure and goals of the YQSR group. A quarterly steering group of all 12 coapplicants, patient panel chairs and additional expert advisors was established to oversee the implementation of the research programme. The steering group alternated meetings in the three main participating centres of Bradford, Leeds and York. Performance management templates were designed for each study to provide a standardised report on (1) progress according to objectives, (2) changes in the original protocol, (3) current challenges and (4) impacts and publications. An independent trials management group was established in 2012 to provide monitoring and scrutiny of the randomised trial.
Patient and public involvement
Patient and public involvement (PPI) was clearly a central commitment of this research programme and significant effort was made at the outset to establish an advisory panel comprising lay people, recruited to contribute to the four individual projects as they progressed from design to completion. Guidance provided by INVOLVE51,52 along with the previous experiences of those researchers on the programme were used to plan and conduct key aspects of a PPI process (recruitment, training, reimbursement), but the team recognised that ongoing evaluation and adaptation were required to promote effective coproduction in research. The lessons from this evaluation were captured and are presented in the final chapter of this monograph.
The programme was also developed with the strong involvement (as coapplicants) of Action against Medical Accidents (AvMA), the national charity for patient safety and justice that has been the champion of patient rights in patient safety since 1982.
Clinical engagement
The programme was set up to promote strong clinical engagement in the research. Although the focus of the research was on patient engagement, we recognised from the start that a unilateral approach could be threatening to NHS staff and so hamper their involvement. Effective clinical engagement has been fundamental to the success of the programme and careful consideration and planning was undertaken from the beginning to consult and involve senior managers and clinical staff in each NHS trust in the design and implementation of all of the research studies. This involvement has been crucial in the ultimate receptiveness of the interventions and future spread and adoption. In the latter 18 months of the programme the team has worked closely with the Improvement Academy, part of the Academic Health Science for Yorkshire and Humberside, to ensure that our research will be embedded into routine clinical practice [see www.yhahsn.org.uk/improvement-academy/ (accessed 29 May 2016)].
Chapter 2 Assessing risk: a systematic review of factors contributing to patient safety incidents in hospital settings
Abstract
Background: Existing frameworks used to understand the factors contributing to PSIs are theoretically informed but are not derived from empirical evidence.
Objective: The aim of this systematic review was to develop a ‘contributory factors framework’ from a synthesis of empirical work that summarises factors contributing to PSIs in hospital settings.
Methods: A mixed-methods systematic review of the literature was conducted.
Data sources: Electronic databases (MEDLINE, PsycINFO, ISI Web of knowledge, Cumulative Index to Nursing and Allied Health Literature and EMBASE), article reference lists, patient safety websites, registered study databases and author contacts.
Eligibility criteria: Studies were included that reported data from primary research in secondary care aiming to identify the contributory factors to error or threats to patient safety.
Results: In total, 1502 potential articles were identified; 95 papers (representing 83 studies) were included and 1676 contributory factors were extracted. Initial coding of contributory factors by two independent reviewers resulted in 20 domains (e.g. team factors, supervision and leadership). Each contributory factor was then coded by two reviewers to one of these 20 domains. The majority of studies identified active failures (errors and violations) as factors contributing to PSIs. Individual factors, communication and equipment and supplies were the other most frequently reported factors within the existing evidence base.
Conclusions: This review has culminated in an empirically based framework of the factors contributing to PSIs. This framework has the potential to be applied across hospital settings to improve the identification and prevention of factors that cause harm to patients.
Chapter rationale
This chapter deals with the first key priority addressed in our programme of work: assessing risk. If patients are to be in a position to be part of proactive management of patient safety, tools are required that allow the collection of information from patients about factors known to contribute to patient safety. Such tools should be based on extant evidence as well as be theoretically driven. This chapter presents a systematic review of the empirical work considering contributory factors to PSIs in health-care settings.
Introduction
Since the early 1990s high-risk organisations have adopted a systems approach to safety management. 53,54 This approach recognises that the immediate causes of PSIs are errors made by people at the frontline of operations (e.g. in the case of medication administration, this is most likely to be a nurse). However, the importance of a systems approach is that it recognises that the organisations within which people work have inherent weaknesses (latent failures) that can arise from decisions made at more senior levels (e.g. plans agreed, buildings designed, staffing levels approved, equipment procured) and those external to the organisation (e.g. policies imposed, targets set, funding decisions, education provision) and that these failures manifest themselves in local working conditions that promote errors. Thus, a focus on individual responsibility for errors is likely to be ineffective as an incident reduction strategy. Based on this approach it can be argued that there are two main strategies to reduce medical error: reactive and proactive. The first relies on learning from (reacting to) previous incidents to minimise error in the future, whereas the second is concerned with prospectively identifying the latent failures within organisations that represent the preconditions for errors and addressing these before a serious event occurs. Incident reporting systems, root cause analysis of serious incidents and case note review are all tools that have the potential to provide data about the prevalence and/or causes of medical errors. However, there is growing frustration with incident reporting systems, with low rates of reporting, poorly designed reporting tools and inadequate feedback all being blamed for providing data that have little value in improving safety. 55,56 Moreover, learning across all of these tools is predicated on the collection of data about the factors contributing to error. 57,58 To date, there is no evidence-based and standardised list of contributory factors that can be used as a basis for understanding causation. Without this, reactive systems are unlikely to provide the answers we are looking for.
In other industries, such as nuclear power and transport, measurement tools have been developed to assess the extent to which organisational factors (e.g. supervision, planning, communication, training, maintenance) represent a failure in the system. 53,59 These tools do not rely on the retrospective analysis of adverse incidents but instead allow the proactive monitoring of an organisation’s safety. However, before such tools can be developed it is necessary to know what represents a latent failure within that particular industry. This systems approach to patient safety has been well established in health care since the publication of To Err is Human by the US Institute of Medicine17 and subsequent policy documents in the UK29,60 and a number of frameworks for studying latent failures have been proposed [e.g. Eindhoven classification,61 WHO patient safety classification,62 London Protocol,63 Veterans Affairs root cause analysis system,64 Australian Incident Monitoring System (AIMS)57]. However, these frameworks are limited by the lack of an empirical basis and a reliance on classifications from non-health-care settings65,66 that are very different from the structure and nature of health care.
The growing emphasis on systems thinking over the past 20 years in health care67 has meant that there is now a significant body of evidence in the scientific literature (e.g. retrospective interview studies, real-time observational studies and aggregated data from incident reporting studies) that can be used as an empirical basis for generating a classification of the contributory factors that impact on health care in hospitals. Such a classification could serve to promote more effective organisational learning through the redesign of incident reporting systems and more effective root cause analysis of health-care incidents. Such a classification system could also inform the development of intervention strategies to improve safety defences or directly address systems failures68,69 and guide the measurement tools used to evaluate policy and service level interventions. 70
Thus, the main aim of this literature review was to produce a framework of contributory factors that contribute to PSIs within hospital settings. As such, it represents the first attempt to summarise the empirical evidence in this area and to use this evidence to develop a clearly defined and hierarchically ordered framework that describes contributory factors from proximal (sharp end) to distal (latent).
A secondary aim was to identify contributory factors that feature most strongly in the literature and which might therefore be appropriate targets for interventions designed to improve patient safety. Finally, we sought to assess the extent to which the contributory factors that were identified most frequently varied as a function of method of elicitation, hospital setting and whether or not a human factors expert was involved in their identification [A PICOS (participants, interventions, comparisons, outcomes, study design) statement is not relevant here because the review does not address a question relating to the effectiveness of an intervention. The review was not registered and no protocol exists.]
Methods
Data sources and searches
The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines were followed in conducting this systematic review [see www.prisma-statement.org/ (accessed 8 August 2016)]. A variety of strategies was used to search the literature to 20 November 2010. Clear identification of studies that identified the contributory factors in active failures was hampered by the lack of consistent terminology used across studies. First, search terms were developed and electronic database searching was performed across the following databases: MEDLINE, PsycINFO, ISI Web of knowledge, Cumulative Index to Nursing and Allied Health Literature (CINAHL) and EMBASE. Second, the reference lists of all downloaded articles were manually searched to identify potentially relevant papers. Third, a number of patient safety organisation websites were searched to identify other potentially relevant published or unpublished reports. Fourth, registered study databases were searched using the term ‘patient safety’ to identify any ongoing or finished projects relevant to the current review that may have provided relevant material. A summary of these search strategies can be found in Appendix 1. Finally, key patient safety authors were contacted and asked to provide details of any relevant published or unpublished reports. This search strategy identified a total of 1502 potential articles. All article titles and abstracts were reviewed for inclusion (by RM). A random sample of 10% of the titles and abstracts was double coded with respect to inclusion or exclusion (by RS). The kappa value of 0.73 indicated an acceptable level of agreement. If there was disagreement about inclusion or exclusion, the full-text article was obtained and reviewed (by RM and RS) and agreement reached. In total, 95 papers (representing 83 studies) met the inclusion criteria and were included in the review (Figure 2).
Study selection
Studies were included in this review if they reported data from:
-
secondary care or hospital environments
-
primary research that either specifically aimed to identify the contributory factors (often referred to as ‘causes’ within studies) of active failures or threats to patient safety or reported a clear framework for the categorisation of contributory factors of errors or threats to patient safety in the results section.
Studies were excluded if they reported data from:
-
active failures as causes of errors or threats to patient safety rather than underlying latent domains (e.g. only specific human ‘errors’ causing failure of a barcode checking system71)
-
contributory factors of behaviours or processes that were not active failures (e.g. factors affecting the likelihood of staff to report serious medication errors in hospitals72)
-
case studies reporting contributory factors of a specific adverse event (e.g. Chassin and Becher73)
-
studies that applied proactive risk assessment methods to identify potential failures (e.g. failure mode and effects analysis) as these papers focused on exploring potential problems of specific elements of a health-care system or process.
Data extraction and quality assessment
The study characteristics of 83 data sets were coded. All included articles were blind double coded (by RM and RS) and data were extracted and uploaded onto a Microsoft Access® database (2010; Microsoft Corporation, Redmond, WA, USA). Kappa values are reported only for dichotomous variables. Articles were coded according to the following characteristics: country of origin; description of setting; study method; study sample; theoretical frameworks informing the research (following the quality coding framework of Sirriyeh et al. ,74 studies were coded as explicit use of theory, i.e. explicit statement of theoretical framework applied to the research; specific use of theory, i.e. reference to specific theoretical basis; broad use of theory, i.e. reference to broad theoretical basis; or none at all, i.e. no theory mentioned); whether identification of contributory factors was a primary or secondary aim of the study (κ = 0.66); whether contributory factors were identified by a human factors expert (κ = 0.79); and, finally, whether patients or staff reported the raw data used to identify contributory factors (κ = 1, perfect agreement). Studies varied in the extent to which they used primary data collection methods to elicit contributory factors or whether they used a predefined set of contributory factors; therefore, the following additional information was gathered to glean more details about the elicitation of contributory factors: whether the contributory factor list was fully developed before empirical data were collected (yes or no, κ = 0.74); the method for eliciting contributory factors (if different from the overall study method); and any further details about the sample used to elicit contributory factors if different or a subset of the overall study sample. Disagreements were discussed and resolved. As we were interested in how contributory factors were identified, regardless of whether this was the primary aim of the study, we did not engage in any further ‘quality assessment’ coding, as often very little detail about how contributory factors were identified was reported. All included papers and extracted data can be found online [see http://qualitysafety.bmj.com/content/suppl/2012/03/14/bmjqs-2011-000443.DC1/Final_Appendix_v2_2_11_11.pdf (accessed 8 August 2016)].
All contributory factors reported within the papers were transcribed verbatim into a Microsoft Excel® spreadsheet.
Data synthesis analysis
To develop the contributory factor framework, two of the authors (RL, a human factors expert, and RM, a behavioural scientist) first independently grouped all of the transcribed verbatim contributory factor items into categories according to their underlying semantic meaning (e.g. equipment not working, equipment failure and equipment malfunction were all be grouped as equipment failure). Items could be categorised into more than one category. Second, each author further grouped these categories into their higher-order domains (e.g. equipment failure was grouped with equipment unavailability and insufficient supplies to become ‘equipment and supplies’). At this stage the authors did not explicitly distinguish between latent conditions and local working conditions. Next, the two authors met to discuss and agree the number of higher-order domains and label and define them (e.g. ‘equipment and supplies’ was defined as ‘the availability and functioning of equipment and supplies’). A decision was made to include all factors contributing to PSIs in this framework, both the proximal factors (e.g. active failures) and those more distal or external to the organisation (e.g. design of equipment and supplies and external policy context). This process resulted in a framework of 19 domains with a definition for each (Figure 3 and Table 1). Finally, the same two authors applied the framework, again independently, to the raw data to classify each of the contributory factors based on the framework and to assess agreement. At first, 10% of the factors were coded and at this stage agreement was 55%. Following clarification and modification of the definitions (e.g. ‘human factors design of equipment and supplies’ became ‘design of equipment and supplies’), the remaining 90% of the contributory factors were coded. Agreement at this second stage was 90%. Disagreements were discussed and resolved through consensus.
Factor | Definition |
---|---|
Active failures | Any failure in performance or behaviour (e.g. error, mistake, violation) of the person at the ‘sharp end’ (the health professional) |
Communication systems | Effectiveness of the processes and systems in place for the exchange and sharing of information between staff, patients, groups, departments and services. This includes both written (e.g. documentation) and verbal (e.g. handover) communication systems |
Design of equipment and supplies | The design of equipment and supplies to overcome physical and performance limitations |
Equipment and supplies | Availability and functioning of equipment and supplies |
External policy context | Nationally driven policies/directives that impact on the level and quality of resources available to hospitals |
Individual factors | Characteristics of the person delivering care that may contribute in some way to active failures. Examples of such factors include inexperience, stress, personality, attitudes |
Lines of responsibility | Existence of clear lines of responsibility clarifying accountability of staff members and delineating the job role |
Management of staff and staffing levels | The appropriate management and allocation of staff to ensure an adequate skill mix and staffing levels for the volume of work |
Patient factors | Those features of the patient that make caring for them more difficult and therefore more prone to error. These might include abnormal physiology, language difficulties and personality characteristics (e.g. aggressive attitude) |
Physical environment | Features of the physical environment that help or hinder safe practice. This refers to the layout of the unit, the fixtures and fittings and the level of noise, lighting, temperature, etc. |
Policy and procedures | The existence of formal and written guidance for the appropriate conduct of work tasks and processes. Where procedures are available but contradictory, incomprehensible or of otherwise poor quality |
Safety culture | Organisational values, beliefs and practices surrounding the management of safety and learning from error |
Scheduling and bed management | Adequate scheduling to manage patient throughput, minimising delays and excessive workload |
Staff workload | Level of activity and pressures on time during a shift |
Supervision and leadership | Availability and quality of direct and local supervision and leadership |
Support from central functions | Availability and adequacy of central services to support the functioning of wards/units. This might include information technology support, human resources, portering services, estates or clinically related services such as radiology, phlebotomy or pharmacy |
Task characteristics | Factors related to specific patient-related tasks that may make individuals vulnerable to error |
Team factors | Any factor related to the working of different professionals within a group that they may be able to change to improve patient safety |
Training and education | Access to correct, timely and appropriate training, both specific (e.g. task related) and general (e.g. organisation related) |
To ensure that the framework of domains had relevance and meaning beyond the two authors who developed it, 10% of the data sets (n = 9) and their respective contributory factors were extracted from the database and sent to two academic health professionals (IW, a general practitioner, and JW, a hospital physician). Both were provided with instructions and definitions of the domains and were asked to code each of the contributory factors using the framework. They were asked to include ‘can’t code’ when they were uncertain of the correct response. Initial agreement between the first two authors and each academic health professional was 62.5% (RL and RM with IW) and 85% (RL and RM with JW). After discussion with the first independent reviewer (IW) and further minor modifications of the definitions of domains, agreement rose to 80.1%. Given that agreement with the second reviewer was initially high (85%), further discussion with this reviewer was not deemed necessary.
As noted in the introduction, contributory factors can vary according to their level of proximity to the ‘active failure’ being accorded to the individual, local working conditions (e.g. management of staff and staffing levels) or more latent conditions (e.g. design of equipment and supplies). The contributory factors elicited in this review also reflected these distinctions. In a final step, an expert panel of clinicians (n = 5), researchers (n = 8), managers (n = 2) and lay people (n = 2) were provided with a list of all contributory factors and definitions and asked to identify the extent to which each factor was removed in time and space from PSIs on a 5-point scale from 1 (very close in time and space) to 5 (very distant in time and space). Contributory factors scoring 4 or 5 were deemed to be more ‘latent’ organisational factors, whereas those scoring 2 or 3 were deemed to be more related to local working conditions or situational factors. This allowed us to ground the taxonomy in a hierarchical framework, which we have described in Figure 3.
Results
Ninety-five studies fulfilled the inclusion criteria, reporting data from 83 independent data sets. 11,75–168 A total of 1676 contributory factors were extracted. Studies reported a median of 15 contributory factors each [interquartile range (IQR) 8–27]. The lowest number of contributory factors extracted from a study was three158 and the maximum was 100. 107 All coded information about studies can be found online (see http://qualitysafety.bmj.com/content/suppl/2012/03/14/bmjqs-2011-000443.DC1/Final_Appendix_v2_2_11_11.pdf). For clarity of exposition, individual references are not included next to summaries of study characteristics except to highlight individual studies. Interested readers can find this information in the online appendix tables. A table containing all of the extracted contributory factors and their categories is available from the first author on request.
Country of origin
The majority of the studies identified by this review were conducted in the USA (n = 34), the UK (n = 13), Australia (n = 7) and Canada (n = 5). One study reported multinational data from 27 countries,162 one reported data from three countries147 and one reported data from the USA and Canada. 167
Setting
Thirty studies reported data collected from general hospital settings. Other studies focused particularly on intensive care on its own (n = 17), in combination with coronary care (n = 1) or in combination with medicine and surgery (n = 1); surgery settings (n = 16), including in combination with intensive care;163 and anaesthesia (n = 7), maternity (n = 2), pharmacy (n = 2) or transfusion settings (n = 2). Other settings included geriatric and cardiovascular wards158 and the emergency department. 154 Two studies reported incidents from US general reporting systems (the US Vaccine Adverse Event Reporting System92 and the National Medication Error Database166). Finally, one study reported data from a cohort of student nurses. 120
Aim of study (primary/secondary) and theoretical basis
The majority of studies explicitly aimed to identify contributory factors (or more commonly referred to as ‘causes’) of errors or active failures (n = 55). Over half of the included studies made no reference to a theoretical basis driving the identification of contributory factors (n = 48). When theory was explicitly mentioned and related to methodology (n = 8), all studies referred to Reason’s20 model of accident causation. Only six studies included explicit human factors expertise in the elicitation of contributory factors.
Description of empirical data collection methods
One-third of studies (n = 30) reported data collected as part of an incident reporting scheme based within the hospital; see http://qualitysafety.bmj.com/content/suppl/2012/03/14/bmjqs-2011-000443.DC1/Final_Appendix_v2_2_11_11.pdf for details. Typically, these studies reported the frequency with which staff identified contributory factors of a reported incident from a predefined list (e.g. Beckmann et al. 80) but they also included studies in which free-text input from incident reports was analysed qualitatively (e.g. Nast et al. 135). Other papers reported results from observational studies (n = 14), interviews (n = 9) and focus groups (n = 1), surveys (n = 8) or case note reviews (n = 4). Seventeen studies reported using multiple methods; see http://qualitysafety.bmj.com/content/suppl/2012/03/14/bmjqs-2011-000443.DC1/Final_Appendix_v2_2_11_11.pdf.
Use of a contributory factors framework
The coders assessed the extent to which studies had generated a deductive predefined list of contributory factors (e.g. the London Protocol63) that then informed data collection or whether studies used inductive methods to elicit contributory factors from participants. For example, within incident reporting studies, the deductive use of lists would take the form of a tick box list given to participants, whereas within interview studies a list of closed questions might be used to elicit responses about particular contributory factors. The use of a deductive list in these contexts means that no new contributory factors can be elicited from participants; rather, only the prevalence with which they are endorsed can be assessed.
In total, 49 studies used a predefined contributory factor list as a basis for data collection. Twenty-six of these were based solely on previous frameworks (seven studies used a variation of the Australian Incident Monitoring Study framework,57 three studies used the Eindhoven classification,61 two studies used the London Protocol63 and 14 reported frameworks from miscellaneous previous publications). Six studies78,109–111,131,148 used a combination of literature reviewing and author or other expert opinion to identify the list of contributory factors, one of which used previous literature (in addition to pilot work not reported in the paper140), and one study used only expert opinion. 163 Fifteen studies that used a predefined contributory factors list did not specify how that list was obtained. The use of a contributory factors framework was unclear in two studies. Of the 35 remaining studies that elicited contributory factors from analysis of primary data, 25 used qualitative methods such as interviews, focus groups or free-text coding of incident reports, eight used observational methods and two used both.
Identification of contributory factors
As described in the methods section, through the coding of the 1676 contributory factors, a list of 20 contributory factor domains was independently identified by two reviewers (RM and RL) and this list was verified by two further coders (IW and JW, both clinicians). Based on this list we also sought to identify contributory factors that were identified most frequently within the literature. The number of times each of the 20 contributory factors was identified across all of the study settings is shown in Table 2 (total column). Across study settings, the five contributory factors identified most frequently were active failures (slips, lapses, mistakes, deviations from policy) (18.2%), individual factors (11.0%), communication systems (7.9%), equipment and supplies (6.6%) and management of staff and staffing levels (5.8%). This pattern varied little according to the hospital setting in which the data were collected, with active failures and individual factors consistently being the most frequently identified contributory factors. However, there was some variation. For example, team factors (8.5%) was among the top five contributory factors for surgery but for no other setting. For anaesthesia, equipment and supplies was the second most cited contributory factor, accounting for 15.2% of the codes. Physical environment was also among the top five factors for anaesthesia. For the general hospital setting, patient factors (7.4%) was among the highest-ranked contributory factors but equipment and supplies was not.
Domain | Anaesthesia (n = 7) | General hospital (n = 30) | Intensive care (n = 19) | Surgery (n = 16) | Other (n = 11) | Total | ||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
Count | % | Count | % | Count | % | Count | % | Count | % | Count | % | |
Active failures | 17 | 16.2 | 79 | 13.5 | 112 | 29.0 | 51 | 14.0 | 46 | 19.3 | 305 | 18.2 |
Communication systems | 2 | 1.9 | 47 | 8.0 | 35 | 9.1 | 33 | 9.1 | 15 | 6.3 | 132 | 7.9 |
Design of equipment and supplies | 1 | 1.0 | 16 | 2.7 | 9 | 2.3 | 8 | 2.2 | 17 | 7.1 | 51 | 3.0 |
Equipment and supplies | 16 | 15.2 | 20 | 3.4 | 31 | 8.0 | 33 | 9.1 | 10 | 4.2 | 110 | 6.6 |
External policy context | 0.0 | 7 | 1.2 | 0.0 | 0.0 | 2 | 0.8 | 9 | 0.5 | |||
Individual factors | 16 | 15.2 | 74 | 12.7 | 41 | 10.6 | 37 | 10.2 | 16 | 6.7 | 184 | 11.0 |
Lines of responsibility | 0.0 | 9 | 1.5 | 1 | 0.3 | 4 | 1.1 | 1 | 0.4 | 15 | 0.9 | |
Management of staff and staffing levels | 3 | 2.9 | 36 | 6.2 | 23 | 6.0 | 23 | 6.3 | 12 | 5.0 | 97 | 5.8 |
Patient factors | 2 | 1.9 | 43 | 7.4 | 19 | 4.9 | 9 | 2.5 | 4 | 1.7 | 77 | 4.6 |
Physical environment | 5 | 4.8 | 15 | 2.6 | 16 | 4.1 | 16 | 4.4 | 9 | 3.8 | 61 | 3.6 |
Policy and procedures | 0.0 | 27 | 4.6 | 15 | 3.9 | 4 | 1.1 | 5 | 2.1 | 51 | 3.0 | |
Safety culture | 0.0 | 10 | 1.7 | 4 | 1.0 | 4 | 1.1 | 8 | 3.4 | 26 | 1.6 | |
Scheduling and bed management | 0.0 | 7 | 1.2 | 0.0 | 9 | 2.5 | 2 | 0.8 | 18 | 1.1 | ||
Staff workload | 1 | 1.0 | 23 | 3.9 | 9 | 2.3 | 5 | 1.4 | 7 | 2.9 | 45 | 2.7 |
Supervision and leadership | 4 | 3.8 | 17 | 2.9 | 7 | 1.8 | 8 | 2.2 | 4 | 1.7 | 40 | 2.4 |
Support from central functions | 1 | 1.0 | 17 | 2.9 | 9 | 2.3 | 13 | 3.6 | 14 | 5.9 | 54 | 3.2 |
Task characteristics | 1 | 1.0 | 5 | 0.9 | 6 | 1.6 | 4 | 1.1 | 4 | 1.7 | 20 | 1.2 |
Team factors | 1 | 1.0 | 13 | 2.2 | 6 | 1.6 | 31 | 8.5 | 2 | 0.8 | 53 | 3.2 |
Training and education | 1 | 1.0 | 19 | 3.3 | 8 | 2.1 | 3 | 0.8 | 8 | 3.4 | 39 | 2.3 |
Outcomea | 7 | 6.7 | 9 | 1.5 | 1 | 0.3 | 27 | 7.4 | 13 | 5.5 | 57 | 3.4 |
Can’t code | 27 | 25.7 | 91 | 15.6 | 34 | 8.8 | 41 | 11.3 | 39 | 16.4 | 232 | 13.8 |
Grand total | 105 | 100.0 | 584 | 100.0 | 386 | 100.0 | 363 | 100.0 | 238 | 100.0 | 1676 | 100.0 |
Table 3 shows the contributory factors identified by each of the different study methodologies. Studies using incident reporting methodology more commonly identified active failures than interview or observational studies. This is intuitive as generally incident report forms are limiting in terms of the details of the event that can be recounted and the options for contributory factors available to the reporter. Interview studies appeared to more commonly identify individual factors and staff workload as contributory factors. Observational studies tended to identify equipment and supplies marginally more frequently than other methods.
Domain | Incident reporting (n = 30) | Interviews and focus groups (n = 10) | Observational (n = 14) | Other (n = 29) | ||||
---|---|---|---|---|---|---|---|---|
Count | % | Count | % | Count | % | Count | % | |
Active failures | 149 | 22.6 | 22 | 9.8 | 24 | 12.6 | 110 | 18.2 |
Communication systems | 38 | 5.8 | 12 | 5.4 | 16 | 8.4 | 66 | 10.9 |
Design of equipment and supplies | 28 | 4.3 | 9 | 4.0 | 0.0 | 14 | 2.3 | |
Equipment and supplies | 55 | 8.4 | 4 | 1.8 | 20 | 10.5 | 31 | 5.1 |
External policy context | 4 | 0.6 | 0.0 | 1 | 0.5 | 4 | 0.7 | |
Individual factors | 68 | 10.3 | 54 | 24.1 | 12 | 6.3 | 50 | 8.3 |
Lines of responsibility | 2 | 0.3 | 4 | 1.8 | 0.0 | 9 | 1.5 | |
Management of staff and staffing levels | 37 | 5.6 | 15 | 6.7 | 7 | 3.7 | 38 | 6.3 |
Patient factors | 39 | 5.9 | 6 | 2.7 | 6 | 3.2 | 26 | 4.3 |
Physical environment | 29 | 4.4 | 7 | 3.1 | 6 | 3.2 | 19 | 3.1 |
Policy and procedures | 16 | 2.4 | 5 | 2.2 | 4 | 2.1 | 26 | 4.3 |
Safety culture | 9 | 1.4 | 5 | 2.2 | 0.0 | 12 | 2.0 | |
Scheduling and bed management | 2 | 0.3 | 1 | 0.4 | 3 | 1.6 | 12 | 2.0 |
Staff workload | 10 | 1.5 | 17 | 7.6 | 4 | 2.1 | 14 | 2.3 |
Supervision and leadership | 10 | 1.5 | 8 | 3.6 | 2 | 1.1 | 20 | 3.3 |
Support from central functions | 23 | 3.5 | 0.0 | 9 | 4.7 | 22 | 3.6 | |
Task characteristics | 6 | 0.9 | 6 | 2.7 | 2 | 1.1 | 6 | 1.0 |
Team factors | 13 | 2.0 | 9 | 4.0 | 11 | 5.8 | 20 | 3.3 |
Training and education | 17 | 2.6 | 2 | 0.9 | 5 | 2.6 | 15 | 2.5 |
Outcomea | 9 | 1.4 | 1 | 0.4 | 25 | 13.2 | 22 | 3.6 |
Can’t code | 94 | 14.3 | 37 | 16.5 | 33 | 17.4 | 68 | 11.3 |
Grand total | 658 | 100.0 | 224 | 100.0 | 190 | 100.0 | 604 | 100.0 |
We also investigated variation in the identification of contributory factors as a function of whether or not a human factors expert was involved in the identification. Caution must be exercised because of the low number of studies explicitly utilising a human factors expert in the elicitation of contributory factors. However, there was some evidence that, compared with others, human factors experts tended to identify active failures less frequently (11% vs. 19%) and more latent contributory factors such as team factors (10% vs. 3%) and the physical environment (7% vs. 3%) more frequently. However, despite some evidence that human factors experts were more likely to identify distal than proximal causes, they were more likely to identify individual factors (e.g. fatigue, inexperience; 16%) than others (11%). A similar pattern of findings was apparent when comparing studies that employed a theoretical framework in developing their contributory factors coding scheme with those that did not.
Figure 3 is a diagrammatic summary of the findings of the review, which represents the speculated hierarchical nature of the identified domains. This figure depicts the domains as a series of concentric circles, with active failures at the centre and external policy context as the outer circle. This diagram helps to illustrate the extent to which a domain is proximal to the active failure.
Discussion
As early as 1998, Vincent et al. 169 produced a framework for analysing risk and safety in clinical medicine. In this influential article, Vincent et al. refer to Reason’s20 model of organisational safety, making a clear distinction between the active failures (slips, lapses, mistakes and violations) of health-care professionals and the latent organisational failures that provide the conditions in which active failures occur. The past 20 years have seen a proliferation of research using this framework or similar models to understand the causes of PSIs. However, to date, there has been no systematic review of this research and, therefore, existing frameworks for risk management have a theoretical, but not an empirical, basis.
In this review we identified 95 studies (83 independent data sets) that reported on primary research work with the aim of identifying the factors that contributed to PSIs. A systematic review and analysis of these studies suggests that, despite the availability of frameworks and models that encourage the elicitation of latent and active failures (e.g. the Australian Incident Monitoring Study system57 asks people to record any physical environment, equipment or work practice or policy issues that contributed to the incident), the overwhelming majority of contributory factors that were identified (irrespective of hospital setting or methodology) were active failures or individual factors. This tendency to focus on the proximal causes of the incident – although slightly less prevalent in our data set when the reviewer was a human factors expert – was ubiquitous, with approximately 25% of the contributory factors identified as falling into one of these two domains (active failure or individual factor). In fact, despite claiming to investigate the causes of incidents, some studies did not go much beyond the immediate behaviour, performance or skills of the individual who was ‘responsible’ for the incident. 120,143,144 Moreover, even when frameworks include systems factors, it is revealing that more attention may be given to the human factors than the systems factors. For example, within the Australian Incident Monitoring Study, 33 codes refer to human factors, whereas 21 refer to systems factors. Within the Eindhoven classification61 there are nine codes that refer to human failure but only four referring to technical failure and five referring to organisational failure. This emphasis on human failure rather than latent failure is much less profound in the London Protocol and WHO classifications. However, our review found that, to date, these frameworks have been used less frequently in published empirical work that identifies contributory factors.
Our review has informed the construction of a framework of contributory factors that includes 20 key domains and suggests the extent to which these are proximal or distal (active or latent failures). This pictorial representation is based on previously described accident causation models53,67 together with the ratings of our expert group. Thus, it should be noted that, although the evidence for the domains reflected within the framework is strong, future research is needed to clarify the exact positioning of the domains within the outer rings and the weighting of each domain (perhaps by varying the size of each segment). Although this framework has a greater number of domains than others (e.g. the London Protocol includes just seven domains and the WHO classification specifies five main contributing factors) and therefore might be criticised for being more complex, it captures the full range of contributory factors (across different hospital settings) and gives a greater weighting to systems rather than human failures. Moreover, some interesting findings have arisen from the work reported here, not least the slight differences in the identification of contributory factors for different settings. The fact that this framework differentiates between surgery, where teamwork was frequently identified, and anaesthesia, where equipment and supply issues were more pronounced, highlights its potential to be generalisable across specialties and error types and yet sufficiently detailed to pick up subtle differences between areas of the hospital to allow the targeting of appropriate interventions. Indeed, this framework has the potential to be used in a number of ways to support improvements to patient safety in practice. It could be used to improve the root cause analysis of serious PSIs. For example, it could be used to analyse PSIs to identify the prevalence of contributory factors and to provide feedback on the quality of existing incident analysis processes. The framework could also be used as a basis for the systematic collection of data about the factors contributing to PSIs through the redesign of local and national reporting systems. The quality of the data elicited through existing reporting systems is often poor55,56,58 because health-care professionals who are responsible for reporting errors focus predominantly on the individual and situational factors that are proximal to the error. Without guidance on other factors we may learn little about the organisational interventions that might better support safer practice. The framework may also help clinicians or managers to identify proactively poor safety performance at an organisational level and therefore guide risk management strategies. For example, the framework could be used as the basis for developing a measurement tool for patients to report on the local and organisational factors that impact on their care.
The findings reported here are important but should be treated with caution for two reasons. First, although we identified that active failures, individual factors such as knowledge and experience of the health-care professionals, communication and equipment and supplies were the contributory factors most frequently recorded in the literature, this should not be interpreted as reflecting the reality of accident causation. Almost half of the studies included in this review (n = 48) did not refer to the use of a theoretical framework to support the identification of contributory factors and only eight made explicit links between theory and the identification of contributory factors. One-third of the studies were based on analysing the data from incident reports, data that are often reported to be of poor quality. 170 For example, some studies simply referred to active failures (e.g. doctor prescribed the wrong drug dose) to explain another active failure or incident, rather than make any attempt to understand the reasons for this behaviour. Typically, incident reporting frameworks rely on those doing the reporting to select probable causes from a given list. This is problematic because those completing the reports may have very little understanding of the factors, active and latent, that contribute to incidents. In addition, when a tick box of contributory factors is available, this might not represent a complete list of possible contributory factors. Second, most staff are not trained in the identification of systems failures and may neglect to look further than the proximal cause of an error (e.g. a slip or lapse) when attributing causes to an incident. Together, the lack of a theoretical framework, the paucity of data available in many of the articles about the underlying causes of the incidents and the lack of detail about contributory factors also meant that it was impossible to code approximately 15% of the contributory factors. It is also pertinent that only two of the studies reported here involved patients in defining the nature of a PSI or in identifying causes. 11,124 Therefore, it must be acknowledged that this framework does not encompass a patient perspective on the causes of safety incidents. This is certainly a worthy future endeavour.
Although the findings about the prevalence of the contributory factors identified within the studies should be treated with caution, the variety of methods and the reach of the research across a range of hospital specialties provide strong grounds for arguing that this work captures the full range of contributory factors. Moreover, the rigorous process employed for coding the contributory factors and developing the classification of these factors means that the resulting framework has a strong evidence base. This is supported by the extent to which our own framework coincides with existing frameworks in this field. 61–64 The framework (see Figure 3) explicitly presents contributory factors at a number of different levels (active failures, situational factors, local working conditions and organisational and external latent factors), which is a welcome addition to the literature. The majority of studies in this review focused on understanding the contributory factors through interviews with frontline staff and their observations and analyses of accidents. These staff may not have a sufficient grasp of the higher-level organisational factors or external policy context that impact on their performance and behaviour. Thus, future research should attempt to further verify the factors in the two outer circles of the framework. Finally, the clear definitions presented within the framework should aid its practical application and the reliable attribution of contributory factors. In fact, without these definitions the coding task here was made much more difficult and distinguishing between some domains was problematic (e.g. communication and teamwork). Initial pilot work using the framework to categorise contributory factors from 44 serious untoward incident reports within three UK hospital sites has been encouraging, with agreement of 80% between two independent assessors. This compares favourably with the published inter-rater reliability of the Eindhoven classification (68%, κ = 0.63). 171
Conclusions and policy implications
The poor quality of the current evidence base and the lack of a consistently adopted framework limits the accurate reporting of factors that contribute to error and hence the opportunity to learn from error. We conducted a systematic review of contributory factors identified from a wide range of settings using multiple data collection methods. We then developed an empirically based framework of contributory factors. This framework has the potential to be applied across hospital settings to improve the identification and prevention of factors that cause harm to patients.
Chapter summary
This chapter has outlined a review of the extant evidence of contributory factors to PSI and the development of a framework for their conceptualisation within a systems approach to safety. Alongside the obvious implications for improving the identification of errors for health-care professionals, it provides a framework for the further development of a tool to allow patients to systematically feed back about the safety of their care, effectively engaging in the assessment of risk within hospital settings. The next chapter describes the development and validation of such a tool: the Patient Measure of Safety.
Publication statement
This chapter is based on a previously published paper. 172 We reproduce it here with permission. This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.
Chapter 3 Assessing risk: developing and validating the Patient Measure of Safety
Abstract
Background: Patients are often able to provide feedback on the quality and safety of their care when in hospital and can identify safety issues that staff may not have noticed. Existing patient experience measures ask some questions about safety but no tool exists that captures patients’ views on their safety to allow ward-based improvements to be made. This chapter reports on two studies concerning the PMOS (Patient Measure of Safety) tool: (1) the development of the PMOS tool and (2) the validation of this measure.
Methods: The development study used qualitative methods in two stages. First, it was ascertained which contributory factors patients could identify as being relevant to patient safety. From these data, PMOS items were developed and tested with health professionals and patients to assess face validity. Next, the validation study used a large survey with patients to assess their perceptions of factors contributing to safety incidents and another survey with staff in the same hospital to assess convergent validity.
Results: The results of the development study showed that patients are able to identify a broad range of contributory factors, with communication being the factor most recognised. It also showed that patients have a willingness to complete the PMOS tool, with few barriers identified. The results of the validation study showed the tool to be reliable and valid.
Conclusions: The PMOS tool offers an important mechanism for hospitals to engage with their patients about safety and to gather data on how wards are performing in relation to the safety and quality of care they are delivering.
Chapter rationale
The previous chapter outlined a detailed examination of the empirical evidence for the factors that contribute to PSIs, with the conceptual framework providing a basis for the development of tools to better assess risk in hospital settings. This chapter deals with the next stage of our work, namely the development of a theory- and evidence-based tool to allow the systematic capture of feedback from patients about the safety of care. It describes the development and validation of the PMOS (Patient Measure of Safety) tool, the first tool of its kind to elicit the patient perspective on the safety of care using a contributory factors framework. First, we describe a qualitative study in which patients were asked to identify factors that might contribute to PSIs and how we used what patients told us in the formulation of the PMOS. This is called the development study. Second, we describe a study which shows that the PMOS tool has acceptable reliability and validity. This is called the validation study.
Background
The elicitation of feedback from patients about satisfaction with their care or their experiences of care is relatively well established. However, patient satisfaction surveys25,173 are often criticised for producing mostly positive ratings from patients174 that are not comparable with the lower levels of satisfaction revealed through interviews with the same patients. 175,176
There is growing evidence to suggest that patients are willing and able to provide feedback on the quality and safety of their care. 15,36,177–180 This is particularly valuable as patients can identify safety issues that staff may not notice or be willing or able to report. 34 Patients are also uniquely placed to observe the processes of their care such as the scheduling of procedures, their treatment, such as inappropriate drug administration, and the physical environment, such as temperature and cleanliness. 181 Recent reports focusing on patient safety in the English NHS have highlighted the importance of listening to and acting on patient concerns about patient safety issues. 182 There is growing evidence that patients can be an important source of knowledge in reducing avoidable harm and improving health care. 34,36,178
Measures of patient experience have been developed to capture data on specific aspects of health-care processes and events. 24,183,184 Although patient experience measures such as the widely used Picker Patient Experience Questionnaire183 ask some questions that are relevant to patient safety (e.g. about medication side effects and communication with patients), to date, no tool has been developed that asks patients to provide feedback on the safety of their care, particularly as a way of capturing information that can be used as a basis for improving safety at ward/unit level. This, along with the international impetus for the mobilisation and empowerment of patients with regard to their safety185 and increasing emphasis on the importance of patient feedback as key indicators of the patient experience makes the time ripe for the development of a systematic way of collecting safety information from patients. 186–188 Furthermore, within the English NHS there is a clear call for hospitals to engage patients and collect their feedback on services in real time and consistently across different organisations. 182 Thus, a tool to allow patients to directly report on their safety is timely and important in the drive for hospitals to proactively manage safety.
A systematic review,172 reported in Chapter 2, generated a comprehensive taxonomy of the factors contributing to PSIs, the Yorkshire Contributory Factors Framework (YCFF) (see Figure 3). These include factors such as the physical environment, communication, leadership and teamwork. Using the YCFF as a starting point, the development study aimed to (1) explore the extent to which patients are able to provide feedback about the contributory factors represented in this framework, (2) develop indicators of each of these contributory factors in the form of questionnaire items and (3) test the acceptability of the PMOS tool to staff and patients. The validation study aimed to test the reliability and validity of the PMOS tool in a hospital setting. The objectives were to explore the factor structure and internal reliability of the scale (Cronbach’s alpha, test–retest reliability), the extent to which the scale discriminates among wards (discriminant validity) and the extent to which it converges with staff measures of patient safety (convergent validity).
Methods
Development
The development study can be divided into two distinct stages. Stage 1 involved a series of qualitative interviews with patients to explore which contributory factors they were able to identify, using the YCFF as a basis. The data from these interviews were used to inform the development of items for the PMOS tool. Stage 2 involved testing the PMOS tool with health professionals and patients using a ‘think-aloud’189 approach. A multidisciplinary panel of experts including policy makers, health professionals, academic researchers and patients informed the design of the study and the patient panel aided the development of the questionnaire.
Patients in stage 1 (interviews) and patients and health professionals in stage 2 (think aloud) were recruited from six units (maternity, renal, physiotherapy outpatients, vascular surgery, ear, nose and throat and cancer services) in a NHS trust in the north of England. These units were selected using purposive sampling to ensure that the views of a broad range of patients were elicited encompassing those with regular interaction with the health service (renal patients) and those with relatively short one-off stays (maternity), young and old patients, men and women and patients of different ethnic backgrounds. The data were collected by two researchers, one of whom was able to speak Mirpuri (a dialect spoken by three-quarters of the Pakistani population in Bradford) [see www.30-days.net/muslims/muslims-in/europe/mirpuris-britain/ (accessed September 2012)]. Ethical approval for this study was gained from a local research ethics committee (reference number 09-H1302–115).
Stage 1: qualitative interviews
Qualitative interviews were used as the basis for identifying which contributory factor domains patients could identify and for developing PMOS questionnaire items. The project steering group, consisting of a multidisciplinary panel of experts, was consulted in the development of interview schedules. This panel recommended that, although the YCFF172 (see Figure 3) could be used to define the interview questions, some of the interviews should take a more unstructured approach to ensure that the views of the patients were fully represented and not constrained by an a priori framework. Thus, two approaches to interviewing were used: unstructured and structured. In the first case (unstructured), interviews (n = 18) were based on a narrative approach190 in which participants were asked to describe their most recent/current hospital experience. These interviews were preceded by three pilot interviews. Participants were asked to describe their hospital experience with an emphasis on patient safety. However, during the pilot interviews it became clear that using the term ‘patient safety’ (which was not familiar to many patients) appeared to discourage participants from engaging in the interview. Thus, it was decided to omit the phrase ‘patient safety’ from any subsequent interviews and to allow the interviewer to explore any experiences that related to patient safety (e.g. delays in waiting for medication, insufficient information given to patients regarding their condition/treatment/procedure, delays in treatment/procedures/operations, poor communication), should participants describe such experiences. In the case of the structured interviews the patient panel for the project was consulted and asked to select which of the 18 contributory factors (contained within the YCFF172) they felt that patients would definitely not be able to identify/comment on. Based on this assessment, patients were not asked to comment on safety culture, policy and procedures, external policy context, task characteristics and design of equipment and supplies. Active failures were also excluded from the structured interviews as the PMOS tool was designed to assess those factors contributing to errors, but not the errors themselves. The structured interviews (n = 15) asked patients which of the remaining 13 factors they felt that they were able to comment on or were in a position to notice and/or make judgements about. They were encouraged to provide examples based on their own experiences.
Patients (except those deemed too unwell by staff or those having undergone a general anaesthetic in the preceding 24 hours) were approached in the ward by the researcher who explained the study and gave them an information sheet. Willing participants were then consented. When possible, interviews took place in the unit, often at the bedside. Some interviews were conducted elsewhere, for example in the hospital canteen or in the waiting room. Interviews ranged from 15 minutes for the structured interviews to 2 hours for the unstructured interviews. The interviews continued until no new themes emerged and theoretical saturation was achieved. 191 All interviews were recorded using a digital recorder and fully transcribed.
Stage 1: analysis
Interview transcripts were imported into NVivo 8 (QSR International, Warrington, UK) and then coded using content analysis. 192 The YCFF was used as a coding framework with a particular focus being on the 13 domains deemed relevant to patients. The frequency with which each of the domains was coded was recorded. Any text that could not be coded using the YCFF was coded separately and new themes created. Initially, three members of the research team reviewed three of the transcripts each to ensure that there was consensus in the interpretation of the factors within the YCFF. Following this, the remainder of the transcripts (n = 30) were divided equally between two of the researchers, who used the above process to code the transcripts.
Stage 2: think aloud
The ‘think aloud’ process took place after the PMOS tool was produced. The purpose of this phase of the research was to test the face validity of the PMOS tool. Twenty-four ‘think aloud’ interviews (12 patients and 12 health professionals) were conducted in the six units described above. As staff and patients are potential users of the survey, the views of both groups were important. Participants were asked to talk aloud about their thoughts and feelings as they read and decided how to respond to each question in the draft PMOS tool.
Participants were also asked to comment on (1) perceived barriers to completing the PMOS tool, (2) the timing of completion during the care pathway and (3) the questionnaire format. Minor revisions to the PMOS tool were made following the ‘think aloud’ procedure.
Recruitment of patients took place within the six units in the same way as for stage 1. The health professionals were identified using existing contacts within each of the areas. These ‘think-aloud’ sessions were arranged by e-mail or telephone and took place at a location chosen by the participants. Interviews were between 10 and 30 minutes long. All interviews were digitally recorded and transcribed in full.
Stage 2: analysis
The ‘think-aloud’ transcripts were imported into NVivo 8. Two of the researchers listened to the recordings and read through the transcripts to identify and code comments made when completing the draft PMOS tool. In addition, both researchers collated the responses to the short questionnaire that was completed following each ‘think-aloud’ session. These, together with the transcripts, informed any changes that were made to the PMOS tool. The final PMOS tool was then tested for readability using the Flesch Reading Ease and Flesch–Kincaid Grade level [see https://readability-score.com/ (accessed 19 July 2016)].
Validation
The validation study used two separate cross-sectional surveys (one with patients and one with staff) within a large acute teaching hospital in the north of England. Data were collected between 1 September 2011 and 30 November 2011 (10 wards) and between 3 and 5 April 2012 (one ward).
In line with recommendations, a minimum sample size of 250 was considered acceptable for the patient survey (in relation to planned factor analysis). 193 There was no a priori required sample size for the staff survey (against which the patient survey was validated) although we aimed to sample a minimum of 50% of staff on participating wards.
Participants in the patient survey
In total, 402 patients or their parents/carers across 11 wards (including medical, surgical, maternity and paediatric wards) were approached to take part in the study. Of these, 344 consented and 297 valid PMOS questionnaires were collected (47 patients consented to take part in the study but did not return a questionnaire). Reasons for refusal were not recorded. The mean age of patients across the 10 adult wards was 54 years [standard deviation (SD) 18.13 years]. Within the paediatric ward, the mean age of children admitted was 6 years (SD 4.45 years); the age of parents completing the questionnaire was not recorded.
Table 4 contains a summary of demographic characteristics by ward. The majority of the sample was classed as white British (78%, n = 254), with 14% (n = 49) of Asian origin (Pakistani n = 37, Indian n = 7, Bangladeshi n = 2, other Asian or Asian British n = 3).
Ward/unit | Consented, n (response rate, %) | Age (years), mean (SD) | Sex (consented) | PMOS complete, n | Completed staff AHRQ questionnaire (staff approached) |
---|---|---|---|---|---|
Medicine_1: coronary care | 36 (72.0) | 57.2 (15.5) | 26 male, 25 female, 1 missing | 32 | 46 (50) |
Medicine_2: renal, rheumatology, gastroenterology | 44 (78.6) | 56.7 (15.5) | 29 male, 15 female | 33 | 18 (65) |
Medicine_3: respiratory, endocrinology | 37 (92.5) | 59.3 (14.1) | 17 male, 20 female | 33 | 7 (40) |
Surgery_1: ear, nose, throat and eyes | 54 (91.5) | 52.2 (16.9) | 29 male, 25 female | 51 | 39 (41) |
Surgery_2: general | 37 (84.1) | 51.9 (21.1) | 37 female | 29 | 7 (26) |
Surgery_3: vascular | 37 (86.1) | 61.2 (16.8) | 25 male, 11 female, 1 missing | 29 | 21 (37) |
Surgery_4: general | 27 (79.4) | 51.4 (16.6) | 27 male | 26 | 16 (22) |
Maternity_1: postnatal | 17 (61.9) | 28.6 (6.8) | 1 male, 16 female | 12 | 7 (26) |
Maternity_2: postnatal | 12 (100) | 27.3 (5.2) | 2 male, 10 female | 10 | 15 (31) |
Paediatric_1: medicine | 28 (93.3) | (Child) 5.5 (4.5) | 19 male, 7 female, 2 missing | 27 | 14 (37) |
Admissions unit | 15 (100) | – | 8 male, 7 female | 15 | – |
In total, 73% (n = 238) had been admitted to the hospital at least once previously in the past 5 years and 55% of patients were receiving ongoing treatment as an outpatient at the time of the stay. Those who did not complete the PMOS questionnaire after consenting (n = 47) were more likely to be female (χ2(1) = 6.84, p = 0.009) but did not differ on age, ethnicity, previous hospitalisation or ongoing treatment. To assess test–retest reliability, 55 patients were sent a second PMOS questionnaire at home approximately 1 week after they completed the first. Of these, 25 (45%) were returned.
Procedure for data collection
Staff on wards identified patients to be approached by the research team. Exclusion criteria included being too ill to talk to the research team (although relatives/carers could take part on a patient’s behalf) or being unable to consent. A member of the research team then approached the patients and/or their relatives/carers and asked if they would be willing to complete a brief questionnaire about safety. Once consented, the patients’ demographic information was recorded, along with a unique patient identification code. Patients were then asked to complete the PMOS tool either on their own or with a member of the research team. Those who wished to complete the questionnaire on their own were provided with a sealable envelope. Locked post boxes were available on each ward for patients to post their completed questionnaires. A subsample of 55 respondents who were due for discharge either on the day they were recruited or imminently afterwards were sent a second questionnaire to their home address approximately 1 week after they completed the first to assess test–retest reliability. They were instructed to complete the questionnaire as soon as possible and return it to the researchers in the postage-free envelope enclosed.
Staff survey
To assess convergent validity, all staff working on 10 wards (118 medical staff, including 54 staff working in obstetrics and gynaecology or paediatrics who had no clear designated ward, and 321 nursing, midwifery or auxiliary staff) were asked to complete the outcomes measures of the Agency for Healthcare Research and Quality (AHRQ) Hospital Survey on Patient Safety Culture. 194 In November 2011 staff on the 10 wards were sent personalised envelopes containing the AHRQ survey, a pen and an internally addressed reply envelope. Questionnaires were either placed in pigeon holes or distributed by the ward manager. Return of questionnaires was incentivised such that the ward returning the most questionnaires (as a percentage of total staff) would receive a £200 prize. Staff were unaware of patient scores on the PMOS scales when completing the safety culture questionnaire.
In total, 212 (48.3%) responses were returned. Of these, 22 were excluded as they indicated that staff worked across more than one of the wards included in the study (n = 19) or in a ward not included in the study (n = 3). Response rates for the wards varied from 17.5% to 95.1%.
Measures
Patients completed the PMOS tool. The questionnaire assesses patients’ perception of the factors contributing to patient safety. These include latent, local and situational factors across a number of hypothesised domains, with examples given here:
-
communication – ‘I got answers to all the questions I had regarding my care’
-
individual factors – ‘I felt that the attitude of staff towards me was good’
-
physical environment – ‘There was not enough space on the ward’ (reverse coded)
-
scheduling and bed management – ‘My treatment/procedure/operation did not always happen on time’ (reverse coded)
-
training and education – ‘On at least one occasion a member of staff was not able to carry out a task that they should have been able to do’ (reverse coded)
-
lines of responsibility – ‘I have always known which person/team was responsible for my treatment’
-
management of staff and staffing levels/workload – ‘Too few staff meant that things didn’t get done on time, e.g. attending to call bells, removing bodily fluids, toileting patients, feeding patients’ (reverse coded)
-
equipment and supplies – ‘Equipment needed for my care was always working properly’
-
supervision and leadership – ‘It was clear who was in charge of staff‘
-
team factors – ‘A doctor changed my plan of care and other staff didn’t know about it’ (reverse coded)
-
support from central functions – ‘My test results were always available when required, e.g. scans, blood tests, X-rays’.
All 42 items were responded to on a 5-point Likert scale from 1 (strongly disagree) to 5 (strongly agree). Respondents could also select a ‘not applicable’ option. Items relating to active failures (errors and violations) were not included in this tool as the focus was on the identification of the factors contributing to these failures.
Staff on participating wards were asked to complete the outcome measures of the AHRQ staff safety culture questionnaire,194 whose validation has been reported elsewhere. 195,196 In line with guidance, four outcome scales were calculated:
-
overall perceptions of safety (mean of four items – ‘our procedures and systems are good at preventing errors from happening’; 1 = strongly disagree to 5 = strongly agree; higher scores = safer)
-
frequency of event reporting (mean of three items – ‘when a mistake is made, but is caught and corrected before affecting the patient, how often is this reported?’; 1 = never to 5 = always)
-
patient safety grade (please give your work area/unit in this hospital an overall grade on patient safety, coded as excellent = 1, very good = 2, acceptable = 3, poor = 4 and failing = 5)
-
number of events reported in the past 12 months (0 = no event reports, 1 = one to two event reports; 3 = three to five event reports; 4 = six to 10 event reports; 5 = 11–20 event reports; 6 = ≥ 21 event reports).
Analysis
Principal components analyses (PCAs) were performed on correlation matrices, with pairwise deletion using PASW Statistics (for Windows, v17; SPSS Inc., Chicago, IL, USA), using orthogonal varimax rotation to explore the internal structure of the questionnaire. PCA was chosen because of the exploratory nature of the PMOS questionnaire. Kaiser’s criterion was used for choosing the number of factors to retain as it has been shown to be relatively accurate when samples sizes are > 250 and when average communalities are > 0.60.
Loadings of > 0.40 were retained as significant in line with Stevens’197 recommendations. Internal reliability of the retained factors was inspected via Cronbach’s alpha, with ≥ 0.8 interpreted as good, ≥ 0.7 as acceptable and ≥ 0.6 as questionable. 198 For factors containing fewer than seven items, average inter-item correlations were calculated based on the recommendation of Briggs and Cheek. 199 Pearson’s correlations were used to assess test–retest reliability of the PMOS questionnaire across participants and to explore convergent validity of the PMOS with the AHRQ Hospital Survey on Patient Safety Culture at ward level. In line with Cohen’s guidelines,200 correlations of 0.1 were interpreted as a small effect, 0.3 as medium and 0.5 as large. A multivariate analysis of variance (MANOVA) was used to assess discriminant validity with ward as the independent factor and the PMOS scales as dependent factors. Missing data were excluded list-wise.
Results
Development study
All participants approached for interview at both stages agreed to take part. This high uptake was because patients who were too unwell or who did not have the capacity to complete the interview were filtered out by staff. In the stage 1 interviews, participants (14 male patients and 19 female patients) ranged in age from 18 to 83 years.
Stage 1: qualitative interviews
Can patients identify contributory factors within the Yorkshire Contributory Factors Framework?
Table 5 shows the number of times patients highlighted a contributory factor domain and provides examples, in the form of interview excerpts, of the way in which patients talked about these factors. As expected, none of the five domains excluded by the expert panel was identified during the qualitative interviews. All remaining 14 domains were identified by patients. The domains most frequently identified were communication (identified 80 times) and individual factors (in relation to staff, e.g. stress, and patients, e.g. attitudes of carers, identified 62 times). Domains such as team factors and support from central functions were identified less frequently. Although the domain active failures was excluded from the structured interviews, participants did identify active failures during the unstructured interviews. There appeared to be no major differences in the nature of information collected using the structured and unstructured approach to the interviews. A further theme not captured within the YCFF was identified: dignity and respect. Little is known about the relationship between dignity and respect and patient safety outcomes, but it is an important factor in patients’ experience of their care, being associated with feeling comfortable, in control and valued. 201 Although not identified as a contributory factor within the literature, our patient panel members felt that patients who were not treated with dignity and respect may be reluctant to communicate important information about their well-being (including symptoms) to health professionals, which might compromise their safety.
Domain | Definition | Times domain identified | Illustrative quotes from interviews |
---|---|---|---|
Communication | Effectiveness of the exchange and sharing of information between staff, patients, groups, departments and services | 80 | No. I don’t think they really understand anyway. But they don’t talk. We were saying this before, I have got five or six different people, and they don’t communicate between them, and a lot of the time that would helpWhite British female, 30 years, physiotherapy |
Individual factors/patient factors | Individual factors refer to characteristics of the person delivering care that may contribute in some way to active failures. Examples of such factors include inexperience, stress, personality and attitude | 62 | The nurses, they can be quite arrogant and they got no manners. They don’t know how to talk to patients, reallyPakistani female, 22 years, ear, nose and throat |
Physical environment | Features of the physical environment that help or hinder safe practice | 25 | If your blood pressure drops they need to be able to lay you flat and there isn’t enough room to move your chair downWhite British male, 64 years, renal |
Scheduling and bed management | Adequate scheduling to manage patient throughput minimising delays and excessive workload | 25 | All I can say to start off with is I’ve had five operations on my leg and I’ve had 13 cancellations. Now we’re being not always getting into the hospital because some of the cancellations were, I would have to be . . . at the weekend and then ring up Monday morning expecting to come in and then said no there’s no bed you can’t come in ring up next Monday morning but don’t stop taking your medicationWhite British male, 60 years, ear, nose and throat |
Management of staff and staffing levels/staff workload | The appropriate management and allocation of staff to ensure adequate skill mix and staffing levels for the volume of work; level of activity and pressures on time during a shift | 21 | Cause they’re dealing with other patients. There’s like six rooms with four in them, dealing with 28 patients . . . They’re understaffedWhite British male, 47 years, renal |
Dignity and respect | Associated with patients feeling comfortable, in control and valued | 13 | This one lady she has had a line on her chest over here and a few times they have not put the curtain around and you know they just dealt with her. I have looked over and closed my eyes and I have thought to myself that this is just not donePakistani male, 39 years, renal |
Training and education | Access to correct, timely and appropriate training: specific (e.g. task related) and general (e.g. organisation related) | 13 | When you get the trainees, they sometimes don’t seem to know what they are doing, it’s a worry reallyWhite British male, 45 years, renal |
Lines of responsibility | Existence of clear lines of responsibility clarifying accountability of staff members | 12 | They’ll just shake your hand and say ‘hello I’m doctor so and so’ or whatever, but you don’t know who they’re working for or why they’re there, they just keep coming in to see you, you think why has he come to see you, and who’s that, they don’t exactly say why they’re there, that’s it really!White British female, 54 years, cancer services |
Equipment and supplies | Availability and functioning of equipment and supplies | 10 | One of the things – just on today’s experience, one of the things is about the hand gel, when I came in today I noticed that the first obvious hand gel on that side was emptyWhite British female, 34 years, ear, nose and throat |
Supervision and leadership | The availability and quality of direct and local supervision and leadership | 10 | They don’t know the machines – they have to go get some help, but what I’m saying is I think it’s unfair on them, and they start getting a bit stressed do you know what I mean? I haven’t seen it for a while, it’s when we had a batch of young nurses in which they are all pretty much fine now, but at that time I just thought it was wrong and unfair on them to be left without an experienced member of staff that did know the machinesWhite British female, age unknown, renal |
Team factors | Any factor related to the working of different professionals within a group that they may be able to change to improve patient safety | 8 | (Partner) Wasn’t there before when you had the two consultants, one was asking for . . . and the other one was ignoring it, things like that, [name given]’s had. Two specialities clashingWhite British female, 28 years, physiotherapy |
Support from central functions | Availability and adequacy of central services to support the functioning of wards/units | 8 | Yeah the practical side was good, the only downside I would say that’s let this particular ward down and the patients is the pharmacy. When you come for appointments you can be 2 hours waiting for the drugs to come up from the pharmacy and that’s annoying for the staff, for the patients because your’re waiting around for 2 hours before you get your treatment, which is probably a couple of hours anyway! And it’s quite annoying! So you just sit in the waiting areaWhite British female, 54 years, cancer services |
All of these domains, with the exception of active failures (deemed to be an outcome rather than a contributory factor), were used as a basis for the PMOS.
Developing the items for inclusion in the draft Patient Measure of Safety tool
The interview transcripts (structured and unstructured) were used to develop the individual items for each domain. For some domains, particularly those that patients referred to less in the interviews, only a small number of questionnaire items were necessary to adequately represent the codes (e.g. training was represented by two items). For other domains, such as communication, which was a broad category, more items were necessary to reflect the assigned codes. Forty-three questionnaire items were developed in total. A mixture of positive and negative items was included to avoid acquiescence response set bias (the tendency to give the same response to questions when the direction of wording is the same202) and items were designed to be responded to using a Likert scale. Two pairs of domains, ‘management of staff and staffing levels’ and ‘staff workload’ and ‘patient factors’ and ‘individual factors’ were merged based on data from the interviews which suggested that patients did not distinguish between these domains. One item was also included in the questionnaire under the heading of ‘dignity and respect’ to capture this aspect of the safety of care.
Consulting with the research team and patient panel
The items in the draft PMOS tool were developed and revised based on discussions with the research team (consisting of psychologists, social scientists and clinicians) and the patient panel for the project to maximise content validity. 203 A small number of changes was suggested by both groups. This resulted in a draft version of the PMOS tool to be used in stage 2, the ‘think-aloud’ process.
Stage 2: think aloud
All those involved in the ‘think-aloud’ process (n = 24) felt that, on the whole, patients would be interested and willing to complete the PMOS tool. Participants identified some potential barriers to completion of the PMOS tool, such as eyesight, language and age (i.e. elderly patients finding it a challenge). During the ‘think aloud’, the majority (n = 20) of participants were able to complete the questionnaire within 15 minutes and were able to understand and respond to the items. The same participants also stated that they were satisfied with the length of the PMOS tool. Only a small number expressed concerns with understanding or found it difficult to respond to questions. There were three key areas of concern that arose during the ‘think-aloud’ process. First, negative statements appeared to present a problem for a small number of participants (n = 5). Second, some participants (n = 2) were unfamiliar with the terminology used within the questionnaire (e.g. care plan). Finally, some patients found it difficult to answer questions that they had no experience or knowledge of, but were often able to use the ‘not applicable’ option if this was the case. These issues were discussed and addressed in consultation with the patient panel and, when necessary, revisions to items were made.
Finalising the Patient Measure of Safety
Following the ‘think-aloud’ process and consultation with the patient panel, some changes were made to the draft PMOS tool, including the format (more space between questions and space to comment after each) and changes to some of the wording used in the questionnaire (e.g. ‘care plan’ was changed to ‘plan of care’). Once the changes to the draft PMOS tool had been made, this resulted in the main outcome of this study, the PMOS tool (see Appendix 2). This was then tested for readability using the Flesch Reading Ease (65.7%, i.e. easily understandable by 13- to 15-year-old students) and Flesch–Kincaid Grade level (6.9, i.e. indicates that the text is expected to be understandable by an average student in the sixth grade) tests.
Validation study
Factor analysis
Using Kaiser’s criterion, nine factors were extracted explaining 58.15% of the variance. Seven of the original items were complex, loading > 0.4 on more than one factor. Cross-loading variables were assigned to scales on the basis of theoretical considerations. Two items did not load on any factor.
Constructing the Patient Measure of Safety index and factors
Using the items retained from the factor analysis, an overall PMOS positive index was constructed by summing the number of items that patients responded to by using one of the two positive response options (strongly agree or agree for a positively worded item; strongly disagree or disagree for a negatively worded item). Thus, patients could score a total of 35, with higher responses equating to better safety. The mean PMOS index score for the entire sample was 24.43 (SD 7.28), meaning that on average patients responded positively to around 24 items out of the retained 35 in the PMOS questionnaire.
A mean score was calculated for each of the PMOS factors, taking into account reverse coded items. To calculate a mean score patients were required to rate at least half the items relevant to the scale (on a five-item scale a minimum of three responses was required). Scores could range from 1 to 5 with higher scores indicating better patient safety. These mean scores are provided in Table 6. Mean scores were generally positive across domains. Patients rated wards higher on the communication and teamwork (mean 4.24), information flow (mean 4.07) and equipment (mean 4.06) factors. Factors rated as poorer included the access to resources factor (mean 3.66).
Factor | n | Min. | Max. | Mean | SD | r test–retest |
---|---|---|---|---|---|---|
PMOS positive index | 297 | 2.00 | 35.00 | 24.43 | 7.28 | 0.75*** (n = 24) |
Factor 1: communication and team work | 297 | 2.22 | 5.00 | 4.24 | 0.65 | 0.75** (n = 24) |
Factor 2: organisation and care planning | 295 | 1.00 | 5.00 | 3.92 | 0.88 | 0.52** (n = 24) |
Factor 3: access to resources | 294 | 1.00 | 5.00 | 3.66 | 0.90 | 0.69*** (n = 24) |
Factor 4: ward type and layout | 296 | 1.60 | 5.00 | 3.78 | 0.84 | 0.47* (n = 23) |
Factor 5: information flow | 292 | 1.67 | 5.00 | 4.07 | 0.77 | 0.69** (n = 24) |
Factor 6: staff roles and responsibilities | 294 | 1.00 | 5.00 | 3.70 | 0.98 | 0.81*** (n = 24) |
Factor 7: staff training | 258 | 1.00 | 5.00 | 4.01 | 1.00 | 0.53* (n = 19) |
Factor 8: equipment (design and functioning) | 271 | 1.00 | 5.00 | 4.06 | 0.82 | 0.48* (n = 20) |
Retained item 8: my treatment/procedure/operation did not always happen on time (reversed) | 273 | 1.00 | 5.00 | 3.34 | 1.40 | 0.29 (n = 22) |
Retained item 26: too few staff meant that things did not get done on time, for example attending to call bells, removing bodily fluids, toileting patients, feeding patients (reversed) | 282 | 1.00 | 5.00 | 3.35 | 1.40 | 0.07 (n = 21) |
Test–retest reliability
The second PMOS questionnaire was completed on average around 2 weeks after the first (median 12 days, IQR 9.5–16 days). The PMOS tool was found to have acceptable test–retest reliability on the positive PMOS index (r = 0.75). Test–retest reliability was acceptable for all PMOS factors (≥ 0.47, a large effect size, according to Cohen’s200 32 guidelines). The two retained items relating to delays did not exhibit acceptable test–retest reliability.
Discriminant validity
A MANOVA was performed to assess the extent to which the PMOS tool discriminated among the 11 wards. The PMOS positive index, the PMOS factors and the two retained items relating to delays were entered as dependent variables with ‘ward’ entered as a fixed factor. Data were deleted list-wise, leaving a total sample of n = 221. The MANOVA showed an overall main effect of ward [Wilks’ λ = 1.67, degrees of freedom (df) = 11, 1510, p < 0.001, partial η2 = 0.08]. Tests of between-subject effects revealed that three factors discriminated between hospital units. Specifically, significant differences were apparent in the:
-
staff roles and responsibilities factor (F = 3.03, df = 10, 210, p = 0.001, partial η2 = 0.13)
-
ward type and layout factor (F = 2.26, df = 10, 210, p = 0.016, partial η2 = 0.10)
-
equipment factor (F = 1.89, df = 10, 210, p = 0.048, partial η2 = 0.08).
Item 26, which referred to having enough staff to get things done, just missed the standard cut-off for significance (F = 1.87, df = 10, 210, p = 0.051, partial η2 = 0.082). Post hoc tests using the Bonferroni correction were used to explore significant differences between wards. Some significant differences were apparent, which was encouraging given the reduced power and relatively small sample sizes within each ward. Within the staff roles and responsibilities factor, the admissions ward appeared to be significantly worse on this scale than eight of the 10 remaining wards. For the equipment factor, maternity ward 2 scored lowest on this scale (mean 3.00) and was significantly lower than three other wards scoring near the upper end of the scale (medicine 3, surgery 1, surgery 4), although care should be taken in this interpretation because of the low sample size within this ward (n = 6, because of list-wise deletion). Finally, there was one significant difference in the access to resources factor, with medical ward 3 scoring significantly higher than surgery ward 2 (p < 0.08), and one difference on retained item 26 [too few staff meant that things did not get done on time (reversed)], with medical ward 2 scoring worse than medical ward 3 (p = 0.051).
Convergent validity
To assess the convergent validity of the PMOS tool, the mean PMOS positive index for each ward was correlated with the four patient safety outcome measures of the AHRQ across 10 wards. The PMOS positive index correlated highly with the perceptions of safety outcome scale (r = 0.79, p = 0.007, k = 10), indicating that the more positive the PMOS scores among patients, the higher staff rated the ward on perceptions of safety. Additionally, the PMOS positive index correlated highly with the patient safety grade (r = −0.81, p = 0.005, k = 10; note that the patient safety grade was coded so that lower scores indicate better patient safety), indicating that the more positive the patients’ evaluation of the ward, the better ‘grade’ staff gave their ward on patient safety. There was no relationship between the PMOS positive index and either overall staff frequency of event reporting (r = 0.25, p = not significant, k = 10) or individual staff event reporting (r = –0.46, p = not significant, k = 10).
Discussion
The development study is innovative in that it clarifies the types of contributory factors that patients can identify in hospital settings and it details the development of the first health-care questionnaire to assess safety from the perspective of the users of health care: patients. To date, patients have had minimal opportunity to input into traditional incident reporting and risk procedures for managing safety despite being able to offer valuable feedback. 11,15 In addition, unlike other patient-reported outcome measures,183 patients have played a more central role in the development of the PMOS tool, as research participants and as part of the research team. With the increasing emphasis on patient-reported outcome measures in the UK204 and USA,205 the PMOS tool provides a useful and timely addition to current tools that focus on safety culture194 and patient experience.
The fact that patients identified and described contributory factors from 13 domains of the YCFF also provides evidence of the face validity of this framework. Patients were best able to identify contributory factor domains inherent in local working conditions (e.g. communication, availability of equipment), as these are often issues that a patient can explicitly observe while on a ward. Those contributory factors at a more upstream organisational level (six of the 20 original domains) appear to represent factors that patients do not recognise as impacting directly on their safety or, even more generally, on their hospital experience (such as policy context or design of equipment). Being outside the scope of experience for most patients these factors were not measured within the PMOS tool. In addition, the study identified one further area that patients felt was strongly linked to safety in hospital settings – dignity and respect – therefore confirming that patients are able to give a more holistic view of issues relating to their safety. Future work using the PMOS tool will allow the relationship between dignity and respect and the other contributory factors to be investigated.
The validation study explored the reliability and validity of the PMOS tool, a tool that allows patients to proactively identify potential risks to safety in hospitals settings. The results of the validation study suggest that the measure shows acceptable validity and reliability, although further development is necessary to refine items and to ensure that each construct is adequately represented (seven items cross-loaded and three domains just missed the 0.7 Cronbach’s alpha criterion of acceptable reliability). Additionally, only three factors discriminated among wards, although this may have been because wards were all from one hospital site (implying a shared safety culture). Future research should explore differences between wards in different hospital settings or between different hospitals.
The PMOS tool was based on the YCFF, with Reason’s67 error causation model as an underlying theoretical tenet. However, there were some differences in structure between the YCFF and what was observed within the PMOS tool. For example, items relating to communication and team working co-occurred within the current questionnaire, despite being separate factors in the original framework. The access to resources factor contained items relating to lack of support from other staff, as well as equipment and other resources, whereas within the original framework these were separated into support from central functions and team factors. These differences may stem from the fact that the YCFF was developed from empirical studies of staff-reported data, whereas the current measure is rooted within the patient’s perspective. Unsurprisingly, staff and patients, having a different vantage point within the ward, appear to conceptualise local conditions and safety differently. However, it was reassuring that some clear similarities remained across the PMOS measure and the YCFF (e.g. roles and responsibilities, training and equipment).
As far as we are aware, this is the first tool developed to allow acute health-care services to systematically and routinely collect information from patients about the safety of their care. Given the current spotlight on patient safety,9,182,206 and the potential role of patients as an extra piece in the ‘error detection jigsaw’,15,207 it is felt that this tool shows promise as a means for hospitals to collect and act on this important new angle on safety intelligence. Our current work aims to use this new tool as part of an intervention, to support hospitals to make ward-based patient safety improvements on the basis of patient feedback about the safety of their care. However, this intelligence might also prove useful for health-care services in other ways, perhaps as part of a safety dashboard, or to help systematise the process of obtaining feedback from patients about safety as part of the wider clinical governance processes. Authors have recently described the need for patients to be more involved in complaints or serious incidents207 and this measure could perhaps allow wards in which safety events have occurred to understand from a range of patients how widespread specific safety issues or concerns are.
The PMOS tool is an example of how patients have the potential to provide valuable quality and safety improvement data at a micro level. Current information on quality and safety comes predominantly from incident reporting systems that rely on health-care professionals to report and this suffers from high levels of under-reporting. 208 Furthermore, a key strength of the PMOS tool is that it assesses eight key domains related to safety that can be used to help staff proactively identify areas of weakness within their clinical areas. An attempt was made to name factors in such a way that they would be meaningful to staff to support the use of the tool in practice. Responses by patients to the PMOS tool might be a useful ‘smoke detector’209 for safety problems on a ward, particularly as the confirmation of discriminatory validity here suggests that the tool is sensitive to differences among wards.
Limitations: development study
The development of the PMOS tool was based on the YCFF, which in turn was based on a review of studies conducted almost exclusively with health-care professionals. This means that there is a danger that the PMOS tool may not truly reflect the views of patients if patients tend to identify different contributory factors. However, taking an unstructured approach to half of the qualitative interviews meant that we were not constrained by this a priori structure. It is worth noting that, aside from the dignity and respect domain, all contributory factors that patients identified mapped onto the YCFF model.
In addition, the PMOS tool contains only those domains from the YCFF that patients were able to comment on, which means that five of the domains within the YCFF are not represented. Although a major strength of the PMOS tool is that it allows identification of contributory factors from a patient’s perspective, it must be recognised that it does not capture all possible contributory factors. It must therefore be used in conjunction with other tools, such as staff safety culture surveys and failure modes and effects analysis,210 to explore upstream organisational failures.
Limitations: validation study
Some of the dimensions within the PMOS tool failed to achieve the recommended Cronbach’s alpha guideline of 0.70. Despite this, the average inter-item correlations suggest that the dimensions are located within the optimum range of good reliability (0.20–0.40). The PMOS positive index was also found to relate to only two of the AHRQ patient safety culture survey outcomes; no significant correlations were found with the two incident reporting outcomes. However, given the unclear relationship between levels of incident reporting and actual PSIs, one might not expect to see a clear relationship between the number of incident reports and a measure of safety. The AHRQ hospital survey of patient safety was chosen as a measure of convergent validity as it has been used and validated a number of times; however, recent research has questioned its reliability and validity in a UK setting. 211 The response rates for this staff survey also varied widely, with an overall response rate of 48%, much lower than responses to the patient survey, which has implications for assessing the convergent validity of the measure. The two retained single indicator items exhibited poor test–retest reliability and, as such, caution should be exercised in their interpretation.
General limitations
Finally, the PMOS tool could be completed only by patients (or their relatives/carers) who understood English. Despite the above limitations, patients were found to be very willing to engage with the measure, demonstrated by the high consent rate into the study (86%). The PMOS tool was developed specifically for patients in hospital settings, which may limit its generalisability to other settings. Further work should explore whether this tool can be used for different patient groups in a range of settings (e.g. community-based outpatient services or general practice). The PMOS tool may not represent the views of the general population internationally because, despite being developed with a diverse ethnic population, the sample was drawn from a single hospital in the north of England. However, future work will explore the utility of the PMOS tool with a larger number of patients across a number of NHS trusts within England.
Recommendations for future research
-
The PMOS tool should be tested between wards in different hospital settings or between different hospitals to determine whether it has acceptable reliability and validity outside of the validation study described in this chapter. The research team plan to undertake a large multicentre evaluation of the tool in hospitals in the north of England.
-
Linked to the above, further work should explore whether this tool can be used for different patient groups in a range of settings such as community-based outpatient services or general practice. To achieve this, further validation work will be necessary to tailor the PMOS tool to these specific patient groups and clinical environments.
Conclusion
Patients are in a very good position to observe the safety of their care and that of others on the same ward/unit and are able to respond to items in the PMOS tool that are indicative of the safe organisation of their care. In this study we have developed a tool that patients can complete during their hospital stay.
Overall, the tool exhibited acceptable reliability and validity, subject to the caveats mentioned above. Based on a clear theoretical framework, the PMOS tool uses the patient’s perspective with the intention of identifying ‘latent’ weaknesses that could contribute to future PSIs. The development of this tool has implications for practice and research. Clinicians can use this tool to gain an important patient perspective on safety, capturing areas of weakness that might otherwise go unreported or unidentified. There is no other such tool available at present to collect this type of information from patients. The tool also offers an opportunity for clinicians and hospitals to track changes in safety over time by repeated assessments within wards.
Chapter summary
This chapter has outlined the development and validation of the first known tool to allow patients to feed back about factors associated with their care that are known to contribute to PSIs. This evidence and theory-based tool effectively allows health-care organisations to systematically capture a perspective on safety – the patient’s – that has been missing in the assessment of risk to date. However, alongside reporting prospectively on factors that may impact on patient safety in the future, it is important to understand how patients might also report retrospectively on patient safety problems that they have already experienced. The next chapter outlines a review of the evidence of patient reporting in hospital settings, addressing the question of whether patients can help us learn about errors in hospitals.
Chapter 4 Learning from error: a systematic review of the evidence on patient reporting of patient safety in hospital settings
Abstract
Background: Patients are increasingly being thought of as central to patient safety. A small but growing body of work suggests that patients may have a role in reporting patient safety problems within a hospital setting.
Objectives: This review asks (1) what patients can report, (2) in what settings they can report, (3) at what times patients have been asked to report and (4) how patients have been asked to report.
Method: Five databases [MEDLINE, EMBASE, CINAHL, (The King’s Fund) Health Management Information Consortium (HMIC) and PsycINFO] were searched for published literature on patient reporting of patient safety ‘problems’ (a number of search terms was utilised) within a hospital setting. In addition, reference lists of all included papers were checked for relevant literature.
Results: Thirteen papers were included within this review. All included papers were quality assessed using a framework for comparing both qualitative and quantitative designs and reviewed in line with the study objectives.
Conclusions: Patients are clearly able to report on patient safety but included papers varied considerably in focus, design and analysis, with all lacking a theoretical underpinning. In all papers, reports were actively solicited from patients, with no evidence supporting spontaneous reporting. The impact of timing on accuracy of information is unknown and many vulnerable patients are not currently included in patient reporting studies, possibly introducing bias and underestimating the potential of patient reporting. The future of patient reporting may well be as part of an ‘error detection jigsaw’, used alongside other methods as part of a quality improvement toolkit.
Chapter rationale
Alongside being part of the prospective assessment of risk, patients may also be in a position to provide health services with information about retrospective indicators of patient safety. If tools are to be developed that allow the systematic capture of information from patients about safety problems that they have experienced in hospital, it is important that these are based on a full understanding of the extant evidence on the role of patients in reporting PSIs. This chapter outlines a systematic review of the evidence on patients’ reporting on patient safety within hospital settings.
Introduction
Rationale
The patient ‘voice’ is emerging as a key part of the research, development and management of patient safety both internationally and within the UK. The main driver for this shift in focus was the political move towards ‘patient choice’ as part of creating a more dynamic and responsive health service. 213 This change of policy was aimed at empowering patients to act as partners in their health care and in terms of patient safety has been translated in practice within the UK through the establishment of national initiatives such as the Patient Safety Champion Network214 and, at a local level, the Patient Advice and Liaison Service (PALS). 215 This was brought sharply into focus by the UK coalition government’s White Paper outlining the legal duty of those with health service commissioning responsibility; the principal aim is to facilitate active participation from patients and public. 186 The UK government health minister at the time encapsulated his vision of the patient perspective in the words, ‘No decision about me, without me’. 216
In addition to establishing patient choice in shaping health-care services, patients have also more recently been viewed as key stakeholders in the management of patient safety. The former UK NPSA recognised this by incorporating patient reports into its National Reporting and Learning System, alongside clinician reports. However, the current position within the NHS and health-care services internationally is still very much dominated by clinician-led reporting of PSIs, a position that has also been apparent in the most recent data published by the UK Patient Safety Observatory. 217
It is intuitive that patients would be a useful source of information on patient safety. Patients are often the only common link between the different treatments and consultations that make up their health-care experience and, as such, are uniquely motivated and positioned to contribute to the quality and safety of their own care. 42,218 However, despite this, it has often been commented that one of the main issues for the patient safety movement has been the lack of patient perspective. 7,219–222 Indeed, a central question for patient safety research must be how relevant and effective research and management of patient safety is if one of the protagonists in the patient safety experience is effectively excluded.
To address such questions, and in line with policy development as already discussed, researchers have more recently started to focus efforts on understanding more about how best to engage patients in patient safety initiatives. One emerging area is how best to involve patients in the reporting of PSIs or issues. A recent systematic review of patient reporting across a variety of settings177 concluded that, despite a relative paucity of studies in this area, patient reporting has been shown to be reliable following corroboration of reports. 36,38,223 However, this timely review also revealed that the little evidence that is available comes from a disparate body of work and that further research is required to identify the optimal method for capturing patient reports, cognisant of different clinical settings and the duration of stay.
In line with such a recommendation, we undertook this systematic review as part of a wider research project aimed at developing and evaluating a range of patient-led PSI reporting tools. Crucially, this research addressed the need for a system that can be used across a hospital with its diversity of clinical settings and which can allow patients to ‘hot report’, that is, to report PSIs while receiving treatment in hospital, thus reducing retrospective recall bias known to be an issue in incident reporting systems across high-risk domains. 224,225 Given the developmental nature of this work, it was clear that a detailed examination of the current evidence was required, informed by a human factors perspective, to ensure that any reporting tool developed would both build on the existing knowledge base and contribute to effective clinical governance. This review builds on previous reviews177,226 by widening the search strategy (through increased numbers of databases searched), examining the quality of the included papers and, most importantly, focusing only on studies exploring patient reporting of PSIs experienced within a hospital setting. This focus has allowed us to consider in greater depth how patient reporting has been examined within the wider context of the systems that exist in hospitals dedicated to the measurement of patient safety, the management of this information and the clinical governance or quality improvement agenda.
Objectives
The nature of the literature examining patient-reported safety concerns precluded a full meta-analysis. Therefore, for this review, our specific objectives were to explore:
-
the types of PSIs identified by patients, how this differs from that found for other reporting methods and where patient reporting fits in relation to other methods of measuring patient safety
-
the settings in which patients have been asked to report on PSIs
-
the timing of patient reports of PSIs in relation to the experience of the patient safety ‘event’
-
how patients have been asked to report on PSIs and what has been done with this information.
Methods
Eligibility criteria
The inclusion and exclusion criteria were decided on with respect to the aims of the review and defined in terms of the population, intervention/comparators, outcome measures and study design, as advised by the Centre for Reviews and Dissemination. 227 Studies were included if they satisfied one or more criterion under each of the following headings.
Participants
-
Adult patients in a hospital setting/recently hospitalised adults.
Interventions/comparators
-
Intervention studies in which patients are involved in reporting PSIs.
-
Surveys of or interviews documenting patient-reported patient safety events or incidents.
-
Comparison with staff incident reporting or case note review.
Outcomes
-
Reported error rates.
-
Adverse event/adverse drug event rates.
-
Incidence of complaints.
-
Patient and/or staff satisfaction.
Study design
-
Experimental [randomised controlled trials (RCTs), cluster RCTs].
-
Quasi-experimental (non-randomised studies, pre and post studies).
-
Cross-sectional (during hospitalisation or post-hospitalisation) surveys or interviews.
-
Cohort studies.
Other criteria
Studies were excluded if:
-
they were published in a language other than English
-
they were unpublished
-
they took place in a health-care setting other than a hospital
-
they were case studies, discussions, reviews or editorial articles
-
they related specifically to adverse drug reactions or pharmacovigilance.
Data sources and search strategy
Five databases were searched for this review: MEDLINE, EMBASE, CINAHL, (The King’s Fund) HMIC and PsycINFO. These databases were selected to cover both medical and psychological literatures. The search strategy was developed iteratively, with reference to the study aims and an assessment and ongoing revision of the keywords of target articles. In addition, in conjunction with the specialist librarian at Bradford Teaching Hospitals NHS Foundation Trust, subject headings that mapped onto the keywords were identified, to ensure that papers not using the keywords but within the subject area were also picked up within the search. For each of the databases, the subject headings were identified separately to ensure optimum coverage. The final list of search terms is detailed in Appendix 3. Final searches across all five databases were run on 9 August 2010.
Given that patient reporting is a relatively new phenomenon in both research and practice, that the related terminology lacks standardisation177 and that through pilot searches we identified some inaccuracy of indexing in electronic databases, we opted for a search based on a high-sensitivity, low-specificity strategy. This also necessitated hand searching the literature.
Data extraction
Figure 4 outlines the process for study selection. First, results from the searches were merged using reference management software and duplicates were removed. The titles and abstracts of retrieved citations were reviewed by a researcher (JW) using the inclusion and exclusion criteria, following which the full texts of 51 studies were retrieved for further assessment. Reasons for excluding studies at this stage were primarily because they did not relate to the subject area and because they were non-empirical studies such as letters, editorials or position papers. The retrieved full-text articles were then scrutinised against the inclusion criteria and the reference sections examined, which identified a further 17 articles. At this stage, concordance in the decision for inclusion or exclusion was achieved by two reviewers (JW and GA). Following this, 12 articles were selected for inclusion in the review. Typically, studies excluded at this stage discussed patient involvement in general patient safety initiatives (rather than incident reporting) or concerned patient reporting in a non-hospital setting (e.g. primary care or outpatient/ambulatory care, community-based surveys, etc.). A further paper was included within the review during the manuscript preparation process, resulting in a final total of 13 papers34–36,38,228–236 selected for inclusion in the review.
Quality assessment and data synthesis
The heterogeneous nature of the study designs, aims and key findings precluded a full meta-analysis of the data from this review. For this reason, we do not report here on analytical strategy or summarise measures. Furthermore, given the range of methods used within the identified studies, a validated quality assessment tool was utilised that allows comparative analysis of quantitative, qualitative and mixed-methods studies. 74 This tool facilitates assessment of the robustness of the study design and methods; reference to theory; sample size and representativeness; any validation of measures; extent of user involvement; and evidence of critical discussion and limitations. Studies were rated against 16 criteria (14 applying to qualitative papers only, 14 to quantitative papers only and 16 to mixed-methods papers), with scores ranging from 0 to 3 for each criterion (0 = no evidence, 3 = ‘complete’). Total scores were translated to percentages allowing comparison across studies using different methods. Included studies were judged against the scoring criteria by a researcher (JW), with a random sample of three studies further scrutinised by another researcher (GA). Differences in scores were resolved by discussion.
Data from included studies were extracted under the following headings:
-
authors
-
sample size
-
target population
-
location
-
design
-
aim
-
method of data collection
-
nature of questions eliciting patient-reported PSIs
-
timing of reporting and length of patient recall period
-
comparison with other methods of error/incident detection
-
concordance across error detection methods.
To allow for further comparison, we also extracted data pertaining to the following outcome measures:
-
total number of patient-reported PSIs
-
percentage of patient sample reporting a PSI
-
number of PSIs per patient
-
method of classifying patient-reported PSIs
-
total number of patient incident reports classified as a PSI
-
percentage of patient sample reporting a classified PSI
-
number of classified PSIs per patient.
Results
A total of 3304 citations were retrieved, with a further 19 papers identified from hand searching (one paper was identified during the manuscript preparation process). Following the screening process (see Figure 4), 13 papers34–36,38,228–236 were included in this review. Tables 7 and 8 provide a synthesis of the data following extraction from the identified papers. Most studies were carried out in North America (69%) and employed a cross-sectional design (69%). The mean quality rating for identified papers was 53% (range 38–64%; full scores can be obtained by contacting the lead author). A significant criticism of this body of work centres on the lack of theoretical frameworks for any of the included papers. Similarly, little attempt was made to identify appropriate sample sizes or assess data collection tools for reliability or validity. However, most papers reported studies that appeared to apply appropriate experimental designs and analytical techniques to meet the research aims, with most samples reasonably or very representative of the target population.
Study | Sample size | Target population | Location | Design | Aim | Method of data collection | Nature of questions eliciting patient-reported PSIsa | Timing of reporting and length of patient recall periodb | Comparison with other methods of error/incident detectionc | Concordance across error detection methodsd |
---|---|---|---|---|---|---|---|---|---|---|
Agoritsas 200538 | 1518 | Recently hospitalised adults from across entire hospital population | Switzerland | Cross-sectional | To investigate the frequency of undesirable events reported by recently discharged patients and to identify correlates of undesirable events | Mailed survey | Predefined categories: survey on ‘undesirable events’ | Post discharge: recall period unclear | No | – |
Fowler 2008233 | 2582 | Recently hospitalised adults from medical and surgical units across 16 hospitals | USA | Cross-sectional | To determine rates at which patients experience what they considered to be adverse events, to describe the kinds of events they reported and to identify the correlates of these reports | Telephone survey | Predefined categories using open-ended questions: interviews on four ‘categories of care’ | Post discharge: average 9 months | No | – |
Friedman 2008235 | 201 | Adults attending an emergency department | USA | Prospective cohort | To investigate whether patients or their families can identify adverse events in the emergency department, categorise these events and compare these reports with current methods | Mixed methods: telephone survey and medical record review | Open-ended questions: interviews about ‘problems or mistakes in care received, or hurt suffered as a result of care’ | Post discharge: < 7 days | Medical record review and hospital patient and visitor incident database | None of the patient-reported adverse events were recorded within the patient medical records or in the hospital reporting database |
Kaboli 2010228 | 126 | Adult inpatients in a general medicine unit | USA | Prospective cohort | To analyse and compare four different methods of detecting medication misadventures to determine the optimal system for reporting clinically observed medication misadventures | Inpatient interview | Predefined categories but open-ended questions: interviews on ‘medication misadventures’ | At discharge or < 2 days post discharge: recall period unclear | Physician reports, nurse reports and medical record review | Six patient reports (50% of physician-classified reports) were confirmed by medical record review, one patient report (8% of physician-classified reports) was confirmed by medical record review and nurse report and one further patient report (8% of physician-classified reports) was confirmed by medical record review and physician report |
Lopez 2009232 | 2582 | Recently hospitalised adults from medical and surgical units across 16 hospitals | USA | Cross-sectional | To assess how disclosure of adverse events by medical personnel affects patients’ ratings of the quality of care | Telephone survey | Predefined categories but open-ended questions: interviews on four ‘categories of care’ | Post discharge: average 9 months | No | – |
Schwappach 2008236 | 125 | Adult inpatients from medical and surgical units across two hospitals | Switzerland | Cross-sectional with follow-up | To develop and pilot a brief patient safety survey applicable to inpatient care in Swiss hospitals | Mixed methods: inpatient survey with follow-up interview | Predefined categories: survey on ‘undesirable events’ | Unclear | No | – |
Schwappach 2010229 | 479 | Adult oncology patients | Switzerland | Cross-sectional | To explore chemotherapy patients’ experiences of drug administration safety and to investigate the relationship between perceptions of risk and harm from error, staff safety practices and patients’ engagement in error prevention strategies | Mailed survey | Predefined categories: one item on a survey | Unclear | No | – |
Weingart 200536 | 228 | Adult inpatients in a general medicine unit | USA | Prospective cohort | To elicit incident reports from hospitalised inpatients to identify and characterise adverse events and near-miss errors | Mixed methods: inpatient interviews, telephone follow-up and medical record review | Open-ended questions: interviews about ‘problems or mistakes in care received, or hurt suffered as a result of care’ | For inpatient interviews: < 3 days; post discharge: < 10 days | Medical record review and staff incident reporting database | No overlap with staff incident reporting database. Medical record review confirmed 40% of patient-reported PSIs |
Weingart 200735 | 193 | Adult oncology patients | USA | Cross-sectional | To investigate the ability of patients in ambulatory specialty care to recognise medical errors and iatrogenic injuries | Inpatient interview | Open-ended questions: interviews about ‘anything unsafe experienced during care’ | < 7 days | No | – |
Weingart 2004230 | 209 | Adult inpatients in a general medicine unit | USA | Cross-sectional | To consider lessons learned from a patient partnership intervention to prevent adverse drug events among medical inpatients | Inpatient survey | Predefined categories: survey about medication errors and quality of care | At discharge or post discharge: recall period unclear | No | – |
Weissman 200834 | 998 | Recently hospitalised adults from medical and surgical units across 16 hospitals | USA | Cross-sectional | To compare adverse events reported in post-discharge patient interviews with adverse events detected in medical record review | Mixed methods: telephone survey and medical record review | Predefined categories but open-ended questions: interviews on four ‘categories of care’ | Post discharge: 6–12 months | Medical record review undertaken by (i) trained nurses and then (ii) physicians | Concordance between medical record review and patient survey found to be 77% overall |
van den Bemt 1999231 | 620 | Inpatients in paediatric and internal medicine units | The Netherlands | Prospective cohort | To investigate the relative value of adverse drug events reported by doctors, nurses and patients | Inpatient interview | Predefined categories but open-ended questions: interviews about ‘adverse drug events’ only | < 24 hours | Staff incident reporting database | 1% concordance between physician and patient reports; 2% concordance between nurse and patient reports |
Zhu 2011234 | 2582 | Recently hospitalised adults from medical and surgical units across 16 hospitals | USA | Cross-sectional | To examine the degree to which physician reviewers agreed that patient reports of ‘negative effects’ constituted adverse events and to identify questionnaire items that affected reviewers’ judgements | Telephone survey | Predefined categories using open-ended questions: interviews on four ‘categories of care’ | Post discharge: 6–12 months | No | – |
Study | Total number of patient-reported PSIs | % of patient sample reporting a PSIa | Number of PSIs per patientb | Method of classifying patient-reported PSIs | Total number of patient incident reports classified as a PSIc | % of patient sample reporting a classified PSId | Number of classified PSIs per patiente |
---|---|---|---|---|---|---|---|
Agoritsas 200538 | 1814 | 48 | 1.3 | None | Not reported | – | – |
Fowler 2008233 | Not reported | 29 | – | Physician | 845 | 24 | 0.33 |
Friedman 2008235 | 18 | 9 | 0.1 | Physician | 18 | 9 | 0.09 |
Kaboli 2010228 | 25 | – | 0.2 | Physician and pharmacist | 12 | – | 0.10 |
Lopez 2009232 | Not reported | 29 | – | Physician | 845 | 24 | 0.33 |
Schwappach 2008236 | 128 | 45 | 1.02 | Researcher | Not reported | – | – |
Schwappach 2010229 | Not reported | 16.2 | – | None | Not reported | – | – |
Weingart 200536 | 310 | 49 | 1.4 | Physician | 62 | 20 | 0.27 |
Weingart 200735 | 121 | 43 | 0.63 | Physician and nurse | 20 | – | 0.10 |
Weingart 2004230 | 19f | 11g | 0.11 | None | Not reported | – | – |
Weissman 200834 | Not reported | – | – | Physician and nurse | 304 | 23 | 0.31 |
van den Bemt 1999231 | Not reported | – | – | Status of classification unclear | 311 | 29 | 0.50 |
Zhu 2011234 | 1170 | 29 | 0.45 | Physician | 833 | 24 | 0.32 |
Terminology
All papers concluded that patients were able to report on PSIs in a hospital setting. However, the terminology used to describe such incidents did vary considerably across papers (Table 9). Four papers were concerned only with issues related to medication or treatment. 228–231 When a broader perspective was taken, papers were split between those concerned only with adverse events as categorised by physician review34,232–234 and those that widened this categorisation to also include near misses/close calls and medical errors with minimal or no risk of harm. 35,36,235 Two of these latter papers took a more analytical approach to differentiate between PSIs (adverse events, near misses/close calls and medical errors with a minimal risk of harm) and service quality incidents and process of care problems, respectively. 35,36 The last category was those papers in which patients were asked to what extent they had experienced any ‘undesirable events’ from a prespecified list. 38,236
PSI terminology | Studies using terminology |
---|---|
Adverse event, near miss/close call, medical error with minimal or no risk of harm | Weingart et al.,35 Weingart et al.,36 Friedman et al.235 |
Adverse drug event and medication error | Kaboli et al.228 |
Adverse event only | Weissman et al.,34 Lopez et al.,232 Fowler et al.,233 Zhu et al.234 |
Adverse drug event only | van den Bemt et al.231 |
Undesirable event | Agoritsas et al.,38 Schwappach236 |
Error in (chemotherapy) treatment | Schwappach and Wernli229 |
Mistakes related to medication | Weingart et al.230 |
Nature of patient reports
Types of safety issues identified in patient reports
The type of safety issues identified by patients is influenced by the type of questions asked by researchers. Ten of the included papers chose to restrict their questions to a predefined category or set of categories. 34,38,228–234,236 Clearly, when patients were asked to report on predefined categories of PSIs, this limited their responses. Only three papers reported asking open-ended questions in which patients were not restricted to certain categories of PSI. 35,36,235 Table 10 summarises the types of PSIs reported by patients from the three papers asking open-ended questions. The taxonomy of patient reports is limited to these three papers, as to include others (in which reports were restricted to certain categories) would risk inflating certain PSI types and therefore misrepresent the available data. It is clear from the summary that patient reports span the full clinical spectrum – from diagnosis and testing through to problems with treatment, medication and care procedures. However, patients do seem to report more medication-related PSIs than any other category of PSIs. In addition to the type of clinically focused PSIs that staff might be likely to report, patients also reported other issues, particularly service quality events. It should, however, be borne in mind that ‘service quality’ reports were over-represented in one study. 35 This used as its sample day-case oncology patients, whom it could be argued may have a smaller spectrum of possible incidents and different priorities regarding reporting compared with inpatients from a range of hospital specialties. This is likely to have artificially inflated the total percentage of service quality reports from the available data and in doing so may not be indicative of the reality of reports from the wider hospital inpatient population.
Category of PSI | Number of patient reports (% of sample)36 | Number of patient reports (% of sample)235 | Number of patient reports (% of sample)35 | Total number of patient reports (% of total sample) |
---|---|---|---|---|
Diagnosis-related problems (e.g. diagnosis error) | 3 (4) | 1 (5) | – | 4 (3) |
Medication-related problems (e.g. failure to order drug, wrong dose, wrong route, known allergy) | 47 (65) | 7 (39) | 7 (17) | 61 (46) |
Operative- or procedure-related problems (e.g. post-procedure-related problems) | 4 (6) | 4 (22) | – | 8 (6) |
Problems with clinical services (e.g. failure to draw blood, wrong patient, wrong body part, delays to tests or procedures) | 7 (10) | 3 (17) | 1 (2) | 11 (8) |
Service quality problems (e.g. waits and delays, problems with care environment) | 6 (8) | 3 (17) | 27 (64) | 36 (27) |
Other problems (e.g. failure to follow up, equipment malfunctions) | 5 (7) | – | 7 (17) | 12 (10) |
Total no. of reports | 72a | 18 | 42b | 132 |
Parties involved in patient reports
Only two papers made reference to the parties involved in patient-reported PSIs. 35,36 Nurses were identified more than any other professional group (26%), closely followed by physicians (22%), with other health professionals and visitors also identified (15% and 0.5%, respectively). Interestingly, in a large percentage of reports, the party involved could not be identified or was ill defined (54%). Multiple parties were often identified in patient reports.
Classification of patient reports
In total, 10 papers reported using some form of review and classification of patient-reported PSIs. 34–36,228,231–236 One of these papers reported researcher confirmation of patient-reported PSIs only236 and in one further paper the nature of the personnel undertaking the classification was unclear. 231
Of the remaining eight papers, five used physicians only to undertake the classification of PSIs,36,232–235 two used physicians and nurses34,35 and one reported classification by both physicians and pharmacists. 228 Three of these papers did not report the number of patient-reported PSIs but only those that after review had been categorised as PSIs. 228,234,235 In the five papers that did report on both,35,36,228,234,235 there was wide variation in the degree to which patient reports were judged to constitute classified PSIs (17–100%, mean 51%).
As part of the classification process, a judgement is usually made about two key risk indices: the degree of preventability and the severity of any given report. Of the eight papers that undertook classification of patient reports, seven report enough information from which to draw definitive data about preventability and severity. 34–36,232–235 Three of the included papers are based on the same data set232–234 and therefore, for the purposes of summarising these data, results from only one of these papers are reported. 233 Table 11 details the available information from the five eligible papers about the physician-classified (and other health professional-classified) preventability and severity of patient-reported adverse events. 34–36,233,235
Study | Severitya | Preventabilitya | ||||
---|---|---|---|---|---|---|
Serious/life-threatening | Significant/moderate | Minor/insignificant | Definitely/probably preventable | Definitely not/probably not preventable | Unknown | |
Weingart 200536 | 5 | 65 | 30 | 65 | 35 | – |
Fowler 2008233 | 18 | 57 | 25 | 31 | 69 | – |
Weissman 200834 | 18 | 62 | 20 | 30 | 65 | 5 |
Friedman 2008235 | 0 | 80 | 20 | 60 | 40 | – |
Weingart 200735 | 0 | – | 100b | 50 | 50 | – |
Mean % across studies | 8 | 53 | 39 | 47 | 52 | 5 |
Although patients across the studies included in Table 11 clearly report PSIs across the full range of physician-classified severity, patient-reported PSIs do appear to be towards the significant to insignificant end of the severity spectrum, with fewer patients reporting serious or life-threatening PSIs. In terms of preventability, however, patients do seem to be in a position to report PSIs that are judged by physicians to be preventable.
Concordance with other error detection methods
Only five of the thirteen identified papers sought to examine the degree of concordance between patient reporting and other methods of error or incident detection. 34,36,228,231,235 Medical record review was the method found to have the most concordance with patient reporting (50%,228 77%34 and 40%36), although one paper reported no concordance between these methods. 235 Staff incident reporting was less likely to overlap with patient reports. Physician and nurse reports were found to have 8% concordance each with patient reports in a paper concerning medication misadventures228 and 1% and 2%, respectively, in a paper examining adverse drug events. 231 One further paper found general staff incident reporting to have no concordance with patient reports of adverse events and near misses. 36 It is important to note, however, that for two of the above papers,34,231 only patient reports that had been classified (as adverse events and adverse drug events, respectively) were included in the final sample. It is possible that this may have inflated the concordance above that which might have been found if all patient reports (and not just those ‘confirmed’ as adverse events/adverse drug events) were taken into consideration.
Health-care setting
Although all papers included in this review concerned PSIs experienced within hospitals (for either inpatients or day patients), there was variety in the type of setting in which patients were asked to report PSIs. Three papers were based in general medical units,36,228,230 with five papers sampling patients from both general medical and surgical units. 34,232–234,236 One further paper reported a sample based within medical and paediatric units. 231 Two papers were based in oncology units,35,229 with one paper sampling patients within an emergency department. 235 Only one paper reported a sample from across the full hospital population. 38
Timing of reports
Papers reported a variation in the ‘recall period’ of patient-reported PSIs, that is, the length of time from a patient experiencing a PSI and reporting it. Two out of 13 papers reported surveying patients at discharge or post discharge228,230 and for both of these papers the recall period was unclear. Five papers reported surveying patients at post discharge only. The length of recall period for these papers varied from < 7 days235 to between 6 and 12 months. 34,232–234 Two papers using inpatient interviews specified shorter recall periods of < 24 hours231 and < 7 days,35 with one further paper using both inpatient interviews (recall period < 3 days) and a post-discharge survey (< 10 days after discharge) to identify PSIs experienced between the initial inpatient interview and discharge. 36 Irrespective of the method of data collection for patient incident reports, in five of the papers the recall period was unclear or not reported. 38,228–230,236
Method of eliciting patient reports
Two methods of collecting incident reports from patients dominated included papers. Interviews (often using a quantitative structured survey format) were the norm, with nine papers reporting using this method. 34–36,228,231–235 The other method of collecting patient reports was to administer a survey or questionnaire to patients to complete alone. This method was reported in three papers. 38,229,230 One further paper reported utilising both methods, with a questionnaire first supplied to inpatients and a follow-up interview undertaken for those reporting an adverse event. 236
Relation to clinical governance/quality improvement
With regard to what is done with the information from patient reports, none of the papers in this review mentioned how safety information from patients could be used as part of the wider clinical governance or quality improvement agenda. In addition, no paper made mention of feedback to study participants or staff groups hosting the research.
Discussion
This chapter set out to review the extant literature examining the nature of patient reporting of PSIs within a hospital setting. The literature suggests that, in academic terms, patient reporting is in its infancy, with included papers varying considerably in terms of their focus, design and quality. Indeed, some of the papers seemed only to include patient reporting as a minor part of the research aims. This notwithstanding, we feel confident that this literature allows a number of conclusions to be drawn, which have implications for both research and practice.
Can patients report patient safety incidents in a hospital setting?
It is clear when one considers the results in their totality that patients are in a position to report on safety-related issues experienced in a hospital setting. Furthermore, these studies do suggest that patients are able to identify PSIs from across a range of incident ‘types’, referencing a variety of different parties, and across the full range of preventability and severity. On this last point, although patients generally reported PSIs that were not life-threatening, they did report a large number of PSIs rated as ‘significant’ by physicians, suggesting that the patient role in error detection is unlikely to be limited to information deemed to be clinically insignificant. Furthermore, in those studies undertaking physician classification, on average, nearly half of all PSIs reported by patients were judged to be ‘definitely’ or ‘probably’ preventable. This clearly demonstrates that, if asked the right questions about the incident context, patient reporting may offer health-care providers a valuable source of information about how to proactively manage safety.
Implications for patient reporting: research
None of the reviewed papers used any theoretical underpinning to inform either their design or analysis of patient reports. A number of models may be of value in investigating patient reporting, for example social cognition models such as the theory of planned behaviour. 237 However, we believe that taking a human factors perspective is perhaps the most appropriate foundation for research in this area, because of its focus on the multilevel, multifactorial nature of PSI causation, as well as its increasing adoption by service providers in safety improvements. Additionally, this perspective also attributes a high value to near-miss events as well as harm events, thereby widening the opportunity for learning from PSIs. Developing a method for capturing patient reports without recognising human factors may lead to a superficial interpretation of PSIs and one that may inappropriately focus on the role of individuals in causation. This could be a particular issue for nurses, who as a professional group are frequently mentioned in patient reports, which may be largely because of their ongoing visibility through the ‘patient lens’ and numerous encounters as the last point of direct professional contact during a process of care. It has been suggested that patients do not have knowledge of the reasons for, or consequences of, adverse events. 233 We would contend that has yet to be fully established empirically and would likely vary across different patients and their level of contact with health services. Furthermore, research from staff incident reporting suggests that such schemes fail to routinely capture the context and causes of PSIs. 170,238 As for the value of patient reporting, we can infer from such research that, even when those reporting may understand the clinical reasons for preventable events, reporting schemes may not facilitate the capture of such information, leading to the erroneous conclusion that they are unaware of any causal antecedents.
The length of recall period between experiencing and reporting a PSI remains unexplored within the literature. Some authors have commented that lengthy recall periods may introduce ‘recall bias’ into patient reporting. 232,233 The authors of one study did report that PSI rates did not decrease as a function of time,233 but this was related only to rates of PSI reports, which is different from any impact on the accuracy of information. This literature seems to currently lack a sound understanding of, first, the key biasing influences on patient reporting of PSIs and, second, the optimal period of recall. Further research is needed to clarify the optimum period for recall based on the experience of real patients with associated issues of acuity, length of stay, severity of illness, the emotional impact of a PSI and the potentially disorienting hospital environment.
A related issue is the method used by patients to report PSIs within these studies. All of the included studies actively ‘solicited’ reports from patients, either through an interview or through a written survey. None of the study designs allowed for patients to spontaneously report a PSI. This is significant as it may be that research risks inflating the extent to which patients may be willing to report PSIs, simply through the methods used to collect such reports. Some authors have reflected on this point, highlighting the related issue of how the role of the researcher and the nature of the questions may preferentially elicit certain responses. 35 Perhaps the key research question should no longer be ‘Can patients report?’ but rather ‘Can patients report in a system designed to collect this information routinely in a clinical setting?’ Consequently, there is currently no evidence whether patient reporting is feasible outside of a research study or whether it could be an integral and complementary element of a service provider’s safety intelligence network. To assess the latter and examine the validity of patient reporting, we argue that future studies should routinely compare not only the type but also the quality of patient reports with conventional methods of incident detection such as case note review.
Irrespective of the specific study design or the nature of capturing the patient safety reports, there are ongoing issues that researchers, practitioners and managers need to be cognisant of when designing studies or indeed systems to capture PSI reports from patients. A significant issue is the somewhat paradoxical situation that those who are least able to report PSIs may also be at most risk of experiencing them. A number of authors have commented previously on this paradox with reference to the inherent bias arising from asking only those who are discharged from hospital about PSIs, when those unfortunate enough not to have survived and been discharged may have been at a higher risk of experiencing a PSI. 34,233 It has been demonstrated empirically that older people do experience more PSIs,239 but there is also emerging evidence to suggest that other factors influence the likelihood of experiencing a PSI (e.g. non-native language speaking),240 factors that may also lead to under-representation in the studies conducted so far. Overall, current estimates of patient reporting may be particularly inaccurate on the basis that some of the most vulnerable groups are under-represented in patient safety research. Further research should focus on the best ways to engage with these patient groups, to gain a fuller understanding of patient-reported PSIs.
Implications for patient reporting: practice
If patient reporting is to become a valid tool for measuring ‘performance’ in patient safety terms, consideration must be given as to how it fits with other existing error detection methods. Some authors have discussed the problem of a higher false-positive rate for patient reporting of medication errors than for errors detected through physician and nurse reporting. 228 Perhaps this finding highlights a weakness in the proposition that patient reporting can be a valid error detection tool. However, others have presented the counter argument that, as false-positive reports can be ‘validated’ by clinical review, the bigger issue is that patients might suffer from higher rates of false negatives than clinicians, meaning that many potential PSIs may go undetected. 236 Thus, the evidence seems to suggest that patient reporting may risk both over- and underestimating the PSI rate because of misunderstanding of what is normal within the clinical context. There is evidence from the wider incident reporting literature that, when triangulated, different error detection methods may lack a high degree of overlap in the PSIs identified. 228,241–244 Taking this into consideration, patient reporting may suffer from some of the perennial problems inherent in staff reporting,245 but as a part of an error detection jigsaw it may also prove a valuable, and as yet untapped, resource. Given the continuing policy emphasis on patient involvement in care, it would seem entirely appropriate to integrate patient reporting of safety concerns as a formal component of clinical governance.
Limitations
Because of the focus of the overarching project (on the basis of which this review was conducted), the search was limited to studies within a hospital setting. This clearly does exclude other health-care settings, for example primary/ambulatory care. However, very little information is published about incident reporting (staff or patient) in primary care and so the inclusion of such studies here may skew the findings of hospital-based studies.
Recommendations
As discussed above, future research is clearly needed to demonstrate that patient reporting can move beyond the research domain and become an established part of clinical governance. Some authors make suggestions for implementing patient reporting in practice and one has discussed the possibility of distributing notepads to patients to write down concerns, events or questions to share with health-care staff. 228 Others discuss the possibility of ‘hot reporting’, with systems designed to allow patients to call a dedicated telephone line to the hospital pharmacy, to report medication errors. 231 Both suggestions take patient reporting into the realm of workable systems, but with the caveat that it is combined with other error detection methods to form part of an overall safety strategy. Furthermore, to be successful, there should be a ‘collective responsibility’ for development of any patient reporting system, with ‘coordinated improvement efforts involving all members of the healthcare team (including patients)’ (p. 172). 246 Indeed, for a given system to be workable for patients, methods of reporting should be designed, tested and evaluated in consultation with patients. The work of Bate and Robert247 would be useful here, which suggests that, whatever patients design, it should be part of a carefully managed emancipatory process that incorporates staff as stakeholders, increasing a sense of co-ownership but also ultimately demonstrating a fit with the pragmatics of clinical governance.
As with all patient involvement, before establishing patient reporting tools within clinical settings, consideration does need to be given to the issue of patient burden. It is an issue that has been raised previously,13 with the concern expressed that blanket patient involvement interventions may risk shifting the responsibility of safety onto patients, at a time when they are arguably at their most vulnerable. Furthermore, we know that not all patients will be willing or able to be engaged with such interventions and therefore it is important that such patients are not negatively affected as a result of their lack of engagement. Going forward, both research and practice must be mindful that any approach must be flexible enough to deal with such differing levels of engagement.
Conclusions
Patient involvement is a policy imperative. It would appear that hospitalised patients have the potential to report safety concerns. However, the evidence base is currently equivocal and dominated by studies that have focused on active solicitation to the neglect of ‘hot reporting’. Future study designs should be underpinned by a human factors approach, developed in collaboration with patients, taking account of memory recall and other cognitive biases, and use terminology that is understandable to patients but also reflects the predominant language of patient safety. Samples should be representative of the entire hospital population and the tool or tools developed must complement existing organisational governance and improvement strategies.
Chapter summary
This chapter presents a detailed examination of the evidence for the role of patients in reporting PSIs within hospital settings. The findings suggest that patients may have a role in helping organisations learn from error but that there are many unanswered questions about how best to capture this information from patients. The next chapter presents a study that provides further information about one of these key questions: ‘What is the most effective mechanism for collecting patient feedback about patient safety?’
Publication statement
This chapter is based on a previously published paper. 15 We reproduce it here with permission under the terms of the Creative Commons Attribution Non-commercial License.
Chapter 5 Learning from error: testing three mechanisms for capturing patient-reported safety concerns
Abstract
Introduction: Emergent evidence suggests that patients can identify and report safety issues while in hospital. However, little is known about the best method for collecting feedback, with most research asking patients after discharge and questions based on predefined clinical categories. This study presents an exploratory pilot study of three mechanisms for collecting patient feedback on safety during their inpatient stay.
Method: Three mechanisms for capturing patient feedback were coproduced with health-care professionals and patients before being tested in an exploratory trial, using cluster randomisation at a ward level. Nine wards participated, with each mechanism tested over a 3-month period. Patients were asked to feed back safety concerns via the mechanism on their ward (interviewing at the bedside, a paper-based form or a patient safety ‘hotline’). Safety concerns were subjected to a two-stage review process to identify PSIs. Differences between mechanisms in reports per patient, likelihood of reporting, number of PSIs and ratings of severity and preventability were examined using analysis of variance and chi-squared analyses. Reported safety concerns were analysed qualitatively and a framework developed.
Results: In total, 178 patients were recruited into the study. Patients in the face-to-face interviewing condition provided more reports per patient and were more likely to report one or more safety concerns. No mechanism differed significantly in the number of classified PSIs. Because of low numbers, we were unable to analyse statistically physician-rated preventability and severity.
Discussion: Interviewing at the patient’s bedside is likely to be the most effective means of gathering patient feedback about the safety of care.
Chapter rationale
There is a consensus that asking patients about their experience of care is crucial if health-care organisations are to provide patient-centred services. However, much less is known about the specifics of systematically collecting information from patients about their experiences of services and, in particular, their views on the safety of these experiences. The study presented within this chapter goes some way to answering the important question, ‘What is the most effective means of collecting data from patients about patient safety?’
Introduction
Over the past decade, patient involvement in patient safety has become increasingly important to policy makers, health professionals and researchers alike and there is a slowly emerging consensus that patients can be meaningfully involved in their safety in a variety of ways and across a range of health-care settings. 15,207 Furthermore, the recently published reports by Robert Francis,182 Sir Bruce Keogh9 and the National Advisory Group on the Safety of Patients in England206 have served only to emphasise the need to elicit and respond to patient feedback about safety. Although such reports are focused primarily on the acute care setting, patients do seem to be in a position to report PSIs during acute periods of illness, with a variety of studies collecting information about safety experiences from patients receiving care in a hospital setting. 15,177
To date, most published work has captured patient safety reports retrospectively, usually after discharge. 15 Some authors have commented on the potential of such retrospective reporting to introduce ‘recall bias’ in the reporting process,232,233 although the exact nature of the biases affecting patients in their reporting of safety-related information either close to an event or after a period of time is still unclear. However, it is clear that ‘hot reporting’ that is, capturing patient reports as close as possible to the safety event, has not, to date, been explored in the literature.
A further issue in previous studies of patient reporting has been the structured approach to collecting data from patients, usually based on a checklist approach of known categories of patient safety events. 15 Although this is clearly a useful way of collecting information about patient safety, it does limit patient responses to the range provided by the content of the checklist or questions asked. However, given patients’ key position as observer to their entire care trajectory,218 it is intuitive to think that they might be able to report on a variety of safety issues that might not otherwise be obvious or available to staff on the ward or fit within the normal ‘clinical risk paradigm’. 207 Furthermore, authors have already commented on the broader definition that patients have of ‘safety events’ or ‘patient safety incidents’ in comparison to many medical or nursing staff. 35 Therefore, it would seem entirely plausible that patients may already be primed to provide health-care organisations with information that does not fit the strict definition or boundaries of medical error or adverse events, but which may prove useful for organisational learning. On this basis, it is perhaps important that tools for capturing patient-reported safety concerns are not limited to issues defined by a strictly clinical patient safety focus.
Another feature of the extant literature on patient involvement in safety reporting is the lack of patient involvement in the development of research materials or the design of data capture tools. This is seemingly at odds with the very ethos of patient involvement and is therefore a somewhat ironic feature of studies to date. However, it has been noted by key researchers in this area that to be successful there should be a ‘collective responsibility’ for the development of patient safety systems, which position patients as central to the process alongside health-care professionals. 246
Recent work has considered patients’ willingness to report safety experiences using different methods of data capture, concluding that patients report a greater willingness to speak to researchers as part of a study. 248 However, this study was limited to patients’ intentions, rather than documenting their actual behaviour using different methods of capturing patient reports. Given the known low correlations between intentions and behaviour,249 it is important for researchers to move beyond simply asking patients what they may do and observe what they actually do under real-life conditions. On this basis, perhaps studies should move from examining whether patients can report to whether patients can report in a system designed to collect this information routinely in a clinical setting.
This chapter presents a novel study designed to meet some of these identified gaps in the literature by examining the efficacy of three different (patient-derived) mechanisms for capturing PSIs from patients in an acute hospital setting. Crucially, for each of the three mechanisms, patients were asked to report during their stay in hospital and not following discharge. In addition, to ensure the generalisability of the conclusions from this study, the mechanisms were tested across a variety of hospital settings, including both inpatients and day-case patients and across general medical and surgical wards, as well as medical specialty wards. Lastly, alongside asking patients to provide information about the nature of their safety events or concerns, we also attempted to ask them to think about factors that may have led to the events occurring.
Specifically, we set out to investigate the following research questions:
-
Can patients relay their safety events or concerns using a variety of different mechanisms for capturing patient reports?
-
Which mechanism might be the most successful in collecting patient reports?
-
Are there differences in the types of patient-reported safety events or concerns captured by the different mechanisms in terms of medical classification or assessments of preventability or severity?
Methods
Study design
The study presented here was an exploratory pilot study using the principles of cluster randomisation.
Setting
This study was undertaken in a large acute teaching hospital based in the north of England between 1 March and 31 May 2011.
Sample
Participants were patients within the participating hospital. Nine wards from across the hospital agreed to host the research. The specialties represented across this sample are detailed in Table 12. The wards were selected to reflect a wide range of presenting patient conditions, treatment types and contact with health services (both inpatient and outpatient services).
Specialty | Wards |
---|---|
Surgery | Vascular surgery |
Urology | |
General surgery (two wards: one male, one female) | |
Medicine | Renal, gastroenterology and rheumatology |
Respiratory medicine, diabetes and endocrinology | |
Cardiology | |
Specialty medicine | Haematology (outpatients only) |
Chemotherapy (outpatients only) |
Development of mechanisms for capturing patient reports
The approach taken for developing ideas for the three mechanisms to capture patient reports was very much embedded in the principles of ‘codesign’,247 in which contributions from patients, health-care professionals and researchers were of equal value. Initial ideas emerged from a series of three focus groups with, first, patients, then health-care professionals and, finally, patients and health-care professionals together (see Ward et al. 250 for a full description of this process). At the final focus group members were asked to identify their ‘top three’ from the final list of ideas for mechanisms for capturing patient reports. The three mechanisms that were developed for further testing were (in order of popularity) (1) face-to-face interviewing by researchers, (2) a paper-based form and (3) a patient safety hotline – a centralised telephone line that could be called by patients from their ward or on their mobile phone.
These three ideas were then taken forward to a smaller working group for further development into workable systems. This working group included two patient representatives, two researchers (JW and GA) and two senior health-care professionals. The working group met three times between September and December 2010, with a brief to produce three workable ‘tools’ that could be tested within a hospital setting but that, crucially, would be a fair representation of the ideas that emerged from the focus groups. Final versions of the three tools were ratified by members of the working group before the study commenced, with all accompanying patient information reviewed by the patient representatives for accuracy and comprehension.
Terminology for capturing patient safety reports
In response to the acknowledged criticism of previous work in this area (that terminology for patient safety-related events or incidents is not standardised15,177), it was agreed that one aspect of the working group’s remit would be to agree on the terminology for describing what we wanted participants to report. There was consensus from the health-care professionals and patient representatives that ‘patient safety incidents’ would not be understood by patients and that it may also lead to only harm events being reported, at the expense of events that had the potential for harm, that is, ‘near misses’ or ‘close calls’. It was agreed, therefore, that the term used to describe PSIs to patients would be ‘safety concerns and experiences’, as it was felt by the group that this was inclusive of the potential for harm and specifically focused on safety.
Mechanisms for capturing patient reports
Face-to-face interviewing
This mechanism provided participants with a facilitated process for reporting their safety concerns and experiences through an interview with a researcher or research nurse on their ward or outpatient unit. After the recruitment process was complete, each participant was given the opportunity to report their safety concerns and experiences either at that time or at any time during the course of their inpatient stay (or outpatient treatment), by requesting an interview initiated from a dedicated telephone line based on the ward. This telephone line was installed on participating wards for the purposes of this research only and was always clearly signed and placed in an easily accessible position (e.g. in the main corridor, by the entrance to the ward, etc.).
Paper-based form
This mechanism provided participants with a simple form to complete when they wished to report a safety concern or experience. After participants were consented into the study, they were provided with five forms and advised that, should they wish to report a safety concern or experience during the course of their hospital stay, they could complete one of these forms and post it into a dedicated ‘drop box’ installed in the ward. These drop boxes were for the purposes of this research only and were clearly signed and placed in an accessible area on the ward.
Patient safety hotline
This mechanism provided participants with a facilitated – but remote – process of reporting any safety concerns or experiences that they had within their hospital stay. Once consented, participants were advised that they could call the hotline at any time, using a dedicated telephone line installed on the ward. During office hours this line was answered by a researcher or research nurse, but the facility was also made available outside of office hours through the provision of a sophisticated, bespoke answering service. This service allowed participants to go through the same questions that would have otherwise been asked by the researcher or research nurse, but using a guided automated response system designed by the research team (JW and GA) in conjunction with a company providing answerphone systems. A telephone line was installed in an easily accessible part of participating wards, but one where some degree of privacy was allowed. Telephone handsets were clearly signed and dialled only the patient safety hotline.
Procedure
Randomisation of wards
After securing agreement from the nine participating wards within the hospital, wards were randomly assigned to a condition (mechanism) using a random number generator. 251 Randomisation at the level of the individual participant (rather than cluster randomisation) was felt to be inappropriate because of (1) the prohibitive pragmatic considerations of installing the infrastructure required for the mechanisms (telephone lines and drop boxes) within each ward, (2) the possibility of contamination across participating patients and (3) the potential confusion arising from multiple mechanisms available on each ward.
Recruitment of participants
Recruitment followed the same process across all nine wards. All wards were visited by one or more researcher or research nurse at least three times each week across the 3-month study period. At each visit the researcher or research nurse would first speak to the ward clerk or staff at the central nurses’ station on the ward to ascertain which patients were not approachable on the basis of illness acuity or infection control. Patients were also not routinely approached if it was indicated by staff that they were disorientated or confused or if they were asleep or had visitors at their bedside. When patients were approached, they were first given a very brief description of the study and then left with a small, patient-friendly leaflet summarising the study and what participation meant for them. After a ‘cooling-off period’ of at least 24 hours, patients who had not been discharged were followed up to ascertain if they wished to participate and were given the opportunity to ask any questions. Those wishing to participate in the study were provided with a participant pack providing information about the mechanism that they would be using to report on their ward. They were also asked to sign a consent form indicating their agreement to participate and asked some basic demographic information relating to them and their current hospital stay (age, sex, ethnicity, first language, date of admission). In addition, patients were asked to indicate (yes or no) whether they had been an inpatient within the hospital over the previous 5 years and whether they were receiving ongoing treatment at the hospital. From these two questions a ‘combined prior experience’ score was created, with a response range of 0–2, with higher scores indicating a greater amount of prior experience of the hospital. Participant information packs were also made available in Urdu and approaches to non-English-speaking patients were conducted by researchers who could speak Mirpuri and Urdu.
Capturing patient reports of safety concerns and experiences
Irrespective of the mechanism concerned, all patient reports were captured using the same set of questions. These were developed on the basis of a review of previous studies in which patients were asked to report safety events or concerns15,177 and then ratified by the working group (see earlier) during the process of developing the mechanisms from initial ideas to workable tools. It was agreed that open rather than closed questions would be used because of the potential for closed questions to limit the range of responses from patients, a previously identified criticism of this body of work. 15 Questions were designed to elicit information about a participant’s safety concern or experience in terms of (1) where it happened, (2) who was involved, (3) what happened and (4) why it may have happened. The questions can be found in full in Appendix 4.
Analysis
Categorising the content of patient-reported safety concerns and experiences
Categories were developed using an inductive and iterative process of thematic analysis. 252 This process involved two research nurses (CC, LT) and one researcher (JoH) first individually identifying the emergent themes from the patient reports and then meeting to achieve a consensus.
Classification of patient reports
All patient reports were reviewed using a two-stage process. First, reports were reviewed by two research nurses independently (CC, CR), to ascertain how many met the definition of a PSI, using the previous definition of a PSI by the NPSA, now adopted by NHS England, of ‘any unintended or unexpected incident which could have or did lead to harm for one or more patients receiving NHS care’. 253
All patient reports that were judged to meet this definition were then reviewed by two physicians using a categorisation process adapted from previous studies. 230,234 First, the reviewing physicians independently assessed each report to see whether it met the definition of a PSI. Those reports categorised as a PSI were then further rated against the standard risk indices of (1) preventability (using a 4-point scale, with 1 = definitely not preventable to 4 = definitely preventable) and (2) severity (using a 5-point scale, with 1 = negligible to 5 = catastrophic). Physician review was undertaken across two rounds. The first was an independent review, with a second review undertaken collectively to discuss any disagreements in scores. Following the second collective review, the inter-rater agreement was 85% for categorisation of a PSI. Of those reports categorised as a PSI, the inter-rater agreement was 77% for the likelihood of preventability and 75% for the degree of severity. Only those reports that were jointly agreed on as meeting the definition of a PSI as part of the collective second round physician review were taken forward into the analyses.
Statistical analysis
Quantitative data were analysed using a number of different methods. The difference between conditions (mechanisms) in the number of reports per patient was explored using analysis of covariance (ANCOVA) controlling for age, sex, duration of hospital stay and the combined degree of prior experience score. Significant between-subjects effects were explored post hoc using the Bonferroni test. There were four categorical dependent variables, namely (1) likelihood of reporting (i.e. whether the patient reported or not), (2) number of patient reports classified as a PSI (yes or no), (3) the degree of preventability in reports classified as a PSI (four categorical response options) and (4) the degree of severity in reports classified as a PSI (five categorical response options). To ascertain if there were differences between conditions in these variables, a series of chi-squared analyses were conducted.
This study was approved by the Bradford NHS research ethics committee.
Results
Sample
In total, 432 patients were approached and provided with an information leaflet about the study. Of these, 178 agreed to participate, equating to a response rate of 41%. The sample demographics are presented in Table 13 (two participants did not provide their demographic details).
Variable | n | % of sample |
---|---|---|
Age (years), mean (SD) | 58.14 (16.24) | |
Ethnic origin | ||
White British | 149 | 85 |
White other | 8 | 5 |
Pakistani | 12 | 7 |
Indian | 2 | 1 |
Other | 5 | 3 |
First language | ||
English | 165 | 94 |
Urdu | 4 | 2 |
Other | 7 | 4 |
Condition | ||
Face-to-face interviewing | 58 | 33 |
Paper-based form | 80 | 45 |
Patient safety hotline | 38 | 22 |
Specialty | ||
Vascular surgery | 20 | 11 |
Urology | 11 | 6 |
General surgery (male) | 25 | 14 |
General surgery (female) | 19 | 11 |
Renal, gastroenterology and rheumatology | 39 | 22 |
Respiratory, diabetes and endocrinology | 22 | 13 |
Cardiology | 19 | 11 |
Haematology | 7 | 4 |
Chemotherapy | 14 | 8 |
Contact with the trust during the previous 5-year period | ||
No previous inpatient stay or ongoing treatment | 28 | 16 |
Previous inpatient stay | 127 | 72 |
Receiving ongoing treatment | 128 | 72 |
Previous inpatient stay and receiving on-going treatment | 108 | 61 |
Number of reports per patient
There was a significant effect of condition (mechanism) on the number of reports per patient after controlling for the effects of the covariates (F2,154 = 8.292, p < 0.01). Post hoc tests showed that patients in the face-to-face interviewing condition provided significantly higher numbers of reports than those in the paper-based form condition [t(154) = –3.064, p < 0.01] and those in the patient safety hotline condition [t(154) = –3.781, p < 0.01]. The mean scores for the number of reports per patient are provided in Table 14.
Condition | |||
---|---|---|---|
Paper-based form | Patient safety hotline | Face-to-face interviewing | |
Number of patients | 80 | 38 | 58 |
Number of reports (total) | 70 | 16 | 108 |
Number of patients reporting one or more reports | 33 | 7 | 38 |
Number of reports per patient | |||
Mean | 0.92 | 0.43 | 1.91 |
SD | 1.645 | 1.144 | 2.238 |
n | 71 | 37 | 55 |
Likelihood of reporting
There was a significant association between the condition (type of reporting mechanism) and whether a patient reported one or more safety concerns [χ2(2) = 19.626, p < 0.01]. The contingency table is displayed in Table 15. In total, 64% of patients in the face-to-face interviewing condition reported one or more safety concerns, compared with 41% in the paper-based form condition and 19% in the patient safety hotline condition. Overall, nearly half (49%) of all patients reporting a safety concern were in the face-to-face condition. The standardised residuals indicate that fewer than expected patients reported a safety concern in the patient safety hotline condition and more than expected reported a safety concern in the face-to-face interviewing condition.
Condition | |||||
---|---|---|---|---|---|
Paper-based form | Patient safety hotline | Face-to-face interviewing | |||
Did patient report one or more safety concerns? | No | Observed count | 47 (58.8%) | 30 (31%) | 21 (21%) |
Expected count | 44.5 | 20.6 | 32.9 | ||
Standardised residuals | 0.4 | 2.1 | –2.1 | ||
Yes | Observed count | 33 (41%) | 7 (19%) | 38 (64%) | |
Expected count | 35.5 | 16.4 | 26.1 | ||
Standardised residuals | –0.4 | –2.3 | 2.3 |
Number of patient reports classified as a patient safety incident
No association was found between condition and the number of reports classified as a PSI. Table 16 displays the numbers of PSIs by type of reporting mechanism.
Condition | |||
---|---|---|---|
Paper-based form | Patient safety hotline | Face-to-face interviewing | |
Total number of reports | 70 | 16 | 108 |
Number of classified PSIs | 14 | 3 | 27 |
Preventability and severity of patient reports classified as a patient safety incident
Because of the numbers of patient reports classified as PSIs it was not possible to undertake the planned chi-squared analyses. Summaries of the preventability and severity scores by condition are detailed in Table 17.
Condition | |||
---|---|---|---|
Paper-based forma | Patient safety hotlinea | Face-to-face interviewinga | |
Number of classified PSIs | 14 | 3 | 27 |
Preventability | |||
Not preventable | – | – | – |
Probably not preventable | – | – | – |
Probably preventable | 2 | – | 1 |
Preventable | 9 | 2 | 20 |
Severity | |||
Negligible | 11 | 3 | 18 |
Minor | – | – | – |
Moderate | 1 | – | – |
Major | – | – | – |
Catastrophic | – | – | – |
Nature of patient-reported safety concerns
The categorisation of the content of the patient-reported safety concerns and experiences is summarised in Table 18. To further illustrate this, Table 19 provides typical examples of the patient reports from each of the five overarching categories.
Medication | General care | Immediate care surroundings | Communication | Service management | Other |
---|---|---|---|---|---|
|
|
|
|
|
|
Category | Examples |
---|---|
Medication | Reporting about themselves:Every time you [get discharged] it takes 8 hours to get the medications prior to dischargePatients are having to wait for medication because only a certain number of keys for drug cupboards. Patient having to wait in pain for too long. My pain gets worse thenParacetamol requested by patient and given Monday evening by a nurse. Paracetamol requested by patient and refused Tuesday morning by another nurse. Nurse explained that drug not prescribedI take Doxasozin 8 mg (2 × 4 mg tablets). I was given 2 × 8 mg tabletsReporting about another patient:A patient was awaiting medication (Herceptin) coming up from pharmacy before commencing treatment. Her husband was taking time off work to accompany her and needed to go back to work. Her stress levels were visibly increasing as the time went by (approx. 2 hours) of waiting |
General care | Reporting about themselves:Patient has two hospital numbers. She looked at the nursing records at the end of the bed. They stated that she was registered twice and had two numbers and that it was being looked intoNo wipes to wipe the toilet seats. Had been wipes available before, but there’s none now. This is an infection riskBooked for scan, lost paperwork so had to waitNurse came into the ward at midnight and woke me up to tell me I was being moved to another ward and would I pack my belongingsNebuliser I took over had not been cleaned out. Coffee all over it. Bugs and germs readily transferable. Reported to S/N [staff nurse], cleaned properlyLack of knowledge about diabetes. The nurses come round every morning at 6 a.m. to do blood sugars. This isn’t necessary as I administer my insulin at 8 a.m., my blood sugars could change significantly in those 2 hours. I check all my own sugars and manage my own diabetes, and continue to do so when I am in hospital |
Immediate care surroundings | Reporting about themselves:Toilets not clean. Patient had to clean them herself. Patient locker very dirty when first arrived. Nurse cleaned locker when asked toNot enough toilets and showers on ward 6. Have to wait a long time to have a showerReporting about another patient:A patient tried to light a cigarette on the ward. One of the nursing staff noticed straight away and stopped him luckily. Staff manage well but always very busy |
Communication | Reporting about themselves:I have been taking a particular tablet for over 1-years which had potassium in it. The doctors stopped it without discussing it with me. Since then my potassium has been low and I have had to take potassium replacementsI started being given an injection into my stomach – no one told me what it was for – it took 3 days for them to tell me. I have had other changes to my medication but have not been told why and whatUrine sample was left under the bed for a day and then thrown away. Patient told nurses it was there but they didn’t take it and then it had to be thrown away |
Service management | Reporting about themselves:Too few staff on the wardSecurity light shines through window into ward all nightOccasionally smoke comes in through the ward window from belowWard is day chemo unit. Is not big enough for the amount of patients using it. Badly needs expanding to help staff and patients |
Discussion
This study set out to test three mechanisms for capturing patient-reported safety concerns. Specifically, in relation to the research questions, it is clear that patients can report safety concerns using a variety of mechanisms but that the most successful means of collecting volumes of patient-reported safety concerns is through facilitated discussion at the bedside. However, it would appear that no mechanism differs significantly in terms of the number of classified PSIs collected. Finally, although we were unable to statistically examine the assessments of preventability and severity for classified PSIs, it is clear that, across all mechanisms, the PSIs were judged to be probably or definitely preventable and of low severity.
This study is the first of its kind to address a fundamental question at the heart of many patient engagement activities: ‘How should we collect information pertaining to quality and safety from patients?’ Therefore, our findings advance the literature by suggesting that embedding questions within a conversation with patients during their stay in hospital, held by someone external to the ward team, is likely to provide health services with the most information on patients’ experiences of care. Earlier work had begun to explore the likelihood of patients using various mechanisms for eliciting safety information,248 but this is the first known study to ask patients to actively use different mechanisms within a hospital ward setting. Within the context of the expansion of patient engagement activities nationally and internationally, it will be increasingly useful for organisations to consider our findings when making decisions about how to seek direct feedback from patients about the quality and safety of care.
The findings also raise a number of interesting issues. First, patients seem to be able to report on a wide range of patient safety issues. Although this result is in keeping with previous work in this area,177 we believe this is of note given that we provided little structure for patients about what safety events they might report, which was a departure from the approach taken in the literature to date. 15 It is also of note given the importance placed on engaging with patients as part of managing quality and safety within recent policy documents for the English NHS. 9,10,182 It has been argued by a number of authors that patients might be usefully regarded as the ‘smoke alarms’ for health-care organisations,209 with ‘their reports of safety issues . . . the best early warning system we have for detecting the point at which poor care deteriorates into care that is clearly dangerous’ (p. 16). 254 With this in mind, it will be important for those seeking feedback to resist the temptation to restrict patients to a defined set of response formats deemed important from a clinical perspective and to take the rather more patient-centred approach of letting them report care from the perspective of their unique ‘patient vista’.
Related to this, the second point of note is that there was no significant difference in the number of classified PSIs across the three mechanisms. However, given the lack of structure in the reporting format, it is perhaps unsurprising that we find a range of harm- and non-harm events across all reporting mechanisms. In essence, across all mechanisms, patients were central to decisions about what was important to them. It might be argued that the conversations at the patients’ bedside were simply better at drawing out greater volumes of safety concerns.
A third issue raised by the findings concerns the timing of seeking feedback from patients. Much of the literature to date has collected information from patients at or after discharge, with little attention paid to the real possibility of bias in the recollection of events that such a delay may introduce. 15 In testing mechanisms for asking patients for feedback about care during their hospital stay, we were in effect testing the possibility that patients can ‘hot report’, that is, spontaneously report to health services as and when they experience safety concerns. Therefore, one interpretation of our study findings might be that the prospect of true ‘hot reporting’ is perhaps less realistic for health services. The reality of the face-to-face discussions was that most of them happened at the point of consent, with few participants asking for a further interview. This means that, although this study provides a potential tool for systematically collecting patient-reported safety concerns, and provides the first known empirical evidence collected only within an acute care setting (not after discharge), it falls short of true ‘hot reporting’ whereby patients provide information as close to experiencing a safety concern as possible. The reality is that little is known empirically about the ‘optimum’ point at which patients should be asked for their feedback about quality and safety and the likely biases that may distort this information. It is now clear that further research is needed to answer these important questions, with the answers informing a wide variety of patient engagement activity across health services globally.
There are a number of strengths to this work. First, the study design meant that we tested each mechanism across different hospital settings and patient groups, thus allowing us to more confidently generalise our findings outside of the study population. Second, our approach to developing the ideas and structures for each of the mechanisms as a collaboration between patient representatives, health-care professionals and researchers means that the final versions were embedded within, and reflective of, the needs and constraints of each of these stakeholder groups. Third, the mechanisms were actively implemented within each of the participating wards for the study period and, as such, our outcomes are actual behaviours as opposed to the likelihood of engaging or willingness to engage with a potential mechanism. This is an important distinction given the known low correlations between intentions to undertake a behaviour and actually undertaking the behaviour. 249
Our work has a number of limitations. This study was carried out in one acute teaching hospital in England. Nevertheless, our sample of wards included a range of specialties covering medicine and surgery, as well as outpatient facilities. Second, patients were excluded from participating in the study if the ward team representative believed that factors such as their level of illness and level of mental capacity militated against their participation. This may have resulted in patients who are especially vulnerable to PSIs because of the complexity of their care255 and their relative inability to communicate256 not being given the opportunity to comment on their care. However, we argue that approaching patients who were judged to be especially ill or to lack the capacity to feed back their patient safety concerns may have placed an inappropriate physical or indeed mental burden on them, the position also taken by the research ethics committee that approved this study. Our subsequent work (see Chapters 7 and 8) now includes the option for patients’ families or carers to speak on their behalf if they are incapacitated in any way. Finally, the study hospital has one of the highest percentages of hospital admissions from a non-white, non-British population, reflecting the local demography. We recruited only 11% of this population and therefore the findings may not be representative of all sectors of the local patient population. Additional work in this area should consider the option of using a broad range of specially trained multilingual researchers to collect such data or, alternatively, creating patient-facing information technologies that facilitate the recruitment of such patients.
Conclusion
Previously published work in this area has been based on patients’ retrospective reports of safety-related information, the information in these reports having typically been generated using structured surveys. Our work heralds a significant departure from this approach. We have developed and tested a novel intervention, codesigned by patients, health-care professionals and researchers. Mindful of the tendency to locate PSI definitions in the clinical risk paradigm, we gave patients licence to define safety on their terms. We now know that hospitalised patients can report their concerns using a variety of mechanisms, but facilitated discussion at the bedside appears to be the preferred and most fruitful option. The diverse range of safety intelligence that patients can provide substantiates their position at the centre of care and their capacity to actively contribute to a more holistic view of risk.
Chapter summary
This chapter provides evidence about a key but previously unanswered issue: how best to collect feedback about safety. In doing so, it informs not just the further development of tools to collect the patient perspective on safety, but, more generally, it suggests that health services might be best exploring ways to speak to patients face-to-face rather than simply relying on passive methods of feedback collection. In addition, this chapter has explored how using an unstructured approach to soliciting feedback from patients can result in a wide range of safety-relevant data being sourced from an as yet untapped, but potentially crucial, perspective. The next chapter explores the potential role of patient feedback about safety as part of a range of error detection methods.
Publication statement
This chapter is based on a previously published paper. 257 We reproduce it here with permission from How might health services capture patient-reported safety concerns in a hospital setting? An exploratory pilot study of three mechanisms, O’Hara JK, Armitage G, Reynolds C, Coulson C, Thorp L, Din I, et al. ,257 2016, with permission from BMJ Publishing Group Ltd.
Chapter 6 Learning from error: a comparative study of patient-reported patient safety incidents and existing sources of patient safety data
Abstract
Background: Codesigned incident reporting tools make a novel contribution to the vanguard of incident detection methods. Established sources of data are known to vary in effectiveness, with several earlier studies demonstrating that different methods identify different incidents, and all have limitations.
Objectives: We examined the use of the patient incident reporting tool (PIRT). Data were collected face-to-face from hospitalised patients and compared with other established sources of safety data, the overlap was quantified and organisational implications were considered.
Method: Trained recruiters collected data from nine wards in a university teaching hospital spanning several specialties. Patient-reported concerns classified as PSIs were retained; we then searched for PSIs in the corresponding patient case notes, staff incident reports, complaints and reports to PALS specific to the study wards during the study period.
Results: In total, 155 patient reports were received from 77 patients; 68 PSIs were identified from consenting patients and eight of these were also identified from doctor case note review, five were identified in incident reports and two were identified through the PALS. Patient reports covered a range of events, from their immediate environment involving different personnel to the entire spectrum of care.
Discussion: Patients can and should contribute to the design of PIRTs. When hospitalised patients are asked about their care, they can provide a unique perspective on patient safety. Codesigned, patient-sourced safety reporting tools should be part of an integrated approach to gathering real-time patient safety information.
Chapter rationale
There is clearly a growing appetite to speak to patients about all aspects of care and a moral imperative to involve them in the assessment of risk and management of safety. However, much less is known about how the perspective of patients adds to the data on patient safety that we currently routinely collect within health services. The PMOS tool facilitates the patient perspective on the prospective assessment of risk and measuring of safety within hospital settings. As outlined in the previous chapter, the PIRT seems to collect a wide range of ‘retrospective’ information about patient-reported safety concerns, but the extent to which this overlaps with current sources of safety data is unknown. This chapter describes a study seeking to examine how, and if, patient reports of safety are duplicated across other patient safety intelligence collected in hospitals.
Introduction
It is clear that patients can report on patient safety problems specific to their care (see Chapter 4) and that reports can be collected from ill or debilitated hospitalised patients during their care (see Chapter 5). Existing methods for detecting what are often described as PSIs have well-documented limitations, case note review is not without bias258,259 and staff incident reporting is beleaguered by under-reporting. 170,260 These existing sources of data are conceived solely through a health professional lens and as such are unable to draw on an entire patient care experience; it is only the patient who has the potential to supply such a context. 218
Traditional approaches to collecting incident data also appear to be idiosyncratic. In a UK study of hospital prescribing errors, Franklin et al. 241 demonstrated that different methods identify different events. Weissman et al. 34 reached a similar conclusion from their multicentre study, using a random sample survey across 20 Massachusetts hospitals, as did Kaboli et al. 228 in a North American hospital study of medication misadventures, whose methods also included patient reports at discharge. A more recent multicentre study incorporating 14 hospitals in the Netherlands investigated the frequency and potential overlap of adverse events from case note review, incident reports, patient claims and patient complaints. 244 Once again, there was little overlap between methods and the authors concluded that staff incident reports neither were sensitive to adverse events nor have a positive predictive value. This paper focuses on the further development and implementation of a coproduced reporting tool for patients to provide feedback on their safety concerns while in hospital (described in Chapter 5) and compares these data with other routinely collected patient safety data. To our knowledge, this is the first study designed to allow such a comparison.
This chapter will specifically:
-
report on the use of the PIRT to collect patient safety data from patients resident in nine wards in a university teaching hospital
-
compare these data with three other established sources of PSI data collected retrospectively through case note review, staff incident reports and PALS
-
quantify any overlap between these various sources of data
-
discuss the implications for advancing patient involvement in patient safety and the organisational management of safety incident data.
Methods
The format of the PIRT intervention, further developed from the work described in Chapter 5, is shown in Box 1.
Have you spoken to a member of staff about your safety concern?
Which category best describes your safety concern?a
-
Medication issue
-
General care issues
-
Care surroundings
-
Communication
Please tell us where your safety concern or experience happened?
Please tell us who was involved in your safety concern or experience?
Please tell us what happened with your safety concern or experience, in as much detail as you can?
Please tell us why you think your safety concern or experience may have happened?
On a scale of 1–10, how serious do you think your concern was (1 being the least and 10 being the most serious)?
Do you think it would have been possible to have stopped your experience from happening (5-point scale)?
What do you think could be done to stop this from happening again to you or other patients in the future?
The categorisation was based on an inductive analysis of PIRTs from an earlier pilot study; the four categories of safety concern data, however, played no part in the analysis here.
Setting and sample
This study was carried out between 5 September and 29 November 2011 in a university teaching hospital in northern England. The total number of patients to be recruited was driven by the PMOS, an associated intervention necessitating a specific sample size to meet predefined requirements for factor analysis. There were nine study wards, each of which were recruited through liaison with senior managers. The wards included three medical wards, three surgical wards, two maternity wards and a paediatric ward. In the last ward parents were able to report concerns specific to their child’s care.
Recruitment, measures and procedure
Data were collected by a team of trained recruiters, which consisted of research nurses, an operating department practitioner and research fellows, all specifically employed to work on this research programme. They were each allocated to collect data on specific wards; however, there was some minor variation in allocation because of recruiter sickness or absence. PIRT data were collected from all those patients who had disclosed and who were willing to disclose a concern about their safety and who had also completed the PMOS questionnaire described earlier. The team used the PIRT pro forma via face-to-face patient interview in the study wards (see Box 1). The research team visited each ward once a day from Monday to Friday over the study period. They first asked the ward manager’s permission to visit all resident patients except those who were very poorly, who were contagious or who lacked capacity to consent. Following permission, each patient considered appropriate to be approached was invited to participate. Those patients wishing to participate were provided with an information pack before being consented. We did not include patients who were unable to speak English. We excluded all non-reports from the comparative process, that is, those reports that did not include an identifiable safety-related concern.
We reviewed case notes for those patients on the study wards who had submitted patient safety concerns, which for the purpose of this study were defined by the former NPSA definition, now used by NHS England,253 as ‘any unintended or unexpected incident which could have or did lead to harm for one or more patients receiving NHS care’. We did not review case notes for those patients who did not consent to review or whose notes could not be accessed by the research team because of ongoing hospitalisation. To reduce any expectation bias, the review team were informed that the case notes had been randomly selected from all patients in the study wards in the study period, not only those who had submitted reports. Additionally, case notes for patients who had submitted reports were not allocated for review to the research team member who had collected the PIRT submission; this avoided the reviewer associating any memory of the patient report with the patient’s clinical profile.
The case notes were reviewed for the admission that coincided with the PIRT submission and covered all aspects of care up until the point of discharge from hospital. The review process was informed by a holistic, two-stage model of case note review, originally used in the Harvard Medical Practice Study239,261 and now routinely employed in clinical audit in the UK. This incorporated two stages: the stage 1 review was carried out by four members of the research team (CC, CR, LT and SM); if a PSI(s) was identified, a stage 2 review was carried out by a medical practitioner. The stage 2 team included five medical staff, four of whom were seconded to the quality and safety research team at the Bradford Institute for Health Research, and one anaesthetist from the study hospital who was not associated with any of the study wards. The stage 1 reviewers did not inform the stage 2 reviewers of the PSIs that they had identified.
Staff incident reports submitted from the study wards over the period of recruitment and data collection were collated and analysed retrospectively via an electronic reporting system. Data from PALS were supplied by the department concerned; there were no patient complaints received from the study wards over the recruitment period.
For the patient reports, two reviewers (CR and SM) individually identified PSIs before forwarding those patient reports with PSIs to two medical reviewers, who then confirmed or did not confirm agreement about the existence of PSIs. Two reviewers (CR and SM) counted PSIs from all sources and cross-referenced between the sources for agreed overlap.
Analysis
To compare the safety incident data from the various data sources with patients’ reported concerns, we matched dates (when possible) and patient identity through use of a specially created patient identity number (PIN) that corresponded with the patients’ unique Patient Administration System number. CR and SM compared all data sources and if there was an agreed overlap between sources this was recorded.
All patients in the study wards were consented before participating in the study. A NHS research ethics committee approved the study.
Results
We recruited 329 patients to the study, which thus allowed completion of the PMOS factor analysis; however, some patients who were recruited were subsequently unable to participate because of clinical interventions or a change in their condition. In total, 155 patient reports were received from 77 patients in the study period. Table 20 shows the numbers of patients recruited across wards who submitted reports and some of their characteristics. As the case notes of three patients could not be obtained and four patients did not give consent to access their case notes, a sample of 70 patient case notes was reviewed at stage 1 and 58 at both stages. Five case notes were not reviewed at stage 2 because the notes were not available. Figure 5 provides a flow chart of the case note sampling process.
Ward type | Total number of patients recruited per ward | Number of reporting patients submitting at least one concern (% of recruited patients) (total n = 77) | Median length of patient stay for those submitting a concern(s) (days) (IQR) | Number of reporting patients with a single concern (% of recruited patients) (total n = 38) | Number of reporting patients with multiple concerns (% of recruited patients) (total n = 39, range 2–8) |
---|---|---|---|---|---|
Medical ward 1 | 37 | 7 (18.9) | 5 (3) | 5 (13.5) | 2 (5.4) |
Medical ward 2 | 43 | 18 (41.9) | 8.5 (16) | 9 (20.9) | 9 (20.9) |
Medical ward 3 | 36 | 6 (16.7) | 10.5 (4.75) | 1 (2.8) | 5 (13.9) |
Paediatric ward | 28 | 5 (17.9) | 5 (1) | 3 (10.7) | 2 (7.1) |
Surgical ward 1 | 65 | 16 (24.6) | 8.5 (6.5) | 6 (9.2) | 10 (15.4) |
Surgical ward 2 | 54 | 12 (22.2) | 5 (11.25) | 7 (13.0) | 5 (9.3) |
Surgical ward 3 | 37 | 8 (21.6) | 11.5 (9) | 4 (10.8) | 4 (10.8) |
Maternity ward 1 | 17 | 2 (11.8) | 10 (7) | 1 (5.9) | 1 (5.9) |
Maternity ward 2 | 12 | 3 (25.0) | 2 (1.5) | 2 (16.7) | 1 (8.3) |
Patient reports
Analysis of each patient narrative revealed a wide range of patient concerns. Box 2 provides a range of illustrative examples of patient concerns that were reported only via the PIRT and, as can be seen, these emanate from observations of the immediate environment, including other patients’ care, the activities of the entire health-care team, from consultants to cleaners, and of course the impact of patients’ particular experiences. It can also be seen that patients appear especially interested in standards of hygiene and lack of attention to detail. Box 3 shows patient concerns that were also evident in other sources of patient safety data.
Before seeing the neurologist I was given a fairly confident diagnosis of intracranial pressure based on a CT [computed tomography] scan, which caused 24 hours of worry and stress. The neurologist contradicted this and felt it wasn’t the case at all. I felt I was given a wrong diagnosis by a non-specialist.
I went to the toilet at 7.30 a.m. and there was blood on the toilet seat. When I went again at lunchtime it was still there. I informed one of the nurses after I had visited the toilet at lunchtime about the problem. I observed the cleaners having finished cleaning at around 2 p.m.
The toilet doesn’t flush properly, sometimes you have to wait a long time for it to fill up, the pressure is too low, sometimes you go in and it hasn’t been flushed.
Nasal cannulae have been on the floor but the nurses have picked them up and used them again on the patient. Patient is using own TCP to clean the nasal cannulae.
There are three doors across the corridor from where my bed is which lead into an area that only staff go into. They bang loudly and constantly, it’s worse on a night as it keeps me awake, and it makes me jumpy and feeling nervy.
I haven’t got a cupboard for my clothes. My table has been missing for a few days, I have to eat my dinner off my bed.
My ex-husband came to see me on the ward, I haven’t seen him for some years. He told me that he was visiting someone on another ward and a member of staff told him where I was. I didn’t want him to know anything about me and I didn’t want him visiting me.
The porter that took me for a scan did not gel his hands on entering and leaving different departments/wards.
Nurse doing medication round dispensed three tablets instead of four. Aspirin was missing. The nurse went and checked, it wasn’t prescribed on the drug chart. This is a vital medication for my condition.
She had been screaming out in pain and been ignored. When she requested analgesia it had not been given until hours later.
PALS (comment received from patient’s spouse while patient hospitalised)
Advice from pain consultant not followed or queried with them so patient didn’t get analgesics.
. . . seen by multiple people from pain team who left good advice in the notes that the clinical team never acted upon.
Case note review (doctor)
During the night I was in a lot of pain. I was prescribed 25 mg pethidine and paracetamol, it wasn’t working. I asked if they could get a doctor. They said they couldn’t get one until the morning. I felt like I was going to die and I wanted to go home.
Patient report
He [patient] keeps getting told, day by day that the procedure would be done but for various reasons it kept getting delayed.
PALS
Surgery delayed because of high INR [international normalised ratio]. Patient report: I was seen in outpatients on Monday, told to be admitted on Wednesday. No one picked up that I was on warfarin, which should have been stopped. This has prolonged my stay for 3 days and taken up a bed unnecessarily.
Case note review (doctor)
Evidence of the reporting of PSIs from multiple sources was very limited. Table 21 shows the overlaps. Only 11 patient-reported concerns were also evident in other sources; of these, four were evident in two or more sources. Case note review yielded a total of 1055 incidents. Staff incident reports and information about safety concerns reported by patients to PALS generated just 24 incidents, with no overlap.
Method of reporting | Doctor case note review: PSI identified | Patient-reported PSIs | Staff incident reports | PALS |
---|---|---|---|---|
Doctor case note review: PSI identified | 1055 | 8 | 11 | 2 |
Patient-reported PSIs | 68 | 5 | 2 | |
Staff incident reports | 19 | 0 | ||
PALS | 4 (one report to PALS included two PSIs) |
Box 4 shows examples of PSIs considered documentation errors. However, it was difficult to determine whether some PSIs were documentation errors or not and examples of these are shown in Box 5.
Incomplete/missing data:
No list of discharge medicine in discharge letter, despite writing that some drugs were started during the admission.
Incorrect terminology/spelling error:
Drug chart Enoxaparin prescribed by brand name.
Illegible entry:
Illegible and untimed consultant notes entry.
Incorrect entry:
Multiple drugs prescribed as regular drugs but with conflicting routes of administration (PO [per os] and IV [intravenous] for aspirin?).
Red allergy sticker on front of notes but all notes record ‘NKDA’ [no known drug allergies] inside.
Incomplete Acute Coronary Syndrome proforma: no evidence of physical examination or drug history.
No input from outreach or ICU [intensive care unit] ever seen in the notes despite multiple high MEWS [Modified Early Warning Score]. No documented decision not to refer to ICU documented either.
Third dose of dextrose/insulin was different in that it was with 15 units of insulin. No explanation as to why dose different in notes.
Consented for LSCS [lower-segment caesarean section] & sterilisation (both consent forms correctly completed) however, no handwritten notes found. Computer summary of operative delivery records ‘condition of fallopian tubes Normal, Condition of ovaries Normal’. Was patient sterilised? Has GP [general practitioner] being informed?
Multiple tests mentioned as requested, (abdo X-ray, urinary amylase, repeat LFT [liver function tests]) but not written in notes. Were these even done?
Discussion
This single-centre, prospective cohort study has demonstrated that patients in an acute hospital setting, and resident on a range of clinical wards, can report their perceived safety concerns when interviewed by staff who are independent of the ward team. These concerns are typically focused on events that occur in close proximity to the patient but which are unlikely to incorporate harm. The same concerns are rarely evident in other patient safety data. This is the first study to compare data from spontaneous patient reporting of safety concerns while receiving care with other sources of patient safety data.
Profile of patient reports
There was a variation in the number of patient reports by ward and specialty. Length of stay and complexity of care may be influential, as has been seen in adverse events studies in critical care settings,262 with patients being more likely to experience threats to their safety as a direct consequence of experiencing more interventions. We were unable to examine any association between either length of stay or care complexity for those patients who submitted more than one report. On the basis of our observations when visiting the study wards to recruit patients, we can add that those patients who were receiving the most complex care were also likely to be the most ill. Consequently, we did not collect data from these patients as this would, understandably, have violated the conditions of our ethical approval.
The content of patient reports is inevitably a reflection of care through the lens of patients’ experience and self-knowledge and many aspects of their experiences may be relatively invisible to clinical staff. 207 It would appear that patients have intently watched not only how care is given by the entire staff team but also the different standards of attention given to particular patient needs and their consequences. Whether patients were reporting concerns about their own care or concerns about the care of others, we argue that the content often reflected the possible precursors or antecedents of future PSIs, which possess an intrinsic learning value. A good example of this is the porter (see Box 2) who apparently did not ‘gel his hands’ prior to patient contact. However peripheral porters may be to clinical procedures, they routinely handle large numbers of patients, within and across wards, and possess considerable potential as a vector for infection. Conversely, many reports might also be judged by some staff as ‘soft issues’, especially those who conceptualise patient safety through a (clinical) outcome-driven lens. 207 Indeed some of the examples illustrate that patients’ reports could be described as perception driven and, consequently, they are affected by individual patients’ values, expectations and of course knowledge of their care and treatment. Furthermore, they can be contentious, as seen in the first report in Box 2. Although it is therefore accepted that there are limitations in the content validity of patient reports, it is also clear that some patients use this novel medium to identify phenomena that paint a vivid picture of safety in their care environment and which would either be causing distress or be likely to contribute to an eventual harm, albeit not often imminently. In summary, patient-reported concerns are centred on what has been called the ‘whole sequence of care’. 244 As indicated earlier, it is apparent that such phenomena may not be recorded in other sources of patient safety data.
Overlap of definitive patient safety incidents between different patient safety data sources
We found a distinct lack of overlap across different methods of collecting safety data, with different safety data sources identifying different PSIs. Our findings reflect those from several other studies that have compared the overlap between staff-based sources of patient data, such as incident reporting and case note review,241,243,260,263 but also those that have added patient sources to the sampling frame. 228,244
In this study we did not find any event that was evident in all data sources but there was some overlap in PSIs that had been identified through case note review. Although our findings confirm those from earlier studies that case note review is likely to provide the greatest number of PSIs compared with other established methods such as incident reporting,241,260,263 it may also be an indication of the multidisciplinary detail available in case notes and the systematic process that we applied to the process of analysis, one that was uniformly carried out by research staff. The total number of PSIs (n = 1055) identified from the second-stage case note review included a large number of documentation errors, which in many cases may not have led to any form of patient harm (e.g. incorrect terminology as defined in Box 4), although this cannot be certain because we found that the majority of documentation errors did not have any consequences recorded in the case notes. More importantly, however, the overlap between the various forms of PSI data was extremely small. This study has, however, further confirmed that the root causes of harm events are unlikely to be evident in either case notes,264 which record clinical information (measurable signs, symptoms and biometric data), or incident report data. 170
The paucity of the PALS data is significant. This may be a reflection of patient awareness about the service and its purpose but also that contacting PALS is something that must be instigated by patients or their relatives, unlike the PIRT process, which was brought to the patient and facilitated by staff (often with a clinical background) but independent of the study hospital.
Implications for patient safety and organisational learning
This study suggests that organisational learning can be enhanced by gathering patient-reported concerns about their safety and that the content of these reports varies greatly from PSIs identified from case note review, incident reports or other forms of institutionally sourced data. These concerns might also act as a very early warning of forthcoming harm events and, we argue, if part of an integrated analysis based on the full range of safety data sources available, may shed further light on the root causes of incidents that have already occurred. We believe that the first step in realising any potential is through asking patients the right questions at the right time for patients and having trained independent personnel to ask these questions. In this study both the questions and the approach to data collection were developed with patients and ratified by staff to be complementary to their existing means of gathering patient safety intelligence. Coproduction is an ideal method for shared decision making265 and systematically collecting and applying the patient perspective; coproduction equally recognises that a parallel staff perspective is crucial for successful implementation. The health-care staff involved in the PIRT development were able to hear the patient view outside of the frantic environment of practice and question that view without creating tension. We argue that this reduced the likelihood of the intervention being perceived by the greater body of staff as a top-down initiative borne out of an inappropriately idealistic, institutional drive to unconditionally prioritise the patient voice. However, the second step in realising the potential of patient reports is involving patients and staff in their analysis and the subsequent application of this knowledge in the risk management or clinical governance process. This remains a considerable challenge. Furthermore, the study described here was generously funded by a national research institution. In the current financial climate, it is hard to foresee health-care providers being able to implement such an initiative without additional finance unless those collecting these data are volunteers. We are currently investigating the feasibility of implementing this intervention (in tandem with PMOS) across three other acute health-care providers in the UK.
Limitations
This was a single-centre study but one that included a wide range of patients from different specialties. We were unable to recruit all of the patients from the wards during the study period, which would have provided a more reliable estimate of the number of PSIs perceptible to all patient admissions in a given ward in a specific time frame. Although we have confirmed that patients can report their concerns, based on our experience we contend that the volume of patients recruited is also likely to be an indicator of how comfortable UK patients currently feel about formally commenting on their care during its provision. Patients may well have experienced a PSI after having had the opportunity to provide data through the PIRT as they were asked on only one occasion if they had experienced a PSI. We did not review the case notes for all patients on the wards in the study period, which would have allowed identification of those PSIs not identified as concerns by patients (or at least by those who had consented). There were also some difficulties in accessing all staff incident reports through the existing electronic reporting system.
Conclusion
There is a growing evidence base which demonstrates that patients can, when asked, report on their safety and that of other patients. It is also apparent that these data and other sources of routinely collected patient safety data have remarkably little overlap. This study provides an important addition to this evidence base but also shows that patients can work effectively with staff to develop PIRTs that can then be used to gather patient feedback while they are receiving care. The collection of this type of patient-reported data requires a sensitive approach and one that is likely to have resource implications for health-care providers. The importance given to the patient voice, however, demands that health-care providers have a systematic means of collecting these data and ideally one that has been rigorously tested. The content of these reports can collectively provide a unique perspective on the overall process of care from the only participants who are constantly present and who can, through their ongoing experience, identify the antecedents of PSIs and the incidents themselves. Their purpose is to add another, as yet, largely invisible source of intelligence about patient safety as part of what must be a fully integrated, multimethod approach to organisational learning.
Chapter summary
The findings from this chapter suggest that speaking to patients about their experiences of safety may well provide health services with an, as yet, untapped perspective on error detection. However, as with all measurement within health care, this information will be of use only if it is somehow interpreted and acted on as part of ongoing service improvement. The next chapter outlines the bringing together of the PMOS survey and the PIRT to form the basis of a patient-centred patient safety intervention. With patient feedback about safety at the heart of this intervention, we move these two tools from simply measuring safety to being the foundation for patient-centric service improvement.
Chapter 7 Assessing risk and learning from error: the Patient Reporting and Action for a Safe Environment intervention – a feasibility study
Abstract
Background: Patient feedback is increasingly of interest to health-care organisations. No known intervention exists, however, allowing the systematic capture of the patient perspective on safety of care in acute settings as a basis for service improvement. This chapter presents a study to assess the feasibility of a patient-centred patient safety intervention, with three research aims: (1) to explore the feasibility of systematically collecting feedback from patients; (2) to explore the feasibility and acceptability of the intervention for staff and how staff use patient feedback for service improvement; and (3) to explore the feasibility of collecting feedback from staff about the safety culture.
Method: This feasibility study used a wait-list controlled design across six wards within an acute teaching hospital. Intervention wards were asked to participate in two cycles of the Patient Reporting and Action for a Safe Environment (PRASE) intervention across a 6-month period. Participants were patients on participating wards and ward staff completing safety culture surveys.
Results: In total, 379 patients were recruited, with 199 staff returning completed safety culture questionnaires. Findings indicate that the PRASE measurement tools can collect the patient perspective on safety. Ward staff were able to use the feedback as a basis for action planning, although engagement with the process was variable. Ward staff were also able to provide feedback about the safety culture on the ward, although response rates were inconsistent. Recommendations are discussed for amendments to the intervention.
Discussion: The PRASE intervention was found to be acceptable to staff and patients and feasible for further testing as part of a future cluster RCT.
Chapter rationale
A key step in the development of complex interventions is testing the feasibility and acceptability of the content, as well as the potential design of future studies aiming to test their efficacy. This chapter describes the development and feasibility testing of the Patient Reporting and Action for a Safe Environment (PRASE) intervention. This intervention uses the two measurement tools developed in Chapters 2–6 but puts them central to an innovative patient-centred PSI.
Introduction
The perspective of patients has never been more central to the way that we deliver health care both nationally and internationally. Within the English NHS a series of recent high-profile reports have all emphasised the need to elicit, understand and respond to the patient voice about the experience and safety of care. 9,182,206 This focus has been mirrored internationally, with Partnering with Consumers in Australia [see www.safetyandquality.gov.au/our-work/patient-and-consumer-centred-care/national-safety-and-quality-health-service-standard-2-partnering-with-consumers/ (accessed 12 July 2016)] and Better Together in the USA [see www.ipfcc.org/advance/topics/better-together.html (accessed 12 July 2016)].
Traditionally, the way that health services asked for patient feedback was to focus on the quality of care, patient satisfaction and experience. This is clearly hugely important to ensure high-quality care, but over the past decade researchers and practitioners alike have begun to understand how patients may also provide useful information to health-care organisations about the safety of their care. 7,15,181,207,212 The evidence base is building regarding the ability and willingness of patients across a variety of health-care settings to report specific safety concerns or PSIs. 15,177 However, what have been lacking in the literature to date are concrete interventions that allow staff to use patient feedback about safety within the context of service improvement. That is, we know that we can measure patient concerns about specific safety issues, but we do not know how or if these may be used to improve safety, or indeed services more widely.
In addition, the focus of most empirical work has been the reporting of specific safety concerns or PSIs,266 which can be conceptualised in terms of human error theory as being ‘retrospective indicators’ of patient safety. What has not been considered is whether patients may also be in a position to report on factors within the health-care environment that contribute to future error – ‘prospective indicators’ of patient safety. Recent work has aimed to address this, with the development of the PMOS tool,181,212 which allows the systematic and routine collection of patient feedback about contributory factors to PSIs. With this tool, health-care organisations can move beyond the measurement of past ‘harm’, which has been a recent criticism of the way that safety is currently measured,266 to a more forward-thinking position whereby patients are integral to the proactive assessment of patient safety.
This chapter describes the development and feasibility testing of the PRASE intervention. This intervention represents an innovative patient safety approach for acute health-care settings that puts patient feedback about the safety of their care central to service improvement. The intervention uses two tools to ask patients about their experience of care within a hospital setting, sampling both retrospective and prospective indicators of patient safety. This information is then fed back to health-care professionals on hospital wards to be considered within an action planning process, with the aim of improving patient safety. This paper explores the initial process of developing the intervention from the two tools, followed by analysis of how the intervention worked in practice in four wards in a large acute teaching hospital.
The revised Medical Research Council (MRC) guidance for complex interventions267 outlines a systematic approach for their development and evaluation, with an emphasis on feasibility testing and piloting. The PRASE intervention clearly meets the definition of a complex intervention, with a number of interacting components and a variety of behaviours required across a number of groups and organisational levels, resulting in localised tailoring of the intervention and the outcome measures used. 267 This feasibility study allows the testing of procedures, exploration of the acceptability of the design and estimation of recruitment and retention prior to testing within a wider efficacy cluster RCT (see Chapter 8).
Specifically, therefore, we had three key research aims:
-
to explore the feasibility of the process of systematically collecting feedback from patients about the safety of care
-
to explore the feasibility and acceptability of the PRASE intervention for staff as part of a randomised trial and to understand more about how staff use the patient feedback for service improvement
-
to explore the feasibility of collecting feedback from staff about the culture of safety on participating wards.
From these questions we would be able to generate findings that support refinements to the PRASE intervention, processes for collection of associated measures and development of a programme theory to support a wider cluster RCT.
Developing the Patient Reporting and Action for a Safe Environment intervention
Development of the intervention
The PRASE intervention has two measurement tools at its core: the PMOS tool and the PIRT. The development of these two measurement tools has been described in previous chapters. However, to form an intervention, these tools needed to be part of a practical process that included the receipt of patient feedback about safety by ward teams and a means by which they might take action. To achieve this, we formed an intervention development group that included a number of managers responsible for patient safety and experience from the region, medical and nursing professionals, patient representatives and the research team. This group met four times over 4 months between January and May 2012. The remit of the group was to identify and explicate how the measurement tools might be used as a basis for an intervention focused on using patient feedback about safety to improve ward-level patient safety and how this would be implemented within acute NHS trusts.
The Patient Reporting and Action for a Safe Environment intervention
Under the guidance of the intervention development group, the PRASE intervention was developed as a cyclical improvement process, which used as its base the patient perspective on the safety of care. Figure 6 presents the outline cycle for the intervention.
The Patient Reporting and Action for a Safe Environment intervention process
The intervention is based on a traditional action planning process, but crucially, as stated, using the patient perspective on the safety of care as the basis for improvement. The PRASE intervention starts with measuring the patient experience of safety on a ward. This information is then collated and fed back to ward staff in the form of a feedback report, the content of which is discussed in a multidisciplinary action planning group (APG). Within the intervention development group it was felt for a number of reasons that the membership of the APG needed to reflect the different professions that deliver care on a ward. First, it was felt that, to fully understand the nature of the patient feedback, multiple perspectives are required. Second, members of the intervention development group wanted the intervention to be seen as more than simply a nurse-led endeavour, but rather as a ward-level improvement approach. Third, to be effective, any actions agreed on would also benefit from multiple perspectives on care delivery, as well as support from different professional groups. The last key facet of the intervention is that it is a cycle in which the process of measurement, feedback and action planning is repeated. The prevailing view of the intervention development group was that the cycle could be achieved within 3 months and this therefore formed the basis of this feasibility study design.
Methods
Design
The study used a wait-list RCT design (Figure 7). A randomised design was used to test the feasibility of randomising wards (including impact on the recruitment of participants) and the acceptability of randomisation to a wait-list control group. To allow focus on intervention fidelity this randomisation was in a 2 : 1 ratio of intervention to control wards. Outcomes were measured at three points (baseline, 3 months and 6 months). Intervention wards received the PRASE intervention, with two 3-month cycles of patient feedback and action planning. Patients on control wards were also measured at three time points but these data were not fed back to staff until the end of the study. Data were collected between July 2012 and March 2013. Wards were randomly assigned to condition using a random number generator.
Setting and participants
This study was undertaken in a large acute teaching hospital in the north of England. Six wards were purposively sampled from the trust to represent a wide range of patient demographics, clinical specialty and type of admission (acute/elective). Specialties included in the study were (1) paediatric surgery, (2) ear, nose and throat surgery, (3) medical admissions, (4) orthopaedics, (5) renal, gastroenterology and rheumatology and (6) urology. There were two participant groups in this study: patients on participating wards and staff on participating wards.
Procedure
The Patient Reporting and Action for a Safe Environment intervention
Patient feedback about the safety of their care
Recruitment of patients within participating wards took place in a staggered series of 2-week blocks across each 6-week data collection period. Recruitment proceeded within two wards in the first 2 weeks, followed by another two wards in weeks 3–4 and the final two wards in weeks 5–6. The process of recruiting a patient started by asking staff on the ward to identify those patients who were unable to participate. Exclusion criteria included being too ill to talk to the research team (excluding paediatrics, where relatives were able to take part on behalf of the patient) or not having capacity to consent (as judged by a senior nurse on the ward). A member of the research team then approached those patients who were eligible to participate (or their relatives if the patient was aged < 16 years) and asked if they would be willing to complete a brief questionnaire about safety. Once consented, patients’ demographic information was recorded and the PRASE measurement tools were completed. All data were captured on computer tablets at the bedside, using software developed by the research team (the software was developed by software developers employed at the Bradford Institute for Health Research, Bradford, UK). This software allowed the direct input of data and secure transfer to a database using an internal Wi-Fi connection within the hospital. Participants were asked if they would like to complete the PMOS questionnaire themselves, with the software recording completion status on the database. Although participants could self-complete, the default for collection of these data was through a facilitated discussion with a researcher, as this has been found to be the best means of collecting this type of information from patients (see Chapter 5). 248
When completion of the questionnaire was facilitated, the researcher went through each of the questions on the PMOS questionnaire in turn, noting the participants’ preferred response. When patients volunteered other information to provide context for their answers, this could be recorded in three ways: (1) as a general comment, (2) as a positive experience of care or (3) as a ‘safety concern or experience’ through the PIRT. If no positive experiences of care or safety concerns were volunteered during the completion of the PMOS questionnaire, the researcher prompted the participant at the end of the process to identify whether they would like to volunteer any information at that point.
When participants identified a ‘safety concern or experience’, the researcher recorded this using a series of prompts, asking patients to consider what happened, why it was a safety concern for them and what could be done to prevent it happening again, as well as their perspective on the preventability and severity of the experience. The full list of prompts, response options and categories is outlined in Appendix 5.
The research team aimed to collect between 20 and 30 responses per ward over a 2-week period. It was anticipated that this would provide a robust representation of the patient experience of safety on the ward, be feasible within the 2-week time frame and provide an adequate amount of feedback on which to base the action planning process.
The Patient Reporting and Action for a Safe Environment feedback report
All feedback was collated and fed back to ward staff. Wards in the intervention group received patient feedback at three time points: after baseline, after 3 months and at the end of the study (6 months). Wards in the control group were provided with feedback from all three time points at the end of the study. Feedback reports were created for each ward for each time point. The style and content of the feedback report was developed through discussion with staff and our patient representatives. An example feedback report is provided in Appendix 5. PMOS scores were represented for each domain, with each domain then broken down into its composite items. A traffic light system was used to allow staff to quickly identify where positive and negative responses had been received from patients. Alongside each PMOS domain were general comments relating to questions for that domain and positive experiences of care or safety concerns that had been categorised by researchers as related to this domain. This allowed staff to consider the PMOS scores in conjunction with the relevant free-text information from patients.
Action planning
Intervention wards were asked to create a multidisciplinary APG within which the feedback could be considered and action plans made. The APG nominated someone from the group to write up and take responsibility for the action plan, using a standardised action planning pro forma supplied by the research team (see Appendix 6). The APG was asked to provide the research team with a copy of the completed pro forma, to evidence the outcomes of the meeting and allow enquiry about implementation of the actions at the end of the 6-month study period. Each APG meeting was attended by a member of the research team, to observe the action planning process.
Exploring the feasibility of the Patient Reporting and Action for a Safe Environment intervention
Research aim 1: exploring the feasibility of collecting feedback from patients about the safety of care
This research aim was explored through a detailed analysis of the data generated by the PRASE measurement tools. The focus of this feasibility study was not on formal comparisons between the intervention group and the control group. Rather, we were concerned with (1) exploring recruitment of participants and the impact of randomisation and (2) examining the data for variation across wards and time points.
To examine the recruitment of participants, participant demographics were calculated across participating wards and confidence intervals (CIs) for the response rates were calculated across intervention and control wards. To examine the variation in the data generated by the PRASE measurement tools, an overall mean PMOS score together with nine domain scores were calculated using the mean of two or more responses. In addition, summary scores were calculated for the item ‘Were you treated with dignity and respect?’, which was regarded as a stand-alone item, not grouped within a domain, but presented to health-care professionals within the feedback report. Patient-reported safety concerns were examined by calculating for each ward (1) the total number of patient-reported concerns, (2) the number of patients reporting one or more safety concerns, (3) the mean number of reports per patient, (4) the mean severity of reported safety concerns, (5) the range of the patient-assessed severity of reported safety concerns and (6) the average level of patient-assessed preventability of reported safety concerns (expressed as the mode as it is a categorical variable).
Research aim 2: exploring the feasibility and acceptability of the Patient Reporting and Action for a Safe Environment intervention process for staff
To explore the feasibility and acceptability of the PRASE intervention as a basis for service improvement and to understand more about how staff use the patient feedback, three qualitative methods were used. First, research staff made structured observational notes about each APG meeting that they attended. Second, follow-up telephone interviews were held with the appointed PRASE lead for each of the intervention wards. Finally, members of the research team recorded their impressions and tacit knowledge about the study through fortnightly team meetings, which were minuted.
Data from the three qualitative methods used (observational notes, telephone interviews and recording of tacit knowledge) were each subjected to a thematic analysis268 and then synthesised. Our coding framework related to directly addressing research aim 2 (exploring the acceptability and feasibility of the PRASE intervention). Therefore, coding specifically pertained to issues of acceptability and feasibility to address our set research questions. The combined results from the quantitative and qualitative analysis described above were then triangulated and considered within the research team with the aim of directing refinements to the PRASE intervention and to specify precise details of the design of the wider cluster RCT.
Research aim 3: exploring the feasibility of collecting feedback from staff about the culture of safety
Health-care professionals’ perceptions of safety culture are often measured as an outcome of patient safety interventions. 269,270 It was anticipated by the research team that safety culture data would be collected as a primary or secondary outcome for the cluster RCT. Therefore, it was important to test the process of collecting feedback from staff about the safety culture on their ward as part of this study. Perceptions of safety culture were assessed at two time points, baseline and 6 months, using the Hospital Survey on Patient Safety Culture. 194 Specifically, there were three ‘outcome’ measures: (1) perceptions of patient safety (four items, from 1 = ‘strongly disagree’ to 5 = ‘strongly agree’), (2) patient safety ‘grade’ (one item, from 1 = ‘failing’ through to 5 = ‘excellent’) and (3) frequency of events reported (three items, from 1 = ‘never’ through to 5 = ‘always’). Respondents were also asked to provide information on their job title and years spent on the ward.
Named surveys were prepared for each member of staff using staff lists collected from the ward prior to the recruitment period. When research nurses began recruitment of patients they met with the senior nurse on each ward and asked them to distribute the surveys to staff and encourage completion. Staff returned questionnaires using drop boxes, which were collected from the ward no longer than a month after patient recruitment was complete. This was to ensure that feedback from staff and patients was collected during the same time period. To allow for the collection of data from staff who were not directly employed to work on a particular ward (e.g. doctors), five unnamed (non-personalised) surveys were also made available for each ward and the sister was asked to ensure that these were all distributed. All surveys were labelled with a code that allowed identification of the trust and ward, but not the individual.
The aim was to recruit a minimum of 50% of the staff on each ward. When this target was not reached at the end of the patient data collection period, research nurses visited the ward on one further occasion to encourage questionnaire completion (blank copies were made available if necessary). As a way of incentivising the process, we offered £1 per completed questionnaire, with all money going to the ward.
To examine the variation in the data generated by the safety culture survey tools, the following variables were created for each ward: (1) perceptions of patient safety, (2) patient safety ‘grade’ and (3) frequency of events reported. Mean scores were then calculated for each ward at baseline and 6 months.
Results
Research aim 1: exploring the feasibility of collecting feedback from patients about the safety of care
Feasibility of patient recruitment
In total, 379 patients were recruited into the study over the 6-month study period. Table 22 shows participant characteristics by ward. The average number of participants recruited per ward at each time point was 20.9, with a mean response rate of 86.8%. Length of stay at point of consent ranged between 1 and 9.7 days, with an overall mean of 3.4 days. The primary ethnic origin of participants was white British (79%), with the overall ethnic group representation across the study approaching that of the local population. 271
Ward | Patients consented (response rate, %) | Mean age (years) | Sex | Ethnicity | Mean length of stay at consent (days) | Self-completion of questionnaire (% of patients recruited) | ||||
---|---|---|---|---|---|---|---|---|---|---|
Baseline | 3 months | 6 months | Baseline | 3 months | 6 months | |||||
Ward Aa (control) | 20 (95) | 24 (100) | 21 (100) | 8 | 38 male, 25 female | 49% white British, 41% Pakistani, 10% other | 1 | 9 (45) | 11 (46) | 12 (57) |
Ward B (intervention) | 26 (100) | 21 (88) | 20 (80) | 50 | 35 male, 32 female | 85% white British, 9% Pakistani, 6% other | 1.3 | 6 (23) | 4 (19) | 4 (20) |
Ward C (intervention) | 21 (95) | 21 (70) | 20 (80) | 52 | 30 male, 32 female | 85% white British, 8% Pakistani, 7% other | 9.7 | 2 (10) | 1 (5) | 6 (30) |
Ward D (intervention) | 22 (96) | 20 (67) | 20 (95) | 58 | 42 male, 20 female | 79% white British, 13% Pakistani, 8% other | 4.3 | 6 (27) | 5 (25) | 6 (30) |
Ward E (control) | 20 (77) | 20 (74) | 20 (87) | 48 | 34 male, 26 female | 83% white British, 12% Pakistani, 5% other | 2.5 | 6 (30) | 5 (25) | 8 (40) |
Ward F (intervention) | 21 (100) | 19 (79) | 20 (80) | 59 | 29 male, 31 female | 92% white British, 5% Pakistani, 3% other | 7.3 | 5 (24) | 3 (16) | 4 (20) |
Summary – control | 40 (95.24; 95% CI 88.8–100) | 44 (81.4; 95% CI 71.1–91.8) | 41 (97.6; 95% CI 93.0–100) | 28 | 72 male, 51 female | 66% white British, 27% Pakistani, 7% other | 1.75 | 15 (38) | 16 (36) | 20 (49) |
Summary – intervention | 90 (97.83; 95% CI 95.0 to 100) | 81 (77.1; 95% CI 69.1 to 85.1) | 80 (81.6; 95% CI 74.0 to 89.3) | 55 | 136 male, 115 female | 85% white British, 9% Pakistani, 6% other | 5.65 | 19 (21) | 13 (16) | 20 (25) |
Summary – overall | 130 (93.5; 95% CI 89.4 to 97.6) | 125 (78.6; 95% CI 72.2 to 85.0) | 121 (86.4; 95% CI 80.8 to 92.1) | 46 | 208 male, 166 female | 79% white British, 15% Pakistani, 6% other | 4.35 | 34 (26) | 29 (23) | 40 (33) |
These participant characteristics demonstrate that we were able to collect feedback from a variety of patients across a range of hospital settings. We recruited both males and females, patients of varying age and patients from a variety of ethnic groups. Of interest is the differing length of inpatient stay at time of consent. Although this reflects the patient turnover across different wards, it also highlights the willingness of patients to report on their experience of safety after a very short period of time in hospital. Lastly, the response rates reflect the acceptability of the PRASE intervention to patients, with the majority of patients approached agreeing to participate in the study.
With the exception of one ward at one time point, we found that we were able to achieve our aim of collecting data from 20–30 patients on all wards within a 2-week period. Achieving the target figure on this particular ward (ward F) generally took longer across all time points because of the higher proportion of geriatric patients and the 3-month time point being during the winter months, at which time this issue would have been exacerbated. As can be seen from the PMOS and patient-reported safety concerns data, this target figure yielded a variation in scores and a range of qualitative data on safety concerns.
Exploring the impact of randomisation
As can be seen in Table 22, the 95% CIs for summary response rates (intervention and control), indicate that the consent rate was slightly higher in the control group than in the intervention group. This is probably because one of the control wards was a paediatric ward, where those consented were parents, who were therefore not unwell and who were motivated to feed back their experiences of care. Overall, however, the consent rate was very good and did not drop significantly over time. This indicates that the process of randomisation was acceptable to both control wards and intervention wards, with little evidence of a differential impact of attrition.
Exploring variation within the Patient Reporting and Action for a Safe Environment measurement tools
Table 23 displays the means and SDs for each of the PMOS domains by ward at baseline and 6 months. Data are not presented for the second time point as some APGs were not able to meet within the first 3-month cycle. This is discussed in more detail in Timing of the Patient Reporting and Action for a Safe Environment intervention cycle.
Domain | Ward A (control), mean (SD) | Ward B (intervention), mean (SD) | Ward C (intervention), mean (SD) | Ward D (intervention), mean (SD) | Ward E (control), mean (SD) | Ward F (intervention), mean (SD) | Summary – control, mean (SD)a | Summary – intervention, mean (SD)a | ||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Baseline | 6 months | Baseline | 6 months | Baseline | 6 months | Baseline | 6 months | Baseline | 6 months | Baseline | 6 months | Baseline | 6 months | Baseline | 6 months | |
Dignity and respectb | 4.47 (0.61) | 4.19 (0.93) | 4.54 (0.76) | 3.75 (1.37) | 4.19 (0.68) | 4.30 (0.47) | 4.45 (1.05) | 4.38 (0.61) | 4.25 (0.71) | 4.45 (0.60) | 4.00 (1.09) | 4.10 (1.02) | 4.36 (0.16) | 4.32 (0.18) | 4.29 (0.25) | 4.14 (0.29) |
Access to resourcesc | 3.85 (0.60) | 4.16 (0.31) | 3.93 (0.55) | 3.94 (0.34) | 3.80 (0.40) | 3.55 (0.40) | 4.08 (0.49) | 3.86 (0.44) | 4.00 (0.24) | 3.69 (0.38) | 4.07 (0.52) | 3.89 (0.32) | 3.93 (0.11) | 3.91 (0.35) | 3.97 (0.13) | 3.81 (0.18) |
Communication and teamworkingc | 4.07 (0.50) | 4.18 (0.40) | 4.10 (0.40) | 3.96 (0.58) | 3.92 (0.48) | 3.99 (0.43) | 4.10 (0.57) | 4.16 (0.31) | 4.14 (0.25) | 3.94 (0.48) | 4.02 (0.65) | 3.94 (0.44) | 4.10 (0.05) | 4.06 (0.17) | 4.03 (0.08) | 4.01 (0.10) |
Delaysc | 3.29 (1.16) | 4.18 (0.46) | 3.83 (0.69) | 3.14 (0.99) | 2.89 (0.91) | 3.44 (0.64) | 4.03 (0.62) | 3.97 (0.76) | 3.73 (0.78) | 3.75 (0.79) | 3.53 (0.90) | 3.36 (0.95) | 3.51 (0.31) | 3.94 (0.33) | 3.57 (0.49) | 3.49 (0.36) |
Equipmentc | 4.24 (0.50) | 4.21 (0.40) | 3.82 (0.64) | 3.84 (0.35) | 3.83 (0.47) | 3.91 (0.32) | 4.10 (0.74) | 4.10 (0.39) | 4.03 (0.34) | 4.03 (0.40) | 4.19 (0.68) | 3.97 (0.58) | 4.13 (0.15) | 4.11 (0.13) | 3.98 (0.19) | 3.96 (0.11) |
Information flowc | 3.71 (0.74) | 3.95 (0.57) | 3.87 (0.62) | 3.84 (0.64) | 3.76 (0.42) | 3.61 (0.86) | 3.89 (0.54) | 3.82 (0.49) | 3.78 (0.85) | 3.68 (0.60) | 3.75 (0.65) | 3.74 (0.57) | 3.75 (0.47) | 3.81 (0.20) | 3.82 (0.07) | 3.75 (0.10) |
Organisation and care planningc | 3.94 (0.63) | 4.16 (0.43) | 4.00 (0.54) | 3.83 (0.57) | 3.86 (0.54) | 3.98 (0.50) | 4.06 (0.65) | 3.97 (0.44) | 4.04 (0.57) | 3.86 (0.38) | 3.84 (0.63) | 3.85 (0.63) | 3.99 (0.07) | 3.99 (0.24) | 3.94 (0.11) | 3.91 (0.09) |
Staff roles and responsibilitiesc | 3.39 (0.90) | 3.65 (0.77) | 3.27 (0.90) | 3.54 (0.74) | 3.52 (0.77) | 3.42 (0.88) | 3.56 (0.96) | 3.39 (0.76) | 3.85 (0.60) | 3.11 (1.07) | 3.27 (0.95) | 3.32 (0.91) | 3.62 (0.32) | 3.40 (0.35) | 3.41 (0.16) | 3.41 (0.09) |
Staff trainingc | 3.75 (1.05) | 4.23 (0.39) | 3.63 (0.85) | 3.90 (0.43) | 3.87 (0.33) | 3.69 (0.84) | 3.97 (0.89) | 3.97 (0.28) | 4.08 (0.46) | 3.83 (0.79) | 4.03 (0.72) | 4.03 (0.13) | 3.92 (0.24) | 4.06 (0.24) | 3.87 (0.18) | 3.88 (0.14) |
Ward type and layoutc | 3.68 (0.70) | 3.86 (0.64) | 3.70 (0.59) | 3.48 (0.60) | 3.49 (0.37) | 3.67 (0.46) | 4.03 (0.59) | 3.77 (0.51) | 4.00 (0.48) | 3.82 (0.52) | 3.79 (0.55) | 3.81 (0.26) | 3.84 (0.23) | 3.85 (0.03) | 3.76 (0.22) | 3.68 (0.14) |
PMOS overall scorec | 3.73 (0.59) | 4.18 (0.36) | 3.72 (0.32) | 3.70 (0.39) | 3.65 (0.38) | 3.70 (0.27) | 4.05 (0.39) | 3.89 (0.26) | 3.90 (0.28) | 3.73 (0.36) | 3.80 (0.51) | 3.82 (0.32) | 3.81 (0.12) | 3.95 (0.32) | 3.80 (0.18) | 3.78 (0.09) |
The PMOS domain scores show variation across wards and also within wards across different measurement points. For example, for some domains there is an increase in scores between baseline and 6 months, whereas for others there is a reduction. The dignity and respect measure also shows similar patterns of variation.
Table 24 details the summary statistics for the patient-reported safety concerns by ward at baseline and 6 months. The figures suggest that patients are able to feed back safety concerns as part of the PRASE tool and that these reports show variation in terms of their perceived severity and preventability.
Safety concerns | Ward A (control) | Ward B (intervention) | Ward C (intervention) | Ward D (intervention) | Ward E (control) | Ward F (intervention) | Summary – control, meana | Summary – intervention, meana | ||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Baseline | 6 months | Baseline | 6 months | Baseline | 6 months | Baseline | 6 months | Baseline | 6 months | Baseline | 6 months | Baseline | 6 months | Baseline | 6 months | |
Number of patient-reported safety concerns | 6 | 1 | 7 | 17 | 6 | 35 | 16 | 8 | 8 | 9 | 15 | 19 | 7 | 5 | 11 | 20 |
Number of patients reporting one or more concerns | 3 | 1 | 3 | 8 | 4 | 12 | 6 | 7 | 5 | 5 | 8 | 8 | 4 | 3 | 6 | 9 |
Number of reports per patient recruited | 0.3 | 0.05 | 0.27 | 0.85 | 0.29 | 1.75 | 0.73 | 0.4 | 0.4 | 0.45 | 0.71 | 0.95 | 0.35 | 0.25 | 0.5 | 0.93 |
Average severity | 1.5 | 4 | 9 | 8 | 7 | 7 | 7 | 5 | 6 | 7 | 6 | 6 | 3.75 | 5.5 | 7.5 | 6.5 |
Range of severity | 1–2 | – | 8–10 | 5–10 | 5–10 | 2–10 | 1–10 | 2–8 | 2–8 | 3–9 | 3–10 | 2–9 | – | – | – | – |
Average preventabilityb | Definitely yes, definitely no | Probably yes | Definitely yes | Probably yes | Definitely yes | Probably yes | Definitely yes | Probably not | Definitely yes | Probably yes | Probably yes | Probably not | – | – | – | – |
Research aim 2: exploring the feasibility and acceptability of the Patient Reporting and Action for a Safe Environment intervention process for staff
Sample
Observational notes were taken for all intervention wards at both stages of the project (n = 7), apart from for one ward at stage 1 because of an administrative error. Staff from three of the four intervention wards took part (n = 6) in the telephone interviews, with staff from the remaining ward declining because of lack of availability. It is noteworthy that every intervention ward involved in the study found it difficult to allocate protected time for their staff to be interviewed about the process. This made the interviews difficult to arrange and they were sometimes interrupted. For the three wards that did nominate staff for interview these were all postponed and rescheduled at least once.
From a synthesis of the qualitative methods, four themes arose, which are useful to examine in depth and are detailed under the following headings.
Engagement with the intervention
The attitude of ward staff towards the intervention and their general engagement was a consistent theme from our data. When a ward was positive about being involved in the study, this was reflected in both the observational notes and what ward staff discussed during the telephone interviews. Equally, the converse was true, with ward staff clearly stating using both methods if they were disinterested in the intervention.
Overall, staff from three wards were largely positive about the PRASE process, welcoming the opportunity to receive detailed patient feedback and facilitated action planning. They valued the patient feedback itself as well as the process of involving a multidisciplinary team in addressing the concerns that their patients had raised. An interesting point to note, which arose through both methods, is that when staff viewed the intervention positively they often talked about how data from this study had reinforced data that they were receiving through other safety initiatives or even what was known tacitly about the ward. It seems that the PRASE intervention often provided ‘objective’ data about what was commonly known about the safety and quality of care on the ward. Ward staff valued this reinforced knowledge, which they believed to have come from a robust source (an independent research team).
In contrast, staff representatives from one ward were largely indifferent about the process of action planning. This highlights the variability in willingness of ward leaders to engage with the study and the process of action planning for change more widely. It is reflective that, on this ward, no action plans were made in either phase of the project. From the telephone interview it was found that this ward questioned the usefulness of receiving feedback from patients about quality and safety as staff believed that they already did everything within their control to meet the needs of patients. They stated that most of the issues that patients raised in the feedback report would already be addressed on a daily basis through their ongoing attention to patients’ needs.
It is important to recognise that engagement with the intervention was not related to whether the feedback received from patients was on the whole largely positive or largely negative. One of the wards that was most welcoming of the intervention had a significant amount of negative issues that the patients had identified. By contrast, the ward that was most disengaged with the study had some of the most positive patient feedback. Therefore, it cannot be concluded that the ward with poorer patient feedback had staff who were too busy or ‘stressed’ to be able to engage with the intervention. Rather, the opposite was true. One interpretation of this might be that, when a ward believes that it is performing well, negative feedback may be taken as a personal criticism of the ward staff. Indeed, for the ward that was not engaged with the study, staff stated in the telephone interview that they were not that interested in receiving information from their patients.
Implementation of action plans
Wards made action plans that contained one or more action points. Action plans tended to consist of two or three main action points, although some wards had a higher volume of action points. Of the four intervention wards involved in the study, one chose not to make an action plan at either time point. Of the other three wards, 21 action points were made in total. In conjunction with ward representatives (as part of the telephone interviews), all action points were assessed to see whether or not they had been implemented, with a three-point scoring system: ‘fully’, ‘partially’ or ‘not at all’. The three APGs who made action points were most likely to only partially implement them (57%), with 33% of plans fully implemented and just 10% not implemented at all. It is useful to explore what types of plans were most likely to be achieved as opposed to those that were only partially implemented or not at all, using illustrative examples.
Actions that were achieved were those that had a defined and relatively simple remit and could be accomplished by members of the APG. Examples include a review and alteration of furniture layout to provide more space around patient bedsides and a new procedure for obtaining information about patients’ existing medications on arrival into hospital. This new procedure involved the provision of a medication box in the ward fridge so that patients can bring their medication from home, a simple but explicit change in the admissions procedure which ensures that full details of medications are available.
The two actions that were never achieved were dependent on engagement from members of staff in other departments. One was an action to resolve regular delays in porter staff by planning to transport patients from the ward to receive treatment elsewhere in the hospital. Ward staff decided as part of their action point to contact porters about this issue to see whether or not patient transportation could be performed in a timelier manner and to investigate delays at the time when they occurred. However, this approach did not resolve the issue because the process of influencing change in the porter system seemed opaque to the ward staff.
The majority of actions were partially resolved and there were two main reasons for this. First, many actions were dependent on the speed and effectiveness of wider trust initiatives. An example is the training of staff in the specialisms of the ward, which some patient feedback had revealed was not currently adequate. A trust initiative to develop all junior nursing staff across a range of training indicators was being rolled out across the hospital at the time of the study but had not achieved completion by the time the study ended. Second, many actions were dependent on addressing staff habits, which in some cases were felt to be too deeply entrenched to be open to modification. An example is for staff involved in a patient’s care to introduce themselves to that patient on a daily basis. Some staff responded well to this but others did not feel comfortable doing so.
The telephone interview data revealed that ward staff need support from senior management to address patient feedback relating to issues that are beyond the direct control of a ward-based APG. One ward vocalised that the patient feedback about the ward needed to be directed to senior management for concrete change to be implemented. In addition, a longer time period was identified as being necessary to implement agreed actions, especially when they were dependent on wider trust initiatives or cultural change.
The researchers’ role in the action planning group
Research staff initially attended APG meetings purely in an observational capacity. At the outset of the study, the ward staff in the APG were tasked with convening their own meetings, autonomously setting their own agenda and discussing the patient feedback among themselves. However, it quickly became apparent that researchers were being drawn into a quasi-facilitation role, otherwise the APG did not function as intended. In some APGs, research staff were asked for their opinion on the patient feedback and even what action plans the researcher would want to make if they were a member of the ward team. As the study progressed, most researchers relinquished their purely observational role to ensure that action planning took place.
Within the research team we held a reflective group discussion about the function of the facilitation role that researchers had found themselves in. The group of researchers agreed that facilitation of an APG was necessary to ensure that (1) staff actually met in an APG to consider the feedback, (2) staff talked to each other about the patient feedback, (3) each member of the APG had an equal voice, (4) action plans were produced as a result of the APG meeting (as the facilitator encouraged this) and (5) the focus was kept on action plans based on the data from the study rather than changes that the staff wanted to make that were perhaps not patient focused or identified.
Timing of the Patient Reporting and Action for a Safe Environment intervention cycle
A final finding from the synthesis of the qualitative data was that staff struggled to hold an APG meeting within the anticipated 3-month period. Although this time period was recommended by the intervention development group, in reality we found this timescale too short. Two out of four intervention wards failed to meet within the first 3 months, with the other two meeting just before the second measurement point. Given that the intervention cannot be considered to be starting until the APG meets to discuss the patient feedback, it was clear that no actions could be expected to have been implemented in the initial 3-month period.
Research aim 3: exploring the feasibility of collecting feedback from staff about the culture of safety
Sample
A total of 199 staff completed surveys at baseline and 6 months. Table 25 shows staff respondents’ characteristics by ward. Response rates varied across the wards and across the two measurement points. However, our aim of achieving a response rate of ≥ 50% was achieved in four out of six wards at baseline and five wards at 6 months. Nursing was the staffing group most represented in the survey responders, followed by other (non-medical) health-care professionals.
Ward | Completed staff questionnaires, n (%) | Job category breakdown of staff completing questionnaires, % | |||
---|---|---|---|---|---|
Baseline | 6 months | Medical | Nursing | Other | |
Ward A (control) | 18 (58) | 21 (64) | 9 | 80 | 11 |
Ward B (intervention) | 8 (24) | 7 (16) | 0 | 86 | 14 |
Ward C (intervention) | 13 (33) | 26 (59) | 11 | 50 | 39 |
Ward D (intervention) | 16 (70) | 18 (62) | 0 | 66 | 34 |
Ward E (control) | 26 (93) | 22 (71) | 2 | 77 | 21 |
Ward F (intervention) | 13 (57) | 15 (63) | 8 | 54 | 38 |
Collecting staff feedback on the patient safety culture
Table 26 details the summary statistics for the patient safety culture survey by ward at baseline and 6 months. The Hospital Survey on Patient Safety Culture is a well-validated patient safety culture assessment tool and, as such, this study was not intended to replicate such validation. However, the results do suggest that wards within the same hospital do vary across the three outcome measures used and that there is also variation over time in these scores.
Ward A (control), mean (SD) | Ward B (intervention), mean (SD) | Ward C (intervention), mean (SD) | Ward D (intervention), mean (SD) | Ward E (control), mean (SD) | Ward F (intervention), mean (SD) | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
Baseline | 6 months | Baseline | 6 months | Baseline | 6 months | Baseline | 6 months | Baseline | 6 months | Baseline | 6 months | |
Perceptions of patient safety | 2.90 (0.70) | 2.44 (0.72) | 2.25 (0.58) | 2.96 (0.70) | 2.31 (0.64) | 2.82 (0.66) | 3.46 (0.55) | 3.35 (0.96) | 4.20 (0.38) | 3.73 (0.75) | 3.35 (0.93) | 3.21 (0.74) |
Frequency of events reported | 3.74 (0.98) | 3.63 (0.77) | 3.96 (1.0) | 4.05 (0.70) | 3.82 (0.98) | 3.88 (0.84) | 3.88 (0.84) | 3.59 (1.27) | 4.13 (0.99) | 4.06 (0.87) | 3.90 (0.92) | 3.64 (0.99) |
Patient safety grade | 2.94 (0.64) | 2.57 (1.06) | 3.00 (1.07) | 4.14 (0.69) | 2.62 (0.65) | 3.21 (0.88) | 4.07 (0.62) | 3.72 (1.01) | 4.19 (0.57) | 4.18 (0.50) | 3.62 (0.65) | 3.14 (0.86) |
Discussion
This chapter has described the process of developing and testing the feasibility of the PRASE intervention across six wards within a large acute NHS trust in the north of England. Our research aims were threefold: (1) to explore the feasibility of the process of systematically collecting feedback from patients about the safety of care, (2) to explore the feasibility and acceptability of the PRASE intervention for staff and to understand more about how staff use the patient feedback for service improvement and (3) to explore the feasibility of collecting feedback from staff about the culture of safety on participating wards. We then were able to use this understanding to make changes to the PRASE intervention prior to a wider cluster RCT.
The findings suggest that the process of systematically collecting feedback from patients on the safety of care can be achieved within the PRASE intervention. Our response rates illustrate that patients were very willing to provide feedback using the PRASE measurement tools, either through self-completing the questionnaire or through a facilitated conversation at the bedside, the latter being the overwhelming preference for patients. Furthermore, we achieved these response rates across hospital settings and patient populations (e.g. elective and acute surgery wards and medical wards), across short- and longer-term inpatient stays and from a representative range of ethnic groups. We were able to achieve our desired figure of 20–30 patients per ward across all but one ward at one measurement point. This figure has also been demonstrated to show variability in scores and provide comprehensive and interpretable feedback reports for staff. Lastly, there was no evidence that randomisation caused any problems for recruitment of participants or retention of wards within the study. These findings provide important information for the further testing of the PRASE intervention within a wider cluster RCT.
With respect to the second research aim, our qualitative enquiry raised a number of important issues that affected the use and usefulness of the PRASE intervention feedback for ward staff and any resultant action on the basis of this feedback. First, the expectation that a group of multidisciplinary health-care professionals can meet together to discuss patient feedback within a 3-month period was ultimately unrealistic. It had been the research team’s intention at the beginning to try and use time in already scheduled multidisciplinary meetings on participating wards. However, the reality was that these did not routinely happen and so the first challenge was to convene such a coming together of professionals providing care on a ward. Given the very real service pressures currently experienced within the NHS, the second challenge was then to ensure that those ward staff intending to participate could get time away from patient care to attend. This ultimately highlighted the need for the PRASE intervention to include explicit guidance regarding the investment in time required to support it, as well as a more realistic time frame for the measurement–feedback–action planning cycle.
Second, our findings suggest that there will be an inherent variation in the way that health-care professionals perceive the role of patient feedback as a means of service improvement. Although we witnessed much enthusiasm for this role, there was also some reticence and defensiveness, resulting in a lack of engagement with the process of action planning. Although some of this reticence will undoubtedly be because of service demands, our findings suggest that some of this will also reflect the attitude of participating staff. Although the former is outwith the control of an isolated intervention, the latter is something that future iterations of the PRASE intervention could, and should, address.
Third, despite the widespread view of health-care professionals involved in helping develop the PRASE intervention that action planning was a simple process that staff would not need support with, our findings suggested that this was not the case. Although we intended at the outset to simply observe the meetings and the action planning process, the reality was that observers were often drawn into a more facilitative role, with staff almost looking to them for guidance about how to interpret, synthesise and act on the patient feedback provided. In retrospect this is perhaps unsurprising and it could be argued that assuming a ‘light-touch’ intervention would uniformly fit all levels of service improvement expertise and understanding was naive. However, given that the level of support was guided by professionals advising the research team within the intervention development group, the discrepancy between perceived action planning competence and the reality is of note.
Finally, a significant factor in the success of implementing actions was found to be the ability of staff to link with other units or support services to make more systemic changes. Some actions, which arguably take a more ‘upstream’ systems approach, were stymied by the perceived helplessness of staff to affect change outside of the ward in which they are based. It would seem that to achieve change that has implications beyond individual wards it will be necessary to engage the support of senior managers for the intervention from the outset. Senior managers with a hospital-wide remit will bring with them access to information that may be relevant to desired actions (which may avoid duplication), potential resources (time, funding, influence) and a public commitment to using patient feedback to improve services.
With respect to our final research aim, we found that the process of collecting feedback from staff about the safety culture of participating wards was feasible. Response rates did vary, with some wards more difficult to engage with the feedback than others. Our attempts to incentivise the process did not appear to have an effect on response rates. It is likely that the view of the senior nursing staff on wards of the value of collecting safety culture feedback may be a key predictor of response rates. This highlights the need to ensure that those involved in the process of testing the PRASE intervention within a cluster RCT have a good understanding of what the measurements are and why they are important.
Recommended changes to the Patient Reporting and Action for a Safe Environment intervention and cluster randomised controlled trial design
In light of our findings from the feasibility testing, the authors recommend the following changes to the PRASE intervention process.
-
An increase in the time between cycles of measurement, feedback and action planning from 3 months to 6 months.
-
Facilitated action planning meetings to encourage teams to integrate the different types of patient feedback and support longer-term systems-based change over inappropriate ‘quick fixes’.
-
An augmented action planning pro forma to encourage staff to consider how they will measure success in addition to what action they will undertake.
-
Better engagement with and support from senior management to support action planning and the implementation process.
-
An initial training session to introduce the intervention to staff and describe more fully the process of using patient feedback for service improvement.
-
Update sessions to provide an opportunity for shared learning between groups and with senior managers. Issues that transcend ward boundaries can be discussed and a wider improvement approach taken.
-
Explicitly recognise (within the APG facilitation guidance) the need of the PRASE intervention to fit within other service improvement activity to avoid duplication of effort.
-
Better support the process of collecting staff feedback about safety culture through education at training sessions as well as informal information sharing when meeting with ward representatives.
Developing a logic model
Having an explicit theory of change for the implementation and evaluation of service improvement interventions is becoming increasingly recognised as crucial for both researchers and improvers. 16 Figure 8 outlines the logic model based on the findings and recommendations of this study. This model helps to explicate the hypothesised relationships between the programme activities and the proximal and distal outcomes, providing a structure for further testing and evaluation. It was developed iteratively through discussion within the research team and is based on the findings from the qualitative research described in this chapter. The logic model forms the basis of the testing of the PRASE intervention in the cluster RCT described in the following chapter.
Limitations
This study had a number of limitations. First, we were limited to six wards within which to undertake this study. Although this was sufficient to address the feasibility study aims, it did not allow for estimates of effectiveness, which may have informed the subsequent study further. Second, with respect to the recruitment of patients into the study, it is possible that, in asking nursing staff about the capacity of patients to participate, a bias was introduced into the sampling regimen. However, it is strongly felt by the research team that this risk must be balanced against the greater risk of causing distress to patients unable to participate because of acute illness or capacity to consent. Furthermore, given that engagement with the intervention by ward staff is an important factor, it was felt by the team that we must at all times work with staff in ways that were acceptable to them.
Conclusion
The PRASE intervention represents an innovative approach for health services wishing to systematically collect feedback from patients about the safety of their care and use this as a basis for making improvements in the way that they deliver care. It is possible that the intervention will be as much about changing culture as about the specifics of service improvement. To this end it will be important for health services seeking to use the intervention to provide adequate resources to support it, in terms of senior management support, as well as the time and space for staff to learn about, interpret and act on patient feedback about safety. It will be crucial not to introduce this as just a top-down safety intervention (the next ‘audit’ to achieve), focused as it is on encouraging ‘bottom-up’ change from the patient perspective upwards. If wards engage wholeheartedly with the intervention, it is possible that this will aid the journey towards achieving greater transparency in how we deliver care, as well as a public commitment to codesigning shared solutions to patient-centred problems.
Chapter summary
This chapter described the feasibility testing of an innovative patient-centred patient safety intervention. The study demonstrated not only that it feasible to systematically collect information from patients about the safety of their care but also that staff – with support – were willing and able to engage with the feedback to make improvements to service design and care delivery. The following chapter will present a more rigorous examination of the intervention within a cluster RCT across three NHS trusts in the Yorkshire and Humber region.
Chapter 8 Assessing risk and learning from error: evaluating the Patient Reporting and Action for a Safe Environment intervention – a cluster randomised controlled trial
Abstract
Trial design: A multicentre, cluster, wait-list design randomised controlled trial to assess the efficacy of the PRASE intervention in achieving patient safety improvements over a 12-month period.
Methods: The trial was conducted in 33 hospital wards across three NHS hospital trusts. All patients able to give informed consent were eligible to take part. Wards were allocated to the intervention or control condition on a 1 : 1 basis using minimisation. Research nurses collecting data from patients were blind to ward allocation. The ward-level intervention included two tools: (1) a 44-item questionnaire that asks patients about factors contributing to safety PMOS and (2) a pro forma for patients to report both patient safety incidents (PSIs) and positive experiences. A report to wards was produced summarising this feedback and ward staff were asked to plan and implement actions with the aim of improving safety. The control group received care as usual but patients also completed the PMOS outcome measure. Control wards did not receive feedback reports until after the study had finished. The two primary outcomes were (1) routinely collected harm-free care score and (2) the PMOS questionnaire. We also measured patient safety culture and patient experience (friends and family test) as secondary outcomes. Statistical analyses used regression modelling (accounting for clustering for patient-level analyses). A naive cost–consequence analysis was used to estimate the impact of the intervention on both implementation costs and outcomes (clinical and non-clinical).
Results: Intervention uptake and retention of wards was 100% and patient participation was high at 86%. However, adherence to the intervention, particularly the implementation of action plans, was poor. We found no significant effect of the intervention on any outcomes at 6 or 12 months. Given this, and the cost of delivery, it is highly unlikely that the PRASE intervention is cost-effective in improving patient safety. However, we did find some improvements in the intervention wards compared with the control wards for new harms (i.e. those for which the wards were directly accountable) and these differences were largest among wards that showed the greatest compliance with the intervention.
Conclusion: Despite patients being willing and able to provide feedback using the PMOS and PIRT tools and wards engaging with this feedback, we were unable to demonstrate any significant effect of this intervention on patient safety.
Trial registration: Current Controlled Trials ISRCTN07689702.
Chapter rationale
The previous chapter outlined the development and feasibility testing of the PRASE intervention, a novel and innovative approach to using patient feedback about safety as a basis for service improvement. What this testing was not designed to achieve, however, was to identify whether or not this feedback and patient-centred change to care delivery actually resulted in improved patient safety outcomes. To address this, the PRASE intervention was further tested within a cluster RCT. This chapter presents this cluster RCT, which was conducted across three NHS trusts within the Yorkshire and Humber region.
Background
Despite both national and international emphasis on patient involvement in safety, there is a dearth of evidence on how best to involve patients and whether or not this leads to improvements in safety. The evidence that exists indicates that patients are willing and able to participate in error prevention strategies,10 which have the potential to improve safety. 7,11–14 Previous chapters describe the development of tools that allow patients to provide feedback on the safety of their care environment to inform local and organisational changes. Following a feasibility study, in this chapter we address the question of whether or not such tools (PMOS and PIRT), deployed as part of an intervention based on two key intervention strategies, monitoring and feedback (of patient experiences) and action planning (by ward staff), has a positive effect on safety.
This chapter reports on the evaluation of the PRASE intervention. This study represents the first evaluation of the effect of patient feedback on safety.
Methods
Trial design
This study was a multicentre, cluster, wait-list design RCT to assess the efficacy of the PRASE intervention in achieving patient safety improvements over a 12-month period. A cluster design was utilised as the intervention was at the ward level, not at the individual patient level.
Participants
The study was conducted in 33 hospital wards spread across three NHS trusts over five different hospital sites (eight wards at a small district general hospital, 10 wards at a medium-sized teaching hospital and 15 wards at a large teaching hospital). We invited five trusts to participate and received responses from four, with one declining and three agreeing to participate. At the small district hospital, all adult non-intensive wards were recruited. At the medium-sized teaching hospital, the chief nurse recruited 10 adult wards, both surgical and medical, to the study. Within the final trust, wards were asked to volunteer to take part. Study participants were patients within participating wards. An average of 25 patients within each ward meeting the following eligibility criteria were recruited at three different time points.
Inclusion criteria
-
Male or female.
-
Aged > 16 years.
-
Able to give informed consent.
-
Minimum period of 4 hours on the ward before questionnaire administered.
Exclusion criteria
-
Has capacity but is too ill or distressed to take part (e.g. breathlessness, pain, bleeding, immediately postoperative).
-
Has already taken part in the study within the previous month.
-
Non-English- or non-Mirpuri-speaking patients (one member of the data collection team spoke Mirpuri, a commonly spoken language among British South Asian patients in the West Yorkshire area).
Interventions
The PRASE intervention (described in earlier chapters) uses patient feedback about the safety of care as a means of achieving patient-centred service improvement and, ultimately, improved patient safety outcomes. In Chapter 7, the ‘logic’ of this intervention was explicated within a programme theory, which outlines the hypothesised ways in which the programme activities will lead to the desired ‘proximal’ and ‘distal’ outcomes. Following the feasibility study, and mindful of the recommended changes to the process of delivering the intervention, the PRASE intervention was modified slightly prior to the cluster RCT described here. In line with the programme theory, the PRASE intervention was designed to collect feedback from patients about the safety of their care, using robust tools. This patient feedback was then collated and presented to each ward in a formal ‘feedback report’. During a facilitated action planning meeting, ward staff used an action planning pro forma to agree on a set of ward-specific improvements/actions to address areas of patient concern. The philosophy of this intervention is that it is an iterative process with a cycle of measurement, feedback and change lasting for a period of 6 months. Ward staff engaged in two cycles during the 12-month intervention period. In response to some of the key moderating factors identified within the programme theory, within this cluster RCT the intervention now included (1) a ‘start-up’ meeting at which all intervention wards within each trust met together to better understand the PRASE intervention, systems approaches to service redesign and the importance of patient feedback for achieving patient safety improvement; (2) a ‘mid-point’ (6-month) meeting for intervention wards at which staff could discuss any collective learning and actions that might require senior-level support; (3) visible support from senior managers, with the director of nursing and medical director invited to attend these two learning sessions; (4) action planning sessions facilitated by members of the research team, to support action planning and discussion of patient feedback; and (5) explicit guidance about the importance of action planning within multidisciplinary teams. As this represented an efficacy rather than effectiveness trial, the intervention required extra resources in the form of (1) data collectors (in this case research nurses), (2) a facilitator to support action planning and (3) the time of health-care staff to develop and implement actions.
Control wards received no intervention during the study period. At the end of the study, control wards received a feedback report for the three time points, amalgamated into one. They were also provided with a 1-hour training session on how to interpret and use their data.
Intervention fidelity
Our assessment of fidelity was informed by the programme theory for the intervention. The fidelity assessment was designed to aid our understanding of the degree to which intervention wards adhered to key components of the programme activities, the effect of a number of the key moderating factors and our achievement of a number of the desired proximal outcomes. Adherence was assessed in relation to eight components of the intervention listed below; each was rated independently by three members of the research team. We followed a framework proposed by Carroll et al. 272
-
Attendance of at least one ward representative at the orientation meeting (0 = no; 3 = yes).
-
Multidisciplinary team present at the phase 1 action planning meeting (0 = no action planning meeting; 1 = one staff group represented; 2 = at least two staff groups represented; 3 = more than two staff groups represented).
-
Creation of action plans in phase 1 (0 = no action plans; 1 = limited action plans including mainly quick fixes; 2 = considered action plans reflecting issues identified, with the potential for short-term impact; and 3 = as 2, but with the potential as longer-term solutions).
-
The extent to which action plans were implemented in phase 1 (0 = no implementation; 1 = at least one action plan partially implemented; 2 = most action plans implemented; and 3 = all action plans implemented).
-
Multidisciplinary team present at the phase 2 action planning meeting (as above).
-
Creation of action plans in phase 2 (as above).
-
Implementation of action plans in phase 2 (as above).
-
Attendance of at least one ward representative at the mid-point meeting (as orientation meeting).
It was decided a priori that scores of 0 and 1 would be classed as non-adherence, whereas scores of 2 or 3 would imply adherence.
Outcomes
Each trust had a start-up meeting following randomisation. There were three outcome measurement periods – at baseline (pre randomisation) and at 6 and 12 months post start-up.
Patient Safety Thermometer data
Patient Safety Thermometer (PST) data are routinely collected monthly within the NHS at the ward level and include information on new and existing cases of four types of possible ‘harm’ [pressure ulcers, venous thromboembolism (VTE), catheter-associated urinary tract infections (UTIs) and falls]. The proportion of patients who are harm free (0–100%) was defined as a primary outcome. A baseline was calculated using the average of the 3 months prior to randomisation; for the 6- and 12-month time points, the month during data collection and the preceding and following months were averaged.
The percentage of harm-free care (HFC) when considering only new harm (new pressure ulcers, falls with harm, catheters and new UTIs and new VTE) was specified post hoc as being of particular interest as these harms could be directly attributed to the participating ward.
Patient Measure of Safety questionnaire
The PMOS questionnaire, a 44-item questionnaire completed by the patient (self-reported) or facilitated by a researcher, was collected at each time point. Forty-three items ask for responses using a Likert scale and one further item asks for a qualitative response. Items are scored from 1 to 5 with high scores indicating a more positive response. Responses of ‘not applicable’ or ‘prefer not to answer’ were treated as missing.
The PMOS questionnaire is scored into nine domains and one single stand-alone item; an overall score is also calculated. Higher scores correspond to more favourable responses.
Secondary outcomes
Commissioning for Quality and Innovation and recommendation to family and friends
Patients completing the PMOS questionnaire at each time point were also asked to complete questionnaire items that are usually routinely collected in hospitals. These were three Commissioning for Quality and Innovation (CQUIN) questions to which responses were categorical relating to the involvement in care, hospital staff and privacy as well as an additional item – ‘How likely are you to recommend this ward to friends and family if they needed similar care or treatment?’ – to which patients responded using a five-point Likert scale (ranging from ‘extremely unlikely’ to ‘extremely likely’).
Staff patient safety ‘culture’
Staff on participating wards were asked to complete four questions from the AHRQ Hospital Survey on Patient Safety Culture194 at baseline and 12 months. These items were chosen because they are designated as ‘outcome measures’ in the AHRQ guidance for using the patient safety culture questionnaire. A detailed assessment of the culture was not necessary here and by choosing these four outcome measures we aimed to minimise response burden and hence maximise response rate. Negatively worded items were reverse coded so that a higher score indicated a more positive response. Four safety culture dimensions were then created from the hospital survey.
-
Perceptions of patient safety (mean of four items, ranging from 1.0 to 5.0, with larger values representing a higher score).
-
Frequency of event reporting (mean of three items, ranging from 1.0 to 5.0, with larger values representing a higher score).
-
Number of events reported in the last 12 months (one item, categorical scale).
-
A ward patient safety grade from 1 to 5 (1 = excellent, 5 = failing).
Ward-level covariates
Per-month staff absence/sickness rates (sum of the total sickness absence days including non-working days divided by the sum of the total days available), number of beds available per ward and ward activity (captured through number of discharges) were collected. It was planned that patient acuity/dependency scores and the nurse to patient ratio would also be collected but these data were not available.
Sample size
The study was powered to detect a small to medium difference (effect size = 0.3) between the allocated groups with respect to the PMOS score (0–100). A small to medium effect size seemed reasonable as each ward would be focusing on developing and implementing tailored action plans, meaning that the intervention might not impact on all areas measured by the PST or PMOS tool. To achieve 80% power (with alpha = 0.05) with an average cluster size of 25 patients and an assumed intracluster correlation coefficient (ICC) of 0.05, 32 wards were required (16 per arm). This estimate of the ICC was considered reasonable given the secondary care setting and patient-reported outcome. 39
Randomisation
Wards were randomly assigned to the intervention or control group on an equal basis by the trial statistician at York Trials Unit using a secure computer system. Minimisation was used to balance the groups with respect to ward type (medical or surgical), age (low, middle or high based on tertiles of the distribution from all wards), male/female/mixed sex wards and ward size (low, middle or high based on tertiles of the distribution from all wards).
Blinding
Baseline data were collected at least 1 month before randomisation. The nature of the intervention meant that it was not possible to blind wards. All meetings with wards, creation of feedback reports and facilitation of action planning meetings were conducted by researchers not involved in data collection. Blinding of research nurses collecting data was planned so as not to bias responses but in reality some would be inadvertently unblinded through conversations with ward managers.
Statistical methods
Analysis was conducted in Stata version 13 (StataCorp LP, College Station, TX, USA) using the principles of ‘intention to treat’, meaning that wards (and associated patients) were analysed according to the trial arm that they were randomised to, regardless of intervention implementation or fidelity. Minimisation factors and clustering were accounted for when required. Statistical significance was assessed at the two-sided 5% level. For regression analyses, the difference in adjusted (least-square) means are summarised with 95% CIs. Mixed regression took a restricted maximum likelihood (REML) approach with random effects to account for any potential clustering by ward. Effect size estimates presented are Cohen’s d effect sizes. Estimates of ICC are presented for both primary outcomes.
Participant flow
Consolidated Standards of Reporting Trials (CONSORT) diagrams were used to show the flow of wards through the trial and cluster size at each time point in relation to primary outcomes. 273
Ward and patient characteristics
Baseline ward and patient characteristics were summarised (no formal statistical comparisons were made), as were patient characteristics of those participating at 6 and 12 months.
Primary analyses
Patient Safety Thermometer
The primary analysis of the PST outcome used a ward-level linear regression model with weighting based on sample size to assess differences between the two allocated groups in the percentage of HFC at 12 months. Adjustment was made for baseline HFC and minimisation factors.
Patient Safety Thermometer sensitivity analyses
The primary PST analysis was repeated twice in prespecified sensitivity analyses:
-
without adjustment for minimisation factors
-
with additional adjustment for ward-level characteristics (as in Ward-level covariates).
A further post hoc sensitivity analysis was conducted as one ward had partially missing baseline data.
Patient Measure of Safety questionnaire
Cronbach’s alpha was calculated at each time point and overall; a value > 0.7 was deemed to show reliability.
The ability of the PMOS questionnaire to distinguish between wards hypothesised to be different with respect to patient safety was assessed using known-group comparisons. Ward-level linear regression models with weighting were used to compare wards with a high/low percentage of HFC (based on tertiles at baseline).
Linear mixed models accounting for minimisation factors and baseline average were used to compare the allocated groups with respect to scores at 12 months. As prespecified sensitivity analyses, these regressions were repeated with further adjustment for ward-level characteristics. A post hoc sensitivity analysis included method of completion as a covariate in further repetitions of these models.
Complier average causal effect analysis
Complier average causal effect (CACE) analysis using an instrumental variable approach with weighted regression was used to account for non-adherence within intervention wards in relation to both primary outcomes. Two adherence cut-off values were applied: 50% and 75%. A ward had 50% adherence if it complied with at least four of the scored components and 75% adherence if it complied with at least six.
Secondary analyses
Patient Safety Thermometer
An analogous model to that used in the primary PST analysis assessed differences between the allocated groups at 6 months.
A post hoc repetition of the primary analysis assessed differences between the allocated groups with respect to new harm at 12 months.
Commissioning for Quality and Innovation and recommendation to family and friends
Responses to the CQUIN questions were summarised and a random-effects logistic regression model was used to compare differences between the allocated groups in terms of finding staff to talk to about worries or fears. Answers of ‘yes, definitely’ and ‘yes, to some extent’ were combined to produce a favourable response. Adjustment was made for baseline proportion and minimisation factors.
A random-effects logistic regression model was used to compare the proportion of patients recommending the ward at 12 months in each trial arm; responses of ‘extremely likely’ and ‘likely’ were considered a recommendation. The model accounted for minimisation factors. Adjustment for baseline proportion was planned but not included because of a large proportion of events occurring in the groups.
Staff patient safety culture
Answers to the four safety culture dimensions were summarised. Separate ward-level linear regression models were used to compare average perception of patient safety and proportion of staff favourably grading their ward across the allocated groups at 12 months, with adjustment for minimisation factors and baseline. A favourable patient safety grade was defined as a response of ‘excellent’ or ‘very good’.
Economic evaluation
Given the limitation of the intervention for the measurement of utilities and as resource use was only PRASE related, a naive cost–consequence analysis was conducted. This format aimed to summarise PRASE-related costs and outcomes in a descriptive and disaggregated way, which represents a more approachable and less complex method to make appropriate decisions on the outcome of interest without overlooking the wider effectiveness of the intervention. 274,275
Resource use and costs
The PRASE intervention requires intensive involvement of ward staff, which is the main cost component. Four categories of resource were considered.
-
Collection of patient data (taking consent and completing the PMOS questionnaire) was estimated as cost per minute according to staff category. All of the researchers involved in data collection were at band 6 according to the Agenda for Change, NHS England [see www.healthcareers.nhs.uk/about/careers-nhs/nhs-pay-and-benefits/agenda-change-pay-rates (accessed 1 September 2016)].
-
Action planning meetings: the identified action planning team consisted of a minimum of one person working in the ward and ideally included both senior and more junior staff from different professional groups. Action planning meetings took place twice during the trial and were facilitated by a senior researcher from the quality and safety research team.
-
Management of the PRASE intervention: there were three sessions: (1) the start-up session (at baseline), (2) the mid-point meeting (at 6 months) and (3) the close-up meeting (at 12 months).
-
Actions: as a result of the intervention the wards proposed actions to improve patient safety.
Resource use was estimated in terms of the mean value per ward in the respective units, for example average staff time.
Unit costs
Unit costs applied to the resource usage (Table 27) were retrieved from the Personal Social Services Research Unit (PSSRU) Unit Costs of Health and Social Care 2013. 276
Item | Unit | Cost (£) | Reference | Notes |
---|---|---|---|---|
Collection of data | ||||
Nurse team leader band 6 | Per hour | 49 | Curtis276 | Field workers collecting data during the trial were band 6 |
Action planning meetings | ||||
Facilitators | Per hour | 57 | Curtis276 | Facilitators in the action planning meetings were band 7; costs including qualifications |
Ward manager | 57 | |||
Senior sister | 57 | |||
Junior sister | 49 | |||
Staff nurse, registered nurse | 41 | |||
Occupational therapist | 36 | |||
Physiotherapist | 36 | |||
Hospital pharmacist | 47 | |||
Registrar | 59 | |||
Foundation house officer 1 | 34 | |||
Matron | 41 | |||
Clinical support assistant, ward clerk and clinical support worker (band 2) | 21 |
Assumptions
Although it was possible to attribute a cost directly to the unit of resource use, some assumptions were made.
-
Collection of data. There were two variables that recorded time spent on data collection: (i) time taken for the questionnaire and (ii) total time spent with each patient. When information regarding time taken for the questionnaire was missing but the total time spent with each patient was available, both were assumed as equal. This assumption was also applied when the total time spent with each patient was available but the time taken for the questionnaire was missing.
-
Action planning meeting (phases 1 and 2). When the duration for only one of the meetings was recorded it was assumed that the other meeting lasted for the same amount of time.
-
Management of the PRASE intervention. Time related to the start-up, mid-point and closing meetings was not available; therefore, it was assumed that one ward manager (Agenda for Change band 7) from each intervention ward attended for 2 hours at each meeting.
-
Implementation of action plans. Staff time connected to the implementation of action plans was difficult to estimate and not directly available. To cost these, action plans were first classified as related to (i) roles and responsibilities, (ii) redesigning the environment, (iii) improving communication and information exchange, (iv) training and education, (v) changing behaviour or (vi) implementing or changing procedures. Intensity of resource usage was then classified as (i) low, (ii) intermediate or (iii) high; associated time assumptions are presented in Table 28. For those actions that were partially implemented only, 50% of the total cost was imputed for the analysis. There was no information available on whether actions plans related to phase 2 were implemented or not. It was assumed that these action plans were partially implemented. This is a conservative assumption and thus costs might be underestimated for the second phase of the trial.
Intensity | Assumption relating to staff | Assumptions relating to time |
---|---|---|
Low | One ward team leader (band 6) | 1 hour per week |
Intermediate | One ward team leader (band 6); one staff nurse (band 5) | 2 hours per week |
High | One ward manager (band 7); one ward team leader (band 6) | 3 hours per week |
Results
Participant flow and recruitment
Thirty-four wards were recruited, with one ward excluded prior to randomisation because of an insufficient number of patients available. Participant data collection periods ran between May and July 2013 for baseline, between January and April 2014 for 6 months post start-up and between June and September 2014 for 12 months post start-up.
Figure 9 shows a CONSORT diagram for the PMOS outcome. The percentage of patients who were approached who provided feedback was 89%, 88% and 82% for baseline, 6 months and 12 months, respectively, giving an overall completion rate of 86%.The average cluster size was 25, as planned in the sample size calculation. Figure 10 shows an analogous diagram for the PST outcome; cluster sizes are large as 3 months’ worth of data were used at each time point. Thirty-three wards were randomised (16 allocated to control, 17 to intervention), with all wards retained throughout the trial.
Implementation of the intervention
Table 29 shows summary statistics for fidelity scores for aspects of the intervention. Multidisciplinary team scores were slightly higher at the first meeting than at the second [1.88 (SD 0.86) vs. 1.18 (SD 0.64)]. Fidelity related to the action plan and implementation was similar in both phases; however, a score was not available for three wards for phase 1. Figure 11 shows the number of components that intervention wards complied with; 11 of the 17 (64.7%) intervention wards complied with at least 50% of the intervention at 12 months and four intervention wards (23.5%) complied with at least 75% of the intervention at 12 months.
Intervention component | n | Mean (SD) | Median (min., max.) |
---|---|---|---|
1. Orientation meeting (baseline) | 17 | 2.65 (1.00) | 3 (0, 3) |
First action plan | |||
2. Multidisciplinary team present at meeting | 17 | 1.88 (0.86) | 2 (1, 3) |
3. Action plan | 17 | 1.35 (0.79) | 1 (0, 3) |
4. Implementation of action plan | 14 | 1.36 (0.84) | 1 (0, 3) |
5. Mid-point meeting (6 months) | 17 | 2.12 (1.41) | 3 (0, 3) |
Second action plan | |||
6. Multidisciplinary team present at meeting | 17 | 1.18 (0.64) | 1 (0, 3) |
7. Action plan | 17 | 1.41 (0.80) | 1 (0, 3) |
8. Implementation of action plan | 17 | 1.35 (0.93) | 1 (0, 3) |
Baseline data
Summary statistics for the minimisation factors are reported in Table 30; all proportions were fairly similar between allocated groups.
Characteristic | Control, n (%) | Intervention, n (%) |
---|---|---|
n | 16 | 17 |
Average age | ||
Low tertile | 5 (31.3) | 4 (23.5) |
Middle tertile | 4 (25.0) | 5 (29.4) |
High tertile | 7 (43.8) | 8 (47.1) |
Ward sex | ||
Female | 1 (6.3) | 4 (23.5) |
Male | 2 (12.5) | 3 (17.6) |
Mixed | 13 (81.3) | 10 (58.8) |
Ward type | ||
Medical | 7 (43.8) | 8 (47.1) |
Surgical | 9 (56.3) | 9 (52.9) |
Ward size | ||
Low tertile | 4 (25.0) | 6 (35.3) |
Middle tertile | 3 (18.8) | 4 (23.5) |
High tertile | 9 (56.3) | 7 (41.2) |
One ward was incorrectly listed as being a female-only ward when it was a mixed ward; summaries reported are corrected. Analyses were conducted treating this ward as a female-only ward (as randomised).
Ward characteristics
Table 31 shows summary statistics for ward characteristics over the study period (May 2013 to September 2014); the allocated groups are generally well balanced. Overall, the mean size of wards was between 25 and 26 beds (range 10–32 beds). The number of discharges ranged between 32 and 223 patients per month with a mean of 121 patients per month and the rate of staff sickness ranged between 1.9% and 9.4% with a mean of 4.9%.
Characteristic | Intervention | Control | Overall |
---|---|---|---|
Number of beds, mean (SD) | 25.0 (4.73) | 26.1 (5.46) | 25.5 (5.13) |
Number of discharges per month, mean (SD) | 120.8 (57.19) | 120.8 (49.29) | 120.8 (53.49) |
Percentage staff sickness per month, mean (SD) | 4.4 (1.64) | 5.5 (1.9) | 4.9 (1.86) |
Summary statistics for patients completing the PMOS questionnaire at each time point are presented by trial arm in Table 32. At baseline, mean age was similar between the randomised groups as was the proportion of women. Most patients were British (92% in both trial arms) and just fewer than 40% of patients in each group were receiving ongoing treatment elsewhere at the time of data collection. The mean number of times that participants had been an inpatient in the hospital trust in the past 5 years was just over two in both groups and the mean length of time spent in hospital until enrolment in the trial was around 7 days. Balance was also observed on all variables at 6 months and 12 months when comparing the intervention and control groups.
Characteristic | Baseline | 6 months | 12 months | |||
---|---|---|---|---|---|---|
Control | Intervention | Control | Intervention | Control | Intervention | |
n | 399 | 423 | 408 | 419 | 393 | 429 |
Age (years) | ||||||
Mean (SD) | 60.2 (18.2) | 61.7 (17.6) | 60.1 (18.4) | 61.4 (17.9) | 58.3 (18.9) | 58.2 (18.6) |
Median (min., max.) | 62 (18, 92) | 64 (18, 92) | 63 (16, 103) | 65 (18, 94) | 61 (16, 93) | 61 (16, 94) |
Missing, n (%) | 1 (0.3) | 2 (0.5) | 1 (0.2) | 2 (0.5) | 8 (2.0) | 2 (0.5) |
Sex, n (%) | ||||||
Female | 189 (47.4) | 199 (47.0) | 181 (44.4) | 185 (44.2) | 200 (50.9) | 201 (46.9) |
Male | 209 (52.4) | 221 (52.2) | 223 (54.7) | 231 (55.1) | 186 (47.3) | 219 (51.0) |
Missing | 1 (0.3) | 3 (0.7) | 4 (1.0) | 3 (0.7) | 7 (1.8) | 9 (2.1) |
Ethnicity, n (%) | ||||||
Asian | 9 (2.3) | 11 (2.6) | 11 (2.7) | 9 (2.1) | 10 (2.5) | 7 (1.6) |
Black | 7 (1.8) | 4 (0.9) | 4 (1.0) | 6 (1.4) | 4 (1.0) | 1 (0.2) |
British/Irish | 369 (92.5) | 392 (92.7) | 380 (93.1) | 386 (92.1) | 375 (95.4) | 407 (94.9) |
Other | 10 (2.5) | 10 (2.4) | 9 (2.2) | 13 (3.1) | 3 (0.8) | 10 (2.3) |
Missing | 4 (1.0) | 6 (1.4) | 4 (1.0) | 5 (1.2) | 1 (0.3) | 4 (0.9) |
Ongoing treatment, n (%) | ||||||
Yes | 148 (37.1) | 166 (39.2) | 154 (37.7) | 189 (45.1) | 117 (29.8) | 135 (31.5) |
No | 248 (62.2) | 254 (60.0) | 251 (61.5) | 228 (54.4) | 272 (69.2) | 286 (66.7) |
Missing | 3 (0.8) | 3 (0.7) | 3 (0.7) | 2 (0.5) | 4 (1.0) | 8 (1.9) |
Inpatient frequency | ||||||
Mean (SD) | 2.3 (6.4) | 2.2 (3.5) | 1.6 (2.6) | 2.4 (6.4) | 1.9 (4.0) | 3.2 (9.4) |
Median (min., max.) | 1 (0, 100) | 1 (0, 40) | 1 (0, 30) | 1 (0, 100) | 1 (0, 50) | 1 (0, 100) |
Missing, n (%) | 2 (0.5) | 8 (1.9) | 1 (0.2) | 4 (1.0) | 1 (0.3) | 2 (0.5) |
Time in hospital to date (days) | ||||||
Mean (SD) | 7.3 (14.3) | 6.8 (12.5) | 6.5 (10.9) | 6.4 (10.2) | 6.0 (9.2) | 6.5 (14.1) |
Median (min., max.) | 3 (0, 143) | 3 (0, 119) | 3 (0, 106) | 3 (0, 130) | 3 (0, 70) | 3 (0, 167) |
Missing, n (%) | 1 (0.3) | 5 (1.2) | 5 (1.2) | 6 (1.4) | 1 (0.3) | 3 (0.7) |
Patient Safety Thermometer
Table 33 shows summary statistics for PST score. The sample size was > 1000 and the mean percentage HFC was high (> 90%) at each time point. Percentages were similar in both allocated groups and were higher at 6 months than at baseline or 12 months.
Summary statistics | Baseline | 6 months | 12 months | |||
---|---|---|---|---|---|---|
Control | Intervention | Control | Intervention | Control | Intervention | |
Patients in sample | 1162 | 1103 | 1172 | 1228 | 1125 | 1210 |
Number of wards (%) | 16 (100.0) | 16 (94.1) | 16 (100.0) | 17 (100.0) | 16 (100.0) | 17 (100.0) |
Percentage HFC | ||||||
Mean (SD) | 92.4 (4.95) | 91.4 (7.25) | 94.1 (6.30) | 94.0 (3.93) | 92.5 (3.47) | 92.4 (6.70) |
Standard error | 1.24 | 1.81 | 1.57 | 0.95 | 0.87 | 1.62 |
Median (min., max.) | 93.0 (82.4, 100) | 93.5 (72, 100) | 95.0 (73.9, 100) | 94.4 (87.2, 100) | 93.4 (85.9, 97.4) | 93.5 (73.1, 100) |
The mean cluster size per ward was similar in both groups. There were no missing data in relation to the PST at 6 or 12 months; however, one ward had no baseline percentage score because of missing data for a required month and hence was excluded from the primary analysis.
Primary analysis of the Patient Safety Thermometer
A weighted ward-level linear regression model to assess differences in the percentage of HFC at 12 months showed no evidence of a difference between the allocation groups (p = 0.99), with a non-significant decrease of 0.03% for intervention wards compared with control wards (95% CI –3.59% to 3.53%, effect size –0.01). The adjusted mean HFC was 92.30% and 92.26% for the control and intervention wards, respectively.
Sensitivity analyses are reported in Table 34. All analyses showed no evidence of a difference between the groups.
PST analysis | Difference (95% CI) (%) | p-value |
---|---|---|
Primary | –0.03 (–3.59 to 3.53) | 0.99 |
No adjustment for minimisation factors | ≈0.00 (–3.22 to 3.22) | 1.00 |
With adjustment for ward characteristics | –1.03 (–4.63 to 2.57) | 0.56 |
Inclusion of previously excluded ward | 0.06 (–3.30 to 3.42) | 0.97 |
CACE (50% compliance cut-off) | –0.06 (–5.61 to 5.50) | 0.98 |
CACE (75% compliance cut-off) | –0.12 (–11.70 to 11.46) | 0.98 |
New harm | 1.60 (–0.62 to 3.83) | 0.15 |
New harm CACE (50% compliance cut-off) | 2.42 (–1.38 to 6.22) | 0.19 |
New harm CACE (75% compliance cut-off) | 5.38 (–3.89 to 14.64) | 0.24 |
Patient Safety Thermometer secondary analyses
An analogous model to the primary analysis to assess differences in the percentage of HFC at 6 months showed no evidence of a difference between the allocation groups (p = 0.58), with a non-significant increase of 1.09% for intervention wards compared with control wards (95% CI –2.94% to 5.12%, effect size 0.20).
Figure 12 shows the mean PST HFC by allocated group at baseline alongside the adjusted HFC at 6 and 12 months. There was a larger, although still non-significant, difference between the two allocated groups at 6 months than at 12 months. The baseline average was lower in the intervention group than in the control group. This was not the case at 6 months; however, very little difference was seen between the two groups at the primary end point.
New harm
When the percentage HFC included only new harms, the results were consistent with the primary analysis in that there was no evidence of a difference between the allocation groups (p = 0.15). There was a non-significant increase of 1.60% for intervention wards compared with control wards (95% CI –0.62% to 3.83%, effect size 0.51). This difference is larger in magnitude than that found in the primary analysis and relates to a Cohen’s d effect size of 0.51. When non-compliance was accounted for through CACE analysis, greater improvements of 2.42% (50% compliance) and 5.38% (75% compliance) were found for the intervention group compared with the control group when new harms was used as the outcome (see Table 34).
Patient Measure of Safety questionnaire
Mean cluster size was very similar between the allocated groups at each time point and was 25 overall as planned, meaning that the study was adequately powered in relation to the PMOS tool.
Table 35 shows summary statistics for the PMOS completion method. The proportion of self-completion was relatively low at each time point, with a maximum of just under 25%; the majority of completions were facilitated.
PMOS completion method | Baseline | 6 months | 12 months | |||
---|---|---|---|---|---|---|
Control | Intervention | Control | Intervention | Control | Intervention | |
Self-completed, n (%) | 71 (17.8) | 84 (19.9) | 62 (15.2) | 84 (20.0) | 93 (23.7) | 72 (16.8) |
Researcher completed, n (%) | 328 (82.2) | 338 (79.9) | 346 (84.8) | 335 (80.0) | 294 (74.8) | 357 (83.2) |
Mixed, n (%) | 0 (0.0) | 1 (0.2) | 0 (0.0) | 0 (0.0) | 6 (1.5) | 0 (0.0) |
Total, n | 399 | 423 | 408 | 419 | 393 | 429 |
Missing data
Over 70% of individuals in each arm at each time point were missing two items or less. The mean number of items without a score was low at each time point, ranging between 0.9 (SD 1.8) and 1.7 (two occurrences; SD 3.6 and 3.9, respectively).
Summary statistics
Table 36 shows summary statistics for the unadjusted overall PMOS scores; summaries for each domain are provided in Appendix 7.
Summary statistics | Baseline | 6 months | 12 months | |||
---|---|---|---|---|---|---|
Control | Intervention | Control | Intervention | Control | Intervention | |
n | 399 | 423 | 408 | 419 | 393 | 429 |
Overall PMOS score | ||||||
n (%) | 352 (88.2) | 340 (80.4) | 382 (93.6) | 385 (91.9) | 301 (76.6) | 355 (82.8) |
Mean (SD) | 3.9 (0.39) | 3.9 (0.42) | 3.9 (0.41) | 3.9 (0.42) | 4.0 (0.42) | 4.0 (0.40) |
Standard error | 0.02 | 0.02 | 0.02 | 0.02 | 0.02 | 0.02 |
A slightly higher proportion of individuals enrolled in control wards at baseline than in intervention wards provided enough data to receive an overall PMOS score (88.2% vs. 80.4%). The proportions receiving an overall score at 6 and 12 months were similar between arms; the highest proportions were seen at the 6-month time point.
Primary analysis of the Patient Measure of Safety
The reliability of the PMOS questionnaire in its entirety was high at each time point (α > 0.9) as were the communication and teamwork and ward type and layout domains. Organisation and care planning and staff roles and responsibilities were found to be reliable at 12 months and > 0.6 at other time points.
Weighted ward-level linear regression was used to assess for differences in the PMOS score in relation to the known groups of wards with high/low levels of HFC at baseline (n = 21). The mean percentage HFC was 89.9% in the lower tertile (n = 10) and 94.8% in the upper tertile. When all data were considered together, the overall PMOS score showed evidence of a difference between the known groups (p = 0.04). Several domain scores similarly showed evidence of a difference with the exception of communication and teamwork, delays, equipment, staff training and ward type and layout.
Regression analyses
Linear mixed regression models were used to compare differences in PMOS scores between the two allocated groups. Neither the overall PMOS score nor any domain scores (Tables 37 and 38, respectively) showed evidence of a difference between the intervention wards and the control wards at 12 months. The ICC ranged between 0.01 and 0.03; these values are < 0.05, which was the ICC assumed for the sample size correlation. This does not imply any loss of power based on correlation.
Overall PMOS score | Difference (95% CI) | p-value |
---|---|---|
Primary analysis | 0.08 (–0.01 to 0.17) | 0.09 |
With adjustment for ward characteristics | 0.05 (–0.04 to 0.14) | 0.27 |
With adjustment for method of completion | 0.08 (–0.01 to 0.16) | 0.08 |
CACE analysis (50% compliance cut-off) | 0.15 (–0.05 to 0.36) | 0.13 |
CACE analysis (75% compliance cut-off) | 0.43 (–0.28 to 1.14) | 0.23 |
PMOS domain | n | Differencea (95% CI) | p-value | Effect size | ICC |
---|---|---|---|---|---|
Access | 807 | 0.00 (–0.09 to 0.09) | 0.93 | –0.01 | 0.01 |
Communication and teamwork | 815 | 0.01 (–0.07 to 0.10) | 0.81 | 0.02 | 0.01 |
Delays | 766 | 0.09 (–0.08 to 0.25) | 0.29 | 0.10 | 0.02 |
Equipment | 734 | –0.01 (–0.13 to 0.12) | 0.93 | –0.01 | 0.03 |
Information flow | 799 | 0.02 (–0.09 to 0.14) | 0.71 | 0.04 | 0.02 |
Organisation and care planning | 810 | 0.02 (–0.08 to 0.11) | 0.69 | 0.04 | 0.01 |
Staff roles and responsibilities | 817 | 0.00 (–0.16 to 0.17) | 0.96 | 0.01 | 0.03 |
Staff training | 726 | –0.01 (–0.12 to 0.11) | 0.92 | –0.01 | 0.01 |
Ward type and layout | 817 | 0.02 (–0.06 to 0.10) | 0.65 | 0.04 | 0.01 |
Table 37 shows a summary of the results from the analyses conducted on the overall PMOS scores. The primary analysis, conducted on 656 individuals, showed a non-significant difference of 0.08 (95% CI –0.01 to 0.17, ES = 0.20; p = 0.09) between the two allocated groups, with no evidence that this difference was significant. The marginal mean was lower in the control group (3.96, 95% CI 3.90 to 4.02) than in the intervention group (4.04, 95% CI 3.98 to 4.10). The ICC associated with the overall score was estimated at 0.03. Sensitivity analyses similarly showed no evidence of a difference between the allocated groups.
Figure 13 shows the mean overall PMOS scores by allocated group at baseline alongside adjusted scores at 6 and 12 months. Intervention group scores were slightly lower at baseline than control group scores; this was reversed at 12 months but the differences were non-significant.
Secondary outcomes analysis
Commissioning for Quality and Innovation and recommendation to family and friends
Patient responses to the recommendation to family and friends question and CQUIN questions are presented in Table 39. At each time point over half of patients felt that they were extremely likely to recommend the ward to their friends or family. Around another one-third of patients reported that they were likely to recommend the ward. Over two-thirds felt that they were always involved in care decisions, between one-third and a half of patients had no worries or fear and over three-quarters felt that they were given enough privacy when discussing their condition or treatment.
Question | Baseline, n (%) | 6 months, n (%) | 12 months, n (%) | |||
---|---|---|---|---|---|---|
Control | Intervention | Control | Intervention | Control | Intervention | |
n | 399 | 423 | 408 | 419 | 393 | 429 |
How likely are you to recommend this ward to your friends and family if they needed similar care or treatment? | ||||||
Extremely unlikely | 6 (1.5) | 6 (1.4) | 5 (1.2) | 8 (1.9) | 4 (1.0) | 5 (1.2) |
Unlikely | 9 (2.3) | 16 (3.8) | 12 (2.9) | 8 (1.9) | 7 (1.8) | 5 (1.2) |
Neither | 15 (3.8) | 22 (5.2) | 33 (8.1) | 20 (4.8) | 9 (2.3) | 13 (3.0) |
Likely | 140 (35.1) | 153 (36.2) | 142 (34.8) | 140 (33.4) | 152 (38.7) | 136 (31.7) |
Extremely likely | 229 (57.4) | 226 (53.4) | 216 (52.9) | 243 (58.0) | 221 (56.2) | 270 (62.9) |
Missing | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) |
Were you involved as much as you wanted to be in decisions about your care and treatment? | ||||||
No | 27 (6.8) | 31 (7.3) | 30 (7.4) | 32 (7.6) | 17 (4.3) | 26 (6.1) |
Yes (sometimes) | 93 (23.3) | 120 (28.4) | 97 (23.8) | 102 (24.3) | 93 (23.7) | 96 (22.4) |
Yes (always) | 274 (68.7) | 271 (64.1) | 280 (68.6) | 283 (67.5) | 280 (71.2) | 304 (70.9) |
Missing | 5 (1.3) | 1 (0.2) | 1 (0.2) | 2 (0.5) | 3 (0.8) | 3 (0.7) |
Did you find someone on the hospital staff to talk to about your worries and fears? | ||||||
No | 109 (27.3) | 124 (29.3) | 143 (35.0) | 137 (32.7) | 128 (32.6) | 126 (29.4) |
Yes (some extent) | 21 (5.3) | 29 (6.9) | 32 (7.8) | 37 (8.8) | 17 (4.3) | 17 (4.0) |
Yes (definitely) | 62 (15.5) | 86 (20.3) | 71 (17.4) | 63 (15.0) | 58 (14.8) | 58 (13.5) |
No worries or fears | 202 (50.6) | 183 (43.3) | 161 (39.5) | 180 (43.0) | 187 (47.6) | 225 (52.4) |
Missing | 5 (1.3) | 1 (0.2) | 1 (0.2) | 2 (0.5) | 3 (0.8) | 3 (0.7) |
Were you given enough privacy when discussing your condition or treatment? | ||||||
No | 28 (7.0) | 33 (7.8) | 26 (6.4) | 28 (6.7) | 19 (4.8) | 24 (5.6) |
Yes (sometimes) | 47 (11.8) | 68 (16.1) | 60 (14.7) | 57 (13.6) | 56 (14.2) | 45 (10.5) |
Yes (always) | 319 (79.9) | 320 (75.7) | 320 (78.4) | 328 (78.3) | 313 (79.6) | 356 (83.0) |
Missing | 5 (1.3) | 2 (0.5) | 2 (0.5) | 6 (1.4) | 5 (1.3) | 4 (0.9) |
There was no evidence of a difference between allocated groups in relation the ward recommendation or finding staff to talk to about worries/fears (p = 0.44 and p = 0.92, respectively).
Staff patient safety culture
Table 40 summarises the four safety culture dimensions by allocated group. The means and proportions were fairly similar in both groups across time. The percentage of staff grading their ward as either excellent or very good decreased over time in both groups although the decrease was larger in the intervention group (Table 41). There was no evidence of a difference between allocated groups in the proportion of staff favourably grading their ward or staff perception of patient safety (p = 0.22 and p = 0.87, respectively).
Safety culture dimensions | Control | Intervention | ||
---|---|---|---|---|
Baseline | 12 months | Baseline | 12 months | |
n | 308 | 269 | 340 | 315 |
Staff perception of patient safety | ||||
n (%) | 294 (95.5) | 251 (93.3) | 328 (96.5) | 297 (94.3) |
Mean (SD) | 3.6 (0.77) | 3.6 (0.76) | 3.5 (0.80) | 3.5 (0.73) |
Standard error | 0.05 | 0.05 | 0.04 | 0.04 |
Median (min., max.) | 3.75 (1, 5) | 3.5 (1, 5) | 3.5 (1, 5) | 3.5 (1.25, 5) |
Frequency of event reporting | ||||
n (%) | 295 (95.8) | 257 (95.5) | 313 (92.1) | 297 (94.3) |
Mean (SD) | 4.1 (0.87) | 4.2 (0.85) | 4.0 (0.94) | 4.0 (0.94) |
Standard error | 0.05 | 0.05 | 0.05 | 0.05 |
Median (min., max.) | 4 (1, 5) | 4.3 (1, 5) | 4 (1, 5) | 4 (1.3, 5) |
Please give your work area/unit in this hospital an overall grade on patient safety, n (%) | ||||
Excellent | 58 (18.8) | 49 (18.2) | 65 (19.1) | 54 (17.1) |
Very good | 159 (51.6) | 152 (56.5) | 168 (49.4) | 142 (45.1) |
Acceptable | 69 (22.4) | 51 (19.0) | 86 (25.3) | 98 (31.1) |
Poor | 11 (3.6) | 12 (4.5) | 11 (3.2) | 17 (5.4) |
Failing | 4 (1.3) | 3 (1.1) | 6 (1.8) | 0 (0.0) |
No response | 7 (2.3) | 2 (0.7) | 4 (1.2) | 4 (1.3) |
In the past 12 months, how many event reports have you filled out and submitted?, n (%) | ||||
None | 92 (29.9) | 84 (31.2) | 126 (37.1) | 104 (33.0) |
1–2 | 74 (24.0) | 63 (23.4) | 85 (25.0) | 85 (27.0) |
3–5 | 58 (18.8) | 56 (20.8) | 62 (18.2) | 63 (20.0) |
6–10 | 48 (15.6) | 27 (10.0) | 23 (6.8) | 30 (9.5) |
11–20 | 14 (4.5) | 17 (6.3) | 13 (3.8) | 14 (4.4) |
≥ 21 | 8 (2.6) | 9 (3.3) | 12 (3.5) | 7 (2.2) |
No response | 14 (4.5) | 13 (4.8) | 19 (5.6) | 12 (3.8) |
Time point | Control (%) | Intervention (%) |
---|---|---|
Baseline | 70.5 | 74.7 |
12 months | 68.5 | 62.2 |
Health economic analysis
Resource use for all health- and non-health-related resources is presented in Table 42.
Resource use | Intervention wards (n = 17) | Control wards (n = 16) | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
n a | Mean | Median | Min. | Max. | n | Mean | Median | Min. | Max. | |
Collection of data (PMOS and PIRT) (average time in minutes) | ||||||||||
Time for questionnaires | 1261 | 15.59 | 15 | 4 | 70 | 1194 | 16.19 | 15 | 4 | 80 |
Total time patients | 1261 | 23.65 | 22 | 8 | 90 | 1194 | 24.02 | 22 | 8 | 100 |
Action planning meetings (average time in minutes) | ||||||||||
Meeting 1 | 17 | 46.23 | 40 | 27 | 80 | NA | ||||
Meeting 2 | 17 | 40.11 | 44 | 0 | 70 | |||||
Management of the intervention (average time in minutes) | ||||||||||
Start-up session | 17 | 120 | 120 | 120 | 120 | NA | ||||
Mid-point meeting | 17 | 120 | 120 | 120 | 120 | |||||
Closing-up session | 17 | 120 | 120 | 120 | 120 | |||||
Sum | Mean | Median | Min. | Max. | Sum | Mean | Median | Min. | Max. | |
Actions | ||||||||||
Total actions phase 1 | 53 | 3.11 | 3 | 0 | 10 | NA | ||||
Roles | 17 | 1 | 1 | 0 | 2 | |||||
Environment | 13 | 0.76 | 0 | 0 | 4 | |||||
Communication | 12 | 0.70 | 0 | 0 | 3 | |||||
Training and education | 4 | 0.23 | 0 | 0 | 2 | |||||
Behaviour | 1 | 0.05 | 0 | 0 | 1 | |||||
Procedures | 6 | 0.35 | 0 | 0 | 2 | |||||
Total actions phase 2 | 36 | 2.11 | 2 | 0 | 4 | NA | ||||
Roles | 12 | 0.70 | 1 | 0 | 2 | |||||
Environment | 4 | 0.23 | 0 | 0 | 1 | |||||
Communication | 12 | 0.70 | 1 | 0 | 2 | |||||
Training and education | 3 | 0.17 | 0 | 0 | 1 | |||||
Behaviour | 3 | 0.17 | 0 | 0 | 1 | |||||
Procedures | 2 | 0.11 | 0 | 0 | 1 |
The mean time spent with patients collecting data was similar between the allocated groups, with a non-significant difference of 0.36 minutes in favour of the intervention (95% CI –0.38 to 1.11; p = 0.34).
Action planning meetings that took place during phase 1 lasted on average 46.23 (SD 14.00) minutes and were attended by 3.82 (SD 1.81) members of staff. During the second phase those meetings were shorter [40.11 (SD 19.02) minutes] and were attended by fewer members of staff [2.23 (SD 1.25)].
A total of 89 action plans were produced. Of the 89 actions, 53 (60%) were planned during phase 1 and 36 (40%) during phase 2. There was only one ward that designed at least one action per category. Table 43 shows the focus of the action plans. One-third aimed to improve roles and responsibilities and over one-quarter aimed to improve communication and information exchange. The mean number of actions associated with roles and responsibilities was 0.85 (SD 0.65) per ward, with a maximum of two actions within this category. The median number of actions was zero for all classes of actions except for the roles and responsibilities and improving communication categories.
Focus of action plans | n (%) |
---|---|
Improving roles and responsibilities | 29 (33.0) |
Redesigning the environment | 17 (19.0) |
Improving communication and information exchange | 24 (27.0) |
Training and education | 7 (8.0) |
Changing behaviour | 4 (4.0) |
Implementing or changing procedures | 8 (9.0) |
Total | 89 (100.0) |
The results show that most wards considered that patient safety and quality can be developed by means of actions focused on improving roles and communication. It was common to find that staff roles were unclear or that patients were unsure which nurse or consultant was responsible for them. Actions relating to roles and communication frequently aimed to (i) ensure guidance and supervision in the absence of senior staff; (ii) ensure clear allocation of staff and planning of the team each day or (iii) define team roles and responsibilities for the prevention of falls and infections. Often actions were straightforward such as buying whiteboards to show different staff roles or bedside handovers with explicit definitions of ‘who is in charge’. Noise was another major concern for patients. Actions to reduce noise ranged from low-cost activities such as earplugs or soft-close bins to more complex activities that involved education of staff about the implications of noise for the patient experience.
The results of the cost–consequence analysis are shown in Table 44.
n a | Total cost (£) | Cost per ward (£) | Contribution to costs (%) | |
---|---|---|---|---|
Collection of patient data | 1261 | 24,463.80 | 1439.04 | 35 |
Phase 1 | 421 | 8691.18 | 511.24 | |
Phase 2 | 417 | 8291.02 | 487.7 | |
Phase 3 | 423 | 7481.68 | 440.1 | |
Action planning meetings | 17 | 4968.82 | 292.28 | 7 |
Meeting 1 | 17 | 2921.86 | 171.87 | |
Meeting 2 | 17 | 2046.96 | 120.40 | |
Management of the PRASE intervention | 17 | 5814.00 | 342.00 | 8 |
Start-up session | 17 | 1938.00 | 114.00 | |
Mid-point meeting | 17 | 1938.00 | 114.00 | |
Closing-up session | 17 | 1938.00 | 114.00 | |
Actions | 17 | 34,639.20 | 2037.60 | 50 |
Phase 1 | 22,111.20 | 1300.65 | ||
Roles and responsibilities | 17 | 5236.80 | 308.04 | |
Redesigning the environment | 17 | 4825.20 | 283.83 | |
Improving communication | 17 | 4840.80 | 284.75 | |
Training and education | 17 | 3382.80 | 198.98 | |
Changing behaviour | 17 | 360.00 | 21.17 | |
Changing procedures | 17 | 3465.60 | 203.85 | |
Phase 2 | 12,528.00 | 736.94 | ||
Roles and responsibilities | 17 | 2504.40 | 147.31 | |
Redesigning the environment | 17 | 916.80 | 53.92 | |
Improving communication | 17 | 4644.00 | 273.17 | |
Training and education | 17 | 1372.80 | 80.75 | |
Changing behaviour | 17 | 2354.40 | 138.49 | |
Changing procedures | 17 | 735.60 | 43.27 | |
Consequences | Difference between groups | 95% CI | p-value | |
PST (percentage of HFC) | – 0.04 | –3.60 to 3.53 | 0.98 | |
PMOS overall score | 0.08 | –0.01 to 0.17 | 0.09 |
Costs
The largest elements of costs were associated with the implementation of the actions (50%) and the collection of patient data (35%). Implementation of the actions had a mean cost of £1018.80 (SD £1500.03) per ward although this cost might be underestimated as action plans related to phase 2 were assumed to be partially implemented. As expected, the highest costs were associated with actions designed to improve communication (mean £278.96, SD £253.24) and actions relating to roles and responsibilities (mean £227.67, SD £206.68).
Wards spent on average £292.28 on action planning meetings, which cost on average £49.72 less than management meetings. This should be cautiously interpreted, as time for management meetings (start-up, mid-point and closing-up session) was not available and assumptions were made instead.
Consequences
There was no evidence of a difference between allocated groups in relation to either primary outcome (PST p = 0.98; PMOS p = 0.09).
Discussion
Summary of the key findings
The cluster RCT reported here ran smoothly. All recruited wards were retained throughout the study and data were collected for an average of 25 patients per ward, as planned. This, together with a lower ICC than expected (0.03 rather than 0.05), means that the trial was adequately powered to detect a medium effect size with respect to the primary outcomes.
Despite the ability to retain wards, adherence with the intervention was poor, with 11 wards adhering with 50% of scored intervention components and only four wards complying with 75%. Ward teams struggled to pull a multidisciplinary team together (particularly in the second cycle) and to develop and implement high-quality action plans.
The primary outcome of mean percentage HFC was high (> 90%), meaning that achieving a substantial increase was challenging. The percentage HFC also demonstrated some seasonal variation but not in the anticipated direction, such that scores were higher (better) at 6 months than they were at baseline or 12 months, despite the 6-month follow-up being in winter when one might anticipate that winter pressures would impact patient safety negatively. These unexpected findings require further investigation.
For the primary PST outcome there was no evidence of a difference between the allocated groups at 12 months (p = 0.99) or 6 months (p = 0.58), despite the intervention wards showing some improvement over time, which was not evident for the control wards. Results from the CACE analysis were consistent, suggesting that intervention wards that adhered to the intervention did not achieve a significantly greater improvement in safety than comparable control wards
There was also no evidence of a difference in HFC between the allocated groups at 12 months when considering only new harms arising during the care of patients on the study wards (p = 0.15), However, the difference between the two groups was larger (1.60%) for this outcome, with an effect size of 0.51. It is perhaps unsurprising that HFC occurring on the study wards themselves rather than elsewhere in the hospital might be a more sensitive measure of a ward-based safety intervention. This is a more positive finding but still non-significant at conventional levels. However, the CACE analyses relating to this outcome identified a difference as large as 5.38% for wards adhering with 75% of the intervention. Thus, those intervention wards that engaged fully with the intervention showed a much larger increase in HFC (for new harms) than comparable control wards.
The second primary outcome was the PMOS questionnaire. This measure was developed during the course of the programme grant (see previous chapters). For this reason, the completion rate, reliability and validity of the PMOS measure were assessed again in this chapter. A high percentage of patients completed a sufficient number of items to receive an overall score (85.6%). Most missing data resulted from patients responding that items were not applicable. The majority of patients were facilitated by the researchers to answer the questions; those self-completing were generally younger and more likely to be male. The overall score showed reliability with high Cronbach’s alpha values at each time point (> 0.9) and predictive validity in known group comparisons such that percentage HFC was higher in wards where PMOS scores were higher.
For this second primary outcome there was no evidence of a difference between allocated groups in relation to the PMOS overall score at 6 or 12 months, although, again, the control group remained the same throughout, whereas the intervention group did show some improvement. CACE analyses also demonstrated a non-significant effect when considering adhering wards although the results do suggest a trend in favour of the intervention wards, with improvements in PMOS scores being largest for those wards that adhered to the intervention.
No significant differences between allocated groups were identified for the secondary outcomes of recommendation by friends and family and staff safety culture. However, the staff grading of the safety of the ward did show some interesting trends. At baseline 70.5% of control ward respondents rated their ward either excellent or very good, whereas at 12 months this proportion was slightly lower at 68.5%. In the intervention wards, 74.7% graded positively at baseline, whereas this dropped to 62.2% at 12 months. The greater decrease in perceptions of safety of the ward in the intervention group may be a direct result of obtaining feedback from patients that highlights possible areas for improvement or from attempts to make improvements on the basis of feedback that have been difficult to implement. Being non-significant, these changes over time may of course just be down to chance.
The main costs associated with the intervention arise from staff time (i) from involvement in the design and implementation of actions and (ii) associated with patient data collection. It is worth highlighting that future models for the PRASE intervention might use volunteers to collect patient data, which would reduce the overall cost of the intervention.
Cost–consequences analyses are not commonly conducted to recommend whether an intervention is worth funding as that judgement is usually related to the gains that would likely be achieved by the patients. However, the intervention was not found to result in any significant differences for outcomes. Combined with the cost of delivery of approximately £2040 per ward and the cost of implementation of £1018.80 per ward it is highly unlikely that the intervention is worth funding. Consequently, given that the opportunity costs are likely to exceed the benefits, policy makers may wish to consider other options for investment to improve health gains for NHS patients.
Interpretation of findings
The main finding of this trial is that an intervention based on feedback from patients to wards about the safety of their care was not effective in improving patient safety during the 12-month study period. There are many possible explanations for this but the two most probable – measurement and fidelity – are discussed in the following sections.
Measurement issues
Patient safety is universally acknowledged to be difficult to measure,277 particularly when the intervention is upstream with the potential to impact on a wide range of safety and quality outcomes. In this study, one of our primary outcomes was the incidence of four patient safety harms on our participating wards. This outcome became available during the development of this programme and, representing routinely collected data available at ward level, seemed to represent a more valid and reliable outcome than other measures such as incident reports. Including percentage HFC as a primary outcome had both advantages and disadvantages. By the time of our trial, trusts were routinely reporting these data and so missing data were rare. Being a new measure, and one that is incentivised, there is the possibility of reporting bias; however, we deemed this to be of low risk because it was unlikely that bias would be more prevalent in one of our two groups. Despite these advantages, little is known about the validity and reliability of these data as the measure has been in use only since 2012.
The intervention provides staff with feedback on the safety of the care environment, to which they are asked to respond by making local changes and flagging broader issues to senior management. This makes it difficult to predict in advance what changes a ward will choose to make and therefore what outcomes it might be appropriate to measure. So, for example, one ward might focus on noise at night because patients report this as a particular problem, whereas another ward might focus on delays and another might focus on staff roles and responsibilities. Improvement in these areas might not be possible to pick up using the measurement of typical patient safety outcomes, for example pressure ulcers, falls or VTEs. For this reason, we also included the PMOS questionnaire results themselves as an outcome measure. We were unable to detect significant differences between the intervention group and the control group on this patient measure although this was much more closely aligned with the actions taken by staff to improve care. Again, however, changes made by the ward might be in response to just a few items on the questionnaire or a cluster of incidents reported by patients. These improvements might be difficult to capture using the PMOS measure.
Midway through the trial and with a better understanding of how the HFC score was being utilised and reported, it became clear to us that the ‘new harms’ score would have been a more appropriate primary end point. This is because, whereas total harm takes into account pressure ulcers, VTEs and catheter-associated infections that have occurred prior to the current admission to the ward, ‘new harms’ are those harms that have occurred after admission to the ward (e.g. for pressure ulcers, those that have developed ≥ 72 hours after admission to the ward; for UTIs, those that are identified any time after admission to the ward; and for VTEs, when treatment started after admission to the ward). In other words, whereas total harms might reflect poor patient safety elsewhere in the hospital, new harms can be argued to be those that are within the control of the admitting ward. The primary outcome for analysis was based on ‘all harms’ but, with hindsight and more experience of the PST data, the ‘new harms’ would have been more appropriate as a primary end point. Although a more appropriate primary outcome measure, guidelines for the design and reporting of RCTs require that the primary outcomes are specified in advance of the trial, meaning that we were unable to make amendments to the reported outcome post hoc. In addition, fewer ‘new harms’ are reported, which means that scores on this outcome are high and it is even more challenging to show differences between intervention and control groups for this measure. Therefore, to have sufficient power to detect differences between the intervention group and the control group would have required more wards to be allocated to each group.
Fidelity of intervention delivery
Following published guidance272 we collected information on the delivery of the intervention, which was coded to provide a quantitative estimation of the fidelity on each of the wards. The majority of wards met to consider feedback from patients and many of these APGs included different levels of nursing staff and members of the wider ward staff, for example occupational therapist, dietitians and ward clerks. However, only three of the action planning teams included doctors of any grade and nine action planning meetings were attended by only two people. One of our aims in this intervention (see programme theory in Chapter 7, Developing a logic model) was to achieve a number of proximal outcomes, namely shared, multiprofessional understanding of the patient perspective of safety, shared action planning and shared implementation of actions. Although these proximal outcomes of the intervention were partially fulfilled in some cases, the lack of medical input in most cases means that we can conclude only that the intervention was not implemented as intended and this may, in part, explain why the intervention was not successful. Despite this, all wards, with the exception of one, developed an action plan. The quality and implementation of these plans was highly variable and this is discussed further below. Appendix 8 shows some examples of action plans.
One interesting finding is that for each of the outcomes, and particularly for the outcome of new harms (those for which the ward are accountable), the CACE analysis demonstrated larger differences between the allocated groups as adherence improved. This implies that when adherence is high and the ward is able to produce good-quality action plans that are then implemented there is potential for safety to improve. Unfortunately, within the current cohort of 17 intervention wards, only four were able to comply at this level. A large number of recruited wards became involved at the request of the senior management team in the trust. When available for use on a voluntary basis, it remains to be seen whether compliance with the intervention and effectiveness may be enhanced.
In 2013, Shekelle and colleagues278 produced a systematic review culminating in recommendations for which patient safety interventions were very strongly or strongly encouraged for adoption based on current evidence. Interventions involving patient feedback or any other kind of patient involvement did not appear on the list, despite growing interest in the potential role of patients in improving patient safety. In recent years a number of studies have investigated direct patient engagement in patient safety, for example patient encouragement of hand hygiene practices or involvement in escalation to the critical care outreach team. A systematic review of patient engagement strategies concluded that there is currently ‘insufficient high-quality evidence informing real-world implementation’ (p. 548). 279 The same conclusion was reached by Mockford et al. in the UK in 2012. 280 There is currently no evidence available that addresses the specific question of whether patient feedback on patient safety has an impact on patient safety and therefore direct comparisons with other similar studies are impossible.
We can, however, turn to other areas of health care in which the impact of patient feedback on practice of health-care professionals has been evaluated for comparison with our work. For example, a review by Boyce and Browne281 found that there was only weak evidence that feeding back patient-reported outcome results to physicians had any impact on patient outcomes. This is supported by a recent systematic review which found that rarely are service outcomes evaluated, but that patient feedback of patient-reported outcomes did have a small to medium effect on the discussion of outcomes during consultations, with some studies showing a reduction in reported symptoms. 282 However, the authors of this study also concluded that clinicians need additional support for how to respond to patient feedback. In another National Institute for Health Research (NIHR)-funded programme grant led by Martin Roland (reference number RP-PG-0608-10050), the impact of patient feedback on general practice is also being evaluated, although this has not yet been reported.
The current and previous studies on the feedback of patient-reported outcomes suggest that patients are willing and able to provide such feedback. What may be more difficult to address is the engagement of staff with this feedback and the use of this feedback by health-care teams to improve services. As with the studies reported above, we found that staff needed additional support to respond to patient feedback.
For most staff on most wards, the difficulty in engaging with the feedback that they received from patients was not that they did not value what patients had to say, but rather that they felt powerless to make the changes necessary to deliver improvements. In fact, staff struggled to implement what seemed like relatively easy changes if they involved the need for support from others in the organisation, such as pharmacy, estates, information technology services or portering services. This meant that wards sometimes did not attempt or failed to achieve changes that might have the most significant impact on safety. For example, staff struggled to negotiate with pharmacies to develop a directive that would allow them to give painkillers to patients while waiting for prescriptions to be written up or issued. This meant that they were unable to address one of the problems that patients reported, which was that they waited too long for pain medication. In other cases, resources, for example a table and chairs to allow patient notes to be written in the bays, were not forthcoming and staff were not able to implement this change to make the nurse in charge of care more visible to patients. Bureaucracy stymied the implementation of other relatively simple action plans such as developing patient information sheets. We witnessed a sense of powerlessness among ward staff to make change happen. However, in our final feedback meetings, still ongoing, senior staff did appear interested in and supportive of cross-ward changes that arise from patient feedback. It may be the case, therefore, that some further impact of our intervention will occur post trial completion. These findings resonate strongly with the recommendations of a recent report by Dixon-Woods et al. ,283 who evaluated the Health Foundation’s Safer Clinical Systems initiative. They conclude that:
Small scale locally led improvement work has an important role to play in improving patient safety. However action also needs to be taken at an organisational and system level, to address problems beyond the control of clinical teams. Without it, there will be a limit to the level of improvement that can be achieved at the frontline.
www.health.org.uk/publication/safer-clinical-systems-evaluation-findings (accessed 8 August 2016)
Although for most staff, patient feedback was deemed an appropriate basis for making change, for the minority this was not the case. Two of the 17 wards were largely disengaged and disinterested. These wards went through the motions by holding action planning meetings, but did little else. Indeed, one ward was outwardly hostile towards the ethos of the intervention, highly critical of the comments that patients had made to researchers and defensive of staff members and the ward itself. Neither ward made action plans.
With this in mind, we would argue that for the PRASE intervention to be successful there would need to be a greater degree of action at the organisational and system level. Although we tried to encourage this by having an orientation, mid-point and final meeting attended by senior leaders in our participating trusts, by deploying tools that encouraged people to think at a system level and by providing feedback about upstream contributory factors such as environment, training, roles and responsibilities and equipment, our experience was that local teams tended to focus on short-term local fixes to problems that they felt they had some control over. Teams appeared to lack the confidence or will to make more ambitious plans for change, possibly because these changes are so much harder to achieve, something that was evident when they did attempt to drive these changes forward. This may have been exacerbated by the lack of medical involvement in the action planning teams. This interpretation also finds support in the staff safety culture survey findings, which showed a greater decrease in the grading of safety on intervention wards (12%) than on control wards (2%) over the 12 months of the trial. This may be because staff in the intervention group became more aware that improvements to safety were required but they felt unable to fully address these problems.
Limitations
This trial was adequately powered and conducted across different wards from a range of hospitals. Data collection proceeded as anticipated. One of the key limitations of this trial was that we did not collect information about the impact of completing a questionnaire about safety on the patients themselves. It is possible that completing such a measure might lead to improvements in patients’ knowledge and understanding of safety, which may change behaviour during the current or subsequent hospital visits. Our model for change focused on the feedback given to staff and the changes in their behaviour rather than patient-mediated change. Although we attempted to blind research nurses collecting the data, several wards identified themselves as being intervention wards through comments about engaging in action planning. Although unlikely, this may have had an impact on the way that research nurses asked questions when collecting feedback from patients.
Generalisability of the findings
We attempted to recruit three very different NHS trusts to this trial (large, medium and small; teaching and non-teaching; one and two or more sites) and to involve a range of different types of surgical and medical wards to make the findings generalisable across the acute care sector. However, we did not engage any paediatric, emergency, critical care or maternity wards/units and therefore it is not possible to generalise our findings to these populations. Critical care patients are unlikely to be able to provide feedback because of their status as seriously ill patients. Patients in accident and emergency care departments stay for only a short period and may also be seriously ill, meaning that the intervention may not be appropriate in these settings. Although possible to deploy the intervention in maternity and paediatric wards, none was recruited into the current study. This may be because patient engagement is already strongly promoted in these settings, but it is possible that this cultural maturity may serve to enhance the impact of an intervention such as the PRASE intervention and therefore evaluation of this intervention in these settings is important.
Conclusion
This intervention, based on collecting information from patients on the safety of their care environment and feeding this back to staff, was not effective based on our primary outcome measures. A secondary analysis revealed that for those wards that were complying with the intervention and in which only new harms were measured, the impact of the intervention was greater. This is worthy of further exploration.
Together with the cost of delivering the PRASE intervention, these two factors would suggest that it is highly unlikely that the PRASE intervention is worth funding.
The collection of data from patients was feasible, with 86% of patients approached willing to respond. The main challenge in implementation was the development and implementation of action plans by ward staff to improve safety.
Chapter summary
This chapter has presented the findings from an ambitious cluster RCT testing the efficacy of a novel patient-centred patient safety intervention. Although the authors found no direct effects on the patient safety outcomes measured, what is clear is that patients are willing to provide information that supports the proactive assessment of risk, as well as the retrospective identification of safety concerns, and that this combination of data can be used as a basis for service improvement. However, there are other potential ways that patients can be involved in improving patient safety within hospital settings and the next chapter focuses on one of these – directly engaging with health-care professionals to improve safety and reduce harm.
Chapter 9 Direct engagement: developing and piloting the ThinkSAFE intervention
Abstract
Background: Evidence suggests that existing initiatives to promote a patient role in improving their own safety have poor acceptability to patients, relatives and health-care staff (users) and that they have the potential to damage the patient–professional relationship. Previous initiatives have lacked user involvement in their development, a theoretical underpinning and evidence of effect.
Aim: To develop an intervention(s) that supports patients to work collaboratively with health-care staff to improve safety and reduce the risk of harm.
Methods: Systematic development over three phases, guided by the MRC framework, involving users at every stage: (1) evidence collation: scoping of evidence, theory and best practice and qualitative methods to elicit user experience; (2) intervention development: iterative, codesign workshops with users; and (3) evaluation study: controlled, pre–post exploratory trial using mixed methods to assess feasibility and the impact on user motivation and behaviour and on the frequency of medication errors at admission and discharge.
Results: Phases 1 and 2 identified the need for a supported, collaborative approach. Four components for ThinkSAFE emerged: a patient safety video; a patient-held health-care logbook incorporating tools to facilitate information sharing; a theory- and evidence-based educational session for staff; and Talk Time, a dedicated opportunity for users to interact. Evaluation showed that all users were highly motivated to engage with the ThinkSAFE intervention, but we were unable to demonstrate a measurable impact of the intervention on participant-reported behaviour, possibly because of ceiling effects but also partly because of low questionnaire response rates. Triangulation of findings from interview data did, however, suggest some impact on staff and patient interactional behaviours. In this pilot evaluation, admission prescriptions on intervention wards contained fewer medication errors. Intervention fidelity was problematic.
Conclusions: The ThinkSAFE intervention is a robustly developed initiative that is grounded in user experience and best evidence and underpinned by a strong conceptual and theoretical rationale. The approach is acceptable to users and has the potential to improve patient safety. Although aspects of the evaluation also suggest that the ThinkSAFE intervention is feasible, important structural and organisational barriers were identified that need to be addressed in future models of implementation.
Chapter rationale
Chapters 2–8 outlined the development and testing of a patient safety intervention that uses as its foundation information provided by patients about their experience of safety in an acute hospital setting. The findings suggest that patients are in a position to provide information about their experience of safety and that this feedback can be meaningfully used by health-care professionals as a basis for making patient-centred service improvements. However, another avenue for the role of patients in patient safety is through direct engagement with staff and systems, to improve safety and reduce the risk of personal harm. This chapter presents the development and piloting of the ThinkSAFE intervention, an intervention designed to support a collaborative approach between health-care professionals and patients in the management of safety in acute hospital settings.
Introduction
Background
A number of current patient safety improvement initiatives prompt patients to be vigilant about their health care and to intervene directly if they have a concern or think that there has been a mistake (e.g. the Speak Up initiative in the USA39 and the Please Ask campaign in the UK40). A common goal of these patient-mediated initiatives is to ‘push’ improvement by encouraging patients to ‘check’ the care that they receive and to speak directly to staff if they have any concerns or questions. Evidence suggests that patients would be willing to take on such a role in improving their safety284,285 but that this is contingent on a number of factors, including illness severity and type,286,287 the proposed action12,179 or question12,13,179,288,289 and their self-confidence. 222,290 Patient involvement also depends on the receptiveness of staff (real or perceived)291–297 and related concerns about damage to the patient–provider relationship. 297 This questions the acceptability and effectiveness of patient-push approaches and highlights a need to better understand how best to involve patients in improving their own safety.
A key criticism of initiatives using a patient-push approach has been the apparent lack of involvement of patients, their families or frontline staff in their development, a lack of theoretical rationale for the choice of intervention approach or materials and a dearth of robust evaluation. This project aimed to address this major gap through the systematic development of a fully piloted intervention grounded in patient and professional experience, underpinned by relevant theory and informed by current evidence and best practice.
Ethical statement
The research was approved by the Newcastle and North Tyneside 2 Research Ethics Committee (reference 10/H0907/24 for phases 1 and 2) in May 2010 and by the National Research Ethics Service (NRES) Committee (proportionate review) East of England – Norfolk (reference 12/EE/0277) in June 2012. NHS research and development research governance approval was obtained from both participating trusts for work undertaken from May 2010 and from June 2012.
Clinical setting
The clinical setting was UK secondary care, involving two NHS foundation trusts in the north-east of England. Patient and staff participants were recruited from acute and elective general medical and surgical wards across a range of clinical specialties.
Aim of the overall project
The aim of this study was to develop an intervention(s) that supports patients to work collaboratively with health-care staff to improve safety and reduce the risk of harm.
Design of the overall project
Guided by the MRC framework for developing and evaluating complex interventions267 the project progressed across four phases, using methods appropriate to a series of objectives relevant to each phase (Table 45).
MRC framework stage | Study phase | Study activity |
---|---|---|
Development | Phase 1 | Evidence collation:
|
Development/feasibility | Phase 2 | Intervention development:
|
Feasibility/evaluation | Phase 3 | Exploratory trial:
|
Evaluation | Phase 4 | Service-level trial:
|
Phase 1: comprehensive evidence collation of resources to inform the content and form of a prototype intervention
International scoping exercise of ongoing work to identify best practice
A scoping exercise was undertaken of local, national and international ongoing initiatives, including identifying any evidence of piloting or evaluation, theoretical underpinning and/or evidence of patient and family involvement in the development of the identified intervention approaches and materials (Table 46). Initiatives were identified through personal contacts with organisational leads and experts and by visiting organisations’ websites and related external links.
Organisation | Initiative | Evidence identified for | ||
---|---|---|---|---|
Evaluation | Theory | Patient/public involvement | ||
International | ||||
WHO World Alliance PfPS | Patient champion networksa | No | No | As champions |
WHO Collaborating Centre for Patient Safety | High 5’s standard operating proceduresb | ‘Impact evaluation strategy’ in progress | No | No |
The Joint Commission | Speak Up campaignc | ‘High uptake’ suggested by leaflet download statistics | No | Patient/family advisory group |
AHRQ | Quality and Safety programmed | Planned | No | Patient/family advisory group |
Partnership for Patient Safety (P4PS) | Consumer-led organisatione | No | No | Consumer led |
Consumers Advancing Patient Safety (CAPS) | Patient/provider coalitionf | Evaluation showed ‘the value of community-based patient–provider partnerships’ | No | Local coalition of patients and physicians |
Coalition for Quality and Patient Safety of Chicagoland (CQPSC) | A federally certified patient safety organisation. Learning from error information shared between member organisationsg | No | No | No |
Canadian Patient Safety Institute (CPSI) | Safer Healthcare Now!h | Two-phase evaluation: implementation and uptake (ongoing) | No | No |
Manitoba Institute for Patient Safety (MIPS) | It’s SAFE to ASK!i | No | No | No |
Australian Commission on Safety and Quality in Healthcare (ACSQH) | Several national patient safety campaigns were under development (now national safety and quality standards)j | No | No | No |
Danish Patient Safety Agency (DPSA) | Patient Safety Handbookk | Due to begin 2010 – mixed focus groups with staff and patients/relatives | No | Patients not consulted. Interviewed 10 doctors and nurses |
UK national | ||||
NPSA | Please Ask; cleanyourhands campaign; National Reporting and Learning System | NPSA unpublished feasibility pilot: need for right environment, with staff ‘giving permission’. NOSEC evaluation:298 measurable change in hand hygiene practice but unable to determine specific influence of cleanyourhands campaign. Patient empowerment less successful. Impact on heath care-associated infections too early to say | No | Patient reporting system: patient involvement via LINks and AvMA, patient safety champions networks |
The Health Foundation (THF) | Safer Patients initiative, later Working with Strategic Health Authorities; Safer Patients Network; Safer Clinical Systems | Built on success of previous Safer Patients Initiative | No | No |
National Institute for Health and Care Excellence (NICE) | Patient guidelines | No | No | No |
Care Quality Commission (CQC) | Quality and Safety Standards | No | No | No |
NHS Institute for Innovation and Improvement (NII&I) | Safer Carel | Planned | No | No |
AvMA | PfPS, PfPS Champions | Allford299 | No | 22 champions from a variety of backgrounds and with a range of professional and personal experience |
Regional | ||||
North East Strategic Health Authority | Safer Care North East; Patient Safety Strategic Forum; Patient Safety Action Team; Patient, Public and Community Engagement | Planned | No | Public, community and patient engagement: network of patient representatives from regional patient safety groups (e.g. PfPS Champions, patient and public involvement, LINks) |
Examples of intervention materials identified as a result of the scoping exercise were used to elicit patient and staff perceptions about the use of these approaches by patients in the UK NHS acute care setting during subsequent qualitative interviews (see Qualitative study). The learning and resources were also used during the intervention development phase (phase 2) to guide the form of the prototype intervention(s) and the chosen mode of delivery.
Systematic review
A previous systematic review of the effectiveness of interventions to involve patients in improving patient safety identified 15 eligible studies reporting on a range of interventions to improve medication safety. 180 This original review found evidence in favour only of patient self-management of medication (mainly in relation to anticoagulant therapy), but little evidence for other approaches to improving safety. The review was updated and expanded to:
-
identify evaluations of interventions published subsequent to the original review
-
extract data across the full review relating to the process of intervention development and the extent, if any, of patient involvement in this process
-
extract data relating to behaviour change techniques (BCTs) used in the interventions described in all included studies, with a focus on BCTs associated with successful interventions [defined for the purposes of this work as present more frequently (> 50%) in positive studies than in negative studies].
Methods
The original search strategy was amended to cover the time period from the close of the previous search (August 2008) to September 2010 and then rerun in the same databases. Potentially relevant articles and reviews identified were first sifted by title and then by abstract, followed by full-text review against the study inclusion criteria by two researchers. Disagreements were reconciled by a third researcher at each stage. Full data abstraction of the final included studies then followed.
Results
Three additional studies were identified,300–302 each focusing on medication safety, resulting in a total of 18 studies reporting evaluations of interventions to promote patient involvement. Two studies examined discharge counselling300,301 and reported a reduction in medication-related readmission and post-discharge service utilisation. The third examined the impact of large print medication labels on patient-mediated medication errors302 and found no difference in condition-specific outcomes at follow-up. Guided by a taxonomy of BCTs,303 12 techniques were identified (Table 47). The most frequently used technique [used in 14/18 (78%) studies] was to ‘provide instruction’ (BCT 4); the next most frequent were ‘provide general information’ (BCT 1) and ‘provide information on the consequences (of doing the behaviour)’ (BCT 2) [both 6/18 (33%)]. As the use of the techniques was not exclusive to successful interventions, no strong conclusions could be drawn to guide the choice of BCTs for the emerging intervention. Two techniques were present more frequently (> 50%) in studies reporting positive outcomes [‘provide information on the consequences (of doing the behaviour)’ (BCT 2) and ‘use of follow-up prompts’ (BCT 11)]. Three of the eight studies reporting in favour of the intervention used BCTs not used in any studies reporting negative or inconclusive outcomes (BCTs 3, 7 and 12).
BCTs303 | Studies identified in the updated review | Studies identified in the original review180 | Total using BCT | % using BCT | More frequent in positive studies | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Shrank et al.302 | aWalker et al.301 | aJack et al.300 | Atkins et al. | Connock et al.a | Fisher et al. | Kennedy et al.a | Kim and Grier | Leya | Mckellar | McMahon | Neafsey | Pereles et al.a | Punekar | Schnipper et al.a | Van haecht et al.a | Varkey | Weingart et al. | ||||
1. Provide information about behaviour–health link (in general) | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | 6 | 33 | No | ||||||||||||
2. Provide information on consequences | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | 6 | 33 | Yes | ||||||||||||
3. Prompt barrier identification | ✗ | ✗ | 2 | 11 | Yes | ||||||||||||||||
4. Provide instruction | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | 14 | 78 | No | ||||
5. Model or demonstrate the behaviour | ✗ | ✗ | ✗ | ✗ | 4 | 22 | No | ||||||||||||||
6. Prompt specific goal setting | ✗ | ✗ | ✗ | 3 | 17 | No | |||||||||||||||
7. Provide feedback on performance | ✗ | 1 | 6 | Yes | |||||||||||||||||
8. Teach to use prompts or cues | ✗ | ✗ | ✗ | 3 | 17 | No | |||||||||||||||
9. Agree on behavioural contract | ✗ | 1 | 6 | No | |||||||||||||||||
10. Prompt practice | ✗ | ✗ | ✗ | 3 | 17 | No | |||||||||||||||
11. Use follow-up prompts | ✗ | ✗ | ✗ | ✗ | 4 | 28 | No | ||||||||||||||
12. Plan social support or social change | ✗ | 1 | 6 | Yes | |||||||||||||||||
Total BCTs per study | 3 | 6 | 8 | 3 | 3 | 0 | 3 | 1 | 1 | 1 | 2 | 2 | 3 | 3 | 2 | 1 | 3 | 3 | Median = 3 |
Qualitative study
A qualitative study297 was undertaken to explore patients’, relatives’ (service users’) and health-care staff members’:
-
understanding and experience of patient safety and the potential patient/family role in harm prevention
-
attitudes towards, and beliefs about, patient and family involvement in improving safety
-
perspectives on how best to support the involvement of patients and families, including potential barriers and facilitators.
Methods
Face-to-face semistructured interviews were used to explore novel ways in which service users might play a role in improving their own safety, and to elicit respondents’ views about current approaches to promoting patient involvement in improving safety (see Appendix 9). Questions initially focused on three core areas of patient safety but were not restricted to these:
-
medicines reconciliation on admission to and discharge from hospital
-
hand hygiene and reducing hospital-acquired infections
-
identifying patient deterioration and alerting staff to concerns.
Interviews explored respondents’ understanding of ‘patient safety’; beliefs and attitudes towards patient involvement in improving their safety; what respondents felt was a feasible and acceptable role for patients; and how such a role might best be supported. Example materials from a range of current campaigns (UK and international) aiming to encourage patients to take a more active role in improving their safety were presented to respondents during the interview. Purposive sampling was used to capture a broad range of perspectives across different health-care contexts, with participants sampled from both acute and elective medical and surgical wards. 297
Participants and process
Eligible respondents were (1) patients who had recent experience as an inpatient on one of eight participating wards within the two participating trusts, (2) their relatives or carers and (3) staff providing care to patients on these wards (doctors, nurses, health-care assistants and pharmacy staff). Patients who did not speak English and patients who lacked capacity were excluded. Relatives of patients lacking capacity were eligible to take part. Service user participants were identified prior to their discharge by ward nurses, who introduced the study using the study information leaflet. The contact details of those expressing an interest were then provided to SH, who telephoned service users 1 week post discharge to complete recruitment. Staff participants were identified by ward leads guided by the purposive sampling frame. 297 All interviews were conducted by SH, in the homes of service users and on hospital premises for staff participants. Service user interviews lasted for 1 hour on average and staff interviews for 30 minutes. All interviews were audio recorded and transcribed verbatim. 297
Analysis
Interview transcripts were analysed iteratively by SH using a grounded theory approach,304 supported by the use of NVivo 9. 305
Emergent themes were discussed at length with RT to develop a coding frame and to guide avenues of exploration in subsequent interviews. Codes were added or revised as new themes emerged. 306,307 Interviews continued until saturation was reached. 297
Results
In total, 16 patients (10 female, 6 male), four relatives (two female, two male) and 39 health-care staff (9 pharmacists, 11 doctors, 12 nurses and 7 health-care assistants) took part in an interview.
What do patients, their relatives and health-care staff currently understand about patient safety?
All interviews began with an exploration of the participant’s understanding of the term ‘patient safety’. Quite consistently, patient safety was presented in terms of personal safety (being safe from physical attack by other patients, safety of personal property) and health and safety (tripping over medical equipment, slipping in the shower, fire evacuation procedures). Once clarified as the risk of harm from the care they receive, understandings of patient safety were still largely restricted to hospital-acquired infection, risk of falls and compliance with medications. This limited understanding, particularly for service users, constrained participants’ ability to generate novel ideas for patient intervention. Examples of current approaches were introduced to the interviews at this point to elicit respondents’ views about participation in the patient safety behaviours promoted.
What do patients, their relatives and health-care staff think about the idea of involving patients in improving their own safety?
All respondents could see benefits from having patients more routinely involved in their care. Generally, patients welcomed the opportunity to ask questions, to be more informed about what was happening to them and to know what to expect during their stay in hospital. Staff reported that they welcomed patients’ questions and anticipated safer care through improved patient understanding and concordance with treatment regimes. Many patient and relative respondents were already heavily engaged with their care in the community setting within the context of self-management of chronic conditions and so felt a degree of capability to help keep themselves or a relative safe, if they were to be advised about how best to do this. There was also evidence of existing service user vigilance over care in the inpatient setting, although in the main this remained passive.
Concerns about involving patients in improving their own safety297
Despite this general positivity, some currently recommended behaviours or actions were seen as problematic by both staff and service users. In particular, service users expressed concerns about actions that encourage them to ‘check’ that their care is being delivered correctly and appropriately and engage directly with staff if they think that something is not quite right. This was felt to be ‘questioning’ or challenging the professionalism of staff and there was a concern that staff would interpret such intervention as criticism. Subsequently, service users felt that their intervention would not always be welcome and feared repercussions as a consequence of upsetting staff.
Several staff identified with this patient perspective, sometimes drawing on personal experience of being a patient or the relative of a patient. For staff, demands on their time and increased workload burden were very prominent concerns but their accounts also mirrored the patient perception that active patient identification of safety concerns involves ‘checking up on’ and ‘challenging’ staff. Some staff expressed this as anxiety about being asked questions to which they might not know the answers and others expressed it as frustration at having their professional status or integrity challenged. There was evidence of staff feeling scrutinised and criticised and of mistrust of patients who asked many questions. This suggests that pushing improvement through patients, although well intentioned, can inadvertently create negative tensions in the patient–provider relationship and lead to a reciprocal erosion of trust. An implication of this is a counterproductive mutual distancing and avoidance of effective patient–provider interactions.
What patient safety behaviours do patients, relatives and health-care staff think are feasible and acceptable?
Several mutually acceptable patient safety behaviours were identified (Box 6). Service users reported a willingness to engage in these behaviours, including vigilance over administered medications and reminding staff about hand hygiene. It was important to patients that their involvement was in the context of helping staff to enhance safety. Traditional role expectations – staff as experts, service users as ‘lay persons’ – were still prevalent. Issues of vulnerability and capacity because of illness were also raised. As well as underlining the need for a collaborative approach, the qualitative study demonstrated that service users perceived their involvement as continuous, rather than as an isolated act at a specific point during their hospital stay. Service users envisaged making different contributions at several time points across the inpatient pathway.
-
Asking about medications and why patient takes them.
-
Keeping an up-to-date list of medicines.
-
Confirming the accuracy of medications at admission.
-
Asking about unfamiliar medicines administered at drug round.
-
Learning about and reporting side effects of medicines.
-
Confirming the accuracy of medicines given at discharge.
-
Adhering to recommended hygiene protocols.
-
Reminding staff to wash their hands.
-
Avoiding falls by being aware of hazards on the ward.
-
Avoiding falls by patient knowing own limitations.
-
Reporting suspected deterioration or change in patient.
-
Confirming and marking the correct surgical site.
-
Asking about what to expect following discharge.
-
Relative acting as the patient’s advocate.
Conclusions
This qualitative work indicated that patients are willing to engage in enhancing their safety provided that staff are receptive to their involvement and that it is done in the context of helping and not checking. This reiterates the importance of staff approval, but also suggests that this is not always sufficient; patients need to be actively encouraged by staff to engage in involvement behaviours. Active patient participation in safety behaviours challenges the traditional boundaries of patient–professional role expectations and the way that patients and professionals interact. A central aim for the study intervention was therefore to support this new way of working together, based on a shared aim to ensure safer care. These relational difficulties are confounded by reports from all respondents that opportunities for important patient–staff interaction are greatly limited by a lack of time, heavy staff workloads and competing priorities. A further aim for the study intervention was to address these more pragmatic barriers to patient involvement in patient safety behaviours.
Novel ideas workshops
Participants in the qualitative study struggled to come up with novel ideas for involving service users in improving patient safety.
To address this, two brainstorming workshops, one with experts in the field of safety in health care, ergonomics and human factors and one with expert patients including regional Local Involvement Networks (LINks) leads and patient safety champions, were held. Using a specific creative thinking technique called ‘bootlegging’,308 these workshops generated ‘out of the box’ thinking and a number of novel approaches to using technology and social marketing to promote and support patient involvement (Table 48). A broad theme across ideas generated within both workshops was again of facilitating the exchange of information between patients and relatives and their health-care providers.
Experts in the field of safety | Expert patients |
---|---|
Patient self-testing at home for a range of conditions (e.g. diabetes, blood pressure, urine) using a community-based biometric sensor. Test results are then communicated to a remote clinical advisor for advice and to initiate potential action | Engagement of homeless people in getting tested for long-term conditions. Attach information to prescriptions or to a beer can and other alcoholic drinks. Testing could be provided via a mobile surgery and community clinicians attending a soup kitchen or hostel |
Use of video calling technology to facilitate direct face-to-face contact/communication with relatives, carers or another advocate at the press of a button. The advocate could reinforce questions that the patient needs to ask as well as contribute to a discussion with their clinician A mobile interactive station in the clinic waiting room. Patient provides information prior to the consultation. The system generates and prints off a list of issues and/or questions to ask the clinician and also sends a copy of this list to the consulting practitioner Use of mobile apps to prompt/alert a family member to inform health-care providers of any allergies that the patient may have or prompt for details of current medications and other important information |
Online pre-admission booking service for secondary inpatient care. Patients could select meals and a bed and take a virtual tour of the ward to show them the ward layout and fire exits, introduce them to staff, etc. Patient safety information included in the tour, supported by a checklist of important things to ask and tell staff about |
Patients given a copy of the referral by their general practitioner (e.g. after a consultation at which a decision has been made to refer them). Patients would better understand why they were referred. An included tip sheet prompts questions they should ask at their referral appointment and signposts to patient advocacy services if required |
Identification of relevant theory
Previous work has found that few studies reported a theoretical underpinning for the choice of intervention approach. 180 A theoretical rationale provides understanding of how interventions are intended to work (i.e. the proposed mechanisms of change) and where and when they might have their effect (i.e. at what point in the causal chain). To guide the systematic development of a theoretical basis for the project intervention a three-step process was used. 309
-
Key themes and related beliefs identified by the qualitative study were mapped onto behavioural dimensions within the theoretical domains framework (TDF)310 (Figure 14). The TDF is a tool derived from psychological theories of behaviour and suggests behavioural determinants that can then be directly targeted within the study intervention. Six TDF domains were relevant to understanding the asking and telling behaviours to be promoted in patients and five were relevant to understanding the ‘fostering engagement’ behaviour to be promoted in staff:
-
domains relevant to both:
-
motivation and goals
-
beliefs about capabilities
-
beliefs about consequences
-
social/professional role and identity
-
nature of the behaviour
-
-
additional domain relevant to patients:
-
social influences.
-
-
-
To further support targeted behaviour change, behavioural domains identified in step 1 were then mapped on to relevant BCTs303 using a related matrix311 (see Figure 14).
-
Identifying relevant theory involved mapping the domains identified as key to explaining patient involvement behaviour back to constituent theories (Figure 15). Systematic methods for doing this are less well developed312 but a number of established theoretical models of behaviour (listed in the appendix of Michie et al. 310) were identified as potential frameworks for understanding patient and staff behaviour in relation to patient involvement in improving patient safety. Constructs within TDF domains identified by step 1 were present in two theories of motivation and three theories of action (see Figure 15). Theories identified included the theory of planned behaviour,313 social cognitive theory,314 self-regulation theory315 and learning theory. 316 The rationale for selecting these theories was supported by evidence on their previous use in studies examining patient involvement in improving patient safety222 and as models for understanding and changing patient and health-care professional behaviours309,317–319 and by their direct linkage to BCTs (social cognitive theory and self-regulation theory). 311
A further theory – role theory320 – which is not included in the TDF list (as it is not a theory of behaviour change), was also included as part of the analytical framework, as most of the themes emerging from the qualitative study related to traditional patient and health-care role perceptions. Support for the utility of this theory for understanding the patient and relative perspective in relation to a patient role in improving safety is provided by previous qualitative work in this area. 291
These theories formed the basis for the initial analytical framework for the pilot evaluation trial undertaken in phase 3 of the study. The framework facilitates modelling of the promoted patient and health-care professional patient safety behaviours and testing of the hypothesised relationships between behavioural antecedents and behaviour, and behaviour and patient safety indicators.
Phase 2: iterative codevelopment and design of the study intervention
Four iterative interactive workshops were convened with mixed and separate groups of patients recently discharged from hospital, their relatives and staff. Using codesign principles the workshops were structured to include presentation of broader findings from the qualitative study to provide context, followed by small group exercises and full group discussion.
During the first workshop participants were presented with lists of the ideas for intervention, identified as a result of the multiple evidence collation activities, along with suggested approaches to how these interventions might be delivered and potential formats and modes of delivery. Building on the ‘pathway’ approach identified during interviews with patients, workshop participants mapped patient safety behaviours, and ideas to support them, across a timeline that spanned pre-admission to discharge.
Participants worked in two groups to identify core intervention components, focusing at this stage on the pre-admission and pre-discharge time points. This exercise resulted in the concept of a pre-admission safety brief and what it might entail and of a patient-held toolkit – some form of document containing information, tools (e.g. medications checklist, question tip sheets) and a question log book – that would be with the patient throughout their stay in hospital. Important pre-discharge components were having someone to ask questions and discharge counselling, which includes, but is not limited to, medication counselling.
Subsequent workshops worked on the inpatient stay period. Core components felt to be important here were a ‘meet and greet’ session on admission to wards and a dedicated time for questions. Later workshops elicited staff and patient feedback about the full conceptual model with all proposed components in place along the inpatient pathway. Subsequent workshops further built on the outcomes of earlier sessions until saturation of ideas and consensus on the conceptual model was reached (Figure 16).
Potential barriers specific to the uptake of the proposed interventions – typically time, resources and additional workload – were identified. Facilitators were therefore actively sought during the final staff workshop by asking staff what they would need to do to support greater patient engagement and involvement using the proposed approach ‘on a ward like theirs’. The rationale for this was to encourage ownership of the process by the staff involved. Importantly, this process reiterated the need for a staff intervention, supporting the findings of the qualitative study and forming the basis for a fourth staff-directed component for the ThinkSAFE approach.
Four core intervention components of ThinkSAFE were thus derived from the above workshops:
-
A safety briefing video for service users (available pre-admission for elective patients or on the ward for acute admissions). The video would address specific educational and informational needs, as well as the facilitators of and barriers to patient involvement identified during the qualitative study, with the recommended behaviours presented at each relevant time point across the inpatient pathway.
-
A patient health-care logbook, linked to the video content, that would provide continuous support to patients throughout their stay in hospital. An A5 document was suggested with content that included sections for recording personal information, a safety tip sheet that reiterated the patient safety behaviours promoted in the video and a range of tools and prompts to support the promoted behaviours (see Figure 16).
-
Talk Time: dedicated one-to-one session with a member of the clinical team. These sessions were conceptualised as a legitimate space in which patients and/or their family members could ask questions or discuss concerns without feeling that they were interrupting busy staff. The health-care logbook, and in particular the question notebook component, was felt to be key in supporting dialogue during these interactions.
-
The staff intervention was envisaged as an educational session, supported by the patient safety video and informed by the findings of the qualitative study, highlighting the key role that staff have in encouraging (permitting) patient involvement.
Phase 3: development and piloting of the prototype intervention materials
Prototype intervention materials were developed by the study research team (including two patient representatives) guided by the conceptual framework and informed by examples of best practice. BCTs also informed the design and format of each individual component.
Intervention components
Patient safety video
The video promotes key behaviours in patients, relatives and staff. This is achieved in two ways. First, the codetermined patient safety behaviours are demonstrated as a series of interactions between service users and members of ward staff. Each behaviour is embedded within a brief scenario based on examples of experience elicited during interviews, for example the scene prompting patients to ask for clarification if they are unsure about what is happening to them arose from a patient’s account of her recent stay in hospital. Second, modifiable antecedent behavioural beliefs that act as key facilitators of or barriers to motivation and behaviour are directly targeted using evidence-based BCTs known to influence change in these factors. For example, the format of the video as brief sketches with patients and health-care staff acting out the target patient safety actions was informed by the technique ‘modelling/demonstrating the behaviour’. There is evidence that this technique promotes behaviour change by its influence on skills development and antecedent beliefs associated with self-efficacy and roles and identity. 311
Other BCTs incorporated into the video include ‘teaching to use prompts’ and ‘providing information about others’ approval’. A story board was first developed that contained brief scripted scenarios. The face validity of this was established by eliciting feedback from members of a NHS patient panel and a group of expert patients from the North East Strategic Health Authority Safer Care group before filming began.
Patient-held health-care logbook linked to the video content
An A5 version of the handbook was developed informed by currently available working models (e.g. the Danish Patient Safety Agency Patient Safety Handbook, Northumbria Healthcare NHS Foundation Trust Stroke Handbook). A range of tools is included. The contents of the logbook are organised into four sections.
-
‘Your personal Information’ – provides space to record pertinent medical history and a medication list template.
-
‘How to enhance your safety’ – includes images of key behaviours demonstrated in the video and tips about what sorts of questions to ask and when; pictures of target interactions are also displayed on a laminated ‘quick to view’ safety brief card in a format similar to the cards used by airlines.
-
‘Your questions and information about your care’ – includes an admission checklist; a week-to-view care calendar to record what patients might expect during their stay; a question notepad to write down questions; note pages to record information for future reference; and a discharge checklist.
-
‘General information’ – provides links to further (reliable) information and patient organisations.
Feedback on initial versions of the logbook was sought from groups of carers, patients and patient representatives (see Appendix 10).
Talk Time
It was envisaged that this component would vary in how it might be operationalised depending on current ward processes and ways of working. Wards could determine how best Talk Time could be incorporated into existing routines and schedules.
Staff intervention
A brief theory-based educational session consisting of five presentation slides and supported by the patient video was developed (see Appendix 15). Staff are prompted to foster patient engagement and involvement in their health care. Each slide was developed around relevant BCTs and each covers a specific issue.
-
Slide 1 – outlines the global problem of patient safety, setting it within its broader context, and provides information about the consequences of doing the behaviour (involving patients) in general.
-
Slide 2 – provides information about the consequences of doing the behaviour for the individual. This slide aims to provide a persuasive message by contrasting ‘old’ traditional roles with new collaborative roles and by outlining to staff the benefits to themselves as safe practitioners.
-
Slide 3 – provides a full description of the patient intervention components and how staff can use the ThinkSAFE approach to help patients help staff to improve patient safety. The content derives from the ‘teach to use prompts/cues (increasing skills)’ BCT.
-
Slide 4 – draws on the BCT ‘provide information about others’ approval’ to explain why patients are reluctant to engage directly with staff, highlighting the crucial nature of the staff role in providing ‘permission’ to service users to directly engage with them.
-
Slide 5 – uses the ‘prompt practice/rehearsal’ BCT and aims to increase skills by providing guidance and ideas about how staff can actively foster service user involvement in their care and safety. It specifically emphasises the need for staff to verbalise permission by saying to patients, ‘It is OK to ask me . . .’ or ‘I want you to tell me . . .’.
Pilot and preliminary evaluation of ThinkSAFE
Methods
The evaluation design was a non-randomised before-and-after study with a parallel control group (Figure 17). Twelve wards (six within each trust) took part in the pilot study; eight received the intervention and four acted as controls. Mixed-methods evaluation318–322 was used to investigate a range of process and outcome measures relevant to the intervention approach (Table 49). Process measures evaluated intervention fidelity, acceptability and feasibility and behavioural antecedents in both service user and health-care staff participants. Outcome measures evaluated the potential impact of the ThinkSAFE intervention on self- and proxy-reported service user and staff behaviours and on medication safety. Medication safety was chosen because of the higher rate of opportunity for patient involvement at different time points across the inpatient stay, the persistently high frequency of particular error types322–326 and the existence of robust, routine methods of measurement of these errors. 259 A power calculation based on published error rates300–302,321–324 for the outcomes targeted by the tracer topic suggested that 230 patient records per group would be needed (460 in total) to detect a difference in proportions of unintended medication omissions of between 50% and 36.5% (13.5% difference) with 80% power, allowing for a 10% loss to follow-up. Sample sizes to allow the reliable capture of safety events relating to hospital-acquired infection and patient deterioration were beyond the scope of a pilot evaluation.
Instrument | Informed by | Outcome measures | Process measures |
---|---|---|---|
ThinkSAFE patient safety questionnaire | Theoretical framework derived from phase 1 project work (theory of planned behaviour, social cognitive theory, learning theory, role theory) UK NHS and US CAHPS national surveys |
|
|
Medications reconciliation audit tool | Review of current audit tools and guidance (UK and international, e.g. NICE 2007; Bradford Institute 2011; ARQH 2008; International SOP 2011a) |
|
|
Interview topic guide | Normalisation process theory, TDF |
|
|
Evaluation framework | Implementation literature272,327 |
|
Measures (see Table 49)
-
Theory-based, self-completion questionnaires for both patients and staff captured pre-intervention (baseline) and post-intervention data for behavioural process and outcome measures (see Appendix 11). Items were measured on a 5-point Likert scale. To enable comparison of the evaluation data with existing survey data, relevant items from validated UK305 and US325 national patient and staff survey instruments were included in the post-evaluation questionnaire. The post-intervention questionnaire asked patient intervention respondents if they had received a logbook and if they had viewed the video/digital versatile disc (DVD).
-
A standardised medications reconciliation audit tool was devised in collaboration with pharmacy teams at both participating trusts, which captured frequency data on target medication errors (see Appendix 12).
-
Post-intervention, face-to-face interviews with patients and/or relatives and health-care staff exposed to the intervention explored user experience and the feasibility and acceptability of ThinkSAFE.
-
Data on the fidelity of the intervention approach was also captured both quantitatively and qualitatively (numbers of logbooks distributed, exposure of ward staff to educational session, staff and patient report of process and use of the intervention).
Setting
The pilot took place on six surgical and six medical wards across three hospitals within two NHS foundation trusts in the north-east of England. Purposive sampling was used to identify wards receiving both acute and elective admissions across a range of specialties and to match intervention and control wards on these features as far as possible. The evaluation began on all control wards on 1 September 2012 and ran without interruption to 14 December 2012. Each control ward aimed to sample around 60 patients at admission and at discharge. Start-up on intervention wards was staggered across this evaluation period, with each intervention ward aiming to sample 30–35 patients during a 2- to 4-week active intervention period.
Participants
Participants were adult elective or acute patients (≥ 18 years) admitted to the participating wards during a 2- to 4-week evaluation period for a stay of ≥ 2 nights (actual or anticipated). An opt-out, blanket approach was used in the delivery and evaluation of the ThinkSAFE approach. When patients lacked capacity or were too ill to participate in the evaluation study, relatives or carers were invited to participate. All staff (nurses, doctors, health-care assistants, ward pharmacists) working on participating wards during the evaluation period were eligible to take part.
Pre-intervention survey
Two weeks prior to their admission, patients on elective lists were sent a study pack containing a letter of invitation from the research team, information about the survey, the theory-based questionnaire and a pre-paid envelope for return of the questionnaire directly to the research team within 1 week, prior to receipt of the intervention materials. Patient questionnaires were returned anonymously.
All staff on participating wards (intervention and control) were provided with a study pack 3 weeks before the ward’s evaluation period. Staff questionnaires were marked with a participant code to enable matching of pre- and post-intervention questionnaire data.
Pre-intervention set-up and delivery of the staff intervention
Intervention wards were set up 2 weeks in advance of their evaluation period to enable the organisational-dependent aspects of the intervention process to be arranged (e.g. time within current routines to provide the Talk Time sessions) and advance rehearsal of the intervention process. Following the pre-intervention survey, staff educational sessions were held on the ward, delivered by the study senior research associate (SH). Reference copies of the patient-focused intervention materials (the video and health-care logbook) were provided to each ward.
Delivery of the patient-focused intervention components
The patient safety video (copied to a DVD) and the health-care logbook were mailed to elective patients 1 week before their planned admission (i.e. 1 week following receipt of the study questionnaire) on intervention wards. Information explaining their purpose and use was included and patients were informed of the availability of Talk Time. Patients were encouraged to read and complete the different sections of the logbook and to watch the DVD prior to admission, sharing this with relatives when possible. Patients arriving on participating wards (or their relatives) following acute or emergency admission were given the logbook by staff, along with the information about its purpose and use. Acute wards were provided with portable DVD players to allow patient viewing of the DVD.
Post-intervention survey
Questionnaires were either given to patients at discharge or, when administrative support was available to wards, posted to patients following their discharge. A pre-paid envelope was included for (anonymous) return of questionnaires to the research team. No reminders were sent to patients.
Staff on intervention wards were invited to complete the post-intervention questionnaire at the end of the evaluation period for their ward; staff on control wards received the questionnaire at the end of the overall evaluation period. A pre-paid envelope was provided for return direct to the study team. Reminders were sent at weekly intervals for 2 weeks following the close of the evaluation period.
Post-intervention interviews
Purposive sampling was used to recruit subsamples of patient and staff participants with a broad range of characteristics (intervention wards only). Informed consent and interviews were undertaken by the study senior research associate (SH) and took around 20 minutes for staff and up to 1 hour for patients. All interviews were audio recorded and transcribed verbatim.
A semistructured interview schedule (see Appendices 9 and 13) was used that was informed by two frameworks for exploring implementation of health-care innovation: the TDF310 and normalisation process theory. 326 Within the domains proposed by these frameworks, interviews explored service user and staff perspectives, attitudes, beliefs and experiences in relation to:
-
use of the intervention materials and participation in the overall intervention approach (service user behaviour; staff behaviour)
-
understanding and knowledge about the approach and patient safety issues
-
acceptability and feasibility of the approach
-
perceived levels of engagement and involvement in identifying patient deterioration and alerting staff to concerns
-
perceived impact on patient safety
-
barriers to and facilitators of implementation
-
unintended consequences (e.g. service user loss of trust in services; service user dissatisfaction with care; damage to staff morale, plans to leave; work burden).
Medication audit
Using a medications reconciliation process, the medication history of patients admitted to participating wards throughout the intervention period was reviewed at admission and discharge. Ward pharmacists coded for discrepancies related to two outcome measures (see Table 49) using a standardised data collection pro forma. No patient-identifiable information was extracted, although data were coded for ward, age group (e.g. 18–30 years, 31–50 years), mode of admission (elective/acute) and sex.
Analysis
Questionnaire data
The internal consistency of multi-item measures was assessed using Cronbach’s alpha (for measures with three items) and Pearson’s product–moment correlation coefficient (for measures with two items), using an acceptability criterion of ≥ 0.6 and r > 0.2, respectively. Relationships between process (explanatory) and outcome variables were examined using Pearson correlations and standard multiple regression analyses.
Intervention and control groups were compared using methods appropriate for comparing two independent samples (t-tests and analysis of covariance to compare two or more groups adjusting for differences in baseline performance). 328 For theory-based questionnaire studies based on the theory of planned behaviour, using multiple regression, a sample size of 80 is generally acceptable based on an assumed moderate effect size (i.e. multiple r of around 0.3). 200,329–331 This analysis sought to provide preliminary information on the size of effect of the intervention approach on the targeted theoretical determinants of service user and staff motivation (willingness) and self-reported behaviours in relation to a patient role in improving their safety.
Interview data
Interview transcripts were coded using thematic analysis. 332 As well as elaborating perspectives on the ThinkSAFE approach and intervention components, the qualitative data generated from the interviews sought to provide important contextual and situational information to inform refinement of the intervention components and future implementation methods. Interview findings were triangulated with the findings of the survey data to provide a more complete description of the overall intervention approach, enhancing the validity of the evaluation study. 329
Medication audit
Error rates were summarised as proportions for the two outcome measures and compared between groups using methods appropriate for comparing two independent samples (t-tests and analysis of covariance to compare two or more groups adjusting for differences in baseline performance). This analysis sought to provide preliminary information on the size of effect of the intervention approach on medicines reconciliation.
Results
The evaluation period was from 1 September 2012 to January 2013. The intervention ran on most wards for 4 weeks but was extended to 5 weeks on two wards to achieve the target sample of 460 patients (230 intervention and 230 control patients). Some post-intervention interviews with staff and patients were completed during January 2013 for these wards.
Delivery and fidelity of the ThinkSAFE intervention during the live intervention period (closed 14 December 2012)
In total, 140 logbooks and DVDs were posted out by ward clerks to elective patients. Interview data suggest that there was varied use of the logbook, with some patients engaging with the whole document and others making use of specific aspects, for example the question note pad, the medication checklist and the ‘how you can help to keep yourself safe’ guidance. Some elective patients reported ‘completing the logbook’ (the personal information section and the medication checklist) and then taking it with them to hospital when they were admitted. However, their expectation that staff would engage with the logbook was generally not met, resulting in the logbook not being used during interactions with staff as was intended. Not all elective patients interviewed had watched the DVD at home prior to admission to hospital. Some did not have a DVD player and others suggested that the written guidance was sufficient so they did not need to watch the DVD. Others watched the DVD and did not feel the need to read or use the logbook.
In total, 125 logbooks were distributed on wards to acute patients. Both staff and patient interviews suggested that distribution of the logbook to acute patients was erratic and that, when it was provided to patients, there was little in the way of explanation of its purpose or encouragement to use it in the way intended. Patients generally reported few, if any, visible signs of other patients having logbooks or using notepads on the wards. Some staff perceived that certain patients would be uninterested in the approach and the intervention materials and so they did not provide them with copies or simply left them on the patient’s tray or bedside cabinet. The use of the portable DVD players on acute wards varied. Staff on these wards suggested that printed and visual materials (e.g. posters and flyers) would be more appropriate. Relatives of acute patients who took part in an interview were not offered the logbook in lieu of the patient at admission.
Delivery of the staff intervention was dependent on the availability of staff during prearranged sessions on wards and sessions were generally attended by nursing staff and health-care assistants. Wards were provided with printed versions of the staff intervention slides for distribution to staff unable to attend and a reference copy of the ThinkSAFE materials. Doctors were sent the slide presentation with supporting notes by e-mail and were invited to view the video provided to their ward. It was not possible to establish if the disseminated materials were accessed outside of the group sessions. During the sessions staff were generally engaged with the information presented to them, asking questions about the approach and identifying and empathising with the issues, and they could acknowledge their role in permitting and fostering patient involvement. A consistent concern expressed was one of time, in particular in relation to delivering the Talk Time component.
Participant flow
A CONSORT diagram (Figure 18) shows the flow of wards through the evaluation trial at each time point in relation to the primary outcomes. 273
Questionnaire data
The pre-intervention survey closed on 12 November 2012. Questionnaires were returned for 268 out of 504 (53%) staff (167/320 from intervention wards and 101/184 from control wards) and for 82 out of 229 (36%) elective patients (57/140 from intervention wards and 25/89 from control wards) (n = 89 was the maximum number of elective control patients meeting the inclusion criteria within the time frame of the study). Patient respondents were 62% male with a mean age of 69 years (range 39–86 years). The post-intervention survey closed on 31 December 2012. Questionnaires were returned for 140 out of 504 (28%) staff (80/320 from intervention wards and 60/184 from control wards) and 72 out of 433 (17%) patients (47/230 from intervention and 25/203 from control wards) (the exact denominator for control wards is unknown but is likely to be much lower than 204 as distribution of the questionnaires by hand to patients, particularly on acute wards, was erratic; for four out of six elective wards, questionnaires were posted out to patients post discharge). Patient respondents were 56% male with a mean age of 67 (SD 13.2) years (range 19–90 years). The findings from all analyses of patient and staff questionnaire data were largely the same at both pre and post intervention; therefore, only pre-intervention data are presented.
The relationship between process (explanatory) and outcome variables targeted by ThinkSAFE
Patients with stronger intention (to ask questions or to tell staff if something is not quite right) also reported being more actively involved in their care (r = 0.806, n = 80), more willing to interact directly with staff to ask questions (r = 0.571) and to tell staff when something does not appear to be right (r = 0.381) and more willing to engage in patient safety behaviours (r = 0.361). These patients also had more positive attitudes towards involvement and greater self-confidence and perceived capability in directly engaging with staff (i.e. asking and telling) (r = 0.524, 0.423 and 0.575, respectively) and a stronger belief that staff would welcome and approve of patient involvement (r = 0.320, p = 0.004). Patients with weaker intention were more likely to fear negative consequences of directly engaging with staff (r = –0.0391) (e.g. being labelled as difficult or demanding; care may be compromised) and to have a more negative attitude (r = –0.486) and were less likely to get involved in their care (r = –0.374). They strongly believed that staff would not approve of patient involvement (r = –0.808) and that asking questions or telling staff when something is not quite right ‘is not the place of the patient’ (r = 0.559). All correlations are two-tailed and significant at the p < 0.01 level unless otherwise indicated. These findings lend quantitative support to the outcomes of the qualitative study undertaken during phase 1 of this project.
Regression analyses guided by the behavioural theories identified during phase 1 (see page 131) identified patient attitude, perceived behavioural control and self-confidence as key drivers of patients’ intention to ask questions and to tell staff if something does not appear to be right with their care. Theory of planned behaviour and social cognitive theory explained 25% and 31%, respectively, of the variance in patient intention to engage in these behaviours. A further key influence on patient motivation was their (self-reported) usual behaviour in relation to being actively involved in their care. Within the learning theory framework, patients’ current level of involvement in their care explained 69% of the variance in intention. In an ad hoc regression analysis, role beliefs were retained in a stepwise regression model that explained 45% of the variance in patients’ anticipation of negative consequences. The belief explaining the largest proportion of this variance was that it is ‘not the place of the patient’ to directly engage with staff about potential safety issues. There was no effect of age or sex. These findings provide empirical support for the behavioural antecedents targeted in patients by the ThinkSAFE approach.
Similarly for staff, feeling comfortable about being asked questions and being told about errors by patients, the attitude that this would help to improve safety and the approval of patients and work colleagues were confirmed as important factors underpinning staff motivation to foster asking and telling behaviours in patients and their relatives as well as their self-reported enactment of fostering behaviour. However, these targeted antecedents explained only small proportions of the variance in staff motivation and self-reported behaviour, suggesting that other factors may mitigate their influence.
The impact of ThinkSAFE on patient motivation (intention) and behaviour
Patient self-reports of engaging in the promoted patient safety behaviours (e.g. ask nurses/doctors questions about your care or treatment, tell nurses/doctors that something is not right/there has been a mistake) varied across the seven items assessing this (7–97%, median 63%) and were similar for both intervention and control groups. Patients reported frequently asking questions about their care (91% and 97% for intervention and control groups, respectively) but reported telling a member of staff that something was not quite right or that there had been a mistake (when they thought that this may be the case) less often (65% and 61% for speaking up to nurses and 52% and 42% for speaking up to doctors, respectively). Fewer patients had reminded nurses (22% and 13%, respectively) and doctors (7% and 13%, respectively) to wash their hands. This pattern of patient engagement with these safety behaviours was confirmed by staff reports of patient behaviour, with similar high levels of asking behaviours reported and similar very low levels of reminding staff to wash their hands. Summary questionnaire responses for patients (at both pre and post intervention) were positively skewed, with a majority of patients scoring ≥ 4 on most items (Table 50).
Variable | n a | Min. | Max. | Mean | SD |
---|---|---|---|---|---|
Willing to engage | 82 | 2.83 | 5.00 | 4.76 | 0.41 |
Willing to tell others | 82 | 1.57 | 5.00 | 4.07 | 0.80 |
Willing to tell staff | 82 | 1.40 | 5.00 | 3.96 | 0.88 |
Willing to tell relative | 81 | 1.00 | 5.00 | 4.33 | 0.91 |
Willing to ask staff | 81 | 2.33 | 5.00 | 4.65 | 0.60 |
Attitude | 81 | 1.60 | 5.00 | 3.80 | 0.78 |
Perceived consequences | 80 | 1.00 | 4.80 | 2.93 | 1.04 |
Confidence | 81 | 1.40 | 5.00 | 4.25 | 0.93 |
Perceived behavioural control (three-item measure) | 81 | 1.67 | 5.00 | 3.66 | 0.80 |
Perceived behavioural control (two-item measure) | 81 | 1.00 | 5.00 | 4.07 | 0.87 |
Subjective norm | 81 | 1.00 | 5.00 | 3.40 | 1.03 |
Social influences | 81 | 2.00 | 5.00 | 4.65 | 0.66 |
Currently involved in care | 81 | 1.50 | 5.00 | 4.14 | 0.93 |
Intention to ask/tell | 81 | 1.00 | 5.00 | 4.25 | 1.01 |
There was no difference in mean scores between groups for any of the measured items at pre and post intervention and there was no change in group means from pre to post intervention.
The impact of ThinkSAFE on staff motivation (intention) and behaviour
Staff generally reported being highly motivated to actively encourage patient involvement in their care and safety and their self-report of fostering patient engagement behaviours (e.g. involve patients in decisions about their care or treatment, sit down with patients to discuss their care or answer their questions) was very high across all five items assessing this (84–100%, median 98%). Patient report of staff engaging in these behaviours was also high, although consistently lower than that of staff themselves (66–92%, median 74%).
Staff were, however, much less certain about questionnaire items that explored potential negative consequences. For example, although 46% of staff disagreed that encouraging greater patient involvement would increase staff workload, 29% agreed and 25% remained uncertain. Likewise, although 44% disagreed that staff would feel criticised or scrutinised, 32% were uncertain and 24% felt that they would be. A significant minority of staff (40%) also agreed that the trust between patients and staff might be damaged and a similar proportion felt that encouraging patients to ask them questions was challenging their professionalism. There was also some concern that patients might lose confidence in staff if they could not answer a question (only 26% disagreed with this statement) and a common belief that patients expected staff ‘to have all the answers’ (61% agreeing). There was general uncertainty over whether or not patients were ‘forgiving’ towards staff who make mistakes (64% uncertain or disagreeing) and a feeling that patients who write things down are usually preparing a complaint (58% uncertain or agreeing). These findings again lend quantitative support for the inclusion and the focus of the parallel staff intervention component that addresses such staff concerns and to the findings of the qualitative study conducted during phase 1 of this project. Additional support is also provided for the inclusion of the Talk Time component. Responses to questionnaire items derived from the NHS staff and US Consumer Assessment of Healthcare Providers and Systems surveys,305,325 and free-text comments on returned questionnaires, implicated a lack of time, heavy workloads and staffing levels as barriers to meaningful interactions with patients and their families. Summary scores for staff on all questionnaire items were similar pre and post intervention and mean scores did not differ between intervention and control groups (Table 51).
Variable | Intervention | Control | ||
---|---|---|---|---|
n a | Mean (SD) | n | Mean (SD) | |
Willingness to foster engagement | 155 | 4.58 (0.55) | 93 | 4.54 (0.63) |
Willingness to foster asking and telling behaviours | 155 | 4.54 (0.62) | 93 | 4.49 (0.75) |
Attitude | 155 | 3.82 (0.72) | 92 | 3.92 (0.73) |
Subjective norm | 155 | 4.00 (0.84) | 92 | 3.82 (0.79) |
Perceived behavioural control | 153 | 2.69 (0.73) | 91 | 2.56 (0.63) |
Perceived consequences | 155 | 3.23 (0.55) | 92 | 3.12 (0.44) |
Self-efficacy/confidence | 154 | 4.39 (0.71) | 92 | 4.19 (0.88) |
Self-reported behaviour (past month) | 153 | 3.94 (0.60) | 92 | 3.85 (0.79) |
Staff-reported patient behaviour (past month) | 153 | 3.06 (0.58) | 92 | 2.77 (0.53) |
Summary
Patients and staff who returned a questionnaire were generally highly motivated to engage in the patient safety behaviours promoted by ThinkSAFE. Although an impact of the intervention approach on motivation, or on the targeted beliefs and cognitions, was not demonstrated, regression analyses confirmed that positive attitudes and self-confidence in asking and telling staff, and the perception that doing so is within a user’s capabilities, are important factors influencing patient intention to adopt the promoted patient safety behaviours. Furthermore, patients whose usual behaviour is to be involved in their care were significantly more willing and confident about engaging directly with staff about their safety. Fear of reprisal remained a significant barrier to ‘speaking up’ and, regardless of the strength of patient motivation to directly engage with staff about safety issues, a majority (89%) of patients agreed (70% strongly) that they would be more likely to directly engage if staff were to say to them ‘it is OK to/I want you to . . . ask me questions/tell me if there has been a mistake’. This confirms the findings of the qualitative work undertaken during phase 1 about the important role of staff in ‘permitting’ patient involvement by demonstrating receptiveness and in actively fostering questioning behaviours in patients and their relatives or carers. However, for staff to engage with this, it is important that they perceive an impact of patient involvement on improving safety and feel comfortable about being asked questions and being told about errors by patients. They also require time to enable them to meaningfully interact with patients.
Post-intervention interview data
Thirty-three staff (4 doctors, 6 pharmacists, 14 nurses, 9 health-care assistants) and 14 patients/relatives (six male, six female, two relatives) were interviewed.
Patient and staff experience
Generally, patients and staff felt that ThinkSAFE was a ‘good thing’ in terms of helping and encouraging greater patient involvement in their health care and allowing patients access to more information about what to expect and what was happening to them. Health-care assistants and pharmacy staff, in particular, spoke about the approach in very positive terms and saw benefits for both themselves and patients:
I think it’s very good in principle . . . it’s quite useful if they [patients] have a written list – which they don’t always have – some of them are very good and do write down everything they take . . . we can go for GP’s [general practitioner’s] fact but that can differ quite significantly from what you are actually taking.
Pharmacist 11
I’ve seen staff sort of glancing through at it [logbook] . . . well I think they think its brilliant you know . . . the ones that I’ve spoke to, they’ve said it’s brilliant because its good for the patients.
Health-care assistant 310
Although the doctors interviewed were supportive of encouraging patients to be more involved in their care, they were only peripherally engaged with ThinkSAFE. Enthusiasm for ThinkSAFE in practice varied among nursing staff, who as a group tended to express misgivings relating to a lack of time to sit with patients or read logbooks or the perceived impact on their already heavy workload. Often their evaluation of the value of ThinkSAFE was influenced by the type of ward they worked on or the characteristics of the patients who they received:
I think it [logbook] would be quite helpful for [nurses] at admission instead of having to keep waiting for the notes and going through all the notes. If all the information was there. I think it’s irrelevant whether it is a bigger workload at the end of the day because [it’s] the patient’s safety that is paramount and, you know, if we can encourage them [patients] to ask about their care that gives us access to the doctors to explain what we can’t to the patients, so that we’re getting more involvement.
Nurse 32
That was my opinion, that’s what I thought, I didn’t think it was going to really work that well on here, probably on elective much better because the patients are following a care-way pattern where they go to see pre-assessment so they know they are coming in for their surgery, they know exactly what will happen, it’s all been discussed while they’re here, they’re not just brought in off the street – a football injury, a laceration . . . but that is the nature of this ward. It’s a totally different . . .
Nurse 18
There was still an air of suspicion among some staff about the purpose of the approach, in particular the logbook and notepad, with staff discussing these resources in terms of eliciting patient feedback or complaints. Some staff did not engage with the logbook as patients ‘did not appear to be interested’. One staff respondent suggested that raising patient expectations for things like Talk Time increases the risk of them feeling frustrated and disappointed with staff, who simply do not have the time to provide this service. A small number of patients and staff also commented on costs given cuts in health-care funding. Many staff interviewed, however, had not attended the educational session, nor seen the ThinkSAFE video or logbook, prior to the approach being introduced to their ward.
Acceptability and feasibility of the ThinkSAFE approach
Patients welcomed the concept of having some kind of patient-held record of their care. There was variation, however, in terms of how comprehensive they felt the record needed to be. For example, some patients preferred and actively used the complete logbook, whereas others engaged with specific sections (e.g. the personal notes sections or the medication checklist) or with varying combinations of components (e.g. the question guide and the question note pad):
I don’t lack confidence but it gave me . . . well it suggested what questions to ask you know? I thought the purpose was to benefit me but also for the hospital staff to read and there was some stuff in there which I thought the purpose of, one of the purposes of the log book was to help them [staff] as well you know? Nobody looked at my logbook you know, which sort of defeats the object a little bit.
Patient 11
But the point is a lot of its [DVD] duplication of what’s already in the book – so if patients don’t see the DVD it isn’t that important if they’ve read the book.
Patient 09
The A5 folder format was not valued by some patients as it was perceived to be slightly cumbersome (‘I would need bigger handbag!’).
There was evidence that the elective patients interviewed had each read and completed sections of their logbooks prior to their stay in hospital, although few logbooks were available during interviews. When they were available (typically for patients who had ongoing or chronic health problems), patients had continued to use their logbook post discharge to record information about their care, to note down questions to ask at future health-care contacts and to purposely facilitate communication of information to multiple care providers. All interviewed patients reported that staff did not engage with the logbook, which appeared to inhibit its use by patients once on the ward:
I saw one or two [logbooks] knocking about and I just thought it was low importance cause I thought the nurse would come in and say can you fill that in for us but nobody, nobody er, approached us.
Patient 13
[Did you see any other logbooks on the ward?] No, not that I noticed, no it was never mentioned. Er well I mentioned it and that was, it was just, ‘no it doesn’t matter’, I was, it was dismissed.
Patient 06
For staff, the logbook approach was felt to be more relevant to elective patients and those who were less acutely ill, although adaptations were suggested for use in the acute setting. These included provision of selected materials, for example the patient safety guidance and the question note book or a simple diary-style booklet, and distribution of the ThinkSAFE materials further upstream in the acute admission pathway, that is, at the point of admission to the emergency room rather than on subsequent admission to a base ward. Staff on acute wards saw a definite benefit to their role (e.g. saving of time and effort in terms of retrieving notes and information about patients) in having a well-documented logbook accompanying patients eventually admitted to them as base wards.
Preference for the different presentation formats of the patient safety guidance also varied, with some patients engaging more with the video than the printed version. Regardless of the format, patients described their experience of this guidance as ‘empowering’, with some directly attributing actual direct engagement with staff to the guidance:
It [video] was good and as I say, I found the whole thing empowering ‘cos it, it sort of emphasised on your own head be it you know, well not on your own head but – you know – it’s a partnership . . . but it was only after I’d had the operation and was in a bed helpless, that I thought ‘this is awful’ – there was a women opposite me and again if it hadn’t been for this [ThinkSAFE] I wouldn’t, I wouldn’t of taken upon myself to say anything [to encourage the patient to tell staff how she felt], but she’d had two kneecaps done and she was crying in pain.
Patient 04
I definitely did sort of say ‘right, I’m not happy about that [height of a toilet seat following hip surgery], I’m going to ask’ – if I hadn’t watched that video, I wouldn’t have, I would have thought well, I’ve just got to sit and use that toilet . . . I had had a quick flick through it [the logbook] but I would say the DVD was enough . . . I don’t think it has to go in to loads of detail, it’s just giving you information on what you can, of what you can do and what you can ask.
Patient 12
I did yes [ask questions] especially sort of on leaving eh, a lot of them were to do on what – you know – what the plan was basically, once I had left the hospital, and those were the questions I’d written down [on the note pad], those were the questions I asked. It definitely helped me yes [to think about what I needed to ask].
Patient 11
Staff were generally positive about the content of the video as a whole and the acceptability of the patient safety guidance it promotes to patients. There was also general support for encouraging patients to ask more questions and to take some responsibility for their care and safety. The video was felt by staff to be a good and useful medium for delivering these messages and they suggested that the video could be played on various screens around the hospital. Despite this, patients still commented on the importance of staff receptiveness to their involvement, highlighting further the key role for health-care staff in the success of ThinkSAFE:
The only comment I would add there, I mean I certainly wouldn’t hesitate to ask questions, but the general atmosphere on a ward can sort of help that or prevent that depending how friendly the nurses are and how open they are. And where I was certainly the nurses were first class.
Patient 11
No, I [don’t] think [ThinkSAFE will change my behaviour] – I still think, personally, I get intimidated by figures in authority which doctors are . . . I find it very difficult to relate to people like that.
Patient 06
Some patient accounts suggested that exposure to ThinkSAFE had influenced patient behaviour beyond the inpatient setting. The note pad in particular appeared to be a popular tool for supporting patient sharing and seeking of information. For example:
I actually used that more when I came out of hospital. I did use that because I think that was a good idea cos it got – you know what it’s like – you think something and you totally forget . . . So for jotting things down yes I reckon that was a good idea. [So what did you use it for after you came out of hospital?] For when I was going to see my GP [general practitioner].
Patient 11
Patients reported that this element of the approach did not formally happen and this was confirmed by the staff interviewed. Despite encouraging wards well in advance of the pilot to discuss and plan how they could incorporate this element of the intervention into current work patterns and ward routines, none of the intervention wards was able to achieve this. However, although Talk Time did not happen at the ward level, there was evidence that some individual staff had attempted to provide additional opportunities for patients to talk to them about their care. Some staff (including doctors, nurses and health-care assistants) reported that they were more deliberately inviting patient questions since attending the educational session. Some also said that they tried to provide ‘time to talk’ during routine contacts with a patient, that is, during ward rounds or when providing personal care:
I remember one patient we had . . . she kept saying ‘oh I didn’t ask this and I didn’t ask that’ and because she wasn’t actually a surgical patient we didn’t have all the answers for her. So I did tell her to write down everything . . . I said the doctor’s maybe not going to be back until tomorrow but at least you’ve got your questions there so that you’re prepared for when he comes to ask them. So I do think that’s a quite useful way of doing that.
Nurse 02
I have noticed a difference in the amount of questions that patients ask you, I think that is one of the biggest differences I have noticed . . . a lot of people now are asking about their blood tests and what their results were and about their X-ray and things and that kind of thing. . . . I don’t know whether we were just not picking up the cue and but it just feels like we are being asked more to explain what we are doing and when you do it becomes a habit and then you start referring things to people as you go and they tend to really appreciate that information that you are giving them . . . I think they should know and it does prompt you to explain things a bit more thoroughly to them rather than ‘we are taking your blood this morning’ – ‘we are taking a bit of blood this morning to check for this’.
Doctor 08
This is an encouraging finding, but in some staff accounts there was also a sense of possible resistance to Talk Time in that it can really only be implemented at the convenience of the staff and not when it is ‘needed’ by patients. It was felt that ‘on-demand Talk Time’ and unpredictable ‘taking time out to talk to patients’ would be too disruptive on busy, task-demanding wards. Some staff also spoke about taking time out to talk with patients in terms of it being a luxury and something secondary to providing clinical care.
The video forms part of the staff educational session but staff very much perceived it as a patient-targeted intervention. The staff-targeted elements were not alluded to during interviews. This suggests that the staff-targeted elements need to demonstrate more explicitly that their behaviour must also change. Although staff did appear to get this message from the slide presentation during the educational session, greater emphasis is needed on the importance of staff actively inviting patient involvement and openly demonstrating receptiveness.
The impact of ThinkSAFE on medication safety
The prescriptions of 522 patients, with a mean age of 65 (SD 19) years, were audited by ward pharmacists at admission. In total, 391 (75%) were acute patients and 279 (53%) were female. At discharge, 409 prescriptions were reviewed, with 200 (49%) patients being female and a mean patient age of 64 (SD 18) years.
The analysis tentatively suggests that prescriptions issued on intervention wards at admission were significantly less likely to require pharmacist intervention [a difference in error rate from 62% (control) to 52% (intervention); p = 0.033] and, when intervention was required, were more likely to contain only one error per patient (73% control vs. 58% intervention; p = 0.024). The most common error recorded was the omission of a ‘home’ medication. This error was 11% less frequent in the admission prescriptions of intervention patients (47% vs. 36%; p = 0.009). High rates of error-free prescriptions were found for both groups at discharge (82%), likely because of error correction at admission and further pharmacist intervention throughout the inpatient stay.
The impact at admission is difficult to untangle, but is unlikely to have been heavily influenced by patient intervention alone, if at all (i.e. by patients presenting completed medication checklists), as acute patients were generally over-represented in the data set and there were proportionately more acute patients in the intervention group (Figure 19). It is feasible, however, that acute patients who received the logbook post admission were prompted by this to volunteer information about their medicines. Although there is no way of establishing from the current data set whether information was elicited by staff or volunteered by patients, ‘the patient’ as the source for information about their medicines was used more frequently in the intervention group than in the control group on two dimensions (patient verbal report: 78% intervention vs. 52% control; patient’s own drugs: 47% intervention vs. 21% control). These findings tentatively suggest that the intervention is having an additive (if not sole) effect through its influence on staff behaviour. However, as these source data were recorded for only 51% of the whole sample (267/522 records), these results need to be treated with caution.
Discussion
The work presented in this chapter describes the systematic development of ThinkSAFE, an intervention approach that supports patients and families in communicating questions and concerns directly to health-care staff, thereby reducing patients’ risk of experiencing harm. Uniquely, ThinkSAFE also provides necessary support to staff to enable them to understand and facilitate this new patient/relative role. A further novelty and strength of ThinkSAFE is its firm basis in user (patient, family members and frontline health-care staff) experience, best evidence and practice and behaviour change theory. Empirical support for the approach is also provided by the robust and transparent approach to the development of ThinkSAFE, which was guided by the MRC framework for developing and evaluating complex interventions. 267
A clear and collective message from service users and staff involved in phases 1 and 2 of the project was that it is the way that patient involvement is approached that matters. Patients are reluctant to engage in behaviours framed as ‘checking up on staff’ and staff are resistant to efforts by service users that they perceive as ‘being checked up on’. Our work further demonstrated that this can actually lead to counterproductive behaviours in both staff and service users. 297 This lends strong support to our overall approach of seeking a collaborative solution to enabling patient involvement. Subsequently, a fundamental aim of ThinkSAFE is to encourage and support this new way of working together, based on a shared aim to ensure safer care.
Even within this collaborative context, it is evident from the phase 3 evaluation that staff approval remains crucial for service user uptake. Furthermore, our findings clearly demonstrate that staff ‘approval’ goes beyond passive receptiveness; evidence is triangulated from both service user and staff interview and survey data that emphasises the need for staff to actively foster patients and their relatives to engage in safety behaviours. This includes staff verbally inviting questions and the communication of concerns, as well as visible demonstrations of their engagement with the ThinkSAFE materials and process.
Staff and service users were highly motivated to engage in the patient safety behaviours promoted by ThinkSAFE, suggesting that the approach is acceptable. Although empirical support is provided for the underlying theoretical rationale, we were unable to demonstrate a measurable impact on participant motivation and other targeted behavioural determinants, possibly because of ceiling effects and problems with intervention fidelity. The qualitative evidence does, however, suggest some influence of ThinkSAFE on staff and patient interactional behaviours. There was also evidence for the potential of ThinkSAFE to improve medications reconciliation on admission to hospital. Although some aspects of the evaluation suggest that the ThinkSAFE approach is feasible, there were evident issues in implementing the complete model (but in particular the implementation of Talk Time). This has, however, provided valuable learning about what needs to change to support feasibility and fidelity in future applications of ThinkSAFE.
Limitations
A key limitation of ThinkSAFE in its present form is that its development was informed exclusively by white, English-speaking service users. Although purposive sampling enabled a broad range of patient experiences and expertise to be considered, this limits generalisation beyond this demographic. The service setting was also limited to inpatient care, although, once again, purposive sampling within two NHS trusts of both acute and elective admissions and across a range of specialties provides a robust analysis within this context. Evaluation of ThinkSAFE in the secondary care hospital setting was challenging. Randomisation within the study design was not feasible and, although control wards were matched as far as possible, there remains the potential for possible bias as well as contamination. Similarly, the use of questionnaire survey methods for large-scale data collection was also extremely labour intensive and difficult to administer consistently and yielded low response rates, which prevented meaningful comparison of group means. Pre-evaluation questionnaire surveys did, however, reach sufficient numbers needed for the planned regression analysis and it is promising that the findings of this analysis are consistent with those of the qualitative study undertaken during phase 1 of the project. As the sample analysed was self-selected, and no data were available about those who did not return a questionnaire, the findings of this analysis may not be representative of patients and staff in general. The quantitative analysis of both patient and staff survey data should therefore be treated with some caution. The analysis of the medications reconciliation data, although adequately powered to detect real differences between groups, is limited by the cross-sectional nature of the data collected and the apparent bias towards acute admission data. Finally, intervention fidelity is questioned, particularly for the acute wards where distribution of the logbook and video to patients was erratic. Several staff were unable to attend the educational session and none of the participating wards was able to provide the Talk Time element of ThinkSAFE. It is impossible to establish use or extent of engagement with the ThinkSAFE materials by service users who received these resources.
Conclusions
ThinkSAFE is a multifaceted intervention that is extensively user informed and robustly developed. It is also underpinned by a strong theoretical rationale to support why and how the approach should have its effect on improving patient safety. Its generic approach to promoting and supporting collaborative interactions between health-care staff and service users provides flexibility for local adaptation without compromising this underlying rationale. Preliminary evaluation, based on the triangulation of findings from several sources of qualitative and quantitative data, suggests that ThinkSAFE is an acceptable, low-risk intervention approach that has the potential for improving medication safety on admission to hospital. Although aspects of the evaluation suggest that ThinkSAFE is feasible, the observed issues of intervention fidelity highlighted important structural and organisational barriers that need to be addressed in future models of implementation.
Chapter summary
This chapter has described the development and initial piloting of an innovative, patient-centred intervention to support the direct engagement of patients with health-care professionals as a means of improving care delivery and reducing harm. Patients directly engaging with health professionals as a means of improving patient safety can, as has been suggested here, be a ‘real-time’ activity in which patients are assumed to be a partner in achieving error reduction. However, it might also be hypothesised that patients can directly engage with health professionals outside of an acute health-care presentation setting and that this engagement might impact on future patient safety outcomes. One such means of achieving this direct engagement with health professionals is using patient narratives within the education of health-care professionals. The next chapter examines this possibility, describing a RCT of an intervention in which patient stories of PSIs are positioned centrally within medical undergraduate education.
Publication statement
Data presented within this chapter have been previously published. 297 We reproduce these data here with permission. © 2013 Hrisos, Thomson. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Chapter 10 Education and training: using patient narratives within medical education – a randomised controlled trial
Abstract
Trial design: An open, multicentre, two-arm, parallel-design randomised controlled trial based in North Yorkshire and East Coast Foundation School (NYECFS).
Methods: All 313 foundation year 1 (FY1) trainees (2011 and 2012) were eligible to participate. Simple randomisation was carried out at an individual level on a 1 : 1 basis. As the sessions took different formats, participants were aware of their assignment. The intervention consisted of patient narratives followed by discussion relevant to the narrative as well as generic safety issues. The control arm received conventional faculty-delivered teaching. The Attitude to Patient Safety Questionnaire (APSQ) and the Positive and Negative Affect Schedule (PANAS) were used to measure the impact of the intervention. Learning points suggested by trainees were used to measure differences in learning outputs between the two groups.
Results: In total, 142 trainees received the intervention and 141 the control. There was no evidence of a difference in APSQ scores between the groups. There was a statistically significant difference in the underlying distribution of both post-positive affect and post-negative affect scores between the two randomised groups (p < 0.001), with an indication of both higher positive and negative affect scores in the intervention group. Analysis of the learning points revealed five overarching themes: risk management and governance; learning about error; communication; processes related to patient safety; and role of education.
Conclusions: We were unable to demonstrate that the intervention was any more effective than standard teaching in changing general attitudes to patient safety. However, the intervention did impact on emotional engagement and learning about communication.
Chapter rationale
The final area of focus for our programme of work on patient involvement in patient safety centred on the role of patients as educators. There is increasing interest across clinical educators on the use of the patient voice within the education of health-care professionals, but there has been little evidence to date testing the efficacy of this approach within patient safety education and training. This chapter describes the testing of an intervention using patient narratives about PSIs as a means of improving medical students’ attitudes towards patient safety.
Introduction
There is increasing evidence that training in patient safety improves knowledge and processes of health care. 333 The recently published Francis8 and Berwick206 reports that followed investigations into poor standards of care in a UK hospital emphasised the need for developing patient safety interventions as part of mandatory training for health-care professionals. A number of these interventions are still new334,335 and need further evaluation to inform educators on how to engage students and recent graduates to learn about safety. There is also a tendency to focus on issues such as causes of safety lapses, root-cause analysis of incidents and the need for promoting organisational patient safety culture. 335 This provides learners with a view of patient safety through a health professional rather than a patient ‘lens’, with limited emphasis on the impact of safety lapses on patients and their families and little or no involvement of patients in the design or delivery of training. There is a drive to involve patients more explicitly in medical education. 336 A systematic review of patient involvement in teaching showed effectiveness in terms of increased learner satisfaction and improved communication skills among health-care professionals. 47 A possible role for such involvement is in patient safety training as part of educational interventions. 206,334,335,337 Patient narratives are now widely employed in health professionals’ training, allowing patients to share their own health-related stories with professionals to facilitate the development of clinical knowledge and skills. 338,339 Patient safety is a particularly appropriate area for narrative-based teaching as it provides an opportunity for patients to share their own real lived experiences of lapses in safety, resulting in harm to themselves and/or their families. These real stories, brought to the classroom, allow the exploration of factors causing the errors, have the potential to increase awareness of the personal impact of such errors on patients and facilitate a wider discussion of safety issues. There is only limited research exploring the use of patient narratives in a safety context. However, preliminary research suggests that this may be a feasible and acceptable method for raising awareness of patient safety concerns among health-care professionals. 337,339
The association between improvements in safety culture (mainly organisational) and climate (mainly attitudes and behaviours of staff) and patient outcomes and staff behaviour is unclear because of the complex nature of this subject and limited empirical research in this area. 340 However, attitudes do influence behavior313 and are increasingly being used as outcome measures for patient safety interventions. 341,342 In addition, emotions influence behaviour either directly through affect (short-term likes/dislikes) or indirectly through cognitive changes in behaviour. 343
Real stories about patient safety tend to trigger strong emotional responses in the listener,344 which, in turn, facilitate greater engagement with345 and better retention of346 the learning messages. An important measure of such an emotional response is the impact that narratives have on a humanistic approach to patients, including empathy,48,347,348 and how this leads to greater patient centredness.
A conceptual framework by Kumagai48 used theories of empathy and moral development to study the use of patient illness narratives in medical education. This work, based on that of Mezirow,349,350 utilises the concept of transformative learning in which there is a ‘process of effecting change in a frame of reference’ (p. 5). 350 These frames of reference include previous experiences, values and feelings, which structure how we make meaning of new learning. Transformative learning encourages ‘critical reflection of assumptions, validating contested beliefs through discourse and taking action on one’s reflective insight and critically assessing it’ (p. 11). 350 Kumagai’s framework is based on the assumption that doctors develop an understanding of the ‘meaning of medicine’ based on the nature of their training and therefore medical education should focus on influencing the way that people learn the meaning of medicine. Stories are used to communicate the meaning of individual experiences to one another and can therefore be used to shape the training of junior doctors. According to Kumagai,48 patient stories may facilitate the development of empathy in trainee doctors in three ways.
-
Narratives allow doctors to appreciate what it is like to experience an illness and, particularly when delivered face-to face, may help to develop an ‘interpersonal link’ in affective, cognitive and experiential domains. This helps to develop a perspective towards patient experiences in a way that is difficult to develop in paper-based scenarios in which there is no patient interaction.
-
Narratives help communicate meaning by triggering fundamental emotional responses such as loss, anger, jealousy, guilt and sadness. This may arouse a sense of urgency in the learners to explore the causes of a patient’s suffering and create moral dilemmas in terms of inequality and social justice.
-
Narratives allow learners to identify better with the patient. If the ideas and beliefs expressed by the patient are not congruent with their own, there may be a dissonance in terms of emotions or cognition. This dissonance stimulates reflection on one’s values and attitudes and discussions on humanistic and ethical practice and patient care.
In this study we adopted the framework suggested by Kumagai48 to deliberately use emotional stories from patients to enhance the learning experience of trainees351,352 and provide the learners with a greater understanding of safety from a patient’s perspective. 353 We wanted to explore if facilitating trainees to reflect on the patient stories and on their own experiences could influence their own beliefs, attitudes and intention of future behaviour.
Objectives
This trial aimed to measure the impact of patient narratives used to train junior doctors in patient safety.
The primary objective was to measure attitudes towards patient safety using the APSQ. 341 A secondary objective was to measure the short-term emotional response to the patient stories using the PANAS. 354
Methods
Trial design
An open, multicentre, two-arm, parallel-design RCT was conducted in NYECFS using five centres: Scarborough, Hull, York, Grimsby and Scunthorpe. There were 20 days of teaching, organised on 10 days during 2 consecutive academic years – 2011 and 2012 (four at Hull, three at York and one at each of the other sites per year); Hull and York employ a higher number of FY1 trainees than the other sites and therefore required more training days. Within each centre, FY1 trainees in their first year following graduation were individually randomised on a 1 : 1 basis to the control arm or the intervention arm. In the control arm, participants received standard faculty-delivered teaching on patient safety; in the intervention arm participants received teaching facilitated by patients. The background information and full details of the intervention can be found in the trial protocol355 and is also described briefly in Intervention.
Participants
All 155 FY1 trainees in 2011 and 158 FY1 trainees in 2012 from the NYECFS were eligible to participate. In the NYECFS, trainees are allocated to mandatory training days (run between January and March each year) at one of the five centres based on their employing hospitals. The intervention was delivered by the ‘patient group’ consisting of six patients and five carers who had experienced harm during health care because of lapses in communication or teamwork, diagnostic errors or poor decision making. The outcomes of the incidents included psychological or emotional stress and sometimes permanent harm and even death.
Ethics
Ethical approval was granted by the National Research Ethics Committee in February 2010.
Intervention
The full details of the intervention have been published previously. 355,356 Patients and carers, recruited from the NPSA, AvMA and advertisements in the local press, were included if they had experience of suffering harm or error to themselves or their families during health care. Four preparatory Patient Learning Journey (PLJ) workshops prepared the patient group for the teaching programme. 356 The workshops created a confidential supportive environment in which the group could comfortably share their experiences, bond with each other, identify key aspects to include within the narrative and adopt a learner-centred approach.
The intervention consisted of two sessions of 1 hour each, developed collaboratively with the patient group and delivered in small groups of between seven and 10 trainees. In each session, one patient narrative that lasted for approximately 15–18 minutes was followed by facilitated discussion between the patients and the FY1 doctors. The narratives were used to focus on specific issues surrounding the individual patient story as well as more generic issues of safety (an example is provided in Box 7). Emphasis was given to analysis of inadequate care and its causes. Each narrative included a factual description of what happened and reflections about the patient’s experience of medical error or inadequate care: what went wrong and why, the impact of the error and what could be done better. During the discussion (cofacilitated by VJ, JS and the patients), trainees reflected on the narrative, identified emergent patient safety themes from the stories and explored their own attitudes and beliefs about patient safety. They also shared their own experiences of safety-related incidents as professionals and as patients or carers. The learning objectives common to both the control group and the intervention group and derived from the UK Foundation Programme Curriculum [see www.foundationprogramme.nhs.uk/pages/home/training-and-assessment (accessed 13 June 2016)] were adhered to throughout the sessions and issues related to the objectives were discussed even if they did not naturally arise during the discussion. The trainees were encouraged to discuss with the researchers any issues related to distress or upset caused by the narratives. The patients were provided with emotional support by JS, who debriefed them after each session.
When the same senior medics that saw her previously with that comforting and attentive attitude, walked into her room the following day, I felt an immense sense of relief. Because for the first time in eleven years, of all we had been through together, I felt completely out of my depth. However, this time their concerned attitude appeared to me to be replaced with annoyance. Annoyance that this girl, that should simply be riding the storm, should be back in hospital, and even put on antibiotics. In a dismissive tone, the medic ordered that the antibiotics should be stopped and instructed both her and I that she simply had to be patient. I was by now, scared for L, and I was scared that despite the medic’s comprehensive knowledge and expertise, they did not have the knowledge of L, and did not seem to grasp her state of health. They of course did not know her normal reactions, her strength of character. They were not privy to the smallest of symptoms or responses that occurred when they were not present, but I was. So I tried to share all this with them to try and convey that she was not merely a little unwell, but very poorly. I knew of course, logically that glandular fever wasn’t treatable but I was getting concerned that something else would be missed that was treatable. However, on trying to inform them of my fears, I felt instantly dismissed, and that I was nothing more than an overprotective mother. And L, who simply could not understand that she just had to be patient and must learn to cope. Over the next week, L was in so much discomfort, she slept for maximum of twelve hours. Twelve hours out of 168. That’s 156 hours of constant pain, but this was of no concern. Her face was so swollen that her eyes were the size of golf balls, which rendered her blinded for three days. This was of no concern. Her face so cracked and scaled, the cream I tried to rub in sat in the crevices created. No one could tell me why her skin was like this. To me, it was like she was being poisoned from the inside/out. But it was of no concern. The bruised marks continued to spread, it was of no concern. Her throat was so sore, that two analgesia tablets broken in half would take me fifteen minutes to administer. Her pain so severe that one of her hands would twist my arm around in pain as she clawed at the sheet with the other hand. Analgesia via her IV [intravenous] drip was refused as something we don’t do on the paediatric ward.
From the seventh day of her second admission, L had been vomiting and feeling very ill. And although her spleen and liver had reduced in size to the touch, at 3 p.m., she started humming to herself. When I asked her what she was doing, she whispered, ‘It comforts me’. She then started talking nonsense: ‘Animals need one energy level, I need another’. I felt as if she was going in and out of consciousness and I was losing her. Trying hard not to scare her, I told her to stay with me and I called for the medical team. When they arrived over two hours later, and after three requests, they felt this confusion was because she was tired. She was asked two questions to assess her orientation to place, which she answered correctly. But these were done with her back towards them as she faced the wall of her room. They never once saw her face. They said they would place her on four hour in neurological obs, but they were not concerned. The first set of observations was carried out by a student nurse. Only two hours later, L suffered a fit, a cardiac arrest, multi-organ failure, and DIC [disseminated intravascular coagulation]. At this point in time, L was only on paracetamol and ibuprofen administered via me. Her IV [intravenous] drip had been discontinued 24 hours previously because of her swelling. Within hours of being admitted to intensive care as an emergency, L was diagnosed with bacterial toxic shock syndrome, this having been the cause of her complete collapse. And on admission to intensive care where she immediately received every treatment imaginable, the ICU [intensive care unit] doctor said to me, ‘She is a very ill girl’. I could hear myself saying, ‘I know that. I had been telling them for a week, but they wouldn’t listen’. But for as much as my soul was screaming, my head was saying, ‘What is the point in telling the intensive care? It is too late’. Later that night, another doctor asked me in a confused manner, with accusation in his tone, ‘Why did you leave it for so long before bringing her to hospital?’ On telling him I hadn’t, he went deathly quiet . . . .
The control group received a clinician-led teaching session using Microsoft PowerPoint® 2010 (Microsoft Corporation, Redmond, WA, USA) presentations and small group work; the group size was similar to that in the intervention group. Researchers observed the control group sessions to compare content with that in the intervention sessions. In one session, typical of the training, trainees were shown a PowerPoint presentation on the General Medical Council regulations and their guidance on the duties of a doctor. A number of safety scenarios developed by the trainers were then presented and discussed in small groups. The scenarios included ethical/legal dilemmas, self-awareness of limitations and how to prevent errors. Further discussions took place around how to stay up to date with developments in the profession and the importance of accurate record keeping and communicating with patients. The order of the session might have implied a hierarchy of importance with regard to the topics covered: regulatory and procedural material came first, followed by ethical/legal issues, with communication with patients and record keeping/handovers in the latter part of the session.
Pilot study
A pilot study356 was carried out to test (1) the feasibility of recruiting patients to develop and implement a patient-led intervention, (2) the acceptability of the intervention among patients and trainee doctors, (3) the practicality of delivering the intervention within an established training programme and (4) the suitability of the outcome measures.
Methods
This study included the development and implementation of the intervention and measurement of the outcomes of the intervention using quantitative and qualitative approaches. It was conducted in the West Yorkshire Foundation School; trainees at one site (Hull) received the intervention, whereas those at the other site (Airedale Hospital) received standard teaching.
The patients were recruited using a number of sources. 357 Patient-safety champions were invited through the NPSA and AvMA; these patients were expected to act as advocates of patient safety with experience of narrating their stories to health professionals. The Patient Voice Group at the University of Leeds and an advertisement in the local press helped recruit local patients to become involved in the study. All 284 FY1 trainees were invited to participate. At the intervention site (Hull), the teaching was delivered using patient narratives; at the non-intervention site (Airedale Hospital), standard teaching was delivered. Outcome measures were completed before the intervention, at the end of the teaching session and 6 weeks later.
The intervention itself has been described in Intervention.
Study measures
The following outcome measures were used to address the study objectives.
-
Success of identifying, recruiting and training patient participants as a measure of the feasibility of recruitment.
-
The attendance of trainee participants, feedback from a course evaluation form, data from individual interviews 4–6 weeks after the teaching session with a volunteer sample of trainees from each group and feedback from a follow-up workshop for the patients and facilitators organised 2 weeks after the last session as measures of the acceptability of the intervention.
-
The success of integrating the intervention into the existing foundation school training programme as a measure of the capability and capacity to deliver the intervention.
-
The suitability of outcome measures – the APSQ23 was the main outcome measure used and administered before, immediately after and 6 weeks after the teaching. All participating trainees were also asked to suggest three learning points that they would try and implement into their practice.
Analysis
All statistical analyses were carried out using SPSS (Statistical Product and Service Solutions) version 15 (SPSS Inc., Chicago, IL, USA). The APSQ data were analysed by using repeated analysis of variance measures for both knowledge and attitude scores. The qualitative data from evaluation forms, interviews and the workshop was analysed using thematic framework analysis. 358
Results
There were a total of 12 training sessions (six each for the intervention and non-intervention groups), with 155 trainees receiving the intervention sessions and 108 receiving the non-intervention sessions. For pragmatic reasons we conducted this study without randomisation; one site received the intervention and the other standard teaching. This allowed us to deliver the intervention to a smaller group and gave us an opportunity to observe the standard teaching offered to the trainees.
The 10 participating patients had a wide range of experiences, with some stories describing explicit isolated safety incidents such as drug errors and others demonstrating specific safety incidents but only within a series of other negative experiences such as poor communication.
Acceptability of the intervention
Although all trainees agreed to participate, the paperwork and questionnaires were not completed by some trainees, which we interpreted as not wishing to take part in the study.
On the evaluation forms, the response to patient involvement was largely positive; in three sessions all trainees felt that the patient input was invaluable, in two of the six sessions > 75% agreed that the patient input was invaluable and in one session only 60% acknowledged the patient input. The intervention group appreciated the patient centredness and highlighted the importance of communicating with patients. However, some trainees found the patient-led teaching to be too negative towards doctors.
During the six post-intervention interviews, trainees reflected on the teaching session, considered logistics such as group size and room layout and discussed how they felt the sessions could be improved and how they experienced patient safety in their own practice. The broad themes from both groups of trainees were similar but demonstrated a difference in understanding of what constituted patient safety. For example, interviewees from the non-intervention group conceptualised patient safety issues from a procedural point of view, for example completion of adverse incident forms, whereas trainees receiving the intervention discussed their interaction with the patient and the responsibilities of their role as a FY1 doctor.
Trainees from the intervention group remarked on the disadvantages of larger group sizes and suggested that working in smaller groups would facilitate the discussion more. A major difference between the two groups was in the emotional impact that the patient stories seemed to have on the trainees, a trend also observed on the evaluation forms. Trainees felt ‘frustrated’ as the patient stories were too complex and beyond the scope of junior doctors’ decision making; ‘intimidated and fearful’ at the attitude of patients with regard to doctor bashing; ‘anxious’ about sessions not containing enough practical knowledge of patient safety issues; ‘disappointed’ that the system had let the patients down; ‘engaged’ by the power of real stories; and ‘pleased’ that patients were given a voice.
During a follow-up workshop for the patients 6 weeks after the final session, most expressed their satisfaction with the teaching sessions. They felt that trainees had engaged well with their stories and that the discussion following the narratives allowed trainees to interact with patients, share their own experiences of safety incidents and reflect on examples of patient safety. There was, however, a suggestion that the relatively large trainee groups restricted audience engagement and interaction with the patients and facilitators. The large groups also resulted in some trainees not contributing at all to the discussion and prevented some trainees from engaging with the patient stories. The facilitators felt that stories with a clearer structure and focus on patient safety messages seemed to work better. A need to develop the narratives so that trainees did not feel that they were being ‘doctor bashed’ was also highlighted. A classroom set-up, as opposed to a lecture theatre setting, was felt to work better as it allowed better interaction with the trainees.
Capability and capacity to deliver the intervention
This study assessed the feasibility of adopting a pragmatic approach to integrating a RCT into an established training programme. To facilitate integration, and ensure that neither group was disadvantaged with regard to their training, the objectives of the intervention group exactly matched those of the non-intervention group. Moreover, all of the intervention sessions were facilitated by members of the research team and the patients; this reduced the administrative and teaching burden on the foundation school. Organisationally, all of the sessions ran on time and, with the support of the administrative staff, the paperwork for both groups was completed adequately and on time. Although all trainees completed the APSQ, evaluation and learning points on the day, the response rates for the 6-week follow-up were relatively poor, with only 38% of trainees responding to reminders by e-mail. On the other hand, a number of trainees agreed to participate in the follow-up interviews despite their very busy schedules, indicating that they were concerned with patient safety issues and wanted to provide feedback and facilitate our research process.
Suitability of outcome measures
The mean attitude APSQ scores pre intervention were not significantly different between the two groups (non-intervention 130.45, intervention 130.32). Mean scores for attitudes to patient safety overall increased after teaching but this was not group dependent (non-intervention 132.61, intervention 133.81). At baseline, the knowledge scores were similar in the two groups (non-intervention 31.4, intervention 32.7); after the teaching session, scores in both groups increased (non-intervention 32.65, intervention 33.0).
Refinement of the intervention
The pilot study revealed areas where the intervention required refinement.
-
There was a need for smaller group sizes for trainee participants. The subsequent RCT ran with a group size of around 20 (10 in each arm). This was possible by randomising the trainees on site.
-
Each story required a clearer focus to convey the key safety messages. The patients worked collaboratively with the research team in identifying these key aspects, sometimes having to restructure their narratives to provide a clearer context and message. The patient pairing was also revised, resulting in a story with a medical error in the same session as one with communication issues or a systems error.
-
The APSQ on its own was not sensitive enough to pick up differences between the intervention group and the non-intervention group. The emotional responses to the patient stories also needed to be specifically measured. We accordingly included the PANAS to assess the mood and emotional engagement of trainees in terms of state in the RCT. 354
-
Trainees from both groups were asked to complete the learning points, which could then be compared for content and coded appropriately. This would allow us to ascertain what types of learning were achieved through the patient-led safety training compared with the standard foundation year patient safety training.
-
The interactions between the patients and trainees were not captured in the form of a permanent transcript in the pilot study. During the RCT, the intervention sessions were video recorded to capture this interaction and allow future analysis.
Randomised controlled trial
Method
Primary outcome
The primary outcome was attitude to patient safety measured by the APSQ (see Appendix 14),341 a 26-item questionnaire addressing patient safety attitudes. An essential component of safety culture in an organisation is the attitudes of practitioners to safety including responsibility for reporting errors and appreciation of causes of errors, areas that are included in the APSQ. The APSQ was originally designed with final year medical students, which is likely to make it applicable to FY1 trainees at the start of their clinical practice. The psychometric properties of the APSQ have been studied using exploratory factor analysis. The primary end point was the overall score on the APSQ completed immediately following the training.
All participants completed a hard copy of the APSQ on three occasions: before the training (baseline), immediately after training prior to the two groups having a chance to meet and discuss their sessions (post) and between 3 and 6 weeks after training (follow-up). Each participant was given a unique identification number that could be used to link the questionnaire data over the three time points. The follow-up period varied as it relied on the availability and convenience of the participants. The pilot study had highlighted problems with relying on trainees to complete follow-up questionnaires online. For this study, researchers made group appointments with trainees and travelled to individual hospitals to allow trainees to complete the questionnaires. Participants gave each item on the APSQ a score between 1 (‘strongly disagree’) and 7 (‘strongly agree’). For most items a high score indicated a positive attitude to patient safety. For seven items (questions 11, 13–17 and 25) a high score indicated a negative attitude and hence scoring was reversed during analysis. An overall APSQ score may range between 26 and 182, with high scores indicating a positive attitude to patient safety.
Secondary outcome(s)
The APSQ was also scored using subscales suggested by the analyses conducted by Carruthers et al. 341 These subscales are formed through grouping the 26 items by topic into nine groups and summing the score for each group. The topic groupings and corresponding question numbers are shown in Table 52. The groupings were patient safety training received (clinical training helping understanding of causes of medical error); confidence in reporting errors (openness/no-blame culture in workplace); working hours as cause of error (shorter working hours, shifts); error inevitability (all doctors make errors); professional incompetence as cause of error (carelessness); disclosure responsibility (reporting error); team functioning (multidisciplinary working); patient responsibility in reducing errors (greater involvement); and importance of training (teaching on safety). Self-reported knowledge about patient safety was an additional subgroup added for the purpose of this study to capture how knowledgeable participants felt in relation to seven aspects of patient safety.
Group | Questions |
---|---|
Patient safety training | 1–3 |
Confidence in reporting errors | 4–6 |
Working hours as causes of error | 7–9 |
Error inevitability | 10–12 |
Professional incompetence as cause of error | 13–16 |
Disclosure responsibility | 17–19 |
Team functioning | 20–21 |
Patient responsibility in reducing errors | 22–23 |
Importance of training | 24–26 |
Knowledge of patient safety | 27–33 |
A further secondary outcome of this study was the effect on participants’ mood as assessed by the PANAS,354 a reliable and validated359 20-item questionnaire (see Appendix 14) that assesses mood and emotional engagement in terms of emotional state. It consists of 10 items on the positive affect (PA) scale (attentive, interested, alert, excited, enthusiastic, inspired, proud, determined, strong and active) and 10 on the negative affect (NA) scale (distressed, upset, hostile, irritable, scared, afraid, ashamed, guilty, nervous and jittery). The PANAS was independently completed by all participants on two occasions, once before the training session (baseline) and once immediately after (post).
Scores for the PA and NA scales were derived separately for both baseline and post-session results. Participants gave each of the 20 items on the PANAS a score between 1 (‘very slightly or not at all’) and 5 (‘extremely’). An overall PA score may range between 10 and 50 with a high score indicating a more positive mood in the participant. An overall NA score may range between 10 and 50 with a high score indicating a more negative mood.
The trainees from both the intervention group and the control group were also asked to suggest three learning points that they would take away from the session that they would try and implement into their practice. The lists were analysed and compared to measure suggested differences in learning outputs between the groups.
Sample size
Initially, the RCT was planned to run over 1 academic year involving approximately 150 participants (15 participants on each of the 10 training days). The 15 participants would be individually randomised into either the control group or the intervention group, that is, seven to eight participants in each arm for each training day. Despite this being an individually randomised trial, it was considered prudent to allow for a clustering effect within each centre using an ICC of 0.05. Assuming seven individuals per group and combining this with the ICC resulted in a design effect of 1.3. Dividing the number of participants in the trial by this design effect gave an effective sample size of approximately 115. Little to no attrition was expected as the study formed part of compulsory training for participants and randomisation occurred directly before teaching. No increment for attrition was therefore made. The effective sample size of 115 gave 80% power to detect an effect size of approximately 0.53 should one exist.
Randomisation
Simple randomisation was carried out at an individual level at each centre on a 1 : 1 basis once consent was obtained. This was carried out on site on the training day by an independent administrator using a randomisation sequence generated using randomly ordered envelopes containing allocations. The assignment in the first envelope was given to the first individual as defined by registration and so on. Trainees who did not consent to the study received the same teaching as the control group and did not complete any questionnaires.
Blinding
The design of educational interventions is complex and problematic. 360 The allocation within this trial was concealed, which is important for randomisation. 361 Participants completed the APSQ baseline questionnaire prior to randomisation and the PANAS baseline questionnaire immediately before the training session started. As the sessions took different formats, participants were aware of their assignment for the training.
Analysis
Statistical methods
Analysis was conducted in Stata version 13 following intention-to-treat principles. Statistical testing was conducted at the 5% significance level using two-sided tests and consisted of t-tests where normality could be assumed and Wilcoxon rank-sum tests where it could not.
For the primary analysis, a regression-based approach compared differences in overall post APSQ scores between the two randomised groups after adjustment for baseline APSQ score and training centre. As a sensitivity analysis, the primary analysis was repeated twice, once without adjusting for training centre and once without adjusting for baseline APSQ score. A t-test was used to compare post APSQ scores in individuals with and without baseline data to assess whether or not data were missing completely at random.
Secondary analyses for the APSQ compared differences in each post APSQ subscale mean score between the two randomised groups using Wilcoxon rank-sum tests where normality could not be assumed and t-tests where normality could be assumed. Appropriate testing was also used to compare differences between the two groups in relation to the additional knowledge subscale. All analyses were repeated to compare differences in follow-up APSQ scores between the two groups.
Analysis of PANAS scores involved using Wilcoxon rank-sum tests to compare post PA and post NA scores between the intervention group and the control group.
Qualitative methods
Qualitative analysis358 of the lessons learned was conducted initially on the data set from 2011. Two researchers independently coded the learning points made by the trainees; these could not be blinded as trainees were easily identifiable by their group. Any discrepancies were resolved by discussion. A large coding frame of 26 codes was developed and collated into five overarching themes. With this framework, two of the patients also analysed the data to interpret the lessons learned in light of the coding frame provided. This provided a consensus on the reliability of the data set. The 2012 data were analysed using the same coding frame; no further subthemes were elicited at this stage. Participants were asked to identify three learning points that they would take into their practice. Some participants listed a variable number of ranked responses; the majority listed three lessons learned but there were one, two and four responses as well as some individuals who did not complete this element within the data set. When the lesson learned could be interpreted in a number of different ways, this was resolved after discussion within the research team and the appropriate code applied.
Economic evaluation
An economic evaluation was carried out in terms of the cost of the intervention over and above the cost of the control group training. Some of this cost was recurrent, for example training for patients, transport, accommodation expenses.
Results
Participant flow
In total, 313 individuals were eligible to participate, of whom five declined (1.6%) and 25 (8.0%) did not attend. Of the 283 participants who attended and consented to participate, 141 were randomised to the control arm and 142 to the intervention arm (Figure 20). There was a large amount of missing or incomplete baseline information for the APSQ (49.8%), with completion being better in 2011 (18.0% of 150 participants were missing data) than in 2012 (85.7% of 133 participants were missing data). For those who completed the APSQ at baseline, the mean baseline scores were similar between allocated groups (Table 53) and between centres (Table 54).
Data | Intervention (n = 142) | Control (n = 141) | Overall (n = 283) |
---|---|---|---|
APSQ dataa | |||
Total score, mean (SD) | 134.11 (11.67) | 132.79 (12.00) | 133.39 (11.83) |
Total score, median (min., max.) | 132 (114, 163) | 135 (99, 153) | 133 (99, 163) |
Total score, IQR | 125–141 | 125–141 | 125–160 |
Complete baseline data, n (%) | 65 (45.8) | 77 (54.6) | 142 (50.2) |
Completely missing baseline data, n (%) | 62 (43.7) | 59 (41.8) | 121 (42.8) |
Partially missing baseline data, n (%) | 15 (10.6) | 5 (3.5) | 20 (7.1) |
PANAS dataa | |||
Total PA score, mean (SD) | 23.85 (7.31) | 22.96 (7.81) | 23.44 (7.54) |
Total PA score, median (min., max.) | 23 (10 to 44) | 22 (10, 43) | 23 (10, 44) |
Total PA score, IQR | 18–29 | 17–28.5 | 17.5–29 |
Total NA score, mean (SD) | 12.89 (3.72) | 12.33 (3.72) | 12.66 (3.74) |
Total NA score, median (min., max.) | 12 (10, 30) | 11 (10, 28) | 11 (10, 30) |
Total NA score, IQR | 11–14 | 10–13 | 10–13 |
Complete baseline data, n (%) | 132 (93.0) | 112 (79.4) | 244 (86.2) |
Completely missing baseline data, n (%) | 8 (5.6) | 23 (16.3) | 31 (11.0) |
Partially missing baseline data, n (%) | 2 (1.4) | 6 (4.3) | 8 (2.8) |
Data | Hull (n = 110) | York (n = 96) | Scarborough (n = 31) | Grimsby (n = 25) | Scunthorpe (n = 21) |
---|---|---|---|---|---|
APSQ data | |||||
Complete baseline APSQ data, n | 44 | 66 | 13 | 8 | 11 |
Total score, mean (SD) | 134.34 (10.01) | 133.12 (13.59) | 132.15 (10.40) | 133.75 (14.41) | 132.45 (7.58) |
Total score, median (min., max.) | 134 (115, 153) | 133 (99, 163) | 132 (111, 148) | 160 (114, 152) | 135 (119, 144) |
Total score, IQR | 127–141.5 | 124–142 | 125–139 | 123.5–148 | 126–138 |
Complete baseline data, n (%) | 44 (40.0) | 66 (68.8) | 13 (41.9) | 8 (32.0) | 11 (52.4) |
Completely missing baseline data, n (%) | 60 (54.5) | 21 (21.9) | 15 (48.4) | 16 (64.0) | 9 (42.9) |
Partially missing baseline data, n (%) | 6 (5.5) | 9 (9.4) | 3 (9.7) | 1 (4.0) | 1 (4.8) |
PANAS data | |||||
Complete baseline PANAS data | 102 | 72 | 26 | 25 | 19 |
Total PA score, mean (SD) | 24.27 (7.87) | 23.40 (7.04) | 24.35 (7.20) | 18.88 (6.08) | 23.89 (8.29) |
Total PA score, median (min., max.) | 24 (11, 44) | 23.5 (10, 42) | 23 (14, 39) | 17 (10, 30) | 23 (10,41) |
Total PA score, IQR | 18–29 | 18–28 | 20–30 | 15–22 | 17–30 |
Total NA score, mean (SD) | 12.78 (3.45) | 12.72 (4.32) | 12.19 (2.95) | 13.00 (4.65) | 11.63 (1.95) |
Total NA score, median (min., max.) | 11 (10, 24) | 11 (10, 30) | 11 (10, 22) | 11 (10, 28) | 11 (10,16) |
Total NA score, IQR | 10–14 | 10–13 | 10–13 | 10–14 | 10–13 |
Complete baseline data, n (%) | 102 (92.7) | 72 (75.0) | 26 (83.9) | 25 (100.0) | 19 (90.5) |
Completely missing baseline data, n (%) | 5 (4.5) | 22 (22.9) | 4 (12.9) | 0 (0.0) | 0 (0.0) |
Partially missing baseline data, n (%) | 3 (2.7) | 2 (2.1) | 1 (3.2) | 0 (0.0) | 2 (9.5) |
The PANAS had a much smaller proportion of completely missing baseline data, varying among the centres between 0.0% and 22.9%. Baseline scores were similar by allocated group for both the PA scale and the NA scale. The baseline total PA score was similar among the centres with the exception of Grimsby, which had a lower total mean score of 18.88 (SD 6.08). The baseline total NA score was also similar across centres.
Attitudes to patient safety
Results from analyses conducted on the overall APSQ scores are shown in Table 55. The primary analysis was conducted on 122 individuals, 56 from the intervention group and 66 from the control group. There was no evidence of a difference in post-training APSQ scores between the allocated groups, with a non-significant increase of 0.17 in overall post score (95% CI –3.96 to 4.29; p = 0.94) for participants in the intervention group compared with those in the control group. This relates to a Cohen’s d effect size of 0.01 (95% CI –0.35 to 0.37). Analysis to assess for a difference in follow-up APSQ scores was conducted on 90 individuals (37 intervention, 53 control). There was no evidence of a difference between the allocated groups, with a non-significant decrease of 1.04 in overall follow-up score (95% CI –5.16 to 3.07; p = 0.62) for participants in the intervention group compared with those in the control group. This relates to a Cohen’s d effect size of –0.09 (95% CI –0.50 to 0.33).
Group | Baseline, mean (SD) | Immediately following training (post), mean (95% CI) | 3–6 weeks following training (follow-up), mean (95% CI) |
---|---|---|---|
Intervention | 134.11 (11.67) | 134.73 (131.71 to 137.75) | 134.16 (131.01 to 137.32) |
Control | 132.79 (12.00) | 134.56 (131.78 to 137.35) | 135.21 (132.58 to 137.84) |
Difference | – | 0.17 (–3.96 to 4.29) | –1.04 (–5.16 to 3.07) |
Statistical testing on all post APSQ subscale means was conducted on 236 individuals (126 intervention, 110 control). Only ‘patient involvement’ showed a statistically significant difference between the two groups (p < 0.01), implying that the intervention may have changed attitudes about the importance of the role of patients in patient safety more than the standard teaching. No difference between the randomised groups was found in relation to the additional knowledge subscale.
Statistical testing on the follow-up (3–6 weeks after training) APSQ subscale means was conducted on 184 individuals (88 intervention, 96 control). There was no evidence of a statistically significant difference between the two randomised groups on any of the subscale scores. One likely explanation is that there was communication between the two groups post intervention.
When the primary analysis was repeated without adjustment for centre as a sensitivity analysis, the results were consistent (p = 0.99). Given the number of missing baseline data, a post hoc analysis repeated the primary analysis without adjustment for baseline APSQ score. Analysis was conducted on 236 individuals; a non-significant increase of 1.60 (95% CI –2.19 to 5.39; p = 0.41) for those in the intervention group compared with those in the control group was found.
A t-test used to compare the difference in post APSQ scores between those with complete baseline data and those without provided no evidence against data being missing completely at random (p = 0.92). This remained the case when each randomisation group was considered separately.
Emotional engagement
Wilcoxon rank-sum tests were used to investigate differences by allocated group for both the PA and NA scores; analyses were conducted on 263 individuals (138 intervention, 125 control). There was evidence of a statistically significant difference in the underlying distribution of both post PA and post NA scores between the two randomised groups (p < 0.001), with indication of both higher PA and higher NA scores in the intervention group. The control group showed an average decrease of 0.13 (SD 6.24) in overall PA score from baseline, whereas there was an average increase of 3.70 (SD 6.75) in the intervention group; this difference of 3.83 points between the randomised groups relates to a Cohen’s d effect size of 0.59 (95% CI 0.32 to 0.85). In terms of overall NA score there was an average increase of 0.21 (SD 3.47) from baseline for individuals in the control group, whereas for those in the intervention group there was an average increase of 3.11 (SD 5.96); this difference of 2.9 points between the randomised groups relates to a Cohen’s d effect size of 0.58 (95% CI 0.31 to 0.84).
Learning points
The codes derived from the analysis were first grouped into 26 categories. These were then grouped into five overarching themes:
-
risk management and governance (report more errors/near misses; working safely; clinical governance; risk assessment/management)
-
learning about error (decision making; attention to detail; minimising error)
-
communication (speaking up about errors; communication in teams; involve patients in discussions; asking for help; no room for arrogance as a doctor)
-
processes related to patient safety (documentation in notes; relying on gut feeling; attention to detail; follow-up on clinical investigations)
-
role of education (Increasing knowledge; safe prescribing; ethical practice).
The ranked frequencies of each theme, when the two randomised groups were compared, demonstrated a difference in the focus of central learning issues.
Participants in the control arm ranked items in risk management and governance (pertaining to elements of reporting errors and near misses) and processes (documenting accurate notes, attention to detail, following up results) most highly, whereas those in the intervention group ranked items pertaining to communication (with both patients and colleagues) most highly. Highlighted within this was the lack of a perception of emphasis on communication by trainees in the control arm. Allied with this theme was the appreciation of the necessity to challenge senior colleagues to prevent errors and of speaking up and honesty after an error has occurred.
Economic evaluation
The cost of running the patient-led intervention for 2 years, including PLJ training, travel, overnight accommodation and subsistence, was £10,524.00. This was over and above the cost of running the control teaching sessions.
Discussion
Interpretation
There are few rigorous evaluations of different methods of delivering training in patient safety. In this study we compared two forms of training, a standard classroom method and a patient narrative approach, using a RCT design. In education research the impact of training is often difficult to measure reliably because of the multitude of variables that need to be considered and a lack of clarity on outcome measures. In evaluating patient safety interventions, although there are a few validated measures of the safety climate,362,363 these tend to measure organisational culture rather than individual attitudes. Subsequent to the start of this study, new measures to test individual junior doctors’ knowledge of and attitudes towards patient safety are being reported. 342 These may need to be adopted as outcome measures in future educational interventions in patient safety.
The APSQ and PANAS baseline scores were similar in the two groups. The mean attitude scores were strongly positive towards patient safety and in keeping with similar cohorts reported in the literature. 364 For the APSQ subscales, as anticipated, there was a significantly increased score for the intervention group in the area of patient involvement in reducing error. Although the intervention did not appear to impact on general patient safety attitudes, it did lead to significantly higher scores for the intervention group in the area of patient involvement in reducing error. It is possible that the lack of identification of a difference in general patient safety attitudes was because the study was, in effect, underpowered because, despite having doubled the sample size, the number of missing baseline APSQ data affected the numbers analysed. However, the results were consistent when the primary analysis was repeated without adjustment for baseline APSQ score.
In keeping with Kumagai’s48 framework, the patient narratives helped ‘communicate meaning’ by evoking an emotional response among the participants. What was interesting was that the narratives seemed to trigger both positive and negative affects, indicating emotional and/or cognitive concordance as well as dissonance among the trainees. In general, doctors learn to distance themselves from their own and their patients’ emotions. 365 However, medical educationalists are now reminding us that medicine ultimately involves interaction with real people, with real emotions,366 and awareness of emotions and how to deal with them should be as much a part of medical education as development of clinical skills.
The emphasis on health organisation and policy and a focus on the technical aspects of risk management by participants in the control group contrasted with the focus on communication and the patient in the intervention group. This highlights the need for safety training programmes to review their objectives to ensure that both aspects of patient safety are addressed. Future programmes may need to combine both professional and patient input to maximise their impact on safety awareness.
The trial had a number of strengths. It demonstrated the feasibility of implementing a RCT, which is unusual in studies on patient involvement in education. It also brought the focus back to the patient as the centre of patient safety interventions, allowed collaborative working between patients and researchers and emphasised to junior doctors the terrible impact of safety incidents on patients and their families.
Limitations
There are difficulties with the design and implementation of randomised trials in education, resulting in a limited amount of literature reporting such studies. 367 Randomised trials in education may suffer from performance bias (i.e. the teacher’s approach may be different because of the novelty of the intervention) as well as a placebo effect because of learners perceiving the novel method of teaching as better. 368 However, randomising students to different interventions allows educationalists to examine differences resulting from the intervention alone rather than other variables. In this trial, attempts were made to standardise the intervention across the 10 sites by trying to adhere to the broad learning outcomes for the sessions, using the same team of researchers to facilitate all of the sessions and the patients being asked to maintain consistency in their narratives. However, the nature of the intervention meant that there were variations in the key safety issues emerging from the patient stories, with subsequent variations in discussions. There were also expected variations in the examples of safety incidents that the trainees brought up for discussion. The irrelevance of post-randomisation blinding in educational trials has been discussed previously.
For the control arm, although the teaching materials were identical for each control group session, the clinician tutor varied between teaching sites. This may have resulted in a variation in emphasis within each session, influenced by the beliefs, values and personal experiences of the clinicians.
This trial was also limited by the number of missing or incomplete baseline APSQ data, leading to a reduction in power. We wished to measure the attitude of trainees prior to randomisation and preferably a few weeks prior to the teaching session. As we had no control over this, a number of trainees turned up to the sessions not having completed the APSQ. On the other hand, the level of missing data for the baseline PANAS questionnaire was very low, probably because the trainees completed this just before the teaching session.
Generalisability
This trial was conducted in one foundation school and a very high proportion of attendees consented across the five centres. Although there is no reason to believe that FY1 trainees in the NYECFS are different from those elsewhere in the UK, the generalisability of the results cannot be confirmed because of a lack of demographic information. The low number of trainees who declined to participate in the trial provided some evidence of the acceptability of the intervention.
Conclusions
This study demonstrated the successful implementation of a RCT in medical education. Involving patients with experiences of safety incidents in patient safety training has an ideological appeal and seems to be an obvious choice when designing safety interventions. However, on the basis of our primary outcome measure we were unable to demonstrate that the intervention was any more effective than standard teaching in changing general attitudes to patient safety. This may well be because of the inherent difficulty in determining valid outcome measures to study the effectiveness of educational interventions. It is also not possible to rule out whether or not the lack of difference was caused by the study being, in effect, underpowered to detect differences on the primary outcome because, despite having doubled the sample size, the number of missing baseline APSQ data affected the numbers analysed. The emotional impact of the narratives and learning about communication suggests an important area that could be targeted to facilitate development of patient safety awareness in medicine. We remain uncertain about whether or not emotional engagement will translate into improved behaviours in the clinical context or indeed if there are any negative effects.
Recommendations for future research
-
To investigate the impact of emotional engagement with patient narratives on behavioural change in clinical practice.
-
To develop valid and sensitive outcome measures to measure the effect of interventions in patient safety training.
Recommendations for practice
-
Patient safety training for doctors may benefit from the incorporation of patients’ views as well as professionals’ views. Patient stories about harm provide an emotionally engaging method that can be adopted into clinical training.
Chapter summary
This chapter has presented a trial of an innovative approach to bringing the patient experience centre stage in patient safety training for medical undergraduates. The findings are consistent with an interpretation that patient involvement in patient safety education may serve to ‘humanise’ content, although the long-term effects of this are, as yet, unknown. There are clear avenues to further extend this work, not least the impact of involvement on the patients themselves. One of the key features of the development of this intervention was the extensive involvement of patient representatives from the outset. Indeed, this has been a key principle of our research across the programme grant as a whole. The following chapter presents an evaluation of the approach to patient involvement in the conduct of the research undertaken within the programme grant.
Publication statement
Data presented within this chapter have been previously published. 369 We reproduce these data here with permission under the terms of the Creative Commons Attribution Non-commercial License.
Chapter 11 Lessons from patient and public involvement: development of a model of coproduction
Abstract
Background: A lay-person panel (LPP) was established to support patient and public involvement (PPI) within this research programme. An evaluation was designed in two stages (mid-point and end), which aimed to (1) inform the team’s own learning around PPI and so support the development of the LPP and (2) develop general recommendations around the process of managing, evaluating and assessing the impact of PPI in health research.
Methods: Senior researchers and lay people were interviewed at the mid- and end points for their perspectives on impact and experience and findings were analysed using existing frameworks for quality and impact. At the end point they were also asked about their perspectives on changes introduced to the LPP as a result of the mid-point evaluation.
Results: At the mid-point lay people reported positive personal experiences and researchers reported some gains to the research design. A diversity of opportunities in different projects was revealed, leading to some tensions that needed resolving to ensure that the LPP developed effectively for the second half of the research programme. As a result a coproduction model for ongoing development of the initiative was subsequently introduced, based on action research principles. Evaluation at the programe end point revealed that this improved the experience for all with respect to relationships between researchers and lay people. However, it did not enable consensus around purpose across projects to be achieved.
Conclusions: A generalised coproduction model is presented that could provide the systematic rigour required to develop impact measures collaboratively at the outset of a PPI initiative that can be regularly reviewed and evaluated. This requires facilitation and time and is likely to be most feasible in single-project teams in which group consensus can be reached.
Chapter rationale
The involvement of patients in patient safety was an overarching aim of the programme grant as a whole. However, in addition to investigating empirically the potential role of patients in improving patient safety, we also aimed to progress and investigate the role of patients as part of the research process itself. This chapter describes the formative and summative evaluations of the PPI strategy for the programme grant, as part of an action research process.
Introduction
Patient and public involvement is central to this research programme and a LPP was established at the outset to contribute to the four individual projects as they progressed from design to completion. Guidance provided by INVOLVE,51,52 along with the research team’s previous experience, informed the plan and conduct of the PPI process (recruitment, training, reimbursement). However, in the absence of generalised guidance on anticipated impacts of PPI, no outcome measures of success were set.
The evaluation of the LPP therefore had two aims: (1) to inform the team’s own learning around PPI and so support the development of the LPP and (2) to develop general recommendations around the process of managing, evaluating and assessing the impact of PPI in health research. The evaluation was conducted at two time points: the programme mid-point (evaluation 1) and the programme end point (evaluation 2). Findings from evaluation 1 were used to inform the LPP development for the rest of the programme. At both time points the evaluation was designed to assess the impact and experiences of the initiative from researcher and lay perspectives. Figure 21 summarises this two-part evaluation design.
The lay-person panel: an evolving patient and public involvement initiative
The LPP was established to facilitate layperson input into the design, conduct, analysis and dissemination of each of the four projects:
-
developing the PMOS181
-
developing a patient-led PIRT (projects 1 and 2 are now combined and operate as one)
-
developing an intervention to help patients improve their own safety
-
a randomised trial investigating patient involvement in education and training. 356
The format was chosen because it was a commonly used approach at the time. One study revealed that most PPI activities (65%) involved membership of an advisory panel or project steering group. 370 Governance arrangements were set by the team informed by members’ own judgements about the role PPI could play. In the early stages of planning it was agreed that this format was not appropriate for project 4, in which patients were already involved significantly within the research design. Figure 22 provides an overview of the LPP as it was established.
The LPP included two members of the public per project as lay representatives to feed back on key decisions and issues as they arose. It was anticipated this would involve recommending how the patient’s voice could be strengthened within interventions being developed through the research, reviewing results and implications for patients and recommending how best to implement findings into practice. Precise involvement activities were not specified at the outset. An additional two lay people were recruited as cochairpersons. Lay representatives would attend individual project meetings every 2 months. The cochairpersons would chair LPP meetings biannually to share progress across all four projects and feed back progress to, and receive updates from, the quarterly programme steering group meetings. All involved would be invited to a relevant annual workshop or conference to develop understanding of the research area.
In recognition that lay people would need training and support and that researchers may need advice throughout the process, a mentor (a member of the programme steering group not working directly on any of the projects) was allocated to the group. A panel convener (a member of the research team) was nominated to provide practical support (ensuring that meetings took place). All LPP members would be remunerated at hourly rates (LPP members £25 per hour; chair £30 per hour) in line with guidance at the time. 51
A collective recruitment process (local newspaper advert followed by selection day) was held for projects 1 and 2. Suitability was not assessed by strict criteria but through the research team’s subjective assessment of ability to express intentions, relevant skills and experience (e.g. community, voluntary or committee work) and a general interest in patient safety. A range of sex, age and sociodemographic characteristics was desired. All those invited to join the panel accepted. Two people from this process expressed interest in the LPP chair role and became cochairpersons. Led by researchers in a different geographical location the recruitment process for project 3 was different. The general public was not targeted; rather, members of an existing patient safety volunteer group were invited to submit a short biography. Selection was based on this information plus an informal telephone conversation.
The mentor provided the new recruits with training in the process of health research and the programme’s aims. A consensus activity was then held, led by the cochairpersons, to collate a general LPP role description. This included meeting arrangements and expectations of researchers and lay people with respect to these meetings and was documented in terms of reference.
Evaluation design
Researchers not directly linked to the projects formed an evaluation team. In planning evaluation 1 this team considered the question of quality in PPI to set some parameters against which to evaluate. By the mid-point it was increasingly recognised that INVOLVE’s categorisation for involvement – (1) consultation, (2) collaboration and (3) user controlled – should not imply a hierarchy of quality but that different approaches suit different research projects and even different stages within projects. Quality is a product of the processes through which approaches are negotiated, planned and reviewed. 291,371,372 The evaluation was therefore framed to reveal not only what had been achieved but also the experience of these processes for all concerned.
With respect to measuring achievement no impact measures had been predetermined but there was emerging knowledge around potential impact areas. A categorisation by Morrow et al. 371 was used to frame the impacts revealed as being on (1) members of the public, patients and research participants involved; (2) the research process (agenda, design, ethics and outcomes); (3) the researchers involved and (4) the wider context (community, community organisations, implementation and change). These categories were used in evaluation 1 to identify progress to date and potential for the future. The factors affecting progress in these areas, the priorities of those involved, how priorities were negotiated and the experience of this process were also examined, guided by a Quality Involvement Framework (QIF). 373
The QIF (summarised in Table 56) presents the PPI experience as a dynamic process involving lay people interacting with a research context. It is informed by theories of power and empowerment, helping us to understand what lay people are able to do (e.g. achieving goals, expressing views, making decisions), what potential they bring to develop their role (e.g. how loyal they are to certain ideas, whether or not they identify their interests) and how the process of involvement makes them feel (e.g. valued, enabled, conscious of power). The research context also has a significant influence on the experience of those involved and the themes of ‘research relationships’ (e.g. incentives for particular activities, communication structures, expectations), ‘ways of doing research’ (e.g. roles available to users) and ‘structures’ (research organisation, ethics) are included to help identify this influence. The QIF was originally designed to interpret the stories of lay people and how their experiences are influenced by the research context, but it is as important to understand the perspectives of the researchers, particularly because they play such an essential role in interpreting the research context. The QIF themes were therefore used also to explore researchers’ perspectives on the process.
Personal factors (users) | Research context |
---|---|
Ability to:
|
Relationships:
|
Findings from evaluation 1 led to some significant changes in LPP functioning that aimed to improve the negotiation process for the second part of the programme. Efforts were made for researchers and LPP members to interpret evaluation 1 results together, revise structures accordingly and establish agreement around intended impact (and actions). At this time there was increasing evidence from the wider PPI community of the need to systematise feedback on progress into ongoing functioning of PPI initiatives. 374 The evaluation team took on this role and applied action research principles375 to present PPI to the LPP members and researchers as a topic for shared learning through reflection on their own practice, setting up an ongoing formative evaluation process for the remainder of the programme (Figure 23).
Drawing on models of action research to support organisational development (e.g. Coghlan and Brannick376), the evaluation team facilitated cycles of action planning and reflection. To start the cycles they hosted two workshops to interpret findings from evaluation 1 to agree shared purpose (impact areas) and from this decide what actions were required to improve the LPP. In addition to these formal workshops, the evaluation team organised a debate so that contentious issues identified in evaluation 1 could be aired in a more informal yet transparent forum. They also ran formative evaluation sessions in which researchers and LPP members were asked to reflect on progress and review actions at 6-monthly intervals. These sessions were conducted with researchers as a group, then with lay people as a group and then together, with a view to addressing any challenges.
Evaluation 2 differed significantly from evaluation 1. The action research cycles led to a better understanding of potential impact against which to summatively evaluate success. Also, there was a baseline of experience revealed in evaluation 1 against which to assess improvements. Any changes and achievements that the action research model itself may have contributed were also examined. Consensus over impact areas was still limited among the research team and so impact was assessed again from the different perspectives of all involved rather than through objective measurement of impact indicators.
Evaluation objectives
-
Assess impact and experience at the mid-point from the researcher and lay member perspectives (evaluation 1).
-
Use findings from evaluation 1 to recommend revision of structures, support LPP development and propose impact areas for future evaluation.
-
Assess impact and experience at the end point from the researcher and lay member perspectives (evaluation 2).
-
Assess the role of the action research approach in supporting LPP development (evaluation 2).
Evaluation methods and analysis
The local research ethics committee and the local NHS trust research and development department held that no ethical approval specific to the evaluation was required. However, research governance principles377 were followed. In evaluation 1, all participants (lay members and researchers) were given an information sheet and consented. For evaluation 2, consent was not taken in writing but verbally at the start of each interview as by this time all those involved were now more aware of the purpose of the evaluation.
Evaluation 1
Data collection
At the mid-point (2.5 years), semistructured interviews with lay members and researchers were conducted to address their views on impact, experience and what could be improved (objectives 1 and 2). In the absence of predetermined outcome indicators, broad interview topic guides were devised to find out the specific activities that lay members had been involved in, what they thought had been achieved, their experience of key aspects of the LPP initiative to date (recruitment process, regular project and panel meetings, steering group liaison, mentorship, training and communication channels) and what could be improved. All those lay members still involved (four lay members and two cochairpersons) were interviewed. Lay members who had left the LPP were invited to take part but declined. Ten researchers linked to the three participating projects were interviewed. One researcher linked to project 4 was interviewed to explore why the LPP had not worked for this project.
Analysis
A thematic analysis grounded in the data268,378,379 was used to derive common themes from the interview transcripts. Consensus over a coding scheme was developed by three members of the evaluation team independently coding four transcripts. The remaining interviews were then analysed by one researcher, supported by the software package NVivo 9. Two frameworks were used to interpret the themes. With respect to ‘experience’ the QIF373 was used. With respect to ‘impact’, findings were organised around the categorisation provided by Morrow et al. 371
Group interpretation exercises
Evaluation 1 findings were presented back to researchers and lay members in a group workshop facilitated by the evaluation team, designed to validate the findings with all participants and start identifying potential for action. Two subsequent facilitated workshops then completed this process. The first was held with researchers and lay members together and the second with lay members only. Facilitation involved ensuring that the findings had been interpreted correctly as well as helping individuals articulate their own responses to these and how they would like to respond.
Evaluation 2
Data collection
Individual interviews were conducted with lay members (n = 3) and researchers (n = 12) at the end point (objectives 3 and 4). Topic guides covered the four potential impact areas around which actions had been agreed after evaluation 1. They also covered ‘experience’, specifically around the new structures that had been put in place. Finally, they included questions about whether and how the action research approach (facilitated interpretation sessions using the results of evaluation 1; debate; facilitated review sessions; other) had helped or not and general reflections on what supports quality PPI.
With respect to objective 4 the evaluation team also met at the end point to reflect on the process of facilitating using the action research model (see Figure 23) as the basis for reflection. In this meeting a summary of the key successes and challenges from the evaluation team’s perspective was derived.
Analysis
Interviews were digitally recorded, transcribed and analysed qualitatively. This time analysis was more structured because there was a clear idea of what issues needed to be covered. Framework analysis358 was therefore used to categorise findings into issues of experience and impact. A more open thematic analysis (as for evaluation 1) was used to interpret the reflections on the action research model from the researcher and lay member interviews and the evaluation team reflections, so as not to presume what this may or may not have achieved.
Evaluation findings
Evaluation 1
Experience
To understand the experiences of all those involved, it is first necessary to outline the different activities that lay people were involved in. They all received the same initial training from the mentor and this was very well received, leading to a consensus activity around a generic role description:
We had an excellent training session . . . this is what the NHS is, this is what it’s about, this is the pattern that you’re going to play and it was great, I think everybody enjoyed that.
Lay member 01
Actual roles undertaken by LPP members evolved differently in different projects (Table 57).
Stage | Activities |
---|---|
Research design | Practical suggestions regarding how to recruit research participants (1, 2 and 3) |
Testing and reviewing patient questionnaires (1 and 2) | |
Testing and reviewing patient safety materials (3) | |
Data collection | Identifying potential patients for recruitment (3) |
Data interpretation | Reviewing focus group transcripts (1), reviewing transcripts and using these to develop themes for a patient questionnaire (2 and 3) |
Dissemination | Programme-wide newsletter (3) |
Explaining PPI to clinicians and student health professionals (1, 2 and 3) |
At the research design stage lay members attached to all projects were very active, advising on the development of patient safety tools for use with patients on wards by attending workshops with researchers and clinicians (projects 1 and 2) and helping to develop patient safety awareness materials (project 3). They also provided suggestions on recruitment of research participants for trialling of these tools. At the data collection stage, when the tools began to be trialled with patients there was no involvement for lay members attached to projects 1 and 2. In contrast, lay members for project 3 did help by identifying which patients were eligible and willing to take part before they were formally consented by researchers. At the analysis stage lay members for all three projects were involved in interpretation of data (reviewing focus group transcripts in project 3; helping to develop themes for a patient questionnaire in projects 1 and 2). At the mid-point it was too early to review the dissemination phase in relation to project findings but lay members from all projects had led presentations to clinicians focusing on the process of PPI and the project aims.
These differences in involvement in data collection became a source of tension among lay members and researchers when those from different projects met together on the panel. The lay members for projects 1 and 2 reported a sense of disappointment:
I misguidedly thought that we would be talking to patients, I wouldn’t say I was disappointed but just surprised . . . it took us several meetings before it dawned on us or was made clear to us that because of the restrictions that were unavoidable we wouldn’t be . . . our work was in the meeting rooms.
Lay member 03
we’re all basically lay people putting in our life experiencesand mine’s no different from [the researchers] . . . so . . . I did expect to be involved in actual hands-on research, yes.
Lay member 06
The interviews with researchers helped understand why roles evolved differently. Perhaps most significant was the different researchers’ own beliefs in the rationale for PPI. For a researcher in project 3 this meant challenging the boundaries between lay input and professional expertise and empowering lay members to take on as much responsibility as possible. This was reflected in their decision to recruit people with considerable experience of patient safety. This combined with circumstance that one lay member on this project was particularly keen to have a central role. On the other hand, researchers in projects 1 and 2 believed that maintenance of a lay perspective was vital and that this could be lost if lay members did too much ‘hands-on’ data collection. As the methods for trialling a standardised questionnaire and reporting system were set, these researchers became clearer in their own opinions about the boundaries of involvement. These researchers felt that the process of identifying, consenting and conducting the questionnaire happened together and could not be broken down into parts that lay members could helpfully carry out. In project 3 it was appropriate to break the process down: patients were recruited for a longer-term study over the duration of their stay in hospital and so it was helpful for lay members to identify potential participants before the consent process was undertaken by researchers.
The tensions that developed around the comparisons of roles became a source of anxiety for some researchers in projects 1 and 2. Some reported that they felt powerless to express their position but that lay members often stated fixed views on how they should be involved. Panel meetings were therefore difficult at this stage. Researchers felt that they were obliged to enable lay members’ views to be aired but not their own:
I think what’s unclear is how we manage conflict, because it’s all about patient involvement and it’s all about empowering patients and, you know, you don’t want to be intimidating or make it difficult for them . . . at the minute, it feels like, when there’s conflict, it’s all about us trying to find a resolution that will be acceptable for a patient.
Researcher 11
The lay members attached to projects 1 and 2 developed their roles largely in isolation from those attached to project 3. The LPP meetings were held biannually and this appeared to be too infrequent to develop a whole-group identity. The divide between projects 1 and 2 and project 3 remained great. Individual project meetings were largely perceived to have worked well as they were always focused on particular tasks. For projects 1 and 2 there were fewer of these at the data collection stage, when less input was required.
As these challenges evolved it was not clear whose role it was to resolve them. The mentor was geographically remote from day-to-day project functioning and expected the cochairpersons to take on this role:
[The mentor] was meant to be there to try and help both sides work together a bit more effectively but [the mentor] saw it as something that they would be stepping back from once the chairs came in [but] that isn’t going to happen.
Researcher 07
Although researchers and LPP members were satisfied with the role of the cochairpersons in chairing the LPP meetings and reporting items from the Programme Steering Group their role in addressing LPP development was perceived to be underdeveloped. The cochairpersons were not directly involved with projects and so became out of touch:
Well I would have thought they would have been more involved in the projects just dropping in to see what’s going on, and phoning us and emailing us saying how you getting on sort of thing but they don’t.
Lay member 05
The role of the convener as administrative support for the LPP also became blurred with the cochairperson and mentor roles, with lay members going directly to the convener rather than to the cochairpersons or mentor:
If I had a problem I’d go to [the convener] . . . so I think the mentoring role needs a lot of clarification.
Lay member 06
Impact
The categorisation provided by Morrow et al. 373 was used to identify the impacts perceived by those involved on (1) the lay people; (2) the research process; (3) the researchers; and (4) the wider context.
Despite the tensions identified above and the fact that not all expectations were met, lay members reported positive personal impacts, especially increased understanding of the NHS:
. . . we don’t know what goes on in the NHS system behind the scenes . . ., you don’t realise how many talented, clever individuals are out there trying to make it work, we all think of doctors, nurses, surgeons and everybody else, we don’t see this kind of work going on so that’s been great.
Lay member 01
At least one lay member developed new skills and experiences:
It’s opened up my life an awful lot, I’ve got onto . . . giving presentations about patient[s] . . . and I seem to have become some local expert in it.
Lay member 06
Some researchers described impacts on their professional lives:
I feel really lucky to have had that opportunity to have gone through all of those experiences and emotions and have reached a point where actually I feel like I’m a much, much better researcher for it.
Researcher 04
Both groups reported positive impacts on the research itself, with one lay member describing a sense of reward from having contributed to the shape of the research:
. . . we had a senior anaesthetist here . . . he made a point that you can’t measure the effect of asking a patient to help has on them.
Lay member 03
The following impacts were described by the researchers:
I’ve probably taken this project further into the hands of the patient than I would have done.
Researcher 10
. . . we developed a set of ideas for patient reporting mechanisms and in order to go on and test them we had to bring them alive and to do that we put together a working group which included our patient panel members but also staff and ourselves as researchers – they had a voice that was the same as the researchers, the same as the consultants, surgeons, senior nurse. It was very much a collaborative effort, and I think all the better for it, I don’t think [as researchers we] would have come up with tools that would have been as good.
Researcher 04
However, not all researchers shared that view:
. . . in terms of making any big impact I don’t think it’s made a huge impact.
Researcher 10
The outcomes from the interpretative sessions held to agree impact areas, actions and revised support structures for the remainder of the LPP are detailed in Tables 58 and 59. It was clear from these sessions that the revision of support structures was the most urgent requirement if the experience of involvement was to improve and the issue of negotiating role diversity was to be addressed.
Impact area | Indicator | Agreed objectives |
---|---|---|
Research process |
|
|
Wider context |
|
|
Those involved as LPP members |
|
|
Researchers involved |
|
|
Who? | What? |
---|---|
Evaluation team |
|
Cochairpersons |
|
Researchers |
|
Mentor |
|
A role was given to researchers to conduct a regular ‘role review’ and the cochairpersons were to link more closely to specific projects to support this. In recognition of the fact that negotiation would require regular exposure of different expectations and viewpoints on the lay role the evaluation team agreed to act in a mediating capacity for the remainder of the programme. An action research model was subsequently adopted through which to support and review agreed actions in relation to each agreed impact area.
Evaluation 2
Experience of involvement: had this improved since evaluation 1?
Evaluation 1 had revealed tensions among both researchers and lay members about the differences in activities that lay people were involved in. In response the evaluation team held two facilitated workshops (one with lay members and researchers together and one with just lay members) to air these grievances and establish support mechanisms to enable ongoing role negotiation (changes to meeting frequency and structure to encompass role review). These two sessions appeared successful and made all concerned feel more positive about the LPP and accept that diversity of roles is inevitable.
The LPP members reported positively about this process:
It went very well . . . it was a full and frank discussion . . . the whole tone of things changed after that York meeting.
Lay member 03
I found establishing relationships with the researchers quite difficult at first. And I think after we’d had the sort of heart to heart as a group, then things became easier.
Lay member 05
The researchers also found it a useful step:
I think it’s got a lot better recently . . . I don’t know what [facilitator’s name] did with them . . . but whatever she did it turned them around.
Researcher 12
However, there was still a sense from both groups of uncertainty and anxiety about what exactly lay members should be able to do. Many researchers felt that the issue of managing expectations was still not resolved:
. . . a lot of work was upfront in that sort of strategic role, but once recruitment had kicked off there wasn’t much for them to do but that was a massive problem . . . in a sense the research team had to almost create jobs for them.
Researcher 12
Although the facilitated sessions helped everyone to move on, the period of low involvement that some had experienced still remained a sore point:
So the second quarter [of the research programme] we were virtually just little goldfish waiting for people to drop crumbs in.
Lay member 03
There was also a sense that this problem could be alleviated if clarity of lay role was achieved at the outset of research projects and some surprise and frustration that this had not occurred here:
. . . it just seems to me that there’s an interesting paradox there, between researchers who are very clear about questions, objectives and planning and then when it comes to PPI not to use the same thinking process.
Researcher 11
One lay member made the following recommendations:
. . . (a) clear goals from the outset i.e. what is trying to be achieved in terms of the research aim, (b) lay members given an indication what their individual roles will be, and (c) to discuss possible barriers to planned involvement activities, so that we are not disappointed if things change over the course of the research programme.
Although some would have liked a consistent approach to roles and boundaries, there were clear differences in opinion among researchers with respect to what these should be. Some felt that lay members should act in an advisory capacity only and not as lay researchers:
. . . because the research team doesn’t have a consensus about what their [the LPP members’] role is. My understanding is that they act in a strategic and advisory role not performing . . . research [activities] because they feel it is in their personal development plan.
Researcher 05
Others felt that involvement became ‘interesting’ at the point at which the lay members assisted to a higher degree, in this case with interpretation of the data:
. . . we’re not creating, sort of new qualitative researchers here; we’re building in a different perspective that enriches the analysis. And of course the leadership of that analysis, in that sort of example, lies with the qualitative researchers who have the skills to do that, but our experience has been that the involvement enriches the process and probably leads to more valid conclusions.
Researcher 11
Impact
Following evaluation 1, actions were identified to achieve progress against all four impact areas. Evaluation 2 was designed to reveal which impacts were most significant from all perspectives. At a personal level some lay members again reported positive impacts:
. . . my knowledge of research when I joined the research team was virtually zero, but I became very interested in research and how research works and they gave me the opportunity of presenting my experiences, now that’s something I’ve never done before . . . and I managed to develop some sort of talent for standing up there and talking in public. So that had a very big impact on me.
Lay member 06
Two lay members reported how they felt that they had learnt so much that they wanted to carry this into other related initiatives and they subsequently did. One lay member became involved in a related research project and another is developing peer support for other lay people across his affiliated research institute.
For researchers the personal impacts overlapped with the impact on the quality of the research. They were pleased to have developed their own PPI practice and recognised benefits to the research itself. One researcher reported the benefit of being able to ask the LPP members to look at data through their lens:
. . . we were struggling as a team to actually interpret the [data] because I think we had a very theoretical model in our head . . . that was almost limiting our ability to see beyond that. So I asked the patient panel to look at the items that were included in each of the factors and then give the factors a name based on their feelings about the items and how they hung together. And that was a really useful exercise and I think when we come to write up the next iteration of the factor analysis we will use that, their mental models in that approach.
Researcher 08
This was echoed by one lay member:
I’d say we’ve helped to bridge to the theory–practice gap because it’s all very well doing research for the sake of research but you have to put it into a patient’s perspective, especially patient safety research.
Lay member 05
With respect to the impact on the community there was nothing reported relating to the public, but significant impact was reported on practice in one of the participating research institutes, where the researchers and lay member now use their experiences to inform development of an institute-wide lay community ‘alliance’, to support the PPI process:
The lay alliance has been absolutely critical and will continue to be.
Researcher 11
The action research approach
Both researchers and LPP members had little to report on the action research approach with the exception of the two facilitated workshop sessions for developing actions in response to evaluation 1. They appeared less aware of this systematic approach and the expectations of them to engage, specifically the additional responsibilities for mentoring allocated to the senior research fellows:
I didn’t know it was a specific action research model and I’ve also never been told that I was supposed to have more of a mentoring role so in terms of the findings, this has not been made explicit to me.
Researcher 12
At the evaluation team’s own reflective meeting, members revealed that they also believed that the facilitated workshops were well received by researchers and lay members and influential in addressing many of the tensions. The evaluation team members were less satisfied with the ambition to involve all researchers and lay members as coresearchers in the process and management of PPI. Although this was agreed at meetings there was a sense that people did not fully commit to this approach. This was demonstrated by a lack of ownership around the impact measures and actions documented and a failure to reach consensus over which impact measures were most important. This made reflection on progress difficult, specifically with regard to what impacts were being reviewed. Despite these sessions not fulfilling the remit outlined in the action research approach, the team felt that they were useful for reminding people about specific actions.
Discussion
Our findings support the need for formative as well as summative feedback mechanisms when evaluating PPI in research. Supporting theoretical models have since been published with respect to the dialogue needed to support both types of feedback. Oliver et al. 380 present a framework for dialogue around role diversity and Stewart and Liabo381 present one for dialogue around role potential, purpose and why different people’s input may be sought at different stages. Although most of those involved reported that they would have liked to have arrived at consensus on roles and purpose at the outset so that expectations could be managed more effectively, caution must be observed here about how prescriptive it is possible to be. Indeed, people also reported the need for flexibility as projects develop, as they learn more and in response to the different skills, experiences and desires of lay people.
Our action research model offers one way of responding to the need for objective setting combined with flexibility. It includes the linking of dialogue (about people’s intentions and expectations) to defined impact measures and action planning and the need for regular progress review as projects develop over time. Independent facilitation was important to mediate the variety of opinions and expectations that surfaced, echoing what was already known: that people hold very different rationales for PPI that will influence the parameters and options they perceive. 370,382,383
We used the QIF,371 combined with a categorisation of potential impact areas,373 to analyse qualitative research findings for the start of our action research process. Other dialogue models may be more appropriate. We present a generalised model of action research to support a coproduced approach to PPI in a research project (Figure 24).
Getting people to view themselves as ‘coresearchers’ within such a process is a challenge and was not fully achieved in this example. The importance of the early communicative stages within action research is significant. 384,385 Reason385 describes the crucial role that ‘contracting’ plays in developing group commitment (p. 23), stating that it usually takes at least two exploratory meetings at which people clearly describe their expectations and needs, learn about the method itself and decide whether or not to take part. This did not happen here as we were introducing the approach at the mid-point and to solve already-existing tensions. There was little time or willingness for discussing this methodological approach. Such an exploratory exercise could be feasible at the outset of a project and would be easier with a smaller, less disparate team.
This evaluation revealed how problematic the issue of impact measures is, an issue not resolved in the wider PPI community. 280 Personal and professional positive impacts were evident for those involved here. There was also a perception that the research itself had been positively enhanced. There would need to be significant consensus over impact intentions to begin to measure impact more objectively than was attempted here. Our evaluation revealed why the establishment of indicators for measurement is problematic but we also use this example to present a model of coproduction that could be used to set impact measures collectively and to use them as the basis for both formative reflection and summative evaluation.
Limitations
The evaluation took place only at the mid-point and beyond. It would have been useful to have conducted interviews with the research team and lay members at the start to obtain a baseline of expectations and perspectives. We were unable to interview those lay members who had left, which meant that a potentially informative perspective was missed. The evaluation approach employed was also constrained by a lack of resources to conduct it as the facilitated action research model was not factored into the original project planning.
Conclusions
The following recommendations are made to others establishing a PPI initiative.
-
Dedicate time within a research team to choose a broad model of PPI that suits the approach and need. A consultative approach may suit teams more than a more collaborative approach; clarity of purpose may be enough to support such a decision.
-
If a more collaborative approach is agreed our coproduction model based on action research principles could help structure the dialogue phases, how to integrate these into cycles of action and reflection to support the evolution of PPI over a research project.
-
Such an approach requires commitment from the research team at the outset.
-
The systematic approach of this model could help provide the academic rigor so desired by research teams, combined with the flexibility required, for engaging the public as volunteers.
-
PPI can be enjoyable and rewarding if there is commitment and support to aid transparency and safe dialogue around different opinions and contributions.
Chapter summary
This chapter has presented the evaluation of the PPI approach taken within the programme grant. The findings suggest that the process of engagement is not straightforward and requires careful planning from the outset to ensure that the needs of all parties are met and that impacts on the research are meaningful and not tokenistic. The evaluation highlights the need for commitment from both researchers and patients in the process of coproduction and collaboration. The findings also suggest that the process of patient and public engagement in research requires an individualised approach with lay representatives and researchers all bringing different skills, needs and expectations to the collaboration. Finally, and perhaps echoing the findings from the programme grant as a whole, measuring the impact of the role of patients and the public on patient safety research is difficult and requires further investigation. The next chapter takes this further by carrying out an expansive assessment of the programme grant as a whole, summarising the key findings and presenting some key conclusions of the research, as well as recommendations for health professionals, health-care services, patients and researchers.
Chapter 12 Conclusions and recommendations
This programme of research has met its original aim to design, develop and evaluate innovative interventions to engage patients in preventing PSIs and protecting themselves against unintended harm. It has done so by focusing on four key areas:
-
assessing risk
-
reporting incidents
-
direct engagement in preventing harm
-
education and training.
In doing so, we have not just studied patient engagement in patient safety but produced measures, reporting systems, interventions and training that have the potential to reduce harm and improve the health of patients. We established comprehensive and effective systems for involving patients and the public in codesign and coproduction of applied health research and evaluated the impact of this involvement to provide lessons for future PPI in health-care research.
Overall, the programme has demonstrated that patients are able and willing to be involved in initiatives to improve patient safety across a wide range of intervention strategies. The findings suggest that future safety and quality improvement programmes should consider approaches involving patients and not just focus on health-care professionals and organisations. However, we were unable to demonstrate a clear impact on safety outcomes using interventions based on patient engagement and further development work and evaluations of such approaches are required.
Assessing risk and reporting incidents
A number of different studies contributed to the key programme areas of patient involvement in assessing risk and incident reporting. Initial research involved two systematic reviews. The first was undertaken to summarise factors contributing to PSIs in hospital settings. The findings of this review were successfully used to develop a framework of contributory factors to PSIs (the YCFF). This framework has the potential to be applied across hospital settings to improve the identification and prevention of factors that cause harm to patients. The second systematic review assessed and summarised research studies that had evaluated patient reporting of errors in health care.
These two reviews informed work to develop a patient-completed measure of organisational safety in a hospital setting and a system for patients to report PSIs.
We successfully developed a reliable and valid tool (PMOS) to offer a mechanism by which patients can provide feedback on factors that might contribute to PSIs in the future. Based on a clear theoretical framework, the PMOS tool uses the patient’s perspective to identify ‘latent’ weaknesses that could contribute to future events. We showed that patients are in a very good position to observe the safety of their care and the care of others on the same ward. These observations can capture areas of weakness in patient care that might otherwise go unreported or unidentified. To our knowledge this is the first such tool that has been developed to collect this type of information. It can be used as an adjunct to other patient safety tools such as incident reporting systems and can also reinforce ‘soft’ intelligence on the quality and safety of care on hospital wards. We would recommend that the PMOS tool is used in conjunction with other safety and patient experience measures to enhance the interpretation of feedback from patients.
We also developed and evaluated a system for patients to report PSIs – the PIRT. This tool was developed with extensive input from patients, drawing on principles of coproduction, and collected safety concerns from hospitalised patients. We compared the data from the PIRT with other sources of safety information (case note review, incident reports) and explored the extent of overlap. This work demonstrated that patients in an acute hospital setting can report their safety concerns and that their reports are rarely evident in other sources of patients safety data. Concerns were typically focused on events that were near to the patient and which were important to them (e.g. failure of health-care staff to demonstrate good hand hygiene practice) but reports were generally unlikely to incorporate mention of direct harm. This study adds to the growing evidence base that patients can, when asked, report on the safety of their care in hospital and also that of other patients in the same shared environment.
The PMOS and PIRT studies were used to inform the development of the PRASE intervention. This consisted of two tools: (1) a 44-item questionnaire that asks patients about factors contributing to safety (PMOS) and (2) a pro forma for patients to report both PSIs and positive experiences. A report to wards was then produced summarising this feedback and ward staff were asked to plan and implement actions with the aim of improving safety. The PRASE intervention was evaluated using a randomised design. Take-up of the intervention by wards and their retention was 100% and patient participation was high at 86%. However, compliance with the intervention, particularly the implementation of action plans, was poor. We found no significant effect of the intervention on either the primary outcome or the secondary outcomes at 6 or 12 months. Based on these findings, the intervention cannot be deemed to be effective. The intervention cost £1018 per ward. However, we did find some improvements in the intervention wards compared with the control wards for new harms (i.e. those for which the ward is directly accountable) and these differences were largest among wards that showed the greatest compliance with the intervention. Introducing such interventions into busy wards with significant existing demands on staff time did highlight barriers in terms of the capacity of some individuals, wards and trusts to innovate and change.
Direct engagement in preventing harm
The ThinkSAFE project involved the development and evaluation of an intervention to support patients to directly engage with health-care staff to enhance their safety through strategies such as checking that their care is delivered as planned and speaking up to staff if they had any concerns. A key criticism of previous initiatives to encourage patients to be more actively involved in contributing to the safety of their care – so-called ‘patient push’ interventions – has been the apparent lack of involvement of patients or frontline staff in their development. There have also been other deficiencies including the lack of any theoretical rationale for the choice of intervention approach or materials and a dearth of robust evaluations. 180,339 This project addressed these concerns through the systematic development of a fully piloted intervention grounded in patient and professional experience, underpinned by relevant theory and informed by research evidence and best practice. Four components of the ThinkSAFE intervention emerged:
-
a patient safety video
-
a patient-held health-care logbook incorporating tools to facilitate information sharing
-
a theory- and evidence-based educational session for staff
-
Talk Time – a dedicated for opportunity for patients and staff to interact.
The piloting of ThinkSAFE showed that the approach is feasible and acceptable to users and has the potential to improve patient safety. ThinkSAFE is currently being further refined and its implementation tested across acute hospitals in the north-east, with support from the North East and North Cumbria Academic Health Science Network (AHSN) Patient Safety Collaborative. It will then be made available for wider implementation.
Education and training
In this component of the programme we developed a patient safety training programme for junior doctors based on patients who had experienced PSIs recounting their own stories. Patient safety training often provides learners with a health professional’s perspective rather than a patient’s. We hypothesised that personal narratives of health-related harm would allow patients to share their stories with junior doctors and influence clinical behaviour by rousing emotions and improving attitudes to safety. This approach was compared with a more traditional method of using expert faculty to teach patient safety to FY1 doctors in an open, multicentre, two-arm, parallel-design RCT. The APSQ and the PANAS were used to measure the impact of the intervention on the junior doctors. Although the study showed that delivering patient safety training based on patient narratives is feasible, we were unable to demonstrate effectiveness of the intervention in changing general attitudes to safety compared with the control. This may be because of the inherent difficulty in determining valid outcome measures to study the effectiveness of educational interventions. We did, however, show a difference in the short-term emotional response of the trainees to the patient stories. Although patient narratives may impact on emotional engagement and learning about communication, we remain uncertain whether or not this will translate into improved behaviours in the clinical context or indeed if there are any negative effects.
Patient and public involvement
Patient and public involvement was a central tenet of this research programme. Significant effort and commitment was made at the outset to establish advisory panels of lay people to promote genuine codesign and coproduction of the research projects. The input from lay members developed considerably over the lifetime of the programme and much valuable experience was gained about how best to optimise the effectiveness of PPI. Many of these insights are captured in our evaluation of the programme’s PPI outlined in Chapter 11, the main finding being that PPI requirements in a research project are likely to be context specific and evolve over the lifespan of the research. On the basis of our experience in this programme opportunities for regular feedback and cycles of learning for all involved are crucial and we would strongly advocate a coproduction model between researchers and lay contributors for developing and maintaini