Functionality and feedback: a realist synthesis of the collation, interpretation and utilisation of patient-reported outcome measures data to improve patient care

Inclusion criteria

• Studies about provider or commissioner views of, responses to, use of, or interpretation of PROMs data, national clinical audits, patient experience data, clinical outcomes (as an indicator of treatment effectiveness and, thus, performance) and mortality data (as an indicator of performance).

• Reports on the process of implementation of local PROMs/patient experience data collection for use as an indicator of service quality, reporting not just the results of the data collection but how data collection was implemented and/or how it was used.

• Reports on the process of implementation and use of local audit to improve care – how it was implemented and how people responded – not just reporting the results of the audit.

Exclusion criteria

• Articles on the development or validation of PROMs data, patient experience data, national clinical audits, mortality data, clinical outcomes.

• Articles about patient involvement in patient experience data.

• Articles reporting just the findings/analysis of audit data.

• Articles evaluating the impact of other QI programmes that did not involve some sort of feedback or public reporting of data.

At this stage we included 124 papers. After rereading the titles and abstracts, we developed a more restrictive set of inclusion and exclusion criteria that were focused on our evolving theories, as follows.

Inclusion criteria

Studies about how clinicians or managers have used or responded to or about clinicians’ views of:

• national or local patient experience data collection and feedback

• mortality report cards or anything described as ‘performance data’

• hospital report cards

• aggregate clinical outcome indicators or process data.

Studies contributing to testing theories about:

• the mechanisms through which feedback is intended to work (intrinsic desire to improve, peer comparison, protecting market share, protecting professional reputation)

• the contextual configurations that might influence how performance feedback works (financial incentives, credibility, ‘actionability’ of these data).

Exclusion criteria

Studies examining:

• views/experiences or implementation of general QI activities

• implementation of, or response to, local clinical audit or guidelines

• results/findings of audits or patient experience data.

Following the application of these criteria, we included 28 studies. We also checked the references of an existing systematic review of public reporting of performance data,27 which identified a further 18 studies, and also checked the reference lists of five key papers18,75,97–99 on which we had conducted a preliminary synthesis and identified 26 studies. We conducted an additional search in MEDLINE and EMBASE for feedback of patient experience data (see Appendix 1) and selected three studies from the 194 identified. This gave us a total of 78 references. We then checked for duplicates and rescreened the papers using the more focused set of inclusion and exclusion criteria listed above, and included 51 papers. These papers focused on providers’ views and responses to performance data and indicators and patient experience data. Table 1 provides a summary of the different sources of papers.

We read the 51 papers and began our synthesis, described in the next section (see Data extraction, quality assessment and synthesis). As the synthesis progressed, we further focused our synthesis on a smaller number of main theories but, at the same time, a number of ‘subtheories’ within this focused selection of ‘main’ theories were identified. This required us to revisit our original search results, further examine documents in JG’s personal library and carry out additional citation tracking of key studies to further test these subtheories.

This identified an additional 30 papers that were included in the final synthesis. At the same time, we found that some of the papers we had originally included were no longer relevant to the main theories in the synthesis. Of the original 51 papers identified from the initial searches, 28 made it into the final synthesis; 23 papers were left out of the final synthesis because they were not relevant to the final theory testing phase of the synthesis or did not progress the theory testing further. Thus, in total, 58 papers were cited in the final synthesis of the feedback of aggregated PROMs and performance data, reported in Chapters 4 and 5. A flow chart of the search process is show in Figure 3.

FIGURE 3.

Flow diagram of the evidence search process for the feedback of aggregate PROMs data.

Summary of unintended consequences

Performance data may give rise to a number of unintended consequences. These include gaming data to give the appearance of changes in performance, without real changes in performance; effort substitution, where efforts are directed to improving what is measured at the expense of what is not; threshold effects, where poor performers feel pressure to improve, but high performers may allow performance to deteriorate towards the mean; and avoidance of treating sicker patients in order to improve performance data. Whether or not and how these effects occur are contested; it will be for our evidence review to ascertain when and why they occur.

Data accuracy

This theory argues that the success of public reporting depends, in the first instance, on the accuracy of the underlying data used to produce performance reports or indicators. In the USA, a particular bone of contention is the formulation of indicators based on routinely collected administrative data gathered by insurance companies at patient discharge to bill payers, which are deemed by many clinicians to be inaccurate, versus the use of data extracted from patient notes by hospital representatives, requiring additional resources to obtain. Similarly, in the UK, many have questioned the accuracy of Hospital Episode Statistics data, either to produce indicators themselves, or in the case of PROMs data, to link PROMs questionnaires to patient episodes. For example, Naylor146 comments:

The struggle with valid description of processes and outcomes of clinical care will continue as long as health services researchers and private profilers are forced to dredge administrative databases. Such databases have well recognized limitations in characterizing patients, clinicians and institutions.

Naylor146

The perceived difficulty is that these databases were originally developed for a different purpose, that of enabling hospitals to bill insurers or commissioners for care provided, rather than for QI purposes.

Case-mix adjustment

This theory addresses the question of whether or not case mix adjustment can ensure that performance data provide a valid indicator of the quality of care provided by an organisation. It is assumed that performance data reflect the quality of care provided by a hospital and will enable differentiation between ‘poor-performing’ and ‘well-performing’ hospitals. However, patient outcomes are determined not only by the quality of care provided by the hospital, but also by the patient’s baseline health, comorbid conditions and other patient characteristics, which are beyond the control of the hospital. Therefore, to represent the outcome that can be attributed to the care provided by the hospital and, thus, enable meaningful comparisons between hospitals, outcome data are adjusted for variations in patient characteristics that may influence the patient’s likely benefit from the care they receive, but are not under the control of the hospital. This is known as case-mix or risk adjustment. Black66 highlights the need for ‘sufficiently robust adjustment for differences in case-mix to achieve credibility’ for PROMs data. Similarly, Marshall and Brook147 argue that the risk of providers refusing to treat disadvantaged groups can be attenuated by ‘careful adjustment of risk and case-mix’.

The precise model that should be included in risk adjustment of performance data has, however, been the subject of much debate. Even using the same variables and data, different models can produce different findings. Iezzoni et al. 148 compared 10 different, commonly used risk adjustment models for mortality data on 100 hospitals. They found only fair to good agreement between different methods of case-mix adjustment and concluded:

For an individual hospital, perceptions of mortality performance could vary according to different severity adjustment methods . . . [which raises] . . . important questions for report card efforts to judge hospital performance by use of severity-adjusted death rates . . . it is important to weigh what actions may be reasonably founded on this information.

Iezzoni et al. 148

Similarly, Grant et al. 149 found that the choice of risk assessment model in the analysis of cardiac surgeon specific mortality data influenced which surgeons would be identified as outliers at 95% control limit.

Others have also warned against the use of outcomes data as an indicator of hospital performance because case-mix adjustment is always imperfect and can never guarantee an unbiased comparison between providers, as there will always be unmeasured prognostic factors that may influence outcomes. For example, Lilford et al. 150 argue:

Therefore, even if an agreed risk-adjustment method could be derived, outcomes could still vary systematically between providers because we can never be sure that risk adjustment is not hampered by unmeasured prognostic factors . . . Making judgements about the quality of care on the basis of risk adjusted comparisons cannot guarantee that like is being compared with like.

Reprinted from The Lancet, Vol. 363, Lilford R, Mohammed MA, Spiegelhalter D, Thomson R, Use and misuse of process and outcome data in managing performance of acute medical care: avoiding institutional stigma, pp. 1147–54, © 2004, with permission from Elsevier150

Others acknowledge that the ‘perfect’ method of case adjustment is unachievable but argue that this should not prevent the publication or feedback of performance data. Instead, they advocate that the credibility of data can be enhanced if those responsible for publishing the data are engaging in an ongoing process of improving the data’s quality. As Marshall and Brooke147 argue:

It is important for those who publish data to show a commitment to investing in the process and progressively improving the quality of the data and the validity of comparisons arising from the data. However, it makes little sense to ‘wait for better data’ – data will always be imperfect and, as one commentator stated, it is important not to let ‘perfect be the enemy of the good’.

Marshall MN and Brook RH. Public reporting of comparative information about quality of healthcare. Med J Aust 2002;176(5):205–206. © Copyright 2002 The Medical Journal of Australia – reproduced with permission147

Still others have argued that the risk of misrepresentation as a result of imperfect case-mix adjustment can also be reduced through the use of multiple different forms of performance data. For example, Naylor146 contends that:

Despite the best risk-adjustment algorithms, clinical outcomes are often confounded by factors other than [the] technical quality of care or clinical decision making. Well balanced performance profiles should include a sensitive array of process-of-care measures, carefully chosen clinical outcomes, and patient perception and satisfaction surveys.

Naylor146

He also reasons that involving all stakeholders in the development of performance indicators can stave off concerns about their credibility. He argues that the development of performance indicators need to be:

[I]nformed by meticulous analytical methods that do everything . . . to level the playing field, eliminating unfair comparisons . . . by working closely with clinicians, hospitals, and other healthcare providers from the outset, quality analysts can forestall criticism of the credibility of their reports and catalyse positive change rather than professional defensiveness.

Naylor146

Although not explicitly stated by Naylor, we can hypothesise that involving stakeholders may increase the credibility of the data through ensuring that the case-mix algorithms are appropriate, through increased ownership of the data, or through both.

When to measure

This theory argues that the choice of time point at which to measure outcomes following a procedure may either fail to capture the full benefit of the procedure on patient outcomes or become confounded by factors not related to the procedure. Black66 acknowledges that ‘judging the best time to assess outcome after an intervention so as to be able to attribute it to that intervention is also contentious’. For knee and hip replacement, postoperative PROMs data are collected 6 months after the operation, and for groin hernia repair and varicose vein surgery, they are collected 3 months after the operation. As Browne et al. 151 observe:

The primary purpose of the English PROMs programme is the detection of deviant performance by surgical providers. Implicit in the methods is the assumption that the performance of these providers can be fairly judged at six months after surgery as the clinically important benefits of surgery have accrued. If this is not the case it is possible that some providers are being unfairly assessed.

Reproduced from Browne et al. 151 under the terms of the Creative Commons Attribution License (http://creative commons.org/licenses/by/2.0)

Browne et al. 151 note that the cut-off point of 6 months to measure the impact of hip and knee replacement was based on ‘clinical consensus’ and on the need for the output to be timely to enhance the likelihood that it will stimulate health-care providers to review and improve the quality of their care. By choosing 6 months, it is not assumed that a patient has reached the maximum benefit of the operation; however, some have expressed concern that the use of 6 months as the cut-off point does not capture the longer-term benefit of the interventions. 152

Maynard and Bloor152 argue that capturing outcomes too long after a procedure makes it difficult to disentangle the relative contribution of the procedure from other aspects of care that may contribute to the patient’s functioning:

Recording ‘success’ after three or even six months [after a procedure] may be incomplete. To go beyond this time period risks factors other than the procedure being evaluated affecting outcome.

Maynard and Bloor152

Their example points to the issue regarding what aspects of care PROMs data are intended to reflect: the skill of the individual surgeon in performing the surgery, the teamwork and co-ordination of the operating team working with this surgeon, the quality of care provided immediately postoperatively (e.g. pain relief) or the quality of community care received by patients in the community in the months following surgery (e.g. from physiotherapists) – or all of these.

Attrition bias

When using PROMs as the outcome measure, it is necessary to have a baseline assessment (pre treatment), as this is the most powerful predictor of the post-treatment PROM, as well as a post-treatment assessment. This adds to the complexity, burden and cost of employing PROMs to assess the quality of care. In addition, the need for two data collection points means that there will be some attrition through patients not responding to the post-treatment PROMs, although, following major interventions such as hip replacements, the response rates are very high. Work by Hutchings et al. 67 revealed that older, sicker patients were less likely to complete PROMs, and they argue that if non-response is associated with outcome, then rates of non-response to postoperative questionnaires would need to be taken into account when these measures are being used to compare the performance of providers or to evaluate surgical procedures. Furthermore, current data on the HSCIC website indicate considerable variation in participation rates between providers, suggesting wide differences in whether or not and how providers administer questionnaires and manage the process of completion. Evidence suggests that providers with lower participation rates are more likely to be erroneously classified as an outlier. 153

Level of analysis

There is also considerable debate about the level of analysis at which PROMs and other performance data should be published. As Black66 notes, currently PROMs data are provided at the level of the hospital, with the implication that it is variation in the care between hospitals that gives rise to variation in outcome. Some have argued that this does not enable patients to distinguish between the quality of individual surgeons, and that surgeon-level data are required. However, others have questioned the utility of surgeon-level data, as the numbers may be too small to permit meaningful comparisons between surgeons, and because surgeons work in teams and therefore outcomes reflect the work of the team and not just of the surgeon. As Sir Bruce Keogh, medical director of the NHS, argued in an interview in the BMJ in 2008:125

Professor Keogh says he does not believe individual level consultant data should be published yet and probably not ‘for some time down the line’ not until people are convinced that the data are accurate, and even then only if the individual data are meaningful. Doctors increasingly work in teams, he notes, so individual data may not always tell the story.

Reproduced from NHS goes to the PROMS, Timmins N, vol. 336, p. 1465, © 2008 with permission from BMJ Publishing Group Ltd125

Trusted source

This theory highlights that credibility in the eyes of recipients is influenced not just by the data themselves, but also by the perceived ‘trustworthiness’ of the body supporting or initiating data collection and the perceived drivers of this process. For example, Marshall and McLoughlin77 highlight a number of conditions under which the use of performance data can be optimised, including the idea that the data are perceived as coming from a ‘trusted source’:

It is important that users perceive the information as coming from a trusted source. Information providers that might be regarded as having ulterior motives, such as government or for-profit organisations, may not be perceived to be independent. Partnerships between the health service, and respected professional bodies and academic institutions may be seen as more trustworthy.

Reproduced from How do patients use information on health providers?, Marshall M, McLoughlin V, vol. 341, p. 5272, © 2010 with permission from BMJ Publishing Group Ltd77

It is argued here that the perceived motives of the body promoting the collection and use of performance data can influence the degree to which the data are trusted by recipients. Wolpert16 also shares this view and highlights the tensions that have emerged between the collection and feedback of PROMs data to support clinical decision-making for individual patients and the use of PROMs data for performance management in the context of mental health services. She argues that the perceived primary driver for the introduction of PROMs data collection in clinical practice influences the degree of clinical engagement with the process:

It may be crucial that in introducing PROMs into clinical practice frontline clinicians are introduced to the tools through a prism of collaborative working and shared decision making rather than as tools primarily used for audit or performance review . . . an underlying ethos of collaborative working and shared decision making, and a focus on using PROMs as part of clinical conversations, promotes greater clinician engagement and willingness to trial the use of PROMs.

Reproduced under the terms of the Creative Commons Attribution License, which permits any use, distribution, and reproduction in any medium, provided the original author(s) and the source are credited16

Similarly, Marshall and Brooke147 argue that clinical engagement in selecting the indicators may also increase engagement with the collection and dissemination of performance data:

. . . we know that forcing new initiatives on reluctant professionals is not the most effective way of changing attitudes, and the introduction of report cards is more likely to be successful if doctors are encouraged to take a lead, particularly in selecting the performance measures.

Marshall and Brooke147

The World Health Organization154 also points out that hospitals often require incentives to participate in public reporting programmes. It observes that the perceived rationale and possible benefits and risks of collecting performance data can influence the extent to which other incentives are required to promote participation in such schemes. It argues that if public reporting programmes are perceived by staff to have intrinsic value to them (e.g. through enabling team building, supporting clinical or professional development or assisting with risk management), then hospitals have less need for financial or market incentives to participate. However, they also note that staff are unlikely to be motivated to participate if performance data may lead to public blame or litigation.

Thus, these theories suggest that it is not just the validity of the data itself that can influence how providers respond, but also the perceived rationale for the data’s collection and whether or not providers feel that they provide information relevant to their own work.

Summary of data credibility

The theories reviewed here revolve around that idea that, unless providers perceive PROMs or other performance data to be credible and accurately reflecting the quality of patient care, they will not trust the data and, consequently, will not take steps to improve patient care. These arguments suggest that providers’ trust in the data rests not just on the technical aspects of validity – that is, case-mix adjustment, the level of analysis and timing of assessment (although these are important) – but also on the perceived motivations behind data collection. These theories also highlight potential tensions between the use of the data for performance management and the use of the data to inform the clinical care of patients. They suggest that clinicians are more likely to engage with performance data if data are provided that can also be used to inform the clinical care of patients.

Timeliness

One theory as to why performance data may not lead to improvements in patient care is because of the time it takes to collect data, adjust for case mix and then disseminate. If prolonged, the data may reflect care provided in the past, rather than that provided currently. This, in turn, makes it difficult to identify and then address current areas of poor care. For example, in a paper that attempted to understand why ‘public reports have not had more impact on system performance’, Guerriere155 speculated that:

The impact of healthcare performance reporting may be diminished by the age and quality of the reports. Getting old news about problems you had a year ago does little to motivate change today.

Guerriere155

Similarly, in a paper debating the potential outcomes of the Australian government’s plans to introduce performance data into its health-care system, Braithwaite and Mannion139 warned that:

Performance measurement systems are necessarily back-ward looking, as it takes time to assemble and disseminate data. By the time a problem is spotted, it may be too late to do anything about it.

Braithwaite and Mannion139

A suggested solution is the feedback of ‘real-time’ performance data: that is, feedback in which the delay between data collection and feedback is minimised. Such an approach has been introduced in England, with the real-time collection and feedback of one form of patient experience data, the FFT, implemented in acute hospitals in 2013. The FFT is a single question that asks patients to think about their recent experience of the service and rate how likely they would be to recommend that service to friends and family if they needed similar care or treatment (with response options of extremely likely, likely, neither likely nor unlikely, unlikely, extremely unlikely and don’t know). Recent NHS England guidance156 on the FFT highlights that its timeliness means that it has some advantages over traditional survey methods of collecting feedback:

Compared to traditional survey methods, where there is often a considerable time-lag between the collection of feedback and the survey results, the FFT is a timely feedback tool. This can help providers to understand their areas of strength and weakness – and drive improvements in patient care – very quickly.

p. 18. Contains public sector information licensed under the Open Government Licence v3.0156

However, NHS England acknowledged that real-time feedback also brings disadvantages, such as the loss of the ability to benchmark performance with other providers:

The FFT does not provide results that can be used to directly compare providers because of the flexibility of the data collection methods and the variation in local populations. This means it is not possible to compare like with like. There are other robust mechanisms for that, such as national patient surveys and outcome measures. The FFT can help mark progress over time for organisations and still provides patients with useful data to inform choice.

p. 18. Contains public sector information licensed under the Open Government Licence v3.0156

Here we see the tensions that arise between the timeliness of the data and their credibility and usefulness. The simplicity of the indicator (a single question) and the quick collection and feedback of these data, which ensure that these data are timely, also mean that there is not sufficient time to adjust these data for case mix, and, thus, ensure that like is compared with like, or to present these data in a format that allows comparison with other providers.

Presentation and interpretation of the data

It has also been speculated that the ways in which performance data are presented influences whether or not their intended recipients are able to interpret and make sense of the data and, in turn, take action in response to these data. The most appropriate way of presenting PROMs and other performance data to facilitate accurate interpretation has been the subject of much debate. For example, in 2009, Black and Jenkinson157 argued against the use of rank ordering or ‘league tables’ of providers and in favour of the use of funnel plots:

Ranking of providers by score is also problematic because the performance of most providers does not differ much. Such rank orders may have little meaning and can give an artificial distinction between those at the top and those at the bottom. It is better to present data in funnel plots . . . as it avoids ranking and instead focuses attention on any outliers that require local in-depth investigation to determine the reasons for their poor performance.

Reproduced from Measuring patients’ experiences and outcomes, Black N, Jenkinson C, vol. 339, pp. 202–5, © 2009 with permission from BMJ Publishing Group Ltd157

The ways in which PROMs data have been fed back to providers, managers and commissioners has been criticised for being difficult to interpret. For example, a BMJ review158 of the PROMs conference held in 2013 cites Professor Nick Black expressing a wish that the group assembled to advise the Department of Health on the PROMs programme would discuss how the presentation of PROMs data could be improved:

Black said that after a hiatus caused by the government’s reorganisation of the NHS in England, the PROMs programme was now moving ahead, and the first meeting of a PROMs Advisory Committee was planned by NHS England to be held soon. Among other things he hoped that the committee, of which he will be a member, would discuss the presentation of PROMs data by the NHS Information Centre. ‘All attention has been to input, none to output’ he said. ‘The spreadsheets published by the Information Centre are a complete turn-off. At least 20% of the budget should be spent on outputs.’

Reproduced from Patients’ rating of treatment tells you more about patients than hospitals, research concludes, Hawkes, N, vol. 347, pp. f6916, © 2013 with permission from BMJ Publishing Group Ltd158

Furthermore, performance data have a wide range of intended audiences: clinicians, commissioners, GPs, NHS managers and also patients. It is unlikely that the same method of presenting performance data will meet the needs of each of these different audiences. One proposed solution is to make use of web technologies that allow different audiences to select information they want and how it is presented; as Marshall and McLoughlin77 note:

New web technologies, in particular could be used to enable users to define what information they want and how they want it presented. This would help to meet the needs of disparate audiences and recognises the heterogeneity among potential users.

Marshall and McLoughlin77

Another purported solution to this problem is to simplify the presentation of complex data by combining data into a single score or by using simple ‘star ratings’. However, this approach creates another set of difficulties. Naylor146 argues that simple ratings do not convey uncertainty in the indicator (e.g. as expressed by confidence intervals) and may lead to exactly the problem (noted above by Black and Jenkinson) of creating an artificial or inaccurate distinction between hospitals:

Communication of complex performance data is difficult. Healthgrades.com uses a ‘star system’ to improve public accessibility, and many other agencies use similar qualitative markers to identify higher and lower performing providers. But simplified rating systems preclude the acknowledgement of uncertainty in rankings and may misclassify ‘borderline’ hospitals.

Naylor146

However, others also recognise that the appropriate presentation of data is only one element in the process of using data to bring about change. Guerriere155 argues that organisations require skills in interpreting performance data, but also the ability to translate that information into operational improvements:

A key issue is the lack of understanding throughout the system of what to do with performance information . . . Without the skills needed to interpret available performance data in order to effect operational improvements, performance reports are destined to have a very limited impact.

Guerriere155

This leads us to our next set of theories, which consider whether or not performance information has sufficient ‘diagnostic’ information to direct organisational change.

Performance data as judgement or as a catalyst for investigation

This set of theories address the question of the extent to which performance data can, or should, offer a clear distinction between ‘well-performing’ and ‘poorly performing’ hospitals and, in doing so, provide the necessary information to inform subsequent organisational change. The argument is that performance data, expressed as a single, overall indicator, offer an incomplete picture of hospital performance. Therefore, performance indicators alone do not offer an absolute verdict on hospital performance, but instead prompt inquiry and questioning. Carter et al. 159 summarise this idea in their depiction of performance data acting as dials, tin openers or alarm bells. They argue that, in an ideal world, performance indicators act like a dial and offer a precise measure of inputs, outputs and outcomes premised on a clear understanding of what good and bad performance entails. However, they recognise that for most public sector organisations there are very few such precise measures, nor is there always absolute agreement as to whether such measures represent ‘good’ or ‘bad’ performance. Instead, they suggest that, in most instances, performance indicators act like tin openers; rather than giving answers, they open up a can of worms and prompt the examination and review of performance. Performance indicators alone thus provide only a partial picture of provider performance.

Similarly, Marshall and Brook147 argue that:

The utility of comparative data comes less from making absolute judgements about performance than from the discussion arising from using the data to benchmark performance. There is therefore a strong educational component to the effective use of comparative data and resources are required to facilitate this process.

Marshall and Brook147

Thus, the theory here is that performance data rarely provide a definitive ‘answer’ regarding the quality of care provided; rather, what leads to change is the discussion and investigation of the underlying cause of the level of performance indicated from the data. Furthermore, such investigations and discussion require resources in order to happen.

The issue of interpretation is a particular challenge for outcomes data. One of the key arguments for the use of PROMs and other outcome data is that they represent the ‘ultimate goal’ of health care and, as such, health care’s success ought to be judged on these data rather than on its activity. 5 To improve outcomes, it is necessary to change the processes that give rise to those outcomes; however, outcomes data alone do not always provide an indication about the cause of the poor outcomes or what needs to be improved. Thus, there is a tension between the overall objective of performance monitoring and the means to achieve it. Several commentators have criticised the use of outcome measures as performance indicators because they do not distinguish between ‘good’ and ‘poor’ care, nor do they provide any pointers for where questioning or investigations might begin to discover the cause of ‘poor’ performance. For example, Lilford et al. 70 argue:

The problem that outcome data are poor barometers of clinical quality is viciously confounded by both their inability to discriminate between good and poor performers and the lack of information they convey about how improvements should be made.

Lilford et al. 70

This criticism has also been levelled at PROMs data; for example, an article in the Guardian newspaper about the first release of the 2009–10 PROMs data160 notes:

PROMs do have their limitations. It is hard, for example, to prove what exactly one hospital is doing different from another that improves outcomes.

Others argue that process measures are better able to signpost the causes of poor care; for example, Mant161 observes:

A second advantage of process measures is that they are easy to interpret. A process measure such as use of aspirin in acute myocardial infarction is a direct measure of quality, whereas hospital-specific mortality from myocardial infarction is only an indirect measure . . . if differences in outcome are observed then alternative explanations need to be considered before one can conclude that the difference reflects true variations in the quality of care. Conversely, a process measure is straightforward to interpret: the more people without contra-indications who receive a proven therapy, the better. A consequence of this is that the necessary remedial action is clearer (use the therapy more often). However, if one does conclude that a higher mortality rate is due to poor quality care, it is not immediately obvious what action needs to be taken, unless perhaps audits of the process of care have been undertaken in parallel.

Reproduced from Mant J, Process vs outcome indicators in the assessment of quality of care. International Journal for Quality in Health Care, 2001,13(6):p. 475–80, by permission of Oxford University Press161

Here we see a potential way in which outcome data could be made more useful: to conduct an audit of process alongside the collection of outcome indicators, in order to understand why the variation in outcomes has occurred. Others have also noted that the usefulness of measuring process data depends on the strength of the evidence underpinning the process-outcome link. They also note that complex care may not be amenable to single process measures. For example, Long and Fairfield162 argue that, where there is high-quality evidence that an intervention produces the anticipated outcomes, monitoring process alongside an audit of any adverse events may be sufficient. However, in circumstances where the evidence is mixed or of poor quality, then scrutinising outcomes, together with a ‘close description and knowledge of the nature of the process of treatment’, is important to establish that the desired outcomes of care have in fact been achieved.

These theories encapsulate the idea that performance data alone do not provide an absolute judgement on the quality of care provided. Instead, performance data may raise questions about the quality of care, which then need to be explored. Furthermore, this process of enquiry is also a process of education, which, in itself, is important to stimulate QI. They also contest different strengths and weaknesses for process and outcome data in the extent to which they may give pointers to this enquiry process. This rests on the strength of the causal link between processes and outcomes.

Focus on outliers or shifting the mean to improve care

Finally, a further debate centres on whether the purpose of performance data feedback is to stimulate improvements by providers classed as ‘poor performing outliers’ or whether the aim is to prompt improvements in all providers, irrespective of the level of their performance. This clearly has implications for which providers are expected to respond to performance data and how they are expected to respond. Much of the guidance on the national PROMs programme and the presentation of the data using funnel plots is geared towards enabling the identification of outliers (both positive and negative). Although the guidance issued by the Department of Health109 for identifying potential outliers attempts to give equal emphasis on identifying both positive (i.e. high performing) and negative (i.e. poor performing) outliers, the suggestions for how potential outliers are expected to respond, appear, implicitly at least, to have a subtle focus on stimulating change in poor performing organisations. For example, the summary of key points on page 6 advises that:

The publication of a list of potential outliers will be published as part of or alongside the quarterly PROMs publication. It would be the responsibility of the provider to take to action and explore and improve their performance. We recommend that:

1. The IC’s [HSCIC’s] participation and response rates table be used by providers to assess the quality of their data. When rates are low, providers would be expected to take action to improve them,

2. Providers consider if there are other factors which may explain their presented results, other than variation in performance,

3. Where possible, comparative information be provided to help organisations identified as potential outliers for example, how they compare with other providers on pre-operative scores or on patient characteristics.

Reproduced from Department of Health under the Open Government Licence v3.0109

However, other commentators have argued that larger health gains can be achieved if the focus was on supporting improvement in all organisations, not just poor performers. For example, Lilford et al. 150 contend that:

. . . focus on outliers is only one aspect of improvement. Improving clinical and managerial processes in the remaining organisations can achieve much more health gain by shifting the mean . . . the actions required to pull back an outlier as opposed to shifting the mean . . . are fundamentally different. The first needs investigation of a probable special cause, whereas the second action needs focus on improvement by means of a wide array of process-improving tools such as the plan-do-study-act cycle. For these reasons, hospitals should monitor their own data – institutional processes, throughput, clinical processes, outcome (especially trends over time), and above all clinical process.

Reprinted from The Lancet, vol. 363, Lilford R, Mohammed MA, Speigelhalter D, Thomson R, Use and misuse of process and outcome data in managing performance of acute medical care: avoiding institutional stigma, pp. 1147–54, © 2004 with permission from Elsevier150

Thus, Lilford et al. 150 point out that if the focus of performance data feedback is to improve the performance of all organisations and not just of poor performers, then all organisations need to use a range of QI techniques, rather than expecting poorly performing organisation to focus on the search for a specific cause of their poor performance.

We noted one further reason for interpreting any data that compare providers with caution discussed in the literature: that of ‘regression to the mean’. There is a statistical likelihood that a provider that is an outlier on one set of data will, in the next tranche of data (e.g. the following year) no longer be an outlier. It will regress to the mean for statistical reasons that may not reflect any true change in performance. 163 Two potential solutions to avoid this pitfall were voiced in the literature. The first was the idea that it is prudent to monitor providers over several rounds of data collection to detect those that are persistent outliers so as to avoid inappropriately reacting to one set of data. The second was that the actual difference from that expected, not just whether or not a provider is a statistically significant outlier, should also be considered.

Indicator specification

Linked to the issues raised in the previous section, there is debate about whether performance data should be expressed as a continuous variable, or whether targets or thresholds are needed to both interpret the data and incentivise change. 137 Shaw et al. 137 argue that the use of thresholds or statistical approaches to characterise performance as ‘outlying’ or ‘not outlying’ can lead to arbitrary distinctions between what constitutes ‘good’ and ‘poor’ care. They observe that, within a performance management framework, it is nonsensical to treat performance on one side of this cut-off point as having no cause for concern while considering performance that is not statistically different, but on the other side of the threshold, as problematic.

A further key tension is whether these thresholds should be defined in relation to evidence-based clinical criteria or clinically important changes/cut-off points or in relation to statistical criteria, such as standard deviations above or below the mean population performance. Bird et al. 164 make the following comment about national targets based on statistical significance:

. . . it is unsmart to specify a national target in terms of statistical significance rather than operational terms because, with a sufficiently large denominator, such as from complete enumeration of 600,000 school children’s test results, very small differences of no practical importance achieve statistical significance.

Bird et al. 164

In other words, with a large number of events, small differences in performance between units or organisations can be statistically significant without being practically or clinically meaningful.

The community involved in the development of PROMs has recognised the problem that statistically significant differences in PROMs scores following an intervention within a RCT may not reflect clinically important changes. To address this, Jaeschke et al. 165 introduced the concept of the ‘minimal clinically important difference’ and developed a methodology to identify it for each PROM. The minimal clinically important difference is defined as:

The smallest difference in score in the domain of interest which patients perceive as beneficial and which would mandate, in the absence of troublesome side effects and excessive cost, a change in the patient’s management. 165

This represents a different approach to thresholds based on statistical parameters and instead focuses on changes in PROM scores that would lead a clinician to change the way they manage a patient.

Summary of actionability

The key ideas here are that unless PROMs and other performance data can be understood by recipients, and unless they provide information about the potential cause of any poor performance or offer pointers to how care can be improved, then providers are unlikely to be able to make use of the data to improve patient care. The ‘actionability’ of data depends on the timeliness of feedback, the way data are presented and the capacity of the organisations and individuals to make sense of and respond to the data. There is some debate about whether or not process data are more actionable than outcome data because they are better able to pinpoint and diagnose where the deficiencies in care lie. Questions are also raised about whether we expect performance data to act like a ‘dial’ and clearly distinguish between poor and good performers or whether we expect performance data to act like a ‘tin opener’ and prompt further questioning and investigation and, in doing so, lead to improvements being made. Finally, there is debate about whether the purpose of performance data are to improve the quality of care of all providers, irrespective of their performance, or whether the focus should be on improving the performance of low performers in particular. Linked to this are different ideas about whether such data should be presented as a continuous variable or in relation to a threshold and whether such thresholds should be based on statistical or clinical parameters.

Varagunam et al.173

Varagunam et al. 173 evaluated the impact of the national PROMs programme in England, and tested whether or not the programme delivered on its policy aims of changing the selection of patients for surgery, reducing the variation among providers in the degree of health gain achieved by patients and improving postoperative outcomes for patients. Using the national PROMs data set, the authors analysed changes in mean preoperative scores and mean adjusted postoperative scores using standardised effect sizes on both disease-specific and generic PROMs for each procedure between April 2009 and March 2012. They found that preoperative severity increased slightly for varicose vein procedures and, to a lesser extent, hip surgery, but not for knee or groin hernia surgery, suggesting that the programme did not have a significant impact on the ways in which patients were selected for surgery. They found little variation between providers and that this variation did not change over time. There was only a modest consistency in providers labelled as outliers over time for hip and knee replacement and much less consistency for the other two procedures. The authors attribute this in part to regression to the mean. They also found slight improvements in health gain for hip and knee replacements but, while these changes were statistically significant, they could not be considered clinically significant. They found no changes over time in health gain for groin hernia repair and a slight worsening in the degree of health gain for varicose vein surgery.

Overall, the authors conclude that the national PROMs programme had ‘little impact’ on preoperative severity and outcomes for patients and suggest that more time is needed to fully evaluate the impact of this programme. In their discussion, the authors attribute this finding to two factors. First, the delay between data collection and feedback of PROMs data meant that providers only received the first release of the finalised data on outcomes in April 2011 and, as such, these data were unlikely to have had any impact on provider practices, and thus outcomes, until the fourth year of the programme. Second, until 2011, the data were fed back as a large spreadsheet, which the authors argue was difficult for clinicians, managers and patients to understand. The use of funnel plots, which enabled providers to compare their own performance with that of others, only commenced in 2011. In terms of the theories under test, both the lack of timeliness of the data and the difficulty in interpreting them may have limited the ability of providers both to compare themselves with others and to identify any gaps between their own performance and that of others. Subsequently, the authors have also suggested that the lack of impact may be due to a ceiling effect; there is little variation between providers, with the vast majority delivering good levels of health gain in clinical terms, and, as such, there is little room for improvement in performance for these procedures.

Boyce and Browne174

This study aimed to examine whether or not providing surgeons with peer-benchmarked feedback about the patient-reported outcomes for their patients improved the outcomes of their future patients. The study was conducted in Ireland, where there is no national programme to feed back hospital-level PROMs data, as exists in England. Furthermore, the feedback was provided at the level of the surgeon, rather than of the hospital, and was privately rather than publicly reported. The authors used a cluster RCT design, with surgeon as the cluster unit.

The trial focused on high-volume orthopaedic surgeons, defined as surgeons who were responsible for at least 100 primary hip operations during the previous year. A total of 21 surgeons were recruited to the trial: 10 were randomised to the control group and 11 were randomised to the intervention group. Intervention group surgeons received a case-mix-adjusted peer-benchmarked comparison of their patients’ mean change in the OHS in the form of a ‘caterpillar chart’. This plotted the patients’ mean change in the OHS and confidence intervals for each of the anonymised 21 surgeons, ordered from highest to lowest, with the recipient’s own performance clearly marked. Intervention surgeons also received feedback on the proportion of their patients who reported an overall improvement in their hip problem and the proportion who reported having at least one of four postoperative complications. Data for this feedback were collected during a period of 11 months prior to the provision of the feedback. The surgeons also received a 9-minute educational video about the interpretation of PROMs data. The primary outcome for the trial was the difference in the mean postoperative OHS between patients in the intervention and control arms who were operated on during a period of 11 months following the provision of feedback. The secondary outcomes included the Hip Osteoarthritis Outcome Score and the EQ-5D.

Oxford Hip Score data were analysed using a linear mixed-effects regression model, adjusting for differences in patient characteristics. The authors found no statistically significant differences in mean postoperative OHS, Hip Osteoarthritis Outcome Score or EQ-5D. They also found no statistically significant differences in the proportion of patients who experienced a complication between the intervention and control arms. The authors conclude that ‘embedding PROMs within quality assurance and improvement frameworks is unlikely to lead to patient benefit’. 174 The authors attribute the lack of impact to a number of issues, including surgeons’ scepticism about the value of the data, the burden of data collection and the lack of sensitivity and specificity of PROMs as indicators of clinical performance. They also note in their conclusions that ‘Performance monitoring can provide information about whether healthcare professionals perform better or worse than their peers, but does not explain why performance differs’. 174 This echoes the theory identified in Chapter 3, that outcome measures do not provide information on which aspects of process are attributable to poor performance. The authors go on to argue that in order to identify the source of any problems, clinicians will have to undertake additional audit activities, which ‘assumes that professionals have the time, resources, knowledge, expertise, flexibility and willingness to implement such activities’. 174

Totten et al.27

This was a large systematic review of over 198 studies to examine whether or not the public reporting of performance resulted in improvement to quality of health care or health-care delivery structures and processes. These studies largely focused on evaluating reporting systems implemented in the USA and Canada, both of which have a longer history of public reporting initiatives than the UK. As such, the transferability of the findings from this review to the UK context will be limited by differences in the funding of the UK, US and Canadian health systems and will also vary depending on the characteristics of the individual reporting system under study. The majority of the studies evaluated the public reporting of mortality rates, although some programmes also included other outcome, process and structure indicators. The precise nature of these programmes varied enormously, but the review was not designed to provide a detailed exploration of the ways in which these variations might have influenced the outcomes. Although the review aimed to be comprehensive and used systematic search strategies, the authors acknowledged that they could not rule out the possibility that some studies might have been missed. The main analyses in the review focused on quantitative studies. The authors provided a descriptive summary of qualitative studies and their findings, but made but no attempt to synthesise this evidence.

Across all the different reporting systems studied, overall, the review found a small decline in mortality following public reporting after controlling for trends in a reduction of mortality; however, individual studies varied in their findings. For example, studies examining the impact of cardiac public reporting programmes on mortality rates found a variable picture: eight studies found a decrease in mortality rates over time,29–36 whereas another four studies37–40 found no changes in mortality rates over time. Similarly, although most studies examining the impact of public reporting on process indicators found an improvement in hospital quality, this varied from a ‘slight’ improvement to a ‘significant’ improvement in quality. The review also examined studies that specifically attempted to understand the contextual factors that may influence the impact of public reporting and concluded that QI is more likely to occur in a competitive market and following low performance.

van der Veer et al.176

This systematic review examined whether or not feedback from medical registries had an impact on the quality of patient care, as well as charting the details of how these data were fed back and the factors noted as barriers and success factors by the authors of the studies reviewed. The authors included both randomised and non-randomised studies that had explored whether or not feedback impacted on the process and outcomes of patient care. They included only studies in which feedback was provided to health-care professionals or the departments in which they worked, rather than to patients or policy-makers. Therefore, this review enables us to test the theory that the private feedback of performance improves the process or outcomes of patient care.

A total of 53 papers reporting on 47 different registries were included in the review. The authors highlighted that the ways in which feedback from medical registries was fed back to providers was heterogeneous in terms of the main purpose of the registry, the medium through which feedback was provided, the specificity of the feedback, the benchmark used to compare provider performance, the frequency with which feedback was given and the time lag between data collection and feedback. As such, the authors found it difficult to ‘make straightforward comparisons between feedback initiatives and to draw definite conclusions on the effectiveness of feedback’. Of the 53 included papers, the authors report that 22 studies (reported in 24 papers) evaluated the effect of feedback on one or more primary clinical outcome or process measure. The authors provide little detail about the different process and outcome measures used in the studies they reviewed.

The authors reported that, of these 22 papers, four found a positive effect on all measures, eight found a mix of positive and no effect and 10 did not find any effect. Across the 22 studies, the impact of private feedback was evaluated on 43 process measures and 36 outcome measures, although it is not clear how these measures were distributed among the 22 studies. A total of 26 out of the 43 process measures and 5 out of the 36 outcome measures were positively affected by feedback. This may reflect a provider’s preference for process measures because they offer clearer guidance on actions needed to stimulate change. However, it may indicate that it is simply more difficult to show an impact of any changes on outcome measures because they do not have a one-to-one relationship with process-of-care measures but rely on a multifactorial set of processes. The authors also conducted one subgroup analysis to examine the effect of feedback for those registries in which QI was not the primary goal of the medical registry but was developed for other reasons (n = 9 studies). Of these nine studies, two found a positive effect, eight found a mix of positive and negative effects and 10 did not find any effect.

The authors also recorded the perceived barriers to and facilitators of effective feedback. It is not clear from their analysis whether exploring the barriers and facilitators was the primary goal of the studies reviewed, or whether these were the opinions of the authors of the reviewed studies, based on their experiences during the study. They note that a lack of trust in data quality was the most common issue raised with the studies, while the most common success criteria were the timeliness of the data and trust in data quality.

This was not a high-quality systematic review; it is not clear how the authors assessed the quality of the studies included in their review, as they provided little detail on the primary outcomes measured in the studies reviewed and their analysis of the effect of feedback on outcomes also lacked clarity. However, their recording of the heterogeneity of the feedback characteristics was detailed. Despite this review’s flaws, we can draw a number of conclusions from it. The first is that medical registries vary in their primary purpose and in how the feedback on performance is fed back to providers. Variation in the quality of the studies apart, it is, therefore, not surprising that the impact of feedback from medical registries was also heterogeneous. The review also suggested two possible theories that could be explored further: that provider trust in these data may be an important determinant of their impact and that process-of-care measures are preferred by providers as a target for change and/or better reflect changes made to patient care.

Taylor et al.177

The authors conducted a survey of clinical audit leads responsible for four national clinical audits across NHS trusts in the UK, followed by interviews with a convenience subsample of respondents. The audits chosen for examination were:

• National Oesophago-Gastric Cancer Audit

• National Bowel Cancer Audit

• National Head and Neck Cancer Audit

• National Lung Cancer Audit.

The survey explored respondents’ perceptions of the purpose of, impact of and barriers to using feedback. The interviews further probed the findings of the survey and explored what lay behind respondents’ responses. Of 607 audit leads contacted, 274 (45%) completed a questionnaire, with response rates being higher for the National Oesophago-Gastric Cancer Audit and the National Bowel Cancer Audit. From 274 questionnaire respondents, 32 volunteered to be interviewed. Both surveys and interviews may be subject to recall and respondent bias; it may be that those who responded to the survey and interviews were those who had more favourable opinions of audit. However, the study provides useful evidence to test the theory that providers are motivated to respond to private feedback.

The findings from the survey suggested that the majority of respondents perceived the audits as useful to identify opportunities for QI (88%), to facilitate team discussions on quality and safety issues (86%) and to benchmark outcomes at their own NHS trust with those of their peers (84%). Roughly two-thirds of respondents indicated that the audits had increased their awareness of levels of performance and practice patterns among their peers. These findings did not differ among the different audits. These findings support theories of audit and feedback and benchmarking, that the feedback increases awareness of peer performance and enables recipients to identify opportunities for improvement. Just over half of all respondents (56%) indicated that they had implemented service improvements and 42% indicated they had changed aspects of their own clinical practice. Respondents from the National Lung Cancer Audit were significantly more likely to report implementing service improvements and changing practice. This might have been because these audit teams were also supported by the Improving Lung Cancer Outcomes Project,178 which ran between 2010 and 2012 and supported multidisciplinary teams to come together, identify practice variation and share best practice. The other audit teams did not receive such additional support.

The interviews suggested that the audits were perceived as an authoritative snapshot of current clinical practice, owing to the credibility of the professional societies associated with the audits. One interviewee stated: ‘I don’t think DAHNO [National Head and Neck Cancer Audit] has ever told us anything we didn’t already know, it’s just given us data that’s authoritative . . . it’s providing data that makes people realise we’re telling the truth.’177 This credibility was perceived as important when making a case to hospital managers for investment to implement changes. The interviews also explored the barriers to the use of audit data to make QI. A frequently cited issue was that the quality of these data, in terms of both the methods used and the resources available to collect the data, often limited their use. One interviewee highlighted that audit data had to compete for priority with national targets and, as such, fewer resources were devoted to ensuring audit data were collected accurately: ‘The trust . . . doesn’t appear to recognise the need for quality of clinical data to be collected accurately, because . . . it isn’t a national target that they are immediately judged on.’177 The second key issue was the availability and appropriate presentation of locally benchmarked data to enable trusts to compare themselves with local NHS trusts. The authors note that such comparative data ‘motivated trusts to improve, driven on by a competitive spirit’. 177 The authors observed that those interviewees who had been able to make changes had done so in the light of looking at how they compared with trusts locally, while those who did not make changes did not have these data available to them. These findings support theories of benchmarking, that peer comparison motivates providers to improve through peer competition.

Guru et al.39

In Ontario, Canada, coronary artery bypass graft surgery performance report cards, measuring risk-adjusted in-hospital mortality, were first introduced in 1993. Between 1993 and 1999, the report cards were shared confidentially with hospitals, with the aim of motivating QI activities. From 1999 onwards, with the agreement of the cardiac surgeons in the province, the report cards were made publicly available. This ‘natural experiment’ enabled Guru et al. 39 to examine the additional impact of public reporting over and above the impact of private reporting, by examining changes in mortality rates during this time. To do this, the authors compared changes in risk-adjusted rates of in-hospital 30-day mortality in Ontario during the period of private reporting (1993–9) with those during the period of public reporting (1999–2001). They also compared risk-adjusted in-hospital 30-day mortality rates in Ontario with those in the rest of Canada to assess whether or not the changes in rates in Ontario were different from those in rest of Canada, where neither private nor public reporting took place. The authors also examined changes in all-cause mortality and length of stay in Ontario during the same time period, indicators that were fed back to surgeons in Ontario during the observed time period. This allowed a comparison between changes in indicators that were and were not fed back to clinicians.

The authors found a steady decline in risk-adjusted 30-day mortality rates following the introduction of private reporting, with the greatest decrease occurring in 1994 (29% reduction), immediately after private reporting began. The introduction of public reporting appeared to have no additional impact on decreasing mortality rates, and there was a relative increase of 2% in 30-day mortality rates immediately following the introduction of public reporting. There was also a decline in 30-day mortality rates across the rest of Canada during the same time period, but this took longer to reach the low rates seen in Ontario. There were no clinically significant changes in 30-day all-cause mortality or readmission rates in Ontario during the same time period. In their discussion, the authors note that, for hospitals exhibiting poor performance, hospital managers requested reports at the level of the surgeon or undertook additional data collection to establish the potential cause of the problems and took action to deal with any issues identified. They also acknowledged that they could not rule out the possibility that private reporting might have simply accelerated improvements that would have occurred anyway, or that, had public reporting been introduced at the same time as private reporting, the impact on mortality rates might have been larger or have occurred more quickly.

In terms of the theories under test, the authors argued that these findings support the hypothesis that private reporting of performance to providers stimulates QI efforts and that public reporting has no additional impact on efforts to improve quality. As such, the additional pressure created by public reporting had no impact on the providers’ responses to such data. However, it is difficult to separate the impact private feedback or public reporting from general trend in the improvement in mortality rates following cardiac surgery across all countries during the time period. It is not clear that these improvements can be solely attributed to either private or public reporting of performance, as there were both changes and improvements in the techniques for performing cardiac surgery occurring at the same time.

Bridgewater et al.35

This study was a retrospective analysis of prospectively collected data on the outcomes of cardiac surgery across four centres in the north-west of England. The authors sought to explore whether or not there had been changes in the observed, predicted and risk-adjusted mortality rates following cardiac surgery since these data were publicly reported. The time period observed was from April 1997 to March 2005. The authors sought to test the hypothesis that public reporting had led to improvements in mortality rates by comparing rates before and after the introduction of public reporting. Thus, in this study, the decision about the cut-off point between private reporting and public reporting is a crucial aspect of its validity. The authors argued that the public reporting of surgical mortality rates for cardiac surgery in the UK first began in 2001, when an independent organisation, Dr Foster, published unadjusted named cardiac surgery mortality rates for all hospitals performing such surgery in the UK. Therefore, the authors subdivided the observed time period into ‘pre public disclosure’ (or private feedback) (April 1997–March 2001) and ‘post public disclosure’ (April 2001–March 2005).

The authors found a significant fall in observed mortality from 2.4% in 1997–8 to 1.8% in 2004–5 (p = 0.014). There was also a significant reduction in the observed-to-expected mortality ratio from 0.8 in 1997–8 to 0.51 in 2004–5 (p < 0.05). Both observed and risk-adjusted mortality rates were significantly lower in the post-public disclosure period than in the pre-public disclosure period. The authors are careful in their discussion not to directly attribute these findings to the introduction of public disclosure, instead merely noting that these findings were observed ‘since the introduction of public disclosure’. 35 However, in a later paragraph, they did acknowledge that their intent was to study the ‘effects of publication of results’. 35

The authors acknowledged several limitations to their study. They accepted that, over the study period, an increasing number of patients were treated with percutaneous coronary intervention, and improvements in percutaneous coronary intervention techniques also occurred. Therefore, it is unlikely that improvements in mortality rates were solely due to the introduction of public reporting. Furthermore, the introduction of public reporting may not have equated with providers feeling accountable. Although the Dr Foster report was published in 2001, Bridgewater127 noted, in a different journal article, that this publication was roundly condemned by surgeons at the time, owing to its lack of case-mix adjustment, and it could be argued that it did not result in surgeons feeling accountable or in attempts to improve their practice. In this paper, the authors argued that ‘it became clear to most members of the surgical community that publication of their outcomes was inevitable between 2001 and 2005’ and thus ‘it is reasonable to use this date as a cut off point’. 35 However, they acknowledged that there is not a clearly defined date that demarcates the introduction of public accountability. Finally, they also noted that public reporting was introduced following many years of the provision of structured, private feedback to doctors, which was also likely to have improved surgery outcomes. They admitted that ‘it is not possible for us to separate incremental improvements in outcome due to public reporting from those obtained simply by collection of the data’. 35

Hibbard et al;179 Hibbard et al.118

Hibbard et al. 118,179 conducted an experiment to evaluate whether or not public reporting was necessary to stimulate QI activities in hospitals. The study assessed providers’ responses to the Quality Counts report in Wisconsin. The report compared performance in 24 hospitals in south central Wisconsin. Two summary indices of adverse events occurring within the broad categories of surgery and non-surgery were included, along with indices summarising three clinical areas: hip/knee surgery, cardiac care and obstetric care. Hospitals were rated as better than expected (fewer deaths/complications), as expected or worse than expected. The data were derived from the Wisconsin Bureau of Health Information inpatient public use data sets.

The authors used a quasi-experimental design to assess the relative impact of private and public reporting. The public report group comprised the 24 hospitals that were the subject of and receivers of the Quality Counts report, which was publicly disseminated. The other 98 general hospitals in Wisconsin were randomly assigned either to receive a private report on their own performance or to a control group which received no report. There were no baseline differences between the hospitals in terms of size or pre-report levels of performance. The initial study assessed providers’ views of the reports and their self-reported responses to these reports. 179 The follow-up study118 assessed hospitals’ performance in the 2 years following the release of the Quality Counts report.

Six to nine months after the release of the report,179 the public report hospitals self-reported that they were more likely to have engaged in improvement activities in areas that were publicly reported, such as obstetrics (an average of 3.4 out of 7), compared with the private (an average of 2.5 out of 7) or no report hospitals (an average of 2 out of 7), although the differences were statistically significant only between the public report group and the private and no report groups combined. When only the low-scoring hospitals were considered, low-scoring public-report hospitals showed the highest level of QI activities, while the private-report hospitals showed an intermediate level and the no report hospitals showed the lowest level. Two years later,118 ‘about a third’ (an exact figure is not reported) of hospitals in the public report group significantly improved their performance, while 5% declined. Twenty-five percent of the private report hospitals showed a significant improvement, and 14% declined. When the authors examined only the hospitals that had obstetric performance scores that were worse than expected at baseline (low performers), the differences were more dramatic but showed the same overall pattern. The authors argue that these findings demonstrate that ‘making performance information public stimulates quality improvement in the areas where performance is reported to be low’179 and that there is an ‘added value’ to making performance public.

In terms of our theories under test, this is consistent with the theory that private reporting stimulates QI activities and provides some indirect support for the idea that providers respond because they have an intrinsic desire to improve. However, it also suggests that public reporting stimulates long-term improvements in quality over and above private reporting, suggesting the public reporting places additional pressure on providers to improve. It also supports the theory that low-performing hospitals that are exposed to public reporting experience greater pressure to improve than average or high-performing hospitals.

Totten et al.27

The authors divided their analysis of whether or not public reporting of hospital quality influenced a hospital’s market share into studies focusing on cardiac report cards and studies examining non-cardiac report cards. For the cardiac report cards, the authors identified nine studies; in four29,32,180,181 of these, there was ‘no impact’ of public reporting on a hospital’s market share. For the other five31,38,182–184 studies, there was ‘some’ impact on market share but ‘the effect was small and did not persist over time’. For the non-cardiac studies, one185 study found a small but statistically significant decrease in discharge rates (used as an indicator for market share) for hospitals with higher-than-expected mortality rates, while two186,187 studies reported little or no effect. A further three188–190 studies found small decreases in market share for lower-rated hospitals or for hospitals that did not produce public reports.

Thus, these studies present a mixed picture of whether or not public reporting of performance leads to changes in a hospital’s market share. We now look at some of these studies in greater detail to explore when and how changes in market share did or did not occur. Many of these studies have focused on one of the first and most comprehensively studied report card systems in the US New York State Cardiac Reporting System (NYSCRS). This reporting system has been associated with a decline in risk-adjusted mortality rates for cardiac surgery following the publication of the reports. 30 We also pay particular attention to the authors’ explanations of why and how changes to market share may have occurred, with a particular focus on why patients may or may not have moved hospital and why providers may or may not have responded to these changes.

Chassin191

Chassin191 compared the market share of high-performing hospitals before and after they were labelled as high performers by the NYSCRS between 1989 and 1995 and the market share of low performers before and after they were labelled as low performers for the same years. To do this, he compared the percentage of all coronary artery bypass graft procedures performed at hospitals classified as outliers for the year before they were labelled as such by the report with the year after. He found either very small or no changes in the market share of groups of hospitals after they were labelled as either high or low performers. He concluded that improvements in New York occurred owing to individual hospitals and cardiac surgery programmes using the data to make specific changes in the way they provided care to coronary artery bypass graft patients.

Chassin cited these findings as evidence that managed care companies (e.g. health maintenance organisations) did not use the data to reward well-performing hospitals or to steer patients towards them. He also argued that patients did not actively search for hospitals with low mortality or avoid those with high mortality. To bolster his argument he presented five case studies to illustrate how different providers made changes to the provision of care in response to being identified as poor performers. Chassin191 highlighted that the providers who took steps to improve the quality of patient care were limited to those who were performing poorly. He attributed the motivation for their efforts to make improvements as the ‘opprobrium attached to being named as poorly performing outliers’. In terms of the theories under test, Chassin’s argument is that because hospitals did not see a change in their market share following the release of the report cards, it is unlikely that concern about losing their market share was the motivation behind poor performers taking steps to improve care; rather, it was concern about damage to their reputation. However, the author did not directly ask hospital leaders in low-performing hospitals what their motivation for improving patient care was.

Mukamel and Mushlin182

Mukamel and Mushlin182 also examined changes in the market share of providers following the publication of the New York cardiac surgery report cards. They calculated both the absolute difference and the ratio between the market shares of each of 30 hospitals before and after the release of the report cards. Unlike Chassin,191 they restricted their analysis to only patients who were not part of a health maintenance organisation, as patients in health maintenance organisations are often restricted to certain hospitals and thus their choices are constrained. Therefore, their analysis included only patients who, in theory, were free to choose their hospital. Using regression analysis, they examined whether or not there was a relationship between the change in market share for each hospital and their mortality rate. Caution must be exerted in interpreting their findings, as their sample size was small and they did not account for any potentially confounding variables in their analysis. Furthermore, correlation is not evidence of a causal relationship.

Mukamel and Mushlin182 found an inverse, but non-statistically significant, relationship between the growth rate of a hospital’s market share and their reported risk-adjusted mortality rates for each occasion the report was published, with the strongest association being found following the first publication of the reports. They also found a stronger relationship between the growth rate in a hospital’s market share in upstate New York than in New York City. They provide two possible explanations for these findings. The first is that patients in upstate New York were more educated and therefore better able to understand the report cards, and also more affluent and therefore more likely to bear the expense of shifting to a higher-quality surgeon. The second is that the New York City market was more competitive and had more information on the relative performance of surgeons, through word of mouth or professional knowledge, to start with. In terms of the theories under test, this study suggests that if report cards do not bring any ‘new information’ about quality to consumers, consumers will not use them to change providers and, consequently, there will be fewer shifts in market share.

Dranove and Sfekas184

This study by Dranove and Sfekas184 tested the hypotheses that when hospital report cards align with prior expectations about quality, there is no change in market share. However, when hospital report cards provide ‘news’ or new information about quality, there will be movement in market share. Using data from 18 New York State Hospital inpatient records between 1989 (before the report was released) to 1991 (just after the first report was released), the authors developed an econometric model to understand trends in the market shares of these hospitals over time. It is important to bear in mind that quality was measured only in terms of mortality and incorporated only one patient group (cardiac), whose preferences and behaviour may not be generalisable to others. The authors attempted to control for trends in market share by creating a time-lagged variable, although it is unclear how successful this was. It is also unclear if other information excluded from the analyses might have influenced choice and market share, for example news items about health-care litigation cases or department investments (such as new scanners and facilities or expansion activities).

Their findings indicated that when hospital report cards provide information that differs from patients’ prior beliefs, patients respond by moving to the highest-quality hospital. They also found that this effect was primarily a result of patients shifting away from hospitals with ‘negative news’, rather than shifting towards hospitals with ‘positive news’. In terms of the theory under test, this suggests that hospitals that previously had a ‘good’ or even ‘average’ reputation and are then rated as ‘poor performers’ by report cards are most likely to have their market share threatened by public reporting.

Hibbard et al.;179 Hibbard et al.118

The studies conducted by Hibbard et al. ,118,179 discussed earlier, also explored the impact of public reporting on market share and providers’ views of whether they were concerned about losing their market share or wished to protect their reputation. The initial study assessed providers’ views of the reports179 and the follow-up study conducted 2 years later118 also assessed changes in market share for hospitals in the public report group by using claims data to compare the proportion of discharges from different hospitals 1 year before the release of the report and 1 year after the release of the report. They also conducted a survey of both community respondents and employees prior to the release of the report to assess their knowledge of hospital quality and resurveyed both groups immediately after the release of the report and 2 years after. It is these findings we consider here.

Six to nine months after the initial release of the report,179 the performance scores of private and no report hospitals were unrelated to what respondents thought a report would do to their hospital’s public image or their market share. Most hospitals considered that it would neither damage nor enhance either their reputation or their market share. However, among the public report hospitals, those with poor scores were more likely to indicate that the report would detract from their public image, whereas those with good scores were more likely to say that the report would enhance it. However, the performance scores of public report hospitals were not related to the anticipated impact of the report on their market share. In terms of the theories under test, this suggests that hospitals whose performance was publicly reported were concerned about their reputation, rather than about their market share.

The follow-up study also found no significant changes in the market share of public report hospitals before or after the report, suggesting that consumers did not change hospitals in response to public reporting. Despite this, consumers exposed to the report were more likely than those not exposed to the report to have an accurate understanding of the relative quality of different hospitals. In terms of our theories under test, this supports the idea that public reporting does increase consumers’ knowledge about the relative quality of hospitals and, as such, may affect a provider’s reputation. However, consumers may not act on this information and, in turn, this serves to reduce a provider’s perceived threat to their market share. Furthermore, the authors also noted that there was little existing market competition between providers, which may also explain why providers felt that only their public image, and not their market share, was threatened. For both reasons, providers did not see the threat of market competition as significant, nor was this the primary motivation for providers to respond to performance data.

Tu and Cameron192

This study reports on a survey of hospitals in Ontario 1 year after the publication of the first cardiac mortality report card, the Institute for Clinical Evaluative Sciences Cardiac Atlas, that was made available to the public in 1999. Prior to this date, the data were fed back privately to hospitals. As such, the survey provides a snapshot of the initial reaction of hospitals following the public release of the report card. The report card contained information on 12 acute myocardial infarction performance measures, and covered outcomes (e.g. 30-day risk-adjusted acute myocardial infarction mortality rates, 1-year post-myocardial infarction readmission rates) and process measures (e.g. 90-day post-myocardial infarction beta-blocker rates). A survey was sent to 121 hospitals in Ontario and complete responses were received from 51 doctors (41% response rate); 55% of respondents were chief of cardiology or medicine, while 28% were intensive care unit or coronary care unit directors. The survey contained 24 questions that explored doctors’ views on the utility of the different performance measures in the atlas, their limitations, their coverage in the media and the ways in which hospitals had responded to the release of the information. Here, we focus on the findings relevant to testing the theory that providers perceived the public release of performance data to be a threat to their market share or reputation.

Seventy-nine per cent of respondents felt that the Cardiac Atlas had had no impact on the reputation of their hospital, while 15% considered that their hospital’s reputation had improved and 6% felt that it had been harmed. Eighty-four per cent of respondents considered that the proportion of patients going to their hospital after Cardiac Atlas publication had remained the same, while 4% felt that it had increased and 12% did not know. None of the respondents felt that the percentage of patients going to their hospital had decreased. Eighty-one per cent reported that none of their patients had discussed the atlas with them and 19% reported that < 10 patients had discussed the atlas with them. Overall, 65% supported the public release of hospital-specific acute myocardial infarction mortality data.

In terms of the theories under test, these findings suggest that doctors in Canada did not perceive the public release of performance data to be a threat to their market share or their reputation. This may be because the survey was conducted only 1 year after the public release of the report card and, as such, the full impact of the report card on market share or reputation had not yet been felt by clinicians. Furthermore, the study findings may also have been affected by selection bias, as only 41% of questionnaires were completed and it is unclear if non-responders had a different perspective from that of those who did respond. Surveys in general can also be influenced by recall bias, in which respondents’ recall of events is partial or influenced by their current situation, beliefs or attitudes.

Guru et al.193

This study reports on a subsequent survey sent to cardiac surgeons in Ontario, Canada, in 2003, 4 years after the public disclosure of the Cardiac Atlas, described above in the study by Tu and Cameron. 192 The survey was sent to all 55 practising cardiac surgeons in Ontario, and 52 (95%) responded. The survey included both closed Likert scale questions and open questions and explored surgeons’ attitudes towards report cards and their beliefs about the impact of report cards.

Eighty per cent of respondents felt that public reporting was important in influencing patients choosing a cardiac surgeon and 84% felt that public reporting was important in influencing the referral patterns of cardiologists. Overall, 51% supported the public release of hospital-specific outcomes and 26% supported the public release of surgeon-specific outcomes. In responding to the open-ended questions, some surgeons expressed positive views about the role of the public disclosure of report cards in enabling public accountability and patient choice: ‘Public [have the] right to know and choose improved quality so surgeons cannot hide behind false impressions.’193 However some also expressed the opinion that surgeons, rather than the public, should receive feedback so that they can improve care, as the public do not understand report cards. As one respondent commented:

It is more important for the surgeon and department to be aware and put in place ‘checks’ to improve quality. Lay people see only a percentable mortality and assume that is either or good or bad without further understanding. 193

They also expressed concern that damage to an organisation’s reputation can outlive problems with poor care; as one respondent commented:

Has potential detrimental effects that can be long-lasting even after issues have been corrected. 193

It is not clear from the paper how widespread these concerns were among participants; in their discussion, the authors indicated that the responses to open-ended questions ‘mainly focused on the negative implications of public reporting’. The respondents and questions in this survey are different from those included in Tu and Cameron’s study192 and differences in question wording can have a significant influence on responses. Despite these caveats, in terms of the theories under test, these findings suggest that, in Ontario, concerns about the impact of public reporting on both market share and professional reputation had increased over time following a switch from private to public reporting of performance. This may be a function of the history of the programme in this setting: clinicians in Ontario had previously been accustomed to the private reporting of performance.

Hildon et al.194

Hildon et al. 194 conducted a qualitative study of providers’ (and patients’, although we do not consider these here) views of PROMs data collected by the national PROMs programme in England. Seven focus groups were conducted with a total of 107 clinicians, including consultant surgeons, junior doctors, nurses and allied health professionals. Some caution must be exercised in interpreting the findings; focus groups can be subject to ‘chatty bias’, whereby those with strong opinions or those higher up in the professional hierarchy may be more likely to make their views known and silence the views of those with different opinions. Furthermore, the focus of the study was to elicit providers’ views of different presentation formats of PROMs data; however, the facilitator allowed participants to express broader opinions about the role of PROMs data in QI. As such, the study provides a useful insight into how clinicians perceived the public reporting of PROMs might impact on their professional reputation.

Clinicians were anxious that such data might be reported in the media and misrepresented by both the press and politicians and, thus, might be misunderstood by patients. Clinicians observed:

You have to remember a politician’s going to play with these . . . the Sunday Times and the politicians are going to mess about with them. 194

You’ve got to be very careful because any data that you give out will be interpreted by people who don’t necessarily understand what it all means. 194

Clinicians also expressed grave concerns about the impact on their professional reputation of working in a trust that had been labelled a poor performer by PROMs data. Clinicians commented:

If I believe that I’m providing a five star service . . . then I’m stuck with the label because I’m in this Trust that has been labelled as a three star operation . . . You may be a very good surgeon in that Trust . . . but you’re just performing low down because your colleagues brought you down. How do you protect yourself against something like that?

Reproduced from Hildon et al. 194 with permission from John Wiley and Sons. © 2012 Blackwell Publishing Ltd

The damage to their reputations was also perceived to have an impact on their private practice, as one clinician expressed:

If I feel I’m performing above two stars . . . then I would feel that I’m being hard done by . . . It’s a serious issue . . . if people’s livelihoods depend on this. Their private practice depends on this. 194

In terms of the theory under test, this study suggested that clinicians perceived that the data may be misrepresented by the press and politicians, and misunderstood by patients. They were concerned about the potential damage to their professional reputation if they were working in a hospital labelled a ‘poor performer’ by PROMs data. In turn, they perceived that this may impact on their private practice rather than on NHS patients choosing or not choosing to attend their hospital.

Mannion et al.18

This study examined the impact of the NHS hospital ‘star ratings’ on acute hospital trusts in England. Recall from Chapter 3 that the star ratings were a single summary score of hospital performance based on a hospital’s achievement on a range of indicators and were made publicly available. Hospitals achieving three stars were judged to have the highest level of performance, two-star hospitals were performing well overall but not consistently in every area, one-star hospitals were a cause for concern and zero-star hospitals had the lowest level of performance against government targets. The authors used a multiple case study design with purposeful sampling of high-performing (n = 2) and low-performing (n = 4) trusts based on 2000–1 performance data. They undertook documentary analysis (CHI reports and internal governance reports) and semistructured interviews with between 8 and 12 key managers and senior clinicians in each site. As such, their interview findings reflect the views of senior rather than frontline staff. Here we focus on their findings in relation to the perceived impact of the star ratings on a hospital’s reputation.

One low-performing trust reported that they had been subjected to a hostile local media campaign that had resulted in adverse public reaction and an erosion of public confidence in the trust, as a quotation from a participant from this trust reveals:

[The public perception was] ‘you go to [trust B] and you die!’ We had people on the wards demanding the self-discharge forms and getting crushed in the rush to leave! It was just awful. Nurses demanding changing rooms because they didn’t want to go outside the trust [in uniform] because they were being accosted in the streets . . . And in the shops, people were saying ‘God, you don’t work for that place do you? How many have you killed today?’

Mannion R, Davies H, Marshall M, Journal of Health Services Research and Policy (vol. 10, issue 1), pp. 18–24, copyright © 2005 by Sage Publications. Reprinted by permission of Sage Publications, Ltd18

In terms of the theories under test, this quotation suggests that participants in this trust perceived that damage to their reputation as a result of the star ratings had led to patients wanting to avoid the organisation. The authors found that low-performing trusts also perceived that the poor star ratings had impacted on the hospital’s reputation and had affected both the reputation of staff within the trust and the hospital’s ability to recruit staff:

I think it [the star rating] gives a very negative view to staff, because if they are working for a one star organisation then it affects the sort of staff who want to come and work for you, but it also makes people who are currently employed here feel that they are working for a third class organisation.

Mannion R, Davies H, Marshall M, Journal of Health Services Research and Policy (vol. 10, issue 1), pp. 18–24, copyright © 2005 by Sage Publications. Reprinted by permission of Sage Publications, Ltd18

This suggests that the star ratings were perceived as affecting the hospital’s reputation not only in the eyes of the public, but also in the eyes of staff working within that organisation and those of peers in other hospitals. This highlights that a hospital’s reputation depends not only on the opinions of patients, but also on those of other clinicians (i.e. their peers).

Mehrotra et al.195

Mehrotra et al. 195 conducted interviews with 17 employers and 27 hospital managers to explore their views of and responses to employer-initiated report cards within 11 regions of the USA. The authors attempted to include hospital representatives who were supportive of and those who were opposed to report cards. Hospital managers were either chief executives or QI directors.

The authors explored the theory that the public release of report cards would capture the attention of the media and consumers and that providers would respond by improving the quality of care. They found that, in most instances, the initial public release of performance data had led to resentment, as hospitals were unhappy that data ‘they perceived to be inaccurate’ were shared with consumers. Their respondents felt that media attention had been patchy and tended to focus on ‘controversies’ such as a hospital with a previously good reputation performing poorly. Some participants felt that they were being ‘beaten over the head’ by the press and were concerned that this would scare consumers and lead to them ‘stampeding the doors wanting change or getting politicians involved’. 195 They found that, in general, participants in communities that already had public report cards were less fearful of public disclosure than those in which feedback was conducted privately. Participants who had experienced public reporting felt that, after the initial wave of media attention following the first release of the report, coverage decreased over time and consumer interest in the report was low. However, some participants felt that this low level of consumer interest was still sufficient to prompt change as it gave ‘hospitals time to fix their problems without horrible penalties . . . But if they ignore it for five years, all of sudden you’re looking at a more significant market share shift’. 195

The authors also found that hospitals preferred reimbursement (i.e. financial rewards) for high quality rather than employers directing their employees towards ‘better’ hospitals and thus increasing their market share. Chief executives felt that, although employers could threaten to do this, they could not ultimately control where patients chose to go. Furthermore, even if they could, many hospitals were full, and so patients would not be able to attend the high-performing hospital in any case.

In terms of the theories under test, these findings suggest that those who experienced private feedback were more fearful of the impact of public reporting on their market share. This may explain why Guru et al. 193 found an increase in concerns about public reporting when it was introduced following a period of private reporting. They also indicate that media coverage which focused on ‘controversies’, such as when a well-regarded hospital had a poor public report card, was the means through which public reporting had the potential to damage a hospital’s reputation. Participants perceived that this in turn would lead to consumers leaving their hospital. This reinforces the findings of Dranove and Skekas,184 that it is when report cards bring ‘new’ information to consumers that they impact on market share. However, the findings also suggested that such an effect may be short-lived and that, in general, consumers were not interested in public reports. Furthermore, they also suggested that the ‘threat’ that providers could lose their market share was an empty one if ‘better’ hospitals were full.

Greener and Mannion196

This paper presents an ethnographic case study of one hospital trust in the north of England that was part of a larger project to examine cultural change in the NHS. Although the paper is primarily focused on providers’ views of the patient choice agenda in the NHS, findings also touch on providers’ perceptions of the impact of performance data. Data were collected over a period of 2 years (2006–8) and included field notes from observations of meetings and ‘hospital life more generally’ and 60 interviews with staff from different levels of the organisation. Their findings shed light on providers’ responses to attempts by the government to increase competition between providers and on their perceptions of how GPs and patients have responded to increased information about hospital performance.

The authors found that hospital managers perceived GPs, rather than patients, to be their real ‘customers’. Hospital managers perceived that GP referral patterns were ‘long established’ and largely governed by ‘history and tradition’, rather than being influenced by data on mortality statistics. For example, one participant observed:

So people still want to go to their local hospital, and . . . you know, there weren’t, despite increasing availability of statistics on mortality and all those things, people still went, it wasn’t a discerning factor.

Senior service manager196

Managers also perceived that patients were loyal to local services and wanted to go there rather than anywhere else. A senior medical manager observed that there was ‘a huge amount of loyalty in the local population for local services’196 and that this persisted even in the face of competition from independent sector treatment centres that provided similar services. One manager explained:

The independent sector treatment centre programme . . . cherry picked, you know: we’ll do simple hips and things and we’ll do simple cataracts and cream off a bit of money. Very mixed experience, people still want to go to their local NHS for it.

Senior service manager196

This quotation suggests that patients remained loyal to their local NHS hospital and that independent sector treatment centres accepted only less complex patients, thus limiting the perceived competition they posed. Furthermore, managers commented that the demand for services was so high that they were more concerned with limiting the number of patients admitted than with losing patients. As a board member of the trust explained:

To be completely honest, we’re so flush with demand that more of our conversations are about how we can . . . constructively decline . . . patients than anything else.

Board member196

In terms of the theory under test, a notable contrast between this case study and both Mehrotra et al. 195 and Mannion et al. 18 was the lack of media coverage about public reports of mortality statistics. Under these circumstances, the availability of information on hospital performance was perceived as having little impact on the choices of either patients or GPs. Instead, these choices were seen as being driven more by habit and tradition than by external data. Furthermore, this study suggests that other contextual factors, such as a high demand for services and independent sector treatment centres (the ‘competition’) accepting only less complex patients, had also limited the extent to which hospital managers may have felt that performance data posed a threat to their market share. Indeed, it was dealing with too many patients, rather than having too few, that was their main problem.

Vladeck et al.186

Vladeck et al. 186 examined the occupancy rates, as an indicator of market share, for all New York City hospitals labelled as having mortality rates above, below or at the predicted levels according the HCFA data. They compared occupancy rates for the five calendar quarters preceding the release of the reports with occupancy rates for the three calendar periods following the report’s release. They found no statistically significant differences in occupancy rates between the two periods. They also found that hospitals with lower-than-expected mortality rates did not experience an increase in market share, and that hospitals with higher-than-expected morality rates did not experience a decrease in market share; in fact, the authors found an opposite, albeit not statistically significant, trend. The authors concluded that the public release of mortality data does not discourage consumers from utilising poorly performing hospitals.

In their discussion, the authors speculated two possible reasons for their findings. The first was that the version of the HCFA ‘death list’ they studied was ‘seriously methodologically flawed’186 and, as such, consumers might have dismissed its usefulness as an indicator of the quality of hospital care. However, one might question if consumers would have had the necessary knowledge to discern that performance data were flawed. The second is that patients’ choice of hospital more influenced was by ‘preferences for and by doctors, tradition, convenience and word of mouth . . . than with objective information about hospitals’. 186 In other words, consumers rely more on a hospital’s reputation in the eyes of friends, family and referring clinicians than on ‘objective’ information about hospital quality.

Mennemeyer et al.185

Mennemeyer et al. 185 examined discharges (as an indicator of market share) from US community hospitals with a standardised HCFA mortality rate of more than one standard deviation from the mean in any year and 50% of the hospitals that were never outliers between 1984 (2 years before the first HCFA report was released) and 1992. They also collected data about newspaper reports of hospital quality from a full-text online retrieval service, and categorised these according to whether the stories related to the hospital being a high or low outlier on the HCFA data and also to stories unrelated to the HCFA data, such as an ‘unfavourable story’ or an ‘untoward death’. They used a number of econometric models to examine the relationship between hospital discharges and the release of the HCFA reports and the media stories. They found that hospitals with higher-than-expected mortality rates did experience a small but statistically significant reduction in discharges following the release of the HCFA data. Their study also assessed whether or not subsequent press reports of the HCFA data affected the number of discharges but did not find any statistically significant impact. However, they did find evidence of a large and significant impact of press reports of ‘untoward deaths’ on market share (9% reduction in hospital use).

Mennemeyer et al. 185 concluded that the release of the HCFA data had only a small but statistically significant effect on market share and found no support for the theory that media reports of the data influenced market share, although they included the caveat that ‘limited data availability keeps us from making too much of this finding’. They argued that the public ‘essentially ignored a sophisticated attempt to measure quality’ but instead took notice of ‘very simple, unsophisticated . . . stories about untoward deaths’. 185 Pawson21 argued that this study shows that the information contained in the report cards regarding hospital performance did not reach the public’s consciousness, neither via the public release of the information nor via the media’s reports of the data, but sensational stories about ‘tales of the unexpected’ did. Mennemeyer et al. 185 noted that perhaps one reason the media reports of the HCFA data had little impact on consumers was that newspaper reports of the HCFA release often included interviews with a hospital leader who discounted the results owing to the faulty analysis by the HCFA. This suggests that the content of the media reports about public report cards may determine their impact on a hospital’s reputation and, consequently, on their market share. It is this issue that we turn to next.

Rudd and Glanz198

Rudd and Glanz198 analysed newspaper coverage of the HCFA hospital mortality data to examine the type, amount and themes of newspaper coverage of the HCFA mortality data and whether or not this was dependent on the ratings of a hospital’s quality. They took a convenience sample of 68 newspaper articles (published in newspapers serving 47 small, medium and large urban areas from 28 regions across the USA), 60 of which were published the day after the release of the reports (18 December 1987). The authors analysed where the reports appeared in the newspaper, the number of lines the reports received, whether the headline was positive, negative or neutral and the number of hospitals mentioned in the report that had higher-than-expected, lower-than-expected or as-expected mortality rates. They also examined the explanations offered by representatives of hospitals with higher than average mortality rates.

Their analyses indicated that the HCFA reports received high newsplay, with almost half of the stories appearing on the front page. They found twice as many negative headlines as positive headlines. Forty-one percent of the articles had negative headlines that emphasised higher-than-expected mortality ratings, 42.6% had neutral headlines and 16% had positive headlines. Articles with negative headlines were no more likely than articles with positive headlines to mention a high number of hospitals with higher-than-expected mortality ratings. In other words, the headline did not necessarily directly reflect the information contained in the articles. They also found that articles over-represented hospitals with higher-than-expected or lower-than-expected mortality rates and under-represented hospitals with as-expected mortality rates. The most common sources of quotations found in articles were from hospital and medical representatives. The most frequently cited explanations for higher-than-expected mortality rates were to blame some aspect of the HCFA methodology (69% of explanations) or to blame some aspect of the case mix of their patients (59% referred to the patients’ illness levels and 35% to their social characteristics).

The authors concluded that press reports of the HCFA data ‘no doubt raised consumer awareness of the concept of healthcare quality’ but also ‘planted doubt about the so-called “hard” data on quality of care’. 198 They also argued that newspaper reports did not provide ‘concrete guidance on how to obtain useful and valid data on health care quality’. 198 In terms of the theory under test, these findings suggest that the public release of performance data may raise awareness of health-care quality but does not necessarily provide consumers with valid and reliable information to inform their choice of hospital, as the data tend to focus disproportionately on those who are either below or above average. Furthermore, the findings also suggest that these reports might send the message to consumers that these data cannot be trusted.

Berwick and Wald197

Berwick and Wald197 conducted a survey of hospital leaders following the release of the HCFA data. From each state, they randomly selected two hospitals with higher than expected mortality rates, two with expected rates and two with lower than expected mortality rates, giving a total of 250 hospital in their sample. Each hospital was sent a 12-item survey exploring hospital leaders’ opinions of the accuracy and value of the HCFA data, whether or not they had used the data and the degree and extent of any problems. A total of 195 hospitals responded, giving a response rate of 78%.

Seventy per cent of respondents rated the usefulness of the HCFA data as poor, 54% rated the data’s accuracy as poor and 85% rated the data’s usefulness to consumers as poor. Hospitals in the high-mortality group were more likely to report that the public release of HCFA data had ‘caused problems in their dealings with professionals, purchasers, patients or others’. 197 In response to an open question, the most common problem reported was ‘misrepresentation of the data by press and public’. 197 In terms of the theory under test, these findings suggest that, in line with the quotations from hospital representatives contained in press reports found by Rudd and Glanz,198 hospital leaders perceived the data to be inaccurate and not useful for their own QI efforts or for patients choosing hospitals. They also indicate that leaders in hospitals that had a higher-than-average mortality rates experienced problems in their dealings with a range of stakeholders as a result of the perceived misrepresentation of the data in the press.

Faber et al.200

This systematic review aimed to explore what weight consumers give to quality of care information in choosing providers, to understand how presentation format of quality information influences patient choice, and to examine the influence of quality care information on consumer choice in real-world settings. In our synthesis, we focus on the first and last of their research questions, as these are most relevant to testing the theories set out above. The authors conducted a comprehensive search strategy and restricted inclusion to RCTs, controlled before-and-after trials and interrupted time series. The authors assessed study quality using a validated checklist, and two RCTs were excluded from the review because they were of low quality.

The authors identified five laboratory studies that used discrete choice experiments to examine the weight consumers gave to quality information in choosing a health plan in the USA. This is not the same as choosing a hospital; a health plan is essentially a health insurance policy that varies in terms of coverage of services and costs. Health plans often limit consumers to attendance at particular hospitals and these experiments often included quality information on the hospitals included in the health plan, as well information on costs. The authors concluded from the findings of two studies201,202 that ‘restrictions of provider choice seemed to overrule the weight given to quality’. One study201 found that, in the absence of cost information, information on quality only induced consumers to choose higher-quality rated health plans, and reduced the importance of other features only when there were large quality differences between the plans. Three studies examined the trade-offs consumers made between the cost and quality of health plans. Two studies203,204 found that cost data reduced the demand for high-quality health plans, provided that the lower-cost health plans were of high quality. However, another study202 found that within health maintenance organisation-managed care programmes, consumers were less likely to select a low cost plan even when these were also of high quality. Together, these studies suggest that, overall, quality data have little impact on consumer choice of health plans.

The authors also identified four ‘real-world’ experiements205–208 that examined consumers’ stated preferences in terms of their actual choice of health providers, following exposure to the Consumer Assessment of Healthcare Providers and Systems (CAHPS) performance data on new Medicaid beneficiaries or employees who had to select a new health plan. These studies all found no effect of the CAHPS information on consumer knowledge, attitudes or behaviour. However, two studies205,207 grouped together all consumers who reported having seen the CAHPS report, irrespective of their study allocation. One study207 found that the subgroup who saw and remembered the CAHPS data selected health plans with higher CAHPS ratings, while the other found that this subgroup perceived the CAHPS information as being more important when choosing a health plan. These findings suggest that whether consumers remember seeing quality data may be an important determinant of whether or not these data inform their choices.

Dixon209

This report provided an overview of findings from the national patient choice survey. A questionnaire was sent to 205,000 patients in England who had been referred by a GP for a first outpatient appointment between 15 and 28 February 2010; 69,000 (34%) responded. Here, we focus on answers to 3 of the 11 questions asked as part of this survey:

Q3: Were you offered a choice of hospital for your first hospital appointment? (Yes/no.)

Q4: Which was the single most important source of information for you when you chose your hospital? (Patients were presented with a list of eight different sources and asked to choose one.)

Q6: Which was the single most important thing for you when you chose your hospital? (Patients were presented with a list of 14 different factors and asked to select one.)

Dixon209

In response to question 3, 49% of patients were offered a choice of hospital. In response to question 4, 43% of patients who were offered a choice of hospital reported that their GP was the single most important source of information when they chose their hospital, followed by 29% indicating that their own experience or that of friends and family was the most important source. Six per cent indicated that a book or leaflet produced by the local PCT, containing performance information, was the most important source, and 4% stated that the NHS Choices website was the most important (Figure 12).

FIGURE 12.

Sources of information used by patients offered a choice of hospital. Reproduced from Dixon. 209 Contains public sector information licensed under the Open Government Licence v3.0.

In response to question 6, the most important factor affecting choice of hospital for 38% was close proximity of the hospital to their home or work, for 12% was their personal experience of the hospital, for 10% was length of wait for an appointment, for 6% was good previous experience and for 5% was quality of care (Figure 13).

FIGURE 13.

Most importance factors for patients offered a choice when choosing a hospital. Reproduced from Dixon. 209 Contains public sector information licensed under the Open Government Licence v3.0.

In terms of the theory under test, these findings suggest that just under half of patients referred to secondary care were offered a choice of hospital. The most important source of information for patients offered a choice was their GP, followed by friends and family. Few patients consulted the NHS Choices website that provided comparative information on hospital quality. Patients were more concerned about going to a hospital close to their home or work, and were more influenced by their previous experience of the hospital. This suggests that comparative information on hospital quality did not play a significant role in patients’ choice of hospital.

Abraham et al.210

This paper reported on a survey of patients attending four medical centres in Minneapolis, USA. A total of 699 patients were asked to complete the survey, of whom 467 did (a response rate of 66.8%). Patients were asked to rate how important a number of different factors were in choosing a health-care provider or doctor on a 5-point scale from ‘not at all important’ to ‘very important’. Patients were also asked about their awareness of different formal sources of cost and quality information and, if they were aware of the source, whether or not this had influenced their decision-making.

The factors with the highest ratings of importance were the reputation of the organisation (91% rating this as ‘important’ or ‘very important’) and the reputation of the doctor (90% rating this as ‘important’ or ‘very important’). Other factors included appointment availability (72%), referral from a doctor (69%) and a recommendation from family and friends (65%). The least important factors were websites that reported clinical quality data (24.2%) and television, radio and media advertisements (8.97%). The authors reported that 36% of respondents reported being aware of individual health-care providers’ websites and, of those who were aware, 5% felt that these had influenced their choice of provider. Of the other websites providing hospital quality and cost information, that with the highest level of awareness was ‘Angie’s List’, with 36% of respondents being aware of this website; of these, 2% felt that it had influenced their choice. The authors noted that this website also contained reviews and recommendations for other services, such as home improvements, in addition to health care, which suggests that respondents might have been aware or accessed this website for other reasons, rather than as a source of health-care quality information. On the basis of these findings, the authors conclude that patients prefer to rely on more informal sources of information, rather than formal sources of performance data, when choosing a hospital. This was a relatively small survey in one location in the USA. However, it provides further support to the findings from Dixon’s larger UK survey that patients rarely consult formal performance data when choosing a hospital, but instead base their decisions on a hospital’s reputation.

Djis-Elsinga et al.211

This paper reported on a survey of patients who had recently undergone surgery in the Netherlands, to explore the factors that informed their choice of hospital. The sample comprised all 2122 patients who had undergone one of six surgical procedures (aorta reconstruction, cholecystectomy, colon resection, inguinal hernia repair, esophageal resection and thyroid surgery) in three hospitals between 2005 and 2006. A total of 1329 questionnaires were available for analysis, giving a response rate of 62.6%. The survey asked patients to indicate which of 14 information items they had used to inform their choice of hospital; these comprised 11 ‘general information’ items and three items relating to quality of care information.

The most common sources of information patients had used to inform their decision about where to have surgery were the reputation of the hospital (69% indicating that they had used this information) and the friendly atmosphere of the hospital (63%). This was followed by easy access by their own or by public transport (48%), the distance to hospital (44%), good parking facilities (41%), hospital rooms equipped with personal facilities (40%) and the fact that they had already been treated by that hospital/surgeon (40%). The least commonly used sources of information were information items about the quality of care, with 3.2% of respondents indicating that they had considered the percentage of patients with an adverse outcome after surgery, 2.1% who had considered the percentage of patients with little pain and < 1% who had considered the percentage of patients with pressure ulcers. It is not clear from the survey whether patients did not consider quality of care information because they did not have access to the information, because they had access but did not understand it or because patients did not consider this to be important. However, this survey also provides further support to the theory that, when choosing a hospital, patients prefer to rely on ‘soft’ information, such as the reputation of the hospital or its staff, than on performance data.

Marang-van de Mheen et al.212

This study, by the same group of researchers, used a discrete choice experiment to examine the relative importance of quality of care information in patients’ choice of hospital. Respondents to the survey, reported by Dijs-Elsinga et al. ,211 who had recently undergone one of six surgical procedures, were asked if they were willing to be contacted again to take part in a further study; 665 out of 1329 patients agreed to be contacted, and 559 of these responded to an invitation to participate in the study. Of these, 369 agreed to take part in the study and were sent a questionnaire, and 308 returned a questionnaire.

The questionnaire consisted of a discrete choice experiment, in which patients were presented with 12 different comparisons of two hospitals. Each hospital was characterised by six attributes, which varied in terms of two different levels and were based on responses to a previous survey about which factors patients most commonly considered in choosing a hospital. The attributes included:

• previous experience with that hospital or surgeon (previous experience vs. no previous experience)

• hospital reputation (good vs. less than good)

• hospital atmosphere (personal vs. businesslike)

• waiting time for surgery (2 weeks vs. 6 weeks)

• percentage of patients with a ‘textbook’ outcome, defined as no readmissions or operations, no adverse outcome reported and a hospital stay that was no longer than average

• a surgery-specific outcome that varied according to the procedure patients had had and focused on the opportunity for minimally invasive surgery (for cholecystectomy and hernia repair)

• surgeon’s experience of the procedure (for colon resection and thyroid surgery) and volume of procedures (for aorta reconstruction).

The different attributes and their levels were described to patients in the questionnaire, and they varied between the 12 comparisons patients were asked to make. In each instance, patients were asked to choose which hospital they would attend. Patients responses were analysed using choice-based conjoint analysis.

The authors found that, overall, surgery-specific information had the most impact on patient’s choices, followed by the percentage of patients with a ‘textbook’ outcome and then a hospital’s reputation. The atmosphere in the hospital and waiting time for surgery had the least impact. However, the impact of the surgery-specific information varied between surgical groups and had more impact in patients undergoing cholecystectomy than in those undergoing inguinal hernia repair and more impact in patients undergoing aorta reconstruction than in those undergoing oesophageal resection. The authors concluded that surgery-specific information was more important than general information when patients choose between hospitals. In terms of our theory under test, it suggests that patients find quality of care information more informative than more general information in supporting their decision about which hospital to choose. This is important because PROMs data are a much more specific indicator of the quality of a hospital for patients undergoing elective surgery than standardised mortality rates for cardiac surgery. This may reflect that surgery-specific information is a more accurate or representative indicator of the quality of care that patients will receive at a hospital for a particular operation than more general information.

Dixon et al.79

This report detailed a large mixed-methods study that collected a range of data on patient and provider experiences and views of patient choice. Using surveys and interviews, the report examined patients’ revealed preferences and their stated preferences, and also included a discrete choice experiment. Four local health economies were selected to provide a sampling frame and case studies, based on their potential for patient choice (high or low, as indicated by the number of providers within 60 minutes’ travelling time) and the percentage of patients reporting having experienced choice (high or low) reported in a previous Department of Health-commissioned MORI patient choice survey, carried out in 2007. A survey was sent to a random sample of 5997 patients across the four case study areas, resulting in a response rate of 36%. The sample contained disproportionately more patients who attended independent sector treatment centres, which the authors compensated for by using weighting in their analysis of the questionnaire data.

The survey had two main components. The first asked patients a series of closed questions about their recent experience of patient choices, and the factors that informed those choices, to provide an assessment of patients’ stated preferences. Patients were presented with a list of hospital attributes and asked to rate, on a 4-point Likert scale, how important they were in informing their choice. They were also asked to indicate which sources they had used to gain information about the performance of the hospital.

The second component consisted of a discrete choice experiment, where patients were presented with five elements of information about three fictional hospitals and asked to choose one hospital. The information, or attributes, consisted of:

• waiting time and journey time to the hospital, to represent information that could be obtained from the Choose and Book system

• data on performance ratings (number of cancelled operations, hospital infection rates, improvements in patients’ health) and patients’ views from surveys (friendliness of staff, communication with patients, cleanliness, facilities) to represent information that could be obtained from leaflets printed by PCTs and through the NHS Choices website

• previous experiences and the opinions of others, to capture ‘softer’ information about the hospital’s reputation.

The level of each attribute for each hospital was varied in comparison with each other over a series of six questions, and patients were asked to choose a hospital in each instance. This enabled the authors to explore which patients were more likely to be willing to travel to a non-local hospital. Finally, patients were also asked to provide information on their social and demographic characteristics and which hospital they attended, and, using these data, the authors were also able to examine patients’ revealed preferences.

In addition, interviews were conducted with 19 patients, 49 senior members of staff from NHS and independent sector treatment centres and 25 GPs from across the four health economies.

The survey data enabled the authors to test the theory about patients being prepared to travel further for a higher-quality hospital if offered a choice. From their survey, the authors found that 49% of patients recalled being offered a choice. In response to a survey question, 73% of all patients were referred to their local hospital; of those who were offered a choice, 29% were referred to a non-local hospital, while 21% of those not offered a choice were referred to their local hospital. The authors also analysed the distances patients actually travelled to attend their appointments, and found that 39% of those not offered a choice and 53% of those offered a choice travelled beyond their nearest hospital. After weighting the data, the authors argued that, when offered a choice, between 5% and 14% more patients travelled beyond their local hospital. In terms of the theory under test, this suggests that most patients use their local hospital, but offering patients a choice slightly increases the chance that they will travel further than their local hospital for care. In their interviews, both GPs and providers perceived that patients were loyal to their local hospital because there was ‘a strong feeling that people should support the local hospital’ and because patients ‘don’t want to have to travel’ (p. 62). 79

The authors examined which patients were more likely to choose a non-local hospital from the real choices they made. Analysing data from patients who indicated they had been offered a choice, using a binary logistic regression model with choice of a non-local hospital as the dependent variable, the authors found that respondents aged between 51 and 65 years were more likely than those aged 16–35 years to choose a non-local hospital; respondents who lived in villages or rural locations were more likely than those living in cities, large towns or suburbs to choose a non-local hospital; and those holding a degree were more likely than those with no formal qualifications to choose a non-local hospital. Of particular relevance to our theory under test, the authors found that those who had a bad or mixed previous experience with their local hospital were more likely to choose a non-local hospital than those who had a generally good previous experience.

The discrete choice experiment also provided further evidence of which patients were more likely to travel to a non-local hospital. The authors found that, across all choice scenarios, 25% of patients always chose the local hospital, irrespective of the hospital’s characteristics, while 70% weighed up the different alternatives and sometimes chose a non-local hospital. The remaining 5% always opted for a non-local hospital across all choice scenarios. They found that patients without internet access, those with a low formal level of education, those who did not travel by car and those living in a big city were more likely to choose the local hospital. They found that those who would like more information to help choose their hospital, those with bad or mixed previous experience of their local hospital, those who had heard about the performance of hospitals in their area from the local media, those being referred for trauma or orthopaedics and those who had visited their GP six times or more in the last 12 months were more likely to choose a non-local hospital. In terms of our theories under test, both studies found that patients with a mixed or previous bad experience of their local hospital would be more likely to choose a non-local hospital, providing lateral support for the theory that if patients have a bad previous experience of a hospital they are more likely to ‘exit’ from a local hospital.

Findings from the survey indicated that, for patients who had been offered a choice, the five factors with the highest mean rating or importance (on a scale of 1–3) were hospital cleanliness, quality of care, standard of facilities, organisation of clinic and hospital reputation (Figure 14). Although patients rated cleanliness and quality of care as most important in their choice of hospital, in their interviews most GPs felt that patients did not judge the quality of a hospital based on clinical outcomes or performance ratings. As one GP indicated:

I don’t think I’ve ever had a patient say ‘I don’t want to go there because I’ve looked at their outcome figure and I don’t like the look of it’.

GP, Area B. Reproduced from Dixon et al. 79 with permission from The King’s Fund

FIGURE 14.

Factors that influenced patients’ choice of hospital. Reproduced from Dixon et al. 79 with permission from The King’s Fund. Note: in this figure, factors rated as ‘essential’ were scored as 3, ‘very important’ as 2, ‘somewhat important’ as 1 and ‘not important’ as 0. The scores shown are the mean score given by respondents for each factor. Scores for each factor have been rounded to one decimal place. The length of the bar represents the non-rounded score.

A hospital’s reputation ranked fifth in patients’ mean ratings of importance. In their interviews, GPs also perceived that this was an important factor for patients, and recalled instances in which patients had expressed a preference for or against a hospital based on its reputation. In their interviews, patients themselves also concurred that a hospital’s reputation informed their choices. Their quotations suggest that patients form opinions about a hospital’s reputation based on their own personal experiences, word of mouth and reports in the media:

One of the hospitals does have a bit of a reputation in the media and people do say ‘Well, actually, I would rather go to [another hospital name] because of that’.

GP, Area A. Reproduced from Dixon et al. 79 with permission from The King’s Fund

It’s got a bit of a bad reputation lately. And I think things like that do put me off. [Is that from stories in the press or just generally what you hear locally?] Word of mouth, people I’ve known use the hospital . . . And I think maybe things I’ve read in the paper, and, of course, you do read things like cleanliness and stuff like that, which, I think, straight away does put you off.

Female patient, aged 57, area B. Reproduced from Dixon et al. 79 with permission from The King’s Fund

In terms of the theory under test, these findings suggest that patients rate ‘cleanliness’ and ‘quality of care’ as the most important factors that influence their choice of hospital. It is not clear from the report whether or how the term ‘quality of care’ was defined or how patients interpreted this term when completing the questionnaire. However, it appears that patients make judgements about these factors based on their perception of a hospital’s reputation, which is informed by their experience, what they read in the press and word of mouth.

In the survey, patients were also asked to indicate whether or not they had received any advice about their choice of hospital. Of patients who had been offered a choice, 40% had received advice from their GP, 35% had consulted family and friends and 14% had received advice from a booking advisor. In their interviews, GPs reported that they tended to advise patients based on ‘local knowledge’ and ‘feedback from previous referrals’ (p. 87). 79 They also indicated that they tended to:

. . . stick to consultants we’re happy with . . . to people that we know . . . [that] we’ve had a good positive experiences in the past.

Reproduced from Dixon et al. 79 with permission from The King’s Fund

Patients were asked to indicate which sources of information they had used to choose their hospital. The most commonly cited source was their own experience (41%), followed by the GP (36%) and then friends and family members (18%). Six per cent of patients indicated that they had consulted a booklet or leaflet produced by the PCT, and 4% had consulted the NHS Choices website. Furthermore, patients were asked from which sources they had heard about the performance of the hospital. The most common sources cited by patients were personal experience (56%) and the experience of friends and family (52%), followed by the local media (28%), newspapers (22%) and the ‘grapevine’ or gossip (21%). In contrast to their role as a source of information to inform choice, GPs were much less commonly cited as a source of information on hospital performance (13%). Finally, 7% of patients had consulted official performance reports and 3% had consulted the internet. In their interviews, patients also reported that they ‘don’t look anything up on the internet’ but preferred to ‘base my opinions on my own experience, experience of close family’ (p. 89). 79

This was a large and comprehensive study, which used a range of data to examine how patients make choices about hospitals. In terms of the theory under test, the findings suggest the majority of patients still go to their local hospital, but, if offered a choice, around 50% of those patients opt to travel further than their local hospital. Patients were more likely to choose a non-local hospital if they had had a bad previous experience with their local hospital, and indicated that ‘cleanliness’, the ‘quality of care’ and the reputation of the hospital were important considerations influencing their choice of hospital. However, patients formed opinions about these factors not through consulting hospital performance data but through their own experiences, the experiences of friends and family, word of mouth and, to a lesser extent, the media. GPs were a source of advice about the choice of hospital, but were much less cited as a source of information of hospital performance; GPs themselves tended to provide advice to patients based on their own relationships with and experience of a provider, rather than on performance information. Thus, the authors concluded that providers were motivated to ‘maintain [their] reputation in order to ensure patients returned or, through word of mouth, spoke highly of their experience’ and that ‘reputation and loyalty, combined with patient’s ability to choose not to return to a hospital, create pressure on providers to deliver a high quality service’ (p. 159). 79 This suggests that providers are motivated to improve care in order to maintain their reputation and thus avoid patients going elsewhere. However, this study’s findings suggest that performance data only contribute indirectly to influencing a provider’s reputation, via what patients hear about the hospital in the newspapers or media. For the most part, a provider’s reputation depends directly on the experiences of patients at the hospital, their families and the GPs who refer them.

Lindenauer et al.215

This study makes use of a ‘natural experiment’, in which changes in hospital performance measured by quality indicators for hospitals that voluntarily participated in public reporting of performance scheme are compared with the performance of a subset of these hospitals who also voluntarily participated in a pay for performance scheme in the USA. The public report scheme, the Hospital Quality Alliance (HQA), was initiated in 2002, with all acute hospitals in the USA invited to participate and incentivised to do so by linking participation with the annual Medicare payment update; 98% of hospitals participated in the scheme. They were expected to collect and report on a minimum of 10 quality measures across three conditions: heart failure, myocardial infarction and pneumonia.

The pay-for-performance scheme in the Centers for Medicare and Medicaid Services (CMS)-Premier Hospital Quality Incentive Demonstration (HQID) was initiated in 2003. A total of 421 hospitals who subscribed to a quality benchmarking database were also invited to participate; 266 agreed but 11 later withdrew, leaving 255 hospitals. As part of this programme, hospitals were expected to collect and publicly report on 33 quality measures for five clinical conditions (heart failure, myocardial infarction, pneumonia, coronary bypass grafting, and hip and knee replacement), which also included the 10 indicators as part of the HQA programme. In addition, for each clinical condition, hospitals in the top decile on a composite measure of quality for a given year received a 2% bonus payment, while hospitals in the second decile received a 1% bonus payment. Hospitals that, at the end of the third year of the programme, had failed to exceed the baseline performance of hospitals in the lowest two deciles incurred financial penalties of between 1% and 2%.

The authors included hospitals if they submitted data on a minimum of 30 cases for a single condition annually as part of the HQA programme. They matched the 255 hospitals that participated in the HQID programme with at least one hospital that participated in the HQA programme alone on the basis of number of beds, teaching status, region, location and ownership status. They matched 199 HQID hospitals each with two HQA hospitals and eight with only one; thus, a total of 207 HQID and 406 HQA only hospitals were included. They compared the change in adherence to performance on 10 indicators shared by both programmes over eight quarters for each hospital between 2003 and 2005, and also calculated change in adherence to two compound measures for each of the three conditions (myocardial infarction, heart failure and pneumonia). They also recalculated the differences adjusting for confounding variables using a linear regression model.

The authors found that pay-for-performance hospitals showed statistically significantly greater improvement for 7 out of the 10 individual measures and in all of the composite measure scores. When differences in baseline performance and other confounding variables were taken in to account, the incremental effect of financial incentives decreased from 4.3% to 2.6% for the composite measure for myocardial infarction, from 5.2% to 4.1% for heart failure and from 4.1% to 3.1% for pneumonia; all differences remained statistically significant. The authors concluded that this suggests that the financial incentives have a modest effect on ‘catalysing quality improvement efforts among hospitals already engaged in public reporting’. However, some caution must be exercised in interpreting these results. Those who participated in the HQID programme were likely to be enthusiasts, and it is likely that they were the better performers at baseline and, thus, more likely to continue to improve their performance. It is unlikely that matching or statistical control of confounders accounted for all the possible differences between the HQA and HQID hospitals. Furthermore, the study assessed the HQID hospitals performance on only 10 indicators but they publicly reported on 33. However, in terms of the theories under test, this study provides some evidence that financial incentives and sanctions can accelerate or amplify provider responses to public reporting programmes.

Friedberg et al.216

This study examined doctors’ groups’ responses to and use of the publicly reported patient experience data, and compared the characteristics of groups that had different levels of ‘engagement’ or use of the data. The data were collected by the Massachusetts Health Quality Partners (MHQP) collaborative, which had been publicly reporting patient experience data since 2006. A total of 117 doctor groups were invited to participate in a 30-minute semistructured interview and 72 (62%) of group leaders responded. The interviews explored group leaders’ use of patient experience reports, what sort of improvement activities had been initiated as a result and data on the characteristics of the group (e.g. group size, organisational model, employment of doctors and exposure to financial incentives).

The initial step of their analysis identified three different levels of doctor group engagement with patient experience data: level 1 groups did not recall receiving patient experience surveys and did not use them, apart from distributing the reports to their staff (17% of their sample); level 2 groups took one or more actions to improve quality, but these were largely directed at doctors or sites that were low performers (22%); and level 3 groups reported one or more group-wide initiatives to improve patient experience, which included most or all staff or sites in the group. They found that level 3 groups were statistically significantly more likely to be integrated medical groups, to employ their own doctors, to be network affiliated and to be exposed to financial incentives based on measures of clinical quality. The authors concluded that their findings indicated that the improvement strategies require a ‘managerial infrastructure capable of starting and directing improvement activities’ improved by ‘payment incentives based on patient experience’. Some caution must be exercised in interpreting these results, as the authors’ findings were based on self-report; respondents may have over-reported their QI activities; and there may have been other, unexplored factors that may have explained differences in the level of engagement in doctors’ groups. Nonetheless, in terms of our theory under test, the results provide some evidence to support the idea that financial incentives can increase the likelihood that providers will initiate QI activities in response to the public reporting of performance.

Alexander et al.217

This study examined the impact of public reporting and financial incentives on the extent to which small and medium-sized doctor practices in the USA engage in ‘care management practices’ (CMPs). The authors define CMPs as ‘organized processes implemented by doctor groups to systematically improve the quality of care for patients’. 217 These include the use of patient registers, electronic medical records, doctor performance feedback and provider education. As such, the study does not examine the impact of public reporting and incentives on the quality of patient care per se, but on processes that are thought to lead to improvements in the quality of patient care.

Their analysis is based on survey data collected as part of the national study of small and medium-sized physician practices; these are practices with < 20 practising doctors. This survey focused on the 14 ‘communities who were in receipt of support from the Aligning Forces for Quality programme’. Practices that were part of the Aligning Forces for Quality programme were provided with both grants and support from people with expertise in QI to support them to both measure and publicly report on the quality of care, and to take steps to improve care and involve patients in this process. The questionnaire was sent to a stratified random sample of the Aligning Forces for Quality practices (n = 1793), of whom 67% responded (n = 1201). The authors of this paper focused on a subsample of 643 practices who were engaged in either private or public reporting of quality. They used the Physician Organisation of Care Management Index (PCOMI) as an indicator of the level of CMP use. The PCOMI is a summary measure (ranging from 0 to 24) of whether the practice uses reminders for preventative care, uses doctor feedback, has a disease register, has clinical practice guidelines and employs non-doctor staff educators in the care of people with four chronic conditions (heart failure, depression, asthma and diabetes). To explain variation in the PCOMI, the authors constructed binary indicators of (1) whether the quality performance of the practice was publicly reported; (2) whether the practice received a financial reward on the basis of its quality performance during the past year; and (3) whether the practice was aware of quality reports.

The authors found that, controlling for patient and practice characteristics, practices that received financial rewards engaged in a statistically significantly higher number of CMPs, with a PCOMI score 7 points higher than those practices that were not in receipt of financial rewards. They also found that practices that discussed quality reports at their physician meetings also engaged in a statistically significantly higher number of CMP practices, with a PCOMI score 17 points higher than those of practices who did not discuss quality reports. Practices that were subject to public reporting did engage in a higher number of CMP processes than practices that were not subject to public reporting, with PCOMI scores 8 points higher than those of practices not subject to public reporting; however, this difference was not statistically significant. Finally, they found that practices who were subject to both public reporting and financial rewards engaged in a higher number of CMPs, with PCOMI scores 10 points higher than those of practices subject to only public reporting or financial rewards. The authors argued that their findings demonstrate there is a ‘significant joint effect of having both PR [public reporting] and financial incentives above and beyond just having one of them’. 217 In their discussion, they acknowledged that their findings may indicate that doctors who participate in the public reporting of quality require financial rewards in order to invest the time required to change clinical care, or that those practices which have CMPs already in place are more likely to produce quality outcomes worthy of financial reward. Their study did not allow for an exploration of doctors’ motives.

This was a cross-sectional study and, therefore, it is not possible to infer a causal relationship between financial incentives, public report and CMPs. We can also question the validity of the PCOMI as a measure of provider engagement in CMP practices. However, in terms of the theory under test, this study provides a further layer of evidence supporting the idea that financial incentives may amplify the impact of public reporting on the quality of patient care. The study also suggests that whether or not practices were aware of and discussed the public reports of quality was more important than whether or not they were subjected to public reporting.

Doran et al.218

This paper reported on a longitudinal analysis to compare achievement rates for 23 activities included in the English primary care QOF incentive scheme and 19 activities not included. The achievement of the QOF indicators was also publicly reported through the HSCIC website and on the NHS Choices website. As such, the study allows for an exploration of the theory that financial incentive schemes can lead to the neglect of activities that are not included in the scheme, the so-called ‘tunnel vision’ hypothesis. The authors’ analyses drew on data from the General Practice Research Database, which contains patient data on morbidity, treatment, prescribing and referral from 500 general practices and covers 7% of the UK population. A sample of 148 practices that provided data continuously throughout the study period (2000–7) was selected to reflect practices with a range of list sizes. A random sample of 4500 patients was drawn from each practice.

The 42 indicators were selected from a pool of 428 indicators identified by the research as being already established indicators or those based on clinical consensus, as expressed in national guidelines. They excluded indicators that may have been affected by significant changes in the underlying evidence base or were dropped from the QOF scheme in an attempt to rule out some of the possible effects of other changes in the policy environment on achievement of the indicators. They classified the indicators into two subtypes: those relating to measurement (e.g. blood pressure measurement) and those relating to prescribing, as previous research suggested that response to these types of indicators may be different. Thus, the indicators were classified into four groups by type and whether or not they were incentivised.

The difference between the expected achievement rate and the actual achievement rate for the indicators was analysed using multivariate regression models over four different time periods: before the introduction of the QOF scheme (2000–3); during preparation for the QOF – when practices knew the scheme would be introduced but did not yet know details of the indicators (2003–4); immediately after the introduction of the QOF (2004–5); and longer term (2005–7). The authors examined the impact of the incentive scheme on the four indicator groups separately and then compared incentivised and non-incentivised indicators for each of the two types (measurement and prescribing).

The authors found that, prior to the introduction of the QOF (2000–3), achievement increased significantly for 32 out of the 42 indicators, decreased significantly for two and did not change for eight. The authors argue that these findings suggest that quality initiatives introduced over this period such as clinical audit, and the introduction of statutory bodies focused on QI (such as NICE), had an impact of improving the quality of care in general practice. The authors found that achievement rates improved at the fastest rate prior to the introduction of the QOF (between 2000 and 2003) for those indicators that were subsequently incentivised under the QOF system. Thus, the QOF indicators focused on areas of care that previously had already shown the greatest improvements, suggesting that they reflected areas of care in which practices were already performing well and/or perceived to be important.

In the first year following the introduction of the QOF, achievement rates for incentivised indicators increased substantially for all measurement indicators, with increases above the predicted rates in 2004–5 of up to 38%. The prescribing indicators had a higher baseline rate than the measurement indicators, increased at a slower rate during the pre-intervention period and, although they also saw significant increases above the predicted rates in the first year, these increases were smaller than those found for the measurement indicators (1.2–8.3%). However, collectively, the incentivised indicators ‘reached a plateau in the second and third years of the scheme’ where ‘only 14 of the 23 incentivised indicators had achievement rates significantly higher than rates projected from pre-intervention trends after three years’.

For the non-incentivised indicators, achievements rates immediately following the QOF improved in line with achievement rates projected from pre-intervention rates. However, in the second and third years, the rate of QI slowed down relative to expected achievement rates. By 2006–7 the authors found that quality was significantly worse than expected from pre-intervention trends, especially for measurement activities. Improvement rates were also significantly lower, relative to projected rates, than achievement rates for the incentivised indicators.

This study did have a number of limitations, acknowledged by the authors, that may have affected its conclusions. Achievement rates may have been affected by changes in the consistency and accuracy of recording indicators over time, especially for incentivised indicators. They may also have been influenced by changes to the case mix of patients subject to the indicators, especially as the incentivised indicators also encouraged increased case finding (e.g. for depression). The practices selected may not have been representative of the population, although the trends found in this study replicated those seen nationally. The authors focused on a limited selection of indicators, which may limit the generalisability of their findings, but at the same time increased the likelihood of attributing differences in achievement rates to the incentivised/non-incentivised status of the indicators, rather than due to other contextual changes. As such, we can be reasonably convinced that this study provides a useful test of the theory that financial incentives may lead practitioners to focus on incentivised aspects of care, to the detriment of non-incentivised aspects.

This study found that although financial incentives increased the quality of care in the short term, in the longer term their impact petered out. The authors suggest three possible explanations for their findings: (1) that the improvements seen in the first year were due to better recording procedures; (2) that practices reached a ‘ceiling’ limit of quality in the third year, with little opportunity for further improvement; or (3) that practices took their foot off the accelerator for these incentivised activities because they had already reached the threshold at which they would receive the maximum amount of remuneration, and further effort would not result in increased income. Of particular interest to our synthesis is the finding that there was no detrimental impact on aspects of care that were not incentivised in the short term, but in the longer term the scheme had ‘some detrimental effects’ on certain areas of care, particularly measurement activities. In terms of the theory under test, this provides some support for the idea that financial incentives can lead to providers focusing on aspects of care that are incentivised at the expense of other areas of care.

Wilkinson et al.219

Wilkinson et al. 219 explored the views and responses of 52 staff, in 15 general practices, to cardiovascular and stroke performance indicators that were developed by the authors. The indicators largely focused on process and were similar to those later included in the National Service Frameworks for coronary heart disease and later the QOF. However, this study was undertaken before those initiatives were implemented. For half of the practices, the academic team collected the data themselves, and for all practices they fed back data to the practice in a 1-hour presentation; as such, the indicators were fed back privately, rather than publicly. During this presentation, the academic team explained how the indicators were developed, how the indicators for that practice compared with those of other practices in the study and the potential clinical benefits if full uptake of the indicators were obtained by the practice. The practice was encouraged to develop an action plan to address changes that they practice felt were necessary. Two months after the presentation, the authors interviewed a range of staff at each practice, including the GPs who led audit activities within the practice (n = 15), other GPs (n = 14), practice nurses (n = 12) and practice managers (n = 11).

The authors found that almost all of the GPs and nurses, and half of the practice managers, questioned the validity of the data used to generate the indicators because of gaps in the data, computer-related difficulties and confusion in applying Read codes. The most common response to the feedback of the indicators was to improve the number, uniformity and accuracy of recording data, with almost half of the practices attempting to do this. The most common reason cited was to ‘demonstrate to other practices within the primary care group that their own practice was providing good care’219 and, less commonly, to prompt GPs to improve patient care. Three of the 15 practices initiated an audit to validate the data used to produce the indicators. The authors note that ‘all the professionals found the comparative nature of the results useful in interpreting their practice’s performance’. 219 In their interviews, respondents mentioned that the comparative indicators had highlighted a gaps in their own performance compared with their own perceptions, and differences between their own performance and that of their peers:

[O]ne imagines that one is doing a fantastic job, then when you actually see it in writing you think oh that’s not quite as good as you think. I am sure that this sort of presentation really winds you up to do better.

GP

It is helpful to be able to compare to local means and see whether you are doing a bit better or worse, and that perhaps is one of the strongest ways of getting GPs to alter things . . . they like to be seen to doing things a bit better than their colleagues.

GP. Reproduced from Quality in Health Care, Wilkinson EK, McColl A, Exworthy M, Roderick P, Smith H, Moore M, Gabbay J, vol. 9, pp. 166–74, © 2000, with permission from BMJ Publishing Group Ltd219

Out of the 15 practices, 11 developed action plans for change, but these were largely in the form of ‘informal verbal agreements devised by one or two enthusiasts who were usually doctors’ and most focused on single changes. Communication of the action plans within practices was ‘ad hoc and informal’. 219 The authors identified that change was constrained by a lack of time and resources (both financial and human) to act on the information. Change was supported if the indicator represented a personal interest of one of the practice members or there was someone in the practice who had been allocated responsibility for that clinical area. Change was also supported when the indicators ‘accorded with other ongoing local and national initiatives’ which serve to increase ‘the status or relevance of the indicator results’. 219

In terms of the theories under test, this study indicates that, in line with feedback and benchmarking theories, the private feedback of their performance to GP practices highlighted gaps between their own performance and the indicators, and highlighted differences between the performance of their practice and that of other practices. This prompted the practices to reflect on their own practice, both as individuals and as an organisation. However, it also suggests that GPs had little trust in the validity of the indicators and sought to improve the accuracy of their data, or, less commonly, to investigate the reasons behind the indicator findings via audit. We could hypothesise that these activities were important to (a) improve the trust the practices had in their data and (b) provide a further basis on which to initiate any changes. However, the authors found that, beyond these activities, practices made few formal, co-ordinated attempts to change practice because they did not have the time, resources or interest in doing so. The authors argued that the ‘absence of specific incentives to change, either positive or punitive, meant that responses were purely voluntary’. 219 In terms of the theories under test, this suggests that the private feedback of data that are not trusted by their recipients, and without incentives attached to them, may prompt individuals or organisations to reflect on their practice and to improve the accuracy of the data collected, but it does not lead to any longer-term improvements to patient care.

Mannion and Goddard220,221

These papers report on provider and health board responses to the Clinical Resource and Audit Group (CRAG) indicators in Scotland. The CRAG indicators were compiled and disseminated by the then Scottish Executive. They consist of seven reports including 38 clinical indicators covering a range of specialties, and individual trusts and health boards are named in the reports. They were not part of a formal framework of performance management, and the Scottish Executive indicated that they should not be used to make definitive judgements about the quality of services.

The authors conducted case studies of eight Scottish NHS trusts that varied by size, geographical area and performance on the indicators. They focused on the impact of two specific indicators relating to 5-year survival of women with breast cancer and 30-day survival after admission for stroke. All eight trusts were ‘average’ for the breast cancer indicators; six of the eight were average for the stroke indicator and two were worse than average. In each trust, they interviewed the chief executive, medical directors, consultants with responsibility for stroke services, consultants with a responsibility for breast cancer services, nurse managers and junior doctors (n = 48). The authors also interviewed the director of public health (or deputy) at the local health board for each trust, and key staff from the Information and Statistics Division of the Scottish Executive and the CRAG secretariat, to explore the intended purpose of the indicators.

With the trusts, the authors found that the CRAG indicators were ‘rarely cited by staff as the primary driver of QI or sharing best practice between organisations’. 220 The indicators were not integrated into formal clinical governance systems, but were mainly used by trusts to argue for increased resources for services. For example, for the two trusts that were ‘worse’ than average for the stroke indicators, the CRAG data were used to argue (successfully in one case) for a new stroke unit in order to improve care. These trusts also conducted further audits to check the quality of the data, and one introduced patient protocols. Two of the ‘average’ trusts also used the CRAG data to argue for a new stroke unit, while, in another, the health board declined a request for additional funding because the CRAG data were satisfactory. For three other ‘average’ trusts, the CRAG data had no discernible impact on stroke services. In response to the breast cancer indicators, in three trusts the data were used alongside other information to inform or argue for the set-up of new services.

Similarly, they found that CRAG indicators were rarely used by health boards to make definitive judgements about the quality of care in trusts. If they were used at all, they were used to highlight potential problems requiring further scrutiny, especially if the trust was identified as a significant outlier. The health board would then meet senior staff within the trust to ‘express concern and explore the problem in further detail’. 221

The authors identified some of the reasons for the low impact of the indicators. Both health boards and trust staff, especially consultants, questioned the quality of the data (e.g. inconsistent coding and quality of case-mix adjustment) and did not perceive them as credible. The data were also not perceived to be timely, due to the time lapse (at least 1 year) between their collection and publication. The data also appeared to be poorly disseminated to frontline trust staff; consultants and chief executives were aware of the data, but nurse managers and junior doctors were not. Similarly, within health boards, data were discussed at board level and by senior management, but were disseminated downwards only if ‘the Health Board or a specific speciality was identified as being a significant national outlier on the indicators’. 221 The indicators were not part of a formal system of performance assessment and, as such, there were ‘weak formal incentives for staff to perform satisfactorily on the indicators’. 221 Furthermore, health boards ‘did not hold staff accountable for their performance’222 and they were not used as a basis for changing contracts to a different trust. However, some trust staff did acknowledge that the indicators could sometimes enhance their ‘professional status and reputation’. 221

These findings suggest that trust and health board staff did not perceive the CRAG indicators to be credible, and that the indicators were poorly disseminated within these organisations. Both good and poor performance on the indicators was used to justify requests for additional resources but performance on the indicators was rarely used as a basis to initiate improvements patient care. In terms of the theories under test, the CRAG indicators were not linked to any rewards and sanctions, and in their discussion the authors observed that ‘Many Trust and Health Board staff identified this as the key reason why the indicators effected little change in provider organizations’. 221 However, they also noted that ‘the introduction of explicit incentives may lead to reduced performance if this crowds out intrinsic professional motivation’. 221

Mannion et al.18

This study examined the impact of the NHS hospital ‘star ratings’ on acute hospital trusts in England. Recall from Chapter 3 that the star ratings were a single summary score of hospital performance based on their achievement according to a range of indicators that were made publicly available. This system was also accompanied by a combination of financial rewards and sanctions. Trusts achieving a three-star rating were granted ‘earned autonomy’ in the form of less frequent monitoring and inspections for the CHI, retention of profits from the sale of hospital land to reinvest in services and the right to become a foundation trust. Their ratings also determined the level of discretion chief executives had to make use of the ‘NHS Performance Fund’ to incentivise QI at a local level. Trusts with a zero-star rating were required to produce a ‘Performance Action Plan’ indicating the steps taken to improve care, which had to be agreed with the Modernisation Agency and the trust’s Department of Health regional office. The authors used a multiple case study design with purposeful sampling of high-performing (n = 2) and low-performing (n = 4) trusts based on 2000–1 performance data. They undertook documentary analysis (CHI reports and internal governance reports) and semistructured interviews with between 8 and 12 key managers and senior clinicians in each site. As such, their interview findings reflect the views of senior, rather than frontline, staff.

Participants expressed a general view that star ratings did not adequately reflect hospital performance, in terms of either their coverage or their sensitivity to local factors that were perceived to be beyond their control. Low-performing trusts especially felt that areas of excellent practice were not taken into account in the indicators. The study also reported that the star ratings had served to align internal performance management activities with national targets and direct resources on those aspects of performance seen as important by government. Some staff in low-performing trusts reported that star ratings were useful in illuminating dysfunctional senior management that previously remained unchallenged.

However, this study also found a number of unintended consequences of the star ratings. Some trusts reported manipulating and misrepresenting the data (e.g. not accurately reporting the number of 12-hour trolley waits) and gaming the data (e.g. cancelling operations the night before rather than on the day) to improve their ratings. These findings align with the hypotheses from benchmarking and public disclosure theories, that the lower the acceptance of the data, the more likely organisations are to engage in efforts to improve the presentation or appearance of the indicator. Another unintended consequence reported in this study was a perception that public disclosure had led to ‘tunnel vision’, with trusts focusing on the issues measured to the exclusion of unmeasured but important areas. One example reported in the paper was that the waiting time target of 13 weeks in children’s services had ‘forced the trust to concentrate on children referred to it by doctors, rather than professionals, even though the clinical needs of the patients may be very similar’. 18

In their discussion, the authors concluded that hospitals use public reports as ‘a lever to influence staff behaviour’ and noted that the ‘unintended and negative consequences of the star rating system came across loud and clear’. 18 The authors contrasted the high profile of the English star ratings with the relatively low profile of the Scottish CRAG indicators. They hypothesised that one reason for this was the effectiveness of the dissemination strategy and the simplicity of the rating system, making it easier for both professionals and the public to understand. However, a consequence of this simplicity was that few participants in this study felt that the star ratings adequately reflected the quality of the hospital and dismissed the ratings as invalid. We note here that a further key difference was that star ratings had a system of rewards and sanctions attached to them, whereas the CRAG indicators did not. In terms of the theory under test, this study suggests that when rewards and sanctions are attached to public reporting, but providers do not accept the validity of indicators, their efforts may focus on demonstrating the appearance of a high performance, resulting in unintended consequences such as effort substitution and gaming.

Dowrick et al.81

This study aimed to examine both GPs’ and patients’ views about the introduction of the routine collection of standardised measures of depression severity, which was incentivised in the QOF. Under this framework, GPs received QOF points for administering a standardised measure of depression for patients coded as being newly diagnosed with depression, and also received points for reassessing patients using a standardised measure of depression 2 weeks later. Furthermore, GP QOF scores were made available to the public via the HSCIC website from 2005 onwards. The authors interviewed 34 GPs and 24 patients from 38 practices in three locations in England. Here, we focus specifically on GPs’ views of the validity of the standardised depression measures and how the incentives built into the QOF influenced GPs’ use of the measures in practice.

The authors found that GPs questioned the validity of standardised measures of depression, both as clinical tools to aid patient management and as aggregate measures of the prevalence of depression. For example, one GP indicated, ‘I don’t have sufficient confidence that it’s an objective enough tool, really, to measure trends’. 81 They also expressed scepticism about the perceived motivation behind this QOF indicator, suggesting that its introduction was based not on an extensive consideration of the evidence that the indicator would improve care but on the hunches of a few academics and policy-makers to make extra work for GPs. As one GP remarked, ‘I have a horrible feeling that a few academics got together and said this is a good idea and someone at the Department of Health said, oh yes, this is another hoop to make GPs jump through’. 81 Coding a patient as having depression but then not using a standardised depression questionnaire to assess them would result in a lower QOF score and, thus, a loss of income for practices. However, the use of standardised depression questionnaires took up time in the consultation. The authors found that this set of conditions resulted in some GPs being reluctant to code people as having depression to avoid having to use a questionnaire, thereby saving time in the consultation. As one GP explained, ‘I think we stop and pause a little bit before we actually put the depression code in. And, of course, there was a mad scramble around the Read codes to find a Read code that wouldn’t get picked up by the QOF’. 81

This study suggests that GPs did not perceive the use of standardised depression measures to be valid or necessary tools in their management of patients with depression. However, if GPs did not use them, they ran the risk of losing income. Under these circumstances, GPs avoided the potential loss of income through manipulating the ways in which they coded patients suspected of having depression. In terms of the theory under test, this study suggests that if providers perceive that the quality indicators that are subject to public reporting do not reflect what they perceive as good-quality patient care but they are financially incentivised to fulfil them, they may resort to the gaming and manipulation of data to avoid both having to fulfil them and the consequent loss of income.

Mitchell et al.80

This study also explored the impact of the QOF on the diagnosis and management of depression in primary care. The authors took a purposive sample of four GP practices and conducted one focus group in each practice. Focus group participants included GPs, practice nurses, community nurses, primary care mental health workers and practice managers. They were asked to describe how the introduction of the QOF and NICE guidelines had influenced how depression was diagnosed and managed.

The authors observed that GPs found the use of the PHQ-9 (a standardised depression rating scale) time-consuming to use during the consultation and had adapted how they administered the questionnaire to fit with their consultation style. These included letting the patient self-complete the measure in the waiting room, reading out the questions to the patient and recording the answers themselves, recalling the questions from memory during the consultation and working out the score afterwards, and telephone administration of the questionnaire. The authors noted that a number of these ‘workarounds’ may have compromised the validity of the PHQ-9. GPs perceived that the PHQ-9 did not facilitate the clinical management of the patient, as they preferred to rely on their ‘gut feeling’ to determine how depressed a patient was, which they felt was often not reflected by the PHQ-9 score. Rather, the impetus to use the PHQ-9 was ‘the potential for missed targets’. 80 The authors report that the financial incentives attached to the QOF acted as a ‘disincentive to code depression if a PHQ-9 was not completed by the patient’. 80 Instead, GPs used alternative codes, such as ‘low mood’ or ‘stress’, to avoid recording ‘mild’ symptoms as depression. As one GP explained ‘diagnoses of what would be “QOF-able” depression has probably dropped . . . we realised if we kept labelling people as depressed when they perhaps weren’t, then we weren’t going to see them again and lose the points’. 80

This GP is referring to a scenario in which a patient with mild depression or stress consults their GP only once and does not return to the practice for follow-up, perhaps because their symptoms have resolved. In this situation, if the patient was initially coded as having depression but then did not return for a follow-up consultation, they would not be able to complete a PHQ-9 questionnaire at follow-up, the practice QOF score would go down and the practice would lose income. It reflects the clinical uncertainty regarding the diagnosis of depression and how the QOF created a penalty for getting this diagnosis ‘wrong’ that did not exist before its introduction. In response, GPs avoided coding patients as having depression. In terms of the theory under test, this study suggests that attaching financial incentives to quality of care indicators may create perverse incentives unless the quality indicators contribute to patient management and allow for the clinical uncertainty inherent in the practice of medicine. In situations where perverse incentives are created, this may lead to gaming or effort substitution. It also highlights the tensions between the use of PROMs as a tool to reward good practice at an aggregate level and their use as individual patient management tools.

Bradley et al.224

This study aimed to identify successful strategies and common difficulties in implementing data feedback initiatives in hospitals. The authors focused on exploring hospitals’ efforts to improve beta-blocker use after acute myocardial infarction and purposively selected eight hospitals from across the USA whose performance in this clinical area varied substantially. They conducted semistructured interviews with between four and seven staff at each hospital; in total, 45 participants were interviewed, comprising 14 medical staff, 15 nursing staff, 11 staff with responsibility for quality assurance and five senior administrative staff. Interview questions focused on how staff had collected and used the data and the degree to which they perceived the data had been effective in improving care. The authors’ analysis focused on identifying ‘what worked’ and what did not in collecting and implementing these data.

The authors identified seven key themes underpinning what made data feedback effective; three of these themes focused on the credibility of the data. The authors report that medical staff at every hospital felt that the data must be valid and perceived as valid by clinicians in order to have any impact on doctors’ behaviour. If data were perceived as valid, clinicians were less likely to reject or ‘argue with’ them and were more likely to respond to them. However, participants also recognised that gaining clinicians’ trust in the credibility of the data took time and required effort. Strategies used to increase the credibility of the data included nurses sitting alongside doctors to demonstrate that they could accurately abstract information from patients’ notes in order to create the feedback reports and to investigate any perceived inaccuracies with the data quickly ‘until we’re sure it’s clean’. 224 Finally, participants also explained that the timeliness of the data and the ways in which they were presented were also central to their perceived credibility. In particular, participants felt that collecting ‘real-time’ data that were ‘no more than 3–6 months old’ and ensuring that they were presented by someone who was ‘clinically competent’ were important ingredients in maintaining the credibility of the data. This study was based on the views of staff in only eight hospitals across the USA; however, it provides some initial evidence to unpick what is meant by and what supported data credibility. In terms of the theory under test, these findings suggest that the credibility of the data depends on the processes through which these data are collected and presented.

Mehrotra et al.195

Mehrotra et al. 195 conducted interviews with 17 employers and 27 hospital managers to explore their views of and responses to employer initiated report cards in 11 regions of the USA. The authors attempted to include hospital representatives who were supportive of, as well as those who were opposed to, report cards. Hospital managers were either chief executives or QI directors. The aim of the study was to explore the determinants of successful report card efforts and to understand why some report card initiatives failed.

The authors found a mix of successes and failures in terms of whether or not report cards were perceived to have stimulated QI in hospitals. In communities where report cards had been successful, interviewees felt that there had been increased attention to quality and an increase in the presence of quality directors at board of trustee meetings. They identified a number of contextual factors or system tensions that prevented the success of report cards. They found considerable ambiguity and tension between these different stakeholders concerning the purposes of report card initiatives. In some communities, hospitals were unclear what the purpose of the report card was. In others, there was tension between employers, who were perceived to have introduced report card systems to reduce costs, and clinicians, who felt that improving quality should be the primary goal of report card initiatives.

The authors also identified conflicts regarding how quality was measured, including concerns about case-mix adjustment, whether outcomes or process measures should be used, and the validity and cost of data used to produce report cards. In terms of case-mix adjustment, hospital leaders felt that it would never be possible to develop reliable methods of case-mix adjustment, while employers felt that imperfect case-mix adjustment was better than none. Hospital leaders felt that outcome measures did not enable them to identify the source of the quality problem, while employers felt that it was a hospital’s responsibility to undertake additional work to identify this. The authors found that the most ‘contentious’ issue between hospitals and employer coalitions was the data used to generate report cards. Many employer report card initiatives used administrative billing data that hospital leaders considered inadequate for QI, as they perceived these data to provide financial rather than quality information. Employers were more accepting of the use of administrative data. Hospitals preferred the use of clinical data to produce performance indicators but were frustrated that they had to pay for these data to be abstracted from patients’ notes in order to produce report cards that they did not want in the first place. Finally, the authors found that the degree to which hospitals were involved in report card design and modification influenced their acceptance of the data.

The authors concluded that they could find no consistent set of report card characteristics that predicted which report cards were successful in initiating QI activities by hospitals, apart from the finding that successful report cards did not use administrative or billing data. They hypothesise two possible explanations for this finding: (1) that hospitals ignored the report cards because they dismissed administrative data as inaccurate, or (2) that hospitals were not involved in the design of such cards and, as such, felt little ownership over the purpose of the scheme.

Boyce et al.58

This study explored Irish surgeons’ experiences of receiving peer-benchmarked feedback, replicating the same measures used as part of the UK national PROMs programme for hip surgery. However, unlike in the UK national PROMs programme, the feedback provided to surgeons in this study was at the individual surgeon level, rather than at the provider level. The feedback was not being implemented routinely and was not publicly reported, but was a ‘one-off’ private feedback intervention. The format of the feedback was also different from that provided by the national PROMs programme; rather than receiving a funnel plot, surgeons received a ‘caterpillar plot’ that graphically presented the average health gain on the OHS plus 95% confidence intervals for all surgeons (anonymised), with their own score highlighted. In this way, the surgeons were able to see how their own score compared with that of others.

The paper reports on a qualitative research study that was nestled within a larger RCT of PROMs feedback, the Patient Reported Outcomes: Feedback Interpretation and Learning (PROFILE) trial, the results of which were reviewed in the previous chapter. 174 This study aimed to evaluate the effectiveness of the NHS PROMs programme methodology for surgeon-level feedback in an Irish context. PROFILE tests the hypothesis that surgeons who received benchmarked PROMs feedback will have better future outcomes than those who do not. Surgeons were randomised to the intervention arm of the PROFILE trial and received peer-benchmarked feedback. All 11 surgeons in this feedback arm of the trial were invited, and agreed, to participate in face-to-face interviews. The participants varied in terms of the setting of their usual workplace, their relative performance ranking and their previous experience of using PROMs. The interviews explored surgeons’ experiences of using PROMs, their attitudes to using PROMs as a peer benchmarking tool, the methodological and practical issues with collecting and using PROMs data and the impact of the feedback on their behaviour.

The authors found that surgeons had conceptual and methodological concerns about the use of PROMs data, which led the surgeons to question the validity of these data. Unlike other performance indicators, PROMs rely on the subjective judgement of patients, and surgeons questioned patients’ ability to report on issues such as pain and function. Surgeons also confused PROMs with patient experience; they assumed that PROMs captured, and would be unduly influenced by, patients’ experiences of their care, and they were also concerned that patients may either underestimate or overestimate preoperative and postoperative outcomes. Furthermore, many surgeons expressed disbelief about the percentage of patients who reported that they had not improved or had had problems after surgery, as these figures did not match their clinical experience and the verbal feedback received from patients. They also expressed concerns about the impact of patient case mix and differences in hospital resources and levels of community support that may affect comparisons between surgeons or providers. They also questioned the timing of PROMs follow-up and did not feel that 6 months’ follow-up would fully capture the full benefit of the operation.

The surgeons also had difficulty interpreting and understanding the meaning of the data. They felt that PROMs feedback alone was not sufficient to provide an explanation for poor performance and that it did not enable them to identify opportunities for QI. This was because the surgeons perceived there to be a number of causal factors that may lead to poor PROMs scores, and thus felt that the PROMs scores did not, in themselves, highlight which of these factors required addressing. This relates to audit and feedback theories, which hypothesise that feedback must unambiguously provide information on the cause of poor performance and identify ways in which it can be rectified. The study also highlighted a number of practical issues around collecting and using PROMs data that created barriers to positive engagement with the exercise. Data collection added to workload pressures, and many surgeons stated that their supporting staff were not willing to accept this added workload. Political will at a hospital and system level were thought of as important in order to sustain any QI as they required local resource flexibility. There were also concerns about training in the use of PROMs.

The study also sought to understand how surgeons’ attitudes to PROMs data related to their use of these data for QI activities. The authors’ analysis identified three distinct groups of participants in terms of attitudes towards the data (typology): advocates, converts and sceptics. The advocates expressed a positive attitude towards the feedback they received, which they believed had an impact through promoting a reflective process focusing on their clinical practice. However, specific changes to care were not discussed. The converts were uncertain about the value of PROMs, and this reduced their inclination to use these data. This group generally felt that it is important to know what patients think about their outcome, but emphasised the need to provide actionable feedback. The sceptics believed that the PROMs feedback they received was not clinically useful and had no impact on their behaviour. They felt that there were too many methodological concerns to trust these data, and that these data did not provide a useful source of ideas to stimulate QI.

In terms of the theories under test, this study suggests that surgeons questioned the validity of PROMs data because they mistrusted the idea that patients’ subjective experiences formed a valid indicator of the quality of care, and because they felt that the instruments themselves, the timing of measurement and the ways in which the data were adjusted for case mix did not provide an accurate indicator of the quality of patient care. This study provides some support to the theory that, owing to the multiplicity of factors that may be causally linked to an outcome, providers find it more difficult to identify the possible causes of poor outcomes.

California Health Outcomes Project reports

New York State Cardiac Reporting System

We now turn to the NYSCRS. These report cards are based on clinical data abstracted from patients’ notes and were overseen by a committee of cardiologists and cardiac surgeons. On this basis, therefore, we might expect that the cards would have been better received than the CHOP reports. One survey directly compared provider responses to the CHOP and the NYSCRS reporting systems, to test the theory that reporting systems based on clinical data were viewed more favourably by providers than those based on administrative data.

Hafner et al.75

This study explored provider views of the nationally standardised acute myocardial infarction, heart failure and pneumonia performance measures produced and reported by the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) in the USA. The Joint Commission is a non-profit organisation that accredits hospitals in the USA, and public reporting of quality indicators forms a mandatory part of that accreditation. The JCAHO also carries out inspections to check that hospitals are meeting the minimum standards. Thirty-six hospitals were randomly selected from a sampling frame of 555 Joint Commission-accredited hospitals and invited to take part in the study. Twenty-nine hospitals agreed to participate; nine had performance indicators consistently above the mean and were ‘high’ performers, seven were equal to or below the mean (the authors classed them as poor performers, but it could be argued that there is a difference between being at and being below the mean) and 13 hospitals had both high and low scores (‘mixed performers’).

Data were collected as 29 focus groups in each hospital with a total of 201 participants, including managers and frontline staff such as doctors, nurses and administrators. The focus groups consisted of mixed groups of staff and were conducted by Joint Commission staff, who were responsible for both producing the performance indicators and accrediting the hospitals. The questions asked tended to focus on the positive elements of public reporting rather than the negative aspects. Thus, the study was at risk of numerous potential sources of reporting bias; frontline staff may not have felt able to express contrasting views to those of senior managers. The authors report that ‘in interviews involving both leadership and frontline staff, more detailed responses to questions were proffered by those in leadership roles with front line staff affirming response either with non-verbal cues or simple one word responses’,75 suggesting that ‘chatty bias’ might have been at play here. As such, findings are more likely to represent the views of managers than frontline staff. Furthermore, participants may have been reluctant to express negative views about performance indicators to the organisation responsible for accrediting their hospital and producing the indicators.

The paper reports largely positive impacts of performance reporting on QI initiatives. The authors found that the public reporting of performance resulted in managers, clinicians and administrators becoming more engaged in QI activities. A nurse commented that it provided justification for securing additional resources for QI activities. It also served to prioritise and focus attention on issues raised by the performance data. For high-performing hospitals, this arose from their desire to maintain this status. For low performers, the data led to an awareness of the need to critically analyse the data and respond to the findings.

Throughout the paper there is evidence that the drive to improve did not come simply from the intrinsic motivation of the staff but from the fact that the data have been made public. Participants reported that making the data public had ‘drawn their attention to it’ and ‘forced them’ to look at the data and respond to them. For example, an administrator indicated ‘When you tell a surgeon his numbers are going to be out there, you get their attention and they ask what they need to do!’. 75 Similarly, nurses from a low-performing hospital indicated that public reporting had ‘forced us to look at it [the data], to compare it. Before, it just sat there, now it drives us to do better’ and that ‘knowing that the public is aware of the scores makes us more energised to do better’. 75 The findings suggest that media scrutiny of the data put pressure on staff to respond to them, as one administrator noted: ‘I don’t think anyone believed it was going to be public until the newspaper article – that’s when people gasped!’. 75 Media scrutiny was perhaps felt more strongly for lower-performing hospitals, as interviewees felt that the local media typically tended to focus on ‘why the numbers are low and rarely on why they are high’. 75 These findings resonate with public disclosure theory, discussed in the previous chapter, which hypothesises that the media act to reinforce the shaming mechanism of public reporting and prompt the desired response.

A further issue to emerge from this paper was that interviewees in 17 of the 29 organisations reported that the validity and reliability of performance data were challenged by frontline staff. This occurred in both high-performing and low-performing organisations, but occurred more frequently in the low-performing hospitals. Concern was expressed that the data did not fully capture the quality of care in the hospital, that comparisons between hospitals did not adequately reflect differences in case mix and that the data were too old to reflect current practice. The authors noted that high-performing hospitals saw these challenges as ‘learning opportunities’, although they do not fully explain what is meant by this. These findings support audit and feedback and benchmarking theories, namely that when performance feedback is inconsistent with a provider’s own estimation of their performance it may not be accepted.

The methodological limitations relating to the numerous sources of reporting bias may explain why this study tended to find more positive views of the impact of mandatory performance reporting than other studies. In terms of the theories under test, this study suggests that mandatory public reporting had served to place QI at the top of the agenda for providers and that increased resources had been directed to addressing quality issues. It suggests that providers were sceptical about the quality and validity of the data. It also indicates that mandatory public reporting ‘forced’ and ‘energised’ providers to address quality issues because of the media scrutiny they expected to receive or actually received.

Asprey et al.223

This study aimed to explore primary care providers’ views of and responses to feedback from the national GP patient survey in England. This is a national survey of a random sample of patients registered at all GP practices in England, which asks about their experiences of care provided by their practice. The survey includes items on the ease of getting through on the telephone, the helpfulness of receptionists and being able to see your preferred doctor, as well as ratings of how good patients felt their doctor and nurse were on a number of dimensions. The survey is run by the CQC and is, essentially, mandatory. The results from the survey are publicly reported on a special GP patient survey website hosted by the CQC, where patients can compare the results of their own practice with those of two others of their choice. In addition, the results of some elements of the survey are used in calculating GP practice QOF payments. As such, the GP patient survey is mandatory and publicly reported, and has financial rewards and sanctions attached to it.

The authors selected four PCTs to represent different geographical regions across England, and within these four areas they selected the five highest-scoring and five lowest-scoring practices on the GP patient survey item ‘ease of obtaining an appointment with a doctor’,223 an item that contributed to the practices’ QOF scores. Their aim was to recruit one high- and one low-scoring practice from each of the four areas; the first practice to agree to take part within each sampling strata was included in the study. Ten GP practices were recruited, four with high scores and six with low scores, which included two single-handed practices. In each practice, two GPs (except in the single-handed practices), one practice manager and one practice nurse were interviewed, giving a total of 37 interviews.

The authors found that participants were sceptical about the credibility of the survey findings for a number of reasons. They were concerned that respondents were unlikely to be representative of the practice population because the ‘vocal minority’ with negative views were more likely to reply, while other groups, such as older people, working people and people with mental health problems, were less likely to reply. They also felt that the items in the questionnaire did not necessarily represent what constituted ‘good’ care, as not all patients valued them in the same way, or it was unclear that a high score on an item reflected a good experience; for example, a high score on waiting times might indicate that patients were being rushed through appointments in order to shorten waiting times for patients. Participants also felt that the lack of adjustment for case mix also reduced the validity, and thus the utility, of the feedback for improving patient care.

The authors report that the ‘most emotive’ responses provided by interview participants related to their suspicion that the surveys were driven by ‘political motives’. One GP from a low-scoring practice described the ways in which the questions were asked as ‘stacked against us and I think most GPs have a cynical view about that’. 223 A practice nurse from a low-scoring practice also felt that ‘this is a cost cutting exercise and little to do with a real commitment to patient satisfaction or to help those in primary care deliver a better service’. 223 In other words, low-scoring practices in particular felt that the survey was a politically motivated attempt to cut costs rather than improve patient care. Such feelings sometimes led participants to reject and ignore the survey results, as one GP from a low-scoring practice explained:

I’m totally cynical about the government’s motivation and this is just part of that . . . So if they think they’ve got me over a barrel, forget it, because they haven’t. And I can just happily carry on and ignore this survey.

GP from a low-scoring practice223

One GP from a low-scoring practice commented that although the financial element of the QOF was important – ‘we like to maintain a high income, of course we do’ – financial concerns were not ‘paramount’;223 rather, the practice was more concerned about the ‘shame factor . . . information is shared so much, you don’t want to see yourself . . . on a bar chart at the bottom of the pile’. 223

The suspicion that political motives were behind the survey was not limited to low-scoring practices; those in high-scoring practices also felt that survey was being conducted for political ends to make GPs work harder but with few clinical gains. One high-scoring GP commented that the GP patient survey was:

. . . a way of softening up primary care for extended hours, by showing there was a demand out there for it . . . I don’t think there’s going to be huge clinical gains from doing that. 223

Furthermore, the questions in the survey were also perceived to measure what mattered to politicians, to the exclusion of other important aspects of care. As one GP from a low-scoring practice expressed:

It’s a bit of tail and dog isn’t it? . . . Because it has been measured, is it necessarily important? It’s important but is it as important as some of the things that haven’t been measured?223

Some participants also felt that the items in the questionnaire driven by government rhetoric, such as rapid access, had unrealistically raised patient expectations about what they could expect from general practice, but, at the same time, the government had not provided additional funding to enable practices to deliver those promises. Finally, some practices had acted on the survey findings and made changes, such as extending their opening hours, but these had not been reflected in their scores, which led to practices feeling discouraged and thinking:

Well why? What else can we do?

Practice manager from a low-scoring practice223

In terms of the theory under test, these findings suggest that this government-initiated, mandatory public reporting programme was perceived to be driven by political motives rather than a desire to improve patient care. Consequently, primary care providers perceived that the items in the programme reflected government definitions of what constituted ‘good’ patient care and measured what mattered to the government, rather than to primary care practices themselves. In turn, practices were sceptical about the credibility of the data as an aid to improving patient care. This feeling was further reinforced when, even after attempts to respond to the survey by making changes to patient care, these improvements were not reflected in the scores achieved on subsequent surveys, further challenging the credibility of the survey itself.

Pham et al.228

This paper explores provider responses to a range of performance reporting initiatives and pays particular attention to two initiatives. The first is the reporting required by JCAHO, an organisation responsible for the accreditation and regulation of hospitals in the USA; public reporting is a condition of accreditation and thus is mandatory. They also explored the Hospital Quality Initiative, which is a voluntary system launched by the CMS and is reported on the Hospital Compare website. Following the reporting system’s introduction, participation in it was poor until a state law was passed that non-participating hospitals would not receive a 0.4% annual payment update. The authors also examined providers’ views on other forms of reporting systems; as such, the paper provides evidence on providers to the different drivers behind a range of reporting systems, including a contrast between mandatory, voluntary and clinician-led public reporting systems.

The paper drew on data collected as part of the Community Tracking Study. The Community Tracking Study is a longitudinal study based on site visits and surveys of health-care purchasers, insurers and providers, and is focused on tracking changes in the accessibility, cost and quality of health care. The data selected for the study were collected in 2004–5 and consisted of 111 interviews with five hospital association leaders, representatives from the JCAHO and CMS and six state reporting programmes, 21 chief effective officers, 21 vice presidents of nursing, 30 quality officers and 26 clinical directors from 2–4 of the largest hospitals or hospital systems in 12 health-care markets across the USA. The data were collected using semistructured interviews that explored specific reporting programmes and their perceived impact on the hospital’s organisational culture around QI, priorities, budget, data collection and review activities, feedback and accountability mechanisms. Clinical directors were also asked about their use of 11 QI tools targeted at chronic heart failure, as both the CMS and the JCAHO reporting systems included these.

The authors found that respondents mentioned involvement in 38 different reporting programmes, with each hospital participating in a mean of 3.3 programmes. These programmes varied along a number of axes, which correspond closely to the contextual factors identified within the public disclosure, audit and feedback and benchmarking theories. These included:

• ‘Sponsorship’, or who initiated the reporting system: by purchaser, regulator, private insurer, professional groups or other private organisations.

• Data type: hospitals submit primary data for public reporting (JCAHO), primary data for private benchmarking (e.g. hospital consortia) or secondary data (e.g. insurance claims or patient surveys).

• Mandatory versus voluntary: although most programmes are voluntary, this is influenced by the nature of incentives attached to the programme – incentives explicitly tied to participation in the programme perceived as rewards (e.g. pay for performance) or punitive (e.g. loss of accreditation).

• QI support: whether or not programmes provide prescriptive information to guide hospitals’ QI activities.

• Inclusion of clinical outcome measures.

Senior hospital leaders perceived that key drivers, such as linkages to payment, JCAHO accreditation and peer pressure from public benchmarking, had raised the priority given to quality measurement and improvement by hospital leaders. This was manifest in several ways, including the inclusion of QI priorities in strategic planning, boards and senior management taking more responsibility for the formal review of performance data and associated improvement strategies and performance-related pay for chief executives. Respondents also believed that CMS and JCAHO reporting programmes had also positively changed doctors’ attitudes towards quality measurement and improvement. The accreditation and financial consequences of reporting programmes could be used as leverage by quality officers in their dealings with doctors. However, the respondents also felt that these programmes artificially focused on a limited number of indicators, which has directed both attention and resources away from other important clinical areas.

As a result of their mandatory requirements, participants felt that the drive to participate in CMS and JCAHO reporting was of a ‘push’ nature. Hospitals did direct resources at the reported clinical areas but ‘without taking standardised approaches to improving performance’. 228 Motivation to be involved in other reporting initiatives was described more in terms of a ‘pull’ because they offered support by specifying changes in care processes. These forms of reporting were seen as attractive because they ‘don’t leave hospitals flailing about trying to identify evidence-based interventions on their own’228 and because they encouraged a culture of continuous QI. These sorts of programmes were most likely to be those initiated by QI organisations, state QI organisations and professional organisations.

Hospital respondents were divided on whether or not reporting had a significant impact on specific process changes to improve the quality of care. Those who had participated in QI programmes prior to being involved in mandatory public report programmes felt that public reporting had little impact on their QI interventions, as these were largely driven internally. The respondents were also divided regarding whether or not reporting had a ‘spill over’ effect on improving quality for the non-reporting conditions; some believed that QI was limited to the targeted conditions, while others felt that it had raised frontline staff’s ability and eagerness to identify and address problems in non-reported areas too.

In their conclusions, the authors argued that national programmes that mandate participation through regulatory or financial reward mechanisms ‘can influence nearly all hospitals and garner attention from those that would otherwise not prioritise QI highly’,228 whereas voluntary programmes, especially those which also provide ongoing support to implement QI initiatives, ‘help focus priorities at hospitals that are eager to take on the more challenging goals of QI’. 228 Thus, national programmes with mandatory reporting and regulatory or payment consequences have increased hospital leaders’ and frontline staff’s attention to quality and directed resources towards QI. However, at the same time, such public reporting initiatives may ‘artificially narrow the scope of QI in which hospitals might otherwise engage, especially for those with long institutional histories of QI’. 228 In terms of the theories under test, these findings suggest that, for those not already involved in QI activities, mandatory public reporting services serves to raise awareness of quality of care issues and direct resources towards issues raised by such reporting systems. However, similar to the findings of Mannion et al. ,18 these can lead to tunnel vision, where other important clinical areas do not receive the same attention. For those hospitals already involved in QI activities, mandatory reporting systems were perceived to have little additional impact on these activities. Furthermore, hospitals were attracted to systems initiated or run by QI or professional organisations because they offered support to providers to take the further steps necessary to identify the source of the problem and implement QI activities.

Davies98

As described in Chapter 4, this study examined the responses of US providers based in California to both externally produced, publicly reported data and internally produced, privately fed back data, with a specific focus on cardiology. They explored providers’ views of a publicly reported data system, such as the CHOP, which publicly reports data on 30-day mortality for acute myocardial infarctions and also confidential data systems designed for internal use, for example the national register for myocardial infarction by Genetech and Health Care Financing Authority, sponsored peer-review organisations in the state of California. As such, the study provides a useful contrast between provider responses to data that is produced and fed back in different ways.

This was a multiple case study of six hospitals purposively selected because they were ‘high performers’, as the author expected that he would be more likely to find examples of QI in those hospitals (with the assumption that the hospitals had become ‘high performers’ as a result of QI activities). However, a range of different hospitals was in the sample, including two academic medical centres, one health maintenance organisation, two private but not-for-profit medical centres and one public provider ‘safety net’ hospital. The author conducted 35 interviews with 31 individuals lasting between 54 and 90 minutes. Interviews were conducted with key informants in each setting, including the chief executive, senior clinicians with management responsibilities, senior quality managers, the chief of cardiology, a senior nurse manager, and two or three frontline staff within cardiology. As such, this study provides insight into the views of frontline as well as managerial staff.

Participants questioned the validity and reliability of publicly reported data because they perceived that the reporting systems did not take case mix into account. Participants also expressed concerns about inconsistent coding practices and the poor quality of administrative data; differences in performance were seen as artefacts of the data collection process rather than reflecting real differences in performance. This led to efforts to reform the data collection process. Participants also felt that issues that were measured attracted more attention than was warranted, to the detriment of other, unmeasured services. Quotations from respondents suggested that a reason this was perceived negatively by some staff might have been because it drove efforts to address the problem that were not necessarily clinically appropriate: ‘It really fires people up to meet the task, rather than for clinically appropriate reasons’. Participants also reported instances in which clinicians challenged the pressure exerted on them by a purchasing group following the feedback of performance data, because they felt that the priorities being imposed on them by the purchasing group were counterproductive.

The author’s analysis of his data suggested that a provider’s action in response to publicly reported comparative performance data was most likely when external data indicated they were performing poorly: ‘being an outlier does motivate performance. There’s no doubt about that’. 98 A response to these data was less likely if they were a ‘middle ranker’ because some providers were willing to tolerate being ‘middle of the pack’ and did not feel that they had an incentive to improve. However, in some instances being ‘middle of the rank’ was not acceptable when providers were ‘striving to be the best’, while, for others, a provider taking action depended on ‘our own perception as to whether [the data] were an accurate reflection of what we think is happening’. 98 The paper also noted that publicly reported data were seen as only one source of information about the quality of care, with their own assessments and the views of peers and coworkers being as important. For example, one senior clinician noted, ‘It’s the opinion of peers that matter more than anything else about quality’ and another senior clinician explained that comparative performance data ‘merely reinforces already held opinions just based on other factors, you know, day to day experience’. 98

The study also sheds light on the relative roles of externally produced and internally collected data on the implementation of QI initiatives in response to these data. The findings suggest that publicly reported data focused attention and acted as a ‘kick start’ to the QI process; as a senior nurse manager explained, ‘external data are the start of the process . . . that really gets the ball rolling in terms of an [internal CQI] investigation’. 98 However, external data were not able to identify the cause of the problem and thus could not help to identify a solution because these data were not timely and thus lacked relevance to current care – ‘If you’re not doing it yourself and reacting to it immediately, there’s a whole time lag and opportunities for improvement that you’ve missed’ – and did not provide sufficient detail: ‘you just don’t get the details [from the external data].’98 It was the internally collected data that provided the necessary clinical detail to identify the cause of the problem and how to remedy it: ‘it’s the in-house data [that] drives us more than the outside data. I think it’s also better data and it’s more focused; it has many more elements to it’ and ‘our best successes [in using data to improve quality] were our very own internal ones.’98

Therefore, external data were useful to identify what needed to be looked at, but internal, clinically owned process-based data were needed to identify the cause of the problem and how it could be dealt with. To support this, providers also needed practical resources for the analysis, presentation and interpretation of such data and a culture that valued and supported continuous QI processes: ‘we have wonderful motivated people but if we didn’t have the resources to do this, we couldn’t. There’s not only people committed to excellence, there’s resources committed to excellence’. 98 If good local data and supportive resources were absent, little QI was seen: ‘We don’t do it [benchmarking] and we don’t have the resources to do it . . . really, no way, since we don’t have ongoing databases’. 98

Thus, in terms of the theories under test, this paper suggests that when publicly reported data highlighted issues that the clinicians themselves or their peers also felt were a problem, they served to amplify or kickstart an intrinsic desire to improve. It also indicates that although publicly reported data might focus attention on areas that need to be changed, it was only through analysing internally collected process data that providers could understand the cause of the problem and identify ways to address it. To be able to do this, providers needed practical resources and management support for internal data collection and the analysis and interpretation of those data.

Greer199

All quotations from this study are reproduced from Greer,199 with permission from the Commonwealth Fund.

Greer’s report of how the WCHQ was set up is overwhelmingly positive about its impact; this may be due, in part, to its reliance on data collected from the ‘enthusiasts’ who led the collaborative. The author recognises that, as the collaborative was a ‘pioneering effort’, it benefited from the Hawthorne effect. The report is based on 31 interviews with board members and medical practice executives involved in the WCHQ. They included eight CEOs, 10 chief medical officers and five executives responsible for the quality of care in their organisation. As such, it does not capture the perspectives of staff on the ground or their responses to this initiative. Nonetheless, the study provides some useful ideas about the mechanisms through which clinical engagement and trust in a public reporting system was achieved.

As Greer describes, the WCHQ was founded in 2003 by chief executives of several large multispecialty practices and their partner hospitals. The collaborative was brought together with the purpose of promoting QI among member organisations by (1) developing performance indicators, (2) openly sharing provider performance through public reporting and (3) identifying and sharing best practices to improve the performance of all members of the collaborative. One of Greer’s interviewees, a doctor by background who was the CEO of a doctor-led health-care organisation, noted:

What we sensed was that there were reports being published by people who had some knowledge, but perhaps not full knowledge of healthcare and its delivery . . . The reason for our meeting was: it looks like people are going to start writing reports, publishing reports on medical performance and that will be followed by dictating type of care and how care should be delivered. Shouldn’t we, the people responsible for care delivery, shouldn’t we be involved in the process?

p. 2199

In other words, the WCHQ was set up in order to allow clinicians to retain some control over the ways in which public reporting was designed and implemented. The underlying premise of this process was perceived to be one of mutual learning, as an antidote to the competitive climate providers perceived they were expected to work in. One of Greer’s respondents, a founding member of the WCHQ, noted:

I think in many ways WCHQ became an oasis from this highly competitive environment . . . a safe harbour where we are not talking about market dominance and control. We are talking about quality. 199

A chief medical officer who joined the initiative later, when the opportunity for membership was opened up to a wider group of clinic groups and practices, also remarked:

[I enjoy] the sense of collaboration, and what is kind of fascinating, is that the discussion – of how we are doing, how we are doing relative to each other, how we can do better – constantly brings you back to your primary purpose and that is the patient you are taking care of. 199

p. 13199

Here we see that the primary motivation for signing up to the WCHQ was its focus on how to improve patient care, rather than on improving profits through market dominance.

The mechanisms of change reported by interviewees were that (1) doctors were intrinsically motivated to do a good job, and receiving feedback that indicated their performance was poor prompted them to take steps to improve; and (2) doctors can identify what needs to be improved, and how, by sharing and learning from the best practices of other providers. These mechanisms resonate with both audit and feedback and collaborative benchmarking theories. Greer’s interviewees noted:

I think one of the constructs that the Collaborative built on is that physicians want to do a good job . . . By providing information about how physicians perform you can influence physicians behaviour . . . They are driven to change things when their performance does not look good.

p. 14199

WCHQ [provides] the actual benchmarking data for looking at where you are at, how to improve, and building on those connections with other organisations that are similar to you; where you can say ‘our numbers are not good here, how did yours get better? What can we learn from how you are doing it?

p. 17199

Greer argued that the WCHQ had a high level of clinical engagement, as evidenced by its growing numbers, with 50% of Wisconsin primary care physicians being members of the collaborative in 2008. Greer attributed this high degree of clinical engagement to the fact that the collaborative was clinician led, rather than led by government or an insurance company. This resulted in the development of indicators and methods of attributing those indicators that were perceived as valid reflections of clinical care. The indicators were accepted because there was a shared perception among the collaborative that the motivation behind the production of the indicators was that of improving patient care, rather for competition or external regulation. Participants in the WCHQ indicated that they accepted the data collated by the WCHQ rather than dismissed them because they were developed by clinicians and as such were seen as an accurate reflection of clinical care. The fact that the collaborative developed the indicators made it much more difficult for its members to then dismiss their validity. As one of Greer’s interviewees reports:

. . . we promised each other we would report our data, we would not fudge it, we would have it verified, we would make it public and we would not walk away from whatever we found. 199

In contrast, Greer’s interviewees were sceptical of the motivation behind insurer, claims-based indicators that were perceived as inaccurate and designed for competitive advantage rather than claims purposes. Indicators developed by organisations that were representatives of the government, such as the Quality Improvement Organisation or the JCAHO, were seen as motivated by external regulation and were perceived to have only a short-term impact on QI initiatives, as one of Greer’s interviewees observed:

JCAHO is big brother coming in and the response is usually: ‘Oh there is a JCAHO visit. We will get ready for the JCAHO visit that we will have to pass.’ Then they leave and they don’t come back for 5 years and until the next one comes along nobody thinks about them.

p. 17199

In summary, Greer’s case study of the set-up of the WCHQ suggests that clinicians were engaged in setting up the collaborative because they supported its primary motive of improving the quality of care by comparing performance and sharing best practices. The perceived mechanisms of change were those evoked by audit and feedback and benchmarking theories; that is, clinicians have an intrinsic motivation to do a good job and will be prompted to improve if their performance is poorer than they would like or poorer than that of their peers. This engagement, in turn, led to the development of indicators that reflected the information that clinicians needed to improve patient care. As clinicians were involved in the design of the indicators, it was then difficult for them to dismiss the data as invalid. This suggests that clinicians are more likely to trust data that are publicly reported if they have a role in the choosing the indicators and the means through which they are risk adjusted and reported.

Lamb et al.229

This paper reports on a longitudinal cohort study and survey of members of the collaborative to explore whether or not the WCHQ led to improvements in the quality of care provided. For each publicly reported indicator, the authors examined whether or not there was an improvement in mean performance for the collaborative as a whole over time and examined trends over time. Clinician groups were ordered by their rank in first year of reporting and this was correlated with their rate of improvement. They conducted a postal survey of clinic groups to examine whether or not QI projects were undertaken specifically in response to reporting. Finally, for four indicators for which comparative data were available, they compared improvement in care for patients in the WCHQ with (1) patients in Wisconsin who were not part of the collaborative, (2) patients in Iowa and South Dakota, where there was no public reporting, and (3) residents in the remainder of the USA. Their analysis is based on responses from 17 out of the 20 doctor groups who were part of the collaborative, representing 409 out of the 582 clinics.

The authors found that for the WCHQ as a whole, for each reported measure, the indicators for clinics that were part of the collaborative improved as a whole over time, but there was wide variation in the amount of improvement, from 1.2 for low-density lipoprotein control to 17.3 for monitoring kidney function. Not all of these improvements were statistically significant (of 13 measures, 7 showed a statistically significant improvement over time). Groups that were initially low performers improved at a greater rate than those that were initially high performers. The survey found that 15 out of 16 groups reported formally giving priority to at least one QI measure in response to reporting. Nine out of 16 groups indicated they always or nearly always set priorities in response to reporting, while 6 out of 16 indicated that they sometimes did so. The mean number of QI interventions initiated by members of the collaborative increased over time. On three of the four measures for which comparative data were available, there was a trend for patients in the WCHQ to receive better care than patients in Wisconsin who were not part of the collaborative, in Iowa and Dakota and the USA as a whole, but this was not statistically significant. The measure that WCHQ members did not perform better on was not publicly reported by the WCHQ.

It is not possible to attribute the improvements found in this study solely to public reporting, as improvements in the quality of care were also found in other locations where public reporting did not occur. As the authors acknowledge, clinicians in the WCHQ were volunteers and more likely to be enthusiastic about public reporting and their patients were more affluent. Nonetheless, these findings suggest that members of the WCHQ did take steps to improve patient care in response to public reporting. They also indicate that providers’ responses were variable, suggesting that no group was able to respond to all the reported measures and clinician groups prioritised a limited number of indicators to focus on. Of note here is the finding that those initially identified as low performers improved at a faster rate than average or high performers. The authors hypothesise that ‘public reporting creates a milieu in which parties compete for external recognition and strive to avoid the negative aspect of publicly being identified at the bottom of the list’.

Smith et al.230

This paper reports on survey of doctor groups who were members of the WCHQ to explore whether or not they had initiated any improvements in diabetes care in response to the collaborative’s public reporting. The authors invited 21 doctor groups, representing 582 clinics, to participate, of which 17 groups representing 409 clinics agreed to participate. They received group responses from 231 clinics and individual responses from 178 clinics. They carried out two surveys: one for the doctor group as a whole and one for each clinic. They asked the clinics if they had implemented any of 55 QI initiatives, of which 22 were diabetes related, in each year between 2003 and 2008. The groups were asked to identify any year in which a metric was chosen to be a focus for QI and to indicate if this was in response to the public reporting. It is unclear how this was quantified, for example whether it was a simple yes/no answer. They used these data to generate an indication of whether, for each year, the doctor group formally adopted a focus on one or more diabetes metrics in response to the collaborative’s reporting, whether they adopted a focus but not in response to public reporting or whether they did not adopt a focus. Given that this survey is based on self-report, there is a risk of reporting bias in terms of both social desirability and recall bias.

They found that the implementation of diabetes QI initiatives increased between 2003 and 2008. Clinics in groups that focused on diabetes metrics in response to public reporting were more likely to implement both single and multiple initiatives than groups that did not formally adopt a focus on diabetes. In this group, a factor that appeared to influence whether clinics adopted single or multiple interventions was their experience in diabetes QI; clinics with less experience were more likely to implement single interventions. Clinics in groups that focused on diabetes metrics but not in response to public reporting were more likely to implement multiple interventions. The authors asked quality directors from four doctor groups why clinics chose to operate multiple or single interventions in a given year. Their responses indicated that clinics implementing single interventions were in the early stages of QI. The authors of the paper report the views of one quality director:

One quality director commented that, with the group’s participation in the collaborative, its doctors were seeing standard comparative reports for the first time. The director said that these reports motivated clinicians to ‘do something’, but ‘they just didn’t have the bandwidth to do more’.

The authors also noted that clinics implementing multiple interventions sometimes did so in response to public reporting, but they were also often involved in externally sponsored QI projects. The authors report:

In one case, clinics had implemented a single physician-directed intervention as a ‘first step’ but the quality director of that group noted that ‘we needed broader organisational change to sustain improvement’. 230

Despite its methodological flaws, this study’s findings suggest the public reporting of performance data alone does not stimulate sustainable organisational change. The public reporting of performance data is more likely to lead to sustainable QI if it occurs alongside other, large, externally supported QI initiatives. Single QI interventions implemented in isolation may not lead to sustainable QI. The study also suggests that QI occurs incrementally, and that the organisations may achieve sustainable improvement through a process of trial and error; single interventions may be a first step along this pathway.

Vingerhoets et al.231

This study assessed whether or not the structured, individualised, benchmarked feedback of patient experiences to GPs in the Netherlands resulted in improvements to patients’ experiences of care. This study was conducted before national surveys of patient experiences were implemented in the Netherlands and therefore GPs in this study were unlikely to have had the same exposure to patient experience surveys as they do now. The sampling frame for the study was a sample of 700 GPs in the Netherlands stratified by level of urbanisation. From this sample, 60 GPs from 43 practices were recruited to the study and each was asked to recruit two cohorts of 100 consecutive attending patients, one at baseline and one 15 months later. Each cohort of patients was asked to complete a previously validated patient experience questionnaire covering nine dimensions of care. After matching for practice size, each practice was randomly allocated to a control arm and an intervention arm. The control arm practices received no feedback from the questionnaire. GPs in the intervention arm practices received a written 15-page report detailing the patient experience scores provided by their patient on each dimension, total scores and also reference figures for all GPs. The report also contained an abstract for a review of the determinant of patients’ evaluations of their care and a manual that explained how GPs might use the results of the survey. GPs in the intervention arm were also sent a questionnaire enquiring about any changes they had made to their own behaviour or the organisation of care. The results were analysed using multiple regression to examine if there were any statistically significant differences in patient experience scores between the two arms of the trial.

The authors found that, after controlling for the effect of baseline patient experience scores, patients’ evaluations of continuity of care and medical care were statistically significantly less positive in the intervention arm. There were no statistically significant differences in the other seven dimensions of patient experience, despite GPs reporting that they had made changes to their own behaviour or the organisation of care. The authors hypothesise that the lack of effect of their intervention may have been because the follow-up period was too short for any improvements to register, that GPs may have been too busy to implement changes and that the general shortage of GPs in the Netherlands may mean that GPs felt less pressure to respond to feedback. They argue that the intervention may have been more effective if ‘it is embedded in an educational programme or QI activity related to a specific clinical topic or group’. 231 They also suggest that the feedback may have functioned as a means of identifying specific topics for QI that needed to be explored in more detail before implementing specific QI activities.

This study was conducted in a different context from that experienced by current GPs, who had considerably more exposure to feedback from patient experience questionnaires. However, it does suggest that a ‘one-off’ feedback of patient experience data to GPs, without any public reporting or financial incentives attached to it, does not lead to improvements in patients’ experiences of care.

Elliot et al.222

This study examined the feedback and public reporting of the Hospital CAHPS survey, which measures patient experiences and is publicly reported on a quarterly basis. The scheme was initially introduced by the CMS on a voluntary basis, and in 2008, 55% of eligible hospitals in the USA were involved in the scheme. However, CMS implemented a penalty of a reduction of 2% in the annual payment to hospitals if they failed to collect data from 2007 and failed to report it from 2009. By 2009, the percentage of hospitals participating in the scheme rose to 80%. Thus, the Hospital CAHPS scheme had financial incentives linked to hospital participation. It is perhaps also worth noting here that two of the authors who conducted the study worked for CMS, which was responsible for introducing the scheme.

The data from the survey are reported as the proportion of responses in the most positive categories (i.e. ‘definitely yes’, ‘yes’ or ‘always’) across nine domains measuring nurse communication, doctor communication, responsiveness of hospital staff, pain management, communication about medicines, cleanliness of hospital environment, quietness of hospital environment and discharge information. The authors adjusted these data for survey mode and patient characteristics. They compared scores on the survey between data published in March 2008 and March 2009 for 2774 hospitals in the USA that publicly reported data for both time periods to examine whether or not there had been any improvements in patient experience over time.

The authors found statistically significant but very small changes in patient experience scores between March 2008 and March 2009 on all nine domains except doctor communication. Most changes were < 1 percentage point difference in the scores in the top category between the two time periods. In their discussion the authors describe these changes as ‘modest but meaningful improvements’ and argue that their findings provide evidence that ‘healthcare entities are able to use CAHPS feedback to improve patient experience’. 222 However, it is difficult to know the real significance of the changes for patients from these data and the study did not explore whether or how providers did use the data to improve care or what changes, if any, were made. Without a control group, it is difficult to know if such modest improvements in patient experience would also have occurred in the absence of any feedback. Furthermore, the time period over which the study was conducted was short and may not have captured the impact of any changes. In terms of the theory under test, this study suggests that, in the short term, the feedback and public reporting of patient experience data to providers leads to only very small gains in some domains of patient experience but not in doctor–patient communication. We now look at studies that have explored providers’ views and attitudes towards patients experience data, and their reports of whether or not and how they used this feedback to initiate QI activities.

Barr et al.232

This study explored the impact of mandatory public reporting of patient experience on providers’ QI activities. The study focused on the state-wide mandatory reporting of patient experience in Rhode Island, which was initially fed back privately to providers in 2000, and from 2001 onwards was fed back publicly. The 56-item survey was carried out annually on a random sample of patients discharged from each state licensed hospital in Rhode Island. The survey covered nine domains of patient experience: nursing care, doctor care, treatment results, patient education (including discharge information), comfort/cleanliness, admitting, other staff courtesy, food service and overall satisfaction/loyalty. The survey findings were publicly reported as the hospital’s score on each domain expressed as whether it was the same as, above or below the national average. Hospitals also received survey item data (expressed as percentage scores).

The sampling frame comprised four key executives in each of the 11 hospitals (CEO, medical director, nurse executive and patient satisfaction co-ordinator). Of the 52 positions identified, 42 people agreed to take part in the study (13 CEOs, eight medical directors, eight nurse directors and 13 patient satisfaction co-ordinators). The authors interviewed participants 1 year after the initial release of the first public report, either face to face or by telephone, and explored what QI activities had taken place in response to the patient experience survey.

The authors found that every hospital reported at least two QI initiatives within the domains reported in the survey. The most commonly reported areas in which improvement initiatives had taken place were admitting (nine hospitals), patient education (nine hospitals), nursing care (eight hospitals), treatment results (eight hospitals) and food service (eight hospitals). Less common areas were other staff courtesy (six hospitals), doctor care (five hospitals) and comfort cleanliness (four hospitals). Hospitals also reported being involved in other, broader QI initiatives, which could also have impacted on the domains reported in the patient satisfaction survey. However, the authors did not explore how the hospital’s own score on these domains related to the QI efforts. The authors also found that although most hospitals had a decentralised approach for initiating QI initiatives, the reporting of the patient survey results was centralised and focused on senior management. Participants explained that they used the patient experience survey results to prioritise areas for improvement. They also noted that they had the greatest support for QI activities from the board and senior management and the least support from medical staff. Participants cited ‘widespread support for QI, a culture and leadership fostering QI and a team approach’232 as being important for successful QI activities.

This paper was a small-scale interview study in one location of the USA. The authors relied on hospital leaders’ self-report of whether or not QI activities had taken place, which might have been subject to recall or social desirability bias. However, this paper provides another layer of evidence to suggest that some areas of patient experience were more likely than others to be subjected to QI efforts. It also suggests that senior managers played an important role in supporting QI activities.

Boyer et al.233

The study reports on providers’ views and responses to a locally developed and implemented patient experience questionnaire for inpatients in a 2220-bed teaching hospital in France. The patient experience survey had been carried out yearly since 1998 and was ongoing at the time the paper was written (2006). It produced patient experience scores on a number of dimensions (medical information, relation with nurses, relations with doctors, living arrangements and health-care management) for the hospital as a whole and for each clinical department. The authors surveyed staff members in the hospital using a 26-item questionnaire which examined if staff had been informed about the overall hospital results, the results for their ward, how they were informed about the results, if the results were discussed, whether or not any action plans were developed as a result of the survey and staff attitudes towards patient experience surveys. The authors sent 502 questionnaires out to staff in the hospital, although they did not report what their sampling frame was or how it was determined. Of these, 261 (52%) of the questionnaires were returned.

The authors found that the specific results for the ward were less well known than the overall hospital results, with 60% of respondents indicating that they were aware of the specific ward results and 70% indicating that they were aware of the overall hospital results. However, 87% of staff indicated that they were more interested in the ward-specific results, compared with 13% who indicated that they were more interested in the overall hospital results. Respondents placed a higher value on open-ended comments than on standardised patient experience scores. Forty per cent of respondents indicated that the results of the patient satisfaction survey were discussed in staff meetings, 40% indicated that actions were taken to solve problems and 40% indicated that the survey had led to modifications to professional behaviour. In their conclusions, the authors argue that one explanation for the insufficient use of the survey may be explained by ‘a lack of quality management culture’ and a lack of ‘discussion of the results within the department’. 233

This is a poor-quality study; the sampling frame for the study was not clear and the sample size and sample of participants who responded was small. The survey was conducted in one hospital and as such its findings may not be generalisable. Nonetheless, the study provides some indication that ward-level data were perceived as more useful than overall hospital performance. It also suggests that patient experience surveys are not a panacea to QI but that their use depends on the extent to which the data are disseminated within the hospital and whether or not they are discussed in ward meetings. It also implicates the importance of a broader, supportive hospital culture in facilitating the use of patient experience surveys to improve patient care.

Geissler et al.234

This study explored the motivators and barriers to doctors’ use of patient experience data. The authors were particularly interested in doctors’ views of the patient experience surveys distributed confidentially as part of the activities of the MHQP collaborative. This survey was conducted and fed back to clinicians every 2 years. However, they also explored doctors’ views of other forms of patient experience data obtained from other sources. They developed a conceptual model to guide their investigation. They theorised that the degree to which doctor practices were engaged in initiatives to improve patient experience influenced the extent to which they make improvements in patient experiences. The degree to which doctor practices engaged in initiatives to improve patient experiences was influenced by organisational characteristics, such as culture, incentives, IT management and leadership, and by the characteristics of patient experience reports themselves, in terms of how they were disseminated, ease of use, timeliness and level at which the report was fed back. Here we focus on the findings relating to the nature of the reports themselves.

To test their model, Geissler et al. 234 conducted 30-minute semistructured interviews with a sample of doctor groups in Massachusetts. The sampling frame was the 2007 MHQP state-wide doctor directory, with at least three doctors providing care to members of at least one of the five largest commercial health plans in Massachusetts, resulting in 117 doctor groups who were invited to participate. They interviewed leaders from 72 groups, giving a 62% response rate.

Their study did not specifically compare doctors’ views of different types of patient experience surveys, but their findings do provide some insight into how the characteristics of the reports and the way they were fed back served to support or constrain the use of patient experience surveys in improving patient care. Participants indicated that they valued free-text responses from patients and sent positive responses to staff to boost morale, especially if individual staff were named. The negative ones were used to target particular departments or wards that were named in the feedback. This suggests that the free-text responses were valued because they allowed a more specific understanding of what was going well and what was not.

They also valued patient experience surveys, which provided support in interpreting and acting on the findings, such as those which provided a ‘priority list’ consisting of the ‘ten most important areas or things that you could address that would have the biggest impact on improving patient satisfaction’. 234 The timeliness of the data was also mentioned as important, with data provided on a frequent basis being seen as supporting efforts to improve care, and those with a large time-lag between data collecting and analysis being seen as less useful, as the following quotations from respondents illustrate:

This data has been more useful . . . because it’s more timely. The data is available to us on an ongoing basis; we get it literally every day . . . so . . . the feedback is . . . more current.

I will get the MHQP and it’s on stuff that happened a year and a half ago. That’s very hard to go out to . . . practices and say we have got a . . . problem . . . you’ve got to do something about it . . . they say ‘well that was a year and a half ago’.

Doctors also valued reports that provided data at the level of the individual clinician and were benchmarked against other groups’ performance, so that they could compare their performance with that of others.

In terms of the theories under test, this study suggests that support with data interpretation, the level at which the data were reported and the timeliness with which the data were reported were seen by doctors as important in either constraining or supporting their efforts to improve patient experience.

Reeves and Seccombe235

This study aimed to explore providers’ attitudes towards patient experience surveys and understand if and how they were being used in NHS hospitals. At the time of this study, annual patient experience surveys were conducted in specific patient populations: inpatients, emergency departments, outpatients and young patients (aged 0–18 years). Twenty-seven hospitals were purposively sampled from 169 NHS trusts providing acute care; the sampling frame was organised according to the size of the trust and whether they were inside or outside London. The person listed in the Health Care Commission records as being the lead for patient surveys was contacted to check whether they were the lead and, if so, invited to take part in an interview. It is not clear if interviews were undertaken face-to-face or by telephone. The interviews focused on views and uses of patient surveys, but they were not tape recorded and only notes were taken. This study therefore focuses on the views of those who lead in the trust on patient experience surveys and as such may not represent the views of frontline staff. As notes were taken, it is possible that key issues were missed and that participants’ responses were filtered through the interviewer’s frame of reference, leaving open possibilities of misinterpretation, misunderstanding of what the interviewee meant and selective listening or remembering. The study was funded by the HSCIC.

Participants drew attention to the trade-off between the timeliness and robustness of different sources of patient experience data. Participants noted that comment cards and suggestion boxes offered immediate feedback, and comments written on questionnaires were seen as useful in gaining the attention of clinicians and often provided details of incidents of poor care. As one participant commented ‘Reading through the comments, even though our percentage scores are OK, you think “That shouldn’t have happened” ’. 235 However, patient experience surveys were seen as more robust: ‘Without a doubt, the national patient surveys are given the most weight. We have nothing else that is so sophisticated and would give us such useful data’. 235 There appeared to be a distinction drawn by participants between ‘soft’ information, such as comments or complaints, and ‘hard’ evidence such as clinical or routine data. Patient experience surveys appeared to be seen as more ‘robust’ than comments and complaints.

Participants also commented about the methods through which survey findings were disseminated, most commonly through the organisation’s intranet, newsletters, meetings where the contractors came into present findings and special events. In most organisations, the results were sent to senior staff who were expected to cascade the information down to junior staff, but participants reported that some groups of staff, such as doctors or more junior staff, were less likely to receive the results. They also commented on how difficult or easy it was to interpret those data; almost all participants felt that the Healthcare Commissions presentation of the published results were easy to interpret, especially the traffic light system, which shows whether the trust falls within the best 20% of trusts, the middle 60% or the bottom 20%. This helped trusts to ‘see quite clearly where you are and where you should be’. 235

However, when it came to acting on the findings from patient experience surveys, opinions were more varied. Some participants felt that feedback from patient experience surveys was not specific enough to be relevant to recipients, who, it was hoped, would act on the information. This was particularly seen as an issue for doctors, who participants felt were focused on their ‘sphere of influence’. As one participant noted, ‘The main criticism we have from doctors is “Make it specific to the area I work in and I will take notice of it” ’. 235 They also noted variation in clinicians’ ‘receptiveness’ to survey findings, with nurses perceived as being ‘easier to engage’ than doctors.

Almost all participants reported using patient survey results as the basis of action plans, and the authors give two examples of changes providers made. Both were in response to very specific issues highlighted by the survey: one was in response to the surveys highlighting problems with ‘noise at night’ that led to a range of efforts to reduce noise on the ward, and the other was variable information provided at discharge which led to changes in the way information was provided. Here, the surveys appeared to highlight problems with specific areas of care that were addressed. However, some participants also noted difficulties in formulating and then implementing action plans in response to the data. One participant commented that ‘Just giving people the results doesn’t mean they will take action. They need direction to make them do things and the frameworks to help them’. 235 Other participants commented that they found it difficult to identify the reasons behind their successes or failures, and had difficulty knowing how to address shortcomings. Policy documents, published at the time of the study, promoted the idea of spreading best practice across the NHS, in line with theories of collaborative benchmarking. However, participants appeared divided in their enthusiasm for this idea. Some were interested in learning how others had made improvements, while others, in the authors’ words, were ‘not particularly enthusiastic’ about identifying and learning from the best practices of others, although the exact form of their opposition is not reported.

In terms of the theories under test, this study indicates that providers were aware of the trade-off between timely and robust feedback and felt that both types needed to be integrated to provide a fuller picture of their performance. It also suggests that providers preferred feedback that was specific to their ward or ‘sphere of influence’ and that this was an important determinant of whether or not providers took action in response to this. However, it also suggests that providers needed support to identify the reasons behind their successes or failures and, in turn, to take steps to make improvements. This suggests that patient experience data do not always provide a clearer picture of the causes of good or poor care.

Boiko et al.236

This study explored primary care staff views and responses to the confidential feedback of a patient experience survey, similar to that used in the England GP patient survey. A random sample of 25 practices from Cornwall, Devon, Bristol, Bedfordshire, Cambridgeshire and North London agreed to take part in the study, and a random sample of their patients was mailed the patient experience survey. The practices received aggregate-level feedback for their practice, and each individual family doctor also received confidential feedback of their own scores on the patient experience survey. A purposively selected sample of 14 GP practices were then invited to take part in focus groups, which included 128 participants in all (40 GPs, 18 practice managers, 18 nurses, 20 receptionists, 13 administrators and 19 other staff members). The focus groups explored how practices had responded to the findings of the surveys, and also commented on two hypothetical situations in which some doctors in the practice received less favourable patient experience scores than other doctors.

Participants questioned whether or not patient experience surveys could adequately capture the ‘complex reality of healthcare interactions’ and contended that they focused on what was measurable to the exclusion of other important aspects of care. As one GP explained:

A lot of this data that’s collected in a measurable kind of way doesn’t really represent reality. There’s a kind of fixation on measurable outcomes but they don’t really tell us what’s going on. 236

Staff also drew attention to the trade-off between the increased relevance of local surveys that were less robust, versus the robustness of national surveys that were less specific to individual practitioners and did not include free-text comments. As one GP commented, ‘We want to see data tailored to individual practitioner because we all practice [sic] differently’. 236 Patient complaints were seen as more useful because they allowed practices to understand where problems may lie. As one administrator noted, ‘I think we learn a lot more from patients that write to us individually about complaints’. 236

Participants also reported a number of changes they had made to services in response to the survey, including modifying their facilities, appointment systems and providing staff training. The changes largely related to organisational aspects of service delivery and operational matters. However, the authors commented that, for most practices, changes were ‘rarely attributable directly to the survey feedback’;236 rather, the survey had provided a ‘nudge’ to implement changes they were already considering. Participants mentioned a number of difficulties in responding to issues highlighted by the patient survey, including not having the resources to acquiesce to perceived unrealistic patient expectations (e.g. patients wanting the surgery to be open at weekends), balancing the sometimes conflicting demands of different groups of patients (e.g. some patients wanted music in the waiting room, whereas others did not) and the working patterns of GPs making it difficult to always fulfil patients’ preferences. As one doctor summarised:

Would you like the surgery to be open on Saturday? Yeah. Would you like us to go 24 hours? Yeah. Are you going to pay more taxes to have it open on Saturday? No. 236

They also felt that, even though they had made changes to the organisational aspects of the delivery of care, these had not been always been reflected in improved scores on patient experience measures which, as one GP described, were ‘remarkably stubborn in terms of the change in perception by patients’. 236

In particular, they acknowledged that it was very difficult to tackle an individual doctor’s poor performance, especially when findings were fed back confidentially. It was only perhaps when these findings were shared more widely within the practice that change might occur. As one practice manager commented:

If the survey results are between (the survey providers) and the doctor . . . there’s absolutely no reason for them to change their ways is there? What is the motivation to change . . .? It is only when this information becomes available to . . . the practice that things could start to change. 236

However, this respondent was also unsure exactly who in the practice could be expected to put pressure on ‘poor performing’ GPs to change their behaviour. Teams acknowledged the difficulties of having an ‘unmanageable’ GP in the practice but most teams indicated that they would support a doctor who consistently received poor patient feedback through mentoring, peer support sessions and interventions by a partner or manager. They also recognised that some GPs may not be ‘a great communicator but they are great at doing something else’. 236 Finally, staff felt that there was little external support for making changes in response to patient experience surveys. One GP complained that surveys had come out but that there was:

very little support from anyone to say, right, this is how you can improve things that might help or we understand why you might be having problems . . . It has always been: here is your survey results, it is up to you now to sort it out. 236

The authors concluded that primary care staff view the role of patient experience surveys as serving a ‘quality assurance’ function, as they offered evidence that they were providing an acceptable standard of care. However, it was less clear that patient experience surveys fulfilled a QI function. Although patient experience surveys identified potential dimensions for change, ‘actual changes were usually confined to “easy targets” for modification such as décor or playing music’. 236 They note that ‘issues such as the management of GPs with evidence of poor communication skills, or responding to other “interpersonal” aspects of care, were much harder to tackle’. 236 They also argue that patient experience survey findings were only one of the ‘spurs to action’ to address problems that practices were often already aware of. In terms of the theories under test, this study suggests that although patient experience surveys may provide a clearer indication of areas of care that required improvement, there was no guarantee that this led to QI activities. The changes that were made focused on issues that staff were already aware of or on the organisational aspects of service delivery, rather than on the ‘harder to tackle’ issues of communication skills and interpersonal behaviour.

Theory 9a summary

These studies suggest that the timeliness of some forms of patient experience data were valued by providers and that ward-specific data were perceived as more useful than higher-level hospital data. However, they also draw attention to the trade-off in characteristics of different forms of patient experience data. Qualitative or ‘softer’ data from comment cards, patient responses to open-ended questions or complaints were seen as providing data that were useful in providing a more detailed understanding of the nature and causes of problems but were seen as ‘less robust’ by providers, while patient experience surveys were perceived as focusing on measurable but less relevant aspects of patient care but were acknowledged to be more robust. Furthermore, providers questioned whether patient experience surveys were able to capture the real-life complexities of patient care. Providers felt that both sources were needed to provide a more rounded picture of patient experience.

However, although patient experience surveys identified potential dimensions for change, the studies reviewed here suggest that this did not always lead to steps to improve patient care. Furthermore, where changes did occur, these were not always directly in response to the findings of patient experience surveys, but reflect issues staff were already aware of from other data sources. The studies indicate a number of reasons for this. Some forms of patient experience data were perceived as not being specific enough to be ‘in the sphere of influence’ of certain clinicians. Providers were sometimes not clear on how to make changes and wanted guidance with this process. Others felt that they did not have the resources to meet patient demands, such as opening primary care services at the weekend. The studies suggested that, when changes were made, these tended to focus on addressing aspects of the organisation of care, or the so-called ‘easy stuff’, and that the more interpersonal aspects of patient care, relating to the behaviour and communication skills of individual clinicians, were much more difficult to address.

Reeves et al.237

This study was designed as a pilot study to test the feasibility and impact of an intervention designed to improve the ways in which patient experience data were fed back to recipients. The intervention had a number of components, including (1) increasing the immediacy of the feedback, (2) providing specific feedback at ward level, (3) including patients’ written comments in addition to the ward’s scores and (4) an enhanced version of the intervention that also included ward meetings with nurses to discuss the findings of the survey and support them to act on the findings. It was hypothesised that the intervention would increase the likelihood that clinicians (in this case nurses) would take steps to improve patient care and, in turn, the patient experience scores for the ward would improve.

The study design was a RCT in two single site acute hospitals in London. All non-maternity inpatient wards were eligible for inclusion: 18 wards in trust A and 14 wards in trust B. Nine wards in each trust were randomly allocated to one of three arms, although the rationale for this number is not provided. Patient survey data were gathered in each trust specifically for the study, using the CQC’s Inpatient Questionnaire collected through a postal questionnaire. A random sample of 160 patients discharged from each of the included wards over a 2-month period was taken at 4-month intervals during the study period (on six occasions in trust A and three in trust B) and they were mailed a questionnaire. The overall response rate during the study period was 47%.

In the control arm, survey findings were provided to the director of nursing in each trust, with no special efforts made to disseminate them to ward nurses. In the ‘basic feedback’ arm, individual letters were sent to nurses on the wards and their matrons which detailed (1) bar charts of scores on questions about nursing care, comparing the target wards scores with the scores for other wards in the arm and the national average; (2) as the study progressed, graphs of how the ward’s scores had changed over time; and (3) transcription of patients’ responses to a series of open questions on the patient experience survey. The ‘feedback plus’ arm received the letters and feedback, but ward managers were also asked to invite ward nurses to ward meetings with the researchers during working hours to discuss the survey findings. The main outcome measure of the study was the mean score of a subset of 20 questions from the CQC inpatient questionnaire, which was termed the Nursing Care Score. The authors do not provide details on why and how these questions were chosen or on whether or not the scale was psychometrically valid and tapped into a common factor of nursing care. Notes were taken at meetings with nurses in the feedback plus arm and ward managers in the basic feedback arms were also contacted to establish whether or not actions had been taken in response to the survey, although details of how this occurred were not provided.

Multiple-level regression was used to examine changes in the Nursing Care Score in wards in each arm. This analysis suggested that there were no statistically significant differences in the changes in ward patient experience scores over time between the control arm and the basic feedback arm. The changes in the Nursing Care Score from baseline to follow-up for the wards in both basic and feedback arms were negative in all wards except two, suggesting that their patient experience scores had worsened over time. The authors report that, when asked, ‘none of the basic feedback ward managers identified specific actions resulted from the printed results’. This suggests that improving the timeliness of feedback and providing ward-level specific feedback alone was not sufficient to lead to improvements in patient experience scores. There was a statistically significant difference in the changes in scores over time in the feedback plus arm compared with the other two arms, with scores improving over time. However, there was much more variation in changes in the Nursing Care Score from baseline to follow-up in the intervention arm, with some wards staying virtually the same (three wards with changes < 1.0 point either way), some worsening (two wards) and one improving. Graphs showing the aggregated rate of improvement in the three groups also demonstrate that the differences between the three groups was largely due to the fact that control group and basic feedback patient experience scores worsened over the study period, while a very small overall improvement was observed in the intervention arm.

The authors reported some of their own impressions from ward meetings to understand the function of the ward meetings and explain why they lead to improvements in patient experience scores. Attendance at meetings was variable, but the attendance of matrons ‘had a positive influence’, as they offered ‘suggestions for improvement’, encouraged ‘ward staff and nurses to take responsibility for results’ or supported ‘efforts to implement changes’. 237 The authors felt that the ward meetings facilitated ‘nurses engagement’ and noted that the written comments in particular stimulated the nurses’ interest. The authors also noted that staff needed prompting from them to focus on understanding the patient feedback and planning strategies for improvement; without this staff were more ‘inclined to discuss . . . the many difficulties they experienced in fulfilling their duties; staffing shortages, NHS policy or their perceptions of hospital managers’. 237 We can hypothesise that the discussion of more general matters either was a lapse of focus on the nurses’ part or served to explain or justify the difficulties in understanding and acting on the data.

The verbatim quotations that are offered by the authors suggest that nurses felt that some of the survey findings had a rational explanation and did not constitute an indicator of poor care; for example, they explained that ‘Patients think we are talking in front of them as they are not there because we have to talk quietly to maintain confidentiality’. 237 They also appeared to implicate patients themselves in making unreasonable demands by using ‘call buttons for trivial reasons’ so that ‘it would not be good use of our time to answer them all immediately’, suggesting, again, that they questioned if the survey indicators tapped into ‘good care’. 237 The authors also note that it was ‘difficult to ascertain clear examples of innovations in patient care’ as a result of the patient survey feedback and the meetings. The most common responses were that nurse managers raised the issues in ward meetings and handovers and ‘reminded nurses of the importance of fulfilling their duties relating to ensuring patients’ experiences were positive’. 237

This study was a pilot study and, as such, has a number of methodological shortcomings. The randomisation process was not masked, leaving open the possibility that the research team’s preferences or knowledge of the wards influenced the allocation. It was not a cluster RCT; wards in the same hospital were randomised to different arms, leaving open the possibility that contamination between the three arms occurred, diluting the impact of the intervention. It is not clear whether or not the changes in patient experience scores in the intervention arm were statistically significant over time, only that they were statistically significantly different from the changes that occurred in the other two arms. The qualitative data were not collected systematically and many of the insights into the meetings were derived from the researcher’s impressions, with few verbatim quotations from participants.

Despite these limitations, the study does provide some lessons about the elements of the feedback process that may support or constrain the actions taken in response to the feedback of patient experience data. In terms of the theory under test, this study suggests that improving the timeliness of feedback and providing ward-level-specific feedback alone was not sufficient to lead to improvements in patient experience. Indeed, under these conditions, patient experience scores worsened. The facilitated meetings did lead to small improvements in patient experience as measured by patient experience scores, but it is not clear if these in themselves were statistically significant. These meetings functioned as an opportunity for nurses to air their concerns about the data (and their general working conditions) and as a means of raising nurses’ awareness about the data, rather than as sites for the strategic planning of improvements to patient care.

Davies et al.238

This study evaluated the impact of a peer and researcher support intervention coupled with a modified process of collecting and feeding back patient experience survey data on the QI activities of a small number of providers. Nine providers who had previously expressed an interest in learning to use patient experience data more effectively were invited to join a collaborative. One group left the collaborative early on in the project, leaving eight groups involved. They were invited to submit suggestions for how the current CAHPS survey could be modified to make it more useful for QI purposes, and the research team used the suggestions to refine the survey. The resulting survey covered five domains: scheduling and visit flow, access, communication and interpersonal care, preventative care and integration of care. It also included a question that measured a global rating of care, a question on whether or not patients would recommend the service to others and an open-ended question. The survey was administered to a random sample of recently discharged patients and fed back to providers on a rolling monthly basis to provide ‘real time’ data to providers, in the hope that this would ‘support a rapid cycle of quality improvement’. 238

In addition, the providers participated in a patient experience ‘action group’, which followed a ‘model of collaborative learning’. 238 This involved a group leader and team members attending a full-day bimonthly meeting facilitated by QI advisors, which focused on supporting the groups to understand and interpret the survey data, prioritise areas for improvement, set targets and action plans to address any issues raised by the patient experience data, and monitor their progress in implementing these plans. The impact of the intervention was evaluated using a mixed-methods cohort study, including measuring changes in patient experience scores during the 18-month intervention and interviews with collaborative methods to explore their experiences. All eight leaders participated in initial interview but, by the time of the follow-up survey, two groups had left the collaborative and two leaders had changed positions. Consequently, the six original leaders and one new leader were interviewed at follow-up. In addition, four leaders invited other members of their team to take part in interviews.

At follow-up, six groups had used at least one of the suggested tools to explore the possible reasons for their patient experience survey results. These tools included walkthroughs (five groups), patient interviews (two groups), patient focus groups (two groups) and cycle time surveys (one group). Four groups had used one of the suggested interventions designed to improve patient care, including scripting for clinic staff (two groups), communication skills training (one group) and patient education materials. Two groups had developed their own interventions. The leaders reported feeling that the group’s support had been useful in creating momentum and motivation to implement the changes, and had provided an opportunity to learn from others. As one member commented:

It lets you know that you are not alone . . . we tend to blame our workers if we get bad outcomes but if the whole world is getting bad outcomes . . . perhaps . . . it’s a common culture. We all have stories about success and failure and sharing those stories is helpful. 238

Six leaders decided on and attempted to implement at least one intervention to improve patient care; four leaders reported that they had implemented the intervention as originally planned, while two reported problems with implementation. These latter two groups reported that other organisational changes had competed for priority and they had focused on those instead. For the other four groups, however, the impact of these interventions was not always reflected in follow-up patient experience survey results. Many of the changes in the scores were not statistically significant and were sometimes in domains that were not directly related to the focus of the intervention that had been implemented. Of the four groups who had implemented an intervention, ‘three had some change in the direction they had hoped for’. 238 For example, one team had attempted to improve communication between staff and to patients about waiting times and test results, and found that more patients reported feeling that they had been kept informed about this in the last 6 months.

Two groups showed ‘mixed or negative effects’. One group had delivered communication skills training to staff and found a ‘slight improvement’ in patients reporting that doctors explained things to them in a way that was easy to understand, but a decline in the percentage of patients reporting that doctors spent enough time with them. Staff reported that some aspects of the communication skills package were perceived as conflicting with the group culture in suggesting that the problem lay with individual clinicians who need to work on their skills, rather than working together as team. Another had tried to reorganise their clinic to improve visit flow but found that patients reported longer waiting times and not being informed about their wait. This was perceived to be because patients arrived earlier than they were asked to and, therefore, perceived this as an additional delay. Thus, attempts to improve one element of care seemed to have a detrimental impact on other aspects. Following a post hoc analysis of their findings, the authors identified that ‘the two groups that succeeded most clearly in improving their patient experience worked on interventions that required no major change in clinician behaviour specifically for the project’ and which ‘aimed for modest improvements that did not require complex changes’. 238

This was a small case study of six doctor practices that were self-selected, highly motivated to take part in the study and relatively experienced in improvement activities. As such, the findings of this study may not be generalisable to other practices that were less motivated and less experienced. However, it provides some valuable lessons as to the circumstances under which the feedback of patient experience data does and does not lead to the successful implementation of QI strategies. It suggests that teams need considerable support to interpret and understand patient experience data, and that they need to conduct more specific investigations to identify the cause of negative patient experiences. Furthermore, it indicates that implementing change is challenging, and that the more complex the issue, the more challenging it is to effect change. Those that succeeded in this study attempted relatively simple interventions that did not require substantial changes to clinical behaviour. Finally, it also demonstrates that change to one aspect of patient care can have a detrimental impact on other aspects of the systems. Thus, as the authors argue in their conclusion, it is possible to produce small improvements in patient experience by making changes to simple aspects of patient care, but ‘it is difficult to . . . leverage more substantial change without a more comprehensive strategy that is organisation-wide and regarded as fundamental to organisational success’. 238

Theories 9b and 9c summary

Here we have reviewed two different interventions that aimed to increase the impact of the feedback of patient experience data on improvements to the quality of patient care. They suggest that enhancing the feedback alone, through providing timely, ward-level data to nurses, is not sufficient to lead to the implementation of QI strategies in response to these data. Supplementing this feedback with ward-level meetings served an important function of addressing nurses’ concerns about the validity of the data, raising awareness of the data and reminding nurses about their role in supporting positive patient experiences. However, these meetings did not serve as opportunities for the strategic planning of QI activities and led to only small improvements in patient experiences.

The other study evaluated an intervention that focused on hospital leaders who were involved in modifying the patient experience questionnaire used and received regular, timely feedback of patient experience data and both expert and peer support to interpret the data, investigate the causes of poor care and implement interventions to address any issues. Despite this considerable amount of support, the findings from this study suggest that some improvements in patient experience occurred but were hard-won. Those that succeeded focused on simple interventions that did not require complex changes or changes to clinical behaviour. This reinforces the findings from the studies reviewed previously in this section that changes to the behaviour of clinicians are more difficult to achieve. The study also demonstrated that changes to one aspect of patient care can have unintended effects on other aspects. The lesson from this study is that significant and sustained improvement in patient experience in response to feedback can only be achieved with system- and organisation-wide strategies.

Snyder et al.325

This study compared patients’ views of three standardised PROMs designed for use with people with cancer using a RCT design. The three PROMs in question were the EORTC QLQ-C30, six domains from PROMIS, and the Supportive Care Needs Survey-Short Form (SCNS-SF-34). The EORTC QLQ-C30 is a disease-specific PROM originally developed for use in trials to assess the impact of cancer treatment on patients’ quality of life, but it has also been used in trials examining the feedback of individual PROMs data in routine clinical practice. 326,327 It has 30 items that cover five functional domains (physical, role, emotional, social and cognitive) and eight symptom scales (fatigue, pain, nausea and vomiting, dyspnoea, insomnia, appetite loss, constipation and diarrhoea). PROMIS is a collection of short-form scales, item banks and computer-adaptive tests designed to be relevant to people across a range of conditions, including cancer. In this study, the fixed-item scales for anxiety, depression, fatigue, pain impact, physical functioning and satisfaction with social roles were used. The SCNS-SF-34 was designed to assess the care needs of people with cancer. It has 34 items covering five need domains (psychological, health systems and information, physical and daily living, patient care and support and sexual).

Those recruited for the study were patients of 12 clinicians involved in the care of people with cancer (medical oncologists, radiation oncologists and nurse practitioners) in the USA. Of the 301 eligible patients, 224 (74%) agreed to participate and were randomised to complete the EORTC QLQ-C30, the SCNS-SF-34 or PROMIS scales before every clinic visit during their treatment (weekly for radiation oncology, every 2–4 weeks for medical oncology patients) via the online Patient Viewpoint web tool. The PROMs were automatically scored by the Patient Viewpoint tool and for phase 1 of the study were printed and fed back to clinicians prior to the consultation. In phase 2 they were made available via a printed version, online via the Patient Viewpoint website or in the patient’s electronic record. At the end of their treatment, patients were asked to rate their views of the PROM using 11 Likert scales in the form of the patient feedback form. The items in this questionnaire assessed the extent to which patients thought that:

• the PROM was easy to complete

• the PROM was easy to understand

• the PROM was useful to complete

• the PROM helped them to remember when they met the doctor

• the doctor used information for their care

• the PROM improved quality of care

• the PROM improved communication with doctor

• they felt more in control of their care

• they would recommend the PROM to others

• they would want to continue responding in the future.

The primary outcome measure was the percentage of patients who answered strongly agree or agree to all 11 rating items on the patient feedback form. In addition, an exit interview was conducted with all patients to further explore whether or not scores were discussed and addressed.

Of the 224 patients who enrolled to the trial, 181 completed their allocated PROM on one or more occasions and completed one or more items on the feedback form. For these patients, there was a statistically significant difference in the primary outcome measure for the three arms of the trial, with the EORTC QLQC-C30 being rated most favourably (74% strongly agreed/agreed to all items), followed by PROMIS (61% patients strongly agreed/agreed) and then the SCNS-SF-34 (52% patients strongly agreed/agreed). The individual item ratings on the feedback form varied both within and across each arm. All items on the feedback form were relatively highly rated for all PROMs (the lowest was 67% agreeing or strongly agreeing that the SCNS-SF-34 improved the quality of their care). However, there was also a consistent pattern of responses to the feedback form items, whereby, for all PROMs, a higher percentage of patients reported that the PROM was easy to understand and complete and helped them to remember when they met the doctor, than felt that the doctor had used the information for their care or that the PROM improved their quality of care or communication with the doctor. Care must be taken in making inferences from these patterns, as testing for variation within the items across PROMs was not part of the planned analysis of the trial. However, the data suggest a hypothesis that may be tested elsewhere: that the process of PROMs completion itself may have more value to patients than the feedback has to clinicians.

The exit interviews provided some explanations as to why the SCNS-SF-34 was rated less highly by patients. The authors report that ‘several’ patients felt that a number of the items were unlikely to change over time and were therefore ‘not suited to repeat administration’. This may have been because the SCNS-SF-34 used a 1-month recall period but patients were completing the PROM weekly. Thus, the authors concluded that ‘the measure does matter to patients’ and PROMs vary in their perceived usefulness to patients. However, patients did not directly compare the three measures, and the findings might have been different had all patients completed all PROMs. In addition, this study is based on self-reports of patients’ satisfaction with the PROMs under study, which are often overwhelmingly positive. In terms of the theory under test, the study suggests that ‘not all PROMs are equal’, and that they vary in their ease of completion and usefulness to patients. It also tentatively suggests a hypothesis that the process of completing the PROM is perceived as having more value to patients in terms of helping them remember things to discuss things with their doctor than in relation to the impact on doctor–patient communication.

Nilsson et al.328

This study provides a comparative analysis of patients’ views of two commonly used generic PROMs: the SF-36 and the EQ-5D. The SF-36 has 36 items that cover eight subscales (physical functioning, role-physical, bodily pain, general health, vitality, social functioning, role emotional and mental health). Items vary in their response options; some items have three-level response options and others have five-level response options. The EQ-5D has five items covering mobility, self-care, usual activities, pain/discomfort and anxiety/depression. In the version assessed by this study, each item has a three-level response option. As such, this study provides a test of whether shorter, more general PROMs with less complex response options are preferred over longer, more comprehensive PROMs with more complex response options, or vice versa.

A total of 573 patients with a range of long-term conditions were recruited via health professionals from hospitals across Sweden. All patients received an intervention of some kind; these included patient education, surgical and pharmacological treatments. Patients were asked to complete the EQ-5D, followed by the SF-36, before and after their intervention. They were then asked to evaluate each instrument using a questionnaire. Two items asked patients to rate how easy the measure was to understand and respond on a 5-point scale from ‘very easy’ to ‘very hard’. One item asked patients to rate how well they felt each questionnaire described their health in a comprehensive way, from ‘very good’ to ‘very bad’. Patients were then asked if they felt that the routine collection of health outcome assessment would be valuable in the future (yes/no) and, if so, which questionnaire they would prefer (SF-36, EQ-5D or no preference). Patients were also given space on the form to expand on their answers.

Of the 573 patients recruited, 463 (80%) completed both measures and an evaluation form. Table 5 summarises the findings of the study.

This table shows that patients rated the EQ-5D as significantly easier than the SF-36 to understand and respond to. There were no differences in patients’ ratings in terms of the extent to which they perceived each measure to describe their health in a comprehensive way. The majority of patients expressed no preference for either measure to be routinely collected in the future. Of those who did express a preference, significantly more patients preferred the SF-36 than the EQ-5D.

The open-ended responses provide some explanation for these findings. Twenty-seven responses referred to the value of outcome assessment in general and 62 responses explained their preference (or lack of) for one instrument or the other. Patients commented on the positive aspects of outcome assessment, including that it ‘saves time at consultations’, shows that ‘someone cares!’ and may give ‘a better insight into the patient’s experience’. Patients also commented that completing the questionnaires ‘forces you to reflect on your situation’. ‘Negative’ comments included scepticism that it was possible to measure health using questionnaires, reflected in the comments ‘how can you measure health on paper?’ and ‘mental or physical health is difficult to describe in text’.

Of those patients who expressed no preference for one measure or the other, free-text responses reflected that the questionnaires both seemed similar, were equally easy to respond to or answer, or were both equally incomplete. Those who preferred the SF-36 did so because they felt that the content of questions or the nature of the response options provided a more comprehensive assessment of their health, as illustrated in the quotations ‘more detailed questions which give a better picture of how you feel’ and ‘more response options to choose from, which makes it easier to describe your condition’. Those who preferred the EQ-5D did so because they found it easier to respond to – ‘easier to understand’ – or less confusing than the SF-36: ‘SF-36 is awkward to answer’. Some patients also identified problems with the EQ-5D, despite indicating a preference for it; these included that they were ‘too healthy for the questions’ or that the response options were too coarse: ‘I would like to answer in between the response alternatives’.

This study is based on patient responses to a relatively simplistic questionnaire to explore their views of two PROMs. All patients completed the PROMs in the same order and their preferences might have been subject to ‘order effects’. However, the study provides a useful test to explore whether patients prefer shorter, less comprehensive PROMs or longer, more comprehensive PROMs. Overall, quantitative data collected in this study found no differences in the extent to which patients felt that each questionnaire adequately described their health, but they felt the EQ-5D was easier to understand and respond to. Although most patients did not express a preference, of those who did, more patients would prefer the SF-36 to be routinely collected than the EQ-5D. The open-ended responses suggested that patients who preferred the EQ-5D did so because it was easier to understand, while the SF-36 was chosen because it was perceived to provide a better reflection of patients’ health. In terms of the theory under test, this study highlights the tension between simplicity and complexity and between brevity and comprehensiveness in the structure and format of PROMs. Overall, it suggests that patients with long-term conditions are willing to tolerate a questionnaire that is longer and more complex to complete if it provides a more comprehensive assessment of their health.

Neudert et al.329

This study assessed patients’ perceptions of the validity and distress caused by completion of three different PROMs: the Sickness Impact Profile (SIP), the SF-36252 and the Schedule for the Evaluation of Individual Quality of Life Direct Weight (SEIQoL-DW). The SF-36 has been described previously. The SIP is a standardised generic PROM and consists of 136 yes/no items covering 12 domains. The SEIQoL-DW is an individualised PROM developed for use by patients with a range of conditions. 330 During completion, patients are invited to choose five life areas they consider to be the most important determinant of their quality of life. They are then asked to rate how satisfied they are with each area on a visual analogue scale. They are then asked to indicate the relative importance of each area using a weighting instrument, such that a pie chart is produced containing five segments, each representing a chosen area, and where the size of each segment represents the relative importance to patients. As such, this study compared patients’ preferences for standardised generic PROMs versus an individualised PROM. It therefore allows us to test the theory that patients consider individualised measures to have greater face validity than standardised PROMs.

Patients with amyotrophic lateral sclerosis at one neurological centre in Germany were invited to participate in the study; most of them were enrolled on a drug trial. All patients completed the SIP as part of the drug trial and were randomly allocated to also complete either the SF-36 or the SEIQoL-DW. Patients completed the SIP and SF-36 independently or with help from one of the authors, whereas the SEIQoL-DW was administered through an interview. Patients completed the measures at least three times at 2-month intervals. Patients were asked to rate their perceived validity and the degree of emotional distress involved in completing each of their allocated instruments on a visual analogue scale from 0 to 100.

Of the 62 eligible patients, 51 (82%) agreed to take part in the study and, of these, 42 (82%) completed at least three assessments of their allocated measures. Patients’ ratings for the perceived validity of the instrument in measuring their quality of life were significantly higher for the SEIQoL-DW than for the SIP or SF-36. There were no differences in the perceived validity of the SIP or SF-36. Overall, the degree of emotional distress caused by the completion of any of the instruments was low, although values were only reported graphically and absolute numbers were not reported. The SEIQoL-DW had the lowest rating for emotional distress, followed by the SF-36 and then the SIP, with the difference between the SEIQoL-DW and the SIP being statistically significant.

This was a small study of patients with a relatively rare condition, and as such the findings may not be generalisable. Nonetheless, the study provides useful data for theory testing. In terms of the theories under test, these findings can be interpreted in two ways. First, they indicate that patients preferred the freedom to nominate areas of life important to them as a result of the open structure and content of the SEIQoL-DW, compared with responding to the pre-defined set items and response alternatives mandated by the standardised nature of the SF-36 and SIP. The authors also point out that the high distress ratings of the SIP may also be because the items focus on what patients are no longer able to do, rather than what they can do, and might have reminded patients about the progressive nature of their condition. Alternatively, as the authors point out, the differences in perceived validity and emotional distress ratings between the three instruments may be due to the fact that the SEIQoL-DW was interviewer administered and this interaction was preferred by patients and also served to reduce the emotional distress of instrument completion. These explanations are not mutually exclusive and both may account for the differences in the ratings between the three PROMs. However, this study did not collect any qualitative data to further explore how and whether these explanations work together or independently of each other.

Summary of quantitative findings on patient views of patient-reported outcome measure format and structure

The studies reviewed in this section suggest that ‘not all PROMs are equal’, and that they vary in how easy they are to understand and respond to and in the extent to which patients feel that PROM results reflect their health. We found some evidence to indicate that patients are prepared to tolerate a longer questionnaire that is more complex to complete if it provides a more comprehensive assessment of their health. 328 However, PROMs that are too long and/or contain items that remind patients of how little they can do cause them distress. 329,331 We also found some evidence to support the theory that patients view individualised measures as more valid and less distressing to complete. 329 This may be because they prefer the freedom to nominate areas of their life that are important to them. It may also be because they value the interaction with an interviewer, which could facilitate the exploration of these areas and mean that patients feel that they are being listened to which, in turn, may reduce any emotional distress caused by PROM completion.

Dowrick et al.81

This study examined views on the use of standardised questionnaires to assess depression among UK patients and GPs while their use was still incentivised under the QOF. Interviews were conducted with 34 GPs and 24 patients. These participants were recruited from 38 general practices around Southampton, Liverpool and Norwich that took part in a quantitative study on depression severity questionnaires. In the selection of GPs, variation was sought in terms of gender, years of experience, full-time/part-time practice, trainer/non-trainer status, geographical location and size of practice. For patients, selection considered sex, age, ethnicity and sociodemographic background.

Patients were largely positive about the questionnaires, seeing them as an efficient approach and an indication that clinicians took their problems seriously. As one patient explained, ‘It can be perceived as you’re being taken more seriously’. 81

The findings suggest one way in which PROMs may encourage accurate description of symptoms by patients: making it easier for them to record their concerns by providing a less threatening format. One patient explained that the questionnaire helped to indicate their concerns because with some issues:

[Y]ou can’t say how you feel and with answering the questions, it, like, asked you how you felt and things like that and now I think that helped, just like, ticking the boxes, just so that he knew . . . how I felt as well.

PT21, 20–2581

Other patients commented that the questionnaire had a less personal nature, which made it easier to answer the questions and share information with their doctor, as one patient explained:

[I]f it had been that the doctor had asked you the questions . . . then it might be a bit different. I think the impersonal nature of the questionnaire’s probably helpful.

PT11, 107–1081

The authors reported that some patients saw the PROM as an opportunity for reflection, which in turn helped them to communicate their concerns to clinicians more effectively. Some patients reported that the questionnaire had helped to increase their self-understanding, so that they could organise their thoughts and express themselves better to their doctor, as one patient reported:

I think that [completing the questionnaire] helped me in my head as well . . . Well I started to think, you know about why I was getting depressed and that.

PT19, 45–4881

However, the study pointed to different views among patients on the PROMs format. One patient felt that the scoring system was too reductionist and did not allow her to accurately express her status:

I suppose, yeah, it sort of quantifies that you are . . . you do have problems . . . but I, I still feel that it was, like . . . you’re trying to, like, tie a number to a thing which isn’t necessary, isn’t necessarily like a yes or a no . . . it’s a, it’s very difficult to . . . put a description to it, I think.

PT03, 98–10181

This suggests that although a closed format may help some patients to express their feelings, other patients may see this format as a barrier, such that the PROMs questionnaire may not allow what they see as an accurate picture of their health status. The authors also suggested that some patients may not complete the PROMs accurately owing to concerns about the stigma attached to depression. As one patient explained:

You’re more likely to lie, well I found I’m more likely to lie . . . Because I still find a lot of stigma attached to depression.

PT16, 100–581

In addition, some patients reported manipulating or ‘gaming’ their PROMs feedback to influence the care they received:

I said no to anti-depressants as I wasn’t depressed. And then she gave me a questionnaire, I remember quite clearly . . . And I remember reading . . . and I thought, I’m not putting down how I really feel, and so I didn’t.

PT24, 131–4281

These themes seem to have been reported by only a small number of patients within the study sample and it is unclear how widely these views are shared.

General practitioners were more cautious and had concerns about the validity and usefulness of the questionnaires. Furthermore, GPs also reported some manipulation of the data by avoiding coding people with depression to avoid having to administer the depression questionnaires.

This study provides useful evidence about conditions that can affect patient reactions at this stage of completing the PROM, and, through this, whether or not PROMs record accurate information. Based on the findings, it appears that standardised depression questionnaires can make it easier for patients to disclose their concerns, when patients feel more comfortable admitting to sensitive issues through ticking a box rather than articulating concerns verbally. However, some patients may feel that a standardised format does not allow accurate description of their concerns. Second, patients may still not complete the forms openly if they are embarrassed about disclosing sensitive problems, suggesting that the relative ease of ticking a box does not always overcome stigma. In addition, patients may intentionally provide inaccurate information if they know or are concerned that their answers will have unwanted implications for their care.

Leydon et al.336

This study336 was a further analysis of the interviews conducted with the 34 GPs as reported by Dowrick et al. 81 Here, the analysis focused on GPs’ views of the impact of standardised depression questionnaires on the doctor–patient relationship and the flow of the consultation. As such, the paper is particularly relevant to the theory that the use of standardised PROMs may interfere with the doctor–patient relationship.

The findings indicated that doctors placed a lot of emphasis on developing a relationship with patients in order to detect and manage depression. One GP explained that the ‘one to one personal relationship’ was especially important when they were ‘supporting someone with depression’, while another observed that ‘a lot of picking up depression is about rapport and about patients feeling comfortable and establishing a relationship’. 336 This relationship building was achieved through talking to patients and letting patients express themselves in their own words. GPs reported feeling that standardised depression questionnaires threatened this relationship-building process by covering issues that had already been discussed, but in a mechanistic way, and trivialising the strength of patients’ emotions, as the quotations below illustrate:

Getting them to do the tick box thing at the end of all that, when they’ve already bared their soul, can sometimes be a bit . . . intrusive I find.

GP336

If you’ve had a very loaded consultation, very cathartic . . . the HAD [Hospital Anxiety and Depression] scale can appear to trivialise the depth of emotions that are being expressed.

GP336

General practitioners also found it difficult to integrate completion of the questionnaire during the consultation, as one GP explained: ‘the greatest challenge is how to incorporate it tactfully into the consultation.’336 GPs reported that they felt uncomfortable using a standardised questionnaire during the consultation when a patient first came to see them about depression, as the two quotations below show:

. . . when a patient comes with a first presentation of depression . . . I don’t feel comfortable, straightaway with . . . issuing a box-ticking exercise.

Where do you plonk those great big . . . bombshells in the middle of a normal consultation with somebody. 336

These quotations imply that the administration of depression questionnaires was not part of the ‘normal’ activities one might find during a consultation. Instead, the use of words such as ‘box ticking’ and ‘bombshell’ suggest that the questionnaires were viewed as bureaucratic and destructive to the normal flow of conversation. GPs had developed a number of strategies to manage this conflict, including giving patients the questionnaires to complete at home, as one GP explained:

[W]hat I would tend to do is give them a form and say, look, take it away at your leisure. 336

This study suggests that GPs felt that developing rapport and a trusting relationship with the patient was essential to enable patients to share their feelings and emotions in order for the GP to detect and then support people with depression. In terms of the theories under test, the use of standardised depression questionnaires threatened this process because these questionnaires appeared to revisit issues that had already been discussed and to trivialise patients’ emotions. They were not seen as vehicles for enabling patients to share their emotions and feelings; rather, they were seen as bureaucratic and destructive. Furthermore, it was difficult to incorporate the questionnaires into the consultation without disrupting the flow of conversation and, thus, the relationship-building process. At the same time, the use of these measures were incentivised under the QOF, so rather than avoiding their use altogether, GPs had developed ‘workarounds’ which included avoiding coding people as having depression (as reported by Dowrick et al. 81 on the basis of the same interview data), so that the use of the questionnaires was not necessary, and giving patients the questionnaires to complete at home.

Mitchell et al.80

This study also explored primary care practitioners’ views of the use of standardised depression questionnaires in the diagnosis and management of depression in primary care in England. The authors took a purposive sample of four GP practices and conducted one focus group in each practice. Focus group participants included GPs, practice nurses, community nurses, primary care mental health workers and practice managers. Participants were asked to describe how the introduction of the QOF and NICE guidelines, with their requirements for GPs to use standardised depression questionnaires, had influenced how depression was diagnosed and managed.

The authors reported that GPs found the use of the depression questionnaires ‘counterintuitive, intrusive and unnecessary’. 80 One GP explained that:

they . . . break down in tears and tell you how depressed they’re feeling . . . and you ask them questions and then ‘oh now I’ve got this questionnaire to fill out’. I just think it’s so inappropriate sometimes. 80

This same GP also felt that the standardised questionnaire was often unnecessary because:

it’s not going to change your management because it might be someone you know quite well but you have to . . . get the QOF points. 80

In other words, clinicians did not see the questionnaire as essential to the management of the patient, as they were already familiar with their history, but they felt obliged to use the questionnaire because the practice would lose QOF points, and thus income, if they did not.

Consequently, the authors observed that GPs had adapted how they administered a particular questionnaire, the PHQ-9, to fit with their consultation style. These included some methods which did not compromise the validity of the questionnaire, including letting the patient self-complete the measure in the waiting room and administering the measure via telephone. However, other ‘workarounds’ may have compromised the validity of the PHQ-9. These included GPs reading out the questions to the patient and recording the answers themselves, missing out the last item on the PHQ-9 because it was not necessary for the QOF points, recalling the questions from memory during the consultation and working out the score afterwards, and adapting the wording of items via a translator for patients whose first language was not English.

Pettersson et al.337

This study aimed to explore the views of Swedish GPs regarding the use of standardised depression scores to detect and manage patients with suspected depression in primary care. In Sweden, the use of standardised depression questionnaires in primary care has been recommended by national guidelines, but no monetary rewards or sanctions or routine data collection have been mandated. GPs were recruited from an urban area (Gothenburg) and a medium-sized town (Skövde); GPs in Gothenburg were also involved in a trial to examine whether or not the routine collection of a standardised depression rating scale improved recovery in patients with mental health problems. Of the 30 GPs invited to participate, 27 took part in one of five focus groups. GPs were asked to discuss when they did and did not use the instruments, how the use of the instruments affected the consultation and the extent to which they thought that the questionnaires provided an accurate representation of the severity of a patient’s depression. All focus groups were audio recorded and field notes were taken.

The authors found that all GPs emphasised the importance of detecting and managing patients with depression based on their own experience; instruments seldom added any value. As one GP explained:

With questionnaires, it feels like going back in development and starting at a more basic level where you give every detail the same importance although you in reality can drop most information rather quickly and concentrate on a few things in order to get a clue.

M2 FG1337

They felt that the use of standardised depression scores may lead to a more bureaucratic work style and were concerned that this may limit a GP’s ability to think for him- or herself.

General practitioners suggested that the PROMs distracted them from noticing a patient’s non-verbal communication, such as body language, which was as important to GPs in understanding the patient’s feelings. As such, PROMs completion limited communication between the doctor and the patient. As one GP explained:

[. . .] misses the opportunity to pick up that little extra that was of special importance to the patient, to hear the nuances in what is said and not said.

Female 7 (F7), FG2337

Most participants perceived that the consultation was negatively affected when an instrument was used because it constrained the narrative of the patient and interfered with the GP’s ability to listen to the patient. GPs perceived that PROMs limited the patient’s opportunity tell their own unique life story. Furthermore, GPs viewed that this affected the rest of the consultation:

Because then it is said and done and we should of course follow up and have a normal consultation talk but in some way both parties are locked by this protocol.

M7, FG3337

General practitioners also felt that the use of standardised questionnaires restricted their ability to be active and engaged listeners. They observed that patients expected to receive their full attention and to engage in a dialogue and a discussion. GPs perceived that using a standardised depression questionnaire led them to feel ‘alienated’ from patients’ problems. They also felt uncomfortable with trying to reduce patients’ emotions to a score.

Many GPs were uncertain whether standardised questionnaires provided a valid and reliable representation of the patient’s problems. This was for a number of reasons. Many expressed the view that questionnaires could not capture the complexity of such a condition as depression. They also reported that a number of patients had misunderstood the questions and had then gone back and changed their ratings once the GP had explained what the questions meant. We can hypothesise that such help might have compromised the validity of the questionnaire, but not giving help would have rendered the results of the questionnaire meaningless. GPs also questioned the validity of patients’ self-ratings, which they felt would vary depending on the patient’s life situation. One GP observed:

I am convinced that we have patients that always score very high on MADRS-S [Montgomery–Åsberg Depression Rating Scale] and yet live their normal lives. But they only go to the doctor the day they can’t handle their work.

F7, FG2337

Although GPs felt that most patients completed the measures truthfully, some raised concerns that some patients may game or artificially inflate their scores to obtain sick leave:

. . . but I think that we may also be manipulated. Symptom scales are available on the Internet and you have threads on Flashback on how to proceed in order to get a sick leave.

M5, FG2337

However, all GPs agreed that questionnaires were useful in some situations, for example with patients who found it difficult to express themselves. For example, one GP related a case of a patient who only gave ‘weak signals’ that she might be depressed, but, on completing the questionnaire, it became clear to both the patient and the GP that she was depressed. Some GPs used questionnaires after the consultation to check that they had not missed any important aspects. The ratings of various items were often used as starting points for discussion, thereby improving communication. They were also found to be useful for patients who could not accept that they had a psychiatric diagnosis:

Men in their forties can be very hard to convince. They prefer to have an ulcer diagnosis.

F5, FG1337

This was a small qualitative study of a self-selected sample of GPs in two areas of Sweden. It is possible that those with strong feelings (either positive or negative) were more likely to have taken part in the study. However, it provides another layer of evidence to suggest that GPs felt the use of standardised depression questionnaires hampered their relationship with the patient and constrained the patient’s ability to tell their story in their own voice. They also questioned the ability of depression questionnaires to adequately and accurately capture the patient’s problems.

Summary of findings from managing depression in primary care

These four papers provide a useful synthesis of the complex ways in which different drivers and incentives may support or constrain the use of PROMs data by different stakeholders for different purposes. It must be noted here that the majority of the evidence reviewed in this section reflects the views of GPs, who, some might argue, may have felt threatened by the imposition of these measures. It is possible that those who agreed to be interviewed in these studies were simply those who held the strongest opposing views to the use of PROMs in primary care.

The patients in Dowrick et al. ’s81 study were, not surprisingly, unaware that the use of the measures were incentivised within the QOF. For some patients, completing a standardised questionnaire was a less intrusive way of sharing their feelings with their doctors and also enabled a process of self-reflection which helped them to understand why and how they were feeling depression. Other patients felt that the standardised nature of the PROM did not fully reflect the complexity of their experiences. However, some patients suspected that PROMs may be used to limit their access to care and, under these circumstances, they admitted misrepresenting their experiences in order to avoid this outcome.

In contrast, GPs felt that it was essential to develop a rapport and a trusting relationship with the patient, so that the patient felt able to share their feelings and the GP could then identify and support those patients who had depression. GPs perceived that using standardised depression questionnaires constrained this process by trivialising patients’ emotions, and they found it difficult to incorporate the use of standardised measures into the consultation. Although not explicitly mentioned by GPs themselves, their comments referring to the use of standardised depression questionnaires being a ‘tick box’ and ‘bureaucratic’ exercise echo Mallinson’s331 observations about the ‘interactional strangeness’ of standardised surveys, where opportunities for sense making and relationship building are constrained.

General practitioners in England found themselves in a difficult position. They did not perceive that standardised depression questionnaires were helpful in supporting their care of individual patients with depression. However, because the use of such measures was incentivised under the QOF, they stood to be penalised if they did not use them. Under these circumstances, GPs developed a number of strategies to meet these two conflicting drivers, by changing the ways in which they administered the standardised depression questionnaire. Some of these methods, which attempted to mitigate the difficulties they experienced in incorporating standardised questionnaires into the consultation, threatened the psychometric validity of the questionnaire, thus defeating its value in the care of individual patients. Furthermore, GPs were also reluctant to code patients as having depression, to avoid being penalised for not using a standardised depression questionnaire, which challenged the measure’s value as an indicator of the quality of care for people with depression. Here, we see an example that attempting to use the same PROM to support the care of individual patients and as an indicator of the quality of patient care can result in PROMs being used in a way which undermines their use for one or both purposes.

Hall et al.339

The Children and Young People’s IAPT programme is an initiative delivered by NHS England that aims to improve both access to and the quality of CAMHS. A key component of this is the use of session by session outcome monitoring to inform the individual care of service users. To inform the choice of measures for this purpose, this study reports on a study that piloted a session-by-session outcome monitoring tool in three UK CAMHS clinics. The monitoring tool was SxS, an electronic 8-item questionnaire based on the Strengths and Difficulties Questionnaire. Patients and/or their parents or carers completed the questionnaire on a tablet computer in the waiting room before each appointment. The tool automatically created a report graphing the patient’s progress, which was provided to the clinician, the idea being that this would be discussed during the appointment. In this context, the SxS was expected to serve two purposes: the care of individual patients was informed through feeding back individual patient data to clinicians but also data could also be aggregated to monitor the quality of the service. Semistructured interviews were conducted with clinicians (n = 10), administrative staff (n = 8) and families (n = 15) to elicit their views on the monitoring tool.

Clinicians made general comments on their attitudes towards outcome measurement, which suggested that they did not find outcome measures useful for service development because they did not receive any feedback on this aspect of data collection. As one clinician explained:

they are important for the service . . . but . . . I don’t ever see any data coming back from them . . . so I don’t think they are very useful . . . it would be more useful if we were getting feedback from them. 339

In this study, patients completed the questionnaire prior to their session with the clinician. Clinicians felt that this was the most appropriate time to complete the measure. Completing the measure during the session could ‘sometimes just get in the way of the therapeutic process a bit’. 339 Completing the measure after the session would not enable data to be used with patients during the session and could also resulted in flawed data for service monitoring purposes because patients may feel ‘like they are giving us feedback’ instead of reporting on the symptoms and thus could ‘feel under pressure to say something . . . more positive than they really think’. 339

Patients and clinicians held different views of the benefits of PROM collection. The authors observed that families felt that ‘completing the questions was a helpful process in its own right’,339 because it provided an opportunity to reflect and assess their condition, which made them more prepared for discussion with clinicians. The authors further noted that:

Parents and young people reported that it reminded them to ask specific questions to their HCP [health-care provider] and that it gave them an opportunity to reflect on how they had been since their last session. As a result of this, families reported feeling more prepared for their session. 339

Some patients also commented that completing the measure showed them how far they had progressed over time and that this provided the motivation to keep working on themselves in therapy. As one young person explained, ‘It showed you . . . how far you’ve been going and . . . giving you motivation to try and get it up another step’. 339 However, this was possible only if clinicians discussed the findings of the measure with them; otherwise, they felt that there was no point to completing it, as one young person observed: ‘I think there’s not really much point in sitting and doing a questionnaire that isn’t being used to help you.’339 However, some clinicians felt that referring to the measure results in the session could create awkward silences, during which the young people they were working with became distracted, as one clinician described: ‘You’ve got to read it and then they sit there in silence and then you’ve got somebody who has got tics or has ADHD [attention deficit hyperactivity disorder] and they start kicking at blocks, the door, shouting . . .’339

This study provides some evidence in support of the theory that completing PROMs can help support patients to reflect on their problems and to raise their concerns with clinicians. It suggests that PROMs enable reflection that makes patients better prepared to raise issues and questions in consultations. It also suggests that patients value feedback on their progress and this can motivate their continued engagement with therapy. However, this is possible only if patients are also provided with a copy of the scores or if clinicians discuss them with the patient during the session. Clinicians reported that referring to the scores during the consultation sometimes interfered with young people’s engagement.

Wolpert et al.340

This paper aimed to explore the views of clinicians and service users of CAMHS and diabetes services regarding the use of routinely collected PROMs data in England. Semistructured interviews were conducted with 10 participants from CAMHS (six young people and four clinicians) and 14 participants (four young people, seven mothers and three clinicians) from a children’s diabetes service. Service users were recruited through advertisements in voluntary sector organisations and, as such, may not represent the range and diversity of service users of child mental health or diabetes services. This group had not experienced the routine collection of PROMs in their service and therefore interviews explored their views on two measures (a symptom checklist for CAMHS and the PeDsQL for diabetes services). During the interviews, participants were asked what information clinicians should seek in order to understand patients’ priorities for care and their views on implementing the routine collection of PROMs with the services they used.

The young people using the diabetes service felt that explaining their experiences to clinicians was difficult and held mixed views about whether or not PROMs would support them in this process. One young person felt that completing a PROM may ‘show what I feel like . . . so clinicians can help me’, while another could not see how it would help her discuss her concerns: ‘I put it down because it . . . matched how I felt but I’m not sure how to approach it now I have written it.’340 Participants also commented on the format of the PROMs; clinicians reported a ‘real tension’ between the fluid and unique nature of patients’ views and the fixed and standardised format of PROMs. Service users also questioned if the fixed format PROMs could adequately capture the dynamic and continually changing nature of their experiences. For example, one young person with diabetes explained that she felt it would be difficult to complete a PROM before her consultation with the clinician because ‘I might start doubting whether my answers were right . . . if I do it quickly, on the paper, I’ve done it and I can’t go back and change it’. 340 This quotation suggests that the process of completing the PROM gives a fixed and permanent status to their experiences at one point in time; however, when service users’ experiences change rapidly, the PROM may not represent how they feel at the time of the consultation if it was completed days before.

The authors also noted differences between service user and clinician views on the timing of PROMs collection. The authors note that clinicians thought PROMs were most beneficial if collected immediately in order to target interventions to patients’ needs and produce ‘better . . . quicker outcomes’. 340 In contrast, young people felt that they needed to ‘build rapport’ with clinicians ‘instead of going straight to the nitty gritty’. 340 Thus, before young people felt able to share some of the more sensitive information contained in the PROMs data they needed to develop a relationship with the clinician. As one young person explained, for some of the PROM questions, ‘You’d tell your best friend or someone you were really, really close to but you wouldn’t tell this random person you’d just met’. 340

Participants also discussed concerns about how PROMs data would be used. A particular issue for the CAMHS service users was the implications of PROM completion for the service itself and their access to it. For example, one service user expressed concern that if she indicated on a PROM that she was worse than last time then this would be ‘rude’, reflect badly on the service and imply that it has not helped her. Another was worried that the PROM would be used to restrict her access to the service: ‘If I tick “much worse” then I don’t know if [that means] “I don’t need this service”, I don’t know, I’m very confused.’340 Parents of children with diabetes were concerned that focusing consultations on emotional issues might mean that less time was available for discussing diabetes management or the ‘objective hardcore’ and could bring unwanted professional scrutiny of the care they provided to their children.

This was a small study in selective group of clinicians and carers and parents of children who used CAMHS. Nonetheless, it adds a further layer of evidence to our understanding of how patients approach PROMs completion. In terms of the theories under test, this study suggests that the fixed nature of PROMs may not fully capture patients’ experiences when those experiences change rapidly from one day to the next. This study indicates that services users have concerns about the use of PROMs when they are used to judge the quality of care they receive or restrict access to the services they receive. The study also suggests that the extent to which a PROM may enable or constrain the sharing of sensitive information with clinicians also depends on the patient’s relationship with and trust in the clinician.

Stasiak et al.341

This is another study that explored the views held by child and adolescent service users and their parents regarding outcome measures and their routine collection in CAMHS in New Zealand. The authors conducted nine focus groups that included 34 child and adolescent service users and 21 family members. In the focus groups, participants were invited to try out completing some commonly used PROMs and were asked to comment on their ease of use, acceptability and relevance. Family members were also asked about the possible difficulties of and concerns about outcome measure collection. The analysis conducted by the authors was largely descriptive, with little attempt to explain their findings. Here we reframe some of their findings in relation to the theories under test.

Participants commented on the process of completing outcome measures and the acceptability of completing the measures. Two key tensions emerged with particular relevance to our synthesis. First, there was a tension between wanting the measure to be brief and at the same time comprehensive enough to capture the complexity and changing nature of patients’ symptoms. The authors report that young people felt it burdensome to complete long measures on several occasions and, as one young person explained, ‘if it’s too long, you’re just going to make the answers up’. 341 However, at the same time, participants questioned whether or not measures fully captured patients’ specific experiences or the dynamic and changing nature of patients’ symptoms. As one young person explained, ‘I can be really unwell and . . . get better and then I can get better and then I can get really unwell . . . so what’s the point of having these forms?’. 341

The second tension was whether completing the measures supported or constrained young people’s ability or willingness to express sensitive issues to clinicians. Key contextual determinants of this were the timing of completion and the degree of rapport and trust between the young person and the clinician. For example, the authors noted that young people felt completing a written measure to be ‘easier and less embarrassing than talking to a clinician about difficult issues’. However, young people also reported that the questionnaires should be administered by someone with whom they had developed rapport. One young person explained, ‘I think you’d have to have a trust thing built up first to actually share something with that person’. 341 The authors also argued that ‘children and young people may not respond accurately to questions they regard as personal or sensitive unless there is good rapport with clinicians’. 341

Participants also discussed the positive and negative ways aspects of completing a measure and the uses to which data could be put. On the positive side, some parents felt that completing an outcome measure signalled that someone was taking their experiences seriously. As one parent explained, completing an outcome measure ‘felt good because you finally had someone listening to you and trying to help you through what you were going through’. 341 The authors noted that young people felt that having the means to track their own progress was useful and ‘potentially empowering’. 341 Young people also felt that completing an outcome measure at the start of treatment provided a useful baseline ‘so that everyone knows where you are’ and that future progress could be measured against it.

However, young people and family members also expressed some concerns about the value of outcome measurement and the use to which data were put. Parents wanted reassurance that data were going to be used to inform the care of their child: ‘is it going to change how you treat her, is it going to make things better, are you actually going to hear what we’re saying?’. 341 They were also concerned that politicians might use data to restrict access to services. One parent thought that politicians might look at data collected and think ‘we’ve “just got this nice little test that we’ve just proved that they’ve made X amount of progress” so that’s it, end of service’. 341

This was a small study in a self-selected group of services users and families who used CAMHS. However, the findings echo many of those later found by Wolpert et al. 340 In terms of the theory under test, this study suggests that the structure and format of PROMs creates a tension between the requirement for measures to be brief to maintain respondent engagement and the need for the measures to be detailed and flexible enough to capture the complexity and dynamic nature of service users’ experiences. However, the study also suggests that it is not just the format of a PROM that supports or constrains patients’ honest or accurate sharing of their experiences. Rather, it suggests that sharing experiences through a PROM are constrained when patients have not developed a trusting relationship with the clinician caring for them or when they are concerned that PROMs data may be used to restrict their access to services.

Cheyne and Kinn338

This study explored counsellors’ views and experiences of using an individualised PROM, the SEIQoL, during the process of counselling service users with drug and alcohol problems. The SEIQoL was being evaluated as part of a controlled trial, where intervention group counsellors (n = 3) were trained to use the SEIQoL and were then expected to use it during their sessions with clients. During the 6-month trial, 20 clients received counselling using the SEIQoL. This study reports on findings from three focus groups with the counsellors who used the SEIQoL during counselling, carried out at 12 weeks, at 18 weeks and at the end of the study period. Data from these groups were used to inform the development of a questionnaire that was completed by all the counsellors. One of the research team also reviewed patients’ case records to extract data relating to the use of the SEIQoL, services users’ and counsellors’ views of the SEIQoL and service users’ responses to counselling. However, in reporting their findings, the authors do not always make it clear whether the data reported came from case notes, focus groups or questionnaires.

The authors report that the counsellors found the SEIQoL easy to use and that 17 out of the 20 clients fully understood the aims and use of the SEIQoL (three were uncertain). The case notes revealed that one service user experienced some boredom with SEIQoL, whereas the other 19 ‘were directly engaged by it’. 338 The counsellors felt that the tool was flexible and fitted ‘well in an assessment’. 338 Counsellors’ notes also captured services users’ comments on the SEIQoL, which reveal that they found it useful to ‘get a better picture of my life right now’. 338 Counsellors felt that the use of the SEIQoL enabled service users to engage more willingly in self-reflection, which led to service users identifying areas of their life they wanted to tackle first. Both counsellors’ own reflections and their impressions of reactions from service users illustrate this:

Helped the service user look inward and stop and reflect on whether there was another way of looking at their situation.

Counsellor338

I liked the subtle way it allowed the service user to get another perspective through their own thinking.

Counsellor338

It changed the way I saw drink screwing up my life.

Service user338

The disks helped me to weigh things up and choose what I wanted to deal with first.

Service user338

Counsellors perceived that because the SEIQoL supported the service user to engage in a process of self-reflection, this in turn facilitated the counselling process, through enabling the service user to ‘talk more openly’ and accelerating ‘the learning and insight the service user had about their drink problem’. 338 As one counsellor explained, the SEIQoL ‘helped the service user to look at whether they were ready to change [and] . . . decide what was important to them and . . . whether they were confident to get on and do it’. 338 Counsellors also felt that use of the SEIQoL prompted them to listen and reflect more on what the service user was saying. The counsellors also observed that use of the SEIQoL therefore helped them to ‘form a better relationship and trust with the service users’ and enabled them to ‘get alongside the service user more quickly’. 338 The counsellors attributed this to the fact the SEIQoL was ‘person centred’ and ‘person led’ and that it put the ‘control back with the service user’. 338

This was a very small study of the perceptions of only three counsellors in one service who were likely to be enthusiasts about the SEIQoL. The data are all gathered from counsellors and it is questionable how far their views reflect the feelings and experiences of their clients. The findings are overwhelmingly positive and read as a ‘good news’ story about the instrument. Nonetheless, the study provides useful insight into the mechanisms through which use of an individualised measure might enable service users to share their experiences with clinicians. In terms of the theories under test, this study provides some support for the theory that use of the SEIQoL facilitates the process of self-reflection and enables service users to prioritise their problems. In turn, this leads service users to engage more effectively with therapy and supports the relationship-building process with the counsellor.

Summary of findings from secondary care mental health services

These studies echo a number of the findings from the previous section examining the use of standardised depression questionnaires in primary care. They indicate that patients in secondary care also expressed concern that PROMs data would be used to restrict their access to services and that these worries increased the likelihood that they may ‘game’ their answers in order to avoid an unwanted outcome. They also suggest some ambivalence on the part of clinicians for the measures being used for multiple purposes; for example, clinicians did not see the measures as useful for monitoring service quality as they themselves received no feedback of these data.

The studies also suggest that PROMs completion may support patients to engage in self-reflection about their own condition, which can make them feel better prepared to see clinicians. There was also some evidence to suggest that PROMs completion may enable patients to track their own progress, which can motivate them to engage with the therapeutic process. However, this is only possible if patients also received feedback on their progress. The studies highlighted that longer, more comprehensive measures may better capture patients’ experiences but may also lead to lack of engagement with the measure, making it more likely that respondents would make the answers up. This may be a particular issue for younger people. They also raised questions about the extent to which standardised PROMs can fully capture the complex and dynamic nature of patients’ experiences. The studies also suggested that patients preferred to share information contained in PROMs when they had developed trust in the clinician caring for them. The study in which an individualised PROM was used suggested that its use supported this relationship-building process.

Hagelin et al.342

This study investigated palliative care nurses’ experiences of using a PROM within an inpatient palliative care service. Nurses were expected to give patients the European Organization for Research and Treatment of Cancer Quality of Life (EORTC QLQ-C30) questionnaire to complete within 2 days of admission. However, previous research by the same team found that the PROM was often not completed, especially for patients who had a shorter length of stay (indicative of shorter survival times). The study sought to understand reasons for this by asking nurses at the service for their views and experiences of using the instrument in this context. A questionnaire with six questions and room for open-ended responses was distributed to 36 nurses at the service and responses were received from 26 (72%).

Nurses’ experiences of using and attitudes towards the PROM varied; they reported ways in which the PROM had supported their interactions with patients but also identified some of the difficulties they felt patients experienced in completing the questionnaire. In relation to our theories under test, quotations suggest that a written PROM may make it more comfortable for patients to share information. The authors report that ‘Several nurses pointed to its value in identifying symptoms that might not have naturally been discussed without systematic assessment’,342 including pain, emotional and existential issues. As one nurse explained, this was because ‘it captures problems that patients might write down, but don’t speak verbally about’. 342 They also indicated that the PROM provided nurses with a structure for their discussions with patients so that they did not forget to ask about things; as one nurse described, ‘I become more structured, don’t forget important things needed to give good care’. 342

Some nurses described the PROM as supporting the therapeutic process with patients as it gave patients an opportunity to ‘highlight his or her problems and distress’. 342 They also felt that it was more useful when the PROM was completed in conjunction with a discussion with the patient; as one nurse observed, ‘it’s a good basis for a discussion about the patient’s situation’. 342 The authors note that nurses perceived the PROM to be ‘a positive complement but not . . . an alternative to other nursing assessments’. 342

However, nurses’ concerns about the use of the questionnaire also point to aspects that may constrain a patient’s willingness or ability to complete them. The authors report criticisms of the length, wording and format of the questionnaire. Some nurses ‘described it as difficult to complete because it was too extensive and time-consuming – “too many questions to fill in adequately” ’. 342 Nurses also reported ‘difficulties for patients in distinguishing between some items, difficulties with the time window related to the past week, as well as questions said to be “too intimate” for some patients and “too intellectual” for others’. 342 Some criticisms emphasise particular difficulties in using the questionnaire with palliative care patients. These included practical aspects of the format, ‘such as layout that was not tailored to the needs of this patient group, i.e. small print and many pages’, and concerns that the questionnaires was too demanding for these patients; one nurse reported that it was ‘too comprehensive for all patients to have the strength to complete’, and another said that ‘patients are too sick, even dying’. 342 Linked to this, the nurses responses point to a possible negative effect of completing PROMs for patients: ‘tired patients become more tired and irritable.’342

This study provides some evidence about processes through which completing PROMs might enable patients to share their concerns, and about conditions that might reduce accuracy and openness. It is not clear from this study whether the PROM was completed by patients independently or completed in conjunction with the nurses. We can speculate that a variety of different practices were used by nurses in this setting. This suggests that patients may be more at ease disclosing issues through a written PROM than verbally. Written PROMs can be a more comfortable format for sharing concerns and encouraging open responses. The findings also suggest that the use of the PROM can also make nurses’ enquiries more structured so that they do not forget to ask the patient about certain issues. However, this study raises questions about the extent to which patients’ views can be captured by questionnaires that are too demanding for them to complete, such that they are unable to provide complete answers. This ‘unintentional inaccuracy’ related to capacity to provide full responses may be a particular issue with detailed PROMs, inaccessible formats and patients with little energy. In these contexts, patients may be unable to complete PROMs in a way that adequately represents their views.

This was another small qualitative study in a single setting. The suggested value to patients of being able to write down concerns, and the concerns that patients may struggle to complete the questionnaire, are based on nurses’ perceptions. It is unclear how widely these views were shared among nurses in the study, or if they are shared by patients at the palliative care service.

Slater and Freeman343

This study examined the introduction of the Palliative Outcome Scale (POS) in a day hospice. Patients completed the POS questionnaires at the start of the day in a communal lounge, or took them home to complete. Once forms were collected, staff discussed any identified needs with patients during the day. A focus group was conducted with nine patients who had been using the POS for at least 3 months. Note that the PROMs implementation process for this study does not involve a scheduled consultation meeting, but rather nurses approach patients for discussion if concerns are raised on the POS.

Overall, patients felt that the POS was useful in helping them to communicate their individual needs. The study suggests that PROMs can provide a more comfortable format for patients to share their concerns. This is partly about the privacy of a written format, and partly about feeling more comfortable responding to direct questions.

In relation to privacy, one patient noted that:

It gives you a chance to write something that you don’t want the others to hear if you have got some emotional problem with the family or you know and you can write it down knowing that that would be followed up later during the day.

Participant 6343

A written PROM can allow patients to indicate concerns that they do not want other patients to know about.

Other patient comments point to the importance of format for enabling privacy: the questionnaire used in the study had a section on the front asking about the patient’s concerns. This was seen as insufficiently private, with one patient concerned about ‘having personal issues written on the front of the form for all to see’. 343

I think it’s too open as well for everyone else, for everybody to look at, I think that’s one reason I don’t fill it in . . . I never fill it in I never have done, it’s private.

Participant 8343

This could be addressed by changing the format, as the authors note: ‘The group displayed a high level of consensus to move the section identifying the patient’s own problems from the front of the form to the back, which is how it was formatted on the original POS form’. 343

In relation to question wording, patients felt that more direct questions may promote openness. The authors observed that ‘The group expressed very strong views that direct questions were needed on the POS forms on whether patients had had suicidal thoughts and feelings’. 343 Two quotations from patients are given to support this idea that patients may be more likely to share their concerns with direct questions, with a specific suggestion that needing to simply tick a box is more comfortable for patients:

It says have you felt sad, worried or angry it doesn’t specifically say outright . . . suicidal, and I think it should be put.

Participant 3343

It’s easier if you’ve just got to tick the box it’s easier than writing it down [you’re feeling suicidal] particularly for someone I would imagine who was in deep depression.

Participant 4343

Another participant also felt that being able to write down their problems was positive and useful:

It makes you think constructively about how you are feeling, and I find putting it into words comforting, just knowing there is someone who is going to read it, and in some cases has the answers.

Participant 7343

The participants discussed the freedom of being able to express their feelings through the POS form. This may enable patients to adapt to the implications of their illness. The skills of the professionals are paramount in helping patients to reflect on their individual situations at assessment and review. The POS may assist with this process.

However, although patients felt that more direct questions could promote openness, the findings also indicate that some patients found completing the form distressing owing to the phrasing of particular questions. In particular, there were different reactions to the question ‘Have you been feeling anxious or worried about your illness or treatment?’. For some, this question provoked offence and distress:

Well I mean when I read that question first I really felt upset and worried and angry I felt like saying well of course I do . . . [this] disease will finish me off what do you think I feel like.

Participant 3343

Indicating variation in reactions, however, another patient felt that this question allowed them ‘to express their frustration’:

I get angry and upset because I can’t do things, can’t do my buttons up it takes me half an hour to do my coat up, I mean there’s nothing anybody else can do.

Participant 5343

These different comments point to the importance of individual context, with different (positive and negative) reactions to the same PROM. The authors note that palliative care patients may adopt either a tolerating or an adapting coping style and that this may influence their engagement with PROMs completion:

Patients using an adapting style may feel more able to confront their problems, while those using a tolerating style may prefer the opportunity to escape from their sick role.

More direct questions may be emotionally beneficial for those adapting, but upsetting for those tolerating. This suggests one aspect of individual context that may affect patient reactions to the PROM and the suitability of direct questions.

Beyond these aspects related to whether or not the PROMs format means that patients feel comfortable about sharing information, the study also indicates that patients may sometimes misunderstand PROM questions. This could mean that the answers they provide do not accurately reflect their symptoms and concerns. For example, the authors explained that one change to the POS form ‘appeared to cause confusion, as participants reported not knowing how to answer these questions’.

A final point to note from this study is that the authors observed that the patient focus group had ‘a high level of consensus on physical, psychological, and communication aspects of care with less focus on social and existential aspects of care’. Reasons for the more limited attention to social and existential issues are not explored, but this may suggest that patients find the PROM more useful for identifying some of their needs (physical, psychological) than others.

This was a small study with just nine patients from the same palliative care unit. The authors indicate when views are shared by the group, providing some confidence that the quotations do not just reflect the perspective of one patient. However, additional evidence would help to test out some of the ideas raised by patients, for example looking at whether or not a change in PROMs format affects the information provided.

Although this is a small study, it provides further insight into ways in which PROMs might support patients to give an open account of their concerns and conditions that might affect this. In relation to our focus on patient responses at this stage of PROMs completion, the study suggests that one way PROMs can encourage openness from patients is when a written format provides more privacy. This may be a particularly important consideration in settings where verbal discussions may be overheard: the study took place in a day centre with a communal lounge, and privacy may be less significant in clinical settings with individual patient consultations. The study also points to aspects of context that enable this privacy, specifically a format that means information that patients consider to be private is not on open display.

The study also suggests more direct questions and tick-box formats can support openness from patients, by being more comfortable to answer. However, it points to a potential negative impact of direct questions, with some patients finding these upsetting. The authors’ discussion of patients who are ‘tolerating’ or ‘adapting’ suggests a useful aspect of individual context that may affect reactions to direct questions and, through this, potentially openness in their answers. The scope for varied reactions to particular questions points to the potential value of more individualised PROMs. A final issue raised by the study relates to patients’ ability to understand PROMs. If the format or wording mean that patients are uncertain about what information is needed or how to complete the PROM, the accuracy of answers may be reduced.

Summary of findings from palliative care

The studies provide useful data to examine how PROMs completion enables or constrains patients’ willingness to share issues with clinicians depends on the format of the PROM. Similar to the studies examining the use of PROMs in primary care, patients and clinicians held different views about the value of PROMs in palliative care. Patients saw the use of a PROM as a sign that someone was interested in their feelings and some patients felt that a written questionnaire format made it easier for them to share sensitive topics with clinicians. Other patients questioned the standardised nature of the PROM and felt that it did not always fully capture the complexity and dynamic nature of their problems. Furthermore, some palliative care patients were too ill to complete PROMs. It was also clear that for some patients PROMs completion was an emotional experience, as it forced them to confront issues they preferred to deny. The authors explained these findings in terms of differences in patients’ coping strategies: those who preferred to confront their problems may not find PROMs completion problematic, whereas those who preferred to deny their situation would be likely to find PROMs completion difficult. In circumstances in which they wish to continue to ignore these symptoms, patients are more likely to misrepresent their responses on the PROM. As such, PROMs completion may not be a panacea for enabling patients to come to terms with their situation.

In contrast, clinicians in palliative care felt that the standardised PROMs did not fully capture patients’ concerns, and perceived that patients were more likely to share their feelings and emotions with them verbally, and often incidentally, while clinicians were caring for patients, rather than through the completion of a PROM. Like the GPs in studies exploring PROMs in the care of patients with depression in primary care,80,81,336 clinicians in palliative care settings placed greater emphasis on verbal communication in building rapport and trust with patients. In some instances, nurses were reluctant to use the PROM, as they perceived that it may be detrimental to this relationship. In other instances, nurses changed or omitted the items to make the PROM more acceptable to patients. They felt that PROMs would not support patients to share their concerns unless they had developed this trusting relationship and that the PROM was most useful if used in conjunction with a discussion with the patient.

The next four studies reviewed provide a useful test of the intersection between PROMs format and trust in the clinician. Three studies focus on the use of PROMs by nurses in the initial assessment of patients referred to palliative care services. As such, these studies focus on the start of the process of relationship building between patient and clinician in a context in which the goal of care is maximising patients’ quality of life. Eischens et al. 347 and Gamlen and Arber348 examine how nurses used standardised PROMs in this context, while Annells and Koch244 explore nurses’ views on both standardised and individualised PROMs. In addition, we also review a study that explores the use of an individualised PROM, the SEIQoL-DW, in an oncology setting. 243

Summary of findings on contextual enablers of and barriers to use of patient-reported outcome measures

These four studies have allowed us to test and refine our theory about the ways in which the format of PROMs may support or constrain the process of relationship building with patients and, consequently, the sharing of information. Gamlen and Arber348 found that the use of a standardised PROM in initial assessments with palliative care patients could threaten the relationship-building process with patients if it is used before this important relationship building has taken place. In contrast, Annells and Koch244 found that the use of an individualised PROM supported this process by offering a means through which patients could tell their story to nurses in their own words and, consequently, share information that really mattered to them. We can hypothesise that because the individualised PROM allowed patients to identify issues in their life that were important to them and did not place restrictions on or standardise the ways in which patients were able to share their story, it more naturally fitted into the ways in which nurses interacted with patients at initial assessments. As Gamlen and Arber348 identified, nurses preferred to let patients share their stories in their own words; Annells and Koch244 found that the CGI enabled patients to do this and, therefore, this supported the relationship-building process. In contrast, the standardised PROM in Gamlen and Arber’s348 study, the SCC, narrowed the focus of this information-sharing process and thus threatened or disrupted nurses’ attempts to build relationships with patients. Furthermore, Kettis-Linblad243 also found that the completion of an individualised PROM prompted patients to reflect on their life situation and gave them permission to raise issues with clinicians. However, Annells and Koch244 also noted that the CGI was unlikely to be useful for follow-up assessments. The authors do not explain the rationale behind this judgement, but we can hypothesise that this may relate to the difficulties of using individualised instruments as measures of outcome identified by Farquar et al. 334 in terms of the consistency of specifying and then rating cues over time. This supports the theory that individualised measures may have more value as a ‘conversation opener’ than as an outcome measure.