The Arthroplasty Candidacy Help Engine tool to select candidates for hip and knee replacement surgery: development and economic modelling

Article history

The research reported in this issue of the journal was funded by the HTA programme as project number 11/63/01. The contractual start date was in November 2016. The draft report began editorial review in February 2017 and was accepted for publication in April 2018. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

Andrew Price reports personal fees from Zimmer Biomet, DePuy Synthes and Smith & Nephew plc, and grants from the National Institute for Health Research (NIHR) and Arthritis Research UK, outside the submitted work. Alastair Gray reports grants from NIHR, during the conduct of the study. Rachael Gooberman-Hill reports grants from the NIHR Health Services and Delivery Research programme for this work during the conduct of the study. Helen Dakin reports grants from NIHR during the conduct of the study and consultancy for Halyard Health outside the submitted work. David Beard reports grants from NIHR outside the submitted work. Jonathan Cook was a member of the NIHR Health Technology Assessment (HTA) Efficient Trial Designs Board (2014–16). Jill Dawson reports grants from the NIHR HTA programme during the conduct of the study and royalty payments from Oxford University Innovation (a university technology transfer company) outside the submitted work, and is one of the original developers of the Oxford Hip and Knee Scores. Raymond Fitzpatrick is one of the developers of the Oxford Hip and Knee Scores. Ashley Blom is the principal investigator in a research project funded by Stryker Corporation. Andrew Judge reports personal fees for consultancy from Anthera Pharmaceuticals, Inc., and Freshfields Bruckhaus Deringer LLP, outside the submitted work. Nigel Arden reports grants from Bioibérica and Novartis Pharmaceuticals UK Ltd, and personal fees from Bioventus, Flexion Therapeutics, Freshfields Bruckhaus Deringer LLP, Janssen Pharmaceutica, Merck & Co. Inc. and Regeneron Pharmaceuticals, Inc., outside the submitted work. David Murray reports grants from the NIHR HTA programme and grants and personal fees from Zimmer Biomet outside the submitted work.

Copyright statement

© Queen’s Printer and Controller of HMSO 2019. This work was produced by Price et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

2019 Queen’s Printer and Controller of HMSO

Research questions specified in the National Institute for Health Research Health Technology Assessment research call

• Can clinical tools for assessment of a patient’s suitability for knee or hip replacement be used to set thresholds for an operation?

• How does the choice of threshold affect the cost-effectiveness of the procedure and subsequent improvements in patient quality of life?

Background

Hip and knee osteoarthritis is a common musculoskeletal condition causing significant pain and loss of function for patients. Using patient-reported outcome measures (PROMs), joint replacement treatment for end-stage disease has been shown to be an effective treatment. 1 Each year, 150,000 hip and knee replacements are carried out in the UK, with the majority of patients having successful outcomes. 2 However, the nationally collected patient-reported outcome data for hip and knee replacements have identified two striking issues with regard to the provision of joint replacement in the UK. First, there is marked variation in current clinical practice in referring and undertaking surgery in patients with arthritis of the hip and knee. 3 Previous studies from the UK support this observation, with recent evidence showing that access to joint replacement is currently inequitable, with deprived areas associated with greater symptom severity and lower surgery rates. 4–6 A previous large national survey of UK NHS patients undergoing joint surgery7 also concluded that there was no evidence that patients were being prioritised on the basis of the severity of their symptoms and function. Second, the national outcomes data have revealed that 10–15% of patients undergoing hip or knee joint replacement are not satisfied with their treatment, and these findings, particularly for the knee patients, are supported by other recent studies. 8,9 It has been suggested that selecting patients too early in their disease process may play a role in producing dissatisfaction with surgery. 9 Overall, these findings suggest that there is no standardisation to the process by which patients are assessed and selected for hip and knee replacement surgery. This is a particular concern given both the projected increased need for joint replacement over the next decade to accommodate an ageing population and the pressure of potential reductions in NHS funding. 10

Assessing patients for joint replacement surgery within the NHS is generally a two-stage process that begins with the patient presenting to a general practitioner (GP) with hip or knee pain (Figure 1). The assessment process usually takes place over a number of consultations, often including an radiography to confirm osteoarthritis. The GP continues to monitor symptoms and responses to non-operative treatments, eventually deciding when the patient is a candidate for joint replacement surgery, and at this point the patient is referred to secondary care. Currently, there are no widely accepted guidelines within the NHS specifically to help health professionals estimate the level of patient symptoms that warrants referral. 11 In the secondary care setting, the specialist assesses the patient, confirms the diagnosis and radiographic severity, reviews a patient’s symptoms and shares information with them about available treatment options. For each patient, the decision to offer surgery requires a personalised assessment of individual preferences, expectations, functional limitations and requirements, degree of radiographic pathology, comorbidities and predicted outcome. 11 Ideally, the patient and their specialist then make a shared decision, with the patient ultimately deciding whether or not to undergo surgery. We estimate that there are around 1 million GP patient assessments for hip and knee pain each year, with around one-quarter of patients who present being referred to a specialist. Although only a small proportion of these patients receive joint replacement, this amounts to 150,000 procedures a year. 12,13 At the Nuffield Orthopaedic Centre (NOC), our data suggest that approximately 15% of patients who are referred from primary care for possible joint replacement do not have symptoms at a level that requires joint replacement and could have been safely managed without an appointment in secondary care.

FIGURE 1.

Patient pathway framework for the NHS to identify candidates for hip and knee replacement. The selected assessment score may also be used later in the pathway to measure the outcome of surgery and to offer surveillance for joint replacement post surgery. THA, total hip arthroplasty; TKA, total knee arthroplasty.

Given the issues of unwarranted variation and poor outcome in some patients, outlined above, there has been significant interest in trying to standardise the process of referral and selection for joint replacement. The use of certain ‘priority criteria’ (such as the Western Canada Waiting List score,14 the Ontario criteria15 or the New Zealand score16) has been investigated as a more consistent method of selecting patients for referral and treatment. These tools identify candidates for surgery in primary care and are based on estimating a patient’s capacity to benefit from surgery. They are generic and attempt to standardise the patient pathway for joint replacement at the entry point. The New Zealand priority criteria16 have been used in some regions within the NHS but have not reached widespread acceptance, and the current evidence of their reliability and validity is minimal. 17,18 Other tools have been developed but not fully tested in clinical practice within the UK. 19–21 The Osteoarthritis Research Society International (OARSI) Standing Committee for Clinical Trials Response Criteria Initiative and the Outcome Measures in Rheumatology (OMERACT) international initiative has attempted to deliver a standardised approach and has highlighted pain and disability as among the key domains for identifying the capacity to benefit. 22–25 It has therefore been a logical progression to investigate if existing assessment tools used in the joint replacement pathway, that measure pain and disability, could be used as a single score to identify candidates for surgery by referring their preoperative assessment score to a threshold for intervention.

In 2009, the Department of Health and Social Care introduced the routine collection of PROMs for hip and knee surgery to measure the outcome of surgery undertaken in NHS hospitals. 8,26 There has been government support for extending the use of scoring systems preoperatively to create thresholds for referral and candidacy for surgery. 27,28 In fact, many primary care trusts and NHS trusts have already introduced PROM-based severity score thresholds for surgery, although the thresholds used vary widely between regions. 29–36 However, evidence underpinning and endorsing the use of PROMs or any assessment score for thresholds is scant and without validation. This poses a significant risk to patients as an incorrectly set threshold may unfairly restrict access to care or, conversely, inappropriately select patients for joint replacement. 2 The development of a preoperative threshold score to identify candidates for hip and knee replacement offers a significant opportunity to standardise the patient pathway. However, this Health Technology Assessment (HTA) call reflects the pressing need within the NHS to produce evidence to support or refute their use.

A number of scoring systems are used to assess the patients in their care pathway. Many are PROM based, such as the Intermittent and Constant Osteoarthritis Pain Measure (ICOAP),37 the EuroQol-5 Dimensions (EQ-5D) and the Oxford Hip and Knee Scores,38–40 whereas others require a clinician’s involvement (e.g. the New Zealand score16). Some systems were designed to measure the burden of osteoarthritis symptoms [e.g. the Western Ontario and McMaster Universities Arthritis Index (WOMAC^®) or ICOAP],41 whereas others were designed to measure the effect of an intervention (e.g. Oxford Hip and Knee Scores). Some scores were produced to measure more general aspects of health status [Short Form questionnaire-36 items (SF-36) or EQ-5D]42 and others aim to prioritise patients for surgery (e.g. the New Zealand priority criteria16). None of these scores has been developed for the specific role of applying thresholds for access to care for joint replacement within the setting of the NHS. It may be that one or more of these scores may be appropriate for such use but evidence is required to validate and justify this role.

To be fit for purpose as a screening device, any candidate score must satisfy a number of requirements.

First, the score must have adequate measurement properties to enable assessment of patients for joint replacement, namely adequate validity. This includes evidence of adequate reliability at an individual level (test–retest and intraclass correlation coefficient), precision [standard error (SE) of the measurement] and responsiveness to change [smallest detectable change and minimally clinically important difference (MCID)]. The effect of comorbidity on the score must also be established.

Second, valid evidence-based thresholds must be produced. The calculation of thresholds is not straightforward, with several different methods available. To generate upper thresholds (i.e. least severity) in preintervention scores, above which patients should not be considered candidates for arthroplasty, methods must account for the likelihood of a patient’s capacity to benefit (i.e. likelihood of achieving a positive change score) and perceive satisfactory improvement following surgery. 43 The measurement properties of the instrument (as described previously), such as the MCID (i.e. the smallest amount of change in a score that patients detect and consider important) and standard error of the mean (SEM) (which relates to the reliability of the instrument and denotes the amount of change that is ‘real’ and beyond measurement error), are also important operational considerations when calculating thresholds. Furthermore, any chosen threshold must distinguish between cases (patients in need of surgery) and non-cases with a consistent level of diagnostic accuracy (discriminative ability).

The process of calculating absolute thresholds will also produce additional and valuable information for patients who are found to be candidates for surgery. By highlighting an individual’s ‘chance’ of benefit following surgery (based on their preoperative score), patients are provided with key information to help with their decision-making, particularly in secondary care. It would provide evidence to support the use of a score embedded within the NHS direct knee/hip osteoarthritis decision aid. Hence, clearly highlighting the risks and benefits may make the decision to have surgery clearer for many patients. This type of information allows patients to more comprehensively participate in the decisions made about their care.

Third, we must understand how the introduction of thresholds for surgery affects the cost-effectiveness of the treatment. Lower-limb joint replacement has previously been shown to be highly cost-effective, costing between €1276 and €18,300 per quality-adjusted life-year (QALY) gained for the average patient,44–48 which is substantially lower than the £20,000–30,000 per QALY range that the National Institute for Health and Care Excellence (NICE) considers to be cost-effective for use within the NHS. 49 However, it is important for commissioners of hip and knee replacement surgery to understand how cost-effectiveness varies between patient and procedure subgroups, and how thresholds for hip and knee surgery affect the cost utility of the interventions. We have recently demonstrated the feasibility of this approach in a pilot study exploring the relationship between costs and improvements in EQ-5D utility and preoperative PROM scores in total knee arthroplasty (TKA). 50

Finally, having identified and validated a clinical tool and calculated valid and evidence-based thresholds for surgery, within the NHS, it must be established whether or not the tools are acceptable to the ‘end-users’. Despite some thresholds for hip and knee replacement having already been introduced to clinical practice in parts of the country, there has been little or no engagement with the wider stakeholders about the appropriateness of this approach or how thresholds should be used in practice. The introduction of thresholds requires the support of patients, health-care professionals and commissioners.

Although the requirements of threshold scores in primary and secondary care may differ, in order to provide consistency for patients and health-care professionals any scoring system would ideally be applicable to both sectors. One aim is to ensure this compatibility by consciously considering the requirements within each setting. In primary care, the requirement is for a simple-to-use patient-based score linked to the patient’s potential to benefit from arthroplasty. This would provide a distinct upper threshold for referral and candidacy for joint replacement. The thresholds calculated for the identified scoring system will be incorporated into a user-friendly knee and hip replacement candidacy assessment tool – the Arthroplasty Candidacy Help Engine (ACHE) tool. Secondary care involves more complex assessments, involving expectation, comorbidity and age-related factors. The ACHE tool would be a starting point for secondary care assessment, linking to other patient decision-support tools. 51,52

In summary, greater standardisation is required in the patient pathway leading to hip or knee joint replacement surgery. The aim of this study is to develop an evidence-based method for identifying patients in primary care who are possible candidates for surgery, using valid thresholds applied to scoring systems that are already available (see Figure 1).

Research objectives

The following research objectives will be met:

• Create a shortlist of scoring systems that are potentially useful for selecting candidates for arthroplasty surgery.

  • From the literature, establish the scores/instruments available. Published evidence concerning their measurement properties, and their past or projected use in setting thresholds for hip and knee replacement, will be reviewed. This will generate a shortlist of potential scoring systems.

  • Using existing data sets and guidance from users, refine the shortlist by establishing the necessary measurement properties of potential scores/instruments when not available in the literature.

• Identify a scoring system, and a set of threshold values, to be used to select candidates for hip and knee surgery.

  • For each shortlisted instrument, determine score thresholds for candidacy for joint replacement surgery.

  • Determine the relationship between threshold levels and cost-effectiveness of hip and knee arthroplasty surgery.

  • Select the most applicable single score and set of thresholds for incorporation into the ACHE tool.

• Explore the clinical effectiveness and cost-effectiveness of the ACHE tool and determine the potential acceptability of the tool and thresholds to stakeholders and patients.

  • Determine the effect of using the ACHE tool on patterns of referral of hip and knee patients to secondary care.

  • Evaluate user opinion – GPs and patients.

  • Engage with a wider stakeholder group to assess the acceptability of the ACHE tool.

The overall aim was to develop a standardised NHS framework for identifying patients for hip and knee replacement surgery using safe and equitable thresholds. This was achieved by creating the ACHE tool, based on a currently available assessment score, with thresholds that take account of patients’ capacity to benefit from surgery and the cost-effectiveness of the treatment. The new system is applicable in both primary and secondary care.

The user group

From the outset of the original design for this project, all service users’ perspectives were considered integral to its success. The ACHE tool was to be designed to help patients, GPs, secondary care personnel, arthroplasty specialists and health-care commissioning staff. It was therefore decided very early on to utilise a ‘user group’ concept, in which representatives of these identified bodies inputted, critiqued and reviewed the progress of the study at appropriate intervals. Importantly, the user group was established as a proactive entity rather than as a passive and disengaged review and agreement exercise. The decisions of the group were critical to the direction of the project and were instrumental in sanctioning various aspects/decisions and vetoing others. The user group was given full autonomy under the direction of the chairperson. Investigators of the study were allowed to be present but did not participate in the meeting unless invited by the chairperson for clarification purposes only. As a result, the final ACHE tool did not reflect the wishes of the researchers, but the group for whom the instrument was designed.

The user group was assembled and provided input at regular and preset intervals during the course of the study. The sequence was predetermined and the role dovetailed with each stage of the project (Figure 2). One option for this report was to describe user group input in separate sections (chapters) in sequence and inserted within the main report at the appropriate temporal intervals. However, some meetings were introductory or had limited remit and do not contain sufficient content to justify separate chapters. A decision was made to report all user group activity in a single separate chapter (see Chapter 10). Readers of the report are required to cross-reference this section and appreciate that user group input took place for each academic section.

FIGURE 2.

Work plan schema highlighting the user group meetings and input. HES, Hospital Episode Statistics; NJR, National Joint Registry; WP, work package.

Background

The aim of this study was to develop an evidence-based system for identifying patients who might be candidates for hip or knee replacement surgery, introducing valid thresholds based on scores that are already available. The first objective in achieving this aim was to create a shortlist of scoring systems that could be used in this way. After discussion within the user group (see Chapter 10), it was established that candidate scores placed on the shortlist would need to meet certain essential criteria:

• A score must be a patient-reported measure to ensure that patients were engaged in the assessment process and that the score used reflected their perspective on the outcome.

• A score must demonstrate adequate measurement properties and have been validated within the hip and knee replacement populations. 53–58

Many different scoring systems and outcome measures have been used for assessing the outcomes of hip or knee arthroplasty, but not all measures have evidence of, or reach, even the minimum psychometric standards for their proposed uses. 1,59–61 Therefore, the aims of this work were to use systemic review methodology to identify and evaluate English-language versions of PROMs that have been evaluated with patients undergoing hip or knee replacement surgery and to provide a comprehensive profile of their measurement properties so that a shortlist of candidate scores could be established.

Methods

Results

Identification of studies

The initial search in Ovid yielded 3774 abstracts. After the removal of duplicates, the number of abstracts for assessment was 2887. In addition, keyword searches (combination of knee, hip and orthopaedics) in EconLit yielded 162 results, the PROMs database identified 454 results and DARE had no results (Figure 3).

FIGURE 3.

Instrument flow diagram. AMED, Allied and Complementary Medicine Database; EconLit, American Economic Association’s electronic bibliography; OU, University of Oxford. Reproduced with permission from Harris et al. 63 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 3.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/3.0/.

Reproduced with permission from Harris et al. This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 3.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/3.0/.

Hand-searching of titles of the following key journals in the 6 months preceding the search was conducted:

• Health and Quality of Life Outcomes (number of articles, one)

• The Journal of Bone and Joint Surgery (American and British volumes; number of articles, one)

• The Journal of Arthroplasty (number of articles, three).

Screening of articles and instruments

Out of the 167 selected abstracts, 146 eligible full-text articles were then screened for all PROMs that were analysed, identifying 135 instruments. If the instrument was not validated (developed for or subsequently validated) for use in a population of patients undergoing hip or knee replacement surgery, it was excluded, leaving 67 instruments. A reliability exercise was undertaken for 16 full-text articles between two reviewers, and the agreement was 95% (38/40 questionnaires identified). An instrument-specific search was then undertaken for each of the 67 identified instruments. By this method, 21 new validation papers (in addition to 42 developmental papers) in the targeted population were identified. Furthermore, on closer examination of shortlisted instruments, 21 initially identified instruments were additionally excluded.

Data extraction

Relevant data on the psychometric performance and operational characteristics were extracted for each PROM. The summary texts were sent to corresponding authors from the developmental study of each respective PROM, and further information was added as a result of this exercise. The appraisal summaries are presented in Tables 1–4.

TABLE 1 - Hip and knee scores

HOOS, Hip Disability and Osteoarthritis Outcome Score; HRQ, Hip Rating Questionnaire; PSI, patient-specific index; VAS, visual analogue scale.

Notes

Psychometric and operational criteria: 0 = not reported; – = no evidence in favour; + = some limited evidence in favour; ++ = some good evidence in favour; +++ = good evidence in favour; +/– = mixed evidence.

Reproduced with permission from Harris et al. 63 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 3.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/3.0/.

Instrument (groups tested)	Score
	Hip				Knee
	HOOS	HRQ	PSI	OHS	Knee disorders subjective history (VAS)	KOOS	KOOS-PS	OKS	OKS-APQ
Number of studies	5	1	4	20	1	3	2	23	1
Reproducibility	++	+	+	++	0	+	0	+++	+++
Internal consistency	+	0	0	++	0	0	+++	+++	+++
Validity: content	0	0	++	++	+	+	+	+++	+++
Construct	++	+	++	+++	+	+	++	+++	+++
Responsiveness	+	+	++	+++	0	0	++	+++	+++
Interpretability	0	0	0	+++	0	0	0	++	0
Floor and ceiling/precision	+	0	0	–/+	0	+	0	++	++
Acceptability	0	0	0	+++	–	0	0	+++	+++

TABLE 2 - Lower limb and pain scores

AAOS, American Academy of Orthopaedic Surgeons; h, hip; k, knee; LEFS, Lower Extremity Functional Scale; MODEMS-HQ, Musculoskeletal Outcome Data Evaluation and Management System Hip and Knee Core Scale; N/A, not applicable; WOMAC-SF, WOMAC Short Form.

Notes

Psychometric and operational criteria: 0 = not reported; – = no evidence in favour; + = some limited evidence in favour; ++ = some good evidence in favour; +++ = good evidence in favour; +/– = mixed evidence.

Reproduced with permission from Harris et al. 63 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 3.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/3.0/.

Instrument (group tested)	Score
	Lower limb							Pain
	LEFS (h/k)	WOMAC (h/k)	WOMAC (h)	WOMAC (k)	WOMAC SF (h/k)	Lower limb core score (h/k)	MODEMS-HK (AAOS) hip and knee core score (h/k)	ICOAP (h/k)	P4 (h/k)	McGill pain-short form (h/k)
Number of studies	5	25	N/A	N/A	N/A	1	1	2	1	2
Reproducibility	+	++	++	+	0	0	0	+	0	++
Internal consistency	+	+	0	0	+	0	0	+	++	0
Validity: content	+	+	+	+	+	+	+	++	+	0
Construct	++	+++	+	++	++	0	+	+	+	+
Responsiveness	++	+++	++	++	+	0	++	–	0	–
Interpretability	+	++	++	++	0	0	0	0	0	0
Floor and ceiling/precision	0	–/+	–	0	0	0	++	0	0	0
Acceptability	0	++	+	+	0	0	+	0	0	0

TABLE 3 - Utility and generic scores

h, hip; HUI2, Health Utilities Index Mark 2; HUI3, Health Utilities Index Mark 3; k, knee; N/A, not applicable; SF-6D, Short Form questionnaire-6 Dimensions; SF-12, Short Form questionnaire-12 items; SIP, sickness impact profile.

Notes

Psychometric and operational criteria: 0 = not reported; – = no evidence in favour; + = some limited evidence in favour; ++ = some good evidence in favour; +++ = good evidence in favour; +/– = mixed evidence.

Reproduced with permission from Harris et al. 63 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 3.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/3.0/.

Instrument (group tested)	Instrument group
	Utility					Generic
	SF-6D (h)	HUI2 and HUI3 (h)	EQ-5D (h/k)	EQ-5D (h)	EQ-5D (k)	SF-36 (h/k)	SF-36 (h)	SF-36 (k)	SF-12 (h/k)	SF-12 (h)	SF-12 (k)	SIP (h)
Number of studies	1	4	9	N/A	N/A	14	N/A	N/A	3	N/A	N/A	2
Reproducibility	0	0	0	0	0	0	0	0	++	0	0	0
Internal consistency	0	0	N/A	N/A	N/A	0	0	–	0	0	0	0
Validity: content	0	0	0	0	0	0	0	0	0	0	0	+
Construct	0	++	+	0	+	+	0	+	0	0	+	+
Responsiveness	++	+	0	0	+	0	++	+	0	+	+	–
Interpretability	0	0	0	++	++	0	+	+	0	+	+	0
Floor and ceiling/precision	–	0	0	0	++	0	–	0	+++	0	0	–
Acceptability	0	0	0	0	++	0	0	0	0	0	0	0

TABLE 4 - Other scores

Aberdeen IAP, Aberdeen Impairment, Activity Limitation, and Participation Restriction; AQOL, Assessment of Quality of Life; h, hip; HAQ, Health Assessment Questionnaire; k, knee; K10, The Kessler Psychological Distress Scale; MHAQ, Modified Health Assessment Questionnaire; MSK, musculoskeletal; N/A, not applicable; NEADL, Nottingham Extended Activities of Daily Living.

Notes

Psychometric and operational criteria: 0 = not reported; – = no evidence in favour; + = some limited evidence in favour; ++ = some good evidence in favour; +++ = good evidence in favour; +/– = mixed evidence.

Reproduced with permission from Harris et al. 63 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 3.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/3.0/.

Instrument (group tested)	Instrument
	WHOQOL-BREF (h/k)	Aberdeen IAP (h/k)	Aberdeen IAP (modified) (h/k)	NEADL (h)	AQOL (h/k)	MSK functional limitations index (k)	HAQ (k)	MHAQ (h/k)	MHAQ (h)	K10 (h/k)
Number of studies	1	1	1	1	2	1	2	2	N/A	1
Reproducibility	0	0	0	++	0	0	0	0	0	0
Internal consistency	++	+	++	++	0	0	–	0	0	0
Validity: content	+	+	0	–	0	0	0	0	+	–
Construct	0	+	+	+	+	+	++	+	+	+
Responsiveness	+	0	0	–	++	0	–	–	+	–
Interpretability	0	0	0	0	0	0	0	0	0	0
Floor and ceiling/precision	++	+	0	+	0	0	+	++	0	++
Acceptability	0	0	0	0	0	+	0	0	0	0

Table 1 summarises the evidence of measurement and operational performance applying the adapted appraisal criteria for the hip PROMs identified in this review. On the basis of the volume and quality of evidence, the Oxford Hip Score (OHS) clearly has the best evidence of measurement properties within the hip-specific PROM category. Within the ‘knee scores’ subgroup (see Table 1), the Oxford Knee Score (OKS) [with the OKS – Activity and Participation Questionnaire (OKS-APQ)] demonstrated best evidence of its measurement properties within the knee-specific PROM category. The Knee injury and Osteoarthritis Outcome Score (KOOS) and the KOOS – Physical Score (KOOS-PS) have some favourable evidence of their measurement properties, although compared with the OKS, the evidence is lacking and further evaluations are needed.

Table 2 summarises the evidence of measurement and operational performance by applying the adapted appraisal criteria to the lower-limb and pain PROMs identified in these reviews. The best-performing lower-limb measure for hip/knee patients is the WOMAC, followed by the Lower Extremity Functional Scale. The WOMAC also performed best when applied to separate hip or knee groups. Satisfactory evidence of measurement properties was generally lacking for all of the three identified pain measures (ICOAP, P4 and the McGill Pain-Short Form). ICOAP and McGill Pain-Short Form had no evidence in favour of their responsiveness and P4 did not have any reported evidence of its responsiveness. Three utility and generic measures identified in the review are listed in Table 3. As with the pain scores, the evidence for utility PROMs was generally lacking, with the EQ-5D scoring worse on construct validity and responsiveness than the Short Form questionnaire-6 Dimensions (SF-6D) and the Health Utilities Index Mark 2 (HUI2) and Mark 3 (HUI3). On the basis of the volume and quality of evidence, among all identified generic measures, the Short Form questionnaire-12 items (SF-12) is clearly the most promising one.

Nine measures identified in the review were categorised as ‘other’ scales. Table 4 summarises evidence of their measurement properties. The World Health Organization Quality of Life (WHOQOL)-BREF instrument, Aberdeen Impairment, activity limitation and participation restriction [Aberdeen Impairment, Activity Limitation, and Participation Restriction (Aberdeen IAP)] and assessment of quality of life had the best overall evidence in this subcategory (on a mixed hip/knee population). However, the overall evidence of their validity was generally lacking.

Discussion

Our review has identified the WOMAC, OHS and OKS to be the most promising disease-/site-specific scores that perhaps provide best coverage of the construct of interest and better responsiveness. The best-performing generic measure was the SF-12. However, further research on some of the missing measurement properties in these measures is required. For the WOMAC, further evidence on ceiling/floor effect, content validity and acceptability is required in both the hip and the knee groups of patients. The OHS is currently lacking evidence on its ceiling/floor effects. Many other PROMs do not have sufficient measurement property validation to recommend their use. Given its widespread use in this clinical area (e.g. national PROMs data), it was disappointing that the EQ-5D score did not perform better.

Our findings are supported by existing literature. Alviar et al. 60 published a systematic review of measurement properties of 28 PROMs used in hip/knee arthroplasty based on published evidence up to December 2009 and found the WOMAC, OKS and SF-36 to be the most comprehensively tested measures at that time, although the need for more rigorous evaluation of reliability, responsiveness and interpretability was noted. Our review has updated this evidence, both in breadth (we have assessed 67 instruments) and time period (our search was until May 2014). Browne et al. 64 identified the OHS and OKS (used alongside the EQ-5D) as primary outcome measures of choice to be used in the UK PROMs programme for hip and knee replacement.

It should be noted that the standards (and indeed scope/tolerance) for reporting details of qualitative procedures and psychometric analysis have changed over the past 20 years (very much so in the musculoskeletal literature), so that although measures that were devised earlier in that period have had a longer time in which to accrue evidence of their measurement properties, they can frequently lack relevant detail specifically in relation to the development of the instrument. Reporting has improved, probably as a consequence of the evolving methods and the recognition that minimum standards are required [e.g. Streiner et al. ,65 COSMIN (COnsensus-based Standards for the selection of health Measurement INstruments)54 and the US Food and Drug Administration55].

Further detail and supplementary material can be found in the publication based on this work by Harris et al. 63

Background

The systematic review of the measurement properties reported the properties for the EuroQol-5 Dimensions, three-level version (EQ-5D-3L), SF-12, OHS, OKS and WOMAC tools based on the existing literature. The OHS (20 studies) and OKS (23 studies) are good in terms of reproducibility, internal consistency, validity (content), construct, responsiveness, interpretability, floor and ceiling effects, precision (except OHS) and acceptability. WOMAC (25 studies) was reported as good in terms of reproducibility, validity (content), construct, responsiveness, interpretability and acceptability. Only fair outcomes for knee for EQ-5D-3L (nine studies) were reported in terms of interpretability and acceptability when construct and responsiveness were not applicable. SF-12 (three studies) was poor in terms of construct, responsiveness and interpretability. There were a number of outcomes for which there was no or little available evidence on one of more of the measurement properties. To be fit for purpose, any candidate score to be used as a screening instrument must satisfy a number of requirements, one being that the score must have adequate measurement properties to enable assessment of patients for joint replacement {i.e. adequate validity [ACHE protocol version 4, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences (NDORMS), 2015]}.

Methods

Results

Interpretability

Minimal detectable change (90% significance level)

Literature-based minimal detectable change

Minimal detectable change (90% significance) was calculated using the literature review ICC values presented in Table 16. The SF-12 PCS, assuming an ICC of 0.84, had ± 7–8 MDC points. The MDC values were 10–12 points for the SF-12 MCS with an ICC of 0.80 for hip and knee. 78 MDC values for the WOMAC pain, physical function and stiffness for hip and knee using ICCs of 0.73, 0.78 and 0.53, respectively, ranged from 20–27, 20–27 and 23–39, respectively. 79

TABLE 16 - Minimally detectable change (90%): literature-based ICCs for hip and knee measurement tools

ADAPT, Assessing Disability After Partial and Total Joint Replacement.

Measurement tool	Pre operation
	n	ICC	MDC 90%
Hip
ADAPT
SF-12 PCS	72	0.84	±7.94
SF-12 MCS	72	0.80	±9.55
ADAPT
WOMAC pain	110	0.73	±26.60
WOMAC function	110	0.78	±24.27
WOMAC stiffness	110	0.53	±38.71
APEX
WOMAC pain	278	0.73	±22.71
WOMAC function	209	0.78	±20.64
WOMAC stiffness	255	0.53	±36.60
EUROHIP
WOMAC pain	1255	0.73	±21.55
WOMAC function	1253	0.53	±26.67
WOMAC stiffness	1266	0.78	±22.62
Knee
ADAPT
SF-12 PCS	65	0.84	±6.67
SF-12 MCS	65	0.80	±9.54
KAT
SF-12 PCS	2087	0.84	±7.60
SF-12 MCS	2087	0.80	±11.99
ADAPT
WOMAC pain	109	0.73	±22.89
WOMAC function	109	0.78	±19.87
WOMAC stiffness	108	0.53	±32.34
APEX
WOMAC pain	269	0.73	±20.20
WOMAC function	187	0.78	±19.54
WOMAC stiffness	242	0.53	±32.21

Minimal detectable change using assumed intracluster correlation coefficient values

Reported ICC values from previous studies reflect the limitations of those studies in terms of population and precision. The arbitrary ICC figures of 0.5, 0.7 and 0.9 that were also used to calculate MDCs are reported in Table 17 for hip and knee scores.

TABLE 17 - Minimally detectable change (90%): assumed ICC values for hip and knee measurement tools

ADAPT, Assessing Disability After Partial and Total Joint Replacement.

Measurement tool	Pre operation
	n	MDC (90%)
		ICC 0.5	ICC 0.7	ICC 0.9
Hip
EUROHIP
EQ-5D-3L index	1228	±0.54	±0.42	±0.24
APEX
EQ-5D-3L index	250	±0.54	±0.42	±0.24
ADAPT
SF-12 PCS	72	±14.04	±10.87	±6.28
SF-12 MCS	72	±15.11	±11.70	±6.76
EPOS
OHS	1517	±13.12	±10.16	±5.87
EUROHIP
OHS	127	±13.10	±10.15	±5.86
ADAPT
WOMAC total	110	±35.00	±27.11	±15.65
WOMAC pain	110	±36.20	±28.04	±16.19
WOMAC function	110	±36.59	±28.34	±16.36
WOMAC stiffness	110	±39.92	±30.92	±17.85
APEX
WOMAC total	200	±29.47	±22.83	±13.18
WOMAC pain	278	±30.90	±23.93	±13.82
WOMAC function	209	±31.11	±24.10	±13.91
WOMAC stiffness	255	±37.75	±29.24	±16.88
EUROHIP
WOMAC total	1243	±26.41	±20.46	±11.81
WOMAC pain	1255	±29.33	±22.72	±13.12
WOMAC function	1253	±27.51	±21.31	±12.30
WOMAC stiffness	1266	±34.10	±26.41	±15.25
Knee
KAT
EQ-5D-3L index	2120	±0.51	±0.39	±0.23
APEX
EQ-5D-3L index	248	±0.51	±0.39	±0.23
KAT
SF-12 PCS	2087	±13.43	±10.40	±6.01
SF-12 MCS	2087	±18.96	±14.68	±8.48
ADAPT
SF-12 PCS	65	±11.78	±9.13	±5.27
SF-12 MCS	65	±15.08	±11.68	±6.74
KAT
OKS	2112	±12.40	±9.60	±5.54
ADAPT
WOMAC total	108	±28.68	±22.22	±12.83
WOMAC pain	109	±31.15	±24.13	±13.93
WOMAC function	109	±29.96	±23.20	±13.40
WOMAC stiffness	108	±33.36	±25.84	±14.92
APEX
WOMAC total	174	±27.06	±20.96	±12.10
WOMAC pain	269	±27.49	±21.29	±12.29
WOMAC function	187	±29.45	±22.82	±13.17
WOMAC stiffness	242	±33.23	±25.74	±14.86

Hip

Using an ICC of 0.9 provided MDCs of 0.24 for the EQ-5D-3L index, 6 points for the OHS total score, 6 and 7 for the SF-12 PCS and MCS, respectively, and 12–16 points for the WOMAC total score across the data sets. MDCs with ICCs of 0.5 and 0.7 were substantially larger, as would be anticipated.

Knee

Using an ICC of 0.9 provided MDCs of 0.23 for the EQ-5D-3L index, 5 and 6 for the SF-12 PCS and MCS, respectively, 6 points for the OKS total score and 13 for the WOMAC subscales across the data sets. MDCs with assumed ICCs of 0.5 and 0.7 were substantially larger, as would be anticipated.

Minimally important change and minimally important difference

Hip

A suitable anchor was available in two data sets: EUROHIP (EQ-5D-3L, OHS and WOMAC) and EPOS (OHS) and applied to the total score only. In EUROHIP, 244 patients answered for the satisfaction question after operation for EUROHIP. Of these, 59 patients (24%) answered ‘somewhat satisfied’ and 13 patients (5.3%) answered ‘somewhat dissatisfied’. These ‘somewhat satisfied’ versus ‘somewhat dissatisfied’ groups were used for the anchor-based analyses in this report when there was no neutral scale. In EPOS, 1053 patients answered the satisfaction question after operation. Of these, 167 patients (16%) answered ‘somewhat satisfied’ and 39 patients (4%) answered ‘somewhat dissatisfied’. These ‘somewhat satisfied’ versus ‘somewhat dissatisfied’ groups were used for the anchor-based analyses in this report.

Figure 4 shows the EQ-5D-3L index ROC curve for the EUROHIP data. Table 18 gives the pre and post operative values in accordance with the anchor, and Table 19 gives the area under the curve (AUC), MIC (ROC), MIC (group), MIC (ES), MID (group) and MID (ES) estimates, along with the sensitivity and specificity for the optimal cut-off point. The AUC was 0.69. The MIC ROC values were 0.07 for both the Youden and the shortest distance methods. MIC (group) and MID (group) were much larger, at 0.36 and 0.28 points, respectively. The values for MIC (ES) and MID (ES) were both around 1.00.

FIGURE 4.

The EQ-5D-3L index: EUROHIP data set ROC curve.

TABLE 18 - The EQ-5D-3L index by satisfaction for the EUROHIP data set

Note

Change score was calculated as postoperative minus preoperative score.

Score	Satisfaction
	Somewhat satisfied			Somewhat dissatisfied
	n	Mean score	SD	n	Mean score	SD
Preoperative	56	0.33	0.32	13	0.34	0.33
Postoperative	59	0.69	0.25	13	0.42	0.32
Change	56	0.36	0.29	13	0.08	0.33

TABLE 19 - The EQ-5D-3L index MIC/MID by satisfaction for the EUROHIP data set

MIC						MID
Individual-level MIC (ROC analysis)				Scale	MIC
AUC (95% CI)	Optimal cut-off point	Sensitivity	1 – specificity
0.69 (0.52 to 0.85)
Youden’s index	0.07	0.79	0.46	Group	0.36	0.28
Shortest distance	0.07	0.79	0.46	ES	1.26	0.97

Owing to the low number of OHS observations in the EUROHIP data set (Table 20), in which the satisfaction was ‘somewhat satisfied’ and ‘somewhat dissatisfied’, the MIC ROC was not calculated in this report when there was no neutral scale.

TABLE 20 - The total OHS, by satisfaction for the EUROHIP data set

Note

Change score was calculated as postoperative minus preoperative score.

Score	Satisfaction
	Somewhat satisfied			Somewhat dissatisfied
	n	Mean	SD	n	Mean	SD
Preoperative	11	14.36	3.61	3	13.33	10.50
Postoperative	11	30.50	7.99	4	26.50	3.32
Change	10	16.14	6.09	3	13.17	7.12

Figure 5 shows the OHS ROC curve for the EPOS data set. Table 21 gives the pre and post operative values in accordance with the anchor, and Table 22 gives the corresponding AUC, MIC (ROC), MIC (group), MIC (ES), MID (group) and MID (ES) estimates, along with the sensitivity and specificity for the optimal cut-off point. The AUC was 0.69. The MIC ROC values were 7 and 14 points for the Youden and shortest distance methods, respectively. The MIC (group) was larger, at 16 points, but the MID (group) was similar, at 6 points. The MIC (ES) and MID (ES) were 2 and 1, respectively.

FIGURE 5.

The total OHS: EPOS data set ROC curve.

TABLE 21 - The total OHS, by satisfaction for the EPOS data set

Note

Change score was calculated as postoperative minus preoperative score.

Score	Satisfaction
	Somewhat satisfied			Somewhat dissatisfied
	n	Mean	SD	n	Mean	SD
Preoperative	163	14.52	7.10	39	14.80	8.00
Postoperative	164	30.85	9.09	38	25.05	8.13
Change	163	16.33	8.16	39	10.25	8.06

TABLE 22 - The total OHS MIC/MID, by satisfaction for the EPOS data set

MIC						MID
Individual-level MIC (ROC analysis)				Scale	MID
AUC (95% CI)	Optimal cut-off point	Sensitivity	1 – specificity
0.69 (0.59 to 0.78)
Youden’s index	7	0.01	1	Group	16.33	6.08
Shortest distance	14	0.44	0.68	ES	2	0.75

Figure 6 shows the WOMAC total ROC curve for the EUROHIP data. Table 23 gives the preoperative and postoperative values in accordance with the anchor, and Table 24 gives the AUC, MIC (ROC), MIC (group), MIC (ES), MID (group) and MID (ES) estimates, along with the sensitivity and specificity for the optimal cut-off point. The AUC was 0.76. The MIC ROC optimal cut-off point was the same for the Youden and shortest distance methods: 26 points. The MIC (group) and MID (group) were larger, at 31 and 19 points, respectively. The MIC (ES) and MID (ES) were 2 and 1 points, respectively.

FIGURE 6.

The total WOMAC score: EUROHIP data set ROC curve.

TABLE 23 - The total WOMAC score, by satisfaction for the EUROHIP data set

Note

Change score was calculated as postoperative minus preoperative score.

Score	Satisfaction
	Somewhat satisfied			Somewhat dissatisfied
	n	Mean	SD	n	Mean	SD
Preoperative	58	63.37	14.41	13	61.23	13.33
Postoperative	58	32.10	18.63	13	48.64	20.55
Change	55	–31.27	16.60	13	–12.59	17.32

TABLE 24 - The total WOMAC score MIC/MID, by satisfaction for the EUROHIP data set

MIC						MID
Individual-level MIC (ROC analysis)				Scale	MID
AUC (95% CI)	Optimal cut-off point	Sensitivity	1 – specificity
0.76 (0.62 to 0.90)
Youden’s Index	26	0.63	0.15	Group	31.27	18.68
Shortest distance	26	0.63	0.15	ES	1.88	1.13

Knee

None of the available knee data sets had a suitable anchor for which interpretability properties could be assessed.

Discussion

We calculated the measurement properties of the candidate scores using various statistical methods, using multiple hip and knee replacement data sets for the candidate instruments. This enabled us to obtain estimates for some of the properties for which population-specific estimates had not been previously reported. Estimates for most of the missing measurement properties could be calculated given the available data sets for the measurement tools of interest.

The OHS (EUROHIP and EPOS) and OKS (KAT) showed generally positive additional evidence regarding the measurement properties (internal consistency, construct validity, responsiveness and interpretability). Similarly, the WOMAC [ADAPT (Assessing Disability After Partial and Total Joint Replacement), the APEX study and EUROHIP] showed generally positive additional evidence regarding the measurement properties. For the SF-12 PCS and MCS, the evidence was more mixed and the lack of agreement between MCS and PCS was noteworthy and, although perhaps unsurprising, is problematic in terms of use in the proposed context.

There was no sign of a flooring effect in any instruments. The OHS showed high ceiling effects at post operation, whereas the OKS had a ceiling effect of only 2% at post operation. The high ceiling effects of the OHS after the arthroplasty surgery could be examined in relation to patient-reported satisfaction and/or assessments measuring success. The EQ-5D-3L index showed high ceiling effects for both hip and knee data sets, in keeping with previous evidence. 85 In respect of the systematic review, there is a substantial ceiling effect for patient ratings in the EQ-5D-3L measurement tool. Different versions of SF-12 questionnaires were used in different studies, illustrating how it can be difficult to obtain unified data sets for one measurement tool. Further research is still required for responsiveness, between the generic questionnaires [e.g. the EQ-5D-3L/EuroQol-5 Dimensions, five-level version (EQ-5D-5L) and SF-12] and disease-specific questionnaires. Taken overall, there was generally a reasonable amount of positive evidence of the measurement properties of the OHS, OKS and WOMAC.

Although the WOMAC physical function, pain and stiffness scales were reported as reliable in the systematic review of measurement properties presented in Chapter 2, the pain scale was highly related to physical function including item scales. 86 In this study, we focused on the relationship between the WOMAC total score and WOMAC subscales; further research may be required regarding the WOMAC subscales and their use as instruments for developing thresholds. Measurement properties of the SF-12 PCS were more positive than the MCS, which is similar to some previous studies. 87,88 Further research may be required to clarify what values of the SF-12 MCS are plausible as thresholds for referral and candidacy for the joint replacement surgery and the role, if any, it can play in this context.

The study had a number of limitations. Most important were the variations between available data sets in terms of size and also the collection of relevant instruments and variables (e.g. anchor questions). Evidence on a number of properties for a number of instruments is still lacking or limited. None of the data sets was ideal and they only contained a subset of relevant instruments, which made comparison between instruments difficult. For the WOMAC score, only relatively small data sets were available. Imbalance between data sets and outcomes collected makes direct comparison between the instruments very problematic, and, therefore, we have restricted reporting to the individual properties as opposed to the contrasting instruments. Similarly, the version of the instrument used varied and in some cases (e.g. the EQ-5D-3L) a more recent version of the tool has been proposed (the EQ-5D-5L) for which no data sets were available. The methods used to assess the measurement properties were relatively simplistic, although commonly used in the literature and in general do not provide definitive answers, only suggestive findings.

Conclusion

From the data sets available, additional data on measurement properties were calculated for the EQ-5D-3L, OHS, OKS, WOMAC, SF-12 (PCS and MCS) and SF-36. These results were added to the information identified from the systematic review to produce a summary table of the measurement properties of each of the 36 instruments originally identified. This additional information was used to update the summary of instrument measurement property evidence produced as part of the systematic review in Chapter 2. This updated summary, shown in Tables 25–28, was presented to the user group to inform the choice of candidate instrument to take forward.

Background

The selected candidate tools are shown in Table 29. The OKS and OHS consist of 12 questions with 0–48 integer score ranges, with higher scores indicating better health status. The SF-12 has two component scores: PCS, with a theoretical range of 4.3 to 76.4 (US version 2), and MCS, with a theoretical range of –1.1 to 79.6 (US version 2), with higher scores also indicating better health status (see Appendix 2). The WOMAC total score consists of 24 questions: the pain subsection has five questions, the physical function subsection has 17 questions and the stiffness subsection has two questions (all have a score range of 0–100). The WOMAC scores were converted to higher scores indicating better health status. The KOOS-PS consists of seven questions (score range of 0–100); no data were available for the KOOS-PS for evaluating in this study.

Methods

Results

Relative threshold using criterion B

Tables 37 and 38 show the relative threshold with sensitivity and specificity (95% CIs) (see Online Supplement 4 for full results) and the study population coverage (%) by each threshold. The AUC (95% CI) for the OHS, OKS, SF-12 PCS and WOMAC total showed poor (< 0.7) discrimination abilities overall.

TABLE 37 - Hip: relative threshold using criterion B

Notes

Model 1 = linear regression.

Model 2 = logistic regression.

Candidate tools	Probability level	Preoperative threshold	Specificity (%) (95% CIs)	AUC (95% CI)	Population coverage (%)
OHS
NHS PROMs
Model 1		40	4 (4 to 5)		100
Model 2	0.5	43	2 (1 to 2)	0.65 (0.65 to 0.66)	100
	0.75	38	6 (6 to 7)		99
EPOS
Model 1		40	3 (0 to 12)		100
Model 2	0.5	42	2 (0 to 9)	0.62 (0.59 to 0.64)	100
	0.75	39	3 (0 to 12)		99
SF-12 PCS
ADAPT
Model 1		46	24 (9 to 45)		91
Model 2	0.5	47	20 (7 to 41)	0.58 (0.46 to 0.7)	92
	0.75	35	48 (28 to 69)		70
SF-12 MCS
ADAPT
Model 1		42	91 (79 to 97)		26
Model 2	0.5	39	96 (87 to 100)	0.93 (0.85 to 0.98)	19
	0.75	37	100 (93 to 100)		16
WOMAC total
ADAPT
Model 1		81	50 (26 to 74)		90
Model 2	0.5	85	44 (22 to 69)	0.76 (0.67 to 0.84)	94
	0.75	78	56 (31 to 78)		86
APEX
Model 1		86	20 (3 to 56)		99
Model 2	0.5	86	20 (3 to 56)	0.61 (0.53 to 0.67)	99
	0.75	82	20 (3 to 56)		98
EUROHIP
Model 1		83	3 (1 to 8)		99
Model 2	0.5	87	1 (0 to 5)	0.56 (0.53 to 0.59)	100
	0.75	80	4 (1 to 9)		99
WOMAC pain
ADAPT
Model 1		83	36 (13 to 65)		94
Model 2	0.5	89	36 (13 to 65)	0.66 (0.57 to 0.75)	96
	0.75	82	36 (13 to 65)		94
APEX
Model 1		83	18 (4 to 43)		99
Model 2	0.5	88	18 (4 to 43)	0.65 (0.59 to 0.7)	99
	0.75	83	18 (4 to 43)		99
EUROHIP
Model 1		84	6 (3 to 12)		99
Model 2	0.5	89	4 (2 to 9)	0.58 (0.55 to 0.62)	100
	0.75	78	8 (4 to 14)		98
WOMAC physical function
ADAPT
Model 1		81	48 (26 to 70)		86
Model 2	0.5	86	43 (22 to 66)	0.82 (0.73 to 0.89)	91
	0.75	78	52 (30 to 74)		82
APEX
Model 1		86	17 (2 to 48)		99
Model 2	0.5	88	17 (2 to 48)	0.65 (0.58 to 0.72)	99
	0.75	83	17 (2 to 48)		99
EUROHIP
Model 1		82	5 (2 to 10)		99
Model 2	0.5	86	3 (1 to 8)	0.55 (0.52 to 0.58)	100
	0.75	79	7 (3 to 13)		99
WOMAC stiffness
ADAPT
Model 1		82	50 (21 to 79)		95
Model 2	0.5	91	42 (15 to 72)	0.81 (0.72 to 0.88)	100
	0.75	86	50 (21 to 79)		95
APEX
Model 1		82	32 (16 to 52)		97
Model 2	0.5	90	29 (13 to 49)	0.77 (0.71 to 0.82)	100
	0.75	83	32 (16 to 52)		97
EUROHIP
Model 1		65	20 (15 to 25)		98
Model 2	0.5	67	20 (15 to 25)	0.71 (0.68 to 0.74)	98
	0.75	36	81 (75 to 85)		61

TABLE 38 - Knee: relative threshold using criterion B

–, could not get the estimation.

Notes

Model 1 = linear regression.

Model 2 = logistic regression.

Candidate tools	Probability level	Preoperative threshold	Specificity (%) (95% CI)	AUC (95% CI)	Population coverage (%)
OKS
NHS PROMs
Model 1		37	4 (4 to 5)		99
Model 2	0.5	40	2 (2 to 2)	0.62 (0.61 to 0.62)	100
	0.75	33	11 (10 to 11)		96
KAT
Model 1		35	5 (3 to 9)		98
Model 2	0.5	39	3 (1 to 6)	0.62 (0.60 to 0.65)	100
	0.75	29	14 (10 to 20)		93
SF-12 PCS
ADAPT
Model 1		36	25 (11 to 43)		87
Model 2	0.5	34	31 (16 to 50)	0.64 (0.5 to 0.75)	78
	0.75	–	–		–
KAT
Model 1		39	27 (23 to 31)		86
Model 2	0.5	43	16 (13 to 19)	0.65 (0.63 to 0.68)	92
	0.75	22	94 (92 to 96)		13
SF-12 MCS
ADAPT
Model 1		40	93 (81 to 99)		22
Model 2	0.5	49	74 (58 to 86)	0.81 (0.7 to 0.9)	51
	0.75	34	100 (92 to 100)		8
KAT
Model 1		43	85 (83 to 88)		28
Model 2	0.5	49	72 (69 to 75)	0.78 (0.75 to 0.80)	45
	0.75	26	100 (99 to 100)		2
WOMAC total
ADAPT
Model 1		86	7 (1 to 24)		100
Model 2	0.5	81	15 (4 to 34)	0.55 (0.46 to 0.65)	97
	0.75	71	19 (6 to 38)		92
APEX
Model 1		81	10 (1 to 30)		99
Model 2	0.5	85	5 (0 to 24)	0.60 (0.53 to 0.68)	100
	0.75	75	10 (1 to 30)		97
WOMAC pain
ADAPT
Model 1		78	5 (0 to 25)		98
Model 2	0.5	82	5 (0 to 25)	0.56 (0.46 to 0.65)	100
	0.75	71	10 (1 to 32)		98
APEX
Model 1		81	0 (0 to 12)		100
Model 2	0.5	–	–	0.59 (0.53 to 0.65)	–
	0.75	85	0 (0 to 12)		100
WOMAC physical function
ADAPT
Model 1		87	13 (3 to 32)		98
Model 2	0.5	89	8 (1 to 27)	0.47 (0.37 to 0.57)	99
	0.75	82	13 (3 to 32)		96
APEX
Model 1		82	15 (4 to 34)		98
Model 2	0.5	86	11 (2 to 29)	0.62 (0.55 to 0.69)	99
	0.75	75	19 (6 to 38)		94
WOMAC stiffness
ADAPT
Model 1		72	18 (8 to 34)		97
Model 2	0.5	65	18 (8 to 34)	0.69 (0.59 to 0.77)	97
	0.75	24	97 (87 to 100)		24
APEX
Model 1		68	28 (18 to 39)		97
Model 2	0.5	65	28 (18 to 39)	0.71 (0.65 to 0.77)	97
	0.75	34	86 (76 to 93)		58

Hip

The range of relative thresholds was 38–43 with specificity of 2–6 for the OHS. A histogram of the preoperative OHS distribution for the NHS PROMs data sets is given in Figure 7. Threshold ranges of the SF-12 PCS and MCS were 35–47 (specificity 20–48%) and 37–42 (specificity 91–100%), respectively. The threshold ranges were 78–87 (specificity 1–56%) for the WOMAC total, 78–89 (specificity 4–36%) for the WOMAC pain, 78–88 (specificity 3–52%) for the WOMAC physical function and 36–91 (specificity 20–50%, EUROHIP 81) for the WOMAC stiffness. Study population coverages for the thresholds of a 50% probability level were 100% for the OHS, 19% for the SF-12 PCS, 92% for the SF-12 MCS and 91–100% for the WOMAC. Figures 8 and 9 give the linear regression model, absolute threshold and logistic regression model estimates. The CI bands (see Figure 8) show the lack of fit for the variation in the OHS outcome for the linear model, although the point estimate seems reasonable. Linear regression models for the other outcomes (not shown) showed a similar lack of fit.

FIGURE 7.

The OHS: NHS PROMs preoperative histogram with the absolute and linear relative thresholds using criterion B.

FIGURE 8.

The OHS: NHS PROMs change scores.

FIGURE 9.

The OHS: NHS PROMs percentage improved using criterion B.

Knee

The range of the relative (observed) threshold was 29–40 with a specificity of 2–14 for the OKS. A histogram of the preoperative OHS distribution for the NHS PROMs data sets is given in Figure 10. Threshold ranges of the preoperative score of the SF-12 PCS and MCS were 22–43 (specificity 16–31%, KAT 94) and 26–49 (specificity 72–100%), respectively. The threshold ranges were 71–86 (specificity 5–19%) for the WOMAC total, 71–85 (specificity 0–10%) for the WOMAC pain, 75–89 (specificity 8–19%) for the WOMAC physical function and 24–72 (specificity 18–28%, ADAPT 97 and APEX 86) for the WOMAC stiffness. Study population coverages for the thresholds of a 50% probability level were 100% for the OHS, 78–92% for the SF-12 PCS, 45–51% for the SF-12 MCS and 97–100% for the WOMAC. Figures 11 and 12 give the linear regression model, absolute threshold and logistic regression model estimates. The CI bands (see Figure 11) show the lack of fit for the variation in the OHS outcome for the linear model, although the point estimate seems reasonable. Linear regression models for the other outcomes (not shown) showed a similar lack of fit.

FIGURE 10.

The OKS: NHS PROMs preoperative histogram with the absolute and linear relative thresholds using criterion B.

FIGURE 11.

The OKS: NHS PROMs change scores.

FIGURE 12.

The OKS: NHS PROMs percentage improved using criterion B.

Discussion

Conclusion

In this study, various improvement definitions and analytical methods were used systemically to calculate threshold levels for the candidate tools in various data sets. The results demonstrated that thresholds of three candidate tools (the OHS, OKS and WOMAC), which suggested promising initial cross-sectional psychometric properties (from work package 1), were consistent across data sets (except for the SF-12).

Background

This chapter aims to answer the following two inter-related questions:

1. What is the economic threshold for each clinical tool (i.e. what is the highest score at which arthroplasty is cost-effective)?

2. How do the incremental costs, QALYs and cost-effectiveness of arthroplasty vary depending on the threshold and clinical tool used?

We addressed these questions in a UK setting from a NHS perspective by conducting a series of cost–utility analyses, comparing the incremental cost-effectiveness of TKA and THA with no arthroplasty in men and women of different ages with different preoperative scores on each clinical tool.

The analyses presented in this chapter focused on total joint arthroplasty (TJA) because this type of surgery constitutes 92% of knee and 99% of hip arthroplasty procedures conducted in the UK. 97 We compared immediate TJA with having no arthroplasty surgery during the 10-year time horizon used in the analysis. This enabled us to calculate the economic threshold for each clinical tool by comparing the cost-effectiveness in different groups with one another and with the £20,000-per-QALY ceiling ratio typically used in NHS decision-making. 98 The economic threshold simply comprised the highest clinical tool score at which the incremental cost-effectiveness ratio (ICER) is < £20,000 per QALY gained. In practice, patients who are not deemed to warrant immediate surgery may have treatment later, after their condition has worsened. However, there are relatively limited data on how clinical tool scores change over time without surgery,99–102 and modelling the referral pathway and outcomes for patients who undergo surgery at different times would have greatly complicated the analysis. Instead, by directly comparing immediate TJA with no arthroplasty over 10 years, we made the most of existing UK data sets and directly assessed the cost-effectiveness of arthroplasty.

We focused on NHS and Personal Social Services costs in line with current UK guidelines,103 but have narrowed the perspective further to focus on NHS costs because only one of the available data sets (APEX) included non-NHS costs. Health benefits were measured in terms of QALYs, in line with guidelines and in order to capture the effect of surgery on both quality of life and mortality. 103 The cost-effectiveness of TJA versus no arthroplasty was therefore calculated as the difference in cost (between patients undergoing TJA and those having no arthroplasty) divided by the difference in QALYs.

The analysis primarily concerned patients aged between 50 and 90 years and with an ASA (American Society of Anesthesiologists) grade of 1–3 who were undergoing unilateral TKA or THA for osteoarthritis. However, patients not meeting these criteria were not explicitly excluded from the regression analyses used to estimate input parameters to ensure consistency with the clinical analyses. Because the model begins when patients undergo (or do not undergo) TJA, the analyses presented in this chapter would apply regardless of whether the thresholds are applied in the setting of general practice, musculoskeletal hubs or secondary care.

Methods

Costs and QALYs for hypothetical patients with different clinical tool scores and demographic characteristics were estimated using decision-analytic models built in Microsoft Excel^® 2010 (Microsoft Corporation, Redmond, WA, USA). Model-based economic evaluation enabled us to synthesise data from different data sets and extrapolate beyond the end of the available data. Separate models were built for TKA and THA for each of the three clinical tools. Model parameters (e.g. costs, utilities and the probability of death or revision) were based on regression models estimated using patient-level data from existing data sets; such models were used to estimate model parameters for patients with different preoperative characteristics. Hypothetical patients of different ages and genders with a wide range of clinical tool scores were run through the models separately and the costs and QALYs with and without TJA were calculated for each hypothetical patient. The models were probabilistic and took account of uncertainty around regression parameters.

Like the analyses described in Chapter 4, we considered hypothetical patients with different total OKSs or total OHSs on a 0–48 scale. For the WOMAC, for simplicity we focused on total scores, rather than considering pain, physical functioning and stiffness separately. As for the clinical analyses, the scale on the total WOMAC score (the sum of the three subscores ranging from 0 to 96) was multiplied by a factor of 10096 to rescale it, and then reversed by subtracting it from 100, such that 0 indicates severe problems and 100 indicates no problems. However, because there is no single summary score for the SF-12, we considered the physical and mental scores as two separate patient characteristics and evaluated the cost-effectiveness of different combinations of physical and mental scores, estimating threshold SF-12 physical scores for patients with different mental scores.

Model

Literature reviews on modelling approach

A comprehensive literature review was conducted to identify previous decision-analytic models that assessed the cost-effectiveness of arthroplasty, specific types of surgery or prostheses or changes to the timing of surgery (see Online Supplement 7). This review identified 41 previous model-based economic evaluations. Almost all of the published studies used Markov models to allow for repeating cycles. Models typically allowed for a proportion of patients having one or more revision operations, which were sometimes separated into one- and two-stage revisions, revisions for infection or other causes, or total and partial revisions. Some models used separate health states to differentiate between patients with good and poor outcomes or quality of life after surgery. Some of the models differentiated between patient groups based on age, sex, comorbidities, ASA grade and/or obesity, and one analysis estimated results for patients with different Kellgren and Lawrence grades with and without symptoms. 104 However, no model-based evaluations calculated how costs and QALYs varied with clinical tool scores. Five studies using patient-level data assessed how costs and/or QALYs varied with the Oxford Hip and Knee Scores or the WOMAC. 2,68,105–107 One study that was published after our search date assessed how cost-effectiveness varied with the SF-12. 108

Description of the Markov model

We used the results of this literature review to inform the design of our model. Like most studies in the literature, we used a Markov model with annual cycles to allow for the fact that patients are at risk of death or needing a revision each year. We estimated costs and QALYs using cohort simulation, because modelling thousands of individual patient trajectories for up to 1410 hypothetical patients using patient-level simulation would have been too computationally time-consuming. Revision rates and mortality varied depending on the time since primary arthroplasty and the age and sex of the hypothetical individual. Costs and EQ-5D utilities also varied depending on the time since primary arthroplasty, age, sex and clinical tool score.

The model started at the point at which patients in the arthroplasty arm underwent primary TKA or THA (Figure 13). Revision was defined in the same way as in the National Joint Registry (NJR), namely an:

. . . operation performed to remove (and usually replace) one or more components of a total joint prosthesis for whatever reason.

Reproduced with permission from the National Joint Registry (www.njrcentre.org.uk) from the 12th Annual Report (2015)97

FIGURE 13.

State transition diagram for the Markov model.

The base-case analysis used a 10-year time horizon because it was considered clinically unrealistic to assume that patients in the no arthroplasty arm would never have surgery. This also approximated the longest follow-up time available in the data sets used to estimate model parameters. However, different time horizons were examined in the sensitivity analysis. Costs, QALYs and life-years accrued beyond year 1 were discounted at 3.5% per annum. 103,109

The model did not apply a standard half-cycle correction to all health states, because the cost of primary TJA was assumed to be at the start of the year regardless of what subsequently happened to the patient and is not evenly distributed across the first 12-month period. We assumed that patients who die in the same year as having a primary TJA or revision surgery will incur the entire cost of the hospital stay in which the TJA or revision surgery was conducted. The cost of the admission for primary or revision surgery was assumed to be the same regardless of whether patients died during or soon after surgery. Other costs were assumed to be evenly distributed across the year, such that patients who die in any given year were assumed to accrue half of the cost and half of the number of QALYs that they would have accrued if they had lived for the whole year.

The model used a probabilistic sensitivity analysis (PSA) to propagate the uncertainty around all uncertain parameters (see Presentation of results and analysis of uncertainty).