Measuring Advance Care Planning: Optimizing the Advance Care Planning Engagement Survey

Corresponding author: Rebecca L. Sudore, MD, University of California, San Francisco School of Medicine, SFVAMC 4150 Clement Street, #151R, San Francisco, CA 94121, Phone: 415-221-4810 x 23475, Fax: 415-750-6669, ude.fscu@erodus.accebeR

Abstract

Context

A validated 82-item Advance Care Planning (ACP) Engagement Survey measures a broad range of behaviors. However, concise surveys are needed.

Objectives

To validate shorter versions of the Survey.

Methods

The Survey included 57 process (e.g. readiness) and 25 action items (e.g. discussions). For item reduction, we systematically eliminated questions based on face validity, item non-response, redundancy, ceiling effects, and factor analysis. We assessed internal consistency (Cronbach’s alpha) and construct validity with cross-sectional correlations and the ability of the progressively shorter survey versions to detect change one week after exposure to an ACP intervention (Pearson’s correlation coefficients).

Results

501 participants (4 Canadian and 3 US sites) were included in item reduction (mean age 69 years (±10), 41% non-white). Due to high correlations between readiness and action items, all action-items were removed. Due to high correlations and ceiling effects, 2 process-items were removed. Successive factor analysis then created 55, 34, 15, 9, and 4-item versions. 664 participants (from 3 US ACP clinical trials) were included in validity analysis (age 65 years (±8), 72% non-white, 34% Spanish-speaking). Cronbach’s alphas were high for all versions (4-item, 0.84–55-item, 0.97). Compared to the original survey, cross-sectional correlations were high (4-item, 0.85–55-item, 0.97) as were delta correlations (4-item, 0.68–55-item, 0.93).

Conclusion

Shorter versions of the ACP Engagement Survey are valid, internally consistent, and able to detect change across a broad range of ACP behaviors for English and Spanish speakers. Shorter ACP Surveys can efficiently measure broad ACP behaviors in research and clinical settings.

Introduction

Advance care planning (ACP) is a process that supports adults at any age or stage of health in understanding and sharing their personal values, life goals, and preferences regarding future medical care. 1 Increasing attention has been paid to ACP over the past several years by healthcare systems, insurers, and researchers, because ACP can improve patient and family satisfaction, increase alignment of medical care with patients’ values, and reduce costs. 2–4 With the proliferation of new ACP initiatives, tools, and reimbursement models for ACP, measuring the success of an ACP program has become increasingly important. Traditionally, successful ACP has been measured only by the completion of an advance directive. However, several studies have shown that the ACP process is complex and involves many different ACP behaviors, such as identifying one’s values, choosing a surrogate decision maker, and discussing values with surrogates and clinicians, as well as completing advance directives. 5–8

To measure the complex behavior process of ACP, we created and validated the ACP Engagement Survey. 9 Although this survey has robust psychometric properties, 9 the original version is too long (82-items, mean of 49 minutes to complete) 10 for widespread use in research and clinical settings. The purpose of this study was to create and validate progressively shorter versions of the Survey that would be able to detect change in response to an ACP intervention.

Methods

The development and validation of the ACP Engagement Survey has been described in detail elsewhere. 9,11 In short, the survey was guided by input from experts in ACP and based on Social Cognitive and Behavior Change Theories. 9 The Survey focuses on 4 behavior change constructs (i.e., knowledge, contemplation, self-efficacy, and readiness) within 4 ACP domains (i.e., surrogate decision makers, values and quality of life, leeway in surrogate decision making, and asking doctors questions). After pilot testing with patients and ongoing input from ACP experts, 33 questions were added to the original survey concerning desired medical treatment and the involvement of friends and family in ACP, as has been previously described. 10 This resulted in an 82-item version of the Survey with 57 behavior change “Process Measures” (measured with 5-point Likert response options) and 25 ACP “Action Measures” (measured with “yes” or “no” response options) ( Table 1 ). The Survey has good internal consistency (Process Measures Cronbach’s alpha, 0.94) and test-retest reliability (Process Measures intraclass correlation, 0.70; Action Measures, 0.87). 9

Table 1

Original ACP Engagement Survey and Questions Retained in Progressively Shorter Versions After Item Reduction

b Response options for Process-Knowledge items (5-point Likert responses): “Not at all”, “A little”, “Somewhat”, “Fairly”, “Extremely”

c Response options for Process-Contemplation items (5-point Likert responses): “Never”, “Once”, “A few times”, “Several times”, “A lot”

d Response options for Process-Self-efficacy items (5-point Likert responses): “Not at all”, “A little”, “Somewhat”, “Fairly”, “Extremely”

e Response options for Process-Readiness items (5-point Likert responses): “I have never thought about it”, “I have thought about it, but I am not ready to do it”, “I am thinking about doing it in the next 6 months”, “I am definitely planning to do it in the next 30 days”, “I have already done it”

f Response options for Action items: “Yes”, “no”, or “I am not sure”. A “no” or “I am not sure” response are both coded as “no.”

The current study had two phases. The goal of Phase 1 “Item Reduction,” was to systematically reduce items and create progressively shorter versions of the Survey ( Figure 1 ). The goal of Phase 2 “Internal Consistency and Construct Validity,” was to test the internal consistency and the construct validity of the shorter versions and ensure they could detect change in response to an ACP intervention. Informed consent was obtained from all participants, and the study was approved by the respective Institutional Review Boards in the US and Canada.

Methods for Creating and Validating Shorter Versions of the ACP Engagement Survey

Participants

For Phase 1, “Item Reduction”, we included participants from both the United States (US) and Canada. In the US, from February 2013 to September 2014, we included baseline data of English-speaking participants enrolled in three ongoing randomized trials of ACP interventions at the San Francisco Veterans Affairs (VA) Medical center and San Francisco General Hospital (SFGH). The methods of these trials have been published. 12 In brief, participants were included if they were ≥ 60 years of age (VA) or ≥ 55 years of age (SFGH), had ≥ 2 chronic medical illnesses, and had seen a primary care physician ≥ 2 times in the past year. Patients were excluded if they had cognitive impairment or dementia, blindness, deafness, or psychosis determined from administrative data or screening.

In Canada, from April to September 2014, we included participants who reported being able to speak and read English and whose physicians reported they did not have cognitive impairment. A convenience sample of patients from varying age groups was recruited in the following clinical settings as part of ongoing additional studies and clinical activities: (1) primary care in Hamilton, Ontario and Edmonton, Alberta including patients ≥ 50 years of age; (2) inpatient hospital settings in Hamilton and Kingston, Ontario including patients ≥ 80 years of age or ≥ 55 with clinical markers of advanced chronic disease; (3) outpatient cancer centers in Edmonton, Alberta; Hamilton, Ontario; and Kelowna and Vancouver, British Columbia including patients ≥ 19 years of age; and (4) outpatient dialysis care centers in Edmonton, Alberta including patients ≥ 19 years of age.

In Phase 2, “Internal Consistency and Construct Validity,” we included participants from the US who were enrolled between February 2013 and March 2016 in three ongoing US ACP trials. Some of Phase 1 US participants were also included in Phase 2. However, because the validity analyses are descriptive and not predictive, we decided to include all trial participants enrolled at the time of the Phase 2 analysis. This included both English and Spanish-speaking participants at the time of Phase 2 analysis. In the Phase 2, we included both baseline and one-week follow-up trial data to be able to assess the Survey’s ability to detect change in response to an ACP intervention.

Phase 1, Item Reduction

We conducted incremental analyses to create successively shorter survey versions ( Figure 1 ). We considered that longer versions would have superior psychometric properties and may be more appropriate for research, while shorter versions may have lower validity, but may be more appropriate in the clinical setting.

First, we assessed survey items for face validity. Feedback was obtained from all authors and 5 research assistants who were involved in data collection and had obtained feedback from patients about the survey. Over several meetings, we reviewed survey items and discussed the ACP behavior change constructs and domains thought to be most important to retain. During these meetings, the team also flagged questions that appeared particularly difficult for study participants to understand.

Second, we determined the non-response or missing rate for each question as a marker of a difficult-to-answer question. Questions with non-item response rates > 10% were flagged for possible deletion before factor analysis. 13

Third, we assessed for redundant items. Specifically, there appeared to be redundancy between the Action questions (e.g., “Have you already formally asked someone to be your medical decision maker,” with response options of “yes” or “no”) and the 5 th response option in the 5-point Likert scale for the corresponding Readiness questions (e.g., “How ready are you to formally ask someone to be your medical decision maker with “5” being “I already did it”). We compared responses to these potentially redundant items and decided to delete Action Measures with agreement >80%. By removing the Action instead of the Readiness measures we would retain the detail of the 5-point Readiness measures as well as the ability to dichotomize responses as yes/no if desired. To further address redundancy, we then assessed correlations between the remaining Survey items within and across ACP domains. Items that had correlation coefficients >0.80 were removed prior to factor analysis. 14

Fourth, we looked for ceiling effects by assessing the proportion of patients who reported a “5” on the 5-point Likert scale for any Process Measure. We used 70% with a response option of “5” to flag an item for removal. 14

Fifth, we used exploratory factor analysis using Varimax rotation to identify factors in which the items naturally grouped. Factors with eigenvalues of >1 were retained. 14 These factors were reviewed, and then we selected items from each factor that either had the highest factor loading or met our face validity criteria for the most important ACP constructs determined a priori. Items are standardly removed from factor analysis if they have loading factors of

In the sixth and final reduction analysis step, all items were systematically removed, one at a time, in separate analyses for each progressively shorter versions of the Survey, to determine if removal of single items improved the Cronbach’s alpha. We considered deleting items whose removal from the shortened surveys improved the Cronbach’s alpha > 0.05. 14

Methods: Phase 2, Internal Consistency and Construct Validity

We first determined internal consistency (i.e., Cronbach’s alpha) for all versions of the Survey that resulted from Phase 1 analyses. Then, we measured construct validity in 3 ways. We assessed the correlations of the overall average 5-point scores of the original, 82-item version of the Survey with the progressively shorter versions using Pearson’s correlation coefficients ( Figure 1 ). The overall average 5 point-score was created using the 5-point Likert scores for the Process Measures and by assigning a value of 5 to Action Measure response options of “yes” and a zero to a response option of “no.”

Next, we assessed whether each version of the Survey was associated with patients’ self-reports of prior planning, defined as having drafted a will, having made funeral plans, or completed an advance directive, using Chi-squared tests. We hypothesized that both the original and shorter versions of the Survey would be associated with these pre-planning activities. Prior planning questions were asked before and separate from the Survey.

Finally, we felt it was highly important that the shorter versions of the Survey be able to detect change in response to an ACP intervention. The aforementioned ACP trials, from which Phase 2 recipient data were obtained, use the original 82-item version of the Survey. As previously described, each study arm receives an intervention (i.e., controls receive an advance directive and the intervention arm receives a directive plus an ACP website). 15 Therefore, we expect an increase in Survey scores regardless of study arm. To maintain blinding, we deleted the randomization identification and combined both arms into one cohort. We excluded individuals who had missing data for any survey item. We then assessed the change (delta) in total average 5-point Survey scores from baseline to one week after ACP intervention exposure and assessed the correlations of the deltas for the original, 82-item version with progressively shorter Survey versions.

Statistical Analysis

For descriptive statistics, we used percentages, means, and standard deviations (SD). To determine item non-response and ceiling effects, we used percentages. For correlations, we used Pearson’s correlation coefficients with coefficients ≥ 0.80 considered acceptable. For internal consistency, we used Cronbach’s alpha with a coefficient of ≥ 0.80 considered acceptable. For factor analysis, we used Varimax rotation with the PROC FACTOR procedures in SAS ® and created scree plots with statements of “method=principal rotate=varimax scree”. For associations of the surveys with prior planning activities, we used Chi-squared tests. To determine the ability of the surveys to detect change, we used Pearson correlation coefficients to assess the correlations of the deltas in average 5-point Survey scores. For Phase 2 only, we stratified our validation analyses on English and Spanish language.

Results

Participants

For Phase 1, 501 English-speaking patients from the US (n=352) and Canada (n=150) were included, and for Phase 2, 664 English and Spanish-speaking patients from the US were included ( Table 2 ). Two hundred and four English-speaking participants from the US were included in both Phases.

Table 2

Patient Characteristics by Study Phase

Phase 1, Item Reduction

Face validity: The overall behavior change constructs prioritized by the team included more activating constructs, such as self-efficacy and readiness rather than knowledge and contemplation. The team also felt that readiness questions concerning being “ready to decide” were difficult for participants to understand, and two questions were flagged for poor understanding; being ready to decide about a decision maker ( Table 1 , question 12) and flexibility for the decision maker (question 65). In addition, several members felt that the ACP domains of flexibility for decision makers and asking doctors questions were less important than other standard aspects of ACP, and several members felt that flexibility was a new and difficult concept for many participants. Furthermore, the team unanimously felt that the Action Measures were redundant to the readiness questions and supported their deletion if corroborated by the analysis (see below). Finally, the team felt that items about specific questions asked of a provider (i.e., risks, benefits, etc., Table 1 , questions 77–81) were not as helpful as asking about Readiness to ask questions in general ( Table 1 , question 82). Although several items were flagged for possible deletion, none were deleted during the face validity stage ( Figure 1 ).

No items were deleted due to non-response because no item had > 10% missing responses (mean non-response 3.1% (±1.2%), data not shown). Redundancy between Action items (yes/no) and corresponding Readiness items (5-point Likert scale with “5” indicating “I have already done it”) was high (mean 96.1%, ±SD 5.9%), and only 1 of 25 items had discrepancy >20% ( Table 1 , question 80). Therefore, we decided to remove all 25 Action items, leaving 57 Process Measure items measured on a 5-point Likert scale.

For the remaining 57 items, no Pearson correlation coefficients met the 0.80 threshold (data not shown). A correlation of 0.66 was found between items concerning readiness to choose flexibility for a decision maker and readiness to talk to the surrogate about flexibility ( Table 1 , questions 65 and 67). We removed question 65 based on face validity and the correlation, leaving 56 items ( Table 1 , Figure 1 ).

For the remaining 56 items, only one item concerning readiness to decide on a surrogate reached our threshold for a ceiling effect and was deleted ( Table 1 , question 12, data not shown).

In factor analysis, the remaining 55 items loaded on 10 factors, which explained 68.8% of the variance (Appendices 1–5). Within these 10 factors, we chose 34 items which spanned all 10 factors, had factor loading values > 0.3 (all > 0.45), and were prioritized based on behavior change constructs of readiness and self-efficacy in the domains of surrogates and quality of life per our face validity assessment (Appendix 1). In repeated factor analysis, the 34-item survey loaded on 7 factors, which explained 66.7% of the variance. Using the above criteria, we then chose 15 items that spanned 6 factors. The 15-items loaded on 3 factors, which explained 61.7% of the variance. Using the above criteria, we then chose 9 items within the 3 factors. The 9 items loaded onto 2 factors which explained 59.9% of the variance. Finally, using the above criteria, we chose 4 items from 2 factors. These 4 items loaded onto 1 domain which explained 58.5% of the variance. All factor loading values in all analyses were > 0.3.

In the final item reduction step, no step-by-step deletion of any individual item from any of the progressively shorter versions of the Survey resulted in an increase in the Cronbach’s alpha more than 0.0002 (data not shown). Therefore, no survey versions were further item reduced.

Phase 2, Internal Consistency and Construct Validation

Table 3

Overall Internal Consistency of the Original and Progressively Shorter Survey Versions in English and Spanish

Table 4

Correlation of Average Scores of Progressively Shorter Survey Versions to the Original Survey in English and Spanish

a Based on an average 5-point score. Process items scored on a 5-point Likert scale and action items scored as “yes” = 5 and “no” = 0.

Table 5

Association of Survey Scores with Pre-Planning Activities a

b Based on an average 5-point score. Process items scored on a 5-point Likert scale and action items scored as “yes” = 5 and “no” = 0.

c Difference in mean average 5-point scores between having completed and not completed pre-planning activities, as well as the overall average of the mean differences ± the standard deviation (SD).

Table 6

Correlation Between Change Scores in Response to an ACP Intervention Between Progressively Shorter Survey Versions and the Original Survey a

a This is a blinded sample that combined data from 3 ongoing randomized trials and that includes both the control groups (i.e., given an advance directive only) and the intervention groups (i.e., given the PREPARE website plus an advance directive).

b Mean change (deltas) obtained by calculating the difference between the average 5-point scores at baseline to one-week after reviewing an ACP intervention. Based on an average 5-point score. Process items scored on a 5-point Likert scale and action items scored as “yes” = 5 and “no” = 0.

Discussion

Using survey data from diverse study participants from the US and Canada, we used rigorous and systematic item-reduction methods to create progressively shorter versions of the ACP Engagement Survey from the original 82-item version to a 55-, 34-, 15-, 9-, and 4-item version. Then, using data from multi-center randomized trials among English- and Spanish-speaking patients in the US, we demonstrated that these progressively shorter versions of the Survey retained high internal consistency and had high construct validity, including the ability to detect change in response to an ACP intervention.

The original development and validation of the 82-item ACP Engagement Survey allowed measurement of multiple ACP behaviors, in addition to advance directive completion. Although the original Survey included a larger number of ACP behaviors and quantified behavior change constructs (i.e., knowledge, contemplation, self-efficacy, and readiness) for each individual ACP behavior, it was long with a mean administration time of 49 minutes. 10 The current study has helped to produce psychometrically sound shorter versions of the Survey while still measuring a broad range of ACP behavior change constructs and domains. As the internal consistency and construct validity were slightly higher for longer versions of the Survey, researchers may wish to use the longer versions (55, 34, and 15-items) to determine efficacy of a program or ACP tool or to detect nuanced differences in specific ACP behaviors. However, the shorter versions (i.e., 9 and 4-items) remained psychometrically sound and may be appropriate to use for quality improvement initiatives in the clinical settings or research studies with limited resources. Decisions concerning which length of the survey to use will depend on the balance between the needed sample size to detect change in response to an ACP intervention, participant survey burden, and study resources.

The strengths of this study include the strong theoretical basis underpinning the original 82-item survey, which was informed by Social Cognitive and Behavior Change Theories. Additional strengths were the inclusion of a multi-disciplinary team of ACP content experts and racially/ethnically diverse patients from multiple medical centers from 2 countries, the rigorous and systematic psychometric validation in 2 phases, and the ability of the shorter versions of the Survey to detect change in response to an ACP intervention. Furthermore, we included English- and Spanish-speakers in Phase 2 construct validity assessments, and validity remained robust in both languages.

This study has several limitations. Phase 1, item reduction, only included English speakers; however, construct validity remained high for Spanish speakers in Phase 2. In Phase 2, validation of the Survey took place in only one area of the US, which may limit generalizability. Generalizability of our findings beyond North America is also unknown. Furthermore, in addition to rigorous psychometric analysis, our team of ACP experts used face validity to prioritize behavior change constructs and ACP domains. Other research groups may prioritize different ACP constructs and domains. Finally, we still do not know the scoring thresholds of the Survey associated with patient-centered outcomes. Based on Social Cognitive and Behavior Change Theories, any increase in the average 5-point score may be associated with clinically meaningful improvement in patient-centered outcomes, such as receipt of care that is consistent with patients’ goals. However, further longitudinal studies will be needed to determine the exact thresholds for a full range of ACP behaviors that lead to improved ACP outcomes.

In conclusion, several shorter versions of the ACP Engagement Survey are valid, internally consistent, and able to detect change across a broad range of ACP behaviors and ACP domains. The Surveys are also ready for use among English and Spanish speakers. Having several psychometrically sound shortened versions of the Survey provides flexibility to researchers and quality improvement experts when choosing surveys to measure successful ACP programs and tools in the research and clinical setting.

Acknowledgments

Funding/Support

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. Dr. Sudore, however, is supported in part by the following grants: NIH R01AG045043, PCORI-1306-01500, VA HSR&D 11-110-2, the American Cancer Society (ACS) #19659, and NIH U24NR014637.

Role of Funders/Sponsors

No funding bodies had a role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, and approval of the manuscript; or decision to submit the manuscript for publication. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs.

Appendix 1

Factor Analysis with the 55-item Version of the Survey, n=501

a Domains: DM =Medical Decision Maker, QOL = Quality of Life, FLEX = Flexibility for the Surrogate Decision Maker, QUEST = Asking Questions of Medical Providers

Appendix 2

Factor Analysis with the 34-item Version of the Survey, n=501

a Domains: DM =Medical Decision Maker, QOL = Quality of Life, FLEX = Flexibility for the Surrogate Decision Maker, QUEST = Asking Questions of Medical Providers

Appendix 3

Factor Analysis with the 15-item Version of the Survey, n=501

a Domains: DM =Medical Decision Maker, QOL = Quality of Life, FLEX = Flexibility for the Surrogate Decision Maker, QUEST = Asking Questions of Medical Providers

Appendix 4

Factor Analysis with the 9-item Version of the Survey, n=501

Appendix 5

Factor Analysis with the 4-item Version of the Survey, n=501

References

2. Klingler C, In der Schmitten J, Marckmann G. Does facilitated Advance Care Planning reduce the costs of care near the end of life? Systematic review and ethical considerations. Palliat Med. 2016 May; 30 (5):423–433. [PMC free article] [PubMed] [Google Scholar]

3. Silveira MJ, Kim SY, Langa KM. Advance directives and outcomes of surrogate decision making before death. N Engl J Med. 2010 Apr 1; 362 (13):1211–1218. [PMC free article] [PubMed] [Google Scholar]

4. Hammes BJ, Rooney BL, Gundrum JD. A comparative, retrospective, observational study of the prevalence, availability, and specificity of advance care plans in a county that implemented an advance care planning microsystem. J Am Geriatr Soc. 2010 Jul; 58 (7):1249–1255. [PubMed] [Google Scholar]

5. Sudore RL, Fried TR. Redefining the “planning” in advance care planning: preparing for end-of-life decision making. Ann Intern Med. 2010 Aug 17; 153 (4):256–261. [PMC free article] [PubMed] [Google Scholar]

6. Sudore RL, Schickedanz AD, Landefeld CS, et al. Engagement in multiple steps of the advance care planning process: a descriptive study of diverse older adults. J Am Geriatr Soc. 2008 Jun; 56 (6):1006–1013. [PMC free article] [PubMed] [Google Scholar]

7. Fried TR, Redding CA, Robbins ML, Paiva A, O’Leary JR, Iannone L. Stages of change for the component behaviors of advance care planning. J Am Geriatr Soc. 2010 Dec; 58 (12):2329–2336. [PMC free article] [PubMed] [Google Scholar]

8. Fried TR, Redding CA, Robbins ML, Paiva A, O’Leary JR, Iannone L. Promoting advance care planning as health behavior change: development of scales to assess Decisional Balance, Medical and Religious Beliefs, and Processes of Change. Patient Educ Couns. 2012 Jan; 86 (1):25–32. [PMC free article] [PubMed] [Google Scholar]

9. Sudore RL, Stewart AL, Knight SJ, et al. Development and validation of a questionnaire to detect behavior change in multiple advance care planning behaviors. PLoS One. 2013; 8 (9):e72465. [PMC free article] [PubMed] [Google Scholar]

10. Howard M, Bonham AJ, Heyland DK, et al. Measuring engagement in advance care planning: a cross-sectional multicentre feasibility study. BMJ Open. 2016; 6 (6):e010375. [PMC free article] [PubMed] [Google Scholar]

11. Sudore RL, Knight SJ, McMahan RD, et al. A novel website to prepare diverse older adults for decision making and advance care planning: a pilot study. J Pain Symptom Manage. 2014 Apr; 47 (4):674–686. [PMC free article] [PubMed] [Google Scholar]

12. Sudore R, Le GM, McMahan R, Feuz M, Katen M, Barnes DE. The advance care planning PREPARE study among older Veterans with serious and chronic illness: study protocol for a randomized controlled trial. Trials. 2015; 16 :570. [PMC free article] [PubMed] [Google Scholar]

13. Barnett V, Lewis T. Outliers in Statistical Data. New York: John Wiley & Sons; 1984. [Google Scholar]

15. Sudore R, Le GM, McMahon R, Feuz M, Katen M, Barnes DE. The advance care planning PREPARE study among older Veterans with serious and chronic illness: study protocol for a randomized controlled trial. Trials. 2015; 16 (1):570. [PMC free article] [PubMed] [Google Scholar]