Contextual Factors: The Importance of Considering and Reporting on Context in Research on the Patient-Centered Medical Home
Prepared For:Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services, 540 Gaither Road, Rockville, MD 20850, http://www.ahrq.gov/
Prepared by: Kurt C. Stange, MD, PhD (firstname.lastname@example.org, Case Western Reserve University) and Russell E. Glasgow (email@example.com, National Cancer Institute).
This brief focuses on the importance of considering and reporting contextual factors in studies of patient-centered medical home (PCMH) models. It is part of a series commissioned by the Agency for Healthcare Research and Quality (AHRQ) and developed by Mathematica Policy Research under contract, with input from other nationally recognized thought leaders in research methods and PCMH models. The series is designed to expand the toolbox of methods used to evaluate and refine PCMH models. The PCMH is a primary care approach that aims to improve quality, cost, and patient and provider experience. PCMH models emphasize patient-centered, comprehensive, coordinated, accessible care, and a systematic focus on quality and safety.
KC and Glasgow RE. Considering and Reporting Important Contextual Factors in Research on the Patient-Centered Medical Home. Rockville, MD: Agency for Healthcare Research and Quality. May 2013. ARHQ Publication No. 13-0045-EF.
This brief and companion briefs in this series are available for download from pcmh.ahrq.gov.
Dr. Stange’s time was supported in part by the National Cancer Institute through the Intergovernmental Personnel Act and by a Clinical Research Professorship from the American Cancer Society. The authors are grateful to Paul A. Nutting, William L. Miller, Benjamin F. Crabtree, Dana Peterson, Deborah Peikes, Janice Genevro, and David Meyers for helpful comments on earlier drafts of this manuscript.
The findings and conclusions in this document are those of the author(s), who are responsible for its content, and do not necessarily represent the views of AHRQ, the NCI, or the ACS. No statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.
I. Contextual Factors in Research
The patient-centered medical home (PCMH) is diversely manifested in different settings, situations, and stakeholders.1 Understanding the PCMH and making it happen requires evaluating and understanding the context in which it arises.
In this regard, the PCMH is like health care and health. While it may occasionally make sense to try to understand very basic physical and biological processes isolated from their context, even the seemingly fundamental processes related to the human genome increasingly require consideration of environmental and epigenetic phenomena in order to even begin to make sense.
The phenomena of health care and health, even more so than basic biology, are complex systems that are fundamentally context-dependent. Contextual factors with potential to influence how the PCMH manifests and how it affects different outcomes include:
- national, State, local, and organizational policies
- community norms and resources
- health care system organization
- payment and incentive systems
- practice culture, history, and staffing
- characteristics of patient populations and subgroups
- historical factors and recent events
- the culture and motivations surrounding monitoring and evaluation
- changes in these factors change over time
Paying attention to and consistently reporting on context in designing, conducting, and reporting research in health, health care, and the PCMH has great potential to advance our science and explain seemingly inconsistent results. What works in one context often does not work in another, leading to potentially conflicting conclusions, or masking effects when outcomes are obscured by heterogeneity of results in different contexts. As described below, an important next step is to build upon the considerable progress that has been made on the PCMH and other health services research by iterative, interactive investigation that periodically raises its gaze to consider the contextual factors that affect the PCMH—what it is, what it does, what happens as a result, and most importantly, what it means. Health and health care are local, and contextual knowledge enhances transferability of what is discovered and learned. Assessing context throughout a PCMH planning, implementation, evaluation, and reporting initiative can increase the likelihood that ongoing cycles of implementation, learning, and rapid refinement will dramatically and quickly advance learning from the natural experiment of differences in contextual factors, rather than being confounded by them.
A Step-Wise Approach. A step-wise approach to assessing and reporting relevant context involves paying attention during all phases of the research:
- Identifying relevant contextual factors based on theory, local history, and the perspectives of multiple stakeholders at the beginning of a project.
- Collecting and analyzing contextual data at multiple time points during the study.
- Reporting relevant contextual factors and how they affected important processes and outcomes.
Paying attention during all phases of the research
Thinking contextually is generally not amenable to a checklist approach but involves a way of approaching research design, implementation, and analysis that uses and expands upon existing methods to consider contextual factors. Generating a theory-based list of potential domains of contextual factors and how they might interact can be helpful in deciding what is most important to measure. To report the influence of relevant contextual factors at the end of a PCMH study, the relevant stakeholders must be engaged to pay attention to and record their understanding of context during the planning, implementation, analysis, and reporting of the research. Since contextual factors continually interact with each other and with the events under study, it is important to consider context and its interactions throughout the study.
Paying attention involves examining factors that affect interpretation of what happened during the study (internal validity) and considering what others would need to know to transport or re-invent the study elsewhere (external validity). It also involves keeping track of how important contextual factors change during the course of the study.
Below we describe three steps in assessing and reporting contextual factors.
Step 1: Identifying Relevant Contextual Factors
Trying to understand the relevant contextual factors from the earliest planning stage and continuing to evolve this understanding during the study can help to assure the internal and external validity of PCMH research. This process involves a participatory, 2, 3 mixed-method 4, 5 approach that includes:
- Identifying the relevant stakeholders for the current project and likely subsequent dissemination targets, and engaging them in identifying and reporting relevant contextual factors and how they change during the study.
- Developing and refining a list of domains of relevant contextual factors for qualitative data collection and ongoing analysis.
- Identifying quantitative measures of relevant contextual factors and developing a plan for periodically gathering and making sense of data on these measures.
Research that takes a participatory approach and includes relevant stakeholders from the policy, community, health care system, practice, and patient groups is more likely to be relevant at the outset and transportable at the end. 6 Identifying these stakeholders and involving them in framing and planning the study, identifying and monitoring baseline and changing contextual factors, and interpreting and disseminating findings makes it much more likely that the relevant contextual factors will be considered. Stakeholder groups typically are well-positioned to identify relevant context; it is up to the research community to create space in which this wisdom can be gathered, used to guide the study, and conveyed during subsequent dissemination.
The pertinent domains of contextual factors will vary with the purpose, setting, participants, and anticipated dissemination targets for each study and will likely include concepts related to the framework or theoretical models used to guide each study. Table 1 identifies some theories and frameworks to consider when identifying contextual factors to assess and report. Selecting one to three theories most related to the project’s particular evaluation questions can often provide important ‘places to look’ or things to think about when deciding which contextual factors to assess and report during research.
Relevant theories and frameworks can help PCMH investigators and implementers to identify the attributes, actions, history, culture, mental models, and motivations to consider across multiple levels, including public policy, community, health care system, practice, research team, patient/family/ caregiver, and other key stakeholders. Initial and historical conditions and how they evolve over the course of the study are especially important to assess.
Conceptual Places to Look for Relevant Contextual FactorsView table in new window
Having identified the domains of relevant contextual factors, evaluators are in a position to pick and, if needed, modify existing pragmatic measures of these domains, and to develop a plan for assessing those domains for which quantitative measurement is not feasible or desirable. For the most important domains, a mix of quantitative and qualitative assessment will be desirable, with the qualitative assessment providing richness, meaning, and openness to inductive discoveries.
A recent review of quantitative measures of the PCMH and its appendix provide a starting point,1 and databases of survey items such as the growing list available from the Grid-Enabled Measures (GEM) database (www.gembeta.org/Public/wsoverview.aspx?wid=11&cat=8) provide additional options. Qualitative assessment typically involves collection of field notes informed by observation and interview, starting with the identified domains, and adding further domains as additional issues and examples of important contextual factors are identified by the research team and participating stakeholders.
Step 2: Collecting and Analyzing Contextual Data
In most studies, the primary responsibility for collecting and analyzing contextual data will fall on the research team. But engaging stakeholders, participants, and anticipated adopters in at least periodically reviewing the contextual domains being assessed can help to assure their relevance and accuracy. More participatory research involving practice-based research networks 34-36 or community-based participatory research 2, 3 approaches typically involve mechanisms for engaging the key stakeholders at all stages of the research, development, and dissemination process, and in cycles of shared learning and implementation that transcend an individual study.
Once the relevant measures have been identified and the list has been developed, quantitative and qualitative data collection can commence. Typically, a combination of data sources are used, including existing administrative, health, and quality of care data; new data collected by outside evaluators; and process and outcome data collected by participants. Data collection should occur at or prior to baseline, periodically during the study, and at the end, in response to pre-specified time-points or observation that changes appear to be happening.
Analyses of both quantitative and qualitative data almost always will involve a team, and the more participatory approaches will include regular engagement of stakeholders in interpreting the emerging findings. Emerging findings also will be useful to the intervention team in refining the PCMH practice change approach to fit emerging conditions, and to Data Safety and Monitoring Boards in assessing participant safety.
To do all this could require considerable time and effort. However, since contextual factors often are actively ignored during research (and reporting), having at least one team member assigned to pay attention to and record relevant contextual factors during the study, informally gathering and including input from multiple stakeholders along the way, would be a significant advance. Even limited but continuously collected input on contextual factors would be of great help to the next step of reporting.
Step 3: Reporting Relevant Context
Reporting information on contextual factors and their influence on PCMH implementation and outcomes is facilitated by having quantitative and qualitative data co-inform each other, 37 using narratives to provide meaning and context, and using numbers to provide statistics on key contextual factors. One easy way to do this is to use a table to convey quantitative summary measures and key qualitatively assessed domains, and to provide an accompanying narrative explanation. If necessary to meet the page or word limitations of journals, the table of key contextual factors and its narrative interpretation can be included as an appendix. Appendix A provides a template, modification of which is being used by 14 AHRQ practice transformation grantees described in section II below. Vetting drafts of such a table and narrative with stakeholders, and looking for trends in data collected at various points during the study can increase the credibility of the findings. Such a template could be included as an appendix to a scientific paper reporting study findings, or preferably could be used to generate a sentence for the abstract, a succinct paragraph that summarizes the important contextual factors for inclusion in the results section of the paper, and used as the basis for interpreting the meaning and transportability of the findings in the discussion section of the paper.
For those who wish to report context, but did not begin tracking contextual data at the beginning of the study or until the study has been completed, the step-wise approach can still be accomplished by gathering relevant retrospective data. This approach, however, may provide less robust data and includes the possibility of retrospective bias, but is much better than simply ignoring contextual factors.
II. Uses of Contextual Factors in Research
In this section, we describe concrete examples of current or recently completed work related to tracking and reporting contextual factors. Together, these complementary examples show how context reporting can be accomplished.
The step-wise method for identifying and assessing contextual factors described above currently is being used by 14 teams of investigators in an AHRQ-supported series of projects on Transforming Primary Care Practice. 38 Their assessment of relevant contextual factors, which began late in the projects, uses an early version of the worksheet in the Appendix, and is reported with each of their papers in a May 2013 supplement to the Annals of Family Medicine. A companion paper in the supplement describes what these investigators are learning from the process of reporting context for their practice transformation projects.
Formatively, Russell Glasgow, Lawrence W. Green, and Alice Ammerman facilitated a meeting of 13 health research journal editors to consider reporting requirements for external validity that consisted largely of contextual factors. They concluded that external validity and contextual factors related to external validity and settings to which results did and did not apply should be reported more frequently. Although all participants agreed on the importance of better reporting on these factors, they did not come to consensus on a standard set of reporting criteria for contextual factors. Several journals called for increased attention, others provided guidelines for reviewers, and others facilitated additional discussions of the issue. 39-41 Examples of such increased reporting can be explicit listing of the exclusion criteria (and rationale), the participation rate and representativeness of participants at each of the following contextual levels: the settings (e.g., primary care clinics), staff, and patients/ people involved.
The Prescription for Health project, funded by the Robert Wood Johnson Foundation and AHRQ, discusses contextual issues in 27 practice-based research networks to change practice and develop community partnerships to foster health behavior change around diet, activity, tobacco, and alcohol use. The findings have been summarized in numerous scientific publications, including supplements to the Annals of Family Medicine 42, 43 and the American Journal of Preventive Medicine. 44, 45 The paper by Cohen and colleagues shows the need to assess the effect of changing contextual factors over time found that to effectively change practice, flexibility to different contextual situations may be at least as important as fidelity to the original research design and evidence. 46
A final example illustrating the potential contribution of reporting on contextual factors comes from the National Demonstration Project (NDP) for the PCMH that used a multimethod approach and flexible group randomized design that allowed assessment of multiple important contextual factors. 15, 16, 47-56 Together, multiple complementary papers provide both numbers 50, 52, 53 and narratives 15,16, 48-50, 54-56 that describe the contextual factors needed to understand what happened and why it happened in this particular project, as well as providing contextual information that others can use to reinvent the lessons in different contexts.
NDP publications provide an explicit assessment of the historical context and evolving changes in the PCMH environment 51, 55, 56 affecting both the internal and external validity of the study findings. The socio-political context of the PCMH movement, and the values and theory of primary care that underpin the PCMH, are described and related to the state of the PCMH movement as it evolved during the time the project was conducted. 51 Reports specifically address the policy context on the levels of primary care practices, health care systems and payers, and evaluators. 47, 49
The NDP methods changed to meet emerging participant needs to adapt to the shifting environment and as stories from the qualitative data informed statistics from the quantitative data, and vice versa. For example, the NDP developed new methods to assess financial changes when it was discovered that practice fiscal records were inadequate for the planned economic analyses. 52 The adequacy of fit of the measures for the PCMH phenomena under study were ascertained and new measures and qualitative analyses developed, and the challenges of gathering data in change-fatigued practices were described. 52
How the PCMH intervention evolved in response to forces both within and outside of project participants 52, 55 represents a key contextual factor that often is ignored or hidden in research reports. For example, the facilitation process was tailored to match different practice change trajectories that became apparent during the course of the project, and the support technology, communication strategies, and shared learning were updated. 55 The usefulness of off-site practice retreats for practices that “hit a wall” of “change fatigue” was discovered and reported by paying attention to the evolution of the intervention in response to an ongoing assessment of practice needs at different stages of the change process. 54 Importantly, the comparison group of “self-directed” practices self-organized their own meeting and shared learning exchange, 52,, 55 which the intervention and evaluation teams capitalized upon to foster learning around a practice change approach determined by each individual practice. 54, 55 A midpoint report by two of the leaders of the self-directed practice organization 48 provided additional context from the directly reported perspective of participants in the middle of the process. The interventionists’ response to participant change fatigue and the self-organization among the comparison practices are examples of the sort of evolving contextual factors that typically are not reported, but that are vital to both understanding what happened and why in the study, and to be able to knowledgeably transport the findings to other settings and situations.
Reporting the important contextual factor of practices’ baseline and changing practice characteristics on the process of change and the intervention effect on both patients53 and practices 50, 54 provided insights helpful to understanding what happened in the NDP, and insights into how others might intervene effectively in different practice settings. Contextual data collection and analyses also pointed out a key limitation for understanding the limited effect on patient outcomes, 53 and for transporting the findings to other settings—that the NDP was an almost entirely practice-focused change intervention with almost no system-level support. 47, 52, 55 Understanding the key contextual factors in the NDP and adding the perspective of the evaluation team’s 15-year experience with other practice change interventions 56 indentified a developmental model of the practice change that involves strengthening practices’ core, building adaptive reserve, and expanding attentiveness to the local environment. 15 As a result, the important contextual factor of practices’ adaptive reserve has become part of the lexicon of PCMH practice change efforts, and interest by investigators in measuring this concept led the evaluation team to develop, psychometrically evaluate, and publish a shortened measure of practice adaptive reserve. 57
This careful, multimethod approach to paying attention and reporting of contextual factors allows for a deep understanding of what happened and why, and for a thoughtful and informed extrapolation of study findings to different times, situations, and settings.
Paying attention to contextual factors during all stages of PCMH research can help investigators and implementers understand often overlooked factors that affect the reach, relevance, implementation, outcome, and generalization of PCMH interventions.
Another advantage of reporting contextual factors is that it supports the replication of effective PCMH models. Reporting relevant contextual factors can help others to make sense of what happened during the study, for what reason, and in what situations. This information helps future implementers avoid the problem of attempts to translate evidence from one situation into another situation in which the evidence doesn’t fit. 28, 58-63
Considering context represents an opportunity to advance health services research conceptualization and methods that are likely to reduce inconsistencies in findings and more accurately represent the effects of the implementation of PCMH models across diverse settings, people, and times Consistently assessing and reporting contextual factors should help make scientific evidence about the effectiveness of PCMH as a health delivery model more relevant and actionable—and indeed more evolvable and applicable across diverse settings, people, and times.
The approach outlined above shows how to expand the usefulness, internal, and external validity of research by: identifying relevant contextual factors; grounding an assessment process in the relevant theory and stakeholder perspectives; using a multimethod, participatory process to collect and analyze the relevant data; and then reporting contextual factors. However, this approach must be applied with an eye on its potential limitations.
First, it can be time and labor intensive. Considering and reporting context requires thought and reflection so that the most important contextual influences on the intervention are identified. It also requires collecting and analyzing indicators of concepts that are outside those typically considered necessary by researchers, reviewers, and funders focused primarily on internal validity. Considering contextual factors may feel like adding complexity at a time when people yearn for simpler solutions, however unsuited simple approaches may be for complex phenomena such as improving (primary) health care. 64-66
Second, many journals do not have space for or prioritize reporting context. As we have discussed above, and shown in the Appendix, journals’ space limitation may be at least partly overcome by including data and analyses of context in an appendix. However, creating demand for reporting context will require sufficient examples of its real value, and of the perils of its being ignored, before it will become the norm.
Finally, it can be difficult to identify which of the myriad possible contextual factors to track in a study, to engage diverse participant and potential end-user perspectives, and to continue to pay attention to the evolution of contextual factors over time. The greater ease of specifying an immutable a priori design, of focusing on internal validity to the exclusion of external validity, and the greater appeal of decontextualized simple solutions, may make it challenging for context reporting to gain traction.
In conclusion, including contextual factors can make research more relevant to stakeholders, foster understanding, and enable wise dissemination and informed re-invention in different moments in time, settings, and situations. Paying attention to context can help research to support advancement along the continuum from information to knowledge, and from knowledge to understanding. Understanding PCMH research in context can foster the development of shared understanding that opens the possibility of wisdom. 67back to top
5. Creswell JW, Klassen AC, Plano Clark VL, et al. National Institutes of Health (U.S.). Office of Behavioral and Social Sciences Research. Best practices for mixed methods research in the health sciences. Bethesda, MD: The Office of Behavioral and Social Sciences Research; 2011: obssr.od.nih.gov/scientific_areas/ methodology/mixed_methods_research/index.aspx.
8. American Academy of Family Physicians (AAFP), American Academy of Pediatrics (AAP), American College of Physicians (ACP), American Osteopathic Association (AOA). Joint principles of the Patient- Centered Medical Home. 2007; www.medicalhomeinfo.org/Joint%20Statement.pdf.
16. Nutting PA, Crabtree BF, Miller WL, et al. Transforming physician practices to patient-centered medical homes: lessons from the national demonstration project. Health Aff (Millwood) 2011;30(3):439-45.
19. Kaplan HC, Provost LP, Froehle CM, Margolis PA. The Model for Understanding Success in Quality (MUSIQ): building a theory of context in healthcare quality improvement. BMJ Qual Saf 2012;21(1):13-20.
20. Barr VJ, Robinson S, Marin-Link B, et al. The expanded Chronic Care Model: an integration of concepts and strategies from population health promotion and the Chronic Care Model. www.primaryhealthcarebc. ca/phc/pdf/eccm_article.pdf Hosp Q 2003;7(1):73-82.
24. Koh HK, Brach C, Harris LM, Parchman ML. A proposed ‘health literate care model’ would constitute a systems approach to improving patients’ engagement in care. Health Aff (Millwood) 2013;32(2):357-67.
26. Glasgow RE, McKay HG, Piette JD, Reynolds KD. The RE-AIM framework for evaluating interventions: what can it tell us about approaches to chronic illness management? Patient Educ Couns 2001;44(2):119-27.
30. Sandoval JA, Lucero J, Oetzel J, et al. Process and outcome constructs for evaluating community-based participatory research projects: a matrix of existing measures. Health Educ Res 2012;27(4):680-90.
32. Phillips KA, Morrison KR, Andersen R, Aday LA. Understanding the context of healthcare utilization: assessing environmental and provider-related variables in the behavioral model of utilization. Health Serv Res 1998;33(3 Pt 1):571-96.
33. Hawe P, Shiell A, Riley T, Gold L. Methods for exploring implementation variation and local context within a cluster randomised community intervention trial. J Epidemiol Community Health 2004;58(9):788-93.
43. Stange KC, ed. Supplement: Prescription for Health: Changing Primary Care Practice to Foster Healthy Behaviors. Leawood: Annals of Family Medicine Inc.; 2005. Annals of Family Medicine No. 3 (Suppl 2).
50. Nutting PA, Crabtree BF, Stewart EE, et al. Effect of facilitation on practice outcomes in the National Demonstration Project model of the Patient-Centered Medical Home. Ann Fam Med 2010;8(Suppl1):S33-44.
54. Nutting PA, Crabtree BF, Miller WL, et al. Journey to the Patient-Centered Medical Home: a qualitative analysis of the experiences of practices in the National Demonstration Project. Ann Fam Med 2010;8(Suppl 1):S45-56.
58. Stange KC, Breslau ES, Dietrich AJ, Glasgow RE. State-of-the-art and future directions in multilevel interventions across the cancer control continuum. J Natl Cancer Inst Monogr 2012;No. 44:20-31.
65. Thomas P. Applying complexity theory to primary health care organizations In: Kernick D, ed. Complexity and Healthcare Organization. A View from the Street. Abingdon: Radcliffe Medical Press; 2004:323-34.
Appendix: Context Matters Reporting Template
Contextual Factors * Relevant for Understanding and Transporting Findings from
[Name of Project]
- e.g., relevant theory or participant mental models
- e.g., national, state and local public policy affecting the study
- e.g., pertinent community norms and resources
- e.g., health care system organization, payment systems, IT and other support systems
- e.g., practice culture and staffing affecting PCMH uptake
- e.g., particulars of patient populations and subgroups
- e.g., relevant historical factors or recent events
- e.g., the culture and motivations surrounding monitoring and evaluation
The following factors changed in important ways over the course of the study:
The following contributed to identifying the relevant contextual factors and to considering how they might have affected the internal and external validity of the study (list names, viewpoints and/or relationship to the project)
* Factors to consider: Relevant theory or participant mental models, national, State, and local public policy, community norms and resources, health care system organization, payment systems, practice culture and staffing, different patient populations and subgroups, available information, relevant historical factors or recent events, the culture and motivations surrounding monitoring and evaluation, & changes in these factors over time
Interpretation of how these contextual factors affected what happened during the study and what others should know to transport/re-invent the findings in their contexts
[Write interpretive text (typically <500 words) that explains the factors listed in the table, how they affected what happened during the project, how any changes over the course of the project, and why these factors might be important to others attempting to re-invent the project in their own context.]