Skip to main content
  • Study protocol
  • Open access
  • Published:

Protocol for comparing two training approaches for primary care professionals implementing the Safe Environment for Every Kid (SEEK) model

Abstract

Background

Child maltreatment (CM) is a major public health problem, affecting many lives, in the short and long term, and costing individuals, families, and society dearly. There is a need for broad implementation of evidence-based preventive interventions, such as the Safe Environment for Every Kid (SEEK) model, developed for pediatric primary care. Primary care offers an excellent opportunity to help address prevalent psychosocial problems (e.g., parental depression) that are risk factors for CM. By addressing such problems, SEEK can strengthen families and support parents; promote children’s health, development, and safety; help prevent CM; and benefit the health of the US population. This study will examine intervention strategies for optimizing SEEK’s adoption, implementation, and sustainment, and its effectiveness in preventing CM.

Despite strong evidence from two federally funded randomized controlled trials, SEEK has not been widely adopted. The goal of this study is to examine technology-driven implementation strategies to scale-up SEEK—in pediatric and family medicine primary care settings. The aims are to (1) evaluate the effectiveness of training strategies on SEEK’s implementation in primary care practices, (2) evaluate barriers and facilitators to successful implementation and sustainment of SEEK, and (3) examine the model’s effectiveness in preventing CM and the economic costs of implementing SEEK.

Methods

This randomized type III hybrid mixed methods design will examine how advances in medical training can bolster SEEK’s adoption and implementation in pediatric and family medicine practices in different regions of the USA. These are independent online training and in-depth structured training via a quality improvement project, approved by the American Boards of Pediatrics and of Family Medicine. We will also evaluate SEEKonline, software that assists primary care practitioners implement the model, and a “Traditional” paper and pencil strategy for their impact on implementation. The study uses the EPIS framework and the Universal Stages of Implementation Completion, quantitative measures, qualitative interviews, and data abstracted from electronic health records.

Discussion

The knowledge gained should improve pediatric primary care to better address prevalent social determinants of health, benefiting many children and families. The outcomes should enhance the field of implementation science and guide future interventions in primary care.

Trial registration

NCT03642327, Clinical Trials, registered August 21, 2018.

Background

Child maltreatment (CM) is a major public health problem in the USA, affecting many lives, in the short and long term, and costing individuals, families, and our society dearly. In 2017, 7.5 million children were reported to Child Protective Services (CPS) [1]. Of these, 674,000 children (i.e., 9.2 per 1000) were “substantiated” victims of CM. Yet, reported cases capture only the tip of the iceberg. The National Incidence Study-4, using observations by community professionals, estimated that 1.26 million children (i.e., 17.1 per 1,000) were maltreated in 2005–2006; the more inclusive “Endangerment Standard” estimated nearly 3 million victims (i.e., 40 per 1,000) [2].

The consequences of CM can be devastating. In addition to injuries and physical health problems, child and adolescent sequelae include many psychological and behavioral problems [3,4,5,6,7]. CM has also been linked to an array of adult outcomes such as substance use disorders, HIV/AIDS-related sexual risk behaviors [8,9,10,11,12,13,14], and intimate partner violence (IPV) [15,16,17,18,19,20], as well as depression, suicide, criminal behavior, interpersonal problems, academic and vocational difficulties [21,22,23,24,25,26,27,28,29,30], and multiple physical health problems [31,32,33,34]. The financial costs of CM are immense. Two thirds of the medical costs of CM are paid through Medicaid [35]. Additional costs are incurred by the child welfare, educational, mental health, and judicial systems, with estimated US costs of $103.8 billion per year [36]. The human and economic costs of CM point to the need for effective preventive strategies and scaling up evidence-based practices (EBPs).

Despite the compelling need to prevent CM, few interventions other than home visiting programs have been rigorously evaluated and found to be effective. Further, programs have not been developed for the healthcare system with the exception of preventing abusive head trauma, an important but small component of CM. Another is the Safe Environment for Every Kid (SEEK) model, developed for pediatric primary care, and found to prevent CM in two large federally funded, randomized controlled trials (RCTs) [37,38,39,40]. The conceptual basis underpinning the model is shown in Fig. 1. Several theories guided SEEK’s development. Ecological-developmental theory recognizes the multiple and interacting systems surrounding a child [41, 42]. Pediatric care has mostly focused narrowly on the child; SEEK was based on understanding the influence of family and parental functioning on children’s health, development, and safety and on CM. SEEK also was guided by the transtheoretical model, linking an understanding of a person’s stage of change (e.g., pre-contemplative) with interventions tailored to the individual [43,44,45]. Principles of motivational interviewing (MI) have been incorporated [46, 47]. Prevention science, integrating multiple disciplines, also guided SEEK’s development [48]. CM, with its multifactorial etiology, demands collaboration among disciplines. SEEK aims to enhance primary care professionals’ (PCPs’) abilities to address targeted social determinants of health (SDH), working with professionals in other disciplines and agencies. The SEEK model was also guided by social cognitive theory [49]. For example, role plays demonstrate how PCPs can help address problems. In addition, the US Preventive Services Task Force recommends screening for intimate partner violence (IPV), depression, and alcohol misuse [50]. SEEK provides a structured approach to follow these recommendations.

Fig. 1
figure 1

Conceptual model for SEEK

Core components of the SEEK model include the following: (1) training PCPs to identify and help address targeted SDH that are also risk factors for CM—parental depression and major stress, substance abuse, IPV, harsh discipline, and food insecurity; (2) the evidence-based SEEK Parent Questionnaire-R (PQ-R) to screen for the problems at well child visits [51,52,53,54,55,56]; (3) the Reflect–Empathize–Assess–Plan (REAP) approach to help PCPs assess and address problems; (4) principles of MI; (5) SEEK Parent Handouts for targeted problems, customized with local resources; and (6) referrals to community resources.

Preliminary effectiveness of SEEK

There is good evidence supporting SEEK based on two RCTs (SEEK I and II) [37,38,39,40]. The SEEK I sample of 558 families was high risk, very low income, urban, mostly African American and served by pediatric resident clinics in Baltimore; it involved 95 physicians [37,38,39,40]. SEEK II involved 105 pediatricians and nurse practitioners and 1119 relatively low risk, mostly white, middle-income families recruited from 18 suburban private pediatric practices in central Maryland [38, 39].

Practices in both RCTs were randomized to either SEEK or standard care. Following baseline evaluations, PCPs randomized to SEEK received in-person training on addressing the targeted SDH; they were evaluated at 18–36 months after the initial training. Parents were recruited from all practices with initial and follow-up assessments at 6 and/or 12 months. Toward the end of the studies, after 30–43 months (SEEK I and II), the children’s medical records were reviewed for CM-related diagnoses, before and after implementing SEEK. Data were gathered from the state agency on possible CPS involvement. We assessed parents’ satisfaction with the child’s PCP. In SEEK II, medical students observed PCPs during 3 checkups, at baseline and at study end, to observe their approach to and time spent on the targeted problems.

Impact on PCPs

In both studies, PCPs in SEEK practices reported significantly greater comfort and perceived competence in addressing the targeted problems, compared to controls [37,38,39,40]. Improvements were sustained for 18–36 months. Review of medical records revealed that SEEK PCPs were more likely than controls to screen for the targeted risk factors for CM. In SEEK II, this was confirmed by direct observation; screening increased on average across practices from less than 5 to 62% of visits. In SEEK I, parents in SEEK clinics reported more favorable views of their child’s PCP [40]. Importantly, busy PCPs demonstrated they could effectively implement SEEK.

Impact on CM

Three measures from three sources assessed CM:

Parent self-report. SEEK I parents reported fewer “severe physical assaults” than controls (0.11 vs. 0.33, p = .04) [37]. SEEK II parents reported fewer instances of psychological aggression (p = .02) and minor physical assaults (p < .05) than did controls [38].

Medical records. Children in SEEK I practices had less medical neglect than did controls [9]. There was less “non-compliance” with medical care (4.6% vs. 8.4%, p = 0.05) and fewer delayed immunizations (3.3% vs. 9.6%, p = 0.002).

CPS reports. In SEEK I, fewer families were reported to CPS (12% vs. 19.7%, p = .04) [9]. A report was prevented in one of every 13 such families exposed to SEEK.

Time required

SEEK did not require significantly more time, on average, for PCPs to address psychosocial problems [39]. Parents completing the PQ-R before visits saved time; this was offset when addressing problems.

Cost

SEEK II cost $3.38 per child per year and $306 per CM experience prevented. Using a conservative estimate of the healthcare cost per case of CM at $2779, providing SEEK in all practices would have saved society $2,151,878 for 29,610 children [57]. Thus, SEEK has a positive cost benefit. However, it does not necessarily follow that practices can afford to implement SEEK. The current study focuses on examining the costs of implementing the model.

The strong evidence for SEEK’s effectiveness has been underscored by its listing on the websites of the US Centers for Disease Control and Prevention, the Agency for Healthcare Research and Quality, the American Academy of Pediatrics, and by the California Clearinghouse for Evidence-Based Interventions in Child Welfare. Early adopters are increasingly implementing SEEK in primary care settings, including in Sweden. SEEK’s adoption by primary care clinics has, however, been limited, and questions remain as to how best implement this model. Given that early adopters are estimated to be 10% of those eligible and that medical innovations can take 17 years to be adopted, it is unlikely that SEEK’s potential public health benefit will be realized without better methods for scaling up [58]. To help facilitate its implementation, two pragmatic technology-driven platforms have been developed to increase the convenience and accessibility of the model to busy PCPs: (1) an interactive web-based training that utilizes webinars and individualized guidance during start-up and (2) SEEKonline software to facilitate delivery of SEEK within regular checkup visits. This study will examine the effectiveness of these implementation strategies.

Training strategies

Two training strategies will be tested (see Table 1), both increasingly used with PCPs, and developed in accordance with principles of adult leaning [59,60,61,62,63,64]. First, SEEK Maintenance of Certification (MOC-4) is an example of a structured quality improvement (QI) project approved by the American Boards of Pediatrics and of Family Medicine, required of physicians to maintain Board certification. It includes viewing the training videos and implementing SEEK in one’s practice and conducting the QI Plan-Do-Study-Act (PDSA) cycle. This PDSA cycle involves learning from data collected before and during SEEK’s implementation to assess and improve the process [65]. Four 1-hr webinars over 4 months enable collaborative learning and mentoring. Second, independent online training (IND) is a student-centered approach; it involves viewing the training videos over 2–3 hrs and passing the post-test. Both approaches include three 1-hr consultations in the ensuing year and offer CME credits toward state licensure. We hypothesize that the interactive MOC-4 training will lead to more efficient and competent adoption of SEEK.

Table 1 SEEK training strategies

Implementation strategies

SEEKonline is software to efficiently implement SEEK via a secure web-based system interfacing with a practice’s electronic health record (EHR). SEEKonline has been developed and currently is being beta tested. Its effectiveness in improving implementation outcomes compared to the Traditional paper-and-pencil approach has yet to be examined. SEEKonline enables parents to privately complete the SEEK PQ-R before a child’s checkup. Responses are available to PCPs at the start of the visit, and there is real-time decision support for PCPs. Selected documentation is sent to the child’s EHR. We anticipate that some practices will choose the Traditional approach over the software (see Table 2).

Table 2 SEEK intervention strategies

Methods

Aims

The overall aims of the proposed study are to examine technology-driven approaches to implementing SEEK and to understand facilitators and barriers regarding its implementation and short-term sustainment, while also examining the effectiveness of these strategies. Doing so will advance knowledge in implementation science related to primary care and the prevention of CM. The randomized type III hybrid design [66] leverages a commitment by 5 major healthcare systems to implement SEEK, enabling a rigorous evaluation of implementation strategies to optimize the adoption and delivery of SEEK in primary care settings, and subsequent prevention of CM: the independent online (IND) [59,60,61] versus in-depth structured MOC training [62,63,64]. We will also observe the impact of using the SEEKonline and the Traditional approach to guide fidelity of model delivery. Further, SEEK will be examined in pediatric and family medicine settings, increasing the generalizability of findings.

Aim 1: Evaluate the effectiveness of targeted implementation strategies on the implementation of SEEK in primary care settings

Practices will be randomly assigned to one of two training conditions (IND or MOC). (H1) MOC training will lead to more positive PCP attitudes, comfort level, and competence in addressing risk factors for CM. (H2) Users of SEEKonline will deliver the intervention more often, achieving a higher rate of penetration, and will report higher levels of provider and parent satisfaction than the Traditional mode of delivery. (H3) MOC training and SEEKonline will together optimize adoption and sustainment of SEEK.

Aim 2: Evaluate the impact of inner context variables (e.g., variation between pediatric and family medicine) on the SEEK implementation process and understand associated barriers and facilitators to successful service start-up and sustainment of SEEK delivery

Using a mixed methods approach, standardized measures of the implementation process (Stages of Implementation Completion (SIC)) and associated cost (Cost of Implementing New Strategies (COINS)) will be integrated with qualitative interview data focusing on barriers and facilitators during implementation from exploration to sustainment. Variations in adoption, model fidelity, and sustainment, and the economic ramifications of the SEEK training and implementation strategies will be examined.

Aim 3: Examine the effectiveness of the intervention strategies in preventing CM

CM will be measured via prevalence of ICD-10 codes related to CM obtained from EHRs for all children 0–5 attending the practices. (H1) Incidence of CM will be reduced in practices after implementing SEEK. (H2) Practices randomized to MOC training that successfully implement SEEK will have lower incidences of CM than with the IND approach. We will also observe the influence of SEEKonline and the Traditional approach on CM rates. Additionally, implementation success will be examined in relation to CM prevalence rates.

Study design (Fig. 2)

The study uses a rigorous hybrid type III design to examine the effectiveness of technology-driven training strategies to facilitate SEEK’s adoption and implementation in pediatric and family medicine settings, and helping prevent CM. The implementation approach is anchored in four stages of the EPIS framework: Exploration, adoption/Preparation, Implementation, and Sustainment [67].

Fig. 2
figure 2

Sequence of steps from pre-implementation (engagement, feasibility, readiness planning) to implementation to sustainment

Exploration and sample

Leaders in five healthcare systems that strongly supported the grant application will be approached to formally approve participation, without committing individual practices. They include 59 practices with 306 PCPs. Different regions provide diversity in terms of urban, suburban, and rural locations, and racial/ethnic diversity. Three systems have integrated behavioral health professionals. We plan to study 13 pediatric and 35 family medicine practices. In addition to five system leaders, we will recruit 5 groups of participants: (1) 48 practice leaders, (2) 275 PCPs, (3) 16 behavioral health professionals, (4) 96 ancillary staff, and (5) 288 parents (6 per practice). Smaller subsamples will be purposively selected for qualitative interviews according to practice type, training strategy, and presence of integrated behavioral health.

Inclusionary criteria are as follows: (1) practices not already implementing SEEK, (2) practices providing primary care to children, and (3) agreement to participate. A letter will be sent to physician leaders of practices inviting participation in the study. If interested, we will hold a 1-hr webinar with the practice leader, PCPs, and behavioral health professionals and key office staff. We will provide written material detailing what their involvement will entail. Participation will be voluntary, and while we prefer that all the PCPs in a practice participate, this will not be required. We anticipate that behavioral health professionals and office staff will necessarily be involved in practices opting to participate. Parents will be recruited via flyers in the waiting area. If interested, they will be asked to notify staff who will request permission to convey their contact information to SEEK project staff. We will contact them and explain the project by phone and in writing.

Adoption/preparation

We will sign an MOU with participating practices and request a limited informed consent by practice leaders and PCPs who opt out to gather data influencing their decision. We will identify a physician “champion” and an office staff member to lead implementation in each practice. Practices will be randomized to one of the two training strategies (IND vs. MOC). Practices, however, will be able to select the facilitation strategy (SEEKonline or Traditional) as we are unable to require this of participants. The design accounts for heterogeneity in geography, size of healthcare systems, type of primary care (pediatric and family medicine), and presence of integrated behavioral health. As shown in Fig. 2, professionals, office staff, and parents are nested within practices which are nested within the 5 healthcare systems. To ease introduction of the SEEK model, we will address logistical issues such as for which checkups parents will complete the SEEK PQ-R and documentation in the EHR, and the SEEK Parent Handouts will be customized with information on local resources.

Implementation

Informed consent will be obtained online from all participants at the beginning of the baseline survey. Practice leaders, PCPs, behavioral health professionals and office staff will be asked to complete the surveys (see Table 3) prior to the PCP training. Subsamples will be selected for 30-min phone qualitative interviews. At the end of training, PCPs will evaluate it. Interested practices will have SEEKonline connected to their EHR, ensuring interoperability. Procedures for addressing concerns of possible CM will be aligned with federal, local, and professional guidelines.

Table 3 Study measures related to specific aims

Sustainment

Participants will be assessed at 12, 21, and 33 months post-baseline regarding their thinking, feeling, behavior, and experiences (see Table 3) to help inform understanding of SEEK’s implementation and sustainment. Final surveys and phone interviews will assess participants’ thoughts and plans about continued use of SEEK. Following the grant period, practices that wish to continue utilizing the implementation strategies will be guided through the process of establishing real-world, not-grant funded contracting.

Final 12 months

De-identified aggregate data will be abstracted from practices’ EHRs on all children (0–5) for the periods prior to and during SEEK implementation: rate of eligible visits where screening occurred, types of problems identified, types of actions taken, receipt of services, and CM-related diagnoses using ICD-10 codes. This will be facilitated by 4 of the 5 systems using Epic as their EHR and having in-house IT. The data will be analyzed, the final report prepared, and presentations and papers will continue to be developed.

Measures (see Table 3)

Measurement will include survey-based assessments, targeted qualitative interviews, observational implementation assessment, and EHR data on service delivery of SEEK (e.g., rate of screening) and CM.

Organizational assessment

Participants will be asked to complete the SEEK Adoption Survey and the following four measures: (1) Practice Demographics Form, (2) the Evidence-Based Practice Attitude Scales measuring attitudes toward adopting EBPs [68], (3) the Implementation Leadership Scale assessing leader support for EBPs within an organization [69], and (4) the Implementation Climate Scale measuring how an organization views new interventions [70]. A subsample of participants will undergo a semi-structured, 30-min phone interview.

Implementation process and outcomes

The implementation process and outcomes will be measured primarily using the Stages of Implementation Completion (SIC), an observational assessment tool [10, 71, 72]. The SIC has 8 stages, each including subactivities, extending from Engagement with the developers to achievement of practitioner Competency. Sustainment is an ongoing process toward achievement of stage 8. Multiple adaptations of the measure have led to the development of the empirically derived Universal SIC, with items found to be relevant and reliably utilized regardless of service sector or population. For the current study, completion of activities will be monitored by the practice manager and collected monthly by the research team, with data entered into the SIC data collection website. Three scores are calculated for each SIC stage. First, the time that a practice takes for a stage is calculated (Duration Score). Second, the proportion of activities completed within a stage is calculated (Proportion Score). Third, the SIC Stage Score marks the final stage that a site reaches. SIC scores are calculated within each of the three implementation phases: pre-implementation (stages 1–3), implementation (stages 4–7), and sustainment (stage 8).

The SEEK Adoption Survey will capture key influences on the decision whether to adopt SEEK, including perceived barriers and strengths. The SEEK PCP Training Evaluation Form will evaluate PCPs’ perceptions of the training approaches and solicit input as to how the training might be improved. The SEEK PCP Questionnaire, used in both SEEK RCTs [39, 40], assesses PCPs’ thinking and practice with regard to addressing the targeted CM risk factors. We will use the SEEK PCP Survey to assess PCPs’ experience implementing SEEK including their perceptions of its relevance, ease of delivery, helpfulness, and training. The SEEK Office Staff Survey will assess staff experiences with SEEK, including their understanding of their roles and perceived competence in implementing the model. The rate of screening for the targeted problems will be abstracted from EHR data, as will be the receipt of services by parents with positive screens. The SEEK Parent View of Child’s PCP, adapted for pediatric practice from the Patient-Doctor Interaction Scale, will be used to assess change in parent’s views of PCPs associated with stage of implementing SEEK [73].

Child maltreatment

EHR CM-related Diagnoses ICD-10 codes accessible through EHRs [74]. De-identified aggregate data will be gathered toward study end for all children 0–5 attending the practices during the study—for up to 2 years prior to and during the study.

Qualitative assessment: phone interviews

To assess potential barriers and facilitators of the SEEK intervention, we will conduct 30-min semi-structured phone interviews with purposively selected (based on role, practice type, training strategy, and presence of behavioral health) subsamples of 5 groups: system and practice leaders, PCPs, behavioral health professionals, office staff, and parents, at 3 time points. The first 3 groups will be interviewed around the start of the intervention (with a focus on adoption) and 11 and 21 months later (with a focus on implementation and maintenance). The last 2 groups will be interviewed at 11, 20, and 32 months following start of the intervention, after they have experienced SEEK. Interviewers will use a semi-structured guide using the EPIS framework to align questions with stage of implementation [75]. Participants will be asked about their experiences with SEEK, assessment of training and implementation support, challenges in delivering the intervention, and recommendations for addressing the challenges. Interviews will be digitally recorded and professionally transcribed for analysis.

Measuring costs

The cost analysis will provide estimates of the overall practice-level costs associated with implementing SEEK. Cost measurement will be organized using the Cost of Implementing New Strategies (COINS) framework which provides a structure for measuring and categorizing costs [76]. Implementation costs include all resources used to deliver SEEK. COINS maps onto the SIC, by tracking the costs and resources needed to complete each implementation activity. In SEEK II, PCPs in the intervention arm did not require more time per child than did controls. It thus seems reasonable to apply this to the current study, rather than conduct another time study.

Data analysis plan

General considerations

Standard statistical methods for calculating point estimates, confidence intervals, and p values require the assumption of independence. However, due to participants being clustered within healthcare systems and within practices, this assumption cannot be made. To account for the lack of independence among multiple measures within the same system or practice, we will use mixed effects (i.e., hierarchical) models. In these models, we will include random effects for system and for practice. In analyses involving multiple measures from the same person, we will also include a random effect for person. Below, we highlight the main analyses for each aim:

Aim 1: Compare alternative approaches to implementation of SEEK with respect to clinical and implementation outcomes. To address aim 1a, we will compare practices randomized to MOC to practices randomized to IND with respect to clinical and implementation outcomes. These include PCPs’ perceptions of the training, scales from the SEEK PCPQ (e.g., competence in addressing problems), and the PCP Survey (e.g., ease of delivery). We will also compare the groups’ rates of screening and parents’ receipt of services and satisfaction with PCPs. Statistical inference will be based on mixed effects models fitted using restricted maximum likelihood. To avoid possible biases due to selective attrition, the primary analysis will follow the “intention to treat” principle including all those randomized. Secondary analyses will be based on groups defined by training received. To address aim 1b, we will examine practices who choose SEEKonline and those choosing the Traditional approach using the same statistical methods as for aim 1a. Outcomes of interest will include staff satisfaction, costs, rates of screening, receipt of services, and parental satisfaction.

Aim 2: Examine variations in SEEK’s implementation process and impacts and understand associated barriers and facilitators in pediatric and family medicine practices. We will determine the proportion of practices that agree to adopt SEEK. We will then assess the relationship between practice characteristics (e.g., demographics, EBP attitudes) and willingness to adopt SEEK. The most important independent predictors will be determined using multivariable logistic regression models. Among those that do adopt SEEK, we will assess their degree of completeness, speed, and quality of implementation using the SIC. Completeness will be summarized using the final stage attained (0–8). Speed will be summarized by the duration in each stage, and quality by the proportion of activities performed at each stage, and overall. The distribution of time to achieve each phase will be estimated using the Kaplan-Meier approach. Barriers and facilitators to implementing SEEK will be probed quantitatively and qualitatively. Among practices adopting SEEK, we will examine the association between practice characteristics and measures of completeness, speed, and quality of implementation from the SIC using multivariable mixed effects models.

Aim 3: Examine the effectiveness of SEEK in reducing CM. As in our previous studies [37, 38], we will identify CM-related diagnoses, now via EHRs. We will review the EHR for each child (0–5) in each practice for up to 2 years prior to implementing SEEK and during implementation. The presence of CM-related ICD-10 codes will be recorded. The proportion of children with CM diagnoses before and during SEEK will be compared at each practice. Formal inference regarding the best estimate and statistical significance of pre-post differences will be based on a binary regression model with a random effect for site, similar to aim 1.

Cost analysis

Total practice cost of implementation completion will be calculated for both training arms, not for implementing the entire SEEK model. Cost-effectiveness ratios will be calculated as the cost per average SIC component completed, and per average Competence scale and Practice Behavior scale scores. Standard time discounting methods will be applied to cost estimates [77]. We will use sensitivity analyses to derive upper and lower estimates of resource use and implementation costs [77]. Standard errors for use in mean comparisons will be estimated, using bootstrapping methods [77].

Qualitative/mixed methods analysis

We will keep an audit trail of data collected and memos, team meetings indicating time, place, source of data, and persons collecting or analyzing information. We will analyze interview transcripts using a thematic content analysis methodology [78]. First, transcripts will be reviewed by investigators to develop a broad understanding of content related to the project’s aims and to identify topics for discussion and observation. Second, segments of text ranging from a phrase to several paragraphs will be assigned codes based on a priori (i.e., from the interview guide) or emergent themes (or open coding) [79]. Codes will be assigned to describe connections between categories and between categories and subcategories (i.e., axial coding) [79]. Codes will also be assigned to reflect participants’ social and demographic characteristics. Lists of codes developed by each investigator will be matched and integrated into a single codebook. Third, each text will be independently coded by at least two investigators. Disagreements in assignment of codes will be resolved through discussion between investigators and by refining definitions of codes. With the final coding structure, two investigators will separately review transcripts to determine level of agreement. A level of agreement ranging from 66 to 97% depending on level of coding (general, intermediate, specific) indicates good reliability in qualitative research [80]. Fourth, based on these codes, the computer program QSR NVivo will generate a series of categories arranged in a tree-like structure connecting text segments grouped into separate categories of codes or “nodes” [81]. These nodes and trees will be used to further the process of axial or pattern coding to examine the association between different a priori and emergent categories. Fifth, by constantly comparing these categories with each other, the different categories will be further condensed into broad themes using a format that places SEEK’s effectiveness and implementation within the framework of the system characteristics [82]. Finally, the themes will be compared with the results of the analysis of quantitative data relating to PCP experiences with SEEK to identify points of convergence and divergence (triangulation) and to explain potentially unanticipated findings (expansion).

Discussion

CM continues to be a major public health and social problem in all countries. Increasing recognition of the importance of addressing SDH offers an opportunity for scaling up effective EBPs to prevent this problem. This hybrid type III study advances implementation science by applying several theories and established frameworks for evaluating implementation of the effective prevention model, SEEK, in primary care practices. The comparison of two increasingly common technology-driven approaches to medical education should yield valuable information for the SEEK model and other innovations in primary care. The convenience of such modalities is clear; this may be critical for scaling up innovations. While there are options to have such training be interactive, they involve little or no direct human contact, observation and this may be less engaging for students. In addition of how the SEEKonline software influences implementation should guide related efforts in healthcare. Diverse practices in different parts of the USA are being recruited because these are not controlled settings; each practice has different contextual factors that may influence implementation. The application of the EPIS framework offers a useful model for evaluating the introduction of such interventions in primary healthcare practices.

Project status

Although delays from the time of grant submission to funding created challenges in recruitment from the originally engaged health systems (due to commonly reported barriers such as changing priorities and resources), 17 practices have been successfully recruited across the USA with the help of several regional and national networks and organizations, such as the American Academy of Pediatrics. In addition, it seemed that practices lacking social work or integrated behavioral health were less inclined to participate. We obtained supplemental funding for a SEEK Helpline to offer consultation to PCPs and staff and help locate local resources, and to offer a modest stipend partly covering practices’ research-related costs. The impact of the COVID-19 pandemic on primary healthcare has however impeded ongoing recruitment efforts and delayed the study.

Availability of data and materials

Not applicable; just started gathering data.

Abbreviations

SEEK:

Safe Environment for Every Kid

SSNR:

Safe, stable, nurturing relationships

IPV:

Intimate partner violence

CM:

Child maltreatment

EBP:

Evidence-based practice

CPS:

Child protective services

MI:

Motivational interviewing

PCP:

Primary care professional

PQ-R:

Parent Questionnaire-R

REAP:

Reflect–Empathize–Assess–Plan

QI:

Quality improvement

MOC:

Maintenance of Certification

IND:

Independent (training)

EHR:

Electronic health record

SIC:

Stages of Implementation Completion

COINS:

Cost of Implementing New Strategies

EPIS:

Exploration, adoption/Preparation, Implementation, and Sustainment

IT:

Information technology

ICD:

International Classification of Diseases

SDH:

Social determinants of health

References

  1. U.S. Department of Health & Human Services, Administration on Children, Youth and Families, Children’s Bureau. Child Maltreatment 2015. 2017.

  2. Sedlak AJ, Mettenburg J, Basena M, Petta I, McPherson K, Greene L. Fourth National Incidence Study of Child Abuse and Neglect (NIS-4): Report to Congress | Office of Planning, Research & Evaluation | ACF [Internet]. 2010 [cited 2020 May 14]. Available from: https://www.acf.hhs.gov/opre/resource/fourth-national-incidence-study-of-child-abuse-and-neglect-nis-4-report-to.

  3. Bennett EM, Kemper KJ. Is abuse during childhood a risk factor for developing substance abuse problems as an adult? J Dev Behav Pediatr. 1994;15(6):426–9.

    CAS  PubMed  Google Scholar 

  4. Brown GR, Anderson B. Psychiatric morbidity in adult inpatients with childhood histories of sexual and physical abuse. Am J Psychiatry. 1991;148(1):55.

    CAS  PubMed  Google Scholar 

  5. Duncan RD, Saunders BE, Kilpatrick DG, Hanson RF, Resnick HS. Childhood physical assault as a risk factor for PTSD, depression, and substance abuse: findings from a national survey. Am J Orthop. 1996;66(3):437–48.

    CAS  Google Scholar 

  6. Glover NM, Janikowski TP, Benshoff JJ. The incidence of incest histories among clients receiving substance abuse treatment. J Couns Dev. 1995;73(4):475–80.

    Google Scholar 

  7. Ireland T, Widom CS. Childhood victimization and risk for alcohol and drug arrests. Int J Addict. 1994;29(2):235–74.

    CAS  PubMed  Google Scholar 

  8. Johnsen LW, Harlow LL. Childhood sexual abuse linked with adult substance use, victimization, and AIDS-risk. AIDS Educ Prev. 1996/02/01. 1996;8(1):44–57.

  9. Kang S-Y, Magura S, Laudet A, Whitney S. Adverse effect of child abuse victimization among substance-using women in treatment. J Interpers Violence. 1999;14(6):657–70.

    Google Scholar 

  10. Morrow KB, Sorell GT. Factors affecting self-esteem, depression, and negative behaviors in sexually abused female adolescents. J Marriage Fam. 1989:677–86.

  11. Rotheram-Borus MJ, Mahler KA, Koopman C, Langabeer K. Sexual abuse history and associated multiple risk behavior in adolescent runaways. Am J Orthop. 1996;66(3):390–400.

    CAS  Google Scholar 

  12. Runtz M, Briere J. Adolescent “acting-out” and childhood history of sexual abuse. J Interpers Violence. 1986;1(3):326–34.

    Google Scholar 

  13. Sansonnet-Hayden H, Haley G, Marriage K, Fine S. Sexual abuse and psychopathology in hospitalized adolescents. J Am Acad Child Adolesc Psychiatry. 1987;26(5):753–7.

    CAS  PubMed  Google Scholar 

  14. Sheridan MJ. A proposed intergenerational model of substance abuse, family functioning, and abuse/neglect. Child Abuse Negl. 1995;19(5):519–30.

    CAS  PubMed  Google Scholar 

  15. Alexander PC. Childhood trauma, attachment, and abuse by multiple partners. Psychol Trauma Theory Res Pract Policy. 2009;1(1):78.

    Google Scholar 

  16. Banyard VL, Williams LM, Saunders BE, Fitzgerald MM. The complexity of trauma types in the lives of women in families referred for family violence: multiple mediators of mental health. Am J Orthop. 2008;78(4):394–404.

    Google Scholar 

  17. Ehrensaft MK, Cohen P, Brown J, Smailes E, Chen H, Johnson JG. Intergenerational transmission of partner violence: a 20-year prospective study. J Consult Clin Psychol. 2003;71(4):741.

    PubMed  Google Scholar 

  18. Halpern CT, Spriggs AL, Martin SL, Kupper LL. Patterns of intimate partner violence victimization from adolescence to young adulthood in a nationally representative sample. J Adolesc Health. 2009;45(5):508–16.

    PubMed  PubMed Central  Google Scholar 

  19. Herrenkohl TI, Mason WA, Kosterman R, Lengua LJ, Hawkins JD, Abbott RD. Pathways from physical childhood abuse to partner violence in young adulthood. Violence Vict. 2004;19(2):123–36.

    PubMed  Google Scholar 

  20. Lang AJ, Stein MB, Kennedy CM, Foy DW. Adult psychopathology and intimate partner violence among survivors of childhood maltreatment. J Interpers Violence. 2004;19(10):1102–18.

    PubMed  Google Scholar 

  21. Chapman DP, Whitfield CL, Felitti VJ, Dube SR, Edwards VJ, Anda RF. Adverse childhood experiences and the risk of depressive disorders in adulthood. J Affect Disord. 2004;82(2):217–25.

    PubMed  Google Scholar 

  22. Malinosky-Rummell R, Hansen DJ. Long-term consequences of childhood physical abuse. Psychol Bull. 1993;114(1):68.

    CAS  PubMed  Google Scholar 

  23. Perez CM, Widom CS. Childhood victimization and long-term intellectual and academic outcomes. Child Abuse Negl. 1994;18(8):617–33.

    CAS  PubMed  Google Scholar 

  24. Dube SR, Anda RF, Felitti VJ, Chapman DP, Williamson DF, Giles WH. Childhood abuse, household dysfunction, and the risk of attempted suicide throughout the life span: findings from the Adverse Childhood Experiences Study. JAMA. 2001;286(24):3089–96.

    CAS  PubMed  Google Scholar 

  25. Linder JR, Collins WA. Parent and peer predictors of physical aggression and conflict management in romantic relationships in early adulthood. J Fam Psychol. 2005;19(2):252.

    PubMed  Google Scholar 

  26. Smith PH, White JW, Holland LJ. A longitudinal perspective on dating violence among adolescent and college-age women. Am J Public Health. 2003;93(7):1104–9.

    PubMed  PubMed Central  Google Scholar 

  27. Spriggs AL, Halpern CT, Martin SL. Continuity of adolescent and early adult partner violence victimisation: association with witnessing violent crime in adolescence. J Epidemiol Community Health. 2009;63(9):741–8.

    CAS  PubMed  Google Scholar 

  28. Taft CT, Schumm JA, Marshall AD, Panuzio J, Holtzworth-Munroe A. Family-of-origin maltreatment, posttraumatic stress disorder symptoms, social information processing deficits, and relationship abuse perpetration. J Abnorm Psychol. 2008;117(3):637.

    PubMed  Google Scholar 

  29. White HR, Widom CS. Intimate partner violence among abused and neglected children in young adulthood: the mediating effects of early aggression, antisocial personality, hostility and alcohol problems. Aggress Behav. 2003;29(4):332–45.

    Google Scholar 

  30. Whiting JB, Simmons LA, Havens JR, Smith DB, Oka M. Intergenerational transmission of violence: the influence of self-appraisals, mental disorders and substance abuse. J Fam Violence. 2009;24(8):639–48.

    Google Scholar 

  31. Arnow BA, Hart S, Scott C, Dea R, O’connell L, Taylor CB. Childhood sexual abuse, psychological distress, and medical use among women. Psychosom Med. 1999;61(6):762–70.

    CAS  PubMed  Google Scholar 

  32. Dube SR, Felitti VJ, Dong M, Giles WH, Anda RF. The impact of adverse childhood experiences on health problems: evidence from four birth cohorts dating back to 1900. Prev Med (Baltim). 2003;37(3):268–77.

    Google Scholar 

  33. Farley M, Patsalides BM. Physical symptoms, posttraumatic stress disorder, and healthcare utilization of women with and without childhood physical and sexual abuse. Psychol Rep. 2001;89(3):595–606.

    CAS  PubMed  Google Scholar 

  34. Felitti VJ, Anda RF, Nordenberg D, Williamson DF, Spitz AM, Edwards V, et al. Relationship of childhood abuse and household dysfunction to many of the leading causes of death in adults: the adverse childhood experiences (ACE) study. Am J Prev Med. 1998;14(4):245–58.

    CAS  PubMed  Google Scholar 

  35. Florence C, Brown DS, Fang X, Thompson HF. Health care costs associated with child maltreatment: impact on medicaid. Pediatrics. 2013;132(2):312–8.

    PubMed  PubMed Central  Google Scholar 

  36. Wang C-T, Holton J. Total estimated cost of child abuse and neglect in the United States. Chicago Il Prevent Child Abuse America. 2007.

  37. Dubowitz H, Feigelman S, Lane W, Kim J. Pediatric primary care to help prevent child maltreatment: the Safe Environment for Every Kid (SEEK) model. Pediatrics. 2009 Mar;123(3):858–64.

    PubMed  Google Scholar 

  38. Dubowitz H, Lane WG, Semiatin JN, Magder LS. The SEEK model of pediatric primary care: can child maltreatment be prevented in a low-risk population? Acad Pediatr. 2012 Jul;12(4):259–68.

    PubMed  PubMed Central  Google Scholar 

  39. Dubowitz H, Lane WG, Semiatin JN, Magder LS, Venepally M, Jans M. The Safe Environment for Every Kid model: impact on pediatric primary care professionals. Pediatrics. 2011;127(4):e962–70.

    PubMed  PubMed Central  Google Scholar 

  40. Feigelman S, Dubowitz H, Lane W, Grube L, Kim J. Training pediatric residents in a primary care clinic to help address psychosocial problems and prevent child maltreatment. Acad Pediatr. 2011;11(6):474–80.

    PubMed  PubMed Central  Google Scholar 

  41. Belsky J. Etiology of child maltreatment: a developmental€ cological analysis. Psychol Bull. 1993;114(3):413.

    CAS  PubMed  Google Scholar 

  42. Brofenbrenner U, Morris PA. The ecology of developmental processes. In: Lerner RM, editor. Theorectical models of human development. New York: Wiley; 1998. p. 993–1028.

    Google Scholar 

  43. Prochaska J, Diclemente C. Stages and procesess of self-change of smoking: toward an integrative model of change. J Consult Clin Psychol. 1983;51(3):390–5.

    CAS  PubMed  Google Scholar 

  44. Prochaska JO, DiClemente CC. Transtheoretical therapy: toward a more integrative model of change. Psychother Theory Res Pract. 1982;19(3):276–88.

    Google Scholar 

  45. Prochaska JO, DiClemente CC, Velicer WF, Rossi JS. Criticisms and concerns of the transtheoretical model in light of recent research. Br J Addict. 1992;87(6):825–8.

    CAS  PubMed  Google Scholar 

  46. Miller WR, Rollnick S. Motivational interviewing: helping people change. 3rd ed. Guilford press; 2013.

  47. Williams AA, Wright KS. Engaging families through motivational interviewing. Pediatr Clin N Am. 2014;61(5):907–21.

    Google Scholar 

  48. Rubin D, Lane W, Ludwig S. Child abuse prevention. Curr Opin Pediatr. 2001;13(5):388–401.

    CAS  PubMed  Google Scholar 

  49. Bandura A. Social cognitive theory: an agentic perspective. Annu Rev Psychol. 2001;52(1):1–26.

    CAS  PubMed  Google Scholar 

  50. U.S. Preventive Services Task Force 2016. Available from: http://www.uspreventiveservicestaskforce.org/.

  51. Dubowitz H, Feigelman S, Lane W, Prescott L, Blackman K, Grube L, et al. Screening for depression in an urban pediatric primary care clinic. Pediatrics. 2007;119(3):435–43.

    PubMed  Google Scholar 

  52. Dubowitz H, Prescott L, Feigelman S, Lane W, Kim J. Screening for intimate partner violence in a pediatric primary care clinic. Pediatrics. 2008;121(1):e85–91.

    PubMed  Google Scholar 

  53. Feigelman S, Dubowitz H, Lane W, Prescott L, Meyer W, Tracy JK, et al. Screening for harsh punishment in a pediatric primary care clinic. Child Abuse Negl. 2009;33(5):269–77.

    PubMed  Google Scholar 

  54. Kim J, Dubowitz H, Hudson-Martin E, Lane W. Comparison of 3 data collection methods for gathering sensitive and less sensitive information. Ambul Pediatr. 2008;8(4):255–60.

    PubMed  Google Scholar 

  55. Lane WG, Dubowitz H, Feigelman S, Kim J, Prescott L, Meyer W, et al. Screening for parental substance abuse in pediatric primary care. Ambul Pediatr. 2007;7(6):458–62.

    PubMed  Google Scholar 

  56. Lane WG, Dubowitz H, Feigelman S, Poole G. The effectiveness of food insecurity screening in pediatric primary care. Int J Child Heal Nutr. 2014;3(3):130–8.

    Google Scholar 

  57. Lane W, Dubowitz H, Frick K, Semiatin JN, Magder LS. The Safe Environment for Every Kid (SEEK) program: a cost-effective analysis. 139th Annual Meeting of the American Public Health Association Washington, D.C. 2011;.

  58. Boren SA, Balas AE. Evidence-based quality measurement. J Ambul Care Manage. 1999;22(3):17–23.

    CAS  PubMed  Google Scholar 

  59. Dolan BM, Yialamas MA, McMahon GT. A randomized educational intervention trial to determine the effect of online education on the quality of resident-delivered care. J Grad Med Educ. 2015;7(3):376–81.

    PubMed  PubMed Central  Google Scholar 

  60. Donovan AK, Wood GJ, Rubio DM, Day HD, Spagnoletti CL. Faculty communication knowledge, attitudes, and skills around chronic non-malignant pain improve with online training. Pain Med. 2016;17(11):1985–92.

    PubMed  Google Scholar 

  61. Lannon CM, Peterson LE. Pediatric collaborative networks for quality improvement and research. Acad Pediatr. 2013;13(6):S69–74.

    PubMed  Google Scholar 

  62. Peterson LE, Blackburn B, Puffer JC, Phillips RL. Family physicians’ quality interventions and performance improvement for hypertension through maintenance of certification. J Healthc Qual. 2014;.

  63. Riley M, Ahmed S, Lane JC, Reed BD, Locke A. Using maintenance of certification as a tool to improve the delivery of confidential care for adolescent patients. J Pediatr Adolesc Gynecol. 2017;30(1):76–81.

    CAS  PubMed  Google Scholar 

  64. Wiggins RE, Etz R. Assessment of the American Board of Ophthalmology’s Maintenance of Certification Part 4 (Improvement in Medical Practice). JAMA Ophthalmol. 2016;134(9):967–74.

    PubMed  Google Scholar 

  65. Institute for HealthCare Improvement. http://www.ihi.org/resources/Pages/Tools/PlanDoStudyActWorksheet.aspx. Accessed 5/15/17.

  66. Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50(3):217.

    PubMed  PubMed Central  Google Scholar 

  67. Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health Ment Health Serv Res. 2011;38(1):4–23.

    Google Scholar 

  68. Aarons GA. Mental health provider attitudes toward adoption of evidence-based practice: the Evidence-Based Practice Attitude Scale (EBPAS). Ment Health Serv Res. 2004;6(2):61–74.

    PubMed  PubMed Central  Google Scholar 

  69. Aarons GA, Ehrhart MG, Farahnak LR. The implementation leadership scale (ILS): development of a brief measure of unit level implementation leadership. Implement Sci. 2014;9(1):45.

    PubMed  PubMed Central  Google Scholar 

  70. Ehrhart MG, Aarons GA, Farahnak LR. Assessing the organizational context for EBP implementation: the development and validity testing of the Implementation Climate Scale (ICS). Implement Sci. 2014;9(1):157.

    PubMed  PubMed Central  Google Scholar 

  71. Chamberlain P, Brown CH, Saldana L. Observational measure of implementation progress in community based settings: the stages of implementation completion (SIC). Implement Sci. 2011;6(1):116.

    PubMed  PubMed Central  Google Scholar 

  72. Saldana L, Chamberlain P, Wang W, Brown CH. Predicting program start-up using the stages of implementation measure. Adm Policy Ment Health Ment Health Serv Res. 2012;39(6):419–25.

    Google Scholar 

  73. Falvo DR, Smith JK. Assessing residents’ behavioral science skills: patients’ views of physician–patient interaction. J Fam Pract. 1983.

  74. McDonell JR, Ben-Arieh A, Melton GB. Strong communities for children: results of a multi-year community-based initiative to protect children from harm. Child Abuse Negl. 2015;41:79–96.

    PubMed  Google Scholar 

  75. Aarons GA, Palinkas LA. Implementation of evidence-based practice in child welfare: service provider perspectives. Adm Policy Ment Health Ment Health Serv Res. 2007;34(4):411–9.

    Google Scholar 

  76. Saldana L, Chamberlain P, Bradford WD, Campbell M, Landsverk J. The Cost of Implementing New Strategies (COINS): a method for mapping implementation resources using the Stages of Implementation Completion. Child Youth Serv Rev. 2014;39:177–82.

    PubMed  Google Scholar 

  77. Drummond MF, Sculpher MJ, Claxton K, Stoddart GL, Torrance GW. Methods for the economic evaluation of health care programmes. Oxford University Press; 2015.

  78. Hsieh H-F, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88.

    PubMed  Google Scholar 

  79. Strauss AL, Corbin J. Basics of qualitative research: techniques and procedures for developing grounded theory. Thousand Oaks, CA: Sage; 1998.

    Google Scholar 

  80. Boyatzis R. Transforming qualitative information: thematic analysis and code development. Thousand Oaks, CA: Sage; 1998.

    Google Scholar 

  81. Fraser D. QSR NVivo NUD* IST Vivo Reference Guide. Melb QSR Int. 2000.

  82. Glaser BG, Strauss AL. The discovery of grounded theory: strategies for qualitative research. New York: Aldine de Gruyter; 1967.

    Google Scholar 

  83. Palinkas LA, Aarons GA, Horwitz S, Chamberlain P, Hurlburt M, Landsverk J. Mixed method designs in implementation research. Adm Policy Ment Health Ment Health Serv Res. 2011;38(1):44–53.

    Google Scholar 

  84. Fitzmaurice GM, Laird NM, Ware JH. Applied longitudinal analysis, 2nd edition. Wiley and Sons; 2011.

Download references

Acknowledgements

We appreciate the help of Jacqueline Hendricks in preparing the references.

Funding

National Institute of Child Health and Human Development. Dissemination and Implementation of the Safe Environment for Every Kid (SEEK) Model for Preventing Child Abuse and Neglect. 1R01HD092489-01A1.

National Institute of Child Health and Human Development. A supplement to Dissemination and Implementation of the Safe Environment for Every Kid (SEEK) Model for Preventing Child Abuse and Neglect. 1R01HD092489-02S1.

Author information

Authors and Affiliations

Authors

Contributions

HD, LS, LM, LP, and JL helped develop this study proposal. RB and UN assisted in the preparation of this paper and in the implementation of the study to date. The authors read and approved the final manuscript.

Authors’ information

Not applicable

Corresponding author

Correspondence to Howard Dubowitz.

Ethics declarations

Ethics approval and consent to participate

The study was approved by the UMB IRB—study protocol number: HP-00078770.

Consent for publication

Not applicable

Competing interests

Howard Dubowitz MD, MS owns the copyright to the SEEK materials. All revenues received from associated licenses and trainings help sustain the dissemination of the SEEK model.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1:

SEEK Protocol Paper – Supplemental Material [83, 84].

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dubowitz, H., Saldana, L., Magder, L. et al. Protocol for comparing two training approaches for primary care professionals implementing the Safe Environment for Every Kid (SEEK) model. Implement Sci Commun 1, 78 (2020). https://0-doi-org.brum.beds.ac.uk/10.1186/s43058-020-00059-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://0-doi-org.brum.beds.ac.uk/10.1186/s43058-020-00059-9

Keywords