Background: Scientists may be biased against studies with certain attributes, such those labeled as preliminary pilot/feasibility studies. This study assessed the impact of key study characteristics on behavioral scientists’ interpretation of study merit and impact.

Methods: This study used a 2x2x2x2 factorial design. Abstracts from five published obesity prevention studies were modified to generate 16 nearly identical versions of each abstract. Abstract versions varied by four factors: sample size (n=20 vs. n=150), statistical significance (P<.05 vs. P>.05), study design (single group vs. randomized two group) and preliminary study status (presence/absence of pilot/feasibility language). Reported effect sizes, intervention components, population, and duration were maintained across all versions. Respondents were provided with one randomly selected version of each of the five abstracts and blinded to the existence of other versions. Respondents rated each abstract on five attributes; significance of the scientific question, methodological rigor, study innovation, writing clarity, whether the findings warrant further testing, and probability of obtaining meaningful results in a subsequent definitive trial using a 1-10pt Likert-scale. Multilevel mixed-effects models evaluated the main effects and all 2- and 3-way interactions with preliminary study status.

Results: Respondents (N=247, years since terminal degree 9.1±9.5) indicated preliminary study status did not impact scores on any scale. Abstracts presenting statistically significant effects were rated as more significant (mean difference 0.31, 95CI 0.08-0.55, Cohen’s d=0.17), rigorous (0.57, 95CI 0.30-0.84, d=0.30), innovative (0.38, 95CI 0.11-0.65, d=0.22), clearly written (0.45 95CI 0.15-0.74, d=0.30), warranted further testing (0.90, 95CI 0.60-1.20, d=0.55), and had meaningful results (1.06, 95CI 0.77-1.35, d=0.54). Abstracts with a randomized design were rated as more rigorous (1.04, 95CI 0.77-1.31, d=0.55) and innovative (0.30, 95CI 0.03-0.58, d=0.17). No other statistically significant effects were observed.

Conclusion: Contrary to hypothesis, pilot/feasibility studies were not viewed differently compared to non-pilot/feasibility studies. Abstracts presenting statistically significant findings, irrespective of study design, preliminary study status, and sample size were rated higher on scientific merit/impact compared to abstracts presenting the same effect sizes but lacking statistical significance. The high ratings attributed to statistically significant findings is concerning and may lead scientists to overlook other important aspects of studies to evaluate quality. 

Background: Designing, conducting, and clearly reporting pilot/feasibility studies (PFS) informs decisions about the evaluation of behavioral interventions in large-scale trials. Numerous reporting guidelines, checklists, recommendations, and frameworks (GCRF) exist, yet their use in PFS is unknown. A scoping-bibliometric review was conducted to identify the application of GCRF in the reporting of PFS.

Methods: A multi-phase approach identified a final sample of PFS. PubMed, Embase, and Web of Science were searched for PFS published from 01/01/2018-10/01/2021 with “pilot” or “feasibility” and “intervention” in the title/abstract. In these PFS, journals were identified that published ≥50 studies and were an outlet for behavioral interventions. A review of titles/abstracts within these journals identified the final sample of behavioral intervention PFS. A total of 91 GCRFs were identified from the literature and placed into 9 categories: study/methodology (e.g., CONSORT, 8-articles), intervention reporting (e.g., TIDieR, 7-articles), adaptation/modifications (e.g., FRAME, 7-articles), implementation/process evaluation (e.g., RE-AIM, 7-articles), feasibility (e.g., Bowen 2009, 8-articles), pilot guidance (Arnold 2009, 29-articles), translational science (e.g., ORBIT, 6-articles), intervention development (e.g., Bartholomew 1998, 4-articles), and scale-up (e.g., WHO 2010, 14-articles). Citations of a GCFR were captured via text-mining procedures of references in the PFS and analyzed descriptively.

Result: A total of 33,955 unique PFS were identified. Of these, 23 behavioral intervention journals (e.g., Pilot and Feasibility Studies, BMJ Open) published 3,781 PFS (avg. 167; 53-638). After full-text screening, 1496 behavioral intervention PFS were included with mhealth (40%), mental health (33%), and physical activity (29%) the 3 most common topics. Overall, 40.2% of PFS did not cite any GCRFs and an additional 31% only cited one, most commonly a CONSORT-related guideline. Most cited GCRFs were study/methodology (37.9%), pilot guidance (17.8%), and translational science frameworks (12.6%). Remaining categories were cited less (0.3% to 7.6%).

Discussion: The use of GCRFs in PFS is low, with the major focus on internal validity. Few PFS utilized GCRFs considered important for early-stage intervention design and testing such as, intervention development or reporting, or implementation/process evaluation and feasibility testing. Increasing awareness and use of existing GCRFs by authors and journal editors is required to enhance transparency and reproducibility in PFS.

Purpose: In the behavioral sciences, where interventions often consist of delivering complex, multi-component behavior change techniques, reporting key aspects of feasibility during the initial testing of the intervention is essential. Reporting trial and intervention feasibility in pilot/feasibility studies creates a foundation for the optimization and successful scaling-up to larger trials. The aims of this study were to: 1) conduct a scoping review of the reporting of feasibility indicators in behavioral pilot/feasibility studies published through 2020, and 2) describe trends in the amount and type of feasibility indicators reported in studies published across three periods: 1982-2006, 2011-2013, and 2018-2020.

Methods: A search of four online databases (PubMed, Embase, EBSCOhost, Web of Science) for health behavior pilot/feasibility studies published up to 12/31/2020 was conducted and a random sample of 600 studies, 200 from each of the three timepoints (1982-2006, 2011-2013, and 2018-2020) was included in this review. The presence or absence of feasibility indicators, including recruitment, retention, acceptability, attendance, compliance, and fidelity, was identified/coded for each study. Univariate logistic regression models were employed to assess changes in the reporting of feasibility indicators across time.

Results: A total of 16,365 unique articles were identified and screened to arrive at the final sample of 600 studies. For the total sample, 428 (71.3%) studies provided recruitment information, 595 (99.2%) provided retention information, 219 (36.5%) reported quantitative acceptability outcomes, 157 (26.2%) reported qualitative acceptability outcomes, 199 (33.2%) reported attendance, 187 (31.2%) reported participant compliance, and 85 (14.2%) reported intervention fidelity outcomes. When compared to the Early Group (1982-2006), studies in the Late Group (2018-2020) were more likely to report recruitment (OR=1.60, 95%CI:1.03-2.49), acceptability-related quantitative (OR=2.68, 95%CI:1.76-4.08) and qualitative (OR=2.32, 95%CI:1.48-3.65) outcomes, compliance outcomes (OR=2.29, 95%CI:1.49-3.52), and fidelity outcomes (OR=2.13, 95%CI:1.21, 3.77).

Conclusions: The reporting of feasibility indicators within behavioral pilot/feasibility studies has improved across time, but key aspects of feasibility, such as fidelity, are still not reported in the majority of studies. Given the importance of behavioral intervention pilot/feasibility studies in the translational science spectrum, there is a need for improving the reporting of feasibility indicators to better inform the process of scaling-up to larger trials. 

Obesity is a pressing concern for clinical and public health professionals, particularly in rural areas. We used a Hybrid Type-III effectiveness-dissemination trial to examine program reach (i.e., number, proportion, and representativeness of participants), retention, effectiveness, and cost of an evidence-based, digitally-delivered, 12-month weight management intervention for rural primary care patients. Patients were referred to the intervention through 1 of 4 methods: 1) point of care referral from their physician with telephone follow-up (POC+); 2) point of care referral without follow-up (POC-); 3) electronic medical record-derived letter with telephone follow-up (Letter+); 4) electronic medical record-derived letter without follow-up (Letter-).

Over 16 weeks, 573 referrals were made and 97 patients, representative of the region, enrolled in the program (59% female; 94% Caucasian; 97% Non-Hispanic). Letter referrals reached a significantly higher proportion of patients than POC (100% vs 17%) and yielded more participants (12% vs 8%, p<.05). Patients receiving telephone follow-up were more likely to be screened (47% vs 7%; p<.001) and enroll (15% vs 7%, p<.001) when compared to those without follow-up.

Participants completed weigh-ins via Bluetooth-enabled home scale. Participants recorded 7.1 (SD=18.4) weights on average. Among participants with at least two recorded weights (n=54), there was a significant difference in weight between baseline (M=224.0lbs, SD=52.0) and final weigh-in [(M=216.7lbs, SD=47.2); t(54)=4.49, p<.001]. Weight change did not differ among the four referral strategy groups (F(3,51)=1.93, p=.14).

Cost per participant was $372. The cost per participant that weighed in once (n=77) was $469, compared to $669 for those that weighed in at least twice (n=54). Cost per participant achieving 3% or more (n=25) and 5% or more (n=17) body weight loss was $1,445 and $2125, respectively. Total recruitment and intervention costs per participant enrolled in each referral strategy were: POC with ATF, $359; POC only = $371; Letter with ATF = $395; Letter only = $361.

When examining the overall impact of the different referral strategies it is important to consider reach, effectiveness, and cost; results from this study indicate the Letter- condition is the superior referral choice. However, local context should be considered before selecting a referral strategy.

Methods: In total, 50 individual semi-structured interviews were conducted with three groups of stakeholders; scheme referrers (n = 9); scheme deliverers (n = 22); and referred patients (n = 19). Scheme refers were recruited using convenience sampling techniques while scheme deliverers and patients were purposively sampled. Referrer guides covered: frequency of and reasons for referral, facilitators and barriers to referral, patient journey through the scheme and process for referral. Topics covered within scheme deliverer guides included; referral rates, types of referrers and engagement of surgeries. Guides for patients covered: experiences of the referral process and reasons for uptake. Themes were derived following the iterative phases of a thematic analysis approach.

Results: Five key themes were developed from analyses; referrer characteristics, geographical disparities in referral and scheme access, reinforcements for awareness of the scheme, patient characteristics and processes and context underpinning a referral. Barriers and facilitators to referral were found to be entwined within and across themes, with an overall high concordance of views between all stakeholder groups. Findings highlighted referral barriers that have persisted since the earlier trial (e.g. a lack of consultation time and a lack of referral feedback) and newly identified barriers (e.g. a lack of scheme awareness and a referral system perceived to be time intensive and disjointed). Key referral facilitators included patient self-referrals, a growing scheme reputation and promotional activities of scheme deliverers.

Conclusions: NERS is an established, evidence-based programme, which continues to support thousands of patients across Wales each year. Findings provide evidence that could inform the further development of NERS and wider exercise referral schemes to ensure the referral process is timely, efficient and equitable.