Pursuit, in partnership with J-PAL and WorkRise, set out to design a Randomized Controlled Trial (RCT) to evaluate the impact of the Pursuit Fellowship. This process revealed key insights about RCT implementation, challenges, and future considerations that can help other organizations planning similar evaluations.
RCTs provide the most rigorous method for evaluating program impact, ensuring that observed outcomes result from the intervention rather than external factors. Pursuit’s Fellowship trains low-income individuals for high-paying tech jobs, making it an ideal candidate for an RCT to measure its true impact and potential for scaling.
Pursuit’s Mission and Model Pursuit trains low-income adults in New York City to become software engineers, increasing their earnings from ~$18K to over $90K. The Fellowship provides career training, employer partnerships, and financial accessibility through its Pursuit Bond model, ensuring long-term career success.
RCTs Are the Gold Standard Tool for Evaluation RCTs isolate the impact of a program by randomly assigning participants to treatment and control groups. This ensures that income gains and other outcomes are directly attributable to the intervention rather than external factors.
RCT Evaluations can help refine and expand Pursuit Research on sectoral employment programs demonstrates their effectiveness in increasing earnings and job stability. An RCT can validate these outcomes and provide crucial insights for refining and expanding workforce programs.
Conducting an RCT requires careful planning to ensure reliable results and minimize disruptions to program operations. Organizations must evaluate their readiness, ensure sufficient participant numbers, and allocate the necessary resources to support the evaluation.
Dedicated Resources and Planning RCTs require significant investment in funding, staff time, and research expertise. Without proper resources, organizations risk inconclusive results that fail to provide meaningful insights.
Cultural and Organizational Understanding of the RCT Staff buy-in is crucial, as concerns about randomization and ethical considerations can impact implementation. Open communication and training can align teams on the long-term benefits of rigorous evaluation.
Program Maturity and Timing An RCT should only be conducted when a program is stable and producing consistent outcomes. External disruptions, such as the COVID-19 pandemic, can alter program conditions and make evaluation results less reliable.
Understand Sample Size Requirements A large enough participant pool is needed to detect meaningful differences between the treatment and control groups. Organizations may need to adjust recruitment strategies or eligibility criteria to ensure a sufficiently robust sample.
While RCTs are valuable, they present challenges, particularly for programs with selective admissions and long-term interventions. Pursuit encountered unique obstacles in randomizing participants and ensuring the evaluation aligned with its program structure and mission.
Selective Admissions Process Pursuit’s admissions process identifies candidates most likely to succeed, making randomization difficult. The organization had to balance fairness in selection while maintaining a sample size large enough for a rigorous evaluation.
Employer Partnerships Limit Randomization Many of Pursuit’s employer partners provide dedicated hiring pathways, which limits the pool of applicants eligible for randomization. This required adjustments to ensure an adequate sample size without disrupting employer relationships.
Long-Term Engagement Complicates Measurement Pursuit provides ongoing career support, making it harder to determine the right timeframe for measuring success. The organization needed to define clear milestones for evaluation while accounting for Fellows’ varied career trajectories.
Pursuit remains committed to rigorous evaluation but recognizes the ethical and operational complexities of RCTs. The organization must carefully balance the need for strong evidence with its mission to provide life-changing opportunities to as many individuals as possible.
By strategically planning its RCT and addressing challenges proactively, Pursuit aims to refine its model, validate its success, and contribute valuable insights to the workforce development field.
Balancing Evaluation with Mission RCTs require some participants to be placed in a control group, preventing them from accessing services. Pursuit must weigh the benefits of strong evidence against the challenge of withholding opportunities from qualified individuals.
Aligning RCTs with Program Evolution Pursuit continuously refines its curriculum and employer partnerships, meaning an RCT must account for ongoing improvements. A flexible evaluation design can ensure results remain relevant despite program adaptations.
Scaling with Data-Driven Insights Even without a full-scale RCT, Pursuit can leverage rigorous data analysis to refine its model and guide expansion. Future evaluations will integrate findings into long-term strategy, ensuring both sustainability and impact.
