Guide to evaluating interventions (differential attainment)

Evaluating interventions

 

What is an intervention?

By interventions, we mean making changes to the way we work to improve outcomes for medical students and trainees. Research identifies five main types of interventions:

  • Training for trainers, such as unconscious bias training
  • Training for trainees, such as to reduce exam anxiety
  • Leadership, such as raising awareness amongst educational leaders of the challenges for specific groups
  • Transparency around data and engaging with stakeholders. For example, making outcome data available publicly to show differences in attainment across groups
  • Designing recruitment and assessment to minimise bias, such as making tests as fair as possible and removing names from interview forms.

Interventions can be targeted at different levels:

  • System level, such as policy changes that aim to improve the fairness of training systems and processes such as the allocation of training posts.
  • Individual level, such as one to one coaching or support
  • Institutional or regional level, such as structured programmes for both trainers and trainee groups

 

Considerations when implementing an intervention

Work Psychology Group identified some helpful tips for developing an intervention:

  • Consider what your data means within your local context. The main issues giving rise to the variation may be different in different parts of the UK and therefore how you respond is also likely to be different
  • Clearly define the purpose of the intervention
  • Define the intended outcomes of the intervention - what does success look like?
  • Consider who the intervention is aimed at

It’s worth remembering that variation in attainment happens across a number of protected characteristics, including age, gender and ethnicity. As there is no single agreed cause of these variations, it can be difficult to identify one factor or specific area that should be targeted. And that’s why it’s useful to evaluate any interventions you introduce.


The benefits of evaluating interventions

Evaluating your interventions will help to continually improve their impact. We asked the Work Psychology Group to develop practical resources to help you plan your evaluation. By following this, you can:

  • assess the stronger or weaker elements of interventions and work towards improving them
  • identify potential unintended consequences (both positive and negative) of the intervention and correct for them if necessary
  • increase the evidence available to other organisations on the effectiveness of interventions, to help them improve the fairness of training

You should try and plan your evaluation alongside the design for your intervention. If this isn’t possible, evaluation can still be completed effectively and the principles of good evaluation design still apply.