Guide to evaluating interventions (differential attainment)
This list of available measures will help you find the most appropriate way to evaluate the impact of your intervention. Read more about measures.
- Consider what validated measures (i.e. previously published measurement instruments) may already be available rather than designing your own
- Frame the survey questions around the aims of the intervention and the short and long-term success outcomes where appropriate. Consider utilising Kirkpatrick's evaluation model to frame the questions, particularly at the learning and behaviour levels (see examples within this guide).
- Consider the scale that will be used for respondents to answer; scales with descriptions (i.e. strongly agree to strongly disagree) are commonly used. When deciding what scale to use, you should consider the type of data required, how the information will be used and the statistical methods that will be employed. Clearly label each point on the scale and give clear instructions for how to answer questions.
- Make sure the language used is clear and unambiguous.
- Consider the length of the survey; include enough questions to adequately measure the domains of interest but of a length that will maximise completion and engagement.
- Consent statements can be included in survey documents if required, or collected separately.
- It may be helpful to seek feedback from a small number of individuals on the survey content and whether it is all applicable to those responding. This is particularly important if you are using an existing measure rather than designing your own.
Interview/focus group guidance
- Frame the interview/focus group questions around the aims of the intervention and the short and long term success outcomes, where appropriate.
- Consider the length of the interview or focus group session; plan enough time to adequately explore the domains of interest but keep it a reasonable length that will maximise attendance and engagement.
- Provide a clear introduction that outlines the purposes of the research, seeks their consent, provides information about confidentiality and provides them with the opportunity to ask any questions.
- If you wish to audio record the interview, seek permission and be clear about what will happen to the recordings and transcripts after the research has been completed.
- Think about the physical environment. This can be important for encouraging respondents to relax and to be open and honest.
Tips for interviewer
- Clarity - Try and ask simple and short questions, keeping them open ended where possible.
- Considerate and respectful - tolerate pauses and give the respondent time to think. Don't make any value judgments about what they are saying.
- Rapport - Listen to what the respondent is saying and facilitate the interview in a way that allows the respondent to feel comfortable expressing their view.
- Open and flexible - be flexible when running the interview. Adapt and probe where needed. Earlier interviews may act as test runs to help improve the questions before conducting later interviews.
- Exploring - Short responses may be provided first. Follow up by asking probing questions to elicit further detail and seek clarification.
- Balanced - Manage the talking from both sides. Skilled interviewers talk less and listen more. Ensure you are allowing sufficient time for the respondent to present their views.
- Take care with interpreting - It can be helpful to clarify and sum up what the respondent is saying, but this may sway the respondent in a way that is in line with the interviewer's intentions or goals.