GMC multi-source feedback study
What were the key findings?
We commissioned a study of the patient and colleague questionnaires we developed to support the revalidation process, looking in particular at their psychometric aspects.
The study showed that the questionnaires provide a reliable way to assess patient and colleague feedback, based on two important measures, they:
- were found to have good ‘internal consistency’ based on their Cronbach’s alpha coefficient.
- would have acceptable ‘reliability’, with the return of 34 patient responses or 15 colleague responses.
The report makes the following recommendations.
The patient questionnaire has limitations but is sufficiently robust to use in the preliminary rollout of obtaining patient feedback for revalidation. We should bear in mind that it’s possible a patient’s report might be biased by non clinical aspects of care, as well as socio-demographic profile of the patient sample.
The colleague questionnaire is sufficiently robust to use in the preliminary rollout of obtaining colleague feedback for revalidation. Guidance about the mix of colleagues who might participate appears appropriate. We should bear in mind the possibility of systematic bias in a colleague’s report based on non-clinical aspects of care, and the extent of colleagues’ familiarity with the doctor.
In view of a lack of availability of robust specialty-specific benchmarks, questionnaires should be used by doctors for reflection for the first two to three years, until mandatory participation in multi-source feedback becomes the norm.
Results from using the questionnaires should be interpreted within the context of the doctor’s practice, their personal characteristics, and the mix of patients they see.
The process of gathering multi-source feedback should include completion of a self-assessment tool to aid reflection.
Central coordinated support delivered by a professional survey organisation and local administrative support will be needed for doctors undertaking these surveys.
Achieving robust benchmarking across a population of doctors will only be possible by the use of one type, or a limited number of types, of patient and colleague questionnaires.
Further research exploring, refining and informing the use and potential of multisource feedback from the patients and colleagues of doctors is essential.
Why did we commission this research?
To support the introduction of revalidation, we developed patient and colleague questionnaires, with content based on the principles of Good medical practice. We commissioned this research to investigate the potential utility of these questionnaires as a means by which doctors might gather evidence about their professional practice from their patients and colleagues for the purposes of revalidation.
What did the research involve?
Researchers invited all non-training grade doctors from 11 sites in England and Wales to take part in a survey between August 2008 and January 2011. Around 1,000 doctors contributed questionnaire data and feedback was obtained by these doctors from over 30,000 patients and 17,000 colleagues using our developed questionnaires.
Psychometric properties of the questionnaires were investigated via statistical techniques based on Classical Test Theory, Generalisability Theory and Rasch analysis. Regression modelling was undertaken to explore the determinants of patient responses, to determine whether sampling biases might exist.