GMC multi-source feedback study
What were the key findings?
We asked for a study of the patient and colleague questionnaires that we developed to support the revalidation process. We wanted to look in particular at their psychometric aspects.
The study showed the questionnaires provide a reliable way to assess patient and colleague feedback. Based on two important measures, they:
- were found to have good ‘internal consistency’ based on their Cronbach’s alpha coefficient.
- would have acceptable ‘reliability’, with the return of 34 patient responses or 15 colleague responses.
The report makes the following recommendations.
-
The patient questionnaire has limitations but is robust to use in the rollout of patient feedback for revalidation. It’s possible a patient’s report might be biased by non clinical aspects of care, as well as socio-demographic profile of the patient sample.
-
The colleague questionnaire is robust to use in the rollout of colleague feedback for revalidation. Guidance about the mix of colleagues who might take part appears appropriate. We should bear in mind the possibility of systematic bias in a colleague’s report based on non-clinical aspects of care. And the extent of colleagues’ familiarity with the doctor.
-
Questionnaires should be used by doctors for reflection for the first two to three years. Until mandatory participation in multi-source feedback becomes the norm.
-
Results from using the questionnaires should be interpreted within the context of the doctor’s practice, their personal characteristics, and the mix of patients they see.
-
The process of gathering multi-source feedback should include completion of a self-assessment tool to aid reflection.
-
Central coordinated support delivered by a professional survey organisation and admin support will be needed for doctors undertaking these surveys.
-
Achieving robust benchmarking across a population of doctors will only be possible by the use of one type, or a limited number of types, of patient and colleague questionnaires.
-
Further research exploring, refining and informing the use and potential of multisource feedback from the patients and colleagues of doctors is essential.
Why did we commission this research?
To support the introduction of revalidation, we developed patient and colleague questionnaires, with content based on the principles of Good medical practice. We commissioned this research to investigate the utility of these questionnaires for doctors to gather evidence about their practice from patients and colleagues for the purposes of revalidation.
What did the research involve?
Researchers invited all non-training grade doctors from 11 sites in England and Wales to take part in a survey between August 2008 and January 2011. Around 1,000 doctors contributed questionnaire data. Feedback was obtained by these doctors from over 30,000 patients and 17,000 colleagues using our developed questionnaires.
Psychometric properties of the questionnaires were investigated via statistical techniques based on Classical Test Theory, Generalisability Theory and Rasch analysis. Regression modelling was undertaken to explore the determinants of patient responses, to determine whether sampling biases might exist.
