Education data reporting tool help

What do the results mean?

How do I read reports?

Main reports

The single and multiple year reports show the results in a coloured grid.

Each cell contains the mean score and a colour which indicates if the score is an outlier.

  • Red: a red outlier is a score in the bottom quartile of the benchmark group, and the confidence interval does not overlap with that of the benchmark mean.
  • Pink: a score in the bottom quartile, but the confidence interval overlaps with that of the benchmark mean.
  • White: a score in between the top and bottom quartiles of the benchmark group.
  • Light green: a score in the top quartile, but the confidence interval overlaps with that of the benchmark mean.
  • Dark green: a green outlier is a score in the top quartile of the benchmark group, and the confidence interval does not overlap with that of the benchmark mean.
  • Grey: fewer than three results (n<3). We only report results which have three or more responses.
  • Yellow: no results (n=0).

Aggregation and indicator reports

  1. This part of the graph (in white) shows the range between the highest score and 100 for this indicator for the benchmark group. Hover over this segment to see the percentage difference between the highest score and 100. For example if the highest score is 85, the number shown will be 15.
  2. This part of the graph (in grey) shows the 75th percentile to the highest score for this indicator for the benchmark group. Hover over this segment to see the percentage difference between the 75th percentile and the highest score. For example if the 75th percentile is 80 and the highest score is 95, the number shown will be 15.
  3. This symbol (upper black cross) represents the upper confidence limit of the score for this report group. The score for the report group will change according to the filters you apply. Hover over this symbol to see the upper confidence limit value. 
  4. This part of the graph (in light blue) shows the 25th to 75th percentile for this indicator for the benchmark group. Hover over this segment to see the percentage difference between the 25th and 75th percentile. For example, if the 25th percentile is 20 and the 75th percentile is 80, the number shown will be 60.
  5. This symbol (red dot) represents the score for this report group. The score is the mean of the scores of all respondents (n) within the report group. The score for the report group will change according to the filters you apply. Hover over this symbol to see the score.
  6. This symbol (black square) represents the mean result for all respondents within the benchmark group. Hover over this symbol to see the mean value.
  7. This symbol (lower black cross) represents the lower confidence limit of the score for this report group. The score for the report group will change according to the filters you apply. Hover over this symbol to see the lower confidence limit value.
  8. This part of the graph (in dark blue) shows the lowest score to the 25th percentile for this indicator for the benchmark group. Hover over this segment to see the percentage difference between the lowest score and the 25th percentile. For example if the lowest score is 5 and the 25th percentile is 20, the number shown will be 15.
  9. This part of the graph (in white) shows 0 to the lowest score for this indicator for the benchmark group. Hover over this segment to see the percentage difference between the lowest score and the 25th percentile. For example if the lowest score is 5, the number shown will be 5.
  10. Group label. In the example shown, 'Allergy' is a post specialty. Please be aware of the report you are looking at and the filters applied before drawing any conclusions.

Hover over a section to see the percentage that the section covers. For example if the lowest score is 5 and the 25 percentile is 20, the number shown if you hover over the dark blue section will be 15.

  • Mean: the mean score of the report or benchmark group.
  • Lower CI: lower confidence limit for the report or benchmark group.
  • Upper CI: upper confidence limit for the report or benchmark group.
  • n: number of respondents in the report group for this report.
  • n range: number of respondents in the report group for this report as a range.
  • Min: the lowest score of all respondents in the benchmark group.
  • Q1: upper value of the first quartile or the 25th percentile of the benchmark group.
  • Median: the median score of the benchmark group.
  • Q3: the lower value of the first quartile or the 75 percentiles of the benchmark group.
  • Max: the highest score of all respondents in the benchmark group.

 

  This report National
Country Mean Lower CI Upper CI n Range Mean Min Q1 Median Q3 Max Lower CI Upper CI N
England 79.09 78.93 79.25 44216 to 44220 79.30 4.00 71.00 81.00 95.00 100.00 79.15 79.44 53074
Northern Ireland 81.21 80.43 81.98 1626 to 1630 79.30 4.00 71.00 81.00 95.00 100.00 79.15 79.44 53074
Scotland 80.00 79.53 80.48 5001 to 5005 79.30 4.00 71.00 81.00 95.00 100.00 79.15 79.44 53074
Wales 80.46 79.80 81.12 2226 to 2230 79.30 4.00 71.00 81.00 95.00 100.00 79.15 79.44 53074

Question item and programme specific reports

Question item reports display as a bar chart.

These reports show responses to individual questions in the survey displayed in bar charts (private tool only).

The question text and n or n range (the number of doctors who answered the question) are shown above the chart.

You can hover over the bars to see percentages for each answer.

What are indicators and how are indicator scores calculated?

What are indicators?

In the survey doctors answer questions based on their experience of training. Questions are grouped by theme and we refer to these groupings as indicators. We use indicators to measure how doctors feel about specific areas of training.

How are indicator scores calculated?

We use each doctor's score to build reports. For example to show results by site we average the scores of doctors at that site or to show results by specialty we average the scores of doctors in that specialty.

What are benchmarks?

What are benchmark groups?

Results are calculated by comparing a report group to a benchmark group. Doctors' scores contribute to the score for the report groups they are in (specialty, site, training level, etc). The scores are also used to calculate the benchmark score.

The benchmark group is the group of respondents whose scores you are comparing your report group to. For example if your report group is general surgery your benchmark group would be all surgical specialties combined.

The report group is the group you are interested in looking at. For example you can look at a post specialty by trust/board report to see how General psychiatry at your trust compares to General psychiatry at other trusts. In this example the benchmark group would be all psychiatry.

In the example below, in the benchmark group of All psychiatry (including General psychiatry) six out of 12 doctors responded positively. This gives a benchmark score of 50. In the report group General psychiatry, two out of three doctors responded positively. This gives a mean score of 66.67 for the report group. Dr A's score contributes to both the report group and the benchmark score.

Because different reports compare different groups, they use different benchmark groups. For example Dr A is in an F1 General psychiatry post in Big City mental health trust. The scores calculated from Dr A's answers will contribute to, and be compared against, different benchmark groups in the different reports.

How do we use benchmark and reporting groups?

Reports compare the doctor's reporting group (eg level or specialty) against a benchmark group.

Different reports use different benchmark groups. This is to make sure that we are always comparing groups fairly.

For example if you filtered the post specialty by site report for general surgery at a specific hospital, the report would show you how that group compared with all surgery posts at all hospitals.

How to use the benchmark tables to find your benchmark group

  • Find the reporting and benchmark groups table for the report you are looking at
  • Look for your specialty, training level etc to see the benchmark for your reporting group
  • Use the benchmark groups table for more detail on what is included in the benchmark group.

Read the reporting and benchmark groups tables

What are outliers and how do we calculate them?

What are outliers?

Outliers are scores that are significantly higher or lower than the average score. In the reporting tool they are shown as red or green flags.

The benchmark group is the group of respondents whose scores you are comparing your report group to. For example if your report group is general surgery your benchmark group would be all surgical specialties combined.

How do we calculate outliers?

To calculate outliers we first calculate the benchmark group scores. We sort all of the scores into the benchmark group in order. We split the scores into quarters.

We also calculate the mean score for the benchmark group (national mean). The national mean is shown by a black square.

Scores in the bottom and top quartiles result generate outliers. These are shown as red (below) and green (above) flags.

What are confidence intervals?

Confidence intervals are the range of values that, to a certain percentage of confidence (95% in the NTS), we are sure the 'true' mean value lies in, accounting for random error. That is, for 95% of confidence intervals, the true mean lies within these range of values.

Hospital A: 100 doctors complete the survey, most of them agree that training is good. We have lots of data and a high level of agreement within the group. This means that we can be confident about the score and the confidence interval is small.

Hospital B: 3 doctors complete the survey, each of them gives a different response. We have a small amount data and a no agreement within the group. This means that we are less confident about the score and the confidence interval is large.

Quartiles are also used to determine the flags. The first quartile refers to the value at which 25% of indicator scores lie below and 75% of scores lie above. The third quartile refers to the value at which 75% of indicator scores lie below and 25% of scores lie above.

We calculate the confident intervals for the national mean and the report group. Finally, we compare your score (red dot) and confidence interval (red line) to the benchmark score and confidence intervals (black square and line). This shows us if your score is a red or a green flag.

This works equally with the red flags and the bottom 25% quartile. Scores that could range across the bottom quartile will flag as pink.

Below are some examples of how the scoring system works.

Figure 1: Hospital A has scored 33 on the indicator. This falls in the first quartile with confidence so will score a red flag.

Figure 2: Hospital B has scored 44 on the indicator, this falls in the first quartile. However, its confidence interval spans across the quartile so will score a pink flag.

Figure 3: Hospital E has scored 67 on the indicator, it is in neither the first or third quartile so therefore will score a white flag.

Figure 4: Hospital H has scored 78 on the indicator, this falls in the third quartile. However, its confidence interval spans across the third quartile so will score a grass flag.

Figure 5: Hospital H has scored 82 on the indicator. This falls in the third quartile with confidence so will score a green flag.

What happens next?

Deaneries/Health Education local offices

The results are used as a screening tool by deaneries/Health Education local offices to help them decide where there might be problems that need looking into.

Usually, survey results are triangulated with other sources of information to help ensure that resources are allocated to the right areas.

Deaneries/health education local offices and local education providers will review their regional and local results straight away and start planning their quality assurance and improvement activity for the next few months. Deaneries/Health Education local offices report back to us periodically through their dean's reports.

Royal colleges

Royal colleges use results from the programme specific questionnaires to monitor the delivery of curricula.

General Medical Council

We use survey data to help quality assure medical education and training across the UK.