NTS priority lists (private tool only)
NTS priority list for training posts
The NTS priority list is a different way of looking at the post specialty by site report that helps us prioritise which areas may need our support.
It uses four measures to identify the lowest scoring posts in the survey results
- most red flags
- most triple reds
- lowest average scores
- biggest drops in average score.
The score in each measure is calculated firstly as a proportion of possible outcomes, for example 3 red flags from a possible 10 indicators would give 30%. This is to account for the fact that not every post will have a score in every indicator. That percentage is then compared to all other posts and ranked. The number displayed is the percentage of all posts that have a better score (known as percentile rank).
If the percentile rank is equal to or greater than the selected threshold (default 99.5%), then the post will appear on this list.
Finally, posts that are on the priority list, but that are highlighted in grey, are included in the list because there fewer than 5 posts of that particular specialty in total. Because the low scores and changes in scores measures select the bottom 5% of posts in each indicator, posts in these smaller specialties appear more often on the list and should be interpreted with care.
Details about each of the four measures and other technical notes are provided below:
|Red flags||This measure tells us which posts have the most red flags in the current year. For each post, we calculate the percentage of possible indicator outcomes that are red and compare that percentage with all other posts.|
|Changes in scores||This measure is designed to spot posts which have substantially lower mean scores compared to the previous year across many different indicators.
We calculate the difference between the mean scores for last year and this year for every indicator for each post. We then compare each of these figures in groups of the same specialty and indicator. When a score is in the lowest 5% in any of those groups we flag it. We then use the same methodology to compare and rank posts as we do for the 'red flags' measure described above.
|Low scores||This measure is designed to spot posts that have low scores in different indicators, but aren't statistically significantly below the national average, meaning they don't have many red flags.
We compare this years' mean scores in groups of the same specialty and indicator. When a score is in the lowest 5% in any of those groups we flag it. We then use the same methodology to compare and rank posts as we do for the 'red flags' measure described above.
|Triple reds||This measure is designed to find the posts that have the most ongoing poor survey results.
For each post, we calculate the percentage of possible indicator outcomes that are red in each of the three previous years and compare that percentage with all other posts.
NTS priority list for training programmes
This report shows a summary score for each training programme based on the number of red and green flags it received in the programme code report in the national training survey. Programmes with a higher score have more green flags. Indicators included in this report are limited to those relating to educational standards, rather than safety standards.
The score each outlier type is worth is shown in the table below; the sum of these outlier scores divided by the number of possible outcomes is used as the overall score. This is to account for the fact that not every programme will have a score in every indicator. The benchmark score presented in the table is the average overall score for all programmes in the specialty or deanery/HEE local office.
The results are presented as a graph with a table below. Programmes can be compared in two ways by using the groups to compare button:
- Compare different deaneries/HEE local offices to one another for a particular programme specialty.
- Compare different programmes to one another within a deanery/HEE local office.
Use the drop down menu at the top of each graph to select a programme specialty or deanery/HEE local office.
Outliers and scores
|Green (above outlier)||2|
|Light green (in top quartile but with overlapping confidence limits)||1|
|White (non outlier)||0|
|Pink (in top quartile but with overlapping confidence limits)||-1|
|Red (below outlier)||-2|
|Grey or yellow (fewer than three survey respondents)||Not included|
NTS priority list based on individual questions
This report highlights training sites and posts where respondents have reported particular concerns in response to six key questions in the NTS. These questions are designed to highlight specific issues around workload, reporting systems and clinical and education supervision that might not be apparent from the results of an indicator analysis alone.
Training environments that meet the criteria for one of the questions listed below may need our support to ensure that safe working practices are being followed and patient safety is assured.
Questions and criteria
|Question master code||Question text||Priority list inclusion criteria|
|OPENQ10||On average, how many hours per week do you ACTUALLY work in this post (please give the average over your post)?||Multiple trainees in a site working more than 56 hours could indicate unsafe rota design or unreported staff shortages.
Sites are included on the list if there are multiple doctors (representing at least 15% of all doctors in training at the site) who have reported working more than 56 hours in an average week.
|GENHQ155 and 156||To what extent do you agree or disagree with the following statements?
GENHQ155: My educational supervisor is easily accessible should I need to contact them.
GENHQ156: The level of contact from my educational supervisor is appropriate for my training needs.
|Training organisations must ensure trainees have access to an educational supervisor (Promoting excellence R2.15). This report highlights all posts where at least two trainees have chosen 'Not applicable (I don't have an educational supervisor)' to either question.|
|GENHQ184||In this post how often (if at all) have you been asked to or felt pressured to submit a record of hours worked that were less than the hours you actually worked?||Multiple trainees choosing 'more than once' could indicate unsafe practices with regard to workload. This report highlights all posts where at least three trainees have chosen 'more than once'.|
|GENHQ196||In this post do you always know who is providing your clinical supervision when you're working?||Foundation doctors must at all times have on-site access to a senior colleague who is suitably qualified to deal with problems that may arise during the session. (Promoting excellence R1.8). This report highlights all posts where a foundation trainee has chosen 'No, there is no one I can contact'.|
|GENHQ206||In this post, out of hours, do you always know who is providing your clinical supervision when you're working?||Foundation doctors must at all times have on-site access to a senior colleague who is suitably qualified to deal with problems that may arise during the session. (Promoting excellence R1.8). This report highlights all posts where one or more foundation trainees have chosen 'No, there is no one I can contact'.|
|GENHQ100||To what extent do you agree or disagree with the following statement?
I have been made aware of how to report patient safety incidents and near misses.
|Multiple respondents choosing 'Strongly disagree' to this question may indicate an unsafe culture in the training environment.
This report highlights sites where multiple doctors in training (representing at least 10% of all doctors in training at the site) disagreed or strongly disagreed that they were made aware of how to report safety incidents and near misses.