Sunday, 27 April 2014

Subjectivity is the curse on examinations for UK doctors - get rid of it

Subjectivity is the curse on examinations for UK doctors - get rid of it

There is currently a controversy raging in UK healthcare about doctors. For many years it was known and was passively accepted that if you were of Black or Minority Ethnic origins more so if you were an International Medical Graduate (IMG - a doctor with a primary medical qualification outside the UK or European Union) facing a Royal College examination you would have a much lesser chance of passing the examination. If you were of BME or IMG origins and were of a generous persuasion you would call this sub-conscious bias but most called this racism, whether it was racism or not. If you were representing the establishment you put out phrases that are superficial gibberish, such as 'reasons are multi-factorial and complex' but certainly not racism.

The issue came to a head with the MRCGP examination where in the new version of the examination the differences between white and non-white candidates were so gross that you would notice it even if you were colour blind.

The medical post graduate examinations conducted by the Royal Colleges are essentially about medical knowledge both theoretical and applied. Given that these are knowledge tests, why did the results show racial differences? We will not discuss racial supremist reasoning here.  Many of us will remember the days before the MCQs - the essay answers were often a demonstration of your wizardry in medical English. Apparently even in the MCQ based knowledge tests we can use linguistic jugglery so that a non-native English speaker comes out as having poor medical knowledge - we are not discussing that further here.

The curse of subjectivity

Applied knowledge in medicine is tested in vivas, OSCEs, with patients and simulated patients. Here the marking is done by examiners, that is where subjectivity comes in despite current best efforts, subjectivity is ruining careers.
The rest of this blog post is about subjectivity (the collection of the perceptions, experiences, expectations, personal or cultural understanding, and beliefs specific to a person - Wikipedia)

The sad paradox is knowledge especially in medicine is objective but part of the testing process of this knowledge is subjective. The tension that results from an objective topic tested subjectively is where the fundamental flaw lies. Where subjectivity exists, there bias exists and hence unethicality at the best and fraud at the worst exists.

Subjective assessments must not have a place in career make or break decisions such as exit examinations or in any arena where career progress or ability to practice the chosen profession can be stopped. Subjective assessments do have a place and can be used for progressing in learning and development - some of which are known as formative assessments. Must not be used for stop-go decisions where only objective assessments should be used.

Reducing or Eliminating subjectivity

Examiners in vivas, OSCEs, patient encounters, interview and other areas currently suffering due to subjectivity, are generally given questions - they should also be given answers and as long as the candidates answers fit in with the recognised accepted answers the candidate passes, when the answers fit in with recognised unacceptable answers the candidate fails and where the answers fit in with recognised borderline, a published formula for accepted number of borderline for a pass or fail should be defined (no, this is not the 'borderline method' that is used in standard setting).

This may beg the question whether vivas are needed at all - verbal communication is essential in all walks of life and especially so in healthcare; a candidate should be able to answer effectively and accurately under stressful verbal conditions and hence vivas are needed but the subjectivity of the vivas must be eliminated.

Subjectivity cannot be sometimes avoided but when forced to use it the answers should be 'force fit' in a pre-defined uniform manner and the candidates be assessed against that uniform force fit. The candidate does not have to know what the defined force-fit answer is but all candidates would be marked against the same answer.  

Lets look at an example: Let us assume that in a scenario where there is a certain level of oxygen desaturation which does not impact on life or limb but where a candidate has to act - say an peripheral oxygen saturation that has fallen from 98 to 89 but where the patient is otherwise very stable. The candidate has to make preparations for an adverse eventuality but there was no need to act immediately. Let us also assume that currently this is subjective and hence an assessor would mark someone and this would be variable (depending on the other skills of the candidate). Let us try a force-fit answer for this scenario - the examiner would be given a set of answers and would give the candidate a mark for each correct answer, for instance, a) the patient if conscious was asked if she was okay within two seconds 1 mark
b) the pulse oximeter probe was checked and re-applied within 4 seconds
c) capnograph reading checked within 6 seconds
d) the oxygen flow and any gas mix ups were checked within 8 seconds
d) airway tube position checked within 10 seconds
e) airway change kit and reversal drugs asked to be brought in and kept ready with 12 seconds
etc. You get the picture.

These answers may not be based on evidence because there is no evidence to base it on. However, for the purposes of the assessments the answers are defined on the basis of agreement between examiners and are used uniformly with all candidates. Then the chances of the examiner being influenced by mastery of the language, social status of an accent, the image projected by clothes, the false confidence provided by a charming smile or colour of the skin would be less.

Subjective experts are simply socially acceptable influential frauds providing a certain voyeuristic celebrity value when they are reviewing wines, films or restaurants. Techniques similar to those have no place in medical examinations. It is of course a completely different story that the British are not able to trust the training provided to their young doctors for somewhere between a minimum five years (in the case of general practice) or an approximate minimum of twelve years in the case of surgeons that makes an 'exit' exam essential to cross check knowledge (which is then pretty badly due to the subjective components). In the USA exit exams are not mandatory, they are voluntary, the Americans obviously have a great degree of confidence in their trainers, trainees and training system. The British system needs reform and a commitment to eliminate subjectivity when the stakes are high could be core to whether the UK will ever have a equitable outcome in examination results.


Follow me on twitter @HemadriTweets

This blog has argued for reducing or eliminating subjectivity from re-validation
We have discussed differential results in surgical Royal College examinations