As part of the CDEI’s series of introductory blogs, we will also be highlighting specific use-cases of data-driven technologies that we have uncovered through our COVID-19 repository and are currently being implemented. This week we’re looking at an Oxford-based study of a new diagnostic tool, which uses machine learning to predict whether a patient has coronavirus within an hour of them entering a hospital.
An overview of AI-driven testing for COVID-19
In a project at Oxford University Hospitals NHS Foundation Trust (OUH), researchers have built an AI-driven test to screen for COVID-19, in the first hour of a patient arriving at an emergency department. The AI model - trained using laboratory bloods, blood gases, and observations recorded routinely during 115,000 presentations to Oxfordshire’s Emergency Departments - looks for a ‘biochemical and physiological signature’ of COVID-19. Whereas the swab test for COVID-19 can take up to 48hrs (although the average is nearer 12hrs), and is estimated to return a falsely negative result for a third of cases, the AI-driven test is calibrated to achieve a high negative predictive value. This means that a negative result from the AI test safely rules out the disease - especially important in hospitals where patients often fall into the most vulnerable groups. After testing the model prospectively for all patients coming to A&E or admitted across four hospital sites over a two-week period, the results showed that the test correctly predicted the COVID-19 status of patients 92% of the time. This was across over 3,000 attendances to A&E and 1.700 admissions to hospital.
Possible benefits
On presentation at hospital, patients with COVID-19 are often difficult to distinguish from patients with other illnesses, making it difficult for medical professionals to safely triage the different groups. At the moment testing for coronavirus typically takes up to 48 hours, thus exacerbating this triaging issue. Aside from the delays, swab testing requires the use of an expensive PCR machine, which itself needs trained laboratory staff to oversee its use, as well as chemicals that are in high global demand - not all of which will be available to every hospital. In a pilot, the AI-driven testing tool has been shown to be considerably faster - taking less than one hour to turnaround. The model also uses data from existing routine tests, and so requires considerably less financial outlay and infrastructure.
The potential for algorithmic bias does remain a concern for AI-driven testing in a clinical setting, but in this case the researchers have already performed some analyses during the validation study to investigate whether or not the algorithm being used shows ethnic, gender, or age bias. This analysis showed that rates of misclassification were not higher in BAME vs. White British patients, female vs. male patients, or over-60s vs. under-60s.
The pilot
The test is being trialled and studied as a collaboration between Oxford University Hospitals and Oxford University, in a project led by Dr Andrew Soltan. It has been built using data from 115,000 presentations and 72,000 admissions to Oxfordshire’s hospitals, with ethical approval from the HRA (the central NHS authority). However, the test has not been independently validated using patient data from other hospitals. This means that the results are still preliminary, prior even to the clinical trial stage, and researchers cannot yet be certain that the same level of accuracy will be achieved when the model is widely applied.
Making the test more widely available
As the testing is based entirely on data that is routinely collected on admission across NHS trusts, it should require relatively little investment to deploy the model in different hospitals - even those less digitally advanced (Oxford University Hospitals is a foresight trust, and therefore has more advanced capabilities than others). At the start of the trial, researchers did consider integrating the pre-existing health data of patients - including changes from previous blood tests and historical health conditions (e.g. diabetes) - which would have required a high level of digitisation. However, results demonstrated that taking the data from presentation alone provided sufficiently accurate results.
Given that there is no need for new infrastructure to deploy the model, this type of testing could also have a high level of impact in the developing world.
Next steps
Following on from the development and validation of the test, the next stage would be a clinical trial in Oxford. The researchers are collaborating with another trust, University Hospitals Birmingham, and this will allow validation on data from other centres to assess generalisability of the AI test. Although the initial sample size for validation was comparatively large, these are still very early stages. Subsequently, there is a risk that the model doesn’t work with populations outside of those visiting the Oxford University Hospitals, and perhaps more importantly, where there is a lower disease prevalence in the populations being tested. This is due to the clinical demand for a higher performance in these situations to avoid too many false positives, although the researchers did analyse performance with various prevalences of COVID-19 in the sample. These results, alongside more detail on the model and its validation, are available in the team’s pre-print, found on medrxiv.
To find out more about the technical aspects of the model contact Dr. Andrew Soltan at andrew.soltan@medsci.ox.ac.uk. The code underlying the model will be also released in due course, and is available now upon request.
Leave a comment