Read the latest Issue
Bringing AI out of the black box
Love it or hate it, artificial intelligence* (AI) is becoming embedded into all aspects of life. From self-driving cars to analysing legal cases, deciding which social media posts you see or making healthcare more personalised, the applications of AI are endless.
AI has been used for many years in research, but as its practical applications grow, researchers have a responsibility to ensure that their methods are transparent and can be scrutinised.
“In the fields of genomics and bioinformatics, publishing the code and the data associated with a study is commonplace, but other life science disciplines lag behind,” says Alvis Brazma, Functional Genomics Senior Team Leader at EMBL’s European Bioinformatics Institute (EMBL-EBI). “This can be problematic, especially in the case of research focusing on human health and disease. Publishing the details and code of the methods used is important not just for scientific value, but also to ensure the method can be tested and any problems identified.”
AI-powered clinical research
The past few years have seen a shift in healthcare research towards AI and deep learning methods. The applications include personalised medicine, identifying drug targets, accelerating clinical testing, and making predictions about the risk of developing a certain disease, or about its severity or outcome.
“Cancer is one of the many diseases that artificial intelligence and deep learning can shed light on,” says Moritz Gerstung, Research Group Leader at EMBL-EBI. “AI can help process enormous amounts of data faster than ever before, which can refine diagnosis, prognosis, and treatment. There’s significant potential for such applications, but the algorithms and underlying data have to be as transparent as possible. This is necessary to fully scrutinise the performance of AI algorithms and their implications, in collaboration with clinical researchers and also patient advocacy groups.”
AI also has the potential to improve clinical trial design, reducing the time and cost involved in research and development. In some cases, AI is already being used in hospitals, to complement the work of healthcare professionals such as radiologists or histopathologists.
Code, results, and transparency
While many AI-powered studies produce fascinating or encouraging results, a lack of detail regarding the methods and algorithms used is a common problem. Not being able to test the algorithm undermines the scientific value of the research. Scientific progress relies on the ability of independent researchers to scrutinise and reproduce the results of a study, and to build on those results. Without access to the code or the data, this becomes impossible. More worryingly, this kind of opacity could lead to unfounded and potentially harmful clinical trials.
When the method or the code underlying a study are not well documented, the study itself is difficult to validate. Textual descriptions of AI or deep learning models are not enough. By providing open access to the actual computer code used to train a model and arrive at its final set of parameters, researchers are enabling others to reuse the model. They’re allowing peers to test it – sometimes even to break it – to show its worth.
Testing plays a huge part in the development of any new technology, from mobile phones to cars, so why would research compromise on this essential step when it comes to AI?
How to open your code
There are many platforms where researchers can share their code, including GitHub, GitLab, and Bitbucket. In addition, they can use package managers, which are collections of software tools that automate the process of installing and configuring computer programs for a machine’s operating system. Package manager Conda or container and virtualisation systems such as Code Ocean or Docker enable control of the software environment, which is essential for large-scale machine learning applications.
Platforms such as TensorFlow Hub, ModelHub, or ModelDepot allow sharing of deep learning models. Using these resources improves transparency and can speed up model development, validation, and clinical implementation.
Another common challenge for researchers is that data, especially human data, can’t always be shared, due to privacy concerns. The restrictions and protocols in place ensure safe and secure sharing of sensitive data, but they can also be problematic for research reproducibility. Despite these challenges, sharing raw data has become more common in the biomedical literature, growing from 1% in the early 2000s to 20% in 2018.
When data can’t be shared, one solution is for authors to create small artificial examples or use public datasets to show how new data should be processed to train the model and to generate predictions.
“The data and the code behind a publication are almost as important as the results, so sharing them is crucial,” says Jo McEntyre, Associate Director of EMBL-EBI Services. “They put science in context, demonstrate rigour, and allow others to build on the hard work the authors have already done.”
In the majority of cases, there are ways of improving the transparency of AI models. While this does require additional effort – and often creative thinking on the part of the authors – it’s crucial if they want their method to have impact beyond the publication. AI-powered research needs to be reproducible if it’s going to be truly useful in healthcare or in other aspects of life.
*This article uses AI as an umbrella term for a suite of technologies, including algorithms, deep learning, and neural networks.
This post was originally published on EMBL-EBI News.
