Wolfgang Huber discusses why research assessment needs reform and how this could make science more diverse, more inclusive, and better
Research assessment across EMBL is changing. EMBL’s Research Assessment Reform Working Group is leading changes to better measure the quality and impact of our research, valuing diverse outputs and activities from across the organisation.
To find out more, we spoke to Wolfgang Huber, Group Leader and Senior Scientist. Huber co-heads the Working Group together with Jo McEntyre, Associate Director of EMBL-EBI Services. Huber is a physicist by training and now works on statistical method development in computational biology. Through this work, which heavily relies on software, Huber has developed a passion for changing the way people think about research outputs and their impact.
I started out as a scientist in theoretical physics and computing. During my postdoc, I got excited by genomics and bioinformatics and soon realised that the really impactful work in these fields did not only consist of papers, but was accompanied by software and data. I found outstanding colleagues and people to collaborate with in the R and Bioconductor communities. This work, and perhaps even more importantly, this approach, was the reason that I got the opportunity to start a research group at EMBL-EBI.
At the same time, the so-called ‘reproducibility crisis’ was unfolding, with scientists claiming spectacular discoveries in papers in high-profile journals, based on data analyses nobody could reproduce. This convinced me that fully transparent analysis paths, from raw data to published figures, were the future of scientific publishing. These needed to be based on easily available, and wherever possible, open-source software.
Making data analysis fully transparent, or providing re-usable, useful methods to other scientists as software packages is a lot of additional work that was mostly not credited until now. So the incentives in the system were, to put it bluntly, against doing the right thing. This needs to change.
Deciding which researchers to recruit and which projects to pursue is a core responsibility of any research organisation. It is the key to its success or failure as an institution. Nevertheless, there has been a trend to outsource such research assessment: journal editors decide which papers get which level of visibility and prestige, and subsequently which scientists advance in their careers and which type of science gets funding.
It is time for researchers, and for their institutions, to take control of their core business. For EMBL, being an early mover and innovator is simply a competitive advantage. It is also part of EMBL’s mission to be a leader in such matters, to develop and figure out approaches to doing excellent research and training that then hopefully inspire similar change elsewhere.
I’m amazed how rapidly research assessment is changing. Funding bodies are a big driver for change. The Wellcome Trust, for example, was an early adopter of valuing different types of research outputs based on their impact and contribution.
It’s not only top-down; the momentum is also building bottom-up. Especially, early-career scientists are rethinking traditional systems of research assessment that set perverse incentives or are based on extensive gatekeeping and groupthink.
Goodheart’s law states: “When a measure becomes a target, it ceases to be a good measure”. This is how I think about scientific publishing: a journal’s prestige and maybe even its impact factor may be a valid statistical correlate of the quality of the papers in it. But by making it the target to publish in a certain venue, in a very narrowly defined format, rather than focusing on content and allowing a wide range of publication or distribution formats, we have put the cart before the horse. It’s important to go back to basics and focus on what we actually want to achieve.
EMBL’s recently formed Research Assessment Reform Working Group is working to drive changes in EMBL processes. EMBL is a signatory of both DORA and CoARA – two of the main initiatives in this field – and this means we have made a commitment to change our processes and practices in line with their principles.
We are implementing three major actions. The first is a clear statement, externally and internally, of EMBL’s commitment to the principles of DORA and CoARA. Secondly, we updated the type of research outputs we ask candidates to list when they apply for EMBL positions. In their CVs, candidates are now asked to augment the traditional ‘publication list’ with a text that summarises the impact and importance of their research outputs in a more conceptual, narrative form. Thirdly, we have updated the EMBL unit review procedure and awards documentation to provide guidance to internal interviewers and reviewers.
Reforming research assessment across EMBL is a work in progress, and a long-term commitment. There are no easy fixes; it requires a culture change that will take time, effort, and continuous readjustment. Doing research assessment well requires that you actually have the expertise to assess a scientist’s work and are willing to engage with it.
We also must face the fact that research assessment is not always practical. Some degree of outsourcing and reliance on external indicators will be unavoidable. We need to be creative, flexible, and always exert good judgement. Science and technology can move very quickly, but sometimes, to fully assess the impact of research outputs, we need longer timeframes, vision, and willingness to take a risk.
Making these changes to our assessment processes and acting on them now will be of enormous benefit to EMBL in the years to come. It will enable us to recognise excellence in all of its forms and secure a diverse pool of talent.
There has been a great deal of attention by the public to the ‘reproducibility crisis’ in science. Newspaper readers were confused by big claims that turned out to be wrong, or by dubious results hyped up as breakthroughs. Big pharmaceutical companies were complaining that published results from academia could not be replicated. Such things have the potential to shake the public’s perception of science and their trust in scientists. Funders started to take notice and decided they needed to do something about it.
Those who pay for science – scientists’ salaries, but also their equipment and buildings – have a right to expect something in return. We need stringent and effective ways to separate good work from bad work. I am sympathetic to the underlying idea of ‘publish or perish’, but in the past, some of the concrete implementations by institutions have created perverse incentives, leading to dishonourable behaviour and toxic workplace environments. Many good scientists decided that this was not for them and have been leaving academia.
Better research assessment will result in more sustainable and better science. It will also make science and academia a more attractive workplace. These changes should create more inclusive and diverse working environments, increasing the talent pool, which is a competitive advantage for any institution that is leading the charge. Eventually it will improve the quality and impact science has globally.
I welcome these changes and am excited to see how reforming research assessment will benefit the science that will come out of EMBL.
Find out more about research assessment at EMBL and learn how to become a signatory of DORA.
On 12 May 2023, the 10th anniversary of the San Francisco Declaration on Research Assessment (DORA), EMBO and EMBL hosted a hybrid panel discussion reviewing the initiative’s impact on the way research is assessed. The six panellists discussed issues with current methods of research assessment, as well as solutions and actions for improvement in which panel members have been involved. Read more about the discussion here.
Looking for past print editions of EMBLetc.? Browse our archive, going back 20 years.
EMBLetc. archive
