Introducing the COVID-19 Multi-model Comparison Collaboration

The challenge

As the COVID-19 pandemic is evolving, a growing number of countries are making use of information derived from epidemiological mathematical models in policy and public communication. The type of models employed by the countries vary: they range from planning tools for capacity and infrastructure preparation—including test capacity, intensive care units, hospital beds, ventilators, etc.—and impact models that run different mitigation policies and scenarios in different settings. The models represent a diverse range of model structures, assumptions, and fitting procedures; these may result in different projections and conflicting results about, for instance, the impact of the outbreak and mitigating interventions, causing uncertainty for decisionmakers with limited technical background in modelling. At the same time, as we also witness decisionmakers questioning the value of models (perceiving them as “crystal balls”), models need to be scrutinized to improve them and to ascertain whether they can tell decisionmakers what they want to know, particularly in resource-poor settings.

The response

In response to increased requests from countries for guidance on the selection and validation of the models, as well as the interpretation of the model results, the World Health Organization(WHO), the World Bank, the Bill & Melinda Gates Foundation, the international Decision Support Initiative (iDSI) hosted by the Center for Global Development, the Royal Thai Government, and other partners including DFID and the following modelling teams: University of Basel, Institute for Health Metrics and Evaluation (IHME), Imperial College London, Institute for Disease Modelling, London School of Hygiene and Tropical Medicine (LSHTM) and the University of Oxford Modelling Consortium (consisting of several modelling partners in different countries) have agreed to collaborate under the auspices of the COVID-19 Multi-model Comparison Collaboration (CMCC). The purpose of the CMCC is to help enhance the use of mathematical models during the COVID-19 outbreak and help policymakers interpret models, foster collaboration between modelers, and assess the fitness-for-purpose of what COVID-19 models produce in terms of the policy questions that need to be resolved related to disease control.  

The overall goal of this effort is to provide country governments, particularly low- and middle-income countries (LMICs), and other model users with an overview of aims, capabilities, and limits of the the main COVID-19 models currently being used, as well as how their projections differ and what the models’ key assumptions and drivers are. In other words, this exercise is to help policymakers and countries to better interpret the estimates from these tools for planning and strategic decisions. It is not to rank, appraise, or approve the models, or to state whether a model is “right or wrong.” The collaboration aims to improve the relevance of COVID-19 models by having information exchange and encouraging discussions between the different groups participating.

The approach

The work focuses on answering four questions:

• What are the crucial policy decisions which can benefit from epidemic modeling evidence for decision support, and how should the analyses be framed?
• What are the differences in the epidemic trajectory and resource requirement estimates  by different models (under the same assumed conditions) for the current COVID-19 pandemic in given contexts?
• What is the purpose, objectives, characteristics, data, parameters,  and key assumptions driving potential differences in predictions of each model, and how do parameters affect model outcomes?
• How can we better guide the selection and use of models, and make the results’ presentation more meaningful to policymakers?

In Phase 1, the work will focus on an initial set of COVID-19 models: Imperial College London, London School of Hygiene and Tropical Medicine, Oxford University COVID-19 modelling consortium, IHME, Institute for Disease Modeling, and the Unversity of Basel.

At the end of Phase 1, the Consortium will produce:

1. a report comparing the initial set of models in terms of objectives, structure, and key driving assumptions, in user friendly/lay language for policymakers;
2. a report containing recommendations on relevant policy scenarios and questions, and future data input available to modelers, adapted to LMICs; and
3. guidelines for modellers on communicating model results, including on uncertainty, that is appropriate for ongoing research and in the context of an emergency; and for policymakers and research funders for enhancing the value and impact of the research to policy implementation.

In Phase 2, additional modellers who want to be part of the COVID-19 model collaboration will be invited to join. Also, participating modelling groups will be invited to run two or three standardised COVID-19 policy scenarios for a typical LMIC context and compare their forecast estimates. The results will be presented to the policy group members, who will be able to deliberate with modelers and the technical group. This deliberation process will allow policymakers to better understand the models and their results in order to make recommendations on the selection and validation of the model, interpretation of the model results, and improvement of presentation and reporting of results.The effort will result in a report summarizing the quantitative comparative analyses and providing information on the robustness of the individual models, as well as their usefulness for policy.

The work will be overseen by four groups:

A technical group, chaired by Dr. Marc Brisson (Laval University, Canada) and comprising international representatives in infectious disease modelling independent of the COVID-19 models included in the exercise, is tasked with comparing the COVID-19 models based on a standardized template and following best practice principles of multi-model comparison (such as den Boon et al. 2019 and Delva et al. 2012).

The technical group works closely with a COVID-19 modellers group, chaired by Marelize Gorgens (the World Bank) and comprising representatives from the owners of COVID-19 models included in the collaboration.

In parallel, a policy group, chaired by Dr. Suwit Wibulpolprasert (Ministry of Public Health, Thailand) and comprising policymakers from Africa, Asia and Latin America, is meant to identify the highest priority questions for their decision needs and discuss how models can help address these.

A partners group consists of numerous other funders and technical global health agencies who are supporting modeling syntheses or wish to be informed by epidemic modeling for COVID-19 control intelligence. WHO, the World Bank, iDSI and the Gates Foundation are the executive team managing the partners’ group and steering the overall effort.

You can access the CMCC concept note here: www.decidehealth.world/CMCC.

We will aim to publish updates on www.decidehealth.world/CMCC and our respective institutional websites, including CGD, HITAP, WHO, and the World Bank, and we welcome your feedback! For more information or questions, please contact Raymond Hutabessy, Nejma Cheikh, Kalipso Chalkidou, or Yot Teerawattananon.  

*partners listed in alphabetical order