Empirical Testing of Epidemiological Models and Interventions

A study of COVID Cases in New Jersey during spring 2020 demonstrating how theoretical models can (and should) be tested empirically reveals that interventions probably made things worse, not better.

It is a curious thing that countries all over the western world acted in lockstep in March 2020 by media messaging and intervention policy in the face of COVID-19.

For those that have been diligently following matters since the start, you will no doubt be familiar with the seminal paper by Prof Ferguson et al¹ published on 16-Mar-20 which became the de facto policy playbook that the western world followed. In typical epidemiological fashion, it is a SIR model, nicely described and distinguished by my former HART colleague Gerry Quinn et al².

This established the dogma of “social distancing” and various other non-pharmaceutical interventions but did not advocate for full lockdowns or wearing of masks for healthy people under the age of 70 - that was entirely the work of other advisers or the politicians.

However, one thing that has never been done, let alone in real-time, is a proper empirical validation of the modelling (not by the modellers or anyone else on the policy side at any rate).

Attempts have been made to justify the model by producing spurious claims of benefit by comparing empirical outcomes to the a priori assumptions of the model without considering the fact that those assumptions were wrong (all model assumptions are wrong until empirically validated - that’s why they’re called assumptions, not facts).

Many alternative models have instead relied on the Gompertz function³. The main significance of using this function (and why I prefer it) is that it is a continuous function with just two parameters, an initial growth and a decay rate. It means that any point that if it fits a set of data well, there is very little exogenous influence on the data. And, most importantly, it can be easily tested.

In this study, I’m going to do exactly that with some really good quality data from New Jersey. The data is true cases data, based on hospitalizations but by date of infection rather than admission.

In addition, I’m going to use the “Social Distancing” Index published by IHME⁴ which is based on mobility. This acts as a good proxy for the timing and effectiveness of various policies designed to limit human interaction and therefore viral spread (in theory). I’m also going to use IHME’s mask compliance index so we can compare the potential impact of these two main official policies on COVID cases.

Figure 1

We can start as early as 11-Mar-20 when the panic button was first pressed in the western world. It is evident in Figure 1 that COVID had already been present in New Jersey since the start of the year but had not made a remarkable impact.

In the week ending 04-Mar-20, it has re-emerged and a week later we can already see evidence of social distancing and mask-wearing. Mobility is about 10% lower than usual and mask compliance is around 4%.

Using the Gompertz function to calibrate to the reported cases at this time, we can fit the 3 weeks of data between 26-Feb and 11-Mar perfectly. This allows us to forecast the number of cases (see #1 in Figure 1). We expect around 10,800 cases and for the epidemic to be over naturally by the end of May.

However, when we empirically test our forecast one and two weeks later, we see that the reported case data is higher than expected (Figure 2).

Figure 2

What is more of a concern is that this unexpected increase in cases is associated with a rapid increase in social distancing and mask compliance, the opposite of the purported outcome. Social distancing is almost at 60% reduction in mobility and mask compliance has risen to 35%.

Regardless, public health authorities keep pushing the narrative because they don’t know that the interventions are having the opposite effect that they are supposed to because they aren’t even testing the model they are relying on. This is dogma, not science.

If we recalibrate the model to take the new reported data into account, we can produce another Gompertz function with a good fit to the calibrated data (11-Mar to 25-Mar) but we inevitably lose some fit to the first 3 weeks since there can only be one solution for a continuous set of data and this data is not continuous (see #2 in Figure 2). It has become worse, much worse. We are forced to revise our forecast from 10,800 cases to almost 29,000 and an end date of early June instead of May.

However, the following week, we observe reported cases exceeding the newly calibrated model yet again (Figure 3).

Figure 3

We go through the same exercise as before and produce forecast #3 (Figure 3) based on calibrating to the reported data between 11-Mar and 01-Apr.

The following week, reported cases continue to outperform the forecast (Figure 4), although the excess is much smaller now and cases have finally begun to decelerate even if they haven’t turned downwards as the model forecasted.

Figure 4

What is interesting here is that the rate of social distancing slowed down substantially the week before.

Re-adjusting our forecast to take the new data into consideration (see #4, Figure 4), we see very little difference between this and our prior forecast which gives confidence. Our expectation now is around 41,000 cases and an end of the epidemic by mid-June.

Nevertheless, even though reported cases finally begin to fall steadily, they still fall at a slower rate than forecast (Figure 5).

Figure 5

Note that as social distancing declines, mask compliance continues to rise giving the indication that this policy is not reducing cases. It hits 80% at the high point. If it was a beneficial intervention, it ought to be apparent in the data at that point. It is not.

In fact, ceteris paribus, the evidence suggests these interventions are making the situation worse, not better. Finally, there were over 81,000 cases against the original forecast of less than 11,000 and the epidemic ended closer to the end of June than May.

We should, however, not accept that these results are conclusive. There can be other confounding variables. But, given that the interventions do not demonstrate any apparent benefit regardless of any confounding, it is remiss of the public health authorities to have persisted with them.

One significant confounder could be heterogeneity of the State. The three to four distinct waves of cases may be due to the virus spreading geographically across cities and counties or different demographics.

The data is stratified by age and county so in the next study, I will examine this.

1. https://www.imperial.ac.uk/media/imperial-college/medicine/sph/ide/gida-fellowships/Imperial-College-COVID19-NPI-modelling-16-03-2020.pdf

2. https://osf.io/s9z2p/

3. https://en.wikipedia.org/wiki/Gompertz_function.

4. https://covid19.healthdata.org/