Just How Accurate are the Models Wisconsin is Relying on to Close the Economy?

Rick Esenberg, Mike Fischer, and Will Flanders

Although it often seems we can think of little else, the most striking thing about the COVID-19 virus may be how little we know. Across the country, governors have issued stay-at-home orders that would have been inconceivable in February, ordering citizens to “shelter” in their homes, family and friends to remain apart and businesses to close. While not all social and economic disruption can be attributed to the orders, it is fair to say that we have never embarked on a costlier or more risky endeavor. A booming economy has been reduced to one rivaling the depths of the Great Depression. A “full employment” economy has seen over thirty million unemployment claims in two months. Concerns over deferred medical care and mental health impacts are rising. Unprecedented restrictions on fundamental freedoms remain in place.

Is this necessary? Thanks to the current lawsuit over the extension of Governor Evers’ “Safer at Home” order, the curtain has been raised on what went into the thinking for both the original order and the extension. Affidavits from Julie Willems Van Dijk and Dr. Ryan Westergaard provide some insight into the data and modeling that the Evers administration relied upon in choosing to impose restrictions. In a letter to Senator Van Wanggaard dated April 24, 2020, Governor Evers said that a report co-authored by Johns Hopkins University and the Department of Health Services’ Office of Health Informatics was “most compelling” and it was emphasized in the state’s findings before the Supreme Court.

The novel coronavirus is just that — novel. Because it is something that we have not seen before, modeling its impact is theoretically difficult and practically impossible. For that reason, it is imperative, given the damage that lockdown orders cause, that policymakers remain nimble. Models are only as good as the assumptions and data upon which they are based and when we have little or no experience with what we are modeling, a continual re-examination of how unfolding reality relates to what was projected is critical.

As the crisis has gone on, it appears that these models have not performed well. Yet the administration continues to rely on a model that seems to be in need of serious reexamination.

Why the lockdown?

An early projection of the pandemic’s impact in Wisconsin was based on a simple assumption that the “doubling” rates that were present in mid-March would continue. This would lead to 22,000 infections by April 8. The Department of Health Services then apparently applied different assumptions as to the case fatality rate to suggest that there could be 300 to 1,440 deaths by April 8. This is a very unsophisticated analysis, although in fairness, better data and modeling may not have been available at the time. During a period when testing was ramping up, it is difficult, if not impossible, to distinguish the spread of the virus from increased count attributable to testing. Nor is it clear that even exponential growth will remain constant as awareness of the virus and its spread grows.

The administration did attempt to get better information. A model developed for the Department of Health Services by Johns Hopkins projected that, without a stay-at-home order, there would be 94,200 hospitalizations, 22,600 ICU stays and 15,820 patients requiring a ventilator by May 1. This number would overwhelm health care providers. They are alarming. But there is reason to question the model.

The Hopkins model appears to have been wildly off

As noted above, Hopkins projected some truly frightening numbers if nothing was done. We cannot learn much about comparing our actual experience to the projections based upon the absence of restrictions. But Hopkins also projected a sobering outcome if the Governor’s “Safer at Home” order was implemented and kept in place. It projected that, if “Safer at Home” remained in place through May 26, the state would still have experienced the following by May 1: 11,900 hospitalization, 4,500 ICU and 3,150 patients on a ventilator. None of this came close to happening. Our actual experience as of April 30 was much different: There were only 1,512 hospitalizations — only 12.7% of what the Hopkins model projected. We do not have cumulative numbers, but there were only 330 ventilated patients on that date and 137 ICU stays, suggesting that the model was way off on these as well.

The Hopkins model also seems to have been well off on projected deaths in place. With the Governor’s “Safer at Home” order in place through May 26, Hopkins projected 2,100 deaths by May 1. The actual number as of April 30 was 316.

Of course, it is possible that reality did not match the model because social distancing mandates were much more successful than the model assumed. But given the huge discrepancy between the model’s projection and the real world, this seems unlikely. It is more plausible that the model’s projections of what would happen if “nothing” was done were dramatically overstated.

There is no clear evidence that “Safer at Home” worked

Our suggestion that the Hopkins model was too pessimistic is buttressed by the absence of proof that the particular set of restrictions imposed by the “Safer at Home” model have made a difference.

We do not suggest that social distancing has no impact. The hypothesis that reduced human contact will slow the spread of the disease is solid. But the general validity of social distancing as a technique does not mean that the particular mandate strikes the right balance between restriction and allowing community life to continue. Different restrictive measures result in different reductions in human conduct. Some may be of little marginal value. It is possible, moreover, that Wisconsinites may voluntarily adopt certain social distancing measures such that mandating them is superfluous. What evidence do we have of the impact of the “Safer at Home” order?

Not as much as we think. Governor Evers has claimed that any deviation from the Hopkins model is a result of “Safer at Home.” But that claim works only if the models accurately reflect what would have happened in the absence of the order. Put differently, the Governor is claiming victory by assuming that he has won. As we have seen, that assumption simply cannot be made. We do not know what would have happened if Wisconsin had imposed lighter restrictions and allowed Wisconsinites to decide how best to conform their behavior to the threats imposed by the virus. The performance of the Hopkins model in April suggests that it was much, much, too pessimistic. It cannot be a measure of “Safer at Home’s” success.

In documents filed with the Court, the Evers administration claims that infection rates have been lower in states that implemented similar measures to Wisconsin. They include a chart which suggests a decline in the rate of increase in cases in such states relative to ones that did not have a “Safer at Home” order in place. This caught our attention as it is not consistent with what others have found. A statistical analysis of states with and without such orders found no relationship between COVID-19 deaths and the adoption of a lockdown order. Controlling for a number of demographic factors, cases were significantly lower in states that did not go into lockdown. This finding persisted even after the researcher controlled for population, population density, median income, median age, diversity (measured as the percentage of minorities in a population).​ The only variable that was positively correlated with COVID deaths was population density.

No single study is dispositive and it could be that Wisconsin’s experience was different. There are some obvious problems with the chart produced by the state. In comparing the experience between states, most researchers adjust for where a state is in spread of the virus, typically looking at — and comparing — experience from some benchmark like the confirmation of a certain number of cases or the experience of a given number of deaths. The state’s chart does not do that.

But there is a more fundamental — and puzzling — problem. Our own attempts to replicate the finding yielded similar results to those mentioned above. States without “Safer at Home” style orders actually had lower average infection rates in the time frame since implementation.

A more direct indicator of the success of the “Safer at Home” order would be a discernible change in the progress of the virus at some appropriate point after implementation of the order. The chart below shows the daily count of new cases in Wisconsin since March 20th. The date of implementation of “Safer at Home,” March 25th, is marked by the red line. If “Safer at Home” had a material impact, one might expect to see a significant change in the rate of new cases beginning at an appropriate time after implementation of the order. But, rather than a decline in the rate of new cases, we have instead seen a rather steady rate over time, i.e., it does not appear that the “Safer at Home” order has made a discernible difference in the rate of new cases since it was implemented. While we can’t know what would have happened in the absence of the order, this is consistent with the national evidence that such orders haven’t had a huge impact.​

Again, this does not mean that a lockdown order does not result in some additional social distancing that might have an impact on the spread of the disease. It does mean that the impact of such interventions cannot be assumed. And this means that their withdrawal may not be as impactful as assumed.

“Safer at Home” alone may not dramatically change projected deaths

Let’s assume, for a moment, that the Hopkins model is correct. The materials submitted by the state to the court are not consistent. An affidavit submitted to the Court claims that Hopkins projected 4,500 deaths by October 1, if nothing was done to control the outbreak. But according to the same model, deaths would still be as high as 4,000–4,100 in that period with “Safer at Home”. Only if “test + isolate” measures are adopted, could deaths be reduced to 1,600. A table in an exhibit to the affidavit presents very different numbers, projecting 20,000–25,900 cumulative deaths depending on when “Safer at Home” ends. Only if the state adopts “test+isolate” are the projected deaths reduced to 8,900–10,000.

There is reason to doubt these as the Hopkins model appears to have substantially over-projected deaths for April under “Safer at Home” (2,100 projected vs. 300 actual). Nor are we able to explain the discrepancy between the affidavit and its exhibit. Although it is critical, we can put that aside for a moment. DHS appears to believe that “test+isolate” is key. According to the Hopkins model, lifting “Safer at Home” restrictions will cause a secondary peak (distinguishable from the “second wave” that it also projects) that will exceed hospital capacity. This secondary peak can apparently not be avoided by restrictions; only delayed. It can be avoided, according to the model, only by “test + isolate” which it defines as “intensive testing and isolation similar to that implemented in South Korea.”

Just what that means is undefined. The administration’s “Badger Bounce Back” plan calls for 85,000 tests in a week and the hiring of 1,000 contact tracers. This would be slightly more than 12,000 tests each day. Is this enough for South Korean-style “intensive testing and isolation?” A recommendation from researchers at Harvard called for daily tests at a rate of 152 per 100,000 per day. This would amount to a bit under 9,000 tests each day. If Harvard is right, then the administration may be insisting on an unnecessarily high level of testing.

But whether the goal is 9,000 or 12,000 tests, how quickly can we get there? Wisconsin exceeded 3,000 tests per day on April 29, April 30, and May 1. Assuming that level of testing could be sustained given our current capacity, the administration seems to be claiming that testing must triple or quadruple before the “Safer at Home” order can be lifted.

In addition, the state wants 1,000 contact tracers. Harvard’s recommendation would call for a somewhat higher number. The state is currently well short of that goal but, with unemployment approaching 30%, one would not think that this is an unattainable number.

Conclusion

It is customary for public officials to say that “the science” will dictate when to reopen. But the stark reality is that we simply do not know what would happen in the absence of stringent social distancing mandates and cannot know what would happen if we lifted them. There is reason to believe that the modeling that the Evers administration has relied upon to date is too pessimistic. “Erring on the side of safety” is understandable, but itself comes at a substantial human cost, measured in lost jobs, businesses destroyed, deferred medical care, devastated family finances, homes potentially lost, depression, anxiety, educational setbacks, substance abuse and deaths of despair.

It is, quite frankly, a dodge for public officials to say that “the science” leaves them with no choice. COVID-19 is a serious disease requiring a serious response. But the costs of our response cannot be overlooked. Public officials face difficult trade-offs. Making those decisions requires a clear-eyed assessment of what we know — and what we do not know.