Another Open Letter to IHME

The original Open Letter to IHME (below, dated May 13, 2020), was amended on May 16, 2020 to include the summarizing plot above of the accuracy of the IHME model's predictions.

May 13, 2020

Open Letter to IHME

Dr. Christopher J.L. Murray

Institute Director and

Professor of Global Health

University of Washington

Emailed to: covid19@healthdata.org & media@healthdata.org

Dear Dr. Murray:

Following on our earlier letter (dated May 7, 2020 and also enclosed below) and subsequent releases of the new IHME models on May 8th and on May 10th, we critically examined the precision of the IHME model predictions as well as the input data used in the model. Based on this, we would like to share our serious concerns regarding the validity of the predictions of the IHME model and their precision.

We request that such shortcomings be clearly communicated to the public. In addition to elaborating on potential drivers for a dramatic change in the model's predictions on national TV, we feel it is absolutely essential that a scientifically sound model also be presented with the uncertainties associated with its predictions (which seem to be rather large in this case including, unfortunately, biases as detailed below) as well as the past performance of the model throughout its many updates and releases especially given that its precision for making predictions appears to be rather poor, as demonstrated below.

We strongly believe that no such "model" nor its "predictions" could ever be allowed to be published in any peer-reviewed scientific journal. But, general public may not be aware of this and may be putting too much trust into these predictions. We feel that this needs to be clearly communicated to the general public, also in writing. Especially given such bold predictions of so many additional deaths in the USA when the model appears to suffer from huge errors/uncertainties and biases and, therefore, should simply not be used for making such far-reaching predictions.

Our specific concerns are listed below. We have examined the data and the model predictions for the State of New York only.

In our earlier letter (point #2 in the May 7th letter), we expressed our concern that the actual data need to be used as close as possible - and potentially all the way - to the model release date. In the May 4th release, the data used lagged by 4 days, but then in the May 8th release, the data lagged by only 3 days, while in the May 10th release the data seem to have been incorporated all the way though the day just before the release, i.e. through May 9th, which is a great improvement. We thank you for achieving this and feel strongly that it is essential to continue doing so in future releases as well.
There continues to be a rather large discrepancy between the number of deaths used in the IHME model and the same data as presented by Governor Cuomo in his daily briefings. For the period of April 25th through April 30th, the differences between the two sources in the daily data ranged from 45% to as high as 71% as we pointed out in the May 7th letter. Furthermore, for the same period, the total number of deaths for those 6 consecutive days differs by as much as 52%, the IHME always being higher!
In passing, we also note that the actual IHME deaths data for the period of April 25th through April 30th decreased between the two model releases, i.e. between May 4th and May 10th. Specifically, the total number of deaths for this 6 consecutive day period changed from 2,980 down to 2,873, a decrease of nearly 4%.
For the period of May 1st through May 9th, the differences in the daily deaths data between the IHME model and the Gov. Cuomo's data range from -17% to 192%! And the totals for this 9 consecutive day period differ by 32% with IHME being higher!
To look at the precision of the IHME model predictions, we next examine its predictions with respect to the actual (i.e., materialized) daily deaths data used by the IHME model itself as well as to the deaths data as reported by Gov. Cuomo. The comparisons are presented in the table as well as in the plot below. We see huge differences with respect to the Gov. Cuomo data as well as with respect to the date used by IHME itself in their next release!
Interestingly enough, for some reason, the shape of the model predictions seems to follow the Gov. Cuomo's data better (blue on the plot) than the data used by the IHME model itself (red).
Typically, the precision of a predictive model should degrade further into the future the model tried to "look". This fact is of course represented in the uncertainty bands associated with the IHME model. E.g. the upper error grows from ~18% for May 1st to ~47% for May 9th. However, oddly enough, the actual precision of the model prediction does not show such behavior and in fact appears to get more precise the further into the future the IHME model attempts to predicts! This perhaps is the biggest indicator that something is terribly wrong with the IHME model and that its predictions are highly unreliable!
Furthermore, the difference between the predictions of the model and the actual, materialized data clearly demonstrate that not only there appear to be huge uncontrolled uncertainties in the model, but that it also has a significant and large bias towards overestimating the number of deaths. This clearly is a significant deficiency of the IHME model.
Lastly, anyone can simply observe a rather odd jump taking place when going from May 9th actual IHME data to the May 10-12th predictions of the IHME model. Being on the strong downward trend, the actual number of deaths on May 9th is 230 (as used by the model) but then the IHME model would lead us to believe that the number of deaths for the very next day, May 10th would suddenly jump up to 350, for May 11th to 342, and for May 12th to 292! These are 52%, 48%, and 27% increases, respectively, as compared to the May 9th actual number of deaths when, at the same time, the deaths are on a strongly expressed downward trend. Yet another clear indication that something is terribly wrong with the IHME model.

We will close by simply repeating verbatim what we said at the end of our earlier, May 7th letter:

We would think that the shortcomings outlined above should have a huge impact on both the predictions and the accuracy of the IHME model not only for NYS, but also for the entire USA because NYS happens to be such a big fraction of the USA data as far as the Covid-19 epidemic measurements go.
We hope these discrepancies can be corrected promptly and prior to next release/update of the IHME model predictions.
If, on the other hand, these happen to be systemic faults or deficiencies of the IHME model, then they must clearly be highlighted as such to the general public.

Sincerely,

Levan Babukhadia, PhD

201-679-1816

levan@babukhadia.com

www.babukhadia.com

May 7, 2020

Open Letter to IHME

Dr. Christopher J.L. Murray Institute Director and Professor of Global Health University of Washington

Emailed to: covid19@healthdata.org & media@healthdata.org

Dear Dr. Murray:

On May 4, 2020, IHME released the much-anticipated update to its Covid-19 model for the United States. Its estimates for the projected total number of deaths in the USA jumped up from ~65 thousand to ~130 thousand, i.e. nearly doubled.

Given the especially high interest and sensitivity of this matter, we feel it is our duty to point out an apparent inconsistency in the IHME model/data and inquire about the magnitude of the impact of these inconsistencies on your final projections.

1. Why are the numbers of actual deaths used in IHME model for the State of New York (NYS) so different from the numbers reported by Gov. Cuomo in his daily briefings [1]? The IHME actual numbers of deaths, e.g. from April 25th through April 30th, highlighted in the below simple comparison table, are 45-57% higher than the numbers reported by the Governor. Why is that? These obviously are rather huge differences!

2. The IHME released new projections on May 4th. Why are then the numbers of actual deaths not included for the three preceding days, i.e. for May 1st through May 3rd in the released model? Instead, the model uses its projected values for those days when the actual death numbers were already available!

3. Even if there were to be a three to four day lag to fully incorporating any new data into the IHME model ( which would in itself speak rather unfavorably about adaptability of the IHME model and would be seen as its big deficiency ), why would the IHME not highlight or comment on the huge discrepancy between the projected values and the actuals for those three days, i.e. May 1st through May 3rd? The IHME “projections” for those day are whopping 65-104% higher than the actual numbers of deaths reported by Gov. Cuomo!

We would think that the shortcomings outlined above should have a huge impact on both the predictions and the accuracy of the IHME model not only for NYS, but also for the entire USA because NYS happens to be such a big fraction of the USA data as far as the Covid-19 epidemic measurements go.

We have not examined other states. We just happen to notice these glaring discrepancies regarding the IHME’s treatment of the NYS data because, living nearby, we naturally watch very closely all developments in NYS.

We hope these discrepancies can be corrected promptly and prior to next release/update of the IHME model predictions.

If, on the other hand, these happen to be systemic faults or deficiencies of the IHME model, then they must clearly be highlighted as such to the general public.