Aletho News

Public Health Scotland and the misinterpretation of data

Health Advisory & Recovery Team | March 4, 2022

“Oh what a tangled web we weave, when first we practice to deceive” – Sir Walter Scott

Throughout the last two years Public Health Scotland (PHS) has punched above its weight by providing reliable data that has quantified the impact of the Scottish government’s COVID-19 response on the health of the Scottish population. In particular, it has documented the unprecedented excess death that occurred in summer and autumn 2021, prompting the establishment of an official enquiry as to the cause, and uncovered a spike in September 2021 in the number of stillbirths in Scotland that is currently under investigation.

However, in its report of 14^th February 2022, PHS has declared that it will no longer publish data on COVID-19 outcomes (cases, hospitalisations and deaths) classified by vaccination status, a hitherto valuable component of the COVID-19 vaccination surveillance strategy. The reason given for making this change is that ‘PHS is aware of inappropriate use and misinterpretation of the data when taken in isolation without fully understanding the limitations’.

It is certainly true that claims have been made about the deleterious effects of COVID-19 vaccines that go well beyond what can be supported by the data published by PHS. In this case critical appraisal of these unsubstantiated claims, rather than the blanket withdrawal of valuable information, would seem the better antidote to the spread of misinformation.

However, it is important to note that implicit in the decision made by PHS is that the information they provide is above reproach, both in terms of inappropriate use and misrepresentation of the data to which they alone are privy. To investigate whether PHS analysis is indeed above reproach, we can look in a little detail at the way in which they have presented the information on COVID-19 outcomes by vaccination status in their last report of February 2022. We will concentrate on the analysis of death with COVID-19 by vaccination status, unvaccinated or booster, found in Table 15, using the data for week 29 January – 04 February 2022. The relevant data from that table is reproduced below:

A superficial inspection of this table would suggest to the casual reader that the death rate with COVID-19 in those who have received a booster is far lower than that suffered by those who are unvaccinated when the difference in age distributions of the booster and unvaccinated populations are taken into account. Indeed, PHS draw the conclusion that ‘the death rate in individuals that received a booster or 3rd dose of a COVID-19 vaccine was between 4.6 and 9.5 times lower than individuals who are unvaccinated or have only received one or two doses of a COVID-19 vaccine’. Let us look in detail at how the data were treated to arrive at this conclusion, and ask whether this very strong affirmation of the benefits of the booster can be substantiated.

We first look at the way in which the two populations that we are comparing, unvaccinated and booster, are defined. For this we turn to Appendix 6 of the report. Here we learn that the unvaccinated population is not, as we might have assumed, those that have never been vaccinated. Instead, it also includes all those individuals who have received a first vaccine, but for whom the time since vaccination is less than 22 days. Thus, if any deaths occur within the 21 days post first vaccine, these deaths will be attributed to the unvaccinated category. This misattribution may have significant consequences because deaths from adverse vaccination reactions principally occur shortly after vaccination. This idiosyncratic classification of the unvaccinated artificially, and misleadingly, inflates the death rate in the unvaccinated population. Would it not have been better to classify the unvaccinated as those never receiving a vaccine, to preclude the introduction of such bias against the unvaccinated into the analysis?

Turning to the boosted population we find that this is not defined as the number of individuals who have received a booster, but rather the number that have received a booster at least 14 days prior to the reporting period. Therefore, if deaths of boosted individuals occur within the first 14 days of this vaccination, they will not be counted as booster deaths, but as a 2-dose death. The mortality rates given are also dependent on the size of the vaccinated population. If the addition of boostered individuals is a continuous process then, depending on accounting, the last two week cohort added to the boostered population may effectively be excluded from contributing to deaths, while the unvaccinated population during the same time period will not. PHS’s redefinition of the booster population again serves to artificially and misleadingly reduce the reported rate of deaths in the PHS booster population relative to the unvaccinated population. Would it not have been better to classify the booster population simply as those who have received a booster shot, and avoided the inevitable bias in favour of the boosted population that is introduced by the PHS redefinition?

Notwithstanding the biases introduced by PHS’s redefinition of the populations to be compared, we can now concentrate our attention on the methods they have used to correct for the fact that the age distribution of the unvaccinated is likely to be much younger than that of the boosted population. To begin our explanation, it is helpful to use the raw data provided in table 15 for week 29 January – 04 February 2022 to calculate the individual rate of death with COVID-19 per 100,000 per week without making any adjustment for differences in age distribution. This can be compared with the figures PHS calculated from the data to quantify ‘Age Standardised Mortality Rate per 100,000 per week’.

The comparison is illuminating and a little worrying. An unadjusted death rate 2.7 times higher in the booster population than in the unvaccinated population has been converted into an age standardised mortality rate that is now 7.3 times higher in the unvaccinated population than in the booster population. To understand what is going on we have to know both how to calculate an Age Standardised Mortality Rate per 100,000 per week, and to understand what this value actually represents.

The Age Standardised Mortality Rate is a measure of the impact, in terms of mortality, on the whole population rather than a particular age group. Rather than calculating a population Age Standardised Mortality Rate based on the age distribution of the Scottish population, Public Health Scotland used the standard WHO age distribution. In this age distribution there is a much lower representation of older people. The consequence is that a very low weight is given to deaths in older age groups and a disproportionately high weighting to deaths in young age groups. In fact, the weighting of a young death can be 10 times higher than for an old death. Through this unjustified weighting a raw mortality rate which was 2.7 times greater in the vaccinated is turned into an age standardised mortality rate which is 7.3 times greater in the unvaccinated.

The age standardised mortality does not relate to individual risk – we may have much higher risk in old age groups individually, but this translates into a very small effect on overall deaths at a population level because the percentage of old people in the population is very low. The point of calculating age standardised mortality is not to compare risks. It is designed to allow comparison of the relative burden of a disease on a population – what proportion of a population will be lost from that population by a particular disease. Its use to somehow correct for differences in age distributions on risk of death is completely inappropriate.

The important thing to note is that what has been calculated is a measure of population impact of COVID-19 in a hypothetical population; what proportion of the population die in this hypothetical population as a consequence of the disease. It is assuredly not a measure of individual mortality risk from COVID-19. As such it is completely inappropriate and misleading to use it to compare the risk of death with COVID 19 between populations of different vaccination status as has been done by PHS. Therefore, their statement that ‘the death rate in individuals that received a booster or 3rd dose of a COVID-19 vaccine was between 4.6 and 9.5 times lower than individuals who are unvaccinated or have only received one or two doses of a COVID-19 vaccine’ is utterly false and misleading and completely unsupported by the data. The simple and transparent way of comparing individual mortality risk would be to use the data in PHS’s possession to estimate individual risk of death for each age category and population, and compare these values within each age category. Rather than compare the whole population, the risk for each age group by vaccination status would provide useful information. UKHSA do provide this data but PHS never have done. The magnitude of the error in using Age Standardised Mortality Rates as a metric calls into question the competence of PHS to analyse and interpret data that are critical to the formulation of Scottish government health policy which directly impacts the wellbeing of literally millions of people.

The final point to make is that in order to receive a booster, an individual must previously have received both a first and a second dose of vaccine. There is a risk of a bias being introduced whereby only survivors, who are by definition less likely to die, are being measured. Therefore, deaths that occurred after first and second vaccinations should be included with deaths after the booster vaccination itself in order to properly assess the overall COVID-19 death rates in the vaccinated population. In other words, the appropriate comparison to make when assessing the effect of booster doses on COVID-19 mortality is between the unvaccinated population and the vaccinated population, where the latter includes anyone who has received any injection.

In conclusion, by announcing that data on COVID-19 outcomes by vaccination status will no longer be provided due to “misrepresentation and misinterpretation of their analyses”, PHS has drawn attention to their own glaring shortcomings in this area. They have been shown to introduce unwarranted bias into their analyses by manipulation of the definitions of vaccination status, and they have used a wholly inappropriate metric to compare the risk of death with COVID-19 among the vaccinated and unvaccinated in the Scottish population.

Truly they are hoisted by their own petard.

March 4, 2022 - Posted by aletho | Deception, Science and Pseudo-Science, Timeless or most popular | COVID-19 Vaccine, Public Health Scotland, UK