“Ebola Ça Suffit!” is not enough to Prove Efficacy of an Ebola Vaccine

Email: david.crowe@cnp-wireless.com Abstract: A recently published Ebola vaccine trial claimed an almost perfect result with no cases of Ebola after the vaccine had time to take effect a vaccine efficacy of 100%. This article is a detailed review of the information published about the trial and, particularly, this claim. The trial design was sub-optimal because randomization was only partial (and one treatment arm was unrandomized), it was unblinded and did not use a placebo. Less information on the comparison arm that substituted for a placebo was provided, such as adverse events, compared to the active treatment arms. There was little baseline information on the trial participants, which is particularly important to ensure equivalence of the arms of a trial that was not fully randomized. In particular, Ebola tests were not performed at baseline, so the possibility of false positive test results or pre-existing asymptomatic cases exists. Ebola symptoms and adverse reactions following vaccination seen during the trial were very similar, allowing diagnostic bias. The exclusion of Ebola cases among the vaccinated during the 10 days after randomization is an arbitrary decision for an infection with an incubation period of 2 to 21 days and not a substitute for comparing vaccine to placebo. Considering the entire 31-day reporting period, two subgroups of unvaccinated participants had significantly fewer cases of Ebola than the two vaccinated groups. The problems with this trial are so grave that it cannot be taken as even weak evidence that the vaccine trialed is effective at preventing Ebola Virus Disease (EVD).


Introduction
A paper published in 2016 announced highly positive results for an Ebola virus vaccine, with no infections in people vaccinated, after the vaccine had time to take effect (Henao-Restrepo et al., 2016). This was widely acclaimed in the press even after only interim results were published, the year before, with headlines such as, "Ebola vaccine trial proves 100% successful in Guinea" (Boseley, 2015).
This vaccine, once fully commercialized, could be offered to tens or hundreds of millions of Africans. If safe and effective it could reduce the scourge of Ebola epidemics, but if not it could add the cost of adverse reactions to the vaccine to an unchanged burden of Ebola outbreaks and discourage other responses to Ebola prevention.
When Ebola Hemorrhagic Fever was described and named in the 1970s, it was a deadly and horrifying hemorrhagic fever first found near the eponymous river, but current definitions no longer require bleeding and it appears that this is now a relatively rare symptom (Schieffelin et al., 2014). Bleeding has even been removed from its name, which is now Ebola Virus Disease (EVD).
The result of this trial that was most prominent in (Henao-Restrepo et al., 2016) and in media articles, was that no cases of Ebola were diagnosed in vaccinated participants more than 10 days after randomization. However, this 10 day period is arbitrary, ignores the widely cited 2 to 21 day incubation period (WHO, 2017) and avoids consideration that, overall, there were significantly more Ebola cases in the two vaccinated subgroups than in some unvaccinated subgroups, a result for which no explanation was provided.
Perhaps due to the desire to test the vaccine before the Ebola epidemic waned, an unusual and obviously deficient trial design was used, without a placebo, with only partial randomization and without blinding to either participants or trial workers.
The trial also recorded little baseline information, essentially no socio-economic or health data, so the comparability of the groups cannot be properly assessed, which is particularly important for a trial that was not double-blinded, randomized and placebo-controlled.
An important missing piece of baseline data is an Ebola test on each participant, which is needed because high rates of false positives have been documented for some tests (Becquart et al., 2010). Combined with a similarity between Ebola symptoms (CDC, 2014;2015) and adverse events in the trial that were not classified as Ebola, the possibility of misclassification of vaccine side effects as Ebola or vice-versa is immense.
This paper reviews the claims of Henao-Restrepo et al. (2016) that the vaccine being tested is highly effective at preventing Ebola Virus Disease (EVD).

Terminology
The three arms of the trial will be referred to as Immediate, Delayed and Unrandomized. The term "Ebola" will be used to refer to "Ebola Virus" and "Ebola Virus Disease (EVD)" as long as the context is clear. Otherwise it will be fully spelled out. Monitoring periods in the trial (Fig. 1) are referred to as VDP (Vaccine Delay Period, between randomization and consent/vaccination), EP (Exclusion Period, from vaccination to 10 days), MP (Monitoring Period, unvaccinated period, sometimes divided into the first 10 days, MP1 and the remaining time, MP2).

Trial Design
The design of the trial was not optimal. It was not blinded, let alone double-blinded, did not have a placebo arm and was only partially randomized, deficiencies which are known to affect outcomes in favor of interventions (Colditz et al., 1989).
Instead of a placebo, the trial was divided into Immediate and Delayed arms. Those in the former arm were vaccinated shortly after (partial) randomization and those in the latter arm were vaccinated approximately 21 days later.
The design of the trial made blinding impossible. Not only did the trial staff know which arm each participant was in but so did the participants.
The Delayed arm was obviously intended to substitute for a blinded placebo because the participants in this arm could be observed for 21 days without vaccination. However, the participants had not actually received a mock injection and were aware of the fact they would not be vaccinated until later. Worse yet, this group was vaccinated 10 days before the end of the trial, so they do not even constitute a fully unvaccinated comparison group and the data for these last 10 days is not segregated.
The trial was cluster randomized, which means that, for each of the 117 cases of Ebola selected as the root of a new cluster, all that patient's contacts and contacts of contacts, were enrolled in the same randomly chosen arm (an average of 80 in Immediate, 81 in Delayed and 105 in Unrandomized). This means that the sample size of the trial was not really the 9,096 individuals assigned to randomized clusters, but only the 98 clusters.
A further deviation from randomization was that, four months into the trial, randomization ceased and an additional 19 clusters, representing 2,745 participants, were included without randomization. Furthermore, this group allowed children, aged 6-18, to be vaccinated, unlike the two randomized arms and it was included in the analysis of the trial.
Of the 117 index cases, "27 were index cases and also endpoints" (Henao-Restrepo et al., 2016, Results section). There was no discussion of how the overlap of contacts was handled between the parent and child cluster. Additionally, there were another 73 cases that were not used in this way because they were "already included in an existing cluster" (Henao-Restrepo et al., 2016, Fig. 1). There was no discussion of why some Ebola cases were used to start a new cluster, but most were not.

Baseline Information
Very little baseline information was provided either for the index cases or for the participants. Statistics were not provided to show whether the difference between the trial arms were likely due to chance.
Despite the paucity of data there were some obvious differences between the groups on the basis of the limited baseline information for the index cases. This included age, gender, whether the index case was alive at the time of randomization, three parameters based on timing following onset of symptoms, size of the cluster and location in a rural or urban area: • Due to the late inclusion of children, the median age of the index cases for Unrandomized was more than 10 years younger (23) than Immediate and Delayed (35 in both cases) • Even though the median ages of index cases in the two randomized arms were similar, the Interquartile Ranges (IQR) were different, 18-43 for Immediate but 27-50 for Delayed (and 13-42 for Unrandomized) • The fraction of female index cases in Immediate was noticeably lower (53%) than in Delayed (66%) and Unrandomized (63%) • More of the Delayed index cases were dead at the time of randomization (68%) than in Immediate (59%) or Unrandomized (47%) • Many fewer of the Unrandomized index cases were from rural areas (47%) than in Immediate (76%) or Delayed (77%) Similarly, few baseline characteristics were collected for participants, just age, gender, whether they were a direct contact of the index case, or a contact of a contact and whether they were considered a high-risk contact. Furthermore, the data was aggregated for all those who consented, not just for those who were actually vaccinated. This was especially significant for those in Delayed who consented on day 0 as data was presented for all 1,435 and not for the 940 who were actually vaccinated on day 21.
Notable differences in the trial arms in the baseline characteristics for participants were: • The fraction of women among those who consented was much lower (30-35% in the three arms) than in the groups that did not consent (55-57%) • The median age of those who consented in Immediate was 10 years older than those who did not consent One could expect that there would be a strong correlation between the baseline socio-economic and health characteristics of an index case and the members of the associated cluster, but no such information was presented. Random variations in the index case characteristics would be multiplied if this was the case.
Another important missing baseline parameter was an Ebola test, particularly important because previous research has shown a 15.3% rate of positive Ebola test results in healthy people in an area with no Ebola cases during the time of the survey (Becquart et al., 2010). Earlier research cited by the same paper found rates of 13.2% in an Aka Pygmy population in the Central African Republic within which Ebola has never been reported and 9.3% in an area of the Democratic Republic of Congo which had no Ebola cases around the time of the survey.

Consent
One striking difference between Immediate and Delayed was that the latter were given two opportunities to consent (day 0 and day 21). Although the fraction of both groups that was ruled eligible, consented to vaccination and actually were vaccinated, was similar (2,119/4,539 = 47% in Immediate and 2,041/4,557 = 45% in Delayed) the number who consented on day 0 in Delayed was much lower (31%). Not only did those who did not consent on day 0 have the option of consenting on day 21, those who did consent on day 0 also had the option of withdrawing consent or being absent and 480 took one of these options on the day of vaccination. Without adequate baseline information and without information on the group that was actually vaccinated, the possibility that this significant procedural difference increased the differences between Immediate and Delayed and thus lessened the comparability, cannot be excluded.

Exclusion Period
Cases of Ebola in vaccinated groups less than 10 days after randomization were not considered a trial outcome. Reasons given were (a) the incubation period of Ebola; (b) the time between onset of symptoms and diagnosis; and (c) the "unknown period between vaccination and a vaccine-induced protective immune response" (Henao-Restrepo et al., 2016).
This exclusion, however, should be unnecessary in a trial with a comparison group, because the first two considerations would be the same in both groups and part of the goal of a vaccine trial should be to determine the length of the third period. Development of disease that is supposed to be prevented by the vaccine should decline over time and, from this, the delay between vaccination and full efficacy could be estimated.
The 10 day period also cannot be justified on the basis of the incubation period because that is widely believed to vary between 2 and 21 days (WHO, 2017).
Immediate and Unrandomized had the 10 day exclusion period at the beginning of the 31 day period, but for the Delayed group, this occurred at the end. Clearly the 10 day periods after vaccination (not randomization) are most comparable, yet the trial compared the case counts of Ebola in the groups chronologically in Table 4 of (Henao-Restrepo et al., 2016), so the first 10 days of all groups were compared, as opposed to the 10 days following vaccination. This resulted in four cases in the Delayed group, that occurred in the days following vaccination on day 21, being relegated to a footnote in the supplementary materials.
Another exclusion period is the time after the end of the observation period during which, "vaccinated cases of Ebola virus disease with an onset of more than 31 days after random assignment were censored" (Henao-Restrepo et al., 2016, Outcomes). This shows that there is no evidence that the vaccine protects anyone for more than 21 days and leaves open the possibility that cases of Ebola in vaccinated participants were detected, but not published.

Adverse Events in the Vaccinated
The trial found a high rate of minor adverse events in those vaccinated. These events were not reported for Delayed, making it impossible to determine the fraction due to the vaccine. The total fraction of those vaccinated who experienced each adverse effect also cannot be determined because they were reported for three separate time periods (which do not match any of the monitoring periods in the trial) and it is likely that some types of adverse events were reported by the same person in multiple periods.
Considering just the 31 min to 3-day post vaccination time period for adults, during which most adverse effects were reported, the most frequent adverse events were headache (27.7% of vaccine recipients), muscle pain (15.5%), arthralgia (15.1%) and myalgia (14.5%). These are plausibly vaccine side effects, along with the less common injection pain (6.4%), fatigue (2.2%) and induration (1 case), although, as already stated, there was no reporting from the comparison group to determine how many were likely due to the vaccine and how many likely would have occurred anyway.
There were 80 adverse events classified as serious and detailed in supplementary Table S8 of (Henao-Restrepo et al., 2016), including some deaths. The researchers declared only two definitely related to vaccination (a serious fever and a serious allergic reaction) and one possibly (influenza-like symptoms). Serious adverse events were not reported for the unvaccinated, except for some Ebola diagnoses (and one road traffic accident, which may have been included in error). There were only 14 Ebola diagnoses detailed as serious adverse events in the unvaccinated, even though the total reported was 80. It is not clear why only a subset was detailed, when all Ebola cases in the vaccinated were shown.
Comparing the number of serious adverse events in Immediate with the number in the Delayed comparison group would help determine the fraction that are probably due to the vaccine and could indicate whether any types were associated with the vaccine.

Side Effects versus Ebola Symptoms
The definition of Ebola used in Henao-Restrepo et al. The symptoms listed are, "Elevated body temperature or subjective fever or symptoms, including severe headache, fatigue, muscle pain, vomiting, diarrhea, abdominal pain, or unexplained hemorrhage". The list can be enumerated as (1) fever; (2) severe headache; (3) fatigue; (4) muscle pain; (5) vomiting; (6) diarrhea; (7) abdominal pain; and (8) unexplained hemorrhage. A CDC Evaluation Algorithm (CDC, 2015) linked from the same web page requires only headache (not necessarily severe), does not require that hemorrhage be unexplained and adds (9) weakness.
There is significant overlap between these Ebola symptoms and the most frequent adverse events listed in the trial. In the following Given that all participants in the trial were either contacts of an Ebola victim, or a contact of a contact and were in an area with an ongoing Ebola epidemic, the epidemiologic risk factor could probably be satisfied.
This only leaves the positive Ebola test to distinguish an adverse event from a case of Ebola and, as pointed out above, a high rate of false positives is to be expected. This risk could be eliminated by a baseline Ebola test, but this was not done.
There is no explanation in (Henao-Restrepo et al., 2016) of why they decided to classify some symptoms as adverse events and others as Ebola. This is a significant potential source of bias, in this unblinded trial, if trial workers were, consciously or unconsciously, more likely to classify the same symptoms as an adverse event in a vaccine recipient and as a possible case of Ebola in a non-recipient.
Although there are relatively few cases of Ebola to draw inferences from, it is notable that more than half the 25 cases after vaccination occurred on the same day as vaccination (6), the day after (3) or the next day (7) yet there were no cases during the approximately two day VPD, between randomization and vaccination ( Fig.  1 and Table S2 of Henao-Restrepo et al., 2016). This is compatible with the vaccine rapidly producing immunity, but more so with the vaccine inducing side effects that can be misdiagnosed as Ebola in someone with a pre-trial asymptomatic positive Ebola status.

Number of Ebola Cases
During the vaccine trial 105 cases of Ebola were reported among trial participants. The trial managers obviously felt that the lack of Ebola cases among vaccinated participants more than 10 days after vaccination was the most important result, but a close examination of the data calls the certainty of this result into question and reveals other, less positive data.
Using data from Henao-Restrepo et al.  Table 4, S2 and S3) which we have reorganized in our Table 1, the following observations can be made, expressing the case rate as the number of cases per day per 1000 people: • In Immediate, the rate of Ebola over the full 31 day observation period among those vaccinated was 0.165 (cases per day per 1000 people) but 0.477 among those who did not consent, which is compatible with the vaccine working • In Unrandomized, the rate among the vaccinated was 0.192 but among those who did not consent was only about half, 0.098, which is not evidence for the vaccine working • In the Delayed arm, the rate of Ebola among those who consented on day 0 was 0.360, but among those who did not consent was much higher: 1.158. The rate of Ebola among those who consented on day 21 was less than a tenth of those who consented on day 0: 0.029 • There was only a single case of Ebola in Delayed in those who consented on day 21 and it occurred after vaccination • The overall rate of Ebola in those who were vaccinated (0.165 in Immediate and 0.192 in Unrandomized) was much higher than in the unvaccinated in Unrandomized (0.098) and particularly in those in Delayed who consented on day 21 (0.029) No statistics were presented to help determine which of these anomalies can be explained as statistically insignificant and which may be real.

Anomalies and Errors
We discovered a number of book-keeping errors: • The abstract Interpretation section erroneously refers to the monitoring period starting "from day 10 after vaccination", while the rest of the paper, including the Findings section of the Abstract correctly defines it as "10 days or more after randomization" • Figure 1 divides the 476 cases reported in "Basse-Guinée" during the trial period into 117 index cases and 361 excluded cases, but this adds up to 478. This may be because two of the index cases were from Sierra Leone, but this is not clearly stated • Figure 1 indicates that there were 73 Ebola cases not included as index cases because they were also end points but the Results section implies that there were 78 (105 total cases less 27 end points used as index cases) • The Results section states that 281 out of 476 cases were not used as index cases, but Fig. 1 lists 361. Even assuming that this is not supposed to include the 73 cases discovered and excluded during the trial, Fig. 1  (2016) is titled, "Reported Serious Adverse Events in all vaccinated" yet includes cases listed as "Not vaccinated" • Only 14 cases of Ebola in people described as "Not vaccinated" were described in detail in Table S8 of Henao-Restrepo et al. (2016), out of 80 such cases total, with no explanation of why only a subset were documented • Entry 9 is a traffic accident listed as a serious adverse event for an unvaccinated person, which is probably an error since no other serious adverse events (other than Ebola diagnoses) were included for the unvaccinated • Entry 16 indicates vaccination two months after randomization, which is probably wrong since vaccination was supposed to take place shortly after randomization or on day 21 • Entry 51 has an enrollment date from 2013, even though the trial didn't start until 2015

Discussion
The design of this trial deviated from the gold standard of a double-blinded, randomized, placebocontrolled clinical trials in important ways. In fact, the trial was not blinded at all. It was cluster randomized (meaning that there were 98 randomizations instead of one for all 9,096 participants), with one arm completely unrandomized (19 clusters with 2,745 participants). Instead of a placebo (mock vaccine) there was a group that simply was not vaccinated until after a 21day observation period.
The design results in a complex set of monitoring periods of different lengths ( Fig. 1 and Table 1) which makes comparisons difficult, particularly because several monitoring periods include time before and after vaccination.
Comprehensive baseline data is always important, but even more so when a trial deviates from the recognized gold standard. In this trial there was essentially no socioeconomic or health data gathered, so the comparability of the three trial arms cannot be determined. Worse yet, the rules were changed during the trial with the addition of children only into one arm.
Despite the paucity of baseline data there were noticeable differences between the arms of the trial, including an average age more than 10 years younger in Unrandomized (probably most due to the inclusion of children only into this group), differences in the distribution of ages even where the medians were similar, fewer women assigned to Immediate vaccination, more of the fatal Ebola cases being assigned to the Delayed control group and more city dwellers in the Unrandomized group.
Another problem with the baseline data was that it was not reported for those who consented to vaccination (just for those broader group of those ruled eligible for vaccination), so the comparability of those actually vaccinated versus control is even less well known. One thing that can be determined is that the number of women who consented was much lower than men.
Given reports of high rates of false positive tests (or asymptomatic infections) with some types of Ebola tests and the wide variation in incubation periods of 2 to 21 days (WHO, 2017), a baseline Ebola test is critically important, but was not performed.
Another major difference between the Immediate active treatment group and Delayed comparison group, was that the latter was given two chances to consent. Evidence that these two subgroups might be quite different is that the number of Ebola cases, were strikingly different, with only one case in those who consented on day 21 -a case that occurred after vaccination.
Serious adverse events, except for 14 diagnoses of Ebola (and one traffic accident), were only reported for the immediately vaccinated groups. This meant that the determination of whether a serious adverse event was related to the vaccine was left to a clinical judgment. Reporting this information for control groups can help determine whether the number of adverse events is higher in the vaccinated and, for more frequent adverse events, determine if some are particularly associated with vaccination. This is particularly important in an unblinded trial to avoid bias in clinical judgment.
The 10 day exclusion period, during which cases of Ebola among the vaccinated were assumed to be due to the incubation period of Ebola, the time between symptoms and confirmation, or lack of time for the vaccine to produce effective antibodies, was arbitrary. This "a priori" (Henao-Restrepo et al., 2016) decision obscures the possibility that the as yet unproven vaccine would increase the probability of infection for a period after vaccination, or that it would become effective in more or less than 10 days. This would be unnecessary if the trial had a proper control group (i.e., blinded placebo) so that the dynamics of Ebola diagnoses over time in the two groups could be compared and the actual delay between vaccination and effectiveness could be determined, within limits, "a posteriori".
Complicating the analysis of data, the 10 day exclusion period at the beginning of the 31 day observation period for Immediate and Unrandomized was not compared to the comparable 10 day exclusion period at the end of the 31 day observation period for Delayed, but to the first 10 days.   The 10-day exclusion period was, inexplicably, the period after randomization and not after vaccination (about 2 days later), even though all the justifications for this period were related to vaccination.
Perhaps the biggest surprise with this trial is the overlap between common adverse events and symptoms of Ebola, particularly headache, fatigue and muscle pain (including myalgia as a synonym). Bias could lead to Ebola being preferentially diagnosed among the unvaccinated, while being diagnosed as a non-Ebola adverse event among the vaccinated. Although the diagnosis of Ebola was performed by an external reference laboratory (Henao-Restrepo et al., 2016), the decision about whether to forward a sample, or to dismiss the symptoms as minor adverse events, would have been a decision of the trial managers.
Without a baseline Ebola test to eliminate false positive results (e.g., someone who was Ebola test positive, but asymptomatic, before the trial, but developed compatible symptoms during the trial) and without forwarding samples from all participants with symptoms, the number of Ebola cases reported in the trial is unreliable.
Although it was reported that there were no cases of Ebola among the vaccinated, after the arbitrary 10 day exclusion period, the paper did include data showing that two unvaccinated subgroups had a much lower rate of Ebola diagnoses over the entire 31 day monitoring period.
At best, the effectiveness of the vaccine has been shown for 21 days, because all cases of Ebola after the end of the observation period were censored.
While the two vaccinated groups had Ebola diagnosis rates of 0.165 (Immediate) and 0.192 (Unrandomized), the subgroup within Delayed of people who consented on day 21 (and not on day 0) had a case rate of only 0.029. Within Unrandomized, the rate among the unvaccinated was about half (0.098) that among the vaccinated.
Within Delayed, the three subgroups (Consent on day 0, Consent on day 21, No Consent) had widely different rates of Ebola diagnoses (0.360, 0.029 and 1.158, respectively) despite the fact that all three subgroups were similarly believed to be unprotected by vaccination (the consenting groups had not been vaccinated for long enough to gain immunity by the assumptions of the trial and the non-consenting group was never vaccinated).
Some of the differences may be due to random variation due to a small number of cases, but there was almost a 40-fold greater rate of Ebola among those who did not consent in Delayed opposed to those who consented on day 1.

Conclusion
There are many reasons to want to cut scientific corners. Sometimes the justifications are due to ethical considerations, but in the case of this Ebola vaccine trial (Henao-Restrepo et al., 2016) they are described as "pragmatic" (Boseley, 2015), an attempt to quickly prove out a vaccine before the number of cases in the epidemic waned.
Scientific principles of clinical trials were put in place because of the danger of approving treatments that would later be found ineffective or dangerous (or both) after marketing. Quick, but scientifically invalid, approval of a vaccine is hopefully not more important than ensuring that a vaccine truly is safe and effective.
The belief that a safe and effective Ebola vaccine is available could have a significant impact on policy and may reduce interest in non-vaccine preventive techniques and the development of other vaccines.
A valid clinical trial should be double-blinded, randomized and placebo controlled, with adequate baseline data gathered to confirm the randomization. Side effects should be monitored by comparison with the blinded placebo group, not by the judgment of a scientist who may well, at least subconsciously, want the trial to succeed (i.e., prove that the treatment is superior to the control).
These are not just theoretical arguments; there are many signs in this trial that the main treatment group (Immediate) was not comparable at baseline with the main control group (Delayed) and particularly with the Unrandomized treatment group and those in Delayed who consented on day 0 appear to have been much more vulnerable to Ebola than those who gave consent on day 21.
The use of an arbitrary 10 day period, during which the vaccine was absolved of responsibility for any Ebola cases, disguises the fact that over the entire 31 day monitoring period two significant unvaccinated subgroups had a much lower rate of Ebola cases than the two vaccinated subgroups.
Due to overlap between documented common adverse events and the list of symptoms used to diagnose Ebola, combined with the lack of blinding and the consequent possibility of bias, the risk of false positive test results and the lack of baseline Ebola testing, the diagnosis of new Ebola cases in some people and the diagnosis of non-Ebola adverse events in others with similar symptoms, is unreliable and subject to bias.
Even if the above problems with the trial did not exist, it still would only show short term (21 day) protection from Ebola by using the vaccine.
It may be that a proper scientific trial of an Ebola vaccine is impossible until the next Ebola epidemic (and everyone wishes that this never happens). However, I caution against using an improperly trialed vaccine because 'it is better than nothing'. It may well not be.