Dammed lies and statistics

Mark Twain wrote: “There are three kinds of lies: lies, damned lies, and statistics.”

Statistics continue to mislead the unwary in the current milieu of rapid online publication of articles and preprints. A case in point is the study published last month in the fast-tracking online medical journal Cureus. This study, entitled Regular Use of Ivermectin as Prophylaxis for COVID-19 Led Up to a 92% Reduction in COVID-19 Mortality Rate in a Dose-Response Manner: Results of a Prospective Observational Study of a Strictly Controlled Population of 88,012 Subjects, purported to show that prolonged, regular use of ivermectin lessened the incidence and mortality of COVID-19 as measured by total dose taken in the period before infection occurred.

Note that though the study title states it was prospective, planning a study in advance is not sufficient for a study to be prospective. Because subjects were only placed into their treatment groups after the study outcomes were known, this was a retrospective study.

The problem lies in the study's design: total dosage of ivermectin over the entire period before COVID infection was the regression variable, versus mortality in those infected. However, because intention-to-treat analysis was thus bypassed, the study was actually measuring the mortality in the group which took ivermectin for many months and either never got COVID-19 or got COVID-19 only after that period, versus the mortality in those who either never took ivermectin or who took ivermectin for only a short time. This means that during the early months of this study, almost all of the patients who developed COVID-19 were placed in the non-ivermectin or the "irregular user" group even if they would have been placed in the ivermectin group had they NOT come down with COVID (and thus have more time to take doses of ivermectin, as many ultimately did). In other words, the fact that some who got COVID-19 before taking ivermectin might well have taken ivermectin later in any given study period had they not gotten COVID-19 first was ignored.

Was this study "strictly controlled"? Yes, in a manner that biases it beyond repair, unless they had ALSO given intent-to-treat data! Only the authors know why their design was designed the way it was.

----------------------------------------------------------------------------------

REFERENCE: Kerr L, Baldi F, Lobo R, et al. (August 31, 2022) Regular Use of Ivermectin as Prophylaxis for COVID-19 Led Up to a 92% Reduction in COVID-19 Mortality Rate in a Dose-Response Manner: Results of a Prospective Observational Study of a Strictly Controlled Population of 88,012 Subjects. Cureus 14(8): e28624.

DOI: 10.7759/cureus.28624

---------------------------------------------------------------------------------

Here is a computer simulation (written in Julia) which demonstates the study bias mentioned above. In the program, the treatment is set to have no actual effect on the infection rate. The highly significant results are purely from the bias of excluding those who become infected from later entering a treatment arm.

using HypothesisTests

@enum TreatmentClass Untreated Irregular Excluded Regular

mutable struct Subject
    cum_dose::Float64
    treatment_class::TreatmentClass
    had_covid::Bool
    last_dose_day::Int
end

function update!(subjects::Vector{Subject}, day, pcovid = 0.001, pstart = 0.0075, pdosing = 0.25, dosage = 35)
    for subj in subjects
        if subj.had_covid
            continue
        elseif rand() < pcovid
            subj.had_covid = true
        elseif (subj.cum_dose > 0 && rand() <= pdosing  && (day > subj.last_dose_day + 14 || day == subj.last_dose_day + 1)) ||
           (subj.cum_dose == 0 && rand() < pstart)
            subj.cum_dose += dosage
            subj.last_dose_day = day
            subj.treatment_class =
               subj.cum_dose == 0 ? Untreated : subj.cum_dose >= 180 ? Regular : subj.cum_dose <= 60 ? Irregular : Excluded
        end
    end
end

function run_study(N = 10_000, duration = 150)
    population = [Subject(0.0, Untreated, false, 0) for _ in 1:N]
    unt, unt_covid, irr, irr_covid, reg, reg_covid, excluded = 0, 0, 0, 0, 0, 0, 0
    println("Population size $N, daily infection risk 0.1%")
    for day in 1:duration
        update!(population, day)
        if day % 30 == 0
            println("\nDay $day:")
            unt = count(s -> s.treatment_class == Untreated, population)
            unt_covid = count(s -> (s.treatment_class == Untreated) && s.had_covid, population)
            println("Untreated: N = $unt, with infection = $unt_covid")
            irr = count(s -> s.treatment_class == Irregular, population)
            irr_covid = count(s -> (s.treatment_class == Irregular) && s.had_covid, population)
            println("Irregular Use: N = $irr, with infection = $irr_covid")
            reg = count(s -> s.treatment_class == Regular, population)
            reg_covid = count(s -> (s.treatment_class == Regular) && s.had_covid, population)
            println("Regular Use: N = $reg, with infection = $reg_covid")
            exc = count(s -> (s.treatment_class == Excluded) && s.had_covid, population)
            println("Excluded: N = $exc")
        end
        if day == 75
            println("\nAt midpoint, Infection case percentages are:")
            println("  Untreated : ", Float16(100 * unt_covid / unt))
            println("  Irregulars: ", Float16(100 * irr_covid / irr))
            println("  Regulars  : ", Float16(100 * reg_covid / reg))
        end
    end
    println("\nAt study end, Infection case percentages are:")
    println("  Untreated : ", Float16(100 * unt_covid / unt), " of group size of $unt")
    println("  Irregulars: ", Float16(100 * irr_covid / irr), " of group size of $irr")
    println("  Regulars  : ", Float16(100 * reg_covid / reg), " of group size of $reg")
    untreated = [s.had_covid for s in population if s.treatment_class == Untreated]
    irregular = [s.had_covid for s in population if s.treatment_class == Irregular]
    regular = [s.had_covid for s in population if s.treatment_class == Regular]
    excluded = [s.had_covid for s in population if s.treatment_class == Excluded]
    println("\n\n   Final statistics:\n")
    @show KruskalWallisTest(untreated, irregular, regular, excluded)
end

run_study()