Beware of These Common Types of Bias

Quality-related data collection is useful, but statistics can also deliver misleading and even dysfunctional results when incomplete. This is often the case when information is collected only from surviving people or products, extremely satisfied or dissatisfied customers, or propagators of bad news about relatively rare incidents. This is simply an extension of the basic principle that a sample should reflect the entire population rather than just a portion—especially a portion that might self-select for the indicated reasons.

Survivor bias

Survivor bias occurs when samples include only people or items that survive harmful conditions. When World War I started in 1914, the soldiers of most armies wore cloth or leather caps that provided almost no protection against shrapnel and small arms fire. The French, for example, replaced their world-famous kepi with the equally recognizable Adrian helmet only in 1915. The British followed suit and introduced the Brodie, later called “Tin Kelly,” helmet. Even the famous German spiked helmet was made of leather and not steel, to be replaced by the Stahlhelm a year into the war.

However, accounts reported that British casualty stations began receiving far more soldiers with head wounds after the introduction of the Brodie helmet, which promoted the erroneous conclusion that wearing one increased your chance of suffering a head wound.¹ If true, what actually happened was that men who would have otherwise been killed outright reported for treatment of wounds that were now survivable. Hence, survivor bias.

More to the point was a study conducted during World War II by the famous statistician Abraham Wald.² Aircraft manufacturers used a familiar concentration diagram or “measles chart” to show where enemy gunfire had hit Allied aircraft. Their natural inclination was to increase the armor over the locations with the most hits. But Wald suggested, counterintuitively, putting armor over the locations with few (if any) hits. The reasoning was that those locations were vital areas, and the aircraft struck there had not returned for inclusion in the study. This same caveat might apply to failure analysis of products that are returned for repairs while those with nonrepairable failures are simply discarded.

Survey bias

In yet another example of bias within data collection, customer surveys are also apt to be less reliable because they represent only those who take the time to answer them. These people are likely to be one of two extremes: extremely satisfied and extremely dissatisfied. Product and service review sites are often overrun with one-star ratings because angry customers leave bad reviews, while moderately satisfied ones often leave no reviews at all. Extremely satisfied customers, on the other hand, often take time to reward the sellers with five-star reviews. Time-consuming surveys are also less likely to be answered than shorter ones, which can also affect the representativeness of the sample.

The same principle carries over into customer complaints, where a performance measurement, such as complaints-per-month, may be dangerously misleading. Many dissatisfied customers don’t bother to complain; they just stop buying the company’s goods or services.

Bad news bias

Bad news bias, which is actually what prompted me to write this article, is yet another form of poor information. I adopted a dog and saw numerous horror stories about side effects of the Lyme disease vaccine that was recommended by the veterinarian. A Google search on these horror stories brings up anecdotes about dogs that died shortly after getting the vaccine, as well as a story about a dog that developed a painful swelling at the injection site.

However, Pennsylvania is full of ticks that carry Lyme disease, which can wreck a dog’s joints and kidneys, among other things. Accordingly, I took the vet’s advice and got my dog vaccinated. I also looked online to see if one was available for humans for the same reason. (Unfortunately, there is not.) I perceive that what may actually be happening is that people whose dogs had bad reactions to the vaccine, or even those whose dogs experienced problems that coincided with the vaccine but were not necessarily caused by it, published the negative stories. Meanwhile, those without problems said nothing at all.

The same issue has discouraged people from getting the vaccine for Covid-19. The FDA, for example, limited access to the Johnson & Johnson vaccine due to rare occurrences of thrombosis with vaccine-induced thrombocytopenia (VITT), a type of blood clot.³ The reference adds that there have been 60 cases of VITT, and nine fatalities, while 16.9 million people have gotten this vaccine. If the vaccine does in fact cause VITT, then the chance of dying from it is less than one in a million—while the disease it is intended to prevent has already killed a million people out roughly 330 million, or three out of every thousand. Of course, the actual death rate depends on age and health status, but the bottom line is that the risks associated with the disease for which the vaccine exists far exceed any risks associated with the vaccines.

The problem is that the horror stories about vaccine side effects get both attention and repetition, while people who get the vaccine and don’t suffer side effects tend to not talk about their results because they take them for granted. (I got the vaccine and a total of three boosters since March 2021, and nothing bad happened to me. In fact, if I got Covid-19 despite this protection, it must have been so mild as to go unnoticed.) People who get symptomatic Covid-19 despite having had the vaccine tend to report their dissatisfaction, while those who don’t get it, or get it but not badly enough to know it, usually take it for granted that they didn’t get sick.

As a result, many people resist getting a vaccine that is very unlikely to harm them but can easily save them from a life-threatening disease, one that can cause long-term health problems among survivors. These include “...long-term breathing problems, heart complications, chronic kidney impairment, stroke, and Guillain-Barre syndrome—a condition that causes temporary paralysis.”⁴ Guillain-Barre syndrome is, ironically, a very rare complication from various vaccines.

Summary

Information is useful if we interpret it correctly. But it can be worse than useless if we draw the wrong conclusions from it. Survivor bias refers to the collection of information about people or products that survive dangerous or destructive conditions, and disregards information about those who did not survive. This could lead, as proven by the Wald example, to the wrong decision to add armor to nonvital portions of an aircraft rather than the vital ones. Survey bias occurs because surveys report information from only those inclined to answer them, and there might also be bias in customer review sites because only those who have strong positive or negative feelings about a product or service bother to leave reviews. Bad news bias exists because bad news, even about relatively isolated anomalies such as dangerous vaccine reactions, is reported and repeated while no news (e.g., the overwhelming majority of vaccinated people had no problems and also did not get the disease) is exactly that: no news. We must therefore always be careful of how we interpret the information we receive to avoid making the wrong decisions.

References
1. Sloane, Paul. Lateral Thinking Puzzlers. Puzzlewright, 2016.
2. Miller, Brendan. “How ‘survivorship bias’ can cause you to make mistakes.” BBC, Aug. 28, 2020.
3. Ellis, Ralph. “FDA Limits Use of J&J Covid Vaccine Over Blood Clot Risk.” WebMD, May 6, 2022.
4. “Covid-19 (coronavirus): Long-term effects.” Mayo Clinic, Oct. 22, 2021.

About The Author