Featured Product
This Week in Quality Digest Live
FDA Compliance Features
Jill Roberts
Another way to know what’s too old to eat
Patricia Santos-Serrao
Four pharma quality trends
Del Williams
Preventing damage caused by large, suspended particles
Kari Miller
An effective strategy requires recruiting qualified personnel familiar with the process and technology

More Features

FDA Compliance News
Now is not the time to skip critical factory audits and supply chain assessments
Google Docs collaboration, more efficient management of quality deviations
Delivers time, cost, and efficiency savings while streamlining compliance activity
First trial module of learning tool focuses on ISO 9001 and is available now
Free education source for global medical device community
Good quality is adding an average of 11 percent to organizations’ revenue growth
Further enhances change management capabilities
Creates adaptive system for managing product development and post-market quality for devices with software elements
VQIP allows for expedited review and importation for approved applicants that demonstrate safe supply chains

More News

Jim Frost

FDA Compliance

How Effective Are Flu Shots?

Sometimes 60% isn’t all it’s cracked up to be

Published: Monday, January 28, 2013 - 12:15

This flu season has been worse than normal. The Centers for Disease Control and Prevention (CDC) data show that the flu has struck early and hard. Influenza cases shot up during December rather than the more usual January or February, and 47 states report widespread influenza cases.

I get a flu shot every year even though I know they’re not perfect. I figure they’re a relatively easy and inexpensive way to reduce the chance of having a miserable week.

I’ve heard on various news media that the shots’ effectiveness is about 60 percent. But what does 60-percent effectiveness mean, exactly? How much does this actually reduce the chances that I’ll get the flu in any given year? I’m going to explore this and go beyond the news media simplification and present you with very clear answers to these questions. Quite frankly, some of the results were not what I expected.

We’ll find our answers in randomized, controlled trials

I’m a numbers guy. I use numbers to understand the world. My background is in research, so when I want to understand an issue, I look at the primary research. If I can understand the researchers’ methodology, the data they collect, and how they draw their conclusions, I’ll understand the issue at a deeper, more fundamental level than news reports typically provide.

To understand flu shot effectiveness, I’m only going to assess double-blind, randomized controlled trials (RCTs), the gold standard. These studies are more expensive to conduct but provide better results than observational studies. (I discuss the differences between these two types of studies in my article about the benefits of vitamins.)

The two influenza vaccination studies I’ll look at satisfy the above criteria and are listed in a section of references for health professionals on the CDC’s website. Presumably these studies make a good case, using trusted data. Along the way, we’ll use Minitab statistical software to analyze their data for ourselves.

Defining the effectiveness of flu shots

Flu shots contain vaccine for three influenza viruses that researchers predict will be the most common in a given flu season. However, plenty of other viruses (flu and otherwise) also are circulating and can make you sick. Many illnesses with flu-like symptoms are incorrectly attributed to the flu.

Consequently, the best studies use a lab to identify the specific virus that infects each of their sick subjects. These studies count only the subjects with confirmed cases of the three types of influenza virus. Effectiveness is defined as the reduction in these three influenza viruses among those who were vaccinated compared to those who were not vaccinated.

It’s time to dig into the data. For me, this is where it gets exciting. You can hear about flu-shot effectiveness in the media, but this is where the information comes from: counts of sick people in the experimental groups.

The Beran study

The Beran et al. study (see references below) assesses the 2006–2007 flu season and tracks its subjects from September to May. Subjects in this study range from 18 to 64 years old.


Flu count

Group size







Because we want to compare the proportions between two groups, we’ll use the Two Proportions test in Minitab. To do this yourself, in Minitab go to Stat > Basic Statistics > 2 Proportions. In the dialog, choose “Summarized data” and enter the data from the table above. Click “OK,” and you get the results below:

Minitab's Two Proportions test for the flu data

The p-value of 0.000 tells us that there is a significant difference between the two groups. The estimated difference indicates that the vaccinated group has 1.9-percent fewer cases of the flu than the placebo group. Because this is an RCT, it’s fairly safe to assume that the vaccination caused the difference between the groups. However, outside of a randomized experiment, it’s not wise to assume causality.

The vaccine effectiveness (or efficacy) is a relative reduction in risk between the two groups. You simply take the relative risk ratio of (vaccinated proportion/unvaccinated proportion) and subtract that from 1. We can get the proportion for each group from the Sample p column in Minitab’s output:

1 - (0.009602/0.029031) = 0.669

This study finds a 66.9-percent vaccine efficacy for the flu shot compared to the placebo.

The Monto study

The Monto et al. study (see references) assesses the 2007–2008 flu season and tracks its subjects from January to April. Subjects in this study range from 18 to 49 years old.


Flu count

Group size







We’ll do the Two Proportions test again for this study. This time, enter the numbers from the above table into the dialog.

Minitab's Two Proportions test for the flu data

Again, the p-value indicates that there is a significant difference between the two groups. The estimated difference shows that the vaccinated group has 7.3-percent fewer cases than the placebo group. Let’s calculate the effectiveness:

1 – (0.034440/0.107692) = 0.680

This study finds a 68.0-percent vaccine efficacy for the flu shot compared to the placebo.

Conclusions so far

We’ve looked at the data from two gold-standard studies and have drawn the same conclusions that you commonly hear on the news: Flu shots significantly reduce the number of influenza infections, and they are about 68-percent effective.

However, looking at the data and analyses myself, I have new insights. Specifically, the low number of influenza cases in the placebo group for each study caught my eye, and that’s what we’re looking at next.

What this means for you: relative vs. absolute risk

If you’re like me, the 68-percent effective statistic isn’t too helpful. The problem is that it is a relative comparison of risk, not an absolute assessment of risk. To illustrate the difference, consider which type of assessment is more useful:
1. Relative assessment: Your car is travelling half as fast as another car, but you don’t know the true speed of either car.
2. Absolute assessment: Your car is travelling at 30 miles per hour and the other car is travelling at 60 miles per hour.

Clearly, the second assessment is much more useful. Similarly, it would be more helpful to know the absolute risk of catching the flu if you get the shot vs. not getting it.

Vaccine effectiveness is a relative risk

Vaccine effectiveness doesn’t tell you the exact risk of catching the flu for either group. Instead, it involves dividing one proportion by the other for the relative risk. In fact, as you should recall, effectiveness is the inverse of the relative risk, which makes it even harder to interpret. A 67-percent effectiveness indicates that a vaccinated person has one-third the risk of contracting the flu as a nonvaccinated person.

Unfortunately, using these numbers, we don’t know the absolute risk for anyone.

The group proportions are the absolute risks

We can estimate the absolute risk from the studies by looking at the proportion for each group in the Minitab output, and subtracting to calculate the absolute reduction. I’ll summarize this information below as percentages and even add in the results for two more flu seasons from another study that the CDC references (Bridges et al; see references):

Flu season


Flu shot

Risk reduction





















Notice how the risk of getting the flu varies by flu season? The differences are not surprising because the studies use different samples, and the flu seasons have different influenza viruses.

So let’s look at the average of these four flu seasons. If you aren’t vaccinated, you have a 7.0-percent chance of getting the flu. However, if you do get the flu shot, your risk is about 1.9 percent, which is a reduction of 5.1 percent.

Hmm. The “5.1-percent reduction” doesn’t sound nearly as impressive as the “67-percent effectiveness." Both statistics are based on the same data, but I think the estimate of absolute risk is a more useful way to present the results.

Closing thoughts about the flu shot data

I was surprised by the results. Although I knew flu shots were not perfect, I always got them because I thought they reduced my risk by more than what the CDC’s recommended studies actually show. Even if you aren’t vaccinated, your risk of getting the flu isn’t too high.

That probably explains why a number of people have told me that while they never get flu shots, they can’t remember having the flu!

These more subtle results made me wonder about flu vaccinations on a societal scale. Could the flu vaccine possibly reduce flu cases enough to save sufficient money (e.g., from lost workdays, doctor and drug costs) to pay for the vaccinations?

Bridges et al. conducted a cost-benefit analysis in their study. For the two flu seasons where they tracked flu vaccinations, infections, and expenditures, the vaccinations actually increase net societal costs. It would’ve been cheaper overall not to get vaccinated.

In light of this, I wasn’t surprised when I read an article on CNN.com that said, outside of the United States and Canada, other countries do not strongly encourage all of their citizens more than six months old to get a flu shot. According to the article, “Global health experts say the data aren’t there yet to support this kind of vaccination policy, nor is there enough money.”

I understand this viewpoint better now.

However, I’m not trying to talk anyone out of getting a flu shot. I’m on the fence myself. While the risk of getting the flu in any given year is fairly small, if you regularly get the flu shot, you’ll probably spare yourself a week of misery at some point. You should always consult a medical professional to determine the best decision for your specific situation.

In my next column, I’ll look at the long-term benefits of flu vaccinations.


1. Beran, J.; Vesikari T., Wertzova, V.; Karvonen, A.; Honegr, K.; Lindblad, N.; Van Belle, P.; Peeters, M.; Innis, B.L.; Devaster, J.M. “Efficacy of inactivated split-virus influenza vaccine against culture-confirmed influenza in healthy adults: A prospective, randomized, placebo-controlled trial.” J Infect Dis 2009;200(12):1861–9.

2. Monto, A.S.; Ohmit, S.E.; Petrie, J.G.; Johnson, E.; Truscon, R.; Teich, E.; Rotthoff, J.; Boulton, M.; Victor, J.C. “Comparative efficacy of inactivated and live attenuated influenza vaccines.” N Engl J Med. 2009;361(13):1260–7.

3. Bridges, C.B.; Thompson, W.W.; Meltze,r M.I.; Reeve, G.R.; Talamonti, W.J.; Cox, N.J.; Lilac, H.A.; Hal,l H.; Klimov, A.; Fukuda, K. “Effectiveness and cost-benefit of influenza vaccination of healthy working adults: A randomized controlled trial.” JAMA. 2000;284(13):1655–63.


About The Author

Jim Frost’s picture

Jim Frost

Jim Frost is a statistical technical communications specialist at Minitab Inc. He has a background in a wide variety of academic research and became known as the “data/stat guy” on research projects that ranged from osteoporosis prevention to quantitative studies of online user behavior. At Minitab, he is a technical writer who helps people use Minitab software to gain insights from their own data, whether they’re working in quality improvement, academic research, or another field entirely. He also writes in The Minitab Blog about various experiences and practical knowledge he’s learned along the way that may help others’ research endeavors.


By how much do flu shots reduce the chances of catching the flu?


What about the 'flu shots reducing the chances of 'flu transmission in certain populations?  In other words, if I spend October to February largely with my immunized family then my chances of catching flu are much reduced.  Likewise if getting the shots was mandatory for everyone, for example, in the Army.

Also, you said:

"So let’s look at the average of these four flu seasons. If you aren’t vaccinated, you have a 7.0-percent chance of getting the flu. However, if you do get the flu shot, your risk is about 1.9 percent, which is a reduction of 5.1 percent"

I would calculate the reduced risk differently.  The 7.0 becomes 100% of the 'flu sufferers and as part of the 1.9 with my 'flu shot I would 27% as likely to catch 'flu.  A reduction of 73% not 5.1%.

Would you agree?


Relative vs Absolute risk


If you spend most of your time with immunized people, I'd expect your risk to decrease even if you aren't immunized. For my column, the studies I reference look at the general population of healthy adults. It would not surprise me if various factors increased or decreased your specific level of risk.

Your calculations are another form of relative risk which doesn't tell you the absolute risk. Suppose that one car is traveling at 60mph and you call that 100%, and another car is traveling at 30mph, which you call 50%. You could say that is a reduction of 50% or 30mph. Both are based on the same data and true. My point is that you gain valuable contextual information by looking at it in absolute terms.

Or, put it this way. Suppose you have two pairs of cars travelling at different speeds:

1) 200 mph vs 100 mph

2) 60 mph vs 30 mph

For both pairs, the relative reduction is 50%. You get the same number for very different cases, which isn't so helpful by itself. However, if you put that in their absolute terms, suddenly the picture is more clear!

Thanks for reading!