Does the influenza vaccine work? A small observational study

The Pediatric Insider

© 2014 Roy Benaroch, MD

Flu season is in full swing here, and I’m seeing dozens of feverish, miserable kids a week. Since it started early this year, maybe that means influenza will burn out and be over soon—but maybe not. Some years we get a “double dip” as a second strain of flu moves through town.

Influenza does a very good job of working its way through our communities each year. The symptoms of flu, including runny nose and cough, make transmission of infected mucus almost guaranteed.  And the virus itself, already very contagious, changes over time– so neither natural infection nor immunizations provide reliable lasting protection.

Though far from perfect, influenza vaccinations should be an important part of your family’s flu prevention strategy. Their effectiveness varies from year to year, but is probably overall in the range of 50-70%. Not great, but if even half of the cases of flu could be prevented, that’s a whole lot less misery, and far fewer people continuing to spread infection. Remember: for every case of influenza prevented, that’s fewer exposures for the rest of us.

I get asked a lot: how’s this year’s vaccine doing? Does it work? So a few days ago I collected data from my practice. I copied out the log book we keep of flu tests from 12-18-2013 to 12-24-2013—this is a list of all of the rapid flu tests we did in one of my two offices, the names of the patients and the results of the tests. Then I went back through their charts to see if they had been fully vaccinated against influenza this year.

Here’s the data, raw, in a “2×2 contingency table”:

Vaccine Yes Vaccine No total
Flu test POS 2 14 16
Flu test NEG 9 9 18
Totals 11 23 34

I’m not going to go into big-time statistics with this—I’m not pretending that this was a full-scale, professional study. This was just a convenience sample of kids who had flu tests done in my office over a few days in December. But what it does show is striking:

  • If you had a flu vaccine, your chance of testing positive for influenza was about 20%.
  • If you didn’t have a flu vaccine, your chance of testing positive for flu was 60%– three times the risk.

For you statistics types, I did plug these numbers into a web-based statistics package, and based on the Chi-Square calculation for a 2×2 contingency table the difference was statistically significant with p<0.05. However: I’m not 100% sure I did that right. I’m not swearing by my statistical chops here. Anyone out there with a good statistics background: if you want to chew on the data, please do, and post in the comments what you’ve determined.

Now, to be honest, there are some big-time caveats to this “study.” We didn’t systematically test people based on certain criteria. Each doc decided who to test, and it’s possible that some of my docs would be more or less likely to test people depending on whether they had had a flu vaccine. The test itself isn’t perfect—though a positive test is quite reliable, it’s possible that a negative flu test misses up to even 50% of true flu cases. The study wasn’t a randomized clinical trial- whether or not each child was vaccinated was up to the parents, and could have been influenced by their individual child’s risks of influenza exposure. And I didn’t look at the timing of the vaccines that were given—it’s possible that some kids in the “Vaccine YES” column received their vaccines too late in the season to be effective, and should have been counted as “Vaccine NO.” Furthermore: I only had access to my own records. Some of the “Vaccine NO” children could have gotten flu vaccines elsewhere.

Still: from this small sample, it looks like flu vaccines were strongly effective at preventing influenza in children at my practice.

More on preventing and treating influenza

Explore posts in the same categories: Medical problems, Pediatric Insider information

Tags: , , ,

You can comment below, or link to this permanent URL from your own site.

16 Comments on “Does the influenza vaccine work? A small observational study”

  1. CATT Says:

    Here we go again. Scare tactics, finger pointing and intimidation. What a crock! And, all over a nothing disease.


  2. lala Says:

    …spoken like someone who has never had, or had a child with, the flu, presumably. I personally would never refer to it as a “nothing’ disease.

    Intimidation? Really?


  3. anon Says:

    What lala said. Houseful of asthmatics here. “A touch of the flu” may be a nothing disease, but pneumonia sure as hell isn’t. I speak from personal experience. And yes, we get our flu shots every year…one year, I managed to catch a mild flu despite that. The flu wasn’t so bad, but the asthma reaction wouldn’t quit and in my weakened state, I (and one of my sons) couldn’t fight off the nasty upper respiratory virus we all got a few weeks later, leading to a 2ndary pneumonia infection. Fun times. So for the sake of those of us who suffer terribly from flu (that would usually include babies and the elderly as well), please get your flu shots for herd immunity. That is a polite request, not intimidation.


  4. OMDG Says:

    Methodological quibble — Your study is not designed to show whether the flu vaccine works; it is designed to show whether people who are sick are infected with the flu. Your sample is comprised solely of people who came to your office with flu-like symptoms. Does it really matter to them whether they tested positive for the flu specifically, since they got sick with some other similarly unpleasant virus? Those who tested negative might even argue (wrongly), based on your data, that vaccination didn’t matter since they got sick with something else anyway.

    I am completely in favor of the flu vaccine, and know that it does work. I just don’t like bad science. 😛


  5. Samantha Says:

    Your chi-square p-value is not valid here because you have a cell count of less than 5. Because you have a 2×2 table, a valid statistical test is Fisher’s exact test — the two-sided p-value is 0.0296 (using SAS 9.3), which is still less than the magical 0.05.

    Hope this helps.


  6. Dr. Roy Says:

    OMDG, you are right. A much-better study would be to prospectively look at those who got flu imm versus those who did not, and track thru the winter whether they got influenza-like illness. But I don’t have the time (or grant money!) to do such a study. Thus, my little observational study. Which as you said, only shows that of those who present with flu-like illness, those who got the flu vaccine are less likely to be sick with flu virus in particular.

    Many better studies have also documented the effectiveness of influenza vaccine: I don’t think my little study will end up on that list!


  7. Dr. Roy Says:

    Thank you Samantha! I was hoping that someone with better statistics chops than I would help out!


  8. CATT Says:

    Excuse me lala and anon. I raised 4 children all of who came down with the Flu at one time or another. I also have had the Flu a number of times. It does amount to nothing or not much, if handled properly from get go. And, that means Echinacea tincture every hour, along with increased intake of Vit C, D3, Quercitin, garlic and plenty of fresh juices. One of my children developed Asthma at age 5. I had him breathe the steam of Melaleuca Oil in boiling water several times a day for 3 weeks. It cured him. He’s never had a recurrence and that was over 25 years ago. I have also cured Pneumonia in 2 of my children and twice in myself, as well as numerous sinus and bronchial infections, using Melaleuca Oil in boiling water. All of my children had Chicken Pox and from the first pox to completely being dried up was a total of 6 days. I have suffered too much and two of my children, as well, from Vaccines. They cause damage to many, and I will never willingly subject myself to another vaccine, for anything, ever again. When your health is ruined by a vaccine, you learn a very hard and fast rule!


  9. Shasta Says:

    Right from the first response, CATT was exaggerating. she (I assume she) obviously already has her agenda. I don’t believe a word she says. She obviously doesn’t know what asthma, pneumonia, or influenza is.


  10. Dr. M Says:

    Vaccines save and have saved millions of lives. Whether or not you have suffered health problems as a direct result of a vaccine notwithstanding (although, I kind of doubt it), that doesn’t mean that vaccines are evil or cause the same problems in everyone. If they did, they wouldn’t be on the market.

    And yes it definitely makes sense that a “nothing” herb like echinacea would cure a “nothing” disease like the flu. Take your antivaccinationist bs elsewhere please.


  11. CATT Says:

    If you have never tried what I have listed, then you have no room to talk or put down what others have tried and found that work for them. Such closed mindlessness is intolerable and unacceptable. Have a nice life; end of subject.


  12. anon Says:



  13. kerrin Says:

    CATT lets leave the flu vaccine alone a look at other horrible diseases. Things like polio, whooping cough, and measels. Currently people in other countries are giving their lives to try and give polio vaccines to people. We are very spoiled in this generation because we dont see these diseases wreaking havoc anymore. We have these vaccines because people were horrified because of their children coughing so much they died. My grandmother had polio. She remembers how you couldnt go swimming for fear of catching it. She, luckily had a milder case and lived, but she had friends who didnt. Her one friend was in an iron lung and in a hospital for 5 months! Our ancestors would be yelling at us and asking us what we are doing by not vaccinating. These vaccines are the answers to their prayers. But we have become spoiled because we are lucky enough to not see this type of suffering any more. I am not saying dont do your homework, i am not saying there arent bad or greedy doctors out there, and i am not saying to blindly follow and i am not saying the body doesnt have a great capacity to heal. But we have been given a gift in that we have the chance in this day and age to not have to subject our children to these horrible illness’s. If you doubt this, i would be happy to let you speak to my grandmother who at 10 years old lost her best friend to polio.


  14. Dr. Roy Says:

    Here’s a much better study, just published in NEJM:

    Overall effectiveness in children of the current quadrivalent flu vaccine was ~ 60% to prevent all flu disease, and 75% to prevent moderate-to-severe disease. These numbers jibe with what’s been seen in other studies, and are probably a fair assessment.


  15. cphickie Says:

    Here in my clinic we’ve had 12 cases of rapid-test confirmed influenza A in the last 3 weeks. None of those 12 children had received a flu vaccine this year. We have not seen a child who received the flu vaccine this year have a positive flu test. I need to figure out how to make my EMR tell me how many others were tested for the flu who fit the two other chi square categories, but I would say even with this that the trend is definitely you more likely to have flu if you haven’t had this year’s flu vaccine.


  16. Susan Says:

    I couldn’t agree with you more. The influenza vaccine works. My 3 kids got the shots and seemed to have stayed away from the misery of serious coughing and runny noses. Your observational study certainly confirms this. And your concern, I’m sure, is much appreciated by people. Kudos!


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: