As the Delta variant takes over the world, researchers are tracking how well vaccines protect against it – and getting different answers.
In the UK, researchers reported in May that two doses of the Pfizer BioNTech vaccine had 88 percent effectiveness and protected against symptomatic disease caused by Delta. A June study from Scotland concluded that the vaccine was 79 percent effective against the variant. On Saturday, a team of researchers in Canada estimated the effectiveness at 87 percent.
And on Monday, the Israeli Ministry of Health announced that the effectiveness of the Pfizer BioNTech vaccine against all coronavirus infections was 64 percent, up from about 95 percent in May before the Delta variant began its rise to near complete dominance in Israel.
While the range of these numbers may seem confusing, vaccine experts should expect them because it is difficult for a single study to pinpoint the effectiveness of a vaccine.
“We just need to put everything together as little pieces of the puzzle and not put too much emphasis on one number,” said Natalie Dean, biostatistician at Emory University.
In clinical studies, it is (relatively) easy to measure how well vaccines are working. Researchers randomly assign thousands of volunteers to receive either a vaccine or a placebo. If the vaccinated group is at lower risk of disease, scientists can be sure that the vaccine will protect them.
But once vaccines hit the real world, it becomes much more difficult to measure their effectiveness. Scientists can no longer control who gets a vaccine and who doesn’t. Comparing a group of vaccinated to a group of unvaccinated, there could be other differences between the two groups that affect their risk of disease.
For example, it is possible that people who choose not to be vaccinated are more likely to find themselves in situations where they could be exposed to the virus. On the other hand, older people are more likely to be vaccinated, but also have a harder time fending off an aggressive variant. Or an outbreak can hit part of a country where most of the people are vaccinated, leaving undervaccinated areas unscathed.
One way to rule out these alternative explanations is to compare every vaccinated person in a study to someone else who did not receive the vaccine. Researchers often go to great lengths to find an unvaccinated partner by looking for people of the same age and health status. You can even bring people together within the same neighborhood.
“It takes a tremendous amount of effort,” said March Lipsitch, an epidemiologist at Harvard TH Chan School of Health.
For its new study, the Israeli Ministry of Health didn’t go as far as to rule out other factors. “I fear the current Israeli MoH analysis cannot be used to safely assess it one way or another,” wrote Uri Shalit, a senior lecturer at the Technion – Israel Institute of Technology, on Twitter.
Israel’s numbers could also be different based on who is being tested. Much of the country is vaccinated. During local outbreaks of new infections, the government requires testing for anyone – symptoms or not – who has come into contact with a person diagnosed with Covid-19. In other countries it is more common for people to get tested because they already feel sick. This could mean Israel is discovering more asymptomatic cases in vaccinated people than anywhere else, lowering the reported effectiveness rate.
Fortunately, all previous studies agree that most Covid-19 vaccines have been very effective in keeping people out of the hospital and generally protecting them from the Delta variant. The Israeli Ministry of Health estimates that the Pfizer BioNTech vaccine is about 93 percent effective in preventing serious illness and hospitalization.
“Their overall impact is consistent: the protection against serious illness remains very high,” said Naor Bar-Zeev, associate professor at the Johns Hopkins Bloomberg School of Public Health.
Because efficacy studies are so difficult, it will take more work to determine just how great Delta’s threat to vaccines is. Dr. Lipsitch said studies are needed from more countries.
“If there are five studies with one result and one study with another, I think it can be concluded that the five are more likely to be correct than the one,” said Dr. Lipsitch.