When vaccination surveys overestimate coverage, public health programs may misallocate resources, underestimate disease risk, and make policy decisions based on biased data.

In a study recently published in the Journal of Survey Statistics & Methodology, my co-authors and I applied a total survey error (TSE) framework to the National Immunization Survey–Flu (NIS-Flu) to measure bias in this annual survey of U.S. parents about the influenza vaccination status of their children. Our findings create a clear picture of not just whether bias exists, but how much, where it originates, and what can be done about it.

We applied TSE to the NIS-Flu to quantify bias from sampling frame noncoverage, nonresponse, and reporting errors. This approach provided a more complete picture of the root causes of bias in the survey estimates.

Public health programs and researchers need accurate measures of vaccination rates to guide immunization program efforts and influence policy decisions. When a survey providing these measures contains potential bias, they need sound, quantitative measures to understand the direction, magnitude, and sources of error.

Public health surveys face several challenges that can produce errors. The accumulation of multiple forms of sampling and nonsampling error can affect data quality.

• Sampling frames must keep up with changing technology use by potential respondents.
• Respondent participation is more difficult than ever to secure, increasing the potential for nonresponse bias.
• Surveys based on respondent recall can suffer from measurement error, especially when surveys ask about distant experiences, rely on specific dates, or contain confusing language.

All of these issues can create errors in survey estimates. Having a disciplined way to quantify where error originates and how much each error contributes to overall survey bias is essential to understanding which concerns most need to be addressed.

Earlier comparisons of NIS-Flu vaccination rates to auxiliary sources raised concerns about potential overestimation but couldn’t tell data users how large the bias was, which components were driving it, or how to target improvements. Our work applied the TSE approach to three past flu seasons, using simulation to quantify total and component bias in end-of-season estimates among children.

Unlike other NIS programs that rely on provider records, NIS-Flu estimates are based on parental report, a key difference that elevates the importance of measuring reporting error. To strengthen measurement-error inputs, we leveraged linkages between NIS-Child, NIS-Teen, and state Immunization Information Systems (IIS), which provide more complete influenza vaccination data than provider histories alone. These novel approaches let us measure reporting error with greater specificity than previous studies.

• Total bias is real and quantifiable. Across the seasons studied, total bias in end-of-season child vaccination estimates was about 6 to 9 percentage points.
• Coverage error is small. Coverage error arises when the sampling frame does not include all households in the target population. In the NIS-Flu, these uncovered groups are very small.
• Nonresponse bias is a risk but mitigated. About 82 to 90 percent of sampled children resided in nonresponding households across seasons. Nevertheless, the model estimated nonresponse bias to be relatively small in final-weighted estimates, about 0.3 to 1.2 percentage points. This underscores that weighting effectively mitigates nonresponse bias and that understanding nonresponse bias requires more than looking at response rates alone.
• Measurement error is significant and requires special attention in the NIS-Flu. Because NIS-Flu relies on parental recall, measurement error differs substantially from provider-record-based vaccine surveys. The model estimated positive measurement error—meaning NIS-Flu overestimates influenza vaccination rates—of about 5.5 to 7 percentage points. Measurement error was positively correlated with age, suggesting recall may be a larger problem for older children.

• Know which lever to pull to address bias. TSE diagnostics help decide whether to prioritize sampling frame coverage, nonresponse reduction (tailored outreach, incentives, follow-up), or measurement improvements (cognitive testing, recall aids, embedded validation).
• Strengthen recall support where measurement error is likely. For age groups with higher reporting error, refine question wording, add concrete recall cues (month, setting, co-occurring events), and expand opportunities to validate respondent reporting.
• Report the uncertainty. Publish total and component-level estimates of bias with key assumptions. This aids data interpretation, may reconcile differences across surveys and other data sources, and builds trust in the results.

• Use error models like TSE to assess survey fitness-for-use. Understanding bias sources and magnitude helps users interpret estimates appropriately and adjust conclusions when needed.
• Calibrate program decisions using bias-aware estimates. Public health leaders using NIS-Flu data—to monitor coverage or estimate disease burden averted by vaccination—should account for documented total bias.
• Focus on where error reduction will improve accuracy the most. Test nonresponse strategies for hard-to-reach groups and refine recall protocols where measurement error is highest. The TSE model can identify which interventions will have the greatest impact.
• Use data linkage to strengthen accuracy. Incorporating provider data or information from external sources intended for similar measures remains an important tool for validating and improving survey estimates.
• Treat transparency as a quality standard. Reporting error components, assumptions, and uncertainty should be standard practice.

We applied the TSE framework to NIS-Flu data for the 2015–16 through 2020-21 influenza seasons among children ages 6 months to 17 years. The model incorporated three error sources:

• Coverage: noncovered populations included no-phone and landline-only households
• Nonresponse: the proportion of eligible children residing in nonrespondent households
• Measurement: parental reporting error based on comparisons to provider and IIS records

We obtained estimated influenza vaccination rates for noncovered and nonresponding groups from National Health Interview Survey (NHIS) public-use data files. We used Monte Carlo simulation with 10,000 repetitions to calculate average bias and 95 percent interval estimates.

Find more details, assumptions, and limitations in the report published on JSSaM.