Declining response rates, increased costs, and decreasing resources are growing challenges to federal survey programs.

The National Center for Science and Engineering Statistics (NCSES) within the U.S. National Science Foundation (NSF) conducts three biennial education and workforce surveys, the National Survey of College Graduates (NSCG), the National Training, Education, and Workforce Survey (NTEWS), and the Survey of Doctorate Recipient (SDR), which have been experiencing several challenges, including declining response rates, rising data-collection costs, and increasing operational constraints. Simultaneously, NCSES seeks to transition from a survey-only approach to an integrated framework that blends alternative data sources that could support new sampling frames, augment or replace survey items, and support data estimation.

NORC and its partners are investigating the feasibility of implementing data modernization efforts, such as integrating workforce surveys and establishing data linkages to alternative data sources.

Through a detailed case study, NORC is working with its partners, American Institutes for Research and The George Washington University, to integrate the three NCSES education and workforce surveys. The case study examines potential options for integrating these surveys using best practices and evaluates the potential of linking alternative data sources—including administrative records and third-party datasets—to supplement or replace traditional survey data for sampling, data collection, and/or estimation purposes. 

• First, we assessed whether the data collected by the integrated survey would continue to meet the data needs of NCSES, the federal statistical community, and its stakeholders. We accomplished this by conducting stakeholder interviews and evaluating the data priorities and needs, which resulted in a memo titled, “Understanding NCSES Data Needs: A Comparison of NTEWS, NSCG, and SDR.”
• We developed two survey operations plans to document all necessary steps in a life cycle: one that uses the American Community Survey (ACS) as a sample frame and another that considers the use of alternate sample frames.
• We identified the implications of an integrated survey on costs, staffing resources, operational efficiencies, data quality, and data dissemination timeliness. NORC analyzed the pros and cons of several survey options for NCSES, providing the agency with critical information to select the most feasible options, which led to the creation of the two survey operations plans.
• NORC assessed whether alternative data sources could be used to provide information currently provided by NSCG, NTEWs, and SDR surveys at a similar quality level and similar or lower costs than current operations. To further explore the potential use of alternate data for sampling and/or data analysis, NORC is producing a set of memos exploring and evaluating available data sources that could be used for sampling or to replace survey questions.
• We are assessing the impact on trends in key estimates for NCSES’s congressionally mandated reports, and identifying the data element(s) for which no appropriate alternative data sources exist or for which critical information would be lost through an integrated survey, which may justify the need to retain the content as-is in separate survey data collection. NORC has helped NCSES identify the core survey items from the three NCSES workforce survey by creating an inventory of all the survey items and evaluating each item’s value and quality. From there, NORC and NCSES developed an integrated survey questionnaire that combines elements from the NSCG, NTEWS, and SDR surveys, as well as new workforce topics in emerging technologies and R&D sectors.
• NORC will also conduct a series of cognitive interviews to test the newly developed integrated survey instrument and refine the questionnaire content in English and Spanish. The cognitive interviewing process will also ensure that the questionnaire will produce high quality data for NCSES’s congressionally mandated reports.