High-quality survey data are increasingly difficult to collect. Response rates are falling, and organizations that conduct surveys or rely on survey data must contend with falsification. Dealing with these data quality threats across the different modes (e.g., phone, in-person, web), geographies, and cultures surveys are administered can be daunting. They increase the effort and costs of data collection and undermine researchers’ ability to generate timely insights. Now, there is a new challenge: the proliferation of artificial intelligence (AI).

My colleague Joshua Lerner recently described the benefits that AI can bring to survey methodology, and NORC has shown how generative AI might improve the quality of survey interviews. However, AI also has potential downsides. For instance, large language models (LLMs) and AI assistants such as ChatGPT can be used to generate responses to open-ended questions on surveys.

When used to respond to surveys in place of people, AI can pose threats to the quality of survey data collected and any insights derived from the data by reducing—or even completely undermining—the extent to which survey responses represent opinions, beliefs, and behaviors of humans in the sample. Evidence suggests that this is not merely a theoretical possibility but an empirical reality (see Related Presentations below for NORC’s relevant research). How can methodologists and researchers respond to this potential threat to survey data quality?

To address the data quality threats that AI-generated responses pose to surveys, NORC has developed a tool that predicts whether responses to open-ended survey questions were generated with AI. Our tool is very effective, with more than 99 percent precision and recall when determining whether open-ended responses were written by humans or LLM-powered AI assistants.

Our AI detection tool performs so well because we designed it specifically for surveys, thus avoiding many of the pitfalls that more generalist, one-size-fits-all AI detectors are shown to have. We first created training data by asking AmeriSpeak^® panelists and several of the latest LLMs to respond to the same survey questions. We varied the types of open-ended questions to include those that elicit free-form responses and others that tend to get structured lists (such as when asked to name the top issues facing the United States). 

Once we collected the responses from humans and LLMs, we used natural language processing (NLP) and machine learning to train a classifier that predicts whether a given response originated from an LLM or a person.

We evaluated the detector’s performance on surveys in different domains with different study populations and on questions that required very different types of writing styles, and our tool continued to exhibit impressive performance, especially when compared to other currently available AI detection tools, whose accuracy tended to be much worse, hovering around the 50 to 75 percent range. A key benefit of our tool is that it is based on a more general methodological framework that could be easily tailored for new surveys.

Our AI response detector further strengthens the rigorous quality control procedures that NORC puts in place for our data collection efforts. The output from this tool is one additional piece of information included in the holistic decision-making process that NORC methodologists and researchers make about whether survey responses should be retained for analysis.

This human-in-the-loop approach ensures that NORC staff can monitor the performance of the AI detector for any issues that might arise and that the tool is integrated into NORC’s extensive quality assurance processes tailored to survey research. (For example, research has shown that the use of AI detectors that are too general to be fit-for-purpose can actually systematically skew data quality.)

Having access to reliable data and insights is of paramount importance to decision-makers and the public. With our AI-detection tool, NORC surveys have a quality control solution that can scale from small projects to the largest efforts. Even at the largest of scales, our tool and the quality control processes within which it is used are designed to protect the voices of genuine respondents and the integrity of decisions made with their input. NORC clients can rest assured that their survey data will be of the highest quality.

There is no shortage of challenges to survey data quality, and NORC has—for over 80 years—been at the forefront of developing innovative solutions. By using AI to help combat threats to survey data quality that AI itself has brought, NORC’s AI detection tool for survey responses is the next step in this history of innovation.