Evaluating a Web Mode for the American Time Use Survey

From above crop female freelancer typing on laptop keyboard while sitting at table and working on remote project at home

Problem

To explore the feasibility of a web option for the American Time Use Survey, the Bureau of Labor Statistics needed to understand how web and phone data compare.

The Bureau of Labor Statistics (BLS) conducts the American Time Use Survey (ATUS) to collect nationally representative data on how, where, and with whom Americans ages 15 and older spend their time. ATUS respondents complete a time diary, reporting their activities for a 24-hour period from 4am the previous day to 4am of the current day.

Since 2003, the ATUS has been conducted by telephone with live interviewers. To modernize the ATUS and offer an alternative method for reporting activities, BLS developed a self-administered web diary so the ATUS could move to a mixed-mode design. But BLS needed to understand how web and phone data differ, and whether respondents could report their activities accurately and completely without an interviewer’s help.

Solution

Across two studies, NORC tested the web diary against phone interviews to measure differences between the modes.

NORC helped BLS refine and test the web diary in two phases. In 2022, NORC used a within-subjects design, collecting both web and phone diaries from 45 participants—who completed the web diary while an interviewer observed—and comparing each person’s two diaries to isolate differences between the modes.

Building on that first test, BLS revised the diary and in 2023 engaged NORC for a larger pilot examining how people report activities on their own, without an interviewer. Drawing on NORC’s AmeriSpeak^® panel, the team collected 790 self-administered web diaries and conducted debriefing interviews with 60 of those respondents. Respondents used the diary on a phone, desktop, or laptop to recall and enter their activities, selecting from a searchable, browsable lexicon or typing verbatim responses, and adding details such as each activity’s duration, location, and whom they were with.

NORC analyzed the pilot diaries and debriefing data and benchmarked them against 2022 ATUS phone data to learn how respondents interpreted the task, where reporting errors arose, and how the diary could be improved.

Result

The pilot gave BLS an evidence base for refining the web diary and evaluating the potential role of a web mode for the survey.

In the pilot, respondents were generally able to complete the web diary on their own, logging their activities and the accompanying details without an interviewer’s prompts.

Compared with 2022 ATUS phone data, web respondents reported a similar number of activities but tended to record longer durations and more unaccounted-for gaps; those using mobile phones reported fewer activities than desktop or laptop users. The web data also showed some reporting errors relative to phone—omitted activities (underreporting), incorrect activities (misreporting), and extra activities (overreporting). Together, these findings pointed to concrete design fixes, such as adding an illustrative example diary, improving the mobile experience, and simplifying how respondents enter activity durations, and gave BLS an empirical basis for evaluating whether and how a web mode could be incorporated into the ATUS.