NORC--Statistics & Methodology Studies

The Current Employment Statistics
This program of monthly estimates of employment and production statistics are among the country's leading economic indicators. Produced through a cooperative agreement between the Bureau of Labor Statistics (BLS) and the states, the CES survey of business establishments has experienced some disturbing revisions in the estimates in the last few years. An independent panel of experts from the American Statistical Association recommended a number of changes to the decades-old program to bring it up to current statistical standards. NORC, as a subcontractor to the state of Illinois, is part of BLS's team working to implement these changes. The team is designing a probability sample, a method for incorporating new business establishments more quickly, and improved estimation processes.
      NORC has concluded much of the preliminary research, and BLS has selected a new probability sample and is recruiting the sample in the wholesale trade industry. This new sample will be maintained in parallel with the existing sample for a period of time as a production test. NORC will help monitor the estimates produced by the new sample and suggest methodological fine-tuning as necessary. The data obtained through the production test will support the team's further research into sample rotation, seasonal adjustment, benchmarking, and variance estimation. In addition, NORC will continue to help Illinois develop estimators for smaller geographies than those required through the CES program. NORC will also extend the research using unemployment insurance databases as auxiliary data in CES estimation.

The NLSY79 16 Recall Experiment
This experiment was conducted as part of Round 16 of the National Longitudinal Survey of Youth 1979 (NLSY79). The NLSY79 survey was originally fielded with a sample of persons who were 14 to 21 years old at that time. Through the first 15 rounds of data collection, interviews were conducted every year and the questions generally covered the period since the last interview. The questions concern a range of topics, including labor force and educational experiences, health and disability, marital status, income, and program participation. With Round 17, the survey became biennial, doubling the length of the period covered by many of the questions. The Round 16 experiment tested the effects of this change in the data collection schedule.
      The analysis examined several labor force and recipiency variables—the number of jobs the respondents reported, the number of gaps between jobs, and whether they reported receiving unemployment, food stamps, or AFDC payments. The two-year recall period had little discernible effect on means and proportions for the sample as a whole. There was some decrease in the number of jobs respondents reported for the most recent year, but none of the other analyses showed much overall impact of the two-year recall period.
      However, a closer examination of the results found substantial errors in the reports covering the two-year period. The two-year period covered by the Round 16 interviews included the one-year period the respondent had already reported in Round 15. The Round 16 reports sometimes failed to reproduce the information the respondent had provided in the earlier interview. The discrepancies in reports concerning this overlapping period were especially marked among respondents who had complicated job and recipiency histories. On the average, such respondents reported fewer jobs and less recipiency in their Round 16 interview than they had in Round 15. The limited overall impact of the two-year recall period thus appears to reflect the stable circumstances of most of the respondents. It is easy for respondents to remember their jobs if they have not changed jobs in many years; it is far more difficult for them to remember their jobs if they change jobs frequently.

Sample Design For a Statistically Valid Evaluation of Accuracy and Completeness of an Establishment's OSHA Mandated Employer Records
This project reviews previous pilot work evaluating the accuracy of OSHA record keeping and will recommend a statistically valid and cost-effective evaluation scheme and sampling strategy to be carried out by OSHA field staff in the course of their regular cycle of audits.
      The specifications for the sampling of establishments to be audited and for sampling employee records within establishments is to be based on a series of power analyses and frame list specifications provided by the client. A cost benefit analysis evaluating alternative solutions at each stage of the sampling plan will also be provided.

Sampling for an Evaluation of the Illinois Guardianship Waiver Demonstration
This project is developing and implementing a sampling plan to provide a sample of children and guardians for the evaluation of a new subsidized guardianship program in the state of Illinois. Respondents are selected randomly within strata defined by geographic location (relative care or foster care), with probability of selection adjusted by the reciprocal of the age of the eldest child to limit the selection of older children and maximize the time the children will remain in the study (children "age out" at 18). The sample will also fill an experimental design in which selected guardian/child combinations are randomly assigned to an experimental group, a control group, or an administrative follow-up group (which is to be evaluated only for a federally required assessment of cost factors related to the guardianship program).

Research on Response Errors in Survey Questions of Children's Immunization
Sponsored by the National Center for Health Statistics, several surveys were conducted to monitor progress toward meeting the President's goal of increasing the proportion of children who have received all their recommended vaccinations. Unfortunately, parents have difficulty recalling the details of their children's vaccination history and may be embarrassed to admit that a child has not received all of the right shots. The project sought to pinpoint the sources of error in parents' reports about their children's vaccinations and to explore methods for improving the accuracy of those reports.
      NORC staff designed and conducted a series of laboratory and field experiments to examine the cognitive processes associated with parents' answering survey questions on immunization.
      NORC began by conducting cognitive interviews with a convenience sample of 24 parents. Two studies conducted at a pediatric clinic in Oakland, CA, examined the information parents had encoded immediately after the child was vaccinated (encoding). In these studies parents either had to name all the vaccinations their child received (recall) or give a yes/no response when the vaccine names were read to them (recognition). A comparison between recall and recognition performance indicates whether encoding problems or retrieval failure may account for parents' difficulty in reporting on their child's vaccinations. Two studies conducted at a pediatric clinic in the Chicago area tested different methods of asking parents questions about immunizations (the memory-aid and proxy measures studies). In the memory-aid study, some respondents viewed a show card listing the vaccines or filled out a medical history calendar before answering the survey questions. In the proxy measures study, respondents were asked indirect questions about events that may be related to the child's vaccination status (doctor's visits) and were given either a self-or interviewer-administered questionnaire. In all the studies, the survey data were matched to vaccination records to assess the accuracy of the parents' reports.
      The results indicated that many parents are unclear about which vaccinations their children received, even when questioned immediately after leaving the doctor's office. Parents' accuracy in reporting is directly related to the number of vaccines the child received. Reporting accuracy on the recognition test was no better than for recall, which suggests that parents have difficulty reporting not because of retrieval difficulties but because they never encoded the information. The memory-aid and proxy measures studies showed that the use of calendars, show cards, and different types of questions and survey administration did not significantly improve reporting. However, parents who know more about vaccines do seem to report more accurately. Our findings suggest that when parents have difficulty reporting their child's vaccination history there is little that can be done to help them remember the information.
     The lack of improvement in reporting accuracy even with self-administered versions of the questionnaire suggests that parents are not manipulating their reports simply to hide embarrassment from an interviewer. Rather, they are simply guessing when they do not know the answers. The finding that more knowledgeable parents reported more accurately suggests that the key to collecting better data on the vaccination status of children is educating parents about vaccines.

Comparative Study of Cognitive Laboratory Interviewing Techniques
The National Center for Health Statistics is sponsoring this study to examine the reliability of results from cognitive interviews conducted by different interviewers and at different organizations. NORC conducted a set of cognitive interviews on a questionnaire already tested by NCHS cognitive interviewers. Four NORC interviewers each completed ten taped interviews with paid volunteers. During the interviews, they recorded problems by checking a "problem" box and recording notes on unfamiliar terminology, the logic of the questions, response categories, etc.
      To code the interviews, each interviewer paired up with another interviewer. The interviewers reviewed their own ten interviews and the ten completed by their partner. For both their own and their partner's, they reviewed five interviews by consulting only the notes taken on the questionnaires and the other five by listening to the tapes and not viewing the notes. The second interviewer also coded the problems that arose during the same interviews by coding the same five interviews from notes only and the other five interviews from tapes only.
      The study's design allows for testing of both inter-organization and inter-coder reliability. Since the same questionnaire has been cognitively tested by both NCHS and NORC, it will be possible to examine whether interviewers trained by different organizations arrive at the same findings. Similarly, since the same interview session was coded by both the interviewer and another trained observer, it will be possible to test whether interviewers within an organization agree on their findings.

Task Orders for the Department of Transportation
NORC has a task order to prepare two reports, one on Reducing Nonresponse in Travel Surveys and the other a Primer for Panel Surveys in the transportation field. NORC staff worked with a review panel comprised of Project Officers from the Federal Highway Administration, Metropolitan Planning Officers in Chicago and Washington, and researchers and experts in the transportation survey research field. Reducing Nonresponse in Transportation Surveys was written for designers, analysts, and sponsors of household travel surveys, and others who collect, report on, or interpret travel survey data. This report provides a set of guidelines for measuring and reporting nonresponse in household travel surveys and for reducing the level and impact of nonresponse. The report first focuses on measuring and reporting nonresponse, and then moves on to describe the potential consequences of nonresponse in household travel surveys and recommend methods for reducing the level of nonresponse. Finally, the report describes statistical methods for reducing the impact of nonresponse on survey estimates.
      The Primer for Panel Surveys introduces the use of panel designs in surveys of travel behavior. The report (1) highlights the differences between cross-sectional and panel approaches to the study of travel behavior, (2) discusses the limitations of cross-sectional and panel data, (3) identifies situations where panel data are preferable, and (4) provides guidelines for designing and maintaining a panel survey. The report contains a number of recommendations for conducting and using panel designs in travel surveys.

Longitudinal Study of American Youth
Sponsored by the Chicago Academy of Sciences, the objective of the project was to perform an item response theoretic (IRT) analysis of achievement data from the Longitudinal Study of American Youth (LSAY) to obtain scale score estimates of science and mathematics attainment. The achievement data consisted of item responses from repeated administrations of a multiple-form assessment instrument to two nationally representative samples of middle and high school students. Previous IRT analyses of the data were conducted at a time when methods for obtaining comparable estimates of ability across test forms and repeated administrations were not completely understood, even by specialists in the field. Since that time, the issue has received considerable attention in the literature, and new methods for equating test forms across multiple groups have emerged. This project applied those methods to the data from the LSAY survey.
      A multiple-group IRT model for equating nonequivalent groups (Bock and Zimowski, 1997) was fit to the item response data using the BILOG-MG program of Zimowski, Muraki, Mislevy, and Bock (1996). The procedure places the items from all forms on a common scale by jointly estimating the item parameters and the latent ability distributions of the nonequivalent groups, which, in this case, correspond to the groups of students completing each form within each year and cohort of the survey. The item parameter estimates obtained in that calibration were used to produce scale score estimates of individual achievement for all measurement periods in the survey. The scores will be added to the LSAY public- use database.

ANSPACK
NORC staff spent a great deal of time this year developing a working prototype of NORCSuite Sampling Tools, which are a combination of a relational database and software tools to select and maintain NORC's sampling frames and project samples. Initially designed for area probability samples, particularly NORC's exclusive national frame, the system will eventually be able to handle most types of samples that NORC manages. The tools automate many of the sample selection and maintenance functions, enabling NORC to maintain the highest quality sampling standards with greater efficiency. A working prototype of the system was developed with the basic functions for selecting a sample of housing units from a list frame of addresses. We are beginning to migrate the data to a larger, more powerful relational database system that can handle the complete set of our sampling frames.

      Back to Studies

Home  |   About NORC  |   What's New  |   Annual Report  |   Search  |   FAQs
Career & Employment Opportunities  |   The General Social Survey
Library & Publications  |   Site Map  |   Studies  |   Contact Us  |   Links