State-specific Design Parameters for Designing Better Evaluation Studies

This IES-funded project seeks to use State Longitudinal Data Systems (SLDS) to estimate variance components and intraclass correlations for use in multi-level study designs in education.  Having plausible values of intraclass correlations is essential to planning how many schools and students are needed for an evaluation design.  This project is a continuation of efforts by Hedges and Hedberg to provide guidance on these parameters to the research community. Whereas their previous work used national probability samples to estimate these parameters, the current project uses state data.

This project advances previous work in this field for three reasons. Use of SLDS allows for unprecedented ability to estimate variance components and intraclass correlations at several levels of analysis inclusive of district, school, and classrooms. In addition, because state data systems are longitudinal, we will be able to estimate variance structures not only of cross-sectional outcomes, but also the gains student make from year to year.  Third, access to longitudinal data from states allows for estimates of year to year gains in effect size units, providing the research community with a better sense of the effect size of a typical single year increase in achievement for specific grades and subjects. This project will produce reports to IES, academic papers, and for participating states, workshops and technical assistance in their evaluation designs.

This project has already produced new methods of calculating standard errors for intraclass correlations and a codebase to enact these calculations.

Hedges and Hedberg are currently writing their preliminary results. They are finding that two-level intraclass correlations (students nested within schools) vary across the participating states.  However, findings from three-level models (students nested within schools nested within districts) indicate that this variation may be related to district structures and that within-district intraclass correlations are more consistent across states.