Skip to main content

Evaluation of HRSA’s Maternal and Child Health Bureau Discretionary Performance Measures

Mother cradles her 1-month old baby in her arms
Developing accurate measures of whether HRSA’s grants are having an impact
  • Client
    Health Resources and Services Administration
  • Dates
    September 2020 - September 2021

Problem

HRSA/MCHB rely on strong performance measures to maximize the value of services to mothers and families.

To achieve its vision of an America where all mothers, children, and families prosper and reach their fullest potential, the Maternal and Child Health Bureau (MCHB) of the Health Resources and Services Administration (HRSA) relies upon evidence-based strategies to implement programs and monitor their effectiveness. Their efforts include supporting states and jurisdictions through the Title V Maternal and Child Health Block Grant Program. They assess the performance of those grants using the Discretionary Grant Information System (DGIS).

Solution

NORC scientists helped MCHB ensure that their measures addressed critical issues, were feasible to collect and use, and were valuable to the agency and its grantees.

MCHB contracted with NORC at the University of Chicago to assess the validity, reliability, and utility of the DGIS performance measures and data. Between September 2020 and September 2021, we conducted a comprehensive, mixed-methods evaluation of the relevancy, appropriateness, and validity of a subset of the DGIS performance measures and developed recommendations for revisions, enhancements, and additions. Our approach to assessing the Core, Capacity Building, and Population Domain measures and Financial and Demographic forms was guided by the National Quality Forum (NQF) performance-measure evaluation criteria:

  • usability and current use
  • importance to measure and report (e.g., prevalence, evidence base)
  • scientific acceptability of measure properties (e.g., validity)
  • feasibility for grantees to report
  • comparison to related and competing measures
     

To answer questions about the importance and feasibility of the measures, we:

  • reviewed the total number of grantees reporting each measure
  • determined the frequency and distribution of each measure type and its alignment with DGIS program goals
  • assessed variability in performance rates among select measures
  • examined patterns of missing data (e.g., low response rates)
  • assessed the form logic to examine whether the forms collect data at the right level and follow best practices
     

To address questions about the current usability, importance, and scientific acceptability of the DGIS measure set and to identify related or competing measures, we reviewed background documents provided by MCHB, conducted 10 discussions with MCHB stakeholders, and participated in seven virtual internal workgroup meetings with MCHB staff.

Result

MCHB received actionable suggestions on improving the set of measures, supporting grantees’ collection and use of them, and longer-term strategic refinements.

NORC generated a list of recommendations for improving the DGIS measures, ranging from short-term solutions to longer-term strategic planning initiatives that may involve an extensive redesign of the measure set. In the short-term, NORC recommended that MCHB focus on improving the reliability and validity of existing measures and reducing burden where possible through these revisions to the 2022 OMB package resubmission:

  • develop DGIS-focused training programs and ongoing technical assistance (TA) for staff and grantees to provide an orientation to the measures and teach them how to submit data
  • provide standardized definitions and instructional guidance for confusing measures and terms
  • provide more informative response categories for specified measures
  • develop and implement a process for validating the DGIS measures

In the long-term, NORC recommended that MCHB engage in strategic planning with leadership and staff to overhaul the DGIS measure set to make them more useful, valid, and less burdensome to stakeholders. These steps were also aimed at improving stakeholder buy-in:

  • clarify the overall purpose of the DGIS measure set.
  • reduce the number of DGIS measures.
  • develop new measures that better capture grantee activities and MCHB core values.
  • further explore and pilot alternate structures.
  • create a more user-friendly data system so the DGIS data are usable and accessible.

Project Leads

Explore NORC Health Projects

YMCA of the USA Water Safety Awareness Evaluation

An evaluation of the Y-USA’s drowning prevention program to enhance water safety knowledge among children

Client:

YMCA of the USA

Workforce Challenges & Technology Adoption for Health Information Professionals

Examining the current state of the health information workforce and the use of artificial intelligence

Client:

American Health Information Management Association