Assessing grants that support maternal and child health.
To achieve its vision of an America where all mothers, children and families are prospering and reaching their fullest potential, The Health Resources and Services Administration’s (HRSA) Maternal and Child Health Bureau (MCHB) relies upon evidence-based strategies to implement programs and monitor their effectiveness. Their efforts including supporting states and jurisdictions through Title V Maternal and Child Health Block Grants. They assess the performance of those grants using the Discretionary Grant Information System (DGIS). The goal of this project was to update the MCHB data collection instruments to more fully and accurately assess whether MCHB-funded discretionary grant programs are making measurable progress toward their goals.
MCHB contracted with NORC to assess the validity, reliability, and utility of the DGIS performance measures and data. Between September 2020 and September 2021, we conducted a comprehensive, mixed-methods evaluation of the relevancy, appropriateness, and validity of a subset of the DGIS performance measures and developed recommendations for revisions, enhancements, and additions. Our approach to assessing the Core, Capacity Building, and Population Domain measures and Financial and Demographic forms was guided by the National Quality Forum (NQF)’s performance measure evaluation criteria:
- usability and current use
- importance to measure and report (e.g., prevalence, evidence base)
- scientific acceptability of measure properties (e.g., validity)
- feasibility for grantees to report
- comparison to related and competing measures
To answer questions about the importance and feasibility of the measures, we reviewed the total number of grantees reporting each measure, determined the frequency and distribution of each measure type and alignment with DGIS program goals, assessed variability in performance rates among select measures, examined patterns of missing data (e.g., low response rates), and assessed the form logic to examine whether the forms collect data at the right level and follow best practices.
To address questions about the current usability, importance, and scientific acceptability of the DGIS measure set and to identify appropriate related or competing measures, we reviewed background documents provided by MCHB, conducted ten discussions with MCHB stakeholders, and participated in seven virtual internal workgroup meetings with MCHB staff.
NORC generated a list of recommendations for improving the DGIS measures, ranging from short-term solutions to longer-term strategic planning initiatives that may involve an extensive redesign of the measure set. In the short-term, NORC recommended that MCHB focus on improving the reliability and validity of existing measures and reducing burden where possible through revisions to the 2022 OMB package resubmission:
- Develop DGIS-focused trainings and ongoing technical assistance (TA) for staff and grantees to provide an orientation to the measures and teach them how to submit data.
- Provide standardized definitions and instructional guidance for confusing measures and terms.
- Provide more informative response categories for specified measures.
- Develop and implement a process for validating the DGIS measures.
In the long-term, NORC recommended that MCHB engage in strategic planning with leadership and staff to overhaul the DGIS measure set to make them more useful, valid, and less burdensome to stakeholders. These steps were also aimed at improving stakeholder buy-in:
- Clarify the overall purpose of the DGIS measure set.
- Reduce the number of DGIS measures.
- Develop new measures that better capture grantee activities and MCHB core values.
- Further explore and pilot alternate structures.
- Create a more user-friendly data system so the DGIS data are useable and accessible.