The Global Development Network, with Department for International Development funding, undertook a comprehensive 5-year program in 2008 to increase the capacity of 15 policy research organizations and research-advocacy organizations located in as many countries in five regions. The goal of the program is for organizations to complete stronger analytic work on the effectiveness of government budgeting and spending and to be more effective in their related communications and advocacy work.
NORC was contracted to carry out an impact evaluation covering two areas: the project’s impact on (a) the quality of policy research produced by grantee/partner institutions, and (b) through opinion research, an understanding of grantees’ perceived effectiveness in the policy arena. The evaluation employs a before-and-after (reflexive) design, with data gathered at the baseline (summer 2009) and at two future points. NORC designed the survey instruments, is collecting the data, and will complete the analysis.
All 15 project partner institutions are included in the evaluation. Data on the effectiveness of grantee institutions on the policy process was sought in each country from 20-40 policy makers, research users, and stakeholders through an online, email, or mail survey. To measure the quality of policy research, six reports nominated by each grantee were reviewed and scored using a standard form by outside reviewers using criteria typical for international journals. Information on the capabilities of each organization was collected through an online institution survey to be able to control for organizational differences and changes over time. The baseline report was submitted in November 2009.
In 2010 the mid-term review was completed. After controlling for other factors, the variable indicating that the report was prepared after the start of the program is positive and statistically significant in all three models. Overall report scores increased by about 7 points over the observation period, or about 12 percent of the baseline value. This is a substantial accomplishment. Regarding the perceptions of the policy commuity about these organizations performance, overall there was a positive but small change in policy communities’ perceptions of the organizations' performance as sources of useful information, research, and recommendations and in having a positive impact on public policy program administration. Additionally there was a significant change in the perceived impact they are having on government accountability for expenditure quality.