English
The synthesis team will encourage country teams to use an appropriate mix of qualitative and quantitative methods, recognizing that the most rigorous methods are often not feasible and in some cases not even desirable. We propose to use the first workshop to discuss which approaches are most suitable under what circumstances. It would be futile to insist in all cases on the use of gold standard methods such as Randomized Controlled Trials (RCTs), but it would also be wrong to fail to use rigorous methods where that is feasible. In that spirit the country teams will be asked to justify their choice of methods. In the feedback on the proposals and in the workshop discussion the synthesis team will ensure that the evaluation will not only assess impact but also, wherever possible, use data on intermediate outcomes to test theories of change. The evaluation should establish not only what happened as a result of the interventions, but also why. For example, in the case of partner organizations the analysis will focus not just on changes in the organizations capacity to achieve development objectives but also on the role of intermediate capacities (Figure 1 in the Call) to bring this about. The final choice of indicators will be decided after intensive consultation with the country teams. In some cases it may differ from the list of priority areas, e.g. in order to ensure that the contribution of MFS II to the poverty target of MDG1 can be assessed. With inputs from the country teams we will develop a transparent scoring system with scores reflecting (a) the complexity of the intervention (for instance: single outcome versus multi-dimensional outcomes) and (b) the rigor of the evaluation conducted by the country team. The synthesis report will address the five research questions in two ways. First, by combining the findings of the country studies in a descriptive way. This in itself is a massive task. It will differ from a conventional synthesis report in using the scoring system rather than giving equal weights to all studies. Secondly, we will use (at least for the subset of indicators and interventions for which this makes sense) regression analysis on the pooled data sets. Obviously, country teams can and will aggregate findings for different projects in the same country. The synthesis team can move beyond this by running regressions on the full dataset to explain changes in the agreed indicators in terms of interventions (of various types) and various confounding variables (such as changes in the macroeconomic environment). The advantage, just as in a meta-study of drugs trials, is that results that are not statistically significant at the country level may well be so in the larger sample. (Although with 55 very heterogeneous projects the degrees of freedom are very limited.) This approach would fully exploit the massive data collection efforts for the MFS II evaluation. It would also give much richer answers to the questions of what works and why?
