Intersections between Criteria

by Sep 28, 2017

Lately I’m thinking a lot about Evaluation Criteria (in fact, we are presenting a session in AEA2017 about expanding the DAC catalogue), and among my readings, Caroline Heider‘s #Whatworks series has specifically let me pondering.

She basically invites us to Rethink evaluation with a series of posts, and this is what resonated more with me when reading each of them:

  • Is Relevance Still Relevant?, where she makes a review of the first DAC criteria, suggesting analyzing relevance the way we do it presently does not reflect complexity, as relevance at community level may give the (wrong) impression that all its members have the same needs, and at authorities’ level assumes that national and local priorities are identical (so her question is: whose needs or priorities are taken into account?). Also, since donors’, governments’ and agencies’ policies are in general defined in very broad terms, it is relatively easy to conclude that any particular program complies with them.
  • Agility and Responsiveness are Key to Success claims how the pantheon of evaluation criteria – relevance, effectiveness, efficiency, impact, and sustainability – does not address the question of whether timely and responsive course-corrections were made when needed, therefore she suggests considering including Timeliness as a criterion.
  • Efficiency, Efficiency, Efficiency refers to how this criterion is less used, conclusions after undertaking the calculations are often not suitable for decision-making and that it addresses -at best, economic costs (economic efficiency) but hidden costs of social and environmental impacts are left out of the equation.
  • What is Wrong with Development Effectiveness? reviews this criterion and how it measures the intervention regarding the program’s objectives, set by the own organization and its development partners (without any benchmark or standard) and that are usually quite static along the program’s life (regardless of changes in the context). In addition, effectiveness keeps attention mostly focused on intended results and fewer evaluations collect evidence on effects outside the immediate results chain and identify unintended consequences.
  • Impact: The Reason to Exist questions this criterion as we need to find ways that recognize the systemic effects interventions can have on a more complex network of interrelated development processes.
  • Sustaining a Focus on Sustainability shares how the different dimensions of sustainability – economic, fiscal, environmental, and social – are complex and have to be taken together.

Reading these posts while nodding, I thought of how those criteria are not sealed boxes, but that they are permeable areas that in fact touch to each other and overlap in some points, so I created this Venn diagram to explore these intersections and represent it:

These are some of the clear conjunctions that came to mind. But there may be many more. And I know I included an uncommon mix of criteria (Sustainability is not but Responsiveness is, for instance), but I will revisit it – as this is not the only reflection that I will be sharing regarding Caroline’s provocative thoughts.

Keep tuned! Thanks.

You want to see more Visuals?