The evolution from evaluators to evaluative thinkers

As the demands on evaluation grow, the field itself is evolving from a group of evaluators to a profession of evaluative thinkers. It means pushing more creative and responsive evaluation designs that can grapple with increasingly complex questions and issues while delivering rigour in new ways. We think this is a good thing. Do you?

We are undertaking an evaluation with the Open Government Partnership (OGP) that falls into this category. OGP is a partnership of governments and civil society that has coalesced to foster accountable, responsive and inclusive governance.

The evaluation of the OGP is a two-year process designed to be rigorous, to deliver ‘answers’ to high level evaluation questions, but to do so in a way that puts learning front and center. We have developed an evaluation approach that recognises the complexity of these change processes while delivering on rigor and real time learning.

There is no one methodological or analytical approach that is going to get us the answers OGP would find most helpful, so we are bringing a bunch of approaches to bear, under a Developmental Evaluation umbrella:

• Using different evaluation designs to answer sub-questions rather than marrying the entire evaluation to a specific approach. This allows us to benefit from the methods’ strengths without being confined by their limitations. For example, we are using contribution tracing to answer questions about causality while hoping to use QCA to look across heterogeneous cases to identify those combinations of factors that successfully influence change.
• Embedding researchers in the field so that they can have dynamic conversations with key stakeholders over the life of the evaluation. Researchers are documenting change processes in real time, staying open to following the story where it takes them.
• Safeguarding a flexible fund that will enable us to explore new questions, new change processes, new outcomes over the two years. This will allow us to lean into findings as they emerge without undermining our ability to deliver on the questions OGP have now.

It’s complicated, with a lot of moving parts. But we think it will be worth it. More of this work is being championed by some corners of the bilateral (shout out to USAID’s Innovation Lab) and philanthropic world. Understandably, program teams are cautious – this new approach to evaluation takes a lot of their time, and requires a different approach to the relationship between commissioner and evaluator. It’s too early to tell if we’ll manage a true ‘developmental evaluation’ or happily settle for an utilisation-focused evaluation (see Tanya Beer’s excellent blog post on this). But we hope by reflecting as we go, we also grow while sharing its value and pitfalls.

This article was originally posted at the American Evaluation Assosiation (AEA) 365 blog. 

Photo: Shutterstock/sladkozaponi