Saville Kushner Information Age Publishing, Inc. 2017, PB, 296pp, £37.80, 978-1681236889
If, for you, evaluation means applying a technocratic, logic-model approach to assessing whether something was delivered exactly as planned and whether it produced the outcomes and impacts predicted, this book will be of little use to you. Likewise, if you are a dyed-in-the-wool realist evaluator who seeks the theoretical elegance of context– mechanism–outcome configurations, you can probably give it a miss too.
If, on the other hand, you view evaluation as a scholarly practice that is situated, political, negotiated, emergent, linked to democracy, and dependent on serendipity and judgement (as well as on careful measurement), this may be the book you are looking for.
Evaluation is situated because it is about how policies and programmes affect the lives of particular people in particular settings. The most powerful approach is usually the in-depth case study, written up as ‘thick description’, enriched with both qualitative and quantitative data.
Evaluation is political because it is the powerful who decide what (and whether) to evaluate, what counts as ‘success’, whose voices get captured (and whose silenced), and which sections of the report get censored.
It is negotiated because, according to Kushner, one of the key roles of an evaluator is to provide a platform where stakeholders can come together and argue about values and priorities.
Evaluation is emergent because, in the real world, programmes inevitably change as they are implemented. I recall leading an evaluation of a large quality improvement programme a few years ago, in which the (large and inclusive) project steering group could not agree on either the work streams or the main outcome measures — for what turned out to be very good reasons. Our evaluation became immeasurably easier when we abandoned our quest for a logic model and decided to just follow the actors and change horses whenever they changed horses.
Evaluation, in Kushner’s view, is linked to democracy because he draws on Barry MacDonald’s taxonomy: bureaucratic evaluation (in which the evaluator serves the government and generally endorses the government’s view), autocratic evaluation (in which the evaluator seeks to operate as an independent, objective scientist), and democratic evaluation (in which the evaluator engages with questions of inequality and seeks to overcome the biases associated with power imbalances).
Evaluation is dependent on serendipity and judgement because cases unfold unpredictably. It is often the surprises that best explain why a programme worked or failed to work. There is no blueprint that will tell you whether (for example) to pull out a tape-recorder or a notepad and pencil when undertaking a sensitive interview.
In short, this is a book for those who already know that there is no formulaic way of doing evaluation and who understand that an evaluator is situated within, not above or beyond, the prevailing social and political context. It is a book about mess, complexity, and struggle. And it is a beautiful and passionate swansong from someone who has devoted his life to refining and applying the principles of democratic evaluation with the goal of improving social justice.
- © British Journal of General Practice 2017
