How to Measure Impact in Media Development?

Evaluations should be used to adjust media development strategies, with a stronger focus on sustainability and efficiency that measure long-term impact, according to a new study from the Center for Media, Data and Society (CMDS).

As donors increasingly seek impact for their dollars, monitoring and evaluation has become an important part of the work carried out by media organizations, both news outlets and freedom of expression NGOs. However, their approach to evaluation needs improvement.

First of all, they have a propensity to evaluate everything. Some 80% of the organizations canvassed by the study Measuring Impact in Media Development: Key Trends, published today by the Center for Media, Data and Society, said that they try to include “as much as possible” in the evaluation terms of reference. As a rule, they ask evaluation consultants to include all possible criteria in their assessments, rarely choosing to focus only on one or two of criteria that interest them the most.

Moreover, some 69% of them do not use the results of evaluations to alter their strategies, and that is because “either the evaluation report is too general or the recommendations made by the evaluator are not very actionable,”, Marius Dragomir, the report’s author wrote. A fifth of the organizations say that their strategy process is so complicated that they completely ignore the results of evaluations during their strategy discussions.

That has to change, many organizations admit.

According to the study, evaluation has to be shaped and initiated more by the staff working on the organization’s programmatic issues; at the moment, this happens in only 1% of the cases. They have to stop seeing evaluation as a reputation-building tool. “We have to be more critical of our own work and encourage consultants hired to carry out evaluations to be much more critical about what doesn’t work in our organization and about all the factors that limit our impact,” said a London-based expert working for a global NGO.

While impact and outcome evaluations remain important, the focus has to also shift towards sustainability and efficiency-focused evaluations that measure long-term impact and the resources put into these organizations, respectively.

In terms of methodology, the combination of desk research and interviews are the most common evaluation methods, the study found. Although it is the most cost-effective methodological mix, more evaluation methods have to be included to improve the quality and depth of evaluation reports.

Finally, evaluations written in less technical language should be encouraged. The evaluation lingo makes evaluation reports look more professional, some say. But it doesn’t make them more popular. “I would welcome a more journalistic way of writing these reports instead of the heavy magnum opuses written in the dull evaluation jargon that we usually get,” said a London-based journalist working with an international NGO.

The study is based on 77 evaluation projects documented by its author between 2009 and 2019.