More and more, our donors, beneficiaries and the general public are requesting rigorous and objective evidence of the efficiency of our action. Beyond accountability requirements, good indicators also help us to assess the quality of our programmes, and correct them if necessary. In this article, you’ll find a few tips to define and use indicators in the conception and follow-up of your projects, while avoiding the most common mistakes!
But to start with… What is an indicator?
An indicator is a factor or a variable that can be measured objectively. It is, used to reliably assess the changes obtained or the progress accomplished by a project in attaining its results and objectives.
The indicators need to have the following characteristics
- Reliable: they can be measured in the same way at different times and by different observers (also referred to as OVI, Objectively Verifiable Indicators).
- Precise: they can be measured clearly by quantitative or qualitative data.
- Appropriate: indicators should provided measurement of changes that have been brought about by the project.
- They can be verified through varied sources, including external sources, as far as possible (governmental or United Nations data, other sources as reliable as possible…)
Don’ts: the classic mistakes
- Defining too many indicators: Don’t forget to take into account the resources needed for indicator follow-up. Indicators need to be sufficient, in number and variety, to give an indication of the changes that have occurred, while at the same time not drowning the team in monitoring and follow-up!
- Forgetting the baseline: Indicators define a value to attain. What will they be compared to if no reliable data exists for this particular project? If initial data is not available, you’ll need to plan for this data collection before starting the intervention. International norms will also be useful for comparison purposes (for example those of the Sphere project)
- Indicators that are too vague and difficult to measure, such as “an improvement of the community’s well being”. How will we measure this improvement? In this case we’ll need to translate the concept into measurable elements, such as mortality rate, literacy rate, or other elements on which the projects will have an impact.
- Indicators without direct relation to the programme objectives, or taking into account other external factors not addressed by the project. For example, will our project really be able to influence the unemployment rate or improve the legal framework?
- Data that is impossible to verify: For example, there might be confidentiality issues, specific to each culture, or data might not be disaggregated according to the follow-up needs – by gender, etc. You’ll have to take into account taboos and cultural factors, testing the questionnaires for potential any confusion etc. Similarly to project objectives, indicators should, as far as possible, be defined in close collaboration with populations targeted by the project.
- Including a transversal aspect in the project without reflecting it in the indicators: For instance, if a ‘gender’ component is mentioned in the project objectives, the main indicators will need to be disaggregated by gender.
How can we measure indicators?
Ideally, sources should be cross-referenced. You should not rely exclusively on external sources, for example published by the government, because of the risk that the data has been manipulated for political aims. Similarly, you should avoid relying exclusively on internal data, the accuracy of which could be questioned. Consequently, you’ll need to vary your sources as much as possible!
The different types of indicators
- Activity, result or impact? We have a natural tendency to try and define activity indicators. For example, we’ll try to measure the success of a professional training course through the number of people attending the training sessions. But what does this number really tell us about the actual effect of the project? We must go deeper. In the case of a bee-keeping training programme, for example, we must measure the number of people starting their business (and maintaining it for longer than 6 months), the number of hives created etc. In most cases, it is useless to define indicators for levels below the results stage of the logical framework. In some cases, we can even define impact indicators, regarding the long-term effects of the project after the end of its implementation.
- Qualitative or quantitative? Even though defining quantitative indicators is easier, it is important not to limit yourself to them. For example, we could measure if we reached an objective with a quantitative indicator as follows: “At the end of the project, 100% of community members have access to drinking water within 150m of their home”. However, a qualitative indicator will measure the result obtained in a more in-depth way: “At the end of the project, 80% of community members, from a minimal sample of 200 people, consider that the new water point responds to their need for a drinking water supply».
There are however limitations to project assessments based on indicators…
- Indicators have a tendency to simplify cause and effect relationships. Real change always comes from a variety of factors.
- They also pose the problem of the ‘marketing’ effect in regards to donors. We tend to value and maximize the successes obtained, while sometimes concealing the less positive, or even negative, aspects of our action. This does not favour the analysis of our mistakes, which is vital to our learning and for improving our action in the long term.
- Finally, the focus is usually set on quantitative indicators that are easier to measure. This tends to simplify complex societal issues, for which we are inclined to only measure aspects that ‘suit’ our purposes, without necessarily having a longer-term vision at impact level.
In brief, it is important to define good indicators (and we do hope this article will help to do so) but also not to limit yourselves to them! What about you, what do you think?