Evaluation of Solidaridad Practice for Change programme (PfC): Final report

Overview

This is an accountability and learning evaluation of Solidaridad's global Practice for Change portfolio, funded by the Netherlands Ministry of Foreign Affairs. It covers 25 in-scope projects (27 according to the executive summary) across seven regions, including East & Central Africa, for 2016–2020, and was carried out between July 2021 and January 2022. It assessed the three pathways (Good Practices, Supportive Business Ecosystems, Market Uptake) and four innovation themes (gender inclusion, digital solutions, climate change, access to finance) through nine case studies. The only ECA case was the Green Tannery Initiative (leather, Ethiopia). The evaluation's aim was to inform future strategy and programming.

Key findings

• OECD-DAC scores: Relevance Satisfactory; Coherence Highly Satisfactory; Effectiveness Satisfactory for Good Practices, Supportive Business Ecosystems and innovation themes but Moderately Satisfactory for Market Uptake; Efficiency, Impact and Sustainability Satisfactory.
• Portfolio figures from the 2020 progress report: 637,712 farmers and miners trained (99% of target), 483,500 adopting good practices (101%), 683,400 workers adopting good practices (84%), 2,500,000 ha under good practices (139%), and 349,100 producers and workers with an improved income position (105%).
• Case-study projects improved working conditions for about 57,000 workers, 27% of their 2020 target. This was partly because targets rested on wrong assumptions about how many workers were on targeted farms.
• Service providers supported far exceeded targets (6,941 against 1,100). The portfolio reports 309 financially viable service providers (80% of target), but case studies cast doubt on viability claims. About 2,600 green jobs were created, on target but small relative to the need.
• In the Ethiopia leather case, three tanneries ran successful bulk trials of chrome-free leather but did not move to larger-scale production. Market linkages with EU buyers were not finalised, a quality-based payment system was not implemented, and political unrest added to COVID-related challenges.
• PfC had no gender-specific KPIs or monitoring. Where case-study projects focused clearly on gender, intended results were achieved. In Ethiopia, women labour union leaders from four textile and garment factories were trained on handling gender-related grievances.
• The full €44.5 million budget was spent by end-2020, and match funding reached €45.8 million (3% above target). M&E rarely led to strategic changes because it did not capture systemic change or scaling.

In figures

Recommendations

• Base project ToCs on a holistic problem analysis, covering GP, SBE, MU and EPE pathways and the links between them, with explicit and tested assumptions.
• Keep an integrated approach combining good practices, robust infrastructure, market uptake and enabling-policy interventions.
• Focus on fewer sector-geography contexts to contribute meaningfully to sector transformation, and use 'isolated' innovation projects where conditions are not yet in place.
• Start every sector-geography context from a holistic, transformation-oriented understanding of the sector, using a systems framework to choose which systemic issues to address.
• Decide where Solidaridad adds value as facilitator and thought leader in sector coordination, and back that role with long-term funding, local ownership, flexibility and skilled staff.
• Work with corporate partners on their own policies, business models and sourcing practices, give more attention to fair pricing and trading, and develop a KPI for changes within companies.
• Link M&E, digital tools, evidence-based learning and adaptive management better: use mixed methods and granular data, monitor scaling, frame clear learning questions, and fund data-analysis staff.

Lessons learned

• Training smallholders does not automatically pass knowledge to their workers. Workers need to be trained directly, by more innovative means.
• Building market supply first and engaging buyers later (a push strategy) is risky when market demand does not materialise, as happened in the gold and leather cases.
• Digital tools help scale capacity building but cannot replace physical presence, need follow-up visits, and are not necessarily cheap.
• Service providers and producer organisations take several years to become viable. The India soy experience suggests at least four years to build solid governance.
• Innovation themes introduced after projects were designed were hard to resource and were sometimes seen as external requirements. Short (three-year) projects, such as leather in Ethiopia, make it hard to build trust with companies and the public sector.

Methodology

Dual approach: a summative evaluation using reconstructed impact pathways, mixed methods (Solidaridad KPI and M&E data plus semi-structured key informant interviews), contribution and additionality analysis; and a formative deep-dive on four learning themes. It drew on nine purposively selected case studies (seven producing-country projects plus two EU units), studied through desk review, remote interviews, triangulation and validation meetings, alongside a programme-level document review and staff interviews.

Limitations

• The evaluation relies heavily on nine case studies selected non-randomly, which carries a risk of selection bias, for example through the evaluability criterion.
• Resources did not allow extensive interviews in each case study, for example with other donors or stakeholders not directly involved, to get independent views.
• Case studies were done remotely by online or phone interviews and workshops. Technology was sometimes a problem, and only some case studies reached final beneficiaries directly.
• The mixed-methods approach cannot prove that Solidaridad caused changes. It gives levels of plausible contribution only.
• Most projects continued into 2021–2022 under the extension, and activities in 2021 could not be separated from earlier ones (monitoring data was reviewed up to 2020).