Listen to an interview with Associate Professor Brian Cook about our approach to Research for Development and measuring impact.
“By broadening the scope of measuring impact and by implementing a number of support interventions, that are very different, enables us to compare support pathways that typically get done in isolation”.
Associate Professor Brian Cook
Four impact metrics
- Learning
- Psycho-social wellbeing
- Agricultural productivity
- Financial wellbeing
Read more below about these four metrics, our Theory of Change and our approach to monitoring and evaluation for the Next Generation Agricultural Extension: social relations for practice change project. Please note, this framework has been developed in June 2023 and is a work in progress.
Introduction
The Next Generation Agricultural Extension: social relations for practice change project is innovative along two lines of analysis. The first is the connection of deep social research with direct development support (i.e., ‘research 4 development’) while the second is the broadened scope of impact analysis applied to the provision of development support. The Next-Gen project places impact, monitoring, and evaluation (M&E) at the forefront of all project activities. In essence, the project, itself, seeks an improved understanding of the relative differences of alternative forms of supporting farmers, measured rigorously to identify which forms of support are most effective with different groups of smallholder farmers in Northwest Cambodia. Rather than an ‘additional’ element of the project, M&E is the core of the Next-Gen project. In simple terms, Next-Gen asks ‘what works?’ in the context of supporting smallholder farmers. In more detail, the project seeks to answer: ‘what types of support work with different types of famers at different times and places, why, how, and in what ways?’ The theory of change and monitoring and evaluation of impact establish the basis for answering those questions.
In general, M&E is used to measure the ‘impacts’ that the project has on participating farmers. Those impacts are primarily the outcome of Activity 6, where between 4-7 interventions will be provided to different types of farmers (drawing from the household typology established in Activity 2). In order to rigorously determine the impacts of the different types of support, the project establishes a quantitative (Activity 2) and qualitative (Activities 3, 4, and 5) understanding of smallholder perceptions, needs, wants, and practices. More specifically, the baseline census (i.e., Activity 2) is followed by qualitative interviews with smallholders (i.e., Activity 3), focus group discussions within communities (Activity 4), and targeted engagements with successful smallholder farmers and households (Activity 5). These data inform interventions with different types of smallholders (Activity 6), and assessment of the impacts (Activity 7).
The project rests on a simple theory of change (ToC), definition of ‘impact’, and broad commitment to M&E in order to capture the wide range of positive, neutral, and negative changes brought about by the interventions.
Theory of Change
An over-arching aim of the Next-Gen research project is to extend the theory of change that dominates agricultural extension in Northwest Cambodia and to empirically assess an alternative founded on social relations. In general, at present, the dominant theory of change presumes an absence of capital-technology and the associated knowledge, which results in ‘poor’ practices; this linear understanding echoes the classic transfer model, often labelled the diffusion of innovations model, described by Chambers and Ghildyal (1985, p. 18) as:
‘‘The transfer model uses a one-way information flow approach. When rural poor farmers do not adopt a new technology physical and social scientists assumed it was attributed to ignorance. This reinforces the idea that ‘We must educate the farmer’ . . . ‘We’ have the relevant knowledge. Ignorant farmers do not have it. We must teach the ignorant farmers.”
As an alternative, Next-Gen presumes that unsupportive and/or exploitative social relations inhibit farmers from altering their practices and that opportunities to form alternate relations will result in changed practices that benefit smallholders or their households. Further, such changed practices will not arise, only, as a direct result of awareness of potential social relations, but must be supported by extensionists working to improve farmers’ livelihoods via increased or improved social relations.
Within the agricultural extension discourse, researchers argue that the failures of the ‘diffusion of innovations’ theory have prompted its ‘replacement’ (Leeuwis & Aarts, 2011) as the prevailing theory of change (ToC). Alternatives include participatory empowerment (Chambers, 1983), systems thinking (Röling, 1992), transition processes (Geels & Schot, 2007), ‘agricultural advisory services’ (Birner et al., 2009), and appreciation for agro-social systems (Leeuwis & Aarts, 2011, p. 2), in which extension-adoption is defined as “a collective process within nested networks of interdependent stakeholders”. With this turn towards inclusion of socio-political considerations, the challenges of power, people, and place are centred, with growing appreciation for the extensionists who identify problems, develop technologies, implement interventions, measure the impacts of interventions, and publicize their impacts (Cook et al., 2021; Landini, 2016; Landini et al., 2017; Nettle et al., 2018; Nettle et al., 2017).
There is a significant difference between ‘having impacts’ compared to ‘having targeted impacts’, a difference that underscores ongoing debates over the growing prominence of Randomised Control Trials (RCTs) (Donovan, 2018) within agrarian studies. Those debates are, themselves, founded on the critique of agricultural extension as having been unable to ‘prove’ (Reddy, 2012) that DoI-based interventions into farmer decision making have resulted in targeted impacts on practice.
There are two scales of the proposed theory of change, though they are both based on the same logic: 1) at the project scale, we will implement and assess the effectiveness of supportive social relations targeted at expressed needs/wants as a model of extension, including its impacts on farmer practices; and 2) at the regional scale, we will explore whether the social relation-based extension – alongside empirical measurement of its impacts – alters how organisations involved in agrarian-change (i.e., project partners and others within the agricultural sector in Northwest Cambodia) practice extension and partner with smallholder farmers.
At its core, the Next Gen ToC hypothesizes that deep, social analyses of farmers’ lives and livelihoods will enable identification of preferred development support and pathways, allowing measurement of whether uptake and impact of the resultingly ‘targeted support’ (i.e., support that responds to farmers’ expressed wants and needs) is greater than levels of uptake and impact amongst the general population.
Monitoring and Evaluation
As stated in the ACIAR guidelines for impact assessment, the “fundamental task of impact assessment is to trace the way[s] in which the research leads to changes in the world” (Davis et al., 2008, p. 15). Such assessment requires systematic, clear, and consistent metrics. The Next-Gen project recognises that ‘impact’ is “the comprehensive actual (intended and unintended) consequences of the investment [i.e. funding of the project, and more specifically the support implemented through Activity 6]. Impact can also consider the impacts on different groups” (Davis et al., 2008, p. 17). Impacts materialize along an ‘impact pathway’, which is a “description of the causal links between inputs [investments deployed and delivered on the basis of the theory of change], outputs [the ‘interventions’ adopted in Activity 6], outcomes [“changes in practices, products or policy that result from adoption following the [interventions] by initial, next and final users”], and impacts” (Davis et al., 2008, p. 17). Impacts are, themselves, divided into ‘initial impacts’, which are:
“changes in demand, supply, environmental, and social pressures and exposure to risk at the community level due to the sum of outcomes at the individual level, or for common resources at the community level;
and ‘impacts’, which are:
changes in markets (prices, input and output costs, quantities) and in the state of common resources (ecosystem health and biodiversity) and communities (livelihood opportunities, health, security, equity)”
(Davis et al., 2008, p. 17).
In simple terms, impacts are the changes that result from the project activities, which require an analytical framework to implement and conceptualise their measurement (i.e., a logic and the specific measures that will be used to document impacts (i.e., collectively the ToC)).
Measurement of impacts
Traditionally, the measurement of impacts has defaulted to economic accounting. As stated in the ACIAR guidelines for assessing impacts:
“With these impact pathways identified, impact assessment then takes place within a benefit–cost framework that explicitly uses the broad theories of applied welfare economics (through the concept of economic surplus) to value inputs and outcomes”
(Davis et al., 2008, p. 17).
This classical economics measurement is one of the four metrics used by the Next-Gen project to measure impacts, with the three additional metrics broadening consideration of less tangible, social, and relational considerations (Birkhaeuser et al., 1988; Mawdsley, 2018; Sachs et al., 2004). Each of the four metrics are ‘composites’ of numerous measures collected across the project Activities.
While we accept that the impacts and their pathways are co-variable, it is essential that we not default or over-rely on economic accounting. The four metrics applied in the Next-Gen project to measure impact are: 1) learning; 2) psycho-social wellbeing; 3) agricultural productivity; and 4) financial wellbeing. The project determines impact by interrogating the four metrics, looking to triangulate findings while also being attuned for any divergence or conflict between the metrics. For example, it will be possible to measure an increase in Learning, Financial wellbeing, and Agricultural productivity while simultaneously measuring negative Psycho-Social change. Alternately, increased productivity may result in unchanged financial wellbeing with decreased psycho-social wellbeing. Broadly, through contrast and comparison of the four metrics, the project will help to deepen understanding of the diverse range of impacts arising from the different interventions and/or combinations of interventions.
1) Learning
Emergent within the M&E discourse is a ‘push-back’ directed at the over-quantification of impacts, which has accompanied growing desire for increased accountability and efficiency. As argued by Dozois et al. (2010, p. 6):
“The pursuit of ‘accountability’ at all costs and the market mantra of value-for-money have cemented the view that evaluation is designed mostly to provide cover for the giver, not useful information for the recipient.”
This position has resulted in advocacy for less prescriptive models of evaluation such as ‘Developmental Evaluation’, in which the primary focus of M&E is adaptive learning by all parties and impacts that benefit targeted populations with targeted outcomes. ‘Learning’, then, is central to the measurement of impacts following the interventions implemented in Activity 6.
Learning, defined as cognitive, normative, and relational change, will be established using a composite of Ensor and de Bruin (2022) and Baird et al. (2014), with additional attention for ‘spillover effects’ (Altschuler & Corrales, 2012; Galizzi & Whitmarsh, 2019; Zhang et al., 2020). The qualitative measurements of learning will be derived from analysis of interviews with smallholders as part of Activities 3, 4, and 5. Further, a pre-intervention interview will be conducted with all participating farmers involved in Activity 6, which will be contrasted with a post-intervention replication to document immediate changes (Activity 7). Lastly, depending on resources, either a full replication of Activity 2 or a partial sampling of participants from Activity 2 will conclude the project, documenting any longer-term impacts arising from the interventions; this will also establish a control sample to differentiate impacts (likely) arising from Activity 6 relative to wider changes amongst the population.
| Learning | Definition |
| Cognitive (positive) | Acquisition of new knowledge; restructuring of existing knowledge; shifting how situations are comprehended; |
| Cognitive (existing, neutral or negative) | |
| Normative (positive) | Changes in norms; changes in values; changes in paradigms; convergence of group opinion; |
| Normative (existing, neutral or negative) | |
| Relational (positive) | Improved understanding of mindsets of others; building of relationships; enhanced trust and cooperation; |
| Relational (neutral or negative) | |
| Behavioural (positive) | Acquisition or alteration to practices arising from learning(s); |
| Behavioural (neutral or negative) | |
| Spillover effects (positive) | The transfer of learning to non-participants; |
| Spillover effects (negative) | |
| Self reported results or changes | Any self-reported changes that a participant recounts when asked directly. |
A quantitative measurement of learning is established in Activity 2 (Table 2 and 3), with inquiry into social capital, trust, and access to (agricultural) information. These questions establish the source of trusted information used by households while simultaneously establishing a social network for the household (i.e., who the household interacts with). Participants are asked if they can access the internet, if they use social media, if they use chat apps, and if they use Youtube. Importantly, these findings situate data collection in Activities 3, 4, and 5 concerning learning amongst groups (Activity 4) and amongst successful farmers (Activity 5).
| In the last 12 months, have you or anyone in your household participate in the following? (Read out and code one answer for each): 1. Yes 2. No | Can you tell me the sex of your household member who participated in that activity? 1. Male only; 2. Female Only 3. Both male and female household members | ||
| a | Provided agricultural help to family, friends, or neighbours who do not live with you and did not pay you for the help. | ||
| b | Provided any kind of non-agricultural help to family, friends, or neighbours who do not live with you and did not pay you for the help. | ||
| c | Took part in a social event in your village (e.g. attending wedding). | ||
| d | Took part in a religious communal event/activity in your village | ||
| e | Done voluntary or charity work for your village. | ||
| f | Cared for a sick or disabled adult who does not live with you and who did not pay you for the help. |
| Prompt for answers: | What are the three main sources of information about production and postharvest practices? Rank from most to least important | What are the three main sources of information about agricultural prices?- Rank from most to least important |
| 1. Family & friends in village 2. Friends & family elsewhere 3. Traders/processor/other buyers 4. Extension agent 5. Cooperative/producer group/association 6. NGO 7. Research Centre 8. Village signboard/loudspeaker 9. Newspaper 10. Radio 11. Television 12. SMS 13. Videos from the internet 14. Chat apps/WhatsApp 15. Other Social media (Facebook) 16. Other, specify | 1. 2. 3. | 1. 2. 3. |
2) Psycho-social wellbeing (P-SW)
A critical finding of ASEM/2013/003 is the risk averse nature of farmer decision making in Northwest Cambodia amongst smallholder cassava producers but potentially representative of smallholders generally. Central to farmers’ accounts of their practices is a clear risk-aversion, in which low-cost, low-input, and predicable relations are prioritised – with the inverse also true (i.e., resistance towards high-input, high-cost, high-risk production options).
A quantitative baseline of Psycho-social wellbeing is established in Activity 2 with the measurement of subjective satisfaction of personal wellbeing (i.e., satisfied with your life). These data are triangulated with comparison of perceptions concerning the participants’ parents’ lives (i.e., perceived relative wellbeing). Akin to the problem-solution pathways (PSPs) of FUAT, a problem/solution-focused approach has informed the psycho-social measurement of wellbeing to ensure that the focus is not strictly negative or problem oriented, but also includes appreciation for the assets, skills, and perceived strengths of the participating households (emphasised in ABCD methodology guiding Activity 4).
Also informing measurement of P-SW are self assessments of life. These topics are the explicit focus of Activities 4 and 5, building upon Activity 2 measurement of household wellbeing.
| Based on your current household needs, what do you think of your current financial situation? | 1. Very good 2. Good (more than enough to meet our needs) 3. Enough to meet our needs 4. Not enough to meet our needs 5. Really not enough to meet our needs | |
| Comparing your standard of living with your parents’ standard of living when they were about your age, would you say that your household is better off, worse off or about the same? | 1. Better off 2. Worse off 3. About the same | |
| Overall, are you satisfied with your quality of life at the moment? | 1. Very satisfied 2. Satisfied 3. Somewhat satisfied 4. Not really satisfied 5. Not satisfied at all |
More qualitatively, we establish the confidence of households in terms of their agricultural productivity and financial profitability, including any non-cash income or bartering.
3) Agricultural Productivity
Agricultural productivity is first established in Activity 2, with qualitative engagements with farmers as part of Activity 3, 4, and 5. The determination of productivity will establish the household labour directed to agricultural productivity, as well as the primary agricultural outputs and their use (e.g., cash sale, home consumption, exchange). Additionally, the total land farmed is also established (owned with documentation, without documentation, leased, or rented), including its productive use(s) (e.g., livestock, fruit trees, cassava, rice).
Productivity will also be established relative to agricultural assets and expenses as a way of triangulating and interrogating the findings. For example, over-estimation of productivity can be identified if the total land is incapable of producing the stated outputs or if the expenses for ploughing costs do not align with the stated amount of land farmed. This triangulation will be used to clean and check the data.
4) Financial wellbeing
The determination of financial wellbeing is a composite of financial questions initiated in Activity 2 (see Figure 1). The direct measure of income is established directly (see Table 5). Supplementing the measurement of income are accounts of assets (e.g., ICT devices, vehicle ownership, agricultural tools and equipment) within the household, especially the ownership status of the land and housing. Activity 2 establishes the household income, including agricultural and off-farm sources (see Table 5). Importantly, these data are cross-referenced with direct questions of monthly and yearly household income in order to, wherever possible, triangulate the findings.
Figure 1: Household income by source
These data are highly diverse, which allows the Next-Gen project to analyse the nature of different income streams, thereby, ultimately, improving the depth of impact analyses in Activity 7 (see Figure 2 for an example).
Figure 2: Non-agricultural income by increasing order
| Did anyone in your household receive income from the following activity Income source | Can you tell me the sex of your household member who participated in that activity? 1. Male only; 2. Female Only 3. Both male and female household members | What is the relative importance of this income source in your household? 1. Most important 2. Important but not most important 3. Helpful, but least important. | ||
| a | Rice | |||
| b | Maize | |||
| c | Cassava | |||
| d | Beans | |||
| e | Vegetables | |||
| f | Fruit trees | |||
| g | Livestock | |||
| h | Aquaculture | |||
| i | Full-time public-sector employment | |||
| j | Other full-time salaried employment | |||
| k | Agricultural casual labour/job | |||
| l | Non-agricultural casual labour/job | |||
| m | Rental income from land | |||
| n | Rental income from other assets (e.g. tractors) | |||
| o | Remittance | |||
| p | Agricultural trading | |||
| q | Non-agricultural trading | |||
| r | Artisanal (carpenter, mason | |||
| s | Home-based tailoring and textiles | |||
| t | Factory work | |||
| u | Other – specify | |||
| v | Other -specify | |||
| w | Other – specify |
Key to a ‘wellbeing’ understanding of household finances, the project also collects data on the relative importance of different income sources (e.g., remittances, off-farm labour, small businesses). Lastly, Activity 2 establishes a subjective measurement of financial distress (Table 3), which we use to situate the qualitative accounts of household finances. These measures are especially important to the wellbeing of the household.
| Cannot pay bills on time (phone, electricity etc) | Yes | No | |
| Cannot pay back loans/credits to bank or other lenders | Yes | No | |
| Having to pawn/sell something to meet needs | Yes | No | |
| Sometimes do not eat/having to go without food | Yes | No | |
| Cannot purchase new clothes/shoes | Yes | No | |
| Asking for families/relatives/neighbours/friends for financial help | Yes | No | |
| Asking for financial help from community organisation | Yes | No | |
| Having to go without medicine/medical treatment when needed | Yes | No |
References
Altschuler, D., & Corrales, J. (2012). The spillover effects of participatory governance: evidence from community-managed schools in Honduras and Guatemala. Comparative Political Studies, 45(5), 636-666.
Baird, J., Plummer, R., Haug, C., & Huitema, D. (2014). Learning effects of interactive decision-making processes for climate change adaptation. Global Environmental Change, 27, 51-63.
Birkhaeuser, D., Evenson, R., & Feder, G. (1988). The economic impact of Agricultural Extension: A review (World Bank working paper). Washington, DC: World Bank; also published by Economic Growth Center (1989).
Birner, R., Davis, K., Pender, J., Nkonya, E., Anandajayasekeram, P., Ekboir, J., Mbabu, A., Spielman, D. J., Horna, D., & Benin, S. (2009). From best practice to best fit: a framework for designing and analyzing pluralistic agricultural advisory services worldwide. Journal of Agricultural Education and Extension, 15(4), 341-355.
Chambers, R. (1983). Rural development: Putting the last first. Routledge.
Chambers, R., & Ghildyal, B. (1985). Agricultural research for resource-poor farmers: the farmer-first-and-last model. Agricultural administration, 20(1), 1-30.
Cook, B. R., Satizábal, P., & Curnow, J. (2021). Humanising agricultural extension: A review. World Development, 140, 105337.
Davis, J., Gordon, J., Pearce, D., & Templeton, D. (2008). Guidelines for assessing the impacts of ACIAR’s research activities. ACIAR Impact Assessment Series, 58.
Donovan, K. P. (2018). The rise of the randomistas: on the experimental turn in international aid. Economy and Society, 47(1), 27-58.
[Record #2205 is using a reference type undefined in this output style.]
Ensor, J., & de Bruin, A. (2022). The role of learning in farmer-led innovation. Agricultural Systems, 197, 103356.
Galizzi, M. M., & Whitmarsh, L. (2019). How to measure behavioral spillovers: a methodological review and checklist. Frontiers in psychology, 10, 342.
Geels, F. W., & Schot, J. (2007). Typology of sociotechnical transition pathways. Research Policy, 36(3), 399-417.
Landini, F. (2016). How to be a good rural extensionist. Reflections and contributions of Argentine practitioners. Journal of Rural Studies, 43, 193-202.
Landini, F., Brites, W., & y Rebolé, M. I. M. (2017). Towards a new paradigm for rural extensionists’ in-service training. Journal of Rural Studies, 51, 158-167.
Leeuwis, C., & Aarts, N. (2011). Rethinking communication in innovation processes: creating space for change in complex systems. Journal of Agricultural Education and Extension, 17(1), 21-36.
Mawdsley, E. (2018). Development geography II: financialization. Progress in Human Geography, 42(2), 264-274.
Nettle, R., Crawford, A., & Brightling, P. (2018). How private-sector farm advisors change their practices: an Australian case study. Journal of Rural Studies, 58, 20-27.
Nettle, R., Klerkx, L., Faure, G., & Koutsouris, A. (2017). Governance dynamics and the quest for coordination in pluralistic agricultural advisory systems. 23(3), 189-195.
Reddy, S. G. (2012). Randomise this! On poor economics. Review of Agrarian Studies, 2(2369-2021-102).
Röling, N. (1992). The emergence of knowledge systems thinking: A changing perception of relationships among innovation, knowledge process and configuration. Knowledge and policy, 5(1), 42-64.
Sachs, J., McArthur, J. W., Schmidt-Traub, G., Kruk, M., Bahadur, C., Faye, M., & McCord, G. (2004). Ending Africa’s poverty trap. Brookings papers on economic activity, 2004(1), 117-240.
Zhang, A. J., Matous, P., & Tan, D. K. (2020). Forget opinion leaders: the role of social network brokers in the adoption of innovative farming practices in North-western Cambodia. International Journal of Agricultural Sustainability, 18(4), 266-284.