Participatory ex-ante impact assessment for interactive research and development in agriculture and food systems

ABSTRACT

Drawing upon literature from both impact assessment of development projects and agricultural research, the aim of this article is to analyse the pilot testing of a new multi-dimensional assessment framework for defining and evaluating the societal impact of agricultural research and corresponding education. The research approach involves an action research effort of pilot testing in three case studies from three different countries. The framework assumes a systems view, understands impact as socially embedded, and adopts the concepts of contribution and productive interactions rather than cause-and-effect attribution. The emerging experiences include developing both project-specific and universal indicators; gauging impacts at different levels and sustainability dimensions; handling the issues of attribution and time frame; and the role of stakeholder involvement. The results have the potential to support the development of a new role for impact assessment, by enabling principal actors in research and higher education institutions to take responsibility for contributing to concrete and demonstrable sustainability changes in society.

KEYWORDS:

• Evaluation
• monitoring
• education
• productive interactions
• agricultural innovation system

1. Introduction

A deliberate transition within agriculture and food systems is fundamental for a sustainable society, as agriculture is ‘the world’s single largest driver of environmental change, and, at the same time, is most affected by these changes’ (Rockström et al. 2017, p. 4). Yet, the agricultural sector has received limited attention in environmental impact assessment literature and practice (Duffy 2004; Tzilivakis et al. 2011). Agriculture’s embeddedness in local culture and practices points to a need for integrating social and environmental dimensions while assessing impact of development (Cusworth 2020). As research and education play a vital role in agricultural development, assessing their impacts becomes essential (La Rovere et al. 2009; Hiruy and Wallo 2018). An emerging role for agricultural research is not only to serve as a means for generating knowledge, but also to act as an instrument for practical change (Horton 1998). This calls for new forms of participatory ex-ante impact assessment in order to both support and estimate these changes.

The opportunity to advance impact assessment theory and practice rests in academics and practitioners expanding their perspectives beyond their disciplinary boundaries (Fischer and Noble 2015; Ehrlich 2022). There is an expectation that agricultural research and education should lead to reduced negative environmental and social impacts, just like the use of impact assessment of various development projects. However, the interfaces between these fields have remained under-explored. Therefore, this paper draws on both the literature on impact assessment of development projects and the literature on agricultural research impact monitoring and evaluation.

Based on these assumptions, an assessment framework was developed to define and evaluate the impact of practice-oriented agricultural research projects, and the corresponding research activities embedded in education, as part of the Horizon 2020 project Nextfood (www.nextfood-project.eu). The aim was to address some of the perceived shortcomings in current impact assessment methodologies and create a multi-dimensional framework that could be used across different levels within the agriculture and food research and education system. The framework aims to challenge a one-sided focus on economic (Horton 1998) or environmental impacts (Fischer and Noble 2015) by integrating sustainability dimensions. The Nextfood framework was developed in 2019, and was tested and further developed in 2020–2021.

Thus, this paper aims to analyse the results of three pilot tests conducted using the multi-dimensional assessment framework. The results provide the basis for a critical discussion of the framework’s usability in different contexts, how it can be further refined, and how the application of such a framework may support the emergence of a new role for research and higher education institutions, allowing them to become agents of practical change.

The article starts with a literature review, followed by an outline of the theoretical lens used in this study. Next, the methods used for testing the assessment framework are explained. The results are then presented in the form of the lessons learned and experiences gained, as well as questions emerging from its use. This is followed by an analysis and discussion of the results, along with proposals for further research and development. The key contribution of the article is the provision of new perspectives on impact assessment. It emphasises a multi-dimensional, interactive process that evolves within diverse temporal and contextual settings; it also explores how the implication of such perspectives can facilitate practical contributions to sustainable development.

1.1. Literature review

The literature review covers two streams of impact assessment of development projects and agricultural research impact evaluation. We draw upon three questions based on this review, which we will return to in the discussion in section 4.

Historically, the literature on impact assessment of development projects has argued for the combination and integration of environmental, economic and social dimensions (e.g. Parthasarathy 2002), as a broader consideration of impacts will lead to improved impact assessments (La Rovere et al. 2009). The interest in integrating different sustainability dimensions has been sparked by the often complex challenges related to sustainable development (Mottee et al. 2020), and, in particular, by taking social impacts into account (Vanclay 2020). For example, Hiruy and Wallo (2018) report on a growing interest in the social impacts of research programmes, and use social impact assessment (SIA) to analyse the social impacts of a fisheries research programme. Furthermore, Parthasarathy (2002) and Hiruy and Wallo (2018) emphasise the dynamic nature of the impact assessment process, rather than taking a static view. Fischer and González (2021) account for the successive development of the underlying philosophies of impact assessment, from positivist to post-modern and post-factual perspectives.

Historically, the impact assessment of agricultural research has evolved from an economic analysis with a cost-benefit focus towards increasingly embracing the complexity of environmental, social and economic impacts (Horton and Mackay 2003). Nonetheless, economic impact assessment remains important as it serves as a justification for viewing agricultural research as an investment (Horton 1998), with the majority of impact assessments still focusing on economic impact (Weiβhuhn et al. 2018). The new models, on the other hand, acknowledge agriculture as a highly complex system, catalysing a shift in underlying logics principles from positivist to constructivist foundations (Douthwaite et al. 2003).

Hence, the broadening of the considered scope in the literature on impact assessment of development projects reflects a parallel development between agricultural research impact evaluation. However, there are also diverging developments. While the former has blossomed into several strands addressing specific sectors, administrative levels and decision tiers (cf. Fischer and González 2021), the latter has embraced the interaction between researchers, farmers and other relevant stakeholders (Pretty 1995; Horton and Mackay 2003; also acknowledged by e.g.; Parthasarathy 2002; La Rovere et al. 2009). The agricultural research impact evaluation literature imply that impact assessment standards need to be adapted to the context of the specific research project or programme (Horton 1998) and engage an extended peer community of relevant stakeholders (Funtowicz and Ravetz 1993).

For example, the Impact Pathway Evaluation (IPE) model divides the evaluation process into two phases, each related to product and process, respectively (Douthwaite et al. 2003). This model demonstrates how impacts can be seen as stemming from the use and relevance of research results, while also acknowledging the research process itself as potentially impactful. Another framework that acknowledges the processual nature of impact is Outcome Evidencing (OE), an ex-ante ten-step rapid evaluation approach to programme monitoring meant to be repeated throughout the research process for continuous learning and adjustment (Paz-Ybarnegaray and Douthwaite 2017; Douthwaite and Hoffecker 2017). While these models were developed for specific projects, there have been initiatives to try to find models that would be more general and less dependent on a specific context.

Studying the interactions between researchers and stakeholders, using agricultural research as one of their cases, Spaapen and van Drooge (2011) focused on the ‘productive interactions’ between researchers and stakeholders. The concept of productive interactions can be broadly understood as an exchange between researchers and stakeholders in which knowledge is produced that is both scientifically robust and socially relevant. These interactions are seen as productive when stakeholders make use of the research results and thereby contribute to the generation of impact.

This points to the issue embedded in the shift in understanding of impact assessment, moving from a narrow focus on either economic or environmental analysis to a broader societal impact focus, integrating economic, environmental and social dimensions of sustainability. A comprehensive assessment of research efforts within a complex agricultural system will demand a combination of a broad view of societal impact, as well as adaptability to the specific case, which, in turn, challenges the notion of a universal approach to impact assessment, i.e. the ability to compare assessments between projects. It raises the question, is it possible to develop an impact assessment model that caters to the needs of both adaptability and universality? (Question 1)

Stakeholder participation and interaction are key in the impact assessment literature (Parthasarathy 2002; La Rovere et al. 2009). Sadler and Verheem (2023) argue that one of the key issues is developing stakeholder involvement in strategic environmental assessment (SEA). The authors acknowledge the ambivalence associated with using an integrated approach to sustainability in SEA, which is partly related to the degree of stakeholder participation and interaction. Faure et al. (2018) and Douthwaite and Hoffecker (2017) found that vibrant interactions between stakeholders and researchers constitute a major contribution to successful outputs and outcomes in agricultural research for development. They concluded that capacity building for relevant stakeholders within the local context is essential for producing positive outcomes and generating societal impacts (ibid). Joly et al. (2015) built on these notions in their impact assessment model, ASIRPA, focusing on innovation processes, stakeholder networks and the assessment of long-term impacts. The ASIRPA and the earlier described impact assessment models demonstrate how stakeholders tied to social networks constitute valuable opportunities for collective action towards sustainable impact, both within the impact assessment process and within the research process itself. Nonetheless, there are several pitfalls associated with stakeholder interactions, such as those described by, for example, Moreira et al. (2022). It is important, then, to consider how stakeholder participation in impact assessment can be developed and organised successfully. (Question 2)

In assessing impacts, a shift in focus from attribution to contribution has been suggested by Spaapen and van Drooge (2011) and Joly et al. (2015). This approach does not seek to attribute a certain effect from a cause-effect chain as a separate and additive impact; instead, it acknowledges multiple factors and seeks to identify the contributions made, while recognising the multiple factors influencing a result. According to Joly et al. (2015), the shift towards contribution is motivated by the fact that complex systems are characterised by synergistic, i.e. non-additive, effects (cf. Ehrlich 2022). Alvarez et al. (2010, p. 947) involved relevant stakeholders to jointly ‘describe the project’s theories of action, develop logic models, and use them for project planning and evaluation’. Their non-linear, complexity-aware model attempted to account for all stakeholder interests by using a ‘causal loop’ system rather than a linear ‘if/then’ cause- and-effect formulation (Paz-Ybarnegaray and Douthwaite 2017). The ‘causal loop’ system helped depict the dynamics of learning and adaptive change during the research process, rather than afterwards. Additionally, Hiruy and Wallo (2018) opted out of the causal-link perspective, in favour of using pre-determined categories of social impacts.

With this view, it becomes evident that assessing complex and context-dependent societal impacts involves the notion of some effects becoming apparent only after a certain time. This poses a challenge when attempting to understand and adequately measure societal effects (Tzilivakis et al. 2011). Traditionally, ex-post assessments have been the favoured time frame as they allow for conclusive measurements of actual cost and benefit streams of research projects (Horton 1998). However, La Rovere et al. (2009) found that it was more accurate to measure some changes 5 to 10 years after the project ended. In addition, there is a growing awareness of ex-ante evaluation enabling a direct influence on research design, allowing for corrective measures to be undertaken that may mitigate unnecessary work (Horton 1998; Weiβhuhn et al. 2018). These insights have influenced the understanding of the appropriate timing for impact assessments of research, moving from a predominantly ex-post tradition towards a combined ex-ante and ex-post assessment. Hence, there are reasons to ask how impacts can be captured in complex systems, especially when some effects become apparent only after a long time. (Question 3)

1.2. Theoretical lens

According to Ehrlich (2022), in order to properly capture impacts, system thinking can aid the understanding of a certain phenomenon by integrating it into a larger whole. The term ‘systems’ has been defined as ‘networks of connected entities’ (Havelock 1986, p. 77), and the properties of a system can be derived from the way these entities interact (Ehrlich 2022). Similar to Joly et al. (2015), we are inspired by the work on actor-network theory (ANT) by Latour (2005). The ANT framework focuses on local actors within complex networks of connections and communications between the actors. It highlights how the social in ‘societal impact’ lies in what binds actors together (Latour 2005).

In particular, we build on Latour’s (1983) suggestion that the practical work that goes on in scientific laboratories, ‘the very content of the trials made within their walls’ (ibid, p. 159), helps to alter and displace, i.e. ‘translate’ society through the transformative processes by which various entities are combined and linked with others. This means that scientific knowledge is not useful per se; instead, it is made useful by communications and transformations among actors, rendering the knowledge ‘actionable’ (Joly et al. 2015, p. 441). In this way, the concept of translation includes not only ‘interactions’ but also the possible transformation of the actors and networks involved in the process (ibid). This series of translations among actors in networks is especially relevant to understanding socio-technical change.

ANT points to the transformative processes involved in technoscientific practices. This means that for particular technological or scientific ‘effects’ to become ‘impacts’, they need to be translated into terms that make those ‘effects’ recognisable as ‘impacts’ to the individuals or organisations for whom impact, as a way of valuing technology and science, matters in some way. Thus, this article broadly draws upon ANT and the notion of translation to highlight the challenges and actions involved in addressing actors’ action research experiences as impacts. This view highlights impact not as something that simply exists, ready to be measured, but as something that is performed by diverse actors. Therefore, a framework for measuring impact should account for actors’ performances, and how they relate to the performances of other actors.

2. Methods

2.1. Research approach

The research approach was based on the learning loops of desk research and a qualitative action research approach (Aagaard Nielsen and Svensson 2006). The research design started with the development of a preliminary version of a new impact assessment framework, named the Nextfood framework, based on literature reviews and expert interviews. Subsequently, the framework was tested in an action research effort using three case studies: research, education and communication materials. This article is based on the research data gathered during the three case studies. Based on the case studies, the framework was developed further, and the new version was subjected to expert reviews and further refinement (Nextfood Deliverable 5.5).

2.2. Case selection and data collection

The framework was tested in an action research effort using three case studies; research, education and communication materials, conducted in Sweden, Greece and the Czech Republic, respectively.

2.2.1. The research pilot

The aim of the research pilot study was to assess the framework and its use in a research setting. The study involved four practice-oriented agricultural research projects conducted at the Swedish University of Agricultural Sciences. Each project involved university researchers and industry stakeholders in working groups and reference groups. Initially, the participants were interviewed individually to understand their interests and roles in the respective projects and their experiences as part of them, as well as to initiate and follow their dialogues around impact. In an action research approach, the participants were invited to reflect together in focus group interviews on the expected, desired and actual impacts of the projects, as well as how to assess these impacts. One of the projects had a large reference group, in which case we involved a representative sample of stakeholders.

The group discussions on impact indicators were facilitated by two Nextfood researchers. The results of the first round of focus groups were used to adapt the framework to each of the research projects. Analysis of the data yielded a list of impact indicators, which were presented to the participants and further elaborated during a second round of focus group interviews. Hence, there were two rounds of focus group interviews for each of the four projects, where the participants discussed their projects in relation to the steps outlined in the framework. Both the focus groups and individual interviews were audio-recorded and transcribed. The focus group interviews were complemented with participatory observations of meetings. A summary of the pilot projects and the interview respondents is presented in Table 1.

Table 1. A summary of the four pilot tests and the roles of the respondents of individual interviews and in focus groups (n = 20). Two persons participated only in the individual interview (a).

Project, runtime	Project aim	Organisation and the role of respondents, in own organisation and in the research project
A. 2016–2019	To investigate the influence and cost of bureaucracy on farmers and on agricultural business development.	University, Assistant Professor, in working group (wg). Advisory organisation 1, Expert, wg. Advisory organisation 2, Expert, wg. Farmers’ organisation, Expert, in reference group (rg).
B. 2019–2021	To investigate farmers’ perspectives and experiences related to rural crime.	University 1, Professor, wg. University 2, Professor^a, wg. University 1, Researcher, wg. University 1, Technical staff, wg. Farmers’ organisation, Chairperson, rg. Farmers’ organisation, Expert, rg. Farmers’ organisation, Expert^a, rg.
C. 2020–2022	To study resistance to plant diseases in crop production.	University, Professor, wg. University, Researcher, wg. Farmers’ organisation, Chairperson, rg. Industry organisation, Chairperson, rg. Industry organisation, Expert, rg.
D. 2020–2022	To study a health issue in animal husbandry.	University, Researcher, wg. Advisory organisation, Expert, wg. Advisory organisation, Expert, rg. Rural Entrepreneur, rg.

2.2.2. The education pilot of undergraduate research projects

The objective of the education pilot study was to assess the application of the framework in an educational setting. It was used to assess the perceived impact of five undergraduate research projects that were conducted between the American Farm School and the International Hellenic University, in Thessaloniki, Greece. Each undergraduate research project was carried out in multi-stakeholder settings, with stakeholders including professionals, farmers, advisors, students, and professors (see Table 2). The assessment framework was implemented in a minimum of two educational sessions per case, where guided group discussions were embedded within the planned educational activities. A facilitator steered the group conversation towards the identification of impact indicators. The group discussions were recorded in the form of session notes.

Table 2. A summary of the five undergraduate research projects and the participants of the group discussions and individual interviews. Projects D and E shared the same professor and advisor. All projects were conducted in collaboration with the IHU Agricultural Technology Department, except for A, which was in collaboration with the IHU Food Science and Technology department.

Pilot case, runtime	Description of projects	Participants and interview respondents
A. 2021–2022	Students researched the ‘‘Effects of the use of hemp protein in the production of bread’	University Professor Laboratory Technician Student 1 Student 2 Advisor
B. 2021–2022	The students researched plant protection protocols for Tuta absoluta on a tomato farm.	University Professor Farmer 1 Farmer 2 Student 1 Student 2 Advisor
C. 2021	The student researched plant protection protocols in aromatic plants on an oregano farm.	University Professor Farmer Student Advisor
D. 2021	The student researched the ‘Effect of melatonin on bovine sperm characteristics’.	University Professor Farm Manager Farm Veterinarian Student Advisor
E. 2021	The student researched the use of ‘new digital technologies for the early diagnosis of bovine mastitis’ on a sheep farm.	University Professor Farmer 1 Farmer 2 Student Advisor

At the end of each undergraduate research project, all participants were invited to individual interviews, which were audio-recorded and transcribed. All collected data were analysed to produce impact indicators.

In addition to the group discussions and the individual interviews, participants were asked to produce a minimum of one Practice Abstract (see below). This was done during the last session as an indicator of project engagement, understanding and dissemination.

2.2.3. The communication materials pilot

This pilot study aimed to use the Nextfood framework to assess the practical usefulness of the communication materials commonly produced by practice-oriented research projects. One such ‘product’ of EU-funded agricultural research projects is the so-called Practice Abstracts. These are meant to be short, descriptive and innovative communiqués, aimed at facilitating the transfer of research knowledge into agricultural practice. The pilot study involved farmers, agricultural advisors and employees of control and certifying bodies in organic farming, based in the Czech Republic. For the recruitment of participants, the official database of advisors and other relevant practitioners in the register of the Ministry of Agriculture was employed. In total, 40 individual interviews and two focus groups, each with five participating stakeholders, were conducted. The participants were asked to choose a Practice Abstract relevant to their expertise and evaluate it using a questionnaire (available upon request to corresponding author). Nextfood researchers facilitated the two-hour focus group interview and took notes from the discussions. The participants first evaluated the Practice Abstracts orally in a discussion before being asked to fill in the questionnaire. The questionnaire was updated on the basis of the feedback from the first focus group.

2.3. The structure and procedure of the original framework

The original framework comprised a structural matrix and a five-step procedure. The structural matrix was created to be used as an organisational tool, in an approach similar to the tiered decision model of strategic environmental assessment, according to Fischer and González (2021). It is a guide for developing indicators, highlighting both process and product-related impacts, as referred to by Douthwaite et al. (2003), Parthasarathy (2002) and Hiruy and Wallo (2018). Each of these aspects was considered in relation to economic, environmental and social sustainability. The original framework matrix was divided into three interrelated impact levels: the project level, intermediary level and systemic level; see Table 3. At the intermediary level, parallel forms of mediation work were identified as achieving impacts in both product- and process-related realms; hence, the indicators at this level encompassed both categories.

Table 3. The structural matrix of the original framework.

	Process related impacts	Product related impacts
Project level The stakeholder community.	Impact on economic, environmental and social sustainability, related to the process on the project level.	Impact on economic, environmental and social sustainability, related to the results/products on the project level.
Intermediary level Bridging the project level with the systemic level.	Impact on economic, environmental and social sustainability, related to the process and product on the intermediary level.
Systemic level Impacts related to the level beyond the immediate community of stakeholders.	Impact on economic, environmental and social sustainability, related to the process on the systemic level.	Impact on economic, environmental and social sustainability, related to the results/products on the systemic level.

The five-step procedure of the framework was intended to be used as a means of organising stakeholder interactions. The steps included: assembling stakeholders, involving them in impact evaluation, planning a course of action, executing the plan and finally, reflecting on both the process and the results (see Figure 1).

Figure 1. The five-step procedure of the original framework.

The next section presents the emerging lessons learned, experiences and questions derived from the framework testing conducted using the three pilot studies described earlier. The section concludes with the revised version of the framework.

3. Results

3.1. Assembling stakeholders

In the communication material pilot, experts were assembled to examine existing Practice Abstracts. The responses from these experts indicated that the examined Practice Abstracts were rated rather low in general, especially regarding the criteria of relevance and usability in practice. These results indicated that the Practice Abstracts failed to effectively disseminate research outcomes of relevance to practitioners. Hence, there is a considerable potential for improvement in the process of writing Practice Abstracts, for example, by involving stakeholders. For that reason, new guidelines were constructed on how to write useful Practice Abstracts (available upon request to corresponding author).

In the research and education pilots, stakeholders were already involved in the setup of each project (section 2). One of the research projects, however, decided to include an extra stakeholder during the assembly step of the framework procedure (Figure 1). This stakeholder was an industry representative with connections to customers such as wholesalers and public procurement, whom the group saw as adding valuable competencies. This illustrates how the assembly step may be an opportunity for re-evaluating the stakeholder group, even if stakeholders are already tied to a project.

3.2. Impacts on different levels

The framework structure defines impacts across different levels (Table 3). Participants found the impacts on the project level to be considerably easier to imagine and assess than impacts at higher levels. To facilitate the understanding of the ‘intermediary level’, it was specified as the ‘organisational level’, concretised as the organisations represented by individuals who are part of the project level. It was also specified that this level could have both product and process related impacts (see 3.7 for a summary of the developments of the framework).

Effects at the systemic level were considered by stakeholders as difficult to identify and assess. However, there were examples linking project outcomes to higher levels, which suggested a contribution to larger systemic change. For example, representatives from one of the research projects were invited to present their research outcomes to politicians in the national parliament, where the impact potential was clearly at a higher, systemic level. In the case of education, it was observed that both students and professionals had the opportunity to reflect on project aspects that would normally be outside of their educational scope, and the framework’s capacity for training in systemic thinking was thought to be considerable through the organisation and systemic levels.

Furthermore, it was found that applying the framework on an individual basis was particularly relevant, as the personal development of students was a desired educational outcome. Significant areas of individual impact were raised in the interviews that were not necessarily project-related. For example, one of the farmers involved in an undergraduate research project discussed the positive psychological impact of communicating and expressing their views and problems within a group. One student mentioned that it gave them valuable experience in multi-actor communication, which would be useful in a professional context. In another undergraduate research project, the university professor and the lab technician explicitly stated that the capacities developed throughout the project for networking, as well as the inclusion of real economic players within educational activities, would help them evolve their educational activities in the future, thus creating a real, societal, long-term impact on the educational system. In the research pilot, several participants, including researchers and stakeholders, stated that they had gained new insights from each other during the focus group dialogues in the impact assessment process. Based on these findings, the framework was complemented with the individual level (see further in 3.7).

3.3. Impacts on economic, environmental and social dimensions

The framework model identifies economic, environmental and social dimensions of impact (Table 3). In practice, impacts are not always divisible into these general categories, or a given project may not have meaningful impacts in all of these dimensions at all levels. This is illustrated by the fact that the four projects in the research pilot each reflected a different distribution of impacts between the economic, environmental and social dimensions. In the education pilot, some participants found it challenging to think of impacts and indicators related to social and environmental aspects on all levels. The most readily seen impacts were economic at the individual and project levels. Discussions beyond this were a matter of the facilitator prompting and suggesting example indicators for the participants to agree or disagree on, with participants asked to give reasons for their answers. The framework thus became an opportunity to develop the ability to identify and articulate the impacts of the economic, environmental and social dimensions in both the research and educational pilots.

3.4. Developing project specific and universal indicators

In order to measure impacts, indicators were discussed and identified in the research projects and undergraduate research projects, respectively. In the pilot tests, a first set of indicators emerged as related to each project’s goals, such as those stated in the original project description. The project-specific indicators offered significant reference points for the duration of the project and tended to be product-oriented. A second set of indicators was created in the pilot tests to allow for comparability across several research projects. These indicators tended to be process-oriented. The emerging indicators from the data collection were tentatively grouped into five categories: 1) project-specific indicators, 2) communication and cooperation, 3) contribution of resources, 4) social learning and capacity building and 5) dissemination of results (see Table 4). The process of discussing indicators in the pilot groups resulted in considerably more concretised and specified indicators than suggested in the original framework.

Table 4. Examples of indicators, emerging from the pilot testing phase, tentatively grouped into five categories.

Tentative category	Examples of indicators
1) Project-specific indicators	Project specific indicators, for example, from project proposals, etc.
2) Direct communication and cooperation between involved actors	Stakeholders are involved in the project initiative from the start and throughout the process. The number of meetings in the project, face-to-face or digitally. A diversity of involved stakeholders. Level of acceptability of results to stakeholders. Providing a basis for further cooperation between the involved individuals and or/represented organisations.
3) Contribution of resources	Resource commitments by individuals and organisations in the project (e.g. funding, time, materials, networks, knowledge, experiences, other resources).
4) Social learning and capacity building	Increase in social capital, individual learning and motivation. Development of teamwork competencies (communication, collaboration, networking). Expanded networks. Increased empowerment (feeling of choice, competence, meaning and agency). Insights into sustainability issues, systemic and visionary thinking. Increase in capacity/willingness to cooperate.
5) Dissemination of results	Stakeholders or other external groups are engaged in examining drafts of communication materials, for relevance and usability for target groups. Number and type of project publications, such as written reports, Practice Abstracts, films, exhibitions. Dissemination of these publications and materials, and research results used directly in education.

The indicators in the fourth category allows for the assessment of effects regarding the participants’ ability to shape their local conditions, e.g. through more trusting relations in networks and increased capacities among participants to innovate and make decisions about a specific situation. For example, in one of the research projects, rural entrepreneurs gained increased self-efficacy and confidence by knowing that their practices were science-based and that they were using the best practices available. In the educational pilot, students studying the effect of hemp protein in bread were initially planning to stop work in this area upon completion of their thesis. However, as the project progressed and the framework was applied, they decided to proceed with organoleptic testing of the product and to begin producing an academic paper. This progress could be assumed to have emerged through ‘productive interactions’ with the relevant stakeholders and through the development of a perception of trust and a willingness to continue these interactions and their engagement.

3.5. Time frame

Some indicators in Table 4 are meant to be monitored either during or at the very end of the project, while others are best monitored at a certain time after the project has finished. The number of meetings in the project is an example of the first, while further cooperation is an example of the latter. As it is potentially problematic to suggest actions be taken after a project has finished, indicators of this kind could be kept to a minimum. Although the facilitator or project manager may have an idea of when a specific indicator should be tested, stakeholders may give their input on this, as they may have a more accurate idea of an indicator’s capacity to change. Our data suggests four different time frames for measuring indicators: in intervals during the project, upon completion of the project, post project (<2yrs) and long-term effects (>2yrs).

3.6. The role of a facilitator

The education pilot needed a facilitator who was capable of prompting and directing the conversation, as participants were often confused and could not understand what was required of them. Many explanations and examples were generally required, and indicators were often inferred from their responses and rephrased for participants to confirm. This confusion is mostly due to the participants’ lack of experience when thinking about impact, especially long-term and systemic impacts, which were not immediately apparent to most stakeholders. Within an academic setting, it also had a significant educational potential for training competence, such as broadening students’ world views and developing their systems thinking.

In the research pilot study, dialogues were facilitated by Nextfood researchers, but it could be argued that the facilitator role could be performed by a project manager, group leader or similar. Clear instructions introducing the concepts of impact assessment and how these could be translated into indicators of impact would help the project manager to facilitate group agreement on a relevant set of indicators for the project. It would also help in determining how these indicators should be measured, as well as helping them delegate responsibilities to group members for the assessment process. This led to the suggestion to complement the framework procedure (Figure 1) with a preparatory step and a slight restructuring of the subsequent steps, see next section.

3.7. Framework revision after the testing phase

This section summarises the changes made to the framework as a result of the pilot testing phase. The structure of the original framework (Table 3) was redefined, resulting in four impact levels: Individual, Project, Organisational and Systemic levels, along with related product and process categories (see Table 5). In addition, a set of process- and product-related indicators were developed and concretised to further refine the framework (see Table 4).

Table 5. The developed structural matrix of the framework is based on four organisational levels, accounts for both process and product impacts, and each may contain dimensions of economic, environmental or social sustainability.

	Process related impacts Economic, environmental, social	Product related impacts Economic, environmental, social
Individual level The personal, professional and academic development of the individuals directly involved in the project.	Impacts of the process on the involved individuals.	Impacts of the results/products on the involved individuals.
Project level The group of individuals involved in the project working group, reference group and/or stakeholder group.	Impacts of the process at the project level.	Impacts of the results/products at the project level.
Organisational level The organisations that are represented by the individuals who are part of the project level.	Impacts of the process at the organisational level.	Impacts of the results/products at the organisational level.
Systemic level Impacts related to the broader agri-food system, or value chain.	Impacts of the process at the systemic level.	Impacts of the results/products at the systemic level.

Working through the procedural steps of the original framework (Figure 1) resulted in refining them by adding a preparatory step and slightly restructuring the steps (see Figure 2). The preparatory step includes developing a time and resource plan along with a preliminary list of assessment indicators. The flow was made circular instead of linear, reflecting the often iterative nature of group discussions, especially in relation to project stages and sections.

Figure 2. The developed five-step procedure of the framework provides a guide to organise stakeholder interactions and operate the framework in practice.

4. Analysis and discussion

The literature review in section 1.1 highlighted three main questions, which we revisit in this section.

4.1. Adaptability and universality in impact assessment models

Firstly, we address the problem of finding an impact assessment model that integrates economic, environmental and social impacts and caters to the needs of both adaptability and universality. An integrated approach has natural challenges (cf. Horton and Mackay 2003; Fischer and Noble 2015; Weiβhuhn et al. 2018). The Nextfood framework handles this by providing structural and procedural components with the flexibility to allow for the organisation of impact indicators that are both adaptable and universal in nature. The results of the pilots provided project-specific indicators, which tended to be product-oriented, and universal indicators, which were often related to the process. This supports the findings of Spaapen and van Drooge (2011), who suggested the use of universal process-related indicators as a proxy for impact, as they found that an interactive process involving relevant stakeholders increased the likelihood of societal impacts.

Crucially, the data revealed that employing either an adapted or a universal approach did not exclude the other. The results of the education pilot showed that through an impact assessment process that follows a continuous learning approach (cf. Paz-Ybarnegaray and Douthwaite 2017), it was entirely possible to assess and monitor universal process-related indicators that were pre-suggested by the project design while also adapting the framework and producing meaningful project-specific indicators. Similarly, the research pilot showed that it was possible to assess and monitor the product-related goals from the original project proposal while developing universal process-related indicators. Indeed, most cases in the pilot projects indicated the value of applying the framework in an ongoing manner throughout the project duration, as it provided continuous opportunities for stakeholders to develop and relate their work to product- and process-related indicators.

The pilot tests also revealed that if the facilitator or project manager has a set of tentative indicators from the start of the project, these can act as the starting point for conversations with the stakeholders and help guide the thoughts in relation to the levels of the framework matrix. The ex-ante approach of starting the impact dialogues from the very beginning of the project improves the scope, allowing impacts to be identified and captured early in the project.

4.2. Stakeholder participation

Secondly, the literature review addressed the value of seeing stakeholder communities as active participants in the impact assessment process (Pretty 1995; Parthasarathy 2002; Alvarez et al. 2010; Douthwaite and Hoffecker 2017). The view of the ANT literature (Latour 2005), where impact is seen as performed by diverse actors, further highlights the inclusion of actors in the process itself. The Nextfood framework caters to this by encouraging the involvement of a variety of stakeholders at an early stage of the project. In fact, the basic outlook of the framework is that impact assessment is seen as a socially-oriented and socially-embedded activity with the potential to transform actors and networks to create change in the sector and society. This view of more active stakeholder participation and interactions also influences the development of SEA, strategic environmental assessment, according to Sadler and Verheem (2023).

The results from the pilot tests found several illustrations of capacity building, both on a personal and organisational level (Douthwaite and Hoffecker 2017; Faure et al. 2018). In particular, the educational pilot triggered the introduction of an additional, personal level into the framework, accommodating the emergence of indicators related to aspects of personal development, such as competence and values. By reflecting on the processes of the project, stakeholders became aware of how specific aspects of the project design and implementation impacted them beyond the scope of the project, at both an individual and an organisational level. Hence, the individual level indicates a project’s impact on the development of competences valuable to the participants’ personal and professional lives.

The individual level may also provide a way to validate system changes or network effects. When individuals learn and develop their skills or values, they may contribute to a shift within the organisations and networks to which they belong. While such shifts may be hard to capture, the individual level can act as a proxy for broader impacts, assuming that individual shifts have the potential to affect organisations and networks. The framework thus opens up for acknowledging the role of individuals in impact assessment, which calls for further scholarly work. This is in line with the actor-network theory of local actors contributing to the translation of knowledge into practice and possibly transforming themselves and their networks in the process (Latour 2005; Joly et al. 2015).

As an example, the education pilot challenged the conventional view of ‘teachers’ and ‘learners’. It is widely assumed that students and farmers are passive ‘learners’, while professionals are the ‘teachers’. These roles are often assigned almost a priori, with very little opportunity to challenge. They can be very restricting for effective knowledge sharing and creating changes in the mindsets and practices of those considered as ‘the teachers’ or the ‘experienced professionals’. This often creates an additional problem, namely that of low motivation for professionals to participate in multi-stakeholder educational activities. However, through the process of participatory impact assessment, this dynamic was challenged. This was made possible through the impact assessment process, helping with the emergence of common motivators for engagement and commitment among all stakeholders. That is, the process of producing impact indicators highlighted that, in the project, there were indicators specific to each group (e.g. academic performance for students, production of scientific articles for professors, development of more efficient practices for farmers), but there also emerged common indicators of impact (motivators) such as opportunities for networking. This example illustrates the systemic effects of this approach.

4.3. Capturing complex impacts

Thirdly, the literature review highlighted how complex, context-dependent impacts are often challenged when viewed as impacts. Distinguishing between cause and effect from other factors is difficult, especially for effects that may only become apparent after a certain time (cf. La Rovere et al. 2009; Tzilivakis et al. 2011). The Nextfood framework challenges the traditional cause-and-effect attribution view of impact by adopting a systems view, acknowledging the social embeddedness of impact and using the concepts of contribution and productive interactions (Spaapen and van Drooge 2011; Joly et al. 2015). Impact assessment is regarded as a joint responsibility and work for all participants, wherein the planning for the desired impacts and how to evaluate the successes and shortcomings starts early in the project. This facilitates the identification and enabling of impacts. The experiences show that actors learned to think about and exercise impact in new ways through their work with the Nextfood framework.

5. Conclusions

The aim of this article was to analyse the results of the pilot testing of a new multi-dimensional assessment framework for defining and evaluating the societal impact of agricultural research and corresponding education. The study provides new perspectives on impact assessment as a joint, interactive process involving diverse stakeholders, integrating the economic, environmental and social impact dimensions. It encourages thinking and collaboration outside the disciplinary silos. The results show how the economic, environmental and social dimensions can be integrated and possible to use in a practical setting, and how stakeholders learn to think about impact as both a process and a product, understanding their role in creating impacts across different organisational levels and sustainability dimensions.

Following actor-network theory, this paper started from the premise that ‘impact’ is not something out-there, ready to be measured but is something that has to be performed by diverse actors in society, including researchers, institutions, practitioners and other stakeholders. The Nextfood framework assumes impact as something that has to be performed by all relevant actors, making them both accountable for and aware of impact and its assessment. Inviting stakeholders in research and impact assessment processes provides early engagement with societal impacts and results in benefits connected to capacity building at personal, organisational and networks levels.

The theoretical implications include the learning processes involved when actors try to make the effects of their technological and scientific work accountable as ‘impacts’ to both others and to themselves. In this way, we understand the Nextfood framework as providing a structure to these processes, i.e. as something that actors can use as a reference to organise an environment that supports reflexivity and collaboration as a basis for performing impactful work.

The practical implications include that impact assessment frameworks can be more attentive to the social dynamics of impact, take temporality and subjectivity into account, thus enabling joint deliberation and contributions towards future impacts.

The Nextfood framework primarily addresses practice-oriented research projects within agriculture. However, in the next step, the framework can be tested in other sectors, thus contributing to the development of practices that could form the basis of new policy making and impact assessment beyond agriculture. The Nextfood framework has the potential to support the development of a new role for principal stakeholders across a broad range of research and higher education institutions. It enables them to take responsibility for science and education that contributes to concrete and demonstrable sustainability changes in society. The framework is flexible and works in both research and education contexts, allowing for assessment at different organisational levels, e.g. for specific projects, departments or at the university level. Even if the operationalisation of the framework requires someone to act as a facilitator, it has a relatively easy-to-use format that allows usage on a wider scale. The framework will be further strengthened through its use in case studies, both in agriculture and other sectors, with these studies allowing for additional experience to help refine the framework and develop new or revised sets of indicators.

Ethics statement

The research has been conducted under the ethics requirements and guidelines of the NextFood project (Deliverables 8.1, 8.2 and 8.3), all of which comply with regulation (EU) no. 1291/2013 of the European Parliament and of the Council. A data management plan was developed to ensure that data collection and processing was performed in accordance with the GDPR. Written informed consent for the publication of qualitative research data was obtained from all study participants.

Acknowledgments

The authors would like to thank all interviewees for their time and interest in sharing their views. We are also very grateful to Jan Lehejček, Jiří Lehejček, Viktor Květoň, Anna-Maria Krooupa and Georgia Zafeiriou for contributing to the Nextfood work package. A special thank you to Line Lindner and Katherine Flynn at ISEKI-Food Association, a Nextfood partner organisation, for authoring the Practice Abstract writing guidelines.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the grant agreement [No 771738] (Educating the Next Generation of Professionals in the Agrifood and Forestry System [NEXTFOOD]). The funders had no role in the study design, data collection and analysis, decision to publish or preparation of the manuscript.