WORKING DOCUMENT

GUIDELINES

COMMON
ASSESSMENT EVALUATION
OF RDP RESULTS: HOW
QUESTIONS
TO PREPAREFOR RURAL ON
FOR REPORTING
EVALUATION IN 2017
DEVELOPMENT
PROGRAMMES
A 11 - F
NNEX 2014-2020
C E
ICHES FOR ANSWERING OMMON VALUATION
QUESTIONS FOR RURAL DEVELOPMENT PROGRAMMES 2014-
2020
CEQ 1 - 21
MARCH 2015

This document has been drawn up based on the regulations published

in the Official Journal of the European Union on 20 December 2013 and
on therelated delegated andimplementing regulations published in
2014.
Copyright notice

© European Union, 2016

Reproduction is authorised provided the source is acknowledged.

Recommended citation:
EUROPEAN COMMISSION – Directorate-General for Agriculture and Rural Development – Unit E.4 (2016): Guidelines:
Assessment of RDP results: how to prepare for reporting on evaluation in 2017. Brussels.
Disclaimer:
The information and views set out in these Guidelines are those of the author(s) and do not necessarily reflect the official
opinion of the Commission. The Commission does not guarantee the accuracy of the data included in these Guidelines.
Neither the Commission nor any person acting on the Commission’s behalf may be held responsible for the use which may be
made of the information contained therein.

The Evaluation Helpdesk is responsible for the evaluation function within the European Network for Rural Development (ENRD)
by providing guidance on the evaluation of RDPs and policies falling under the remit and guidance of DG AGRI’s Unit E.4
'Evaluation and studies' of the European Commission (EC). In order to improve the evaluation of EU rural development policy
the Evaluation Helpdesk supports all evaluation stakeholders, in particular DG AGRI, national authorities, RDP managing
authorities and evaluators, through the development and dissemination of appropriate methodologies and tools; the collection
and exchange of good practices; capacity building, and communicating with network members on evaluation related topics.

Additional information about the activities of European Evaluation Helpdesk for Rural Development is available on the Internet
through the Europa server ([Link]
Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

GUIDELINES

ASSESSMENT OF RDP RESULTS:

HOW TO PREPARE FOR REPORTING
ON EVALUATION IN 2017

ANNEX 11 - FICHES FOR ANSWERING COMMON

EVALUATION QUESTIONS FOR RURAL DEVELOPMENT
PROGRAMMES 2014-2020

CEQ 1 - 21
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes
2014-2020 (CEQ 1 – 21)

CONTENTS
1. Introduction..................................................................................................................... 1

2. Evaluation Questions related to Focus Areas ................................................................. 5

2.1 Focus Area P1A, Evaluation question 1: To what extent have RDP interventions
supported innovation, cooperation and the development of the knowledge base in rural
areas? ............................................................................................................................ 5

2.2 Focus Area P1B, Evaluation question 2: To what extent have RDP interventions
supported the strengthening of links between agriculture, food production and forestry
and research and innovation, including for the purpose of improved environmental
management and performance? .................................................................................... 8

2.3 Focus Area P1C, Evaluation question 3: To what extent have RDP interventions
supported lifelong learning and vocational training in the agriculture and forestry
sectors? ....................................................................................................................... 12

2.4 Focus area P2A, evaluation question 4: To what extent have RDP interventions
contributed to improving the economic performance, restructuring and modernization of
supported farms in particular through increasing their market participation and
agricultural diversification? ........................................................................................... 16

2.5 Focus Area P2B, Evaluation question 5: To what extent have RDP interventions
supported the entry of adequately skilled farmers into the agricultural sector and in
particular, generational renewal? ................................................................................. 28

2.6 Focus Area P3A, Evaluation question 6: To what extent have RDP interventions
contributed to improving the competitiveness of supported primary producers by better
integrating them into the agri-food chain through quality schemes, adding value to the
agricultural products, promoting local markets and short supply circuits, producer groups
and inter-branch organization? ..................................................................................... 32

2.7 Focus Area P3B, Evaluation question 7: To what extent have RDP interventions
supported farm risk prevention and management?....................................................... 38

2.8 Focus Area P4A, Evaluation question 8: To what extent have RDP interventions
supported the restoration, preservation and enhancement of biodiversity including in
Natura 2000 areas, areas facing natural or other specific constraints and HNV farming,
and the state of European landscape? ......................................................................... 42
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

2.9 Focus Area P4B, Evaluation question 9: To what extent have RDP interventions
supported the improvement of water management, including fertilizer and pesticide
management? .............................................................................................................. 46

2.10 Focus Area P4C, Evaluation question 10: To what extent have RDP interventions
supported the prevention of soil erosion and improvement of soil management? ......... 51

2.11 Focus Area P5A, Evaluation question 11: To what extent have RDP interventions
contributed to increasing efficiency in water use by agriculture? .................................. 57

2.12 Focus Area P5B, Evaluation question 12: To what extent have RDP interventions
contributed to increasing efficiency in energy use in agriculture and food processing? 66

2.13 Focus Area P5C, Evaluation question 13: To what extent have RDP interventions
contributed to the supply and use of renewable sources of energy, of by-products,
wastes, residues and other non-food raw material for purposes of the bio-economy? .. 73

2.14 Focus area P5D, evaluation question 14: To what extent have RDP interventions
contributed to reducing GHG and ammonia emissions from agriculture? ..................... 82

2.15 Focus Area P5E, Evaluation question 15: To what extent have RDP interventions
supported carbon conservation and sequestration in agriculture and forestry? .......... 102

2.16 Focus area P6A, evaluation question 16: To what extent have RDP interventions
supported the diversification, creation and development of small enterprises and job
creation? .................................................................................................................... 106

2.17 Focus Area P6B, Evaluation question 17: To what extent have RDP interventions
supported local development in rural areas? .............................................................. 110

2.18 Focus Area P6C, Evaluation question 18: To what extent have RDP interventions
enhanced the accessibility, use and quality of information and communication
technologies (ICT) in rural areas? .............................................................................. 114

3. Evaluation Questions related to other RDP aspects ................................................... 118

3.1 Operational performance, Evaluation question 19: To what extent have the synergies
among priorities and focus areas enhanced the effectiveness of the RDP? ............... 118

3.2 Technical Assistance, Evaluation question 20: To what extent has technical assistance
contributed to achieving the objectives laid down in Art. 59(1) of Regulation (EU) No
1303/2013 and Art. 51(2) of Regulation (EU) No 1305/2013? .................................... 120

3.3 National Rural Networks (NRN), Evaluation question 21: To what extent has the national
rural network contributed to achieving the objectives laid down in Art. 54(2) of
Regulation (EU) No 1305/2013? ................................................................................ 123
Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes
2014-2020 (CEQ 1 – 21)

1. INTRODUCTION

In section 7 of the annual implementation report submitted in 2017, the Managing Authorities will have
to provide information resulting from evaluation activities regarding the quantification of programme
achievements, in particular through the assessment of result indicators and the answers to the relevant
evaluation questions.

The European Evaluation Helpdesk for Rural Development has developed the Guidelines “Assessment
of RDP results: how to prepare for reporting on evaluation in 2017”, which guide stakeholders in
Members States in preparing, conducting and reporting on the assessment of RDP results in 2017, and
provide the foundation for the RDP evaluation at later stages of the programming period 2014-2020.

Annex 11 of the above Guidelines has been published separately and provides guidance in the form of
CEQ fiches on how to answer each of the common evaluation questions (CEQs) to be reported on in
the AIR 2017, namely CEQ number:

• 1 – 18, relating to each rural development focus area (FA) and

• 19 – 21 relating to other RDP aspects (programme synergies, technical assistance and national
rural networks) 1.

The information in the CEQ fiches:

• complement

o the Working Document on Common Evaluation Questions for Rural Development

Programmes 2014-2020,

o the Working Document on Complementary Result Indicators Fiches for Pillar II,

o the Working Document on Target Indicators Fiches for Pillar II, and

o the Guidelines: Assessment of RDP results: how to prepare for reporting on evaluation in
2017.

• provide additional guidance for answering common evaluation questions mentioned above with the
help of common result/target indicators, complementary result indicators (CEQ: 4, 11, 12,13 and
14), output and common context indicators and additional indicators. Additional indicators are
developed by Member States in case the:

o common indicators are not sufficient to answer the CEQ and to assess the secondary
contributions of the RDP’s operations to the focus area other than how they have been
programmed;

o stakeholders in the Member State employed additional judgment criteria linked to the CEQ.

1
WD Common Evaluation Questions for Rural Development Programmes 2014-2020,
[Link]
1
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

The table below shows the structure and content of the CEQ fiches

Evaluation question: Title of the evaluation question and its related Focus Area

List of measures contributing to the FA linked to the CEQ (only valid for CEQ 1 – 18 and 21):

• primarily programmed measures/sub-measures and

• examples of measures/sub-measures programmed under the other FA, but showing the secondary
contributions to this FA.

Consistency check between CEQ, judgement criteria and indicators

Prior to answering the CEQ, one should make consistency checks between the CEQ, judgment criteria and the
indicators. If data gaps arise during this consistency check (which would impend the answering of the CEQ),
additional judgment criteria and indicators should be developed by the Member State.

Judgement criteria

The Working Document on Common Evaluation Questions for Rural Development Programmes 2014-2020
proposes a set of judgement criteria for each CEQ. Stakeholders in the Member States might add/change the
judgement criteria according to the intervention logic of the FA (selection and combination of measures/sub-
measures).

Indicators

CEQs should preferably be answered by means of common indicators suggested in the Working Document on
Common Evaluation Questions for Rural Development Programmes 2014-2020 or other common indicators
presented in the CMES (output and common context indicators). The Working Document further proposes
Member States collect additional information. Moreover, stakeholders in the Member States might develop
additional indicators. All common and additional indicators are listed in this part.

Qualitative indicators and assessment

Qualitative indicators can be useful in situations where data is not available; and also in order to provide more
explanatory arguments when answering the evaluation questions. The assessment of qualitative indicators
covers using of qualitative methods based on collecting qualitative information, theory of change, etc.

Data needs and data sources

Indicators Data needs * Data sources **

Indicators
Common
Additional
indicators

2
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

*Data needs

The data needs are listed here. This includes:

a) the necessary data for the common indicators (usually monitoring data) as defined in the Working Document
on target indicator fiches for Pillar II; and the Working Document on complementary result indicator fiches for
Pillar II.

b) additional data and/or information (optional) to provide values for additional indicators and to give
explanations/interpretations of the indicator values (common and where they exist, additional indicators).

** Data sources

This part indicates possible data sources.

Whenever possible, data on beneficiaries should be collected via the operations database (application forms,
payment requests, monitoring tables, etc.). Links to the EU level data sources are provided whenever possible.
In specific cases for the complementary result indicators (CRI) the national/regional coefficients or
national/regional EU level data sources to calculate indicators are also provided.

Timing

The best moment where data should be collected to answer evaluation questions is provided in this section.

Methodology to calculate common and additional indicators (quantitative, qualitative, mixed)

The proposed methodology to collect and analyse data and information is explained here.

Taking into consideration the secondary contributions of operations towards other FAs other than those under
which they have been programmed. For more detail on methods, references to specific chapters of the relevant
Guidelines or to other related documents are provided.

A variety of quantitative and qualitative methods are proposed in order to achieve triangulation of the findings
from different sources.

Possible challenges/risks/issues

A list of challenges, risks and/or issues that may hinder or provide difficulties in answering the evaluation question
are listed here.

Proposed solutions

Solutions are proposed for each challenge/risk/issue.

Answer to CEQ

Answers are provided based on 1.) the calculated values of the indicators 2.) the collected qualitative
information or 3.) the qualitative assessment.

Conclusions Recommendations

The main conclusions are stated here for the FA on The main recommendations linked to the conclusions
the basis of the evaluation’s findings. are stated here for the FA.

Using the fiche for programme-specific evaluation questions (PSEQs)

If stakeholders in the Member States define PSEQs for the RDP evaluation 2017, the table above will
serve to both help stakeholders define these PSEQs, as well as act as a source of guidance on how to
conduct the evaluation and provide reliable answers. The PSEQ can be formulated in cases when:

3
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

• the RDP contains a programme-specific focus area that cannot be captured with the CEQs. The
programme-specific focus area is usually developed for national (territorial) priorities, which are not
covered with the EU’s rural development focus areas;
• stakeholders are required to evaluate specific evaluation topics (e.g. RDP delivery, administration,
etc.);
• the programme shows potential indirect, unexpected (positive/negative) effects, which should be
assessed and are not covered by any other evaluation questions.

If the above fiche is used to develop the fiche for PSEQ it should also contain in addition:
• the rationale behind using the PSEQ,

• a list of measures contributing to the programme specific focus area objectives (primarily and
secondarily) in the case of a programme-specific focus area related evaluation question.
• programme-specific judgement criteria and indicators – common and programme-specific to be
used to answer the PSEQ.

4
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

2. EVALUATION QUESTIONS RELATED TO FOCUS AREAS

2.1 Focus Area P1A, Evaluation Question 1

To what extent have RDP interventions supported innovation, cooperation and the development of the
knowledge base in rural areas?

List of measures contributing to the FA 1A

Programmed measures/sub-measures (proposed by EC) 2:

• Measures and sub-measures of Art. 3 15 “Advisory services, farm management and farm relief services”.

• Measures and sub-measures of Art. 14 “Knowledge transfer and information actions”.

• Measures and sub-measures of Art. 35 “Co-operation”.

Measures programmed under other focus areas than FA 1, and contributing secondarily to the objective of FA
1A should be considered.

Consistency check between CEQ, judgement criteria and indicators:

Judgement criteria

Judgement criteria (JC) proposed in the Working Document on Common Evaluation Questions for Rural
Development Programmes 2014- 2020:

• RDP projects have been innovative and based on developed knowledge

• Operational groups have been created

• Variety 4 of partners involved in EIP operational groups

• Innovative actions have been implemented and disseminated by the EIP operational groups

Note: stakeholders in the Member States might add/change judgement criteria in line with the intervention logic
of the FA (selection and combination of measures)

Indicators

The following common indicators should be used to answer the CEQ:

Common target indicator:

• T1 - % of expenditure under Articles 14, 15 and 35 of Regulation (EU) No 1305/2013 in relation to the total
expenditure for the RDP

The following other common indicators may be used to answer the CEQ:

Common output indicators 5 (data collected via operation database):

• O13 Number of beneficiaries advised

• O16 Number of EIP operations

• O16 Number and type of partners in EIP groups

2
WP: Guidelines for strategic programming for the period 2014-2020, European Commission, 04/03/2014, published on
ECAS/circabc
3
All articles mentioned in this section of each fiche for answering CEQ 1 – 18 are of Regulation (EU) No 1305/2013, unless it is
stated otherwise in the text directly
4
Variety is defined by the representation of different socio-economic sectors (private, public, civil, agriculture, food industry,
forestry, etc.) and organizations such as academia, banks, NGO, etc.
5
WD: Data item list for Pillar II operations database and WD RD programming and target setting
5
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Common context indicator (data needs and sources are described in the Working document: Proposed list of
common context indicators):

• CCI 24 Agriculture training of farm managers

Additional indicators and information

Common indicators are not sufficient to answer the CEQ. Working Document: Common Evaluation Questions
for Rural Development Programmes 2014-2020 recommends the collection of the following additional
information:

• % of innovative RDP projects (ratio of innovative projects to total number of projects)

• Composition of EIP operational groups (number and types of partners) 6

• Number of supported innovative actions implemented and disseminated by EIP operational groups divided
by type, sector etc.

The following additional indicator is proposed to be used in answering the CEQ:

• Number of operational groups created

Note: More additional indicators can be developed if needed, e.g. when judgement criteria are added to specify
the evaluation question in the Member State.

Qualitative indicators

For qualitative assessment the following qualitative indicators can be used:

• Type and content of innovation, (description of the innovation created and it’s use by beneficiaries and/or
non-beneficiaries),

• cooperation and knowledge base created in rural areas,

• description of the key factors that have contributed to innovation, cooperation and the development of
knowledge base in rural areas

Data needs and data sources

Indicators Data needed Data sources

T1 - % of expenditure under Articles Realised expenditure for completed Beneficiaries:

14, 15 and 35 of Regulation (EU) operations for measures 1, 2 and 16
Application form (project start)
No 1305/2013 in relation to the total and also, where possible, having
expenditure for the RDP regard to the stage of programme Payment request (project end)
implementation for selected
Common indicators

operation 7 (data item O1)

O.13 – number of beneficiaries Number of beneficiaries Beneficiaries:

supported
Application form (project start)

Payment request (project end)

O.16 - Number of EIP operations Number EIP operations Beneficiaries:

Application form (project start)

Payment request (project end)

6
Similar to common output indicator O16
7
Art. 50 of the Regulation (EU) No 1303/2013.
6
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

O.16 - Number and type of partners Number and type of partners in EIP Beneficiaries:
in EIP groups groups
Application form (project start)

Payment request (project end)

% of innovative RDP projects (ratio Number of innovative projects Beneficiaries:

of innovative projects to total implemented by the RDP measures
Application form (project start)
number of projects) 1, 2 and 16
Payment request (project end)
Total number of projects
implemented by the RDP measures
Additional indicators

1, 2 and 16

Total number of innovative projects

implemented by the RDP

Number of operational groups Number of operational groups Beneficiaries:

created created
Application form (project start)

Payment request (project end)

Number of supported innovative Number of innovative actions Beneficiaries:

actions implemented and implemented and disseminated by
Application form (project start)
disseminated by EIP operational operational groups
groups Payment request (project end)

Timing of data collection

Proposed data on beneficiaries should be collected starting from the beginning of the RDP’s implementation
(1st call for proposal, operations database), for % of cooperation projects continuing after the RDP’s support at
the time of the evaluation (evaluator).

Methodology to calculate common and additional indicators

Common indicator

The common indicator T1 is calculated as a ratio of expenditures to total expenditures, based on the data
collected via the operations database.

Additional indicator

The indicator is assessed via a survey to operational groups:

• number and type of operations,

• number and type and role of different partners.

Qualitative assessment

Proposed methods are:

• Interviews with partners in operational groups

• Structured focus groups 8, including the MAPP method 9

8
For details on how to use surveys and focus groups, see ‘Capturing the success of your RDP: Guidelines for ex post
evaluation of 2007-2013 RDPs’, Evaluation Helpdesk, June 2014
9
Study: Investment support under Rural Development policy, DG Agri, 2014: [Link]
development-reports/2014/investment-support-rdp/fulltext_en.pdf
7
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

• Delphi method (a structured process for collecting and synthesising knowledge from a group of experts
through a series of questionnaires, accompanied by controlled opinion feedback) 10, e.g. with the focus on
innovation.

Triangulation of the findings from different sources (indicator values, findings from interviews, surveys and focus
groups)

Challenges/Risks/Issues

• Low quality data that does not provide evidence based answers to the EQ

• Lack of data to answer the EQ (e.g. low RDP uptake/no uptake, small programmes, etc.).

Proposed solutions

In the case of low uptake/small programmes, this EQ can be answered with the data collected for the respective
indicators from the total population of beneficiaries. In the case of no uptake the CEQ can be answered with the
qualitative assessment/ estimation (via interviews and expert opinions) of the extent to which RDP interventions
can support innovation, cooperation and the development of the knowledge base in rural areas. The Delphi
method (see above) can serve as a good approach as it is used to generate forecasts and serves to shed light
on the evolution of the situation. Using the theory of change is another proposed approach to answer the CEQ
in the situation of no uptake.

In the case of low quality data, information can be triangulated through interviews and surveys.

Answer to CEQ

Answer is provided based on the calculated values of indicators, on the collected information from the
qualitative indicators and/or on the qualitative assessment.

Conclusions Recommendations

Main conclusions are stated here for the FA based on Main recommendations linked to the conclusions are
the evaluation findings. stated here for the FA.

2.2 Focus Area P1B, Evaluation Question 2

To what extent have RDP interventions supported the strengthening of links between agriculture, food production
and forestry and research and innovation, including for the purpose of improved environmental management and
performance?

List of measures contributing to FA 1B:

Programmed measures/sub-measures (proposed by EC) 11:

• Measures and sub-measures of Art. 12 35 Co-operation

Measures programmed under other focus areas than FA 1, and contributing secondarily to the objective of FA
1B should be considered.

10
For a detailed description of the Delphi method, see Chapter 5 of ‘Evalsed Sourcebook: Methods and Techniques’:
[Link]
11
WP: Guidelines for strategic programming for the period 2014-2020, European Commission, 04/03/2014, published on
ECAS/circabc
12
All articles mentioned in this section of each fiche for answering CEQ 1 – 18 are of Regulation (EU) No 1305/2013, unless it
is stated otherwise in the text directly
8
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Consistency check between CEQ, judgement criteria and indicators:

Judgement criteria (JC)

Judgment criteria proposed by the Working Document: Common Evaluation Questions for Rural Development
Programmes 2014- 2020:

• Long term collaborations between agriculture, food production and forestry entities and institutions for
research and innovation have been established.

• Cooperation operations between agriculture, food production, forestry, research and innovation for the
purpose of improved environmental management and performance have been implemented

Note: stakeholders in the Member State might add/change judgment criteria in line with the intervention logic of
the FA (selection and combination of measures)

Indicators

The following common indicators should be used primarily to answer the CEQ:

Common target indicator:

• T2 - Total number of co-operation operations supported under the cooperation measure (Art. 35 of
Regulation (EU) No 1305/2013) (groups, networks/clusters, pilot projects…)

The following other common indicators might be used to answer the CEQ:

Common output indicators 13 (data collected via operation database):

• O4 Number of holdings/beneficiaries supported

• O16 Number of EIP operations

• O16 Number and type of partners in EIP groups

• O17 Number of other cooperation operations (groups/networks/clusters/ pilot projects)

Common context indicators can be also used to answer the CEQ. Their selection depends on the type of
cooperation operations.

Additional indicators and information

Common indicators are not sufficient to answer the CEQ. Working Document: Common Evaluation Questions
for Rural Development Programmes 2014-2020 suggests:

• % of cooperation operations continuing after the RDP support including for the purpose of improved
environmental management and performance (if suitable for 2017 and in each case for the enhanced AIR
2019 and the ex-post evaluation),

• Number and types of partners involved in cooperation projects

The following additional indicator is proposed to be used in answering the CEQ:

• Number of all cooperation projects in the field of environmental management and performance divided by
type (including their content and results envisaged and produced)

Note: More additional indicators can be developed when judgement criteria are added to specify the evaluation
question in the Member State.

Qualitative indicators

For qualitative assessment the following information can be gathered:

13
WD: Data item list for Pillar II operations database and WD RD programming and target setting
9
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

• Description of the type, content and quality of cooperation projects and the results envisaged and/or
produced with respect to research and innovation;

• Description of the key factors that contribute to effective cooperation, especially cooperation for improved
environmental management and performance.

Data needs and data sources

Indicators Data needed Data sources

T2 - Total number of co-operation Number of EIP operations (data item Beneficiaries:

Common indicators

operations supported under the O.17)

Application form (project start)
cooperation measure (Art. 35 of
Number of other cooperation
Regulation (EU) No 1305/2013) Payment request (project end)
operations (groups,
(groups, networks/clusters, pilot
networks/clusters, pilot projects…) to
projects…)
be supported under measure 16
‘Cooperation’ (data item O.17)

% of cooperation operations Number of cooperation operations Beneficiaries:

continuing after the RDP support that continue after the RDP support.
Monitoring tables
including for the purpose of improved
Number of cooperation operations
environmental management and Interviews and surveys to
for the purpose of improved
performance cooperation partners
environmental management and
Additional indicators

performance that continue after the

RDP support

Number and types of partners Number and types of partners in Beneficiaries:

involved in cooperation projects cooperation projects
Application form (project start)

Payment request (project end)

Number of all cooperation projects Number of cooperation operations Beneficiaries:

divided by type of which in the field of divided by the type out of those
Application form (project start)
environmental management and related to improved environmental
performance management and performance Payment request (project end)

Timing of data collection

Proposed data on beneficiaries should be collected starting from the beginning of the RDP’s implementation (1st
call for proposal, operations database), for % of cooperation projects continuing after the RDP’s support at the
time of the evaluation (evaluator).

10
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Methodology to calculate common and additional indicators

Common indicators

The common indicator T2 is collected via the operations database and summed up to the total number of
cooperation operations under the cooperation measures.

Additional indicators

The indicator is calculated as the ratio of total cooperation operations, which continue after the RDP’s support
ends based on the ad hoc survey among beneficiaries, with the representative sample of those which conducted
cooperation operations for improved environmental management and performance.

Qualitative assessment

The proposed methods are:

• Survey to cooperation projects and to final beneficiaries

• Structured focus groups 14

• Delphi method (a structured process for collecting and synthesising knowledge from a group of experts
through a series of questionnaires, accompanied by controlled opinion feedback) 15

Triangulation of the findings from different sources (indicator values, findings from interviews, surveys, focus
groups)

Challenges/Risks/Issues

• Low quality data that does not provide a robust answer to the EQ

• Lack of data to answer the EQ (e.g. low RDP uptake/no uptake, small programmes etc.)

• Number of cooperation operations that continue after the RDP’s support ends will not be known until later
on in the programming period (if relevant for 2017, in each case for the AIR submitted in 2019 and ex post
evaluation).

Proposed solutions

In the case of low quality data, information can be sought through interviews and surveys.

In the case of a lack of data (low RDP uptake and small programmes) this EQ will be answered with data collected
from the entire population of beneficiaries. In the case of no uptake the answer can be based on an expert´s
estimation of expected cooperation projects and expected results of these cooperations, for instance in relation
to the improvement of environmental management.

To assess long-term cooperation (duration after RDP support), the types of cooperation structures created (e.g.
legal structure, composition, statement of commitment, etc.), that may point towards potential sustainable
cooperation, can be analysed as part of the qualitative assessment.

14
For details on how to use surveys and focus groups, see ‘Capturing the success of your RDP: Guidelines for ex post
evaluation of 2007-2013 RDPs’, Evaluation Helpdesk, June 2014
15
For a detailed description of the Delphi method, see Chapter 5 of ‘Evalsed Sourcebook: Methods and Techniques’:
[Link]
11
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Answer to CEQ

Answer is provided based on the calculated values of indicators, on the collected qualitative information or on the
qualitative assessment

Conclusions Recommendations

Main conclusions are stated here for the FA based on the Main recommendations linked to the conclusions are
evaluation findings. stated here for the FA.

2.3 Focus Area P1C, Evaluation Question 3

To what extent have RDP interventions supported lifelong learning and vocational training in the agriculture and
forestry sectors?

List of measures contributing to the FA 1C:

Programmed measures/sub-measures (proposed by EC) 16:

• Measures and sub-measures of Art. 17 14 Knowledge transfer and information actions

Measures programmed under other focus areas than FA 1, and contributing secondarily to the objective of FA 1C
should be considered.

Consistency check between CEQ, judgement criteria and indicators:

Judgement criteria

Judgement criteria (JC) proposed by the Working Document: Common Evaluation Questions for Rural
Development Programmes 2014- 2020:

• The number of rural people who have finalised lifelong learning and vocational training in the agriculture and
forestry sectors has increased

Note: stakeholders in MS might add/change judgement criteria in line with the intervention logic of the FA
(selection and combination of measures)

Indicators

The following common indicators should be used to answer the CEQ:

Common target indicator:

• T3 - Total number of participants trained under Art. 14 of Regulation (EU) No 1305/2013

The following other common indicators might be used to answer the CEQ:

Common output indicators (data collected via operation database) 18:

• O11 Number of training days given

• O12 Number of participants in training

• O14 Number of advisors trained

16
WP: Guidelines for strategic programming for the period 2014-2020, European Commission, 04/03/2014, published on
ECAS/circabc
17
All articles mentioned in this section of each fiche for answering CEQ 1 – 18 are of Regulation (EU) No 1305/2013, unless it
is stated otherwise in the text directly
18
WD: Data item list for Pillar II operations database and WD RD programming and target setting
12
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Common context indicator (data need and data sources are described in the WD: Proposed list of common context
indicators):

• CCI 24 Agriculture training of farm managers

Additional indicators and information

Common indicators are not sufficient to answer the CEQ. WD Common Evaluation Questions for Rural Development
Programmes 2014-2020 suggests:

• % of trainees receiving certificates from recognized educational and training institutions via activities supported
by RDP out of the total number of participants

The following additional indicators are proposed to be used in answering the CEQ:

• Number and % of knowledge transfer and information actions divided by type/content that have been supported
by the RDP in lifelong learning and vocational training in agriculture and forestry of total number of trainings in
agriculture and forestry

• Number/percentage of RDP training activities divided by type/content linked to the agriculture and forestry
sectors main needs

Note: stakeholders in Member States might add/change judgement criteria in line with the intervention logic of
the FA (selection and combination of measures)

Qualitative indicators
Qualitative indicators might be also used, e.g. to assess the quality and type of training and aims of the training that
have been supported.

Data needs and data sources

Indicators Data needed Data sources

T3 - Total number of participants Number of participants trained under Beneficiaries:

indicators
Common

trained under Art. 14 of Regulation measure 1 ‘Knowledge transfer and

Application form (project start)
(EU) No 1305/2013 information actions’ (1.1. vocational
training) (data item O.12) Payment request (project end)

% of trainees receiving certificates Number of trainees supported by the Beneficiaries:

from recognized educational and RDP receiving certificates from
Application form (project start)
training institutions via activities recognized educational and training
supported by the RDP institutions Payment request (project end)

Total number of trainees receiving Survey to the participants of

Additional indicators

certificates from recognized knowledge transfer and information

educational and training institutions actions
out of it those with vocational
Non-beneficiaries:
training/lifelong learning in agriculture
National statistics
and forestry
Interviews to educational and training
institutions

Number of knowledge transfer and Number of Knowledge transfer and Interviews with managers of RDP
information actions divided by type / information actions divided by the interventions in the field of lifelong
content that have supported lifelong type/content that supported lifelong learning and vocational training

13
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

learning and vocational training in learning and vocational training

agriculture and forestry activities

Number / percentage of RDP training Number or % of lifelong learning and Beneficiaries:

activities divided by type/content vocational training activities that
Additional indicators

Application form (project start)

linked to the agriculture and forestry include an assessment of labour
sectors main needs market needs Payment request (project end)

Non-beneficiaries:

Survey/focus group to providers of

training activities

Diagnosis of the training needs

Timing of data collection

Proposed data on beneficiaries should be collected starting from the beginning of the RDP’s implementation (1st
call for proposal, operations database) for non-beneficiaries at the time of evaluation (evaluator).

Methodology to calculate common and additional indicators

Common indicators

The common indicator T3 is collected via the operations database and summed up to total number of participants
trained.

Additional indicators

All 3 additional indicators are calculated as the ratio of RDP support of the total amount.

Qualitative assessment

For qualitative assessment, the proposed methods will enable: a) interpret the quantitative values of the indicators;
b) assess the net effect of the RDP on lifelong learning e.g. if the participants can apply the knowledge in their
economic activities and what the perception of the results of the training as well as the effectiveness of lifelong
learning (i.e. does it bring participants closer to the needs of their economic activities).

Proposed methods are:

• Interviews to educational and training institutions

• Surveys to the participants of knowledge transfer and information actions

• Structured focus groups 19, including MAPP focus groups 20

• Interviews with managers of RDP interventions in the field of lifelong learning and vocational training

Triangulation of the findings from different sources (indicator values, findings from interviews, surveys, focus
groups, diagnosis of the training needs).

19
For details on how to use surveys and focus groups, see ‘Capturing the success of your RDP: Guidelines for ex post
evaluation of 2007-2013 RDPs’, Evaluation Helpdesk, June 2014
20
Investment support under Rural Development policy, DG Agri, 2014: [Link]
development-reports/2014/investment-support-rdp/fulltext_en.pdf
14
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Challenges/Risks/Issues

• Low quality data that does not provide a robust answer to the EQ

• Lack of data to answer the EQ (e.g. low RDP uptake/no uptake, small programmes, etc.)

Proposed solutions

In the case of a lack of data (low uptake/small programmes), the CEQ will be answered with data for indicators
collected from the whole population of beneficiaries contracted. In the case of no uptake the answer can be
developed based on an estimation (via interviews and expert opinions) of envisaged training activities, their
content, estimated number of participants, target groups and expected results. The Delphi method 21 may be a
good approach as it is used to generate forecasts and serves to shed light on the evolution of the situation.

In the case of a lack of data, information can be sought through interviews, surveys, focus groups.

Answer to CEQ

The answer is provided based on the calculated values of indicators, on the collected qualitative information or on
the qualitative assessment.

Conclusions Recommendations

Main conclusions are stated here for the FA based on the Main recommendations linked to the conclusions are
evaluation findings. stated here for the FA.

21
For a detailed description of the Delphi method, see Chapter 5 of ‘Evalsed Sourcebook: Methods and Techniques’:
[Link]
15
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

2.4 Focus area P2A, evaluation question 4

To what extent have RDP interventions contributed to improving the economic performance, restructuring and modernization of supported farms in particular through increasing
their market participation and agricultural diversification?

List of measures contributing to the FA 2A:

Primarily programmed measures/sub-measures (proposed by EC) 22:

• Measures and sub-measures of Art. 23 35 Co-operation

• Measures and sub-measures of Art. 19 Farm and business development

• Measures and sub-measures of Art. 17 Investments in physical assets

• Measures and sub-measures of Art. 15 Advisory services, farm management and farm relief services

• Measures and sub-measures of Art. 14 Knowledge transfer and information actions

Examples of measures/sub-measures programmed under the other FA but potentially showing the secondary contribution to this FA:

• All above measures if programmed under other FAs than FA 2A and contributing to improving the economic performance, restructuring and modernization of supported farms

• Measures and sub-measures of Art. 16, Quality schemes for agriculture products and food stuff

• Measures and sub-measures of Art. 27 Setting up of producers groups and organisations

• Measures and sub-measures of Art. 30 Natura 2000 and Water framework directive payments

• Measures and sub-measures of Art. 31 Payments to areas facing natural or other specific constrains

• Measures and sub-measures of Art. 35 of Regulation (EU) no 1303/2013, Support for Leader local development

Consistency check between CEQ, judgement criteria and indicators:

Judgement criteria

Judgement criteria (JC) proposed by the Working Document: Common Evaluation Questions for Rural Development Programmes 2014- 2020:

22
WP: Guidelines for strategic programming for the period 2014-2020, European Commission, 04/03/2014, published on ECAS/circabc
23
All articles mentioned in this section of each fiche for answering CEQ 1 – 18 are of Regulation (EU) No 1305/2013, unless it is stated otherwise in the text directly

16
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

• Agricultural output per annual working unit of supported agricultural holdings has increased

• Farms have been modernized

• Farms have been restructured

Above judgment criteria could be completed if they do not specify sufficiently expected success of the intervention, additional judgment criteria could be, e.g.:

• Farms’ economic performance has improved

• Farms’ market participation has increased

• Farms’ agricultural diversification has increased

Note: stakeholders in the Member States might add/change judgement criteria in line with the intervention logic of the FA (selection and combination of measures)

Indicators

The following common indicators should be used to answer the CEQ:

• R1/T4 - % of agriculture holdings with RDP support for investments in restructuring or modernisation

• R2 - Change in agricultural output on supported farms/AWU (Complementary result indicator)

The following other common indicators might be used to answer the CEQ:

Common output indicators 24 (Data are collected via Pillar II operations database):

• O3 – Number of operations supported

• O4 – Number of holdings supported for investment in agriculture holdings (for calculation of R1)
Common context indicators 25:

• CCI 26 Agriculture entrepreneurial income

• CCI 27 Total factor productivity income

24
WD: Data item list for Pillar II operations database and WD RD programming and target setting
25
Data needs and data sources for common context indicators are described in the Working document: Proposed list of common context indicators.

17
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

• CCI 14 Labour productivity in agriculture

Additional indicators and information:

Working Document: Common Evaluation Questions for Rural Development Programmes 2014-2020 suggests:

• Economic farm size structure of supported farms

Further examples of possible additional result indicators:

• Gross Farm Income (see JC: economic performance)

• Family farm income (see JC: economic performance)

• Gross investment on fixed assets /agriculture output (see JC: economic performance)

• Net investment on fixed assets / agric. Output (see JC: modernisation/restructuring)

• Sales/total output (see JC: market participation)

• % of agricultural output (by product) (see JC: diversification (1)

• % of agricultural income to total income (see JC: diversification (2)

Note: More additional indicators can be developed when judgement criteria are added to specify the evaluation question in MS.

Qualitative information

The proposal is to collect also qualitative information, e.g. whether farms have increased their market participation with the help of the RDP support (Likert scale).

18
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Data needs and data sources

Indicators Data needed Data source

R1/T4% of agriculture holdings with RDP Total number of farm holdings receiving support for investment in restructuring and/or Application forms/payment request
support for investments in restructuring or modernisation under measure 4 (collected via Pillar II operation database as O4 indicator) (beneficiaries - operations database)
modernisation
Total number of agricultural holdings in base year for the RDP area (CCI 17) collected via Eurostat or national statistics: Farms
statistics (national, Eurostat) structure survey (total farm population)
Common result indicators

R2 (Complementary result indicator) Calculation of programme effects using indicator R2 requires collection of data for both Application forms/payment request
beneficiaries of RD measures affecting P2A and appropriate control groups: (beneficiaries - operations database)

• Numerator: Agricultural output (output of crops and crops products, livestock and livestock Survey on beneficiaries and non-
products = value of sales + balance of stocks + own use or consumption) per farm in years beneficiaries and/or
prior to receiving support from the RDP (i.e. 2013) and after support (i.e. 2016, 2018 and
FADN data base and anonymised paying
ex post)
agency data (no information which could
• Denominator: Total labour input of holding expressed in annual work units (AWU full-time identify entity or person) needed to identify
person equivalents) (= family and unpaid labour AWU + paid labour AWU) per farm in beneficiaries of RDP measures linked to
years prior to receiving support from RDP (i.e. 2014) and after support (i.e. 2016, 2018 focus area 2A 26
and ex post).
Eurostat: Farm structure survey
• Relevant GIS data (to be used for evaluation of environmentally related measures)
Standard outputs per MS and region and
• Information on the total amount of subsidies (RDP subsidies directly related to focus area year – average 2010:
2A + RDP subsidies non-directly related to focus area 2A + non-RDP subsidies) obtained [Link]
by a farm in respective periods before the current programme and during its /so-coefficients

26
Questionnaire of FADN for 2014: to see what data are available for the rural development measures - table M Subsidies: [Link]
content/EN/TXT/PDF/?uri=CELEX:32012R0385&rid=1

19
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

implementation (2014-2016; or 2014-2018; or 2014-2020 <+2 years>) – to be used as Total standard outputs (overall economic
control variables and for the calculation of programme efficiency size of farm) per MS and region and per
year:
In order to calculate effect of RD measures on the focus area P2A using result indicator R2
[Link]
the change of the net-value of the R2 indicator should be estimated, see: description of
e/report_en.cfm?dwh=SO
methodology.
Regional/National agricultural statistics, e.g.
farm bookkeeping data, standard outputs

National institutions: GIS data

Economic farm size structure of supported To be collected for beneficiaries and non-beneficiaries (control groups) FADN
farms
Economic size = economic size of holding expressed in European size units (on the basis of [Link]
Community typology) = Total standard output (SO) of the holding expressed in Euro. The e/database_en.cfm
Additional indicators

SO of the holding is calculated as the sum of the SO of each agricultural product present in
Eurostat: Farms structure survey
the holding multiplied by the relevant number of hectares or heads of livestock of the holding.
Regional/National agricultural statistics (e.g.
farm bookkeeping data)

Gross Farm Income To be collected for beneficiaries and non-beneficiaries (control groups) FADN;

Gross Farm Income (GFI) = SE410 = Output – intermediate consumption + balance current Farms structure surveys
subsidies & taxes
Regional/National agricultural statistics (e.g.
farm bookkeeping data)

20
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Family farm income To be collected for beneficiaries and non-beneficiaries (control groups) FADN;

Family Farm Income (FFI) = SE420 = farm net value added – total external factors + balance Farm structure surveys
on subsidies and taxes on investment
Eurostat - Economic accounts for agriculture
(EAA) for computation of context indicators

Eurostat - Agriculture Labour Input statistics

(ALI) for computation of context indicators

[Link]
a-z/abc

Regional/National agricultural statistics, e.g.

farm bookkeeping data

Gross investment on fixed assets /agric. To be collected for beneficiaries and non-beneficiaries (control groups) FADN;
output
Gross investment on fixed assets = SE516 = Purchases – sales of fixed assets + breeding Farm structure surveys,
livestock change of valuation (divided by total agric. Output)
Regional/National agricultural statistics, e.g.
farm bookkeeping data

Net investment on fixed assets / agric. To be collected for beneficiaries and non-beneficiaries (control groups)
output
Net investment on fixed assets = gross investment – depreciation (divided per total agric.
Output)
Additional indicators

Sales/total output To be collected for beneficiaries and non-beneficiaries (control groups)

Sales/ (total output) = share of marketed agricultural output

% of agricultural output (by product) To be collected for beneficiaries and non-beneficiaries (control groups)

% of agricultural output (by product)

% of agricultural income to total income To be collected for beneficiaries and non-beneficiaries (control groups)

% of agricultural income to total income

21
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Timing of data collection

Proposed data on beneficiaries and non-beneficiaries should be collected prior to RDP implementation (i.e. years 2010-2013) and during RDP implementation (i.e. 2014-2020: 1st
call for proposal, operations database, national/regional statistics, EU sources)

Quantitative method to calculate the complementary result indicator R2 - change in Agricultural output/AWU (= farm labour productivity) 27

Complementary result indicator R2 (Agricultural output/AWU) = farm labour productivity) can be interpreted as a partial measurement of the competitiveness of the agricultural sector.
Both primary (programmed directly under the FA 2A) and secondary (programmed under other FAs than 2A, but contributing to 2A) contributions of all relevant RDP measures to
this indicator have to be taken in consideration.

The main challenge of the use of the R2 in the evaluation of RDPs is to be seen as a fact that an observed change (e.g. in period 2013-2016) of this indicator in supported projects
may result from a number of various factors, e.g.:

a) Primary contributions of investment support received by a given farm from RDP measures linked directly to the focus area 2A (e.g. M3, M4, M6)

b) Primary contributions of non-investment support received by a given farm from RDP measures associated with the focus area 2A (e.g. M1, M2, M9, i.e. training and
advisory services, producer groups)

c) Secondary contributions of support received by a given farm from RDP measures linked to other focus areas, e.g. FA4 or FA5 (e.g. via M10, M11, M12, M14, etc.)

d) Direct and indirect effects of other subsidies not related to RDPs received by a given farm (e.g. from Pillar I)

e) Direct and indirect effects of other subsidies not related to the CAP (e.g. from regional funds - via migration of labour from agriculture to the non-agricultural sector)

f) Effects of other exogenous factors not related to the RDP, e.g. change in agricultural prices, change of price ratios between agricultural outputs and inputs which provide
an incentive for an increase/decrease of agricultural output; etc.

In order to calculate the extent to which changes in the R2 indicator was caused by RDP measures = net effect of the RDP programme in supported projects, the evaluator has to
divide the observed total change of R2 into two independent components: i) a change of R2 which was due to the RDP programme (cases: a-c); and ii) a change in R2 caused by
other factors (cases: d-f as one aggregate). Considering the extent to which RDP measures affect R2 cannot be directly observed, separation of these effects must be carried out

27
The stages and steps for calculation of CRI (adjusted) can be also used for calculation of additional indicators if relevant)

22
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

using advanced quantitative evaluation methodologies. Application of methodologies recommended below further allows for the analysis of various indirect programme effects:
deadweight loss effect, substitution effect, etc.

The proposed approach for assessing the extent to which RDP measures contributed to a change in the R2 indicator involves three stages, each implemented in several steps:

Stage 1: Estimation of primary contributions of RDP measures directly attributable to the focus area 2A (i.e. measures: M1, M2, M3, M4, M6, M9, M16) (Note: all measures
directly attributable to the focus area 2A are expected inter alia to affect the farm labour productivity (indicator R2) of the direct programme beneficiary.

Steps for the calculation of primary contributions:

Step 1: Find a sample of farms/farmers who received in a given period, e.g. 2014-2016 support from measures M1, M2, M3, M4, M6, M9 or M16 (“beneficiaries”),

Note: In case indicator R2 is calculated on a constant sample, please be aware that about 60% of the sample is the same after 4 years (for instance from 2009 to 2012). This is just
an average: the % of constant sample goes from 20% to 90%, depending on the Member State.

Step 2: Select from all farms/farmers who in the same period didn’t received support from measures M1, M2, M3, M4, M6, M9 and M16 (“non-beneficiaries”) a sample which is at
least 2-3 times larger than the sample of “beneficiaries”, Note: MS should be aware of the FADN sample size and see if the sample is sufficient to conduct the counterfactual analysis,
in case not the stakeholders in the Member States might have already developed or develop national FADN which would enlarge the sample.

Step 3: For all farms in both groups collect data on agricultural output and employment (AWU) as well as data on other farms’ major characteristics, e.g., farm area, rented area,
total inputs used, gross farms income, family farm income, total assets, gross investments in fixed assets, other subsidies received, etc.

Step 4: Apply appropriate techniques (e.g. matching) which enables one to identify from the sample of “non-beneficiaries” (see: step 2) a suitable “control group” for a sample of
“beneficiaries” (some of “non-beneficiaries” and/or “beneficiaries” will be dropped from the analysis due to the lack of adequate control units).

Step 5: Check statistically the “similarity” of both groups prior to receiving support from the programme, e.g. by performing statistical tests (average values of farms characteristics
in the group of “beneficiaries” should not significantly differ from the “control group”)

Step 6: Compute for the group of “beneficiaries” and “control group” the average value of R2 indicator prior to the support (e.g. in year 2013) and after support (e.g. year 2016)

23
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Step 7: Perform a calculation of the specific policy indicators, e.g. Average Treatment Effects on Treated (ATT), using R2 as the result indicator. Compute programme net effect on
R2 (by combining calculated ATTs with Difference in Differences (DID) method) 28

Step 8: Perform sensitivity of obtained results

Step 9: Calculate the aggregated value of the net indicator at a programme area level by applying extrapolation techniques (e.g. by multiplying average micro-results computed at
a farm level by a number of supported farms)

Note: application of the methodology described above allows inter alia analysing programme deadweight loss effects

Stage 2: Estimation of secondary contributions of those RDP measures in which the main objective is linked to another focus area (e.g. renewable energy, water efficiency
e.g. M10, M11, M12) but which also are expected to contribute on farm restructuring and competitiveness. This also includes those operations implemented via CLLD strategies,
which show secondary contributions to the farm competitiveness. The most significant contributions should be taken in consideration. Calculation of the above contributions (mainly
environmentally oriented) requires additional data on the environmental variables linked to the location of those supported (and non-supported farms). If there is not sufficient uptake
to calculate the secondary contributions in a robust way a qualitative assessment is also possible (see the Annex 10 of the Guidelines: Assessment of RDP results: how to prepare
for reporting on evaluation 2017).

Steps for the calculation of secondary effects:

Step 1: Find a sample of farms/farmers who received in a given period, e.g. 2014-2016 support from measures M10, M11, M12, M13 and M14 (“beneficiaries”)

Step 2: Select from all farms/farmers who in the same period did not receive support from measures M10, M11, M12, M13 and M14 (“non-beneficiaries”). This sample should be at
least 2-3 times larger than the sample of “beneficiaries”

Step 3: For all farms in both groups collect data on agricultural output and employment as well as data on the farm’s major characteristics (used to construct the control groups),
e.g., farm area, rented area, total inputs used, gross farms income, family farm income, total assets, gross investments in fixed assets, other subsidies received, etc. as well as other
environmentally relevant characteristics of the farms location, e.g. using GIS data

Step 4: Apply appropriate techniques (e.g. matching) which enables one to identify from the group of “non-beneficiaries” a suitable “control group” (some of the non-beneficiary farms
and/or beneficiaries will be dropped from the analysis due to the lack of adequate control units).

28
Capturing the success of your RDP: Guidelins for the ex-post evaluation of 2007-2013 RDP, PART II, Chapter 4: Methods, part [Link] Quantitative approaches, page 94,
[Link]

24
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Step 5: Check statistically the “similarity” of both groups prior to receiving support from the programme, e.g. by performing statistical tests (average values of farms characteristics
in the group of “beneficiaries” should not significantly differ from the “control group”)

Step 6: Compute for the group of beneficiaries and control group the value of the R2 indicator prior to the support (e.g. in year 2013) and after support (e.g. year 2016)

Step 7: Perform calculation of specific policy indicators, e.g. Average Treatment Effects on Treated (ATT), using R2 as the result indicator. Compute programme net effect on R2
(by combining calculated ATTs with Difference in Differences (DID) method)

Step 8: Perform sensitivity of obtained results

Step 9: Calculate the aggregated value of the net indicator at a programme area level by applying extrapolation techniques (e.g. by multiplying average micro-results computed at a
farm level by a number of supported farms)

Note: application of the methodology described above allows inter alia analysis of deadweight loss effects of measures included in the analysis (i.e. M10, M11, M12, M13 and M14)

Stage 3. Estimation of indirect effects of the RDP measures identified at Stage 1 and Stage 2 (Note: it can be expected that support obtained by beneficiaries of RDP measures
attributable to the focus area 2A, e.g. M4, may have expected/unexpected general equilibrium effects, e.g. negative effects on non-beneficiaries located in a close neighbourhood
of programme beneficiaries. For example, due to intensive support of farm investment (M4) the price of land and investment goods in this region may increase and therefore may
affect negatively labour productivity of programme non-beneficiaries): substitution effect. Similarly, <expected/unexpected> indirect effects on labour productivity of programme non-
beneficiaries may occur during implementation of measures linked to other focus areas (renewable energy, water efficiency, etc.)

Steps for calculation of indirect effects:

Programme indirect effects, e.g. substitution, displacement, multiplier, etc. of RDP measures identified in Stage 1 and Stage 2 should be computed and shown separately. The
methodology which can be applied to analysis of programme indirect effects has been described in: “Approaches for Assessing the Impacts of the RDP in the context of multiple
intervening factors”, Brussels, 2010;

Note: The quantitative values resulting from the above methodology can be validated and interpreted with qualitative approaches, involving interviews and focus groups with the
managers of measures and/or with representatives of beneficiaries and non-beneficiaries.

Challenges/Risks/Issues

Although building valid counterfactuals is the most promising technique for the separation of RDP measures’ effects from other programme independent factors, the empirical
application of the recommended methodology to the evaluation of RDP measures expected to affect the net value of the R2 indicator (i.e. labour productivity) requires an abundant

25
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

database (especially when analysing effects of measures linked to environmental targets/focus areas). For the same reason, the institutional entities (MA, regional and/or national
institutes of statistics…) and evaluators should not underestimate the role of a systematic collection of data on programme non-beneficiaries enabling the construction of valid
counterfactuals. Also, the proposed method will not generate reasonable results if there are other important observable characteristics explaining the differences in performance of
programme beneficiaries and control groups but they were not included into the model (it is therefore important that all stakeholders who can facilitate to collect respective information
as early as possible collaborate, in order to insert into the model several indicators showing the farms’ main characteristic and performance). Furthermore, some techniques can be
statistically complex and require advanced quantitative skills (econometrics, modelling, etc.).

Proposed solutions

Low number of non-beneficiaries (or small programmes) and proposed solutions

In case in a given programme area a great deal of farms received RDP support a counterfactual analysis of the effectiveness of programme support on indicator R2 should involve
other quantitative methods, e.g. generalized propensity score matching (GPSM). Given explicit information on the intensity of investment support (e.g. financial flows into a public
investment programme per farm). The effect of the RDP measures on the R2 indicator can be analysed by means of a dose-response function and derivative dose-response function
(part of GPSM). Generally speaking, GPSM method not only allows to estimate the average effect of public investment support on the selected result/impact indicator (e.g. GVA/farm
or GVA/region), but also to assess the marginal effects of the programmes or measures, depending on the support intensity level obtained.

Application in 2017 and low RDP up-take and short time-lags and proposed solutions

The selection of an appropriate time period (after implementation of a given programme) may be crucial for estimating the programme results. Generally, the period which is chosen
should not be too short (unfolding outcomes) nor too long, as other confounding factors or policies (specifically targeting either programme beneficiaries or programme non-
beneficiaries) may systematically influence the effects. As evaluation methodologies described above are quite flexible regarding the selection of an “end period”, it is advisable to
undertake re-estimations of outcomes by including the successive years, in order to verify the stability of the estimated effects. Another possibility is to build 2 or 3 year averages to
represent the “end period”. Unfortunately, there is usually a trade-off between advantages and disadvantage from re-estimations using successive years namely from worsening of
the quality of the database resulting from dropping units/observations from the balanced panel. The marginal effectiveness of such an exercise is to be estimated in each individual
case by an evaluator.

Solutions to a low uptake at the beginning of the programme can be similar as for slowly unfolding results. A solution is a re-estimation of results in the successive years (or building
averages of years as the end period). The cost-effectiveness of using this approach in slow-uptake situations or for a very short period of time of one or two programme years needs
to be reviewed on a case by case basis.

26
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Examples of use of CRI R2

“Approaches for Assessing the Impact of the Rural Development Programmes in the Context of Multiple Intervening Factors”, The European Evaluation Network for Rural Development, March 2010, pp.
1-225, [Link] 31E9-DDFD-E9C9FFC0E30A.

“Guidelines for the ex-post evaluation of 2007-2013 Rural Development Programmes”, The European Network for Rural Development, pp 1-196, April 2014,

[Link] “, Guidelines for ex-post evaluation of RDP programmes”, Brussels, 2014

“Investment support under Rural Development Policy”; final report; Metis, WIFO, AEIDL; November 2014; [Link]
2014_en.htm

Michalek J. (2012), “Counterfactual impact evaluation of EU rural development programmes - Propensity Score Matching methodology applied to selected EU Member States”, Volume 1 – A micro-level
approach.”, European Commission, JRC Scientific and Policy Reports, pp 1-95, [Link]

Michalek, J. et. al. (2015), “Investment Crowding Out: Farm-level Evidence from Northern Germany”, Regional Studies – DOI: 10.1080/00343404.2015.1044957

Examples of application of GPSM method

Michalek J. et. al. (2014), "Capitalization of CAP Single Payment Scheme into Land Value: Generalized Propensity Score Evidence from the EU", Land Economics, University of Wisconsin Press, May
2014 vol. 90:260-289. [Link]

Answer to CEQ:

The answer is provided based on the calculated values of indicators, on the collected qualitative information or on the qualitative assessment.

Conclusions Recommendations

Main conclusions are stated here for the FA based on the evaluation findings. Main recommendations linked to the conclusions are stated here for the FA.

27
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes
2014-2020 (CEQ 1 – 21)

2.5 Focus Area P2B, Evaluation question 5

To what extent have RDP interventions supported the entry of adequately skilled farmers into the agricultural sector
and in particular, generational renewal?

List of measures contributing to the FA 2B:

Primarily programmed measures/sub-measures (proposed by EC) 29:

• Measures and sub-measures of Art. 30 35 Co-operation

• Measures and sub-measures of Art. 19 Farm and business development

• Measures and sub-measures of Art. 17 Investments in physical assets

• Measures and sub-measures of Art. 15 Advisory services, farm management and farm relief services

• Measures and sub-measures of Art. 14 Knowledge transfer and information actions

Examples of measures/sub-measures programmed under the other FA but potentially showing the secondary
contribution to this FA:

• All above measures if programmed under other FAs than FA 2B and contributing to generational renewal and
improved skills of farmers

• Measures and sub-measures of Art. 16, Quality schemes for agriculture products and food stuff

• Measures and sub-measures of Art. 27 Setting up of producers groups and organisations

• Measures and sub-measures of Art. 35 of Regulation (EU) no 1303/2013, Support for Leader local development

Consistency check between CEQ, judgement criteria and indicators:

Judgement criteria

Judgement criteria (JC) proposed by the WD Common Evaluation Questions for Rural Development Programmes
2014- 2020:

• Adequately skilled farmers have entered into the agricultural sector

• The share of adequately skilled young farmers in the agricultural sector has increased

Note: stakeholders in Member States might add/change judgement criteria in line with the intervention logic of the FA
(selection and combination of measures)

Indicators

The following common indicators should be used to answer the CEQ:

Common result/target indicator:

• R3/T5 - % of agriculture holdings with RDP supported business development plan/investments for young farmers

The following other common indicators might be used to answer the CEQ:

29
WP: Guidelines for strategic programming for the period 2014-2020, European Commission, 04/03/2014, published on
ECAS/circabc
30
All articles mentioned in this section of each fiche for answering CEQ 1 – 18 are of Regulation (EU) No 1305/2013, unless it
is stated otherwise in the text directly
28
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Common output indicator - O4 31 (data collected via the operations database):

• Number of beneficiaries receiving the start-up aid young farmers

• Number of holdings supported for investment in agriculture holdings

Common context indicators (Data needs and data sources are described in the Working document: Proposed list of
common context indicators):

• CCI 17 Agriculture holdings

• CCI 22 Farm labour force

• CCI 23 Age structure of farm managers

• CCI 24 Agriculture training of farm managers

Additional indicators and information

Common indicators are not sufficient to answer the CEQ. WD Common Evaluation Questions for Rural Development
Programmes 2014-2020 suggests:

• % of adequately skilled new young farmers in the agricultural sector of the RDP territory

The following additional indicator is proposed to be used in answering the CEQ:

• % of adequately skilled new young farmers in the agricultural sector of the RDP territory

Note: More additional indicators can be developed when judgement criteria are added to specify the evaluation
question in MS.

Qualitative indicators

For qualitative assessment the following information can be gathered: Types and description of skills that new
entrants (young farmers) bring into the agricultural sector of the RDP territory, if the skills of the young farmers are in
line with the supported business development plan/investments for young farmers, if the business plans include future
training.

31
WD: Data item list for Pillar II operation database and WD RD programming and target setting

29
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Data needs and data sources

Indicators Data needed Data sources

R3/T5 - % of agriculture holdings with Total number of farm holdings Beneficiaries

RDP supported business development receiving business start-up aid for
plan/investments for young farmers young farmers over the programming Application form (project start)
period (under measure 6.1)
Payment request (project end)
Common indicators

Total number of farm holdings

supported for investment targeting For total number of agriculture

young farmers (under measure 4.1 in holdings

case measure 6.1 is not programmed)

National/regional statistics
(data item O.4)

Eurostat, Farm Structure Survey, etc.

Total number of agricultural holdings in
(for the total number of holdings):
base year for the RDP area
[Link]
lture/data/main-tables

% of adequately skilled new young Number of new skilled young farmers Beneficiaries
farmers in the agricultural sector of the that entered to the agricultural
RDP territory (gross/net) supported by the RDP, by age and Application form (project start)

level and type of training/education

Payment request (project end)
Total number of farmers in the
Survey to new/young farmers
Additional indicators

agricultural sector of the RDP territory,

supported by the RDP
by age and level and type of
training/education
Beneficiaries and non-beneficiaries

National/regional statistics

Eurostat, Farm Structure Survey, etc.

(for the total number of farmers and by
age and by level of training):
[Link]
lture/data/main-tables

Timing of data collection

Proposed data on beneficiaries and non-beneficiaries should be collected prior to RDP implementation (i.e. years
2010-2013) and during RDP implementation (i.e. 2014-2020: 1st call for proposal, operations database,
national/regional statistics, EU sources)

30
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Methodology to calculate common and additional indicators

Common indicators

The common indicator R3/T5 is collected via the operations database and calculated as a ratio of agriculture holdings
with business plans for young farmers to the total number of holdings supported by the RDP.

Secondary contributions

Secondary contributions to the value of R3/T5 can be tracked also directly through the operations database if in the
project applications is indicated that the project promotor is young farmer. This also includes those operations
implemented via CLLD strategies. The monitoring should be able to eliminate double counting.

Additional indicators

Both additional indicators are calculated as ratio of adequately skilled new/young farmer of total number of farmers in
the RDP territory (the skills of the new/young farmers shall be collected by the application forms and the business
plans).

Secondary contributions

Secondary contributions of operations which might be implemented by young farmers, which are programmed under
other FAs than P2B are also considered when calculating the indicators. This also includes those operations
implemented via CLLD strategies, which show secondary contributions to the generational renewal and enhancement
of farmers´ skills. The contributions are assessed with the means of additional result indicators linked to the FA 2B
based on a survey with samples of those beneficiaries – (new) young farmers which are implementing operations via
different focus areas (e.g. FA 2A, 6A, 6B etc.) and have flagged the contribution to FA 2B in application
forms/payment request. Different types of operations can show different sizes of contributions to the value of result
indicators used to assess achievements towards the FA 2B. Most significant contributions should be taken into
consideration. In cases with a low number of beneficiaries flagging the additional contribution to the FA 2B the total
population of beneficiaries can be taken in consideration in the survey. If there is no sufficient uptake to calculate the
secondary contributions in a robust way, the qualitative assessment is also possible (see the Annex 10 of the
guidelines: Assessment of RDP results: how to prepare for reporting on evaluation 2017). Findings will also be used in
answering the CEQ 19.

Qualitative assessment

For qualitative assessment, the proposed methods will enable one to: a) interpret the quantitative values of the
indicators; b) assess the net contribution of the RDP on skilled young farmers entering the agriculture sectors.
Proposed methods are:

• Survey to new entrants/beneficiaries of measure 6.1

• Survey to supported farms of measure 4.1 (in case measure 6.1 is not programmed)

• Interviews with the managers of the interventions

• Structured focus groups 32, including the MAPP method 33 for the assessment of net effects

Triangulation of the findings from different sources (indicator values, findings from interviews, surveys, focus groups)

32
For details on how to use surveys and focus groups, see ‘Capturing the success of your RDP: Guidelines for ex post
evaluation of 2007-2013 RDPs’, Evaluation Helpdesk, June 2014
33
[Link]
31
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Challenges/Risks/Issues

• Low quality data that does not provide a robust answer to the EQ

• Lack of data to answer the EQ (e.g. low RDP uptake/no uptake, small programmes etc.)

• Young farmers not attracted to the agricultural sector, which may lead to low uptake

Proposed solutions

In case of a lack of data (low uptake, small programmes), this EQ can focus on an estimation (via interviews and
expert opinions) of the expected effect of the RDP on the entry of new and skilled farmers into the agricultural sector.

In case of low uptake due to a lack of interest, explore the reasons that deter young people from entering the
agricultural sector via RDP support (e.g. difficulties to acquire farms that are economically viable, or that provide full
time employment or that enable them to earn sufficient income).

In case of low quality data, qualitative information can be sought through interviews, surveys, and focus groups.

Answer to CEQ

The answer is provided based on the calculated values of indicators, on the collected qualitative information or on the
qualitative assessment.

Conclusions Recommendations

Main conclusions are stated here for the FA based on the Main recommendations linked to the conclusions are
evaluation findings. stated here for the FA.

2.6 Focus Area P3A, Evaluation question 6

To what extent have RDP interventions contributed to improving the competitiveness of supported primary producers by
better integrating them into the agri-food chain through quality schemes, adding value to the agricultural products,
promoting local markets and short supply circuits, producer groups and inter-branch organization? 34

List of measures contributing to the FA 3A:

Primarily programmed measures/sub-measures (proposed by EC) 35:

• Measures and sub-measures of Art. 36 15 Advisory services, farm management and farm relief services

• Measures and sub-measures of Art. 14 Knowledge transfer and information actions

• Measures and sub-measures of Art. 35 Co-operation

• Measures and sub-measures of Art. 33 Animal welfare

• Measures and sub-measures of Art. 20 Basic services and village renewal in rural areas

• Measures and sub-measures of Art. 19 Farm and business development

• Measures and sub-measures of Art. 27 Setting up producer groups and organisations

• Measures and sub-measures of Art. 17 Investments in physical assets

34
The questions concern the share of primary producers at the final price of the agricultural products proposing various
scenarios, e.g. quality schemes, adding value to primary products, participation in short supply circuits, promoting local markets
etc.
35
WP: Guidelines for strategic programming for the period 2014-2020, European Commission, 04/03/2014, published on
ECAS/circabc
36
All articles mentioned in this section of each fiche for answering CEQ 1 – 18 are of Regulation (EU) No 1305/2013, unless it
is stated otherwise in the text directly
32
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

• Measures and sub-measures of Art. 16 Quality schemes for agriculture products and food stuffs

Examples of measures/sub-measures programmed under the other FA but potentially showing the secondary
contribution to this FA:

• All above measures if programmed under a different FA other than focus area 3A and contributing to improving the
competitiveness of supported primary producers by better integrating them into the agri-food chain through quality
schemes, adding value to the agricultural products, and promoting local markets and short supply circuits, or producer
groups and inter-branch organization

• Measures and sub-measures of Art. 35 of Regulation (EU) no 1303/2013, Support for Leader local development

Consistency check between CEQ, judgement criteria and indicators

Judgement criteria

Judgement criteria (JC) proposed by the Working Document: Common Evaluation Questions for Rural Development
Programmes 2014- 2020:

• Competitiveness of supported primary producers has improved

• The share of the final price of agriculture products retained with primary producers has increased

• The added value of agricultural products of primary producers has increased

• Implementation of quality schemes by primary producers has increased

• Participation of primary producers in short circuit schemes, quality-oriented producer group and/or inter branch
organization has increased

Note: stakeholders in the Member State might add/change judgement criteria in line with the intervention logic of the
FA (selection and combination of measures)

Indicators

The following common indicators should be used to answer the CEQ:

Common result/target indicator:

• R4/T6 - % of agricultural holdings receiving support for participating in quality schemes, local markets and short
supply circuits, and producer groups/organisations

The following other common indicators might be used to answer the CEQ:

Common output indicators 37 (data collected via the operations database):

• O4 Number of holdings supported

• O9 Number of holdings participating in producer groups supported

• O9 Number of agriculture holdings participating in cooperation/local promotion among supply chain actors

Common context indicators (Data need and data sources are described in the Working document: Proposed list of
common context indicators):

• CCI 14 Labour productivity in agriculture

• CCI 16 Labour productivity in the food industry

37
WD: Data item list for Pillar II operation database and WD RD programming and target setting
33
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Additional indicators and information

Common indicators are not sufficient to answer the CEQ. WD Common Evaluation Questions for Rural Development
Programmes 2014-2020 suggests:

• Agricultural output on supported farms

• Margin of primary producers in the final price of agricultural products

• % of primary producers introducing quality schemes with RDP support

Further proposed additional indicators:

• Comparison between the margins of producers involved and not involved in quality schemes, short supply circuits,
producer groups and inter-branch organization

• Note: More additional indicators can be developed when judgement criteria are added to specify the evaluation
question in MS.

Qualitative indicators

For qualitative assessment the following information can be gathered:

• types and description of local markets,

• types and description of short supply chains,

• description of the key factors that motivate farmers to participate in quality schemes (other than the financial support)
etc.

Data needs and data sources

Indicators Data needed Data sources

R4/T6 - % of agricultural holdings Total number of agricultural holdings Beneficiaries

receiving support for participating receiving support under measure 3.1
Application form (project start)
in quality schemes, local markets ‘Participation to quality schemes’ (data item
and short supply circuits, and O.4) Payment request (project end)
producer groups/organisations For total number of agriculture holdings
Total number of agricultural holdings
Common indicators

receiving support under measure 9 National/regional statistics

‘Setting-up producer groups and
organisations (data item O.9) Eurostat, Farm Structure Survey, etc. (for
the total number of holdings):
Total number of agricultural holdings [Link]
receiving support under measure 16.4
re/data/main-tables
‘Cooperation among supply-chain actors’
(data item O.9)

Total number of agricultural holdings in

base year for the RDP area

34
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Changes in agricultural output on Agricultural output on supported farms Beneficiaries

supported farms
Total number of agriculture holdings Application form (project start)
divided by sector
Payment request (project end)

Beneficiaries and non-beneficiaries

Survey on beneficiaries and non-

beneficiaries and/or

National/regional statistics

Eurostat, Agricultural output, price indices

and income:
[Link]
explained/[Link]/Agricultural_output,_
price_indices_and_income
Additional indicators

Eurostat, Farm Structure Survey, etc. (for

the total number of holdings):
[Link]
re/data/main-tables

FADN

Margin of primary producers in the Final price of agricultural products

final price of agricultural products
Primary producer margins

[useful the comparison with farmers which

are not involved in quality schemes, short
supply circuits, producer groups and inter-
branch organization]

% of primary producers introducing Number of supported primary producers Beneficiaries

quality schemes with RDP support introducing quality schemes divided by
Application form (project start)
sector
Payment request (project end)
Total number of agriculture holdings
divided by sector For total number of agriculture holdings

National/regional statistics

35
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Comparison between the margins Beneficiaries

of producers involved and not
Application form (project start)
involved in quality schemes, short
supply circuits, producer groups Payment request (project end)
and inter-branch organisation. Survey of supported primary producers

Beneficiaries and non-beneficiaries

Survey on beneficiaries and non-

Additional indicators

beneficiaries and/or

National/regional statistics

FADN

Eurostat, Agricultural output, price indices

and income:
[Link]
explained/[Link]/Agricultural_output,_
price_indices_and_income

Eurostat, Farm Structure Survey, etc. (for

the total number of holdings):
[Link]
re/data/main-tables

Timing of data collection

Proposed data on beneficiaries and non-beneficiaries should be collected prior to RDP implementation (i.e. years 2010-
2013) and during RDP implementation (i.e. 2014-2020: 1st call for proposal, operations database, national/regional
statistics, EU sources)

Methodology to calculate common and additional indicators

Common indicators

The common indicator R4/T6 is collected via the operations database and calculated as a ratio of agriculture holdings
receiving support for participating in quality schemes, local markets and short supply circuits, and producer
groups/organisations of total number of holdings.

Secondary contributions

Secondary contributions to the value of R4/T6 can be tracked also directly through the operations database if in the
project applications is indicated that the project is participating in quality schemes, support local markets and enables
short supply circuits. This also includes those operations implemented via CLLD strategies. The monitoring should be
able to eliminate double counting.

Additional indicators

Counterfactual is proposed in calculation of two additional indicators:

• Agricultural output on supported farms

• Margin of primary producers in the final price of agricultural products

36
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

As techniques, PSM in combination with DID are proposed by the Guidelines for the ex post evaluation of 2007-2013
RDPs 38.

The two other additional indicators are calculated as ratio of farmers, which have introduced quality schemes short supply
circuits, producer groups and inter-branch organizations with RDP support of total number of farmers in the RDP territory.

Secondary contributions

Secondary contributions of operations which are programmed under other FAs than P3A should also be considered for
the calculation of additional result indicators (if relevant) linked to the CEQ for FA 3A when answering it. This also includes
those operations implemented via CLLD strategies, which show secondary contributions to the objectives of FA 3A.
Secondary contributions to the value of result indicators are calculated based on a survey with samples of those
beneficiaries which are implementing operations in the scope of different focus areas (e.g. 2A, 6A, 6B etc.), but have
flagged additional contribution to FA 3A. The assessment may also look at the significance of contributions of different
types of operations: e.g. farm investment measures, of farm diversification measures etc. to the participation in quality
schemes, producer groups, promoting local markets and short supply circuit and compare the contributions. The most
significant contributions should be taken in consideration. In the case of a low number of beneficiaries flagging the
additional contribution to the FA 3A the total population of beneficiaries can be taken into consideration in the survey. If
there is not sufficient uptake to calculate the secondary contributions in a robust way, the qualitative assessment is also
possible (see the Annex 10 of the guidelines: Assessment of RDP results: how to prepare for reporting on evaluation
2017). Findings are also used when answering the CEQ 19.

Qualitative assessment

For the qualitative assessment, the proposed methods will enable one to describe the types of local markets, types of
short supply chains, key factors that motivate farmers to participate in quality schemes etc. Proposed methods are:

• Survey to supported and non-supported primary producers under measures 3.1, 9 and 16.4

• Structured focus groups 39, interviews with managers of these measures / producer groups / inter-branch
organisations. In relation to focus groups. the MAPP method can be used to assess net effects 40

Triangulation of the findings from different sources (indicator values, findings from interviews, surveys, focus groups)

Challenges/Risks/Issues

• Low quality data that does not provide a robust answer to the EQ

• Lack of data to answer the EQ (e.g. low RDP uptake/no uptake, small programmes etc.)

Proposed solutions

In case of a lack of data (low uptake, small programmes) this EQ can be answered based on data collected for the total
population of contracted beneficiaries for the calculation of the respective indicator. In the case of no uptake the answer
can be estimated based on qualitative approaches (via interviews and expert opinions) of the expected improvements in
competitiveness of the supported primary producers and for what reasons (e.g. participation in quality schemes, adding
value to agricultural products, participating in producer groups, etc.).

In case of low quality data, qualitative information can be sought through interviews, surveys, focus groups.

38
For details on how to use surveys and focus groups, see ‘Capturing the success of your RDP: Guidelines for ex post
evaluation of 2007-2013 RDPs’, Evaluation Helpdesk, June 2014
39
For details on how to use surveys and focus groups, see ‘Capturing the success of your RDP: Guidelines for ex post
evaluation of 2007-2013 RDPs’, Evaluation Helpdesk, June 2014
40
Investment support under Rural Development policy, DG Agri, 2014: [Link]
development-reports/2014/investment-support-rdp/fulltext_en.pdf
37
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Answer to CEQ

The answer is provided based on the calculated values of indicators, on the collected qualitative information or on the
qualitative assessment.

Conclusions Recommendations

Main conclusions are stated here for the FA based on the Main recommendations linked to the conclusions are
evaluation findings. stated here for the FA.

2.7 Focus Area P3B, Evaluation question 7

To what extent have RDP interventions supported farm risk prevention and management?

List of measures contributing to the FA 3B:

Primarily programmed measures/sub-measures (proposed by EC) 41:

• Measures and sub-measures of Art. 42 15 Advisory services, farm management and farm relief services

• Measures and sub-measures of Art. 14 Knowledge transfer and information actions

• Measures and sub-measures of Art. 35 Co-operation

• Measures and sub-measures of Art. 36 Risk management

• Measures and sub-measures of Art. 18 Restoring agricultural production potential damaged by natural disasters

Examples of measures/sub-measures programmed under the other FA but potentially showing the secondary
contribution to this FA:

• All above measures if programmed under a different FA other than FA 3B and contributing to farm risk prevention
and management

• Measures and sub-measures of Art. 17 Investment in physical assets

• Measures and sub-measures of Art. 19 Farm and business development

• Measures and sub-measures of Art. 35 of Regulation (EU) no 1303/2013, Support for Leader local development

Consistency check between CEQ, judgement criteria and indicators:

Judgement criteria

Judgement criteria (JC) proposed by the Working Document: Common Evaluation Questions for Rural Development
Programmes 2014- 2020:

• Participation of farms in risk prevention and management schemes has increased

Note: stakeholders in Member States might add/change the judgement criteria in line with the intervention logic of the FA
(selection and combination of measures)

Indicators:

The following common indicators should be used to answer the CEQ:

41
WP: Guidelines for strategic programming for the period 2014-2020, European Commission, 04/03/2014, published on
ECAS/circabc
42
All articles mentioned in this section of each fiche for answering CEQ 1 – 18 are of Regulation (EU) No 1305/2013, unless it
is stated otherwise in the text directly
38
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Common result/target indicator:

• R5/T7 - % of farms participating in risk management schemes

The following other common indicators might be used to answer the CEQ:

Common output indicator – O4 43 (data collected via the operations database):

• Number of holdings supported for premium for insurance

• Number of holdings participating in mutual funds

• Number of farm holdings participating in income stabilisation funds

• Number of beneficiaries for preventing actions – farm holdings

Common context indicator (Data needs and data sources are described in the Working Document: Proposed list of
common context indicators):

• CCI 17 agricultural holdings (farms)

Additional indicators and information

This evaluation question is related to a new measure introduced in the current programming period (measure 17: risk
management). It also covers measure 5.1 ‘Restoring agricultural production potential damaged by natural disasters and
catastrophic events and the introduction of appropriate preventive actions’. Given the novelty of the risk management
measure, additional information should be sought to analyse the typology of risks and actions to prevent/manage them:

Further proposed indicators and information collection:

• Number and % of farms participating in risk prevention and management actions, divided by types, e.g.

o a) actions preventing soil degradation, low tillage and winter green cover;

o b) establishment of agro-forestry systems that provide synergies to improve soil management including soil
carbon stock depletion;

o c) investments in soft and hard infrastructure to manage climate hazards (e.g. flood risk and volatility in water
supply);

o d) investment in equipment for flood risk management;

o e) introduction of climate resilient crops and species, including local crop varieties and livestock breeds;

o f) development of risk analysis models for assessment and management of changing climate hazards;

o g) creation or modification of existing insurance and compensation funds and schemes according to changing
hazards;

o h) introduction of climate resilient practices and crops as requirements for participation in insurance schemes;

o i) other

Note: More additional indicators can be developed when the judgement criteria are added to specify the evaluation
question in the Member State

Qualitative indicators

43
WD: Data item list for Pillar II operation database and WD RD programming and target setting
39
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

For qualitative assessment the following information can be gathered: description of the risks, description of the key
factors that motivate farmers to participate in risk prevention and management or, on the other hand, key factors leading
to low uptake of the concerned measures.

Data needs and data sources

Indicators Data needed Data sources

R5/T7 - % of farms participating in Number of agricultural holdings Beneficiaries

risk management schemes supported for insurance premiums under
Application form (project start)
measure 17.1 (data item O.4)
Payment request (project end)
Number of agricultural holdings
participating in mutual funds under For total number of agriculture holdings
Common indicators

measure 17.2 (data item O.9)

Eurostat, Farm Structure Survey, etc. (for the
Number of agricultural holdings total number of holdings):

participating in income stabilisation tools [Link]

under measure 17.3 (data item O.9) data/main-tables

Number of agricultural holdings investing

in preventing actions under measure 5.1
(data item O.4)

Total number of agricultural holdings in

base year for the RDP area

Number and % of farms Number and % of farms participating in Beneficiaries

Additional indicators

participating in risk prevention and risk prevention and management actions

Application form (project start)
management actions divided by divided by type
type Payment request (project end)

Surveys

Timing of data collection

Proposed data on beneficiaries and non-beneficiaries should be collected prior to the RDP’s implementation (i.e. years
2010-2013) and during the RDP’s implementation (i.e. 2014-2020: 1st call for proposal, operations database,
national/regional statistics, EU sources)

Methodology to calculate common and additional indicators

Common and additional indicator

Both common (R5/T7) and additional indicators (if relevant) are collected via the operations database and calculated as
a ratio of agriculture holdings participating in risk management schemes as a total of the number of holdings. The indicator
can be broken down by types of risk management schemes in addition.

Proposed methods are:

• Survey to supported agricultural holdings

• Interviews with managers of these measures

40
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

• Triangulation of the findings from different sources (indicator values, findings from interviews, surveys, and focus
groups)

Challenges/Risks/Issues

• Lack of data to answer the EQ (e.g. low RDP uptake, small programmes etc.)

• Low quality data that does not provide a robust answer to the EQ

Proposed solutions

In the case of a lack of data (low uptake, small programmes), this EQ can be answered with data collected for the entire
population of beneficiaries contracted for the calculation of indicators. In case of no uptake, the answer can be based on
a qualitative assessment (via interviews and expert opinions) of the expected participation in risk prevention and
management schemes.

In case of low quality data, qualitative information can be sought through interviews, surveys, and focus groups.

Answer to CEQ

The answer is provided based on the calculated values of indicators, on the collected qualitative information or on the
qualitative assessment.

Conclusions Recommendations

Main conclusions are stated here for the FA based on the Main recommendations linked to the conclusions are
evaluation findings. stated here for the FA.

41
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)
2.8 Focus Area P4A, Evaluation question 8

To what extent have RDP interventions supported the restoration, preservation and enhancement of biodiversity including
in Natura 2000 areas, areas facing natural or other specific constraints and HNV farming, and the state of European
landscape?

List of measures contributing to the FA 4A:

Primarily programmed measures/sub-measures (proposed by EC) 44:

• Measures and sub-measures of Art. 45 15 Advisory services, farm management and farm relief services

• Measures and sub-measures of Art. 14 Knowledge transfer and information actions

• Measures and sub-measures of Art. 35 Co-operation

• Measures and sub-measures of Art. 21 Investments in forest area development and improvement of the viability of
forests

• Measures and sub-measures of Art. 20 Basic services and village renewal in rural areas

• Measures and sub-measures of Art. 34 Forest-environmental and climate services and forest conservation

• Measures and sub-measures of Art. 30 Natura 2000 and Water framework directive payments

Examples of measures/sub-measures programmed under the other FA but potentially showing the secondary
contribution to this FA:

• All above measures if programmed under a different FA other than FA 4A and contributing to the restoration,
preservation and enhancement of biodiversity

• Measures and sub-measures of Art. 17 Investment in physical assets

• Measures and sub-measures of Art. 28 Agri-environment-climate

• Measures and sub-measures of Art. 29 Organic farming

• Measures and sub-measures of Art. 35 of Regulation (EU) no 1303/2013, Support for Leader local development

Consistency check between CEQ, judgement criteria and indicators:

Judgement criteria

Judgement criteria (JC) proposed by the Working Document: Common Evaluation Questions for Rural Development
Programmes 2014- 2020:

• Biodiversity on contracted land has been restored, preserved and enhanced

Note: stakeholders in the Member States might add/change judgement criteria in line with the intervention logic of the
FA (selection and combination of measures)

Indicators

The following common indicators should be used to answer the CEQ:

Common result/target indicators:

44
WP: Guidelines for strategic programming for the period 2014-2020, European Commission, 04/03/2014, published on
ECAS/circabc
45
All articles mentioned in this section of each fiche for answering CEQ 1 – 18 are of Regulation (EU) No 1305/2013, unless it
is directly stated otherwise in the text.
42
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

• R7/T9 - % of agricultural land under management contracts supporting biodiversity and/or landscapes

• R6/T8 - % of forest or other wooded areas under management contracts supporting biodiversity

The common indicators can be further broken down by type of actions that are effective in supporting biodiversity and/or
landscapes (e.g. AECM typology, conserving hedgerows, biodiversity enhancing crops, farm operations through adapted
plant selection, reduced pesticide inputs, landscape protection, etc.)

The following other common indicators might be used to answer the CEQ:

Common output indicators 46 (Data collected via the operations database):

• O5 total area (ha)

• O6 Physical area supported (ha)

• O7 No of contracts supported

Common context indicators (Data need and data sources are described in the Working document: Proposed list of common
context indicators):

• CCI 34 - Natura 2000 areas

• CCI 35 - Farmland bird index

Additional indicators and information

Common indicators are not sufficient to answer the CEQ. Working Document: Common Evaluation Questions for Rural
Development Programmes 2014-2020 suggests:

• Number of flora and fauna species on contracted land

Further possible additional indicators:

• % increase/decrease of flora and fauna species on contracted land (and in the programme area) divided by species,
among them those which are endangered

Note: More additional indicators can be developed when judgement criteria are added to specify the evaluation question
in the Member State.

Qualitative indicators

Qualitative indicators might also be used, e.g. to assess the typology of actions that have been effective in supporting
biodiversity and/or landscapes.

Data needs and data sources

Indicators Data needed Data sources

R7/T9 - % of agricultural land Total agricultural land under land Beneficiaries

Common indicators

under management contracts management to improve biodiversity (data

Application form (project start)
supporting biodiversity and/or items O.6, O.7)
landscapes Payment request (project end)
Total agriculture for the RDP area (data item
O.5) For total agriculture land

IACS

46
WD: Data item list for Pillar II operation database and WD RD programming and target setting
43
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Eurostat :
[Link]
explained/[Link]/Farm_structure_statisti
cs#Main_tables

R6/T8 - % of forest or other Total forestry area under land management to Beneficiaries
wooded areas under improve biodiversity (data items O.6, O.7)
Application form (project start)
management contracts
Total forestry area for the RDP area (data item
supporting biodiversity Payment request (project end)
O.5)
Eurostat :
[Link]
explained/[Link]/Farm_structure_statisti
cs#Main_tables

Number and % Number of flora and fauna species on Survey

increase/decrease of flora and contracted land and in the programme area
National and regional environmental
fauna species on contracted divided by species among them those
monitoring
land and in the programme area endangered in timelines (for calculation of
European Environment Agency:
divided by species among them increase/decrease [Link]
Additional indicators

those which are endangered

BISE:
[Link]

National/International Agencies (Bird Life

International)
Our Life Insurance, Our Natural Capital
(2012) [Link]
content/PT/TXT/HTML/?uri=CELEX:52011D
C0244&from=EN

Nível Global – Aichi Biodiversity Targets

(2010) [Link]

Timing of data collection

Proposed data on beneficiaries and non-beneficiaries should be collected prior to the RDP’s implementation (i.e. years
2010-2013) and during the RDP’s implementation (i.e. 2014-2020: 1st call for proposal, operations database,
national/regional statistics, EU sources)

Methodology to calculate common and additional indicators

Common indicators

The common indicators – R6/T8 and R7/T9, are collected via the operations database and calculated as a ratio of agriculture
land or forestry land under managements contracts supporting biodiversity (and landscape in case of agriculture land) of
the total UAA. For those operations programmed under other FAs and contributing to the FA 4A the secondary contributions
to values of above indicators are already captured in the monitoring system.

Additional indicators

Additional indicators will be calculated based on monitoring of selected species of flora and fauna on contracted land in the
timeline. Number and % will be calculated as difference between baselines data and data collected in the year of the
evaluation. For counterfactual similar observation will be done for non-contracted land (as a control group).

44
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

The proposed methods are:

• Survey biogeographical and phytosociological data: beneficiaries of measures/sub-measures 10, 11, 12.1, 8.1 and 8.2,
and non-beneficiaries

• Survey biogeographical and phytosociological data: beneficiaries of forest measures/sub-measures 12.2, 15 and 8.5,
and non-beneficiaries

Secondary contributions (for additional indicators only)

Investment types of operations programmed under FAs other than FA 4A can also show secondary contributions to the
restoring, preserving and enhancing the biodiversity. For investment related operations the additional result indicator should
be used when assessing the secondary contributions to the FA. Secondary contributions to the values of additional result
indicators are estimated based on survey with those beneficiaries which are implementing operations under different focus
areas (e.g. FA 2A, 2B, 4B, 4C, 6A) and have been flagged as additional contributors to FA 4A. This also includes those
operations implemented via CLLD strategies, which show secondary contributions to the enhancement of biodiversity.
Sample sizes of surveyed beneficiaries can be set up in accordance with the expected/actual contributions of implemented
operations (e.g. farm investment measures) to the FA’s objectives. The ratio of the contribution is estimated based on the
above surveys. Most significant contributions should be taken in consideration. If there is no sufficient uptake to calculate
the secondary contributions in a robust way the qualitative assessment is also possible (see the Annex 10 of the guidelines:
Assessment of RDP results: how to prepare for reporting on evaluation 2017). Findings will be used also in answering the
CEQ 19.

Qualitative assessment

For the qualitative assessment the proposed methods will enable one to describe the type of actions that are effective in
supporting biodiversity and/or landscapes, the species involved with special attention to those endangered (the floristic
diversity responds very quickly to pressures and environmental impacts, pesticides application, soil tillage techniques, local
climate change, etc.).

Surveys of beneficiaries, structured focus groups 47 interviews with managers of the measures/sub-measures implemented
under this FA are suitable methods for the qualitative assessment. In relation to focus groups, the MAPP 48 method is
relevant for assessing the effects of interventions on environmental indicators that are difficult to measure (such as
biodiversity in this case).

Challenges/Risks/Issues

• Low quality data that does not provide a robust answer to the EQ

• Lack of data to answer the EQ (e.g. low RDP uptake/no uptake, small programmes etc.)

• Increase/ decrease of targeted species and species diversity of operations cannot be assessed using (general) data
available in RDP operations and other databases (e.g. Natura 2000 areas, habitat maps, etc.). These can be assessed
only through systematic on-site monitoring programmes. Such programmes should be well designed and implemented
across beneficiaries and non-beneficiaries (contracted and non-contracted land) and provided with adequate
resources.

Proposed solutions

47
For details on how to use surveys and focus groups, see ‘Capturing the success of your RDP: Guidelines for ex post
evaluation of 2007-2013 RDPs’, Evaluation Helpdesk, June 2014
47
Investment support under Rural Development policy, DG Agri, 2014: [Link]
development-reports/2014/investment-support-rdp/fulltext_en.pdf
48
Investment support under Rural Development policy, DG Agri, 2014: [Link]
development-reports/2014/investment-support-rdp/fulltext_en.pdf
45
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

In case of a lack of data (low uptake,), this EQ can focus on the estimated land to be covered by management contracts
and/or the expected improvements in biodiversity and/or landscapes (via interviews and expert opinions).

In the case of low quality data, qualitative information can be sought through interviews, surveys, and focus groups. A
screening of existing monitoring activities through different environmental stakeholders might reveal potentially useful
data on specific biodiversity indicators which can be combined with the other data sources for assessment. But
comparability and reliability of different data sets can become a critical issue.

In case of no data (no databases available), these can be assessed only through systematic on-site monitoring
programmes. Such programmes should be well designed and implemented and provided with adequate resources.

Answer to CEQ

The answer is provided based on the calculated values of indicators, on the collected qualitative information or on the
qualitative assessment.

Conclusions Recommendations

Main conclusions are stated here for the FA based on the Main recommendations linked to the conclusions are stated
evaluation findings. here for the FA.

2.9 Focus Area P4B, Evaluation question 9

To what extent have RDP interventions supported the improvement of water management, including fertilizer and
pesticide management?

List of measures contributing to the FA 4B:

Primarily programmed measures/sub-measures (proposed by EC) 49:

• Measures and sub-measures of Art. 50 15 Advisory services, farm management and farm relief services

• Measures and sub-measures of Art. 14 Knowledge transfer and information actions

• Measures and sub-measures of Art. 35 Co-operation

• Measures and sub-measures of Art. 21 Investments in forest area development and improvement of the viability of
forests

• Measures and sub-measures of Art. 20 Basic services and village renewal in rural areas

• Measures and sub-measures of Art. 34 Forest-environmental and climate services and forest conservation

• Measures and sub-measures of Art. 30 Natura 2000 and Water framework directive payments

Examples of measures/sub-measures programmed under other FA but potentially showing the secondary
contributions to this FA:

• All above measures if programmed under another FA other than FA 4B and contributing to the improvement of water
management, including fertilizer and pesticide management

• Measures and sub-measures of Art. 17, Investment in physical assets

• Measures and sub-measures of Art. 28 Agri-environment-climate

49
WP: Guidelines for strategic programming for the period 2014-2020, European Commission, 04/03/2014, published on
ECAS/circabc
50
All articles mentioned in this section of each fiche for answering CEQ 1 – 18 are of Regulation (EU) No 1305/2013, unless it
is stated otherwise in the text directly
46
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

• Measures and sub-measures of Art. 29 Organic farming

• Measures and sub-measures of Art. 27, Setting up producer groups and organisations in agriculture and forestry
sectors

• Measures and sub-measures of Art. 35 of Regulation (EU) no 1303/2013, Support for Leader local development

Consistency check between CEQ, judgement criteria and indicators:

Judgement criteria

Judgement criteria (JC) proposed by the Working Document: Common Evaluation Questions for Rural Development
Programmes 2014- 2020:

• Water quality has improved

Note: stakeholders in MS might add/change judgement criteria in line with the intervention logic of the FA (selection
and combination of measures)

Indicators

The following common indicators should be used to answer the CEQ:

Common result/target indicators:

• R8/T10 - % of agricultural land under management contracts to improve water management

• R9/T11 - % of forestry land under management contracts to improve water management

The following common indicators should be preliminary used to answer the CEQ:

Common output indicators 51 (Data collected via the operations database):

• O4 number of holdings/beneficiaries supported

• O5 total area (ha)

• O6 Physical area supported (ha)

Common context indicators (Data need and data sources are described in the Working document: Proposed list of
common context indicators):

• CCI 40 – Water quality (GNB and nitrates in freshwater)

Additional indicators and information

Common indicators are not sufficient to answer the CEQ. WD Common Evaluation Questions for Rural Development
Programmes 2014-2020 suggests:

• Additional information on water quality of the land under management contracts, e.g.:

o Use of nitrogen and phosphorous (kg/ha)

o Pesticide use (kg/ha)

o Rate of macro-invertebrates in fresh water

o Official water quality indicators (oxygen consuming substances in rivers, content nutrients in freshwater)

51
WD: Data item list for Pillar II operation database and WD RD programming and target setting
47
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Note: More additional indicators can be developed when judgement criteria are added to specify the evaluation question
in the Member State.

Qualitative indicators

Qualitative assessment might be also used, e.g. to assess the typology of actions that have been effective in
improving water quality to understand the key factors of individual and cumulative environmental impact and to interpret
the quantitative data.

Further, the qualitative analysis will enable one to frame the following issues:

• Not all pesticides equally affect water quality – as this depend on their toxicity, persistence, degrades and
environmental fate (behaviour). Reporting on a decrease of pesticide use using a “basket” of pesticides will provide
only partial information on their pressure on water quality.

• Key factors of why the pesticide and nitrogen end-up in water. Take into account that it also depends on many
(external) factors, namely, rainfall, soil type, slope, crop type – this monitoring needs to be very well designed,
organised and results need to be interpreted correctly.

Data needs and data sources

Indicators Data needed Data sources

R8/T10 - % of agricultural Total agricultural land under land Beneficiaries

land under management management contracts to improve
Application form (project start)
contracts to improve water water management (data items O4,
management O.6,) Payment request (project end)

Total number of agricultural holdings For total number of holdings and area
for the RDP area (data item O.4)
IACS
Common indicators

Total agricultural area for the RDP Eurostat :

area (data item O.5) [Link]

R9/T11 - % of forestry land explained/[Link]/Farm_structure_statistics#

Total forestry area under land
under management Main_tables
management contract to improve
contracts to improve water water management (data items O4,
management O.6)

Total number of forestry holdings for

the RDP area (data item O.4)

Total forestry area the RDP area

(data item O.5)

48
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Use of nitrogen and Consumption of N and P from mineral Beneficiaries

phosphorous (kg N or P/ha) fertilisers
Application forms
Consumption of N and P from livestock
Payment request
manure (any form)
Beneficiaries and non-beneficiaries
N fixed in soil by legume crops
WISE:
[Link]
FADN and farm surveys data

Surveys with forest managers

National and regional monitoring data on nutrient

management and / or water quality

National/regional environmental agencies

reports
Additional indicators

Evaluation studies on the environmental quality

Pesticide use (kg/ha) Consumption of pesticide (= all types of

plant protection agents used in
Rate of macroinvertebrates
agriculture or forestry).
in fresh water

Official water quality Additional information on water Beneficiaries and non-beneficiaries

indicators (e.g. oxygen quality
National and regional monitoring data on water
consuming substances in
quality
rivers, content nutrients in
freshwater) National/regional environmental agencies
reports

Evaluation studies on the environmental quality

Surveys

European Environmental Agency (for oxygen

consuming substances in rivers, nutrients in
freshwater): [Link]
and-maps

Timing of data collection

Proposed data on beneficiaries and non-beneficiaries should be collected prior to the RDP’s implementation (i.e.
years 2010-2013) and during the RDP’s implementation (i.e. 2014-2020: 1st call for proposal, operations database,
national/regional statistics, EU sources)

Methodology to calculate common and additional indicators

Common indicators

The common indicators - R8/T10 and R9/T11 are collected via the operations database and calculated as ratio of
agriculture land or forestry land under management contracts to improve water management of the total UAA. For those
operations programmed under other FAs and contributing to the FA 4B the secondary contributions to values of above
indicators are already captured in the monitoring system.

49
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Additional indicators

For investment related operations an additional indicator should be identified, e.g. when assessing the secondary
contributions to the FA (See below).

Additional indicators will be calculated based on data on fertilisers and pesticides use as well as monitoring of water
quality affected by agriculture (pesticides and fertilizers) of underground waters. For counterfactual similar observation
will be done for contracted and not-contracted land (as a control group). For further analysis on types of actions which
might have various effects on water quality, several control groups according the type of actions can be created. However,
data should include sufficient information on farm structure and management practices to enable the setting up of the
control groups and application of advanced matching techniques.

Secondary contributions (for additional indicators only)

Investment types of operations programmed under other than FA 4B can also show secondary contributions to the water
management. For investment related operations the additional result indicator should be used when assessing the
secondary contributions to the FA. Secondary contributions to the values of additional result indicators are estimated
based on surveys with those beneficiaries which are implementing operations under different focus areas (e.g. FA 2A,
4C) and have flagged the additional contribution to FA 4B. This also includes those operations implemented via CLLD
strategies, which show secondary contributions to the enhancement of biodiversity. Sample sizes of surveyed
beneficiaries can be set up in accordance with the expected/actual contributions of implemented operations (e.g. farm
investment measures) to the FA objectives. The ratio of the contribution is estimated based on the above surveys. Most
significant contributions should be taken in consideration. If there is no sufficient uptake to calculate the secondary
contributions in a robust way a qualitative assessment is also possible (see the Annex 10 of the guidelines: Assessment
of RDP results: how to prepare for reporting on evaluation 2017). Findings will be used also in answering the CEQ 19.

Qualitative assessment

For the qualitative assessment the chosen methods should enable one to describe the type of actions that are effective
in supporting water management, fertilizer and pesticide management. Qualitative assessments can be done via surveys
to beneficiaries, structured focus groups 52, and interviews with managers of measures/sub-measures programmed under
the FA. In relation to focus groups, the MAPP 53 method is relevant for assessing the effects of interventions on
environmental indicators that are difficult to measure (e.g. water management).

Challenges/Risks/Issues

• Not all pesticides equally affect water quality – as this depend on their toxicity, persistence, degraders and
environmental fate (behaviour). Reporting on a decrease of pesticide use using a “basket” of pesticides will provide
only partial information on their pressure on water quality.

• Indicators directly measuring the quality of freshwater bodies might not be easily attributable to beneficiaries and
non-beneficiaries.

• No data availability in RDP operations and other databases

• Low quality of data that does not enable a robust answer to the EQ

Proposed solutions

In the case of a lack of data (no databases available), these can be assessed only through systematic on-site monitoring
programmes. Such programmes should be well designed and implemented and be provided with adequate resources. In
fact, systematic, well organised water quality monitoring programmes are most suitable for judging if RDP operations

52
For details on how to use surveys and focus groups, see ‘Capturing the success of your RDP: Guidelines for ex post
evaluation of 2007-2013 RDPs’, Evaluation Helpdesk, June 2014
53
Investment support under Rural Development policy, DG Agri, 2014: [Link]
development-reports/2014/investment-support-rdp/fulltext_en.pdf
50
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

contributing to FA 4B really improved water quality. Strategic sampling of monitoring points increases the
representativeness of monitoring data and enables the creation of multiple comparison groups of beneficiaries and non-
beneficiaries. Thus MS/RDP MAs are encouraged to set-up such programmes to monitor pesticide and nitrogen loss in
water (leaching and run-off) in combination with reliable information on fertiliser (e.g. nitrogen) and pesticide management
of farms. Monitoring should involve both agricultural/forest land participating in RDP operations contributing to FA 4B and
those without RDP support (the later should serve as control group). Monitoring should record both the baseline situation
and progress during the implementation of the RDP’s operations.

In addition, pesticide and nitrogen ending-up in water depends on many (external) factors, notably rainfall, soil type, slope,
crop type – this monitoring needs to be very well designed, organised and results need to be interpreted correctly.
Sufficient resources should be allocated to implement it. To increase the cost-effectiveness of monitoring programmes,
multiple purposes of such programmes should be considered.

In the case of a lack of data (low uptake,), this EQ can focus on the estimated land to be covered by management contracts
and/or the expected improvements in water quality (via interviews and expert opinions).

In case of low quality of data, qualitative information can be sought through interviews, surveys, and focus groups.

Answer to CEQ

The answer is provided based on the calculated values of indicators, on the collected qualitative information or on the
qualitative assessment.

Conclusions Recommendations

Main conclusions are stated here for the FA based on the Main recommendations linked to the conclusions are
evaluation findings. stated here for the FA.

2.10 Focus Area P4C, Evaluation question 10

To what extent have RDP interventions supported the prevention of soil erosion and improvement of soil management?

List of measures contributing to the FA 4C:

Primarily programmed measures/sub-measures (proposed by EC) 54:

• Measures and sub-measures of Art. 55 15 Advisory services, farm management and farm relief services

• Measures and sub-measures of Art. 14 Knowledge transfer and information actions

• Measures and sub-measures of Art. 35 Co-operation

• Measures and sub-measures of Art. 21 Investments in forest area development and improvement of the viability of
forests

• Measures and sub-measures of Art. 20 Basic services and village renewal in rural areas

• Measures and sub-measures of Art. 34 Forest-environmental and climate services and forest conservation

• Measures and sub-measures of Art. 30 Natura 2000 and Water framework directive payments

Examples of measures/sub-measures programmed under the other FA but potentially showing the secondary
contribution to this FA:

54
WP: Guidelines for strategic programming for the period 2014-2020, European Commission, 04/03/2014, published on
ECAS/circabc
55
All articles mentioned in this section of each fiche for answering CEQ 1 – 18 are of Regulation (EU) No 1305/2013, unless it
is stated otherwise in the text directly
51
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

• All above measures if programmed under another FA other than FA 4C and contributing to the prevention of soil erosion
and improvement of soil management

• Measures and sub-measures of Art. 17 Investment in physical assets

• Measures and sub-measures of Art. 28 Agri-environment-climate

• Measures and sub-measures of Art. 29 Organic farming

• Measures and sub-measures of Art. 35 of Regulation (EU) no 1303/2013, Support for Leader local development

Consistency check between CEQ, judgement criteria and indicators:

Judgement criteria

Judgement criteria (JC) proposed by the Working Document: Common Evaluation Questions for Rural Development
Programmes 2014- 2020:

• Soil erosion has been prevented

• Soil management has improved

Note: stakeholders in the Member States might add/change judgement criteria in line with the intervention logic of the
FA (selection and combination of measures)

Indicators

The following common indicators should be preliminary used to answer the CEQ:

Common result/target indicator:

• R10/T12 - % of agricultural land under management contracts to improve soil management and/or prevent soil erosion

• R11/T13 - % of forestry land under management contracts to improve soil management and/or prevent soil erosion

The following other common indicators might be used to answer the CEQ:

Common output indicators 56 (Data collected via the operations database):

• O4 number of holdings/beneficiaries supported

• O5 total area (ha)

• O6 Physical area supported (ha)

Common context indicators (Data need and data sources are described in the Working document: Proposed list of common
context indicators):

• CCI 41 – Soil organic matters in arable land

• CCI 42 - Soil erosion by water

Additional indicators and information

Common indicators are not sufficient to answer the CEQ. WD Common Evaluation Questions for Rural Development
Programmes 2014-2020 suggests:

• Additional information on soil erosion of the land under management contracts, e.g.: official soil erosion indicators (e.g.
estimated soil erosion by water, measured in tonnes/ha/year, estimated soil erosion by wind measured in days/year)

56
WD: Data item list for Pillar II operation database and WD RD programming and target setting
52
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Further proposed additional indicators:

• Decrease of quantity of soil eroded

• Change in soil compaction

• Change in earthworm’s density (Note: earthworms are considered to be a good indicator of good/improved soil
management over a range of soil types and situations. They are easy to detect and count).

• Number and types of actions that contribute to prevent/improve soil erosion

Note: More additional indicators can be developed when the judgement criteria is updated to specify the evaluation
question in the Member State.

Qualitative indicators

Qualitative assessment might be also used, e.g.

• to assess the typology of actions that have been effective in preventing/improving soil erosion

• perception of beneficiaries on various actions taken in this respect, etc.

Data needs and data sources

Indicators Data needed Data sources

R10/T12 - % of Total agricultural land under land management Beneficiaries

agricultural land contracts to prevent soil erosion (data items O.6,
Application form (project start)
under management O.4)
contracts to improve Payment request (project end)
Total number of agricultural and forestry
soil management For total number of agriculture and forestry holdings
holdings for the RDP area (data item O.5)
Common indicators

and/or prevent soil

erosion IACS

Eurostat : [Link]
R11/T13 - % of Total forestry area under land management
explained/[Link]/Farm_structure_statistics#Main_
forestry land under contracts to prevent soil erosion (data items O.6,
tables
management O.4)
contracts to improve National/regional statistics
Total agriculture and forestry area in base year
soil management
for the RDP area (data item O.5)
and/or prevent soil
erosion

Decrease of quantity Quantity of soil eroded by water Beneficiaries and non-beneficiaries

of soil eroded (t
Quantity of soil eroded by wind On-site monitoring of agricultural/forestry land under
eroded soil/ha)
contract and land not under the contract
Additional indicators

National/regional environmental agencies reports

Evaluation studies on the environmental quality

Change in soil Area with compacted soil Beneficiaries and non-beneficiaries

compaction (% of
On-site monitoring of agricultural/forestry land under
area with decreased
contract and land not under the contract
soil compaction)

53
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Change in earth Number of earthworms per square metre. Beneficiaries and non-beneficiaries
worms density
On-site monitoring of agricultural/forestry land under
(number/m2)
contract and land not under the contract

Number and types Number of actions contributing to Beneficiaries

of actions that prevention/improvement soil erosion broken
Application form (project start)
contribute to down by type
Additional indicators

prevent/improve soil Payment request (project end)

erosion Surveys

Official soil erosion Additional information on soil erosion of the land European Environment Agency (for estimated soil
indicators (e.g. under management contracts erosion by water and by wind) 57
estimated soil
National/regional environmental statistics
erosion by water,
measured in
tonnes/ha/yr,
estimated soil
erosion by wind
measured in
days/year)

Timing of data collection

Proposed data on beneficiaries and non-beneficiaries should be collected prior to the RDP’s implementation (i.e. years
2010-2013) and during the RDP’s implementation (i.e. 2014-2020: 1st call for proposal, operations database,
national/regional statistics, EU sources)

Methodology to calculate common and additional indicators

Common indicators

The common indicators - R10/T12 and R11/T13, are collected via the operations database and calculated as ratio of
agriculture land or forestry land under management contracts to prevent soil erosion and improve soil management of the
total UAA. For those operations programmed under other FAs and contributing to the FA 4C the secondary contributions to
values of above indicators are already captured in the monitoring system.

Secondary contributions

Secondary contributions of operations which are programmed under other FAs than P4C are also considered when
calculating the common indicator. However, only operations linked to the area based measures could be considered.
Findings will be considered when answering the CEQ 19.

Additional indicators

For investment related operations an additional indicator should be identified, e.g. when assessing the additional
contributions to the FA. This requires a review of existing soil erosion indicators to select the most appropriate indicator for
the specific intervention logic and causal relationships of the investment measure.

Additional indicators will be calculated based on on-site monitoring of soil erosion of agricultural/forestry land under contract.
For counterfactual similar observation will be done for contracted and not-contracted land (as control group). Monitoring
should record both baseline situation and progress during implementation of RDP operations. For further analysis on types

57
European Environment Agency, Estimated soil erosion by water: [Link]
erosion-by-water-1/assessment
54
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

of actions which that might have various effects on soil erosion, several control groups according the type of actions can be
created. However, data should include sufficient information on farm structure and management practices to enable the
application of advanced matching techniques.

Secondary contributions (for additional indicators only)

Secondary contributions to the values of additional result indicators are calculated based on surveys with those beneficiaries
which are implementing operations via different focus areas (e.g. 2A, 4B, 6A) and have been flagged as providing additional
contributions to FA 4C. This also includes those operations implemented via CLLD strategies, which show secondary
contributions to the enhancement of biodiversity. Sample sizes of surveyed beneficiaries can be set up in accordance with
the expected/actual contributions of implemented operations (e.g. farm investment measures) to the FA objectives. Most
significant contributions should be taken in consideration. If there is no sufficient up take to calculate the secondary
contributions in a robust way a qualitative assessment is also possible (see the Annex 10 of the guidelines: Assessment of
RDP results: how to prepare for reporting on evaluation 2017). Findings can be used in answering the CEQ 19.

Qualitative assessment

For qualitative assessment the following methods are proposed:

• Survey to beneficiaries (and non—beneficiaries) of primarily and secondarily programmed measures/sub-measures

• Structured focus groups 58,

• Interviews with managers of these measures.

In relation to focus groups, the MAPP method is relevant for assessing the effects of interventions on environmental
indicators that are difficult to measure (such as soil erosion).
Challenges/Risks/Issues

• Low quality data that does not provide a robust answer to the EQ

• Lack of data to answer the EQ (e.g. low RDP uptake/no uptake, small programmes etc.)

• Systematised and harmonised data on trends in soil erosion in Europe and Member States are lacking (according to
the European Environment Agency).

• Soil erosion, change in soil compaction and earth worms density cannot be assessed using (general) data available in
RDP operations and other databases (e.g. soil maps, land use data, etc.). This can be assessed only through
systematic on-site monitoring programmes. Such programmes should to be well designed and implemented and
provided with adequate resources. The cost-effectiveness of such monitoring programme needs to be reviewed.

Proposed solutions

In case there is a lack of data (low uptake, small programmes), this EQ can focus on the estimated land to be covered by
management contracts and/or the expected improvements/prevention of soil erosion (via interviews and expert opinions).

In case of low quality data, qualitative information can be sought through interviews, surveys, and focus groups.

In case of a lack of data, use the EEA indicators of estimated soil erosion.

Modelling approaches (non-experimental design) can be used to calculate soil erosion, sediment retention and organic
carbon content in circumstances with limited observational soil data. However, the quality of the model input data determines
the level at which the results can be used and the type of comparison groups that can be designed for the assessment.

58
For details on how to use surveys and focus groups, see ‘Capturing the success of your RDP: Guidelines for ex post
evaluation of 2007-2013 RDPs’, Evaluation Helpdesk, June 2014
55
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

In case data is not available from the RDP’s operations and other databases (e.g. soil maps, land use data, etc.), the
indicators can be assessed only through systematic on-site monitoring programmes. Such programmes should to be well
designed and implemented and provided with adequate resources.

Answer to CEQ

The answer is provided based on the calculated values of indicators, on the collected qualitative information or on the
qualitative assessment.

Conclusions Recommendations

Main conclusions are stated here for the FA based on the Main recommendations linked to the conclusions are stated
evaluation findings. here for the FA.

56
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

2.11 Focus Area P5A, Evaluation question 11

To what extent have RDP interventions contributed to increasing efficiency in water use by agriculture?

List of measures contributing to the FA 5A:

Primarily programmed measures/sub-measures (proposed by EC) 59:

• Measures and sub-measures of Art. 60 15 Advisory services, farm management and farm relief services

• Measures and sub-measures of Art. 14 Knowledge transfer and information actions

• Measures and sub-measures of Art. 17, Investment in physical assets

• Measures and sub-measures of Art. 35 Co-operation

• Measures and sub-measures of Art. 28 Agri-environment-climate

Examples of measures/sub-measures programmed under other FAs but potentially showing secondary contributions to this FA:

• All above measures if programmed under another FA other than FA 5A and contributing to increasing efficiency in water use by agriculture

• Measures and sub-measures of Art. 19 Farm and business development

• Measures and sub-measures of Art. 18 Restoring agricultural production potential damaged by natural disasters and introduction of appropriate prevention

• Measures and sub-measures of Art. 21 Investment in forest area development and improvement of viability of forests (sub-measures 8.1 and 8.2)

• Measures and sub-measures of Art. 27, Setting up producer groups and organisations in agriculture and forestry sectors

• Measures and sub-measures of Art.34 Forest-environment and climate services and forest conservation

• Measures and sub-measures of Art. 35 of Regulation (EU) no 1303/2013, Support for Leader local development

59
WP: Guidelines for strategic programming for the period 2014-2020, European Commission, 04/03/2014, published on ECAS/circabc
60
All articles mentioned in this section of each fiche for answering CEQ 1 – 18 are of Regulation (EU) No 1305/2013, unless it is stated otherwise in the text directly

57
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Consistency check between CEQ, judgement criteria and indicators:

Judgement criteria

Judgement criteria (JC) proposed by the WD Common Evaluation Questions for Rural Development Programmes 2014- 2020:

• Efficiency in water use by agriculture has increased

Note: stakeholders in Member States might add/change judgement criteria in line with the intervention logic of the FA (selection and combination of measures)

Indicators

The following common indicators should be used to answer the CEQ:

Common result indicator:

• R12/T14 - % of irrigated land switching to more efficient irrigation system

• R13 - Increase in efficiency of water use in agriculture in RDP supported projects (complementary result indicator)

The following other common indicators might be used to answer the CEQ:

Common output indicator 61 (Data collected via the operations database):

• O5 Total area (ha),

• O6 Physical area supported (ha)

Common context indicator (Data need and data sources are described in the Working document: Proposed list of common context indicators):

• CCI 39 Water abstraction in agriculture (also impact indicator 10)

Additional indicators and information

Since the common indicator is not sufficient to answer the evaluation question additional quantitative indicators are suggested:

61
WD: Data item list for Pillar II operation database and WD RD programming and target setting

58
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

• Percentage of area under RDP-supported operations practising residue management (e.g. mulching);

• Percentage of area under RDP-supported operations practising conservation tillage;

• Percentage of area under RDP-supported operations practising soil carbon-building measures.

Note: More additional indicators can be developed when judgement criteria are added to specify the evaluation question in MS.

Qualitative indicators

Further on the evaluator can collect additional qualitative information, e.g. perception of beneficiaries on how efficiency in water use by agriculture has increased as a result of the
RDP support (Likert scale)

Data needs and data sources

Indicators Data needed Data source

R12/T14 - % of irrigated land switching Beneficiaries Beneficiaries

Common indicators

to more efficient irrigation system • Number of hectares switching to more efficient Application forms (before the project start)
irrigation systems through the RDP measures 4
Payment request (after the project end)
and 10
Beneficiaries and non-beneficiaries
• Total irrigated land in base year for the RDP area
Eurostat (for the total irrigated land in base year)

59
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

R13 - Increase in efficiency of water use Beneficiaries: Beneficiaries:

in agriculture in RDP supported projects • Identification and basic characteristics (size, type Application forms (before the project start)
yield per crop, quantity of water used for irrigation
Payment request (after the project end)
per crop, technology used) of projects with a water
saving/efficiency component Information from the Beneficiaries and non-beneficiaries
completed project on the situation before and after
Farm structure survey
project implementation about the items mentioned
above Survey on agriculture production methods (SAPM)

Beneficiaries and non-beneficiaries Field research

• Yield per crop FADN - Total output per farm:

• Quantity of water used for irrigation per total output • Formula: [Link]
Common indicators

of farm (FADN)
• Database: [Link]

Eurostat – standard outputs:

Standard outputs per MS and region and year – average 2010:

[Link]

Total standard outputs (overall economic size of farm) per MS and region and per
year:

[Link]

National and regional statistics

Projects:

OpiRIS - the EU 7th framework programme: Irrigation scheduling expert system –

OpIRIS, [Link] which provides the advice on how to set up the
precise irrigation scheduling and increase the water efficiency, which does not relate
only to technology but also to knowledge and management practice. OpIRIS is a low
cost autonomous system dealing with water, fertilizers and energy reduction in
Hydroponic Greenhouses and Trees Orchards. OpIRIS provides free access to a
database that helps appraising the potential impact of knowledge-based, irrigation

60
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

management and precision scheduling strategies on water productivity (final yield

versus irrigation water) in fruit trees orchards and hydroponic crops. Public visitors can
personalise the results by filtering OPIRIS’ database according to a specific
combination of crop, variety, soil, irrigation system and climatic zone.

CCI 39 Water abstraction in agriculture All farmers All farmers

(also impact indicator 10)
indicators
Common

Volume of water which is applied to soils for Farm structure survey

irrigation purposes in m³
Survey on agriculture production methods (SAPM)

National and regional data

Percentage of area under RDP- Beneficiaries and non-beneficiaries Beneficiaries:

supported operations practising residue
• Area under operations practising residue Application forms (before the project start)
management (e.g. mulching)
management
Payment request (after the project end)
Additional Indicators

Beneficiaries and non-beneficiaries

Farm survey

Percentage of area under RDP- • Area under operations practising conservation Beneficiaries:
supported operations practising tillage
Application forms (before the project start)
conservation tillage
Payment request (after the project end)

Beneficiaries and non-beneficiaries

Farm survey

61
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Percentage of area under RDP- • Area under operations practising other carbon – Beneficiaries:
supported operations practising soil building measures
Application forms (before the project start)
carbon-building measures
Payment request (after the project end)

Beneficiaries and non-beneficiaries

Farm survey

Methodology to calculate complementary result indicator R13 - Increase in efficiency of water use in agriculture in RDP supported projects (net value)

Water use efficiency is a function of gathering, storing, conveying and using water efficiently on the crop. Gains should be relatively easy to assess if monitoring equipment is in place
at either the end of the irrigation conveyance system. Some projects may relate to collective management of irrigation systems so sometimes multi-holding collective appraisal will be
needed.

The steps identified below should be followed.

Step 1: Identify sample of completed operations on farms who have adopted enhanced irrigation practices with RDP support

Step 2: Select a control group with the same characteristics but has not adopted enhanced irrigation practices with RDP support.

Step 3: Assess consumption of irrigated water (in m³) per tonne of irrigated crop. The situation as to water consumption and planned situation after the implementation of the investment
before the start of the programming period (baseline) for both beneficiaries (e.g. in application forms) and non-beneficiaries (survey), and the situation 1 year after the investment (for
beneficiaries could be used the payment request, and beneficiaries and non-beneficiaries survey sheets etc.).

Step 4: Assess the change in water use on supported farms and unsupported farms 62 before and after the investment (see above).

This indicator should be expressed as a percentage of the increase in efficiency of water use. This should be calculated by comparing use of irrigated water (in m3) per standard unit
of output both in RDP-supported projects and in the control group using the matching approach. Suppose that one of the variables for matching for beneficiaries and control farms is
the use m3 irrigated water to obtain the standard output. The increase in efficiency of water use in RDP-supported projects for crop A x%. The same can be used for crop B (y%,), C

62
Pay attention whether water-intensive farms are dropping out. If this happens – for instance because of increased water price, one could get a wrong overall picture on the success of RDP measures.

62
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

(y%), etc. Following this approach, the comparison of the increase in efficiency of water use across Member States can be done for standard unit of outputs for which data are available
in the Member States.

Secondary contributions

When calculating the indicator, both – primary and secondary contributions should be taken in consideration. Secondary contributions are understood as contributions of operations
programmed and implemented under FAs other than P5A to the values of result indicators. The secondary contributions are calculated based on survey with those beneficiaries which
are implementing operations via different focus areas (e.g. 2A, 4B and C) and have been flagged as additional contributors to FA 5A. This also includes those operations implemented
via CLLD strategies, which show secondary contributions to the water use efficiency. Samples of surveyed beneficiaries can be set up in accordance with the expected/actual
contributions of implemented operations (e.g. farm investment measures, environment & climate, forestry related measures etc.,) to the FA objectives. Most significant contributions
should be taken in consideration. The ratio of the contribution is estimated based on the above surveys. If there is not sufficient uptake to calculate the secondary contributions in a
robust way a qualitative assessment is also possible (see the Annex 10 of the guidelines: Assessment of RDP results: how to prepare for reporting on evaluation 2017). Findings can
be used in answering the CEQ 19.

Qualitative assessment

The qualitative assessment is done via surveys, interviews and focus groups, which can serve to contextualize the water use and its efficiency, namely:

• identifying and describing all the factors that contribute to the loss of water in an irrigation system (climate, soil type, hydrology, type of irrigation and topography). These factors
are unpredictable and heterogeneous and therefore complicate management and measuring;

• screening and describing unanticipated negative effects in the selected cases 63; and

• considering the water use at multiple scales from field, to farm to drainage basin/catchment. Thus, there is a need to look at the impacts on the whole water system.

As an example of wider system effects, drip irrigation can impact the groundwater recharge (reducing it) and thereby impact downstream users. Better capture of water by the crop
will reduce stream flow and increase the pollution load, so a wider scanning of the catchment for adverse effects should be undertaken where irrigated areas are expanded or
irrigation technologies are changed.

63
See Ward and Pulido-Velazque, 2008 at: [Link]

63
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Principal challenges

• Water use efficiency can be considered at multiple scales from field, to farm to drainage basin/catchment. All are legitimate concerns and scale of evaluation should be contingent
on the scale of the RDP-supported scheme. Arguably there is a need to look at the whole water system and related data linked to the irrigated crop. However, the level of efficiency
that can potentially be attained is not simply the amount of water taken up by a crop compared to the amount of water applied (Fairweather et al.). In the assessment of indicators
and setting up the control groups, it is important to take into account all the factors that contribute to the loss of water in an irrigation system (climate, soil type, hydrology, type of
irrigation and topography). There might be some difficulties in data availability, since many factors are mostly unpredictable and heterogeneous and therefore complicate the
measurement of the system.

• As an example of wider system effects, drip irrigation can impact on groundwater recharge (reducing it) and thereby impacting downstream users. Unanticipated negative effects
should be screened for in selected cases 64 Better capture of water use by the crop will reduce stream flow and increase the pollution load, so a wider scanning of the catchment
for adverse effects should be undertaken where irrigated areas are expanded or irrigation technologies are changed.

• Eurostat reports that there is likely to be significant illegal and unmonitored use of water in some parts of Europe.

• It is very important to note that irrigation is not THE ONLY technique increasing efficiency in water use by agriculture. Several other techniques are also important in increasing
efficiency of water use in agriculture. For instance:

o Residue management (e.g. mulching) leaving plant residue on the soil surface improve the ability of the soil to hold moisture – and reduces water run-off from the field – as
well as surface evaporation.

o Conservation tillage – same effect as above.

o Soil-carbon building practices (e.g. regular application of livestock manure, growth of perennial legumes) increases soil carbon and water holding capacity. Some of these
are part of agri-environment schemes. Employing alternative sources of water for irrigation, such as rainwater harvesting and utilising treated (low salinity) wastewater.

How to use the indicator in the situation of lack of data (low RDP uptake, etc.)

The approach adopted assumes limited data at catchment scale but that crop level data is possible to collect to generate indicator values. Where uptake is low care must be taken
to ensure a sample size to deliver the requisite confidence in estimates.

Example

Weblink: [Link]

64
See for instance Ward and Pulido-Velazque, 2008 at: [Link]

64
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Bio-intelligence Services and others, see BIO Intelligence Service, Cranfield University and Risk and policy Analysts (2012) Water saving potential in agriculture in Europe: findings from the existing
studies and application to case studies.

Anon (2015) Example Irrigated Farm Water Use Efficiency Assessment (IFWUEA)

[Link] (Example of detailed analysis of water use efficiency at farm scale based on Australian (New South Wales)
study).

Sub-surface drip irrigation, for instance: [Link]

Advantages of the Watering Community, for instance at: [Link]

Further information

Anon (2015) Example Irrigated Farm Water Use Efficiency Assessment (IFWUEA) [Link]

BIO Intelligence Service, Cranfield University and Risk and policy Analysts (2012) Water saving potential in agriculture in Europe: findings from the existing studies and application to case studies.
[Link]

Burt, C. M., Clemmens, A. J., Strelkoff, T. S., Solomon, K. H., Bliesner, R. D., Hardy, L. A., Howell, T. A. and Eisenhauer, D. E. (1997) Irrigation performance measures: efficiency and uniformity. Journal
of Irrigation and Drainage Engineering, 123:423-442 available at [Link]

Fairweather, H., Austin, N. and Hope. M., (undated) Water use Efficiency An information package Water Insights 5.

This provides very useful contextual information and efficiency ratios for use in irrigation appraisal. It was prepared for Australia but the ratios and thinking are highly pertinent for Europe.

Answer to CEQ

The answer is provided based on the calculated values of indicators, on the collected qualitative information or on the qualitative assessment.

Conclusions Recommendations

Main conclusions are stated here for the FA based on the evaluation findings. Main recommendations linked to the conclusions are stated here for the FA.

65
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

2.12 Focus Area P5B, Evaluation question 12

To what extent have RDP interventions contributed to increasing efficiency in energy use in agriculture and food processing?

List of measures contributing to the FA 5B:

Primarily programmed measures/sub-measures (proposed by EC) 65:

• Measures and sub-measures of Art. 66 15 Advisory services, farm management and farm relief services
• Measures and sub-measures of Art. 14 Knowledge transfer and information actions
• Measures and sub-measures of Art. 17, Investment in physical assets
• Measures and sub-measures of Art. 35 Co-operation
• Measures and sub-measures of Art. 20 Basic services and village renewal in rural areas

Examples of measures/sub-measures programmed under the other FA but potentially showing secondary contributions to this FA:

• All above measures if programmed under another FA other than FA 5B and contributing to increasing efficiency in energy use by agriculture
• Measures and sub-measures of Art. 19 Farm and business development
• Measures and sub-measures of Art. 27, Setting up producer groups and organisations in agriculture and forestry sectors
• Measures and sub-measures of Art. 35 of Regulation (EU) No 1303/2013, Support for Leader local development

Consistency check between CEQ, judgement criteria and indicators

Judgement criteria

Judgement criteria (JC) proposed by the Working Document: Common Evaluation Questions for Rural Development Programmes 2014- 2020:

• Efficiency of energy use in agriculture and food processing has increased

Note: Member States might add/change judgement criteria in line with the intervention logic of the FA (selection and combination of measures)

65
WP: Guidelines for strategic programming for the period 2014-2020, European Commission, 04/03/2014, published on ECAS/circabc
66
All articles mentioned in this section of each fiche for answering CEQ 1 – 18 are of Regulation (EU) No 1305/2013, unless it is stated otherwise in the text directly

66
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Indicators

The following common indicators should be used to answer the CEQ:

Common result indicators:

• T15 - Total investment for energy efficiency

• R14 - Increase in efficiency of energy use in agriculture and food-processing in RDP supported projects (complementary result indicator)

The following other common indicators might be used to answer the CEQ:

Common output indicators 67 (data collected via the operations database):

• O2 Total investment € (public + private)

Common context indicator (data need and data sources are described in the Working document: Proposed list of common context indicators):

• CCI 44 Energy use in agriculture, forestry and food industry

Additional indicator and information

As additional indicator the following one can be used to look at the economic gains of the energy savings>

• Standard output/MJ energy input

Note: If stakeholders in MS have added additional judgement criteria, then other indicators, consistent with the proposed additional judgement criteria, may be included here

Qualitative information

Additional qualitative information may also be collected in the assessment of whether efficiency in energy use in agriculture and food processing has increased, for example:

• The perception of beneficiaries on how efficiency of energy use in agriculture and food processing has increased as a result of the RDP support (Likert scale)
• The types of actions that have proved effective in improving energy efficiency

67
WD: Data item list for Pillar II operation database and WD RD programming and target setting

67
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Data needs and data sources per indicator

Indicators Data needed Data source

T15 Total investment for energy Total amount of investments in energy saving and Beneficiaries
efficiency efficiency supported under the RDP
Application forms (prior the project start)

Payment request (after project completion)

R14 Increase in efficiency of Beneficiaries: Beneficiaries:

energy use in agriculture and • Total energy use data (fuel, electricity) in T.O.E/per • Application forms (prior the project start)
food-processing in RDP standard unit of output • Payment requests (after project completion)
• Energy audit (especially for processing industries)
supported projects Non- Beneficiaries:
Beneficiaries and non-beneficiaries:
• Total energy use (fuel, electricity) in T.O.E/per
• EUROSTAT – energy statistics
standard unit of output
Common indicators

• Farm survey of selected beneficiaries and non-beneficiaries

• Survey of supported and non-supported food firms
FADN - Total output per farm:

Formula: [Link]

Database: [Link]

Eurostat – standard outputs:

Standard outputs per MS and region and year – average 2010:

[Link]

Total standard outputs (overall economic size of farm) per MS and region and per year:

[Link]

National and regional statistics

68
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Standard output/MJ energy input Standard outputs per crop and animal (see annex of this Beneficiaries and non-beneficiaries:
Additional
indicator
document – standard outputs tables • EUROSTAT – standard outputs
• Farm survey of selected beneficiaries and non-beneficiaries
• Survey of supported and non-supported food firms

Timing of data collection

Proposed data on beneficiaries and non-beneficiaries should be collected prior to the RDP’s implementation (i.e. years 2010-2013) and during the RDP’s implementation (i.e.
2014-2020: 1st call for proposal, operations database, national/regional statistics, EU sources).

Methodology to calculate complementary result indicator R14 - Increase in efficiency of energy use in agriculture and food-processing in RDP supported projects
(net value)

It is only possible to estimate the efficiency gains in energy use from RDPs if all RDP-related agriculture and food processing projects are screened for energy use effects.
However, energy use should be assessed only where there are significant effects. For example, a 5% change as the threshold could be used. Thus, this analysis shall be
restricted to the main measures within Focus Areas which are likely to affect energy use both positively and negatively (using expert judgement as to whether the measure
might reach the suggested 5% threshold value). In this scope all measures under FA 5B should be considered (but the minimum should be a focus on Measure 4 and 7).

Energy use will arise from electricity use (e.g. in cooling systems for dairy farms), in fuel use with machinery and tractors, in electricity use with processing food industries.
These are reduced to a TOE figure based on a standard multiplier (suggested figure: 1 kW/hr= 0.000085984522785899 ([Link]
[Link]).

Coefficients for conversion of various energy sources to T.O.E: e.g. Directive 2009/28/EC; International Energy Agency: [Link]

Standard unit of outputs (Eurostat) are used as a denominator in the calculation of the indicator. An energy audit should be a prerequisite for any farm, processing industry or
rural business (or potentially Leader–funded entity) applying for this measure.

The following steps should then be used in the calculation of the indicator:

Step 1: Specify subset of RDP measures for which assessment is needed. This should be decided at the RDP level.

Step 2: Take stratified samples of all relevant RDP-supported projects (weighted by RDP expenditures).

Step 3: Identify parallel samples of RDP funded projects as the control group.

69
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Step 4: Use energy use records calculated per standard unit of output per commodity (€) and or survey to identify before, and after energy use data on supported and
unsupported farms/firms.

Step 5: Assess before and after energy consumption in T.O.E per standard unit of output (€) in supported and control farms and firms.

Step 6: Assess the net energy savings on supported farms and firms (per standard unit of output) in supported and control farms and firms, this ratio should increase as a result
of the intervention.

Step 7: Compare the ratio on supported versus unsupported farms. This ratio should increase as a result of the intervention

Secondary contributions

When calculating this indicator, both – primary and secondary contributions should be taken in consideration. Secondary contributions are effects of operations contributions
and understood as contributions of operations programmed and implemented under other FAs than P5B to the values of result indicators. The secondary contributions are
calculated based on survey with those beneficiaries which are implementing operations via different focus areas (e.g. FA 2A, 6A, 6B) and have been flagged as additional
contributors to FA 5B. This also includes those operations implemented via CLLD strategies, which show secondary contributions to the water use efficiency. Samples of
surveyed beneficiaries can be set up in accordance with the expected/actual contributions of implemented operations (e.g. farm investment measures, environment & climate,
forestry related measures, etc.) to the FA objectives. Most significant contributions should be taken in consideration. The ratio of the contribution is estimated based on the
above surveys. If there is not sufficient uptake to calculate the secondary contributions in a robust way a qualitative assessment is also possible (see the Annex 10 of the
guidelines: Assessment of RDP results: how to prepare for reporting on evaluation 2017). Findings can be used in answering the CEQ 19.

70
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Principal challenges:

• Where energy use is associated with renewable energy resources the TOE measure is not an appropriate measure of contribution to emissions, but still reflects efficiency.
A savings in GHG emissions will arise but there may be a substitution of renewable for non-renewable energy, which means the net emissions will be much less. Equally,
TOE using an EU average will not reflect member state differences in the renewable proportion of their electricity production.

• Where energy use is associated with a shift from the application of nitrogenous fertilisers which embody large amounts of energy to organic production, this will be reflected
in the reduced use of and demand for nitrogenous fertiliser and will be reflected in fertiliser industry figures not farm level change. ONLY on-farm energy use should be
considered, but evaluators should recognise that some activities can be outsourced (e.g. contracting of machinery operations) which reduces farm energy use but not total
energy use. Such energy savings could be illusory.

• FA 5A is essential for the efficiency of energy use, so the source (non-renewable vs renewable) is, to a degree, immaterial for this indicator. We should not be concerned
with sourcing of energy inputs (which is important for TOE estimates) but with efficiency of energy use. The TOE figure is only used to estimate the amount of energy use
in a single denominator.

• Because of price volatility linked to different products it is proposed to use as the standard output the volume of production per crop (farms) and commodity (food processing
firms).

How to use the indicator in the situation of lack of data (low RDP up-take, small programmes etc.)

This is unlikely to be an issue as energy use is universal. The key challenge is the screening of projects to determine which projects fall into consideration.

Example (including the web page link if existing)

Example 1 Farm: Northern Ireland pig farm. The farm generated the following savings following energy audit (in this case not RDP funded):
• Milling & mixing costs halved per tonne output creating major savings in energy use and costs
• Less electricity used for lighting, through change in lighting system
• Improved temperature control on heat pads in farrowing house for sows
• 1000 litres of oil saved annually in heating
• Over 3.5 t CO2 emissions saved annually.

These figures can readily be converted into a TOE figure using on line conversion tables.

71
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Example 2 Food firm

Chivas Brothers who are malt whisky producers undertook a careful analysis of its sites and the suitability of thermo-compressors, to guard spirit quality before implementing
its rolling programme to retrofit the technology at some of its distilleries with existing heat recovery systems - a programme in place since 2005. The pursutechnology was
included in Chivas' expansion of The Glenlivet distillery in 2009, where the new still house uses 20% less energy than the existing part of the site. Benchmarking of distillery
performance has confirmed these savings. All distilleries employing thermo-compressors now use between 10% - 20% less energy per unit of alcohol than would be expected
for their size category. The actual carbon savings vary by distillery size and fuel type. For example, in distilleries with at least six stills, average annual reductions of 1,750 and
2,000 tonnes of CO2 have been achieved, for gas and oil fuelled sites respectively. Assuming gas or diesel heat source, this amounts to 0.25 kgs CO2 per kW hr of heat
produced using Carbon Trust conversion factors, which then needs to be converted to TOE.

Source: [Link]

Further information

Carbon Trust (2006) [Link]

Carbon Trust (2012) Food and drink processing: Introducing energy saving opportunities for business

Carbon Trust (no date) Industrial Energy Efficiency Accelerator Guide to the dairy sector (one of many guides to different parts of food sector)

DARDNI (2011) Farmer Case Studies: How to reduce costs and cut Greenhouse Gas (GHG) Emissions

[Link] this provides advice on energy reduction measures

European Commission (JRC) (2009) Reference Document on Best Available Techniques for Energy Efficiency

[Link]

[Link]

[Link] film of case of brewery

72
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Answer to CEQ

The answer is provided based on the calculated values of indicators, on the collected qualitative information or on the qualitative assessment.

Conclusions Recommendations

Main conclusions are stated here for the FA based on the evaluation findings. Main recommendations linked to the conclusions are stated here for the FA.

2.13 Focus Area P5C, Evaluation question 13

To what extent have RDP interventions contributed to the supply and use of renewable sources of energy, of by-products, wastes, residues and other non-food raw material for
purposes of the bio-economy?

List of measures contributing to the FA 5C:

Primarily programmed measures/sub-measures (proposed by EC) 68:

• Measures and sub-measures of Art. 69 15 Advisory services, farm management and farm relief services
• Measures and sub-measures of Art. 14 Knowledge transfer and information actions
• Measures and sub-measures of Art. 17 Investment in physical assets
• Measures and sub-measures of Art. 35 Co-operation
• Measures and sub-measures of Art. 20 Basic services and village renewal in rural areas
• Measures and sub-measures of Art. 19 Farm and business development
• Measures and sub-measures of Art. 26 Investments in forestry technologies and in processing, mobilising and marketing of forestry products
• Measures and sub-measures of Art. 21 investments in forest area development and improvement of the viability of forests

Examples of measures/sub-measures programmed under the other FAs but potentially showing secondary contributions to this FA:

• All above measures if programmed under another FA other than FA 5C and contributing to the supply and use of renewable sources of energy, of by-products, wastes, residues
and other non-food raw material for purposes of the bio-economy
• Measures and sub-measures of Art. 29 Organic farming
• Measures and sub-measures of Art. 27 Setting up producer groups and organisations in agriculture and forestry sectors

68
WP: Guidelines for strategic programming for the period 2014-2020, European Commission, 04/03/2014, published on ECAS/circabc
69
All articles mentioned in this section of each fiche for answering CEQ 1 – 18 are of Regulation (EU) No 1305/2013, unless it is stated otherwise in the text directly

73
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

• Measures and sub-measures of Art. 35 of Regulation (EU) No 1303/2013, Support for Leader local development

Consistency check between CEQ, judgement criteria and indicators

Judgement criteria

Judgement criteria (JC) proposed by the Working Document: Common Evaluation Questions for Rural Development Programmes 2014- 2020:

• The supply of renewable energy has increased

• The use of renewable energy has increased

Additional judgment criteria:

• Improved economic performance due to renewable energy use

Note: stakeholders in Member States might add/change judgement criteria in line with the intervention logic of the FA (selection and combination of measures)

Indicators

The following common indicators should be preliminary used to answer the CEQ:

Common result indicators:

• T16 - Total investment in renewable energy production

• R15 - Renewable energy produced from supported projects (complementary result indicator)

The following other common indicators might be used to answer the CEQ:

Common output indicators 70(data collected via the operations database):

• O2 - Total investments € (public and private)

Common context indicators (data need and data sources are described in the Working document: Proposed list of common context indicators):

• CCI 43 Production of renewable energy from agriculture and forestry

• CCI 44 Energy use in agriculture, forestry and food industry

70
WD: Data item list for Pillar II operation database and WD RD programming and target setting

74
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Additional indicators and information

Common indicators are not sufficient to answer the CEQ. WD Common Evaluation Questions for Rural Development Programmes 2014-2020 suggests:

• Total investments for the use of renewable energy supported by the RDP, including by type of renewable energy
• Renewable energy used in supported holdings,

Further proposed additional indicators:

• Changes in gross Farm Income due to energy consumptions (additional JC: improved economic performance due to renewable energy use)
• Changes in family farm income due to energy consumptions (additional JC: improved economic performance due to renewable energy use)

Note: If stakeholders in the Member States have added additional judgement criteria, or other indicators, consistent with the proposed additional judgement criteria, it may be
included here

Qualitative indicators

Qualitative information might be also used, e.g.

• perception of beneficiaries on how the supply and use of renewable energy has increased as a result of the RDP support (Likert scale), and/or
• description of actions to increase the production and supply of renewable energies (e.g. biomass based through cultivating annual and perennial crops such as grasses and
coppice trees, forest biomass based, biogas, etc.)

75
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Data needs and sources

Indicators Data needed Data source

T16 Total investment in Beneficiaries Beneficiaries

renewable energy production • Total investment in renewable energy production under • Application forms (before the project start)
measures 4, 6.2, 6.4, 7.2 and 8.6
• Payment request (after project finalisation)
Beneficiaries and non-beneficiaries
Beneficiaries and non-beneficiaries
• Total investment in renewable energy production in agriculture
and forestry Survey on beneficiaries and non-beneficiaries

National and regional statistics

Energy supply and control entities data

Common indicators

Eurostat – energy statistics

[Link]

R15 - Renewable energy Beneficiaries: Beneficiaries

produced from supported • Installed capacity and efficiency factors via using monitoring • Application forms (before the project start)
projects technology
• Payment request (after project finalisation)
• Renewable energy production
Beneficiaries and non-beneficiaries
Non-beneficiaries:
National/regional statistics
• Renewable energy production in agriculture and forestry
Energy supply and control entities data

Eurostat – energy statistics

[Link]

Surveys/focus groups

76
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Total investments for the use Beneficiaries: Beneficiaries

of renewable energy
Data on total investments in technologies which enable the use • Application forms (before the project start)
supported by the RDP,
including by type of renewable of renewable energies in supported holdings • Payment request (after project finalisation)

energy Beneficiaries and non-beneficiaries

Beneficiaries and non-beneficiaries
Additional indicators

National/regional statistics
Data on investments in energy infrastructure in agriculture and
forestry Eurostat – energy statistics

[Link]

Data on investments in energy infrastructure in agriculture and forestry

Energy supply and control entities data

Surveys/focus groups

77
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Renewable energy used in Beneficiaries: Beneficiaries

supported holdings
Data on used renewable energy in supported holdings • Application forms (before the project start)
• Payment request (after project finalisation)
Beneficiaries and non-beneficiaries
Beneficiaries and non-beneficiaries

Data on used renewable energy

National/regional statistics

Eurostat – energy statistics

[Link]

Surveys/focus groups

Data on used renewable energy

Energy supply and control entities data

Changes in gross Farm Income Beneficiaries and non-beneficiaries Beneficiaries:

Additional indicators

due to energy consumptions

Gross Farm Income (GFI) = SE410 = Output – intermediate Application forms and payment request
consumption + balance current subsidies & taxes
Beneficiaries and non-beneficiaries

FADN; farms surveys

78
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Changes in family farm income Beneficiaries and non-beneficiaries Beneficiaries:

due to energy consumptions
Family Farm Income (FFI) = SE420 = farm net value added – total Application forms and payment request
external factors + balance on subsidies and taxes on investment
Beneficiaries and non-beneficiaries

FADN; farm surveys

Eurostat - Economic accounts for agriculture (EAA)

Eurostat - Agriculture Labour Input statistics (ALI)

[Link]

Methodology to calculate complementary result indicator R15 - Renewable energy produced from supported projects

The production of renewable energy is rarely monitored as energy production, but rather by the capacity of technologies purchased and used by holdings supported via the RDP to
produce renewable energy (RDP beneficiaries). For this analysis, however, it is important to look at the installed capacities and realised production after finalisation of the installation
at the same time.

The following steps should be used to calculate the indicator R15:

Step 1: Define a sample of completed operations in relation to this FA and gather the following data: renewable energy capacity created and generated per renewable due to the RDP
support.

Step 2: Convert to TOE figures using standard formula: e.g. use on-line converter: [Link] and/or
[Link]

Step 3: Express energy produced in TOE

Note: The submitted figure should be based on production not export, as domestic or farm use of renewables may reduce the export of power to the wider energy system.

Secondary contributions

79
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

When calculating the indicator, both – primary and secondary contributions should be taken in consideration. Secondary contributions are effects of operations which are programmed
under other FAs than P5C. The secondary contributions are calculated based on surveys with those beneficiaries which are implementing operations via different focus areas (e.g.
FA 2A, 6A, 6B) and have been flagged as additional contributors to FA 5C. This also includes those operations implemented via CLLD strategies, which show secondary contributions
to the water use efficiency. Samples of surveyed beneficiaries can be set up in accordance with the expected/actual contributions of implemented operations (e.g. farm investment
measures, environment & climate, forestry related measures etc.,) to the FA objectives. Most significant contributions should be taken in consideration. The ratio of the contribution is
estimated based on the above surveys. If there is not sufficient uptake to calculate the secondary contributions in a robust way a qualitative assessment is also possible (see the
Annex 10 of the guidelines: Assessment of RDP results: how to prepare for reporting on evaluation 2017). Findings can be used in answering the CEQ 19.

Negative effects of renewables

It’s important to identify and describe the negative effects of renewable energy production (sustainability) in the programme area. There are some widely recognised negative effects
of renewables, which include the depletion of river reaches and the impedance of migratory fish in the case of hydropower and visual intrusion in the case of wind energy turbines.
Also the use of farmland for biomass cropping for energy production consumes a great deal of energy in the production process.

These are addressed in The Report From The Commission To The Council And The European Parliament (2010) on sustainability requirements for the use of solid and gaseous
biomass sources in electricity, heating and cooling. See [Link] Where a significant proportion of renewable
energy production comes from energy crops, Member States are strongly recommended to assess sustainability impacts. European Parliament protocols should be followed: see
[Link]

80
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Principal challenges regarding CRI

• Many MSs will have other means of funding renewables which are also adopted by farmers, such as national level feed in tariffs. These should be ignored in this calculation as
the renewable energy production should be only from RDP-supported projects. Where national schemes exist, the additionality of the RDP supported schemes may be modest
and this should be noted by the evaluator. For example, currently in the UK, wood energy use is supported by the Renewable Heat Incentive, which is being widely adopted by
farmers. But it falls outside the definition of RDP-supported projects.

• The different technologies have different side effects which may be positive or negative: for example, biogas production produces bio-digestive waste which is a valued fertiliser;
biomass energy has a combustion method that is not clean.

These issues fall outside of this indicator but where they are deemed significant they should be flagged rather than measured. Where countries have high levels of uptake of biogas
measures, the use of EU sustainability criteria for biofuels is recommended (see earlier note).

How to use the indicator in the situation of lack of data (low RDP up-take, small programmes etc.)

The application forms should have a tight specification of the production of renewable energy to allow the estimation of this kind of energy. The confirmation of the quantity of energy
produced through renewable sources should be done by the beneficiaries after the project has reached a “cruising speed”.

In the case of low uptake, this EQ can focus on an estimation (via interviews and expert opinions).

Further information

Banks C, (2009) Optimising anaerobic digestion, conference presentation

[Link]

Bestwick, M. (pers. comm.) Highland eco-design hydropower figures for Scottish example project.

Heinsuu, K, (undated) BioEnergy Farm: Implementation plan for BioEnergy Farm IEE Contract No: IEE/09/637 S12.558213 Intelligent Energy Europe

Held J., Mathiasson,A. and Nylander, A.,(2008)Biogas from manure and waste products - Swedish case studies

Van Foreest, F. (2012) Perspectives for Biogas in Europe, Oxford Institute for Energy Studies NG 70

On-line converter: [Link]

81
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Answer to CEQ

The answer is provided based on the calculated values of indicators, on the collected qualitative information or on the qualitative assessment.

Conclusions Recommendations

Main conclusions are stated here for the FA based on the evaluation findings. Main recommendations linked to the conclusions are stated here for the FA.

2.14 Focus area P5D, evaluation question 14

To what extent have RDP interventions contributed to reducing GHG and ammonia emissions from agriculture?

List of measures contributing to the FA 5C:

Primarily programmed measures/sub-measures (proposed by EC) 71:

• Measures and sub-measures of Art. 72 15 Advisory services, farm management and farm relief services
• Measures and sub-measures of Art. 14 Knowledge transfer and information actions
• Measures and sub-measures of Art. 17 Investment in physical assets
• Measures and sub-measures of Art. 35 Co-operation
• Measures and sub-measures of Art. 28 Agri-environment-climate

Examples of measures/sub-measures programmed under other FAs but potentially showing secondary contributions to this FA:

• All above measures if programmed under other FAs than FA 5C and contributing to the supply and use of renewable sources of energy, of by-products, wastes, residues and
other non-food raw material for purposes of the bio-economy
• Measures and sub-measures of Art. 29 Organic farming
• Measures and sub-measures of Art. 21 Investments in forest area development and improvement of the viability of forests (sub-measures 8.1, 8.5)
• Measures and sub-measures of Art. 30 Natura 2000 and water framework directive
• Measures and sub-measures of Art. 33 Animal welfare
• Measures and sub-measures of Art. 35 of Regulation (EU) No 1303/2013, Support for Leader local development

71
WP: Guidelines for strategic programming for the period 2014-2020, European Commission, 04/03/2014, published on ECAS/circabc
72
All articles mentioned in this section of each fiche for answering CEQ 1 – 18 are of Regulation (EU) No 1305/2013, unless it is stated otherwise in the text directly

82
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Consistency check between CEQ, judgement criteria and indicators:

Judgement criteria

Judgement criteria (JC) proposed by the Working Document: Common Evaluation Questions for Rural Development Programmes 2014- 2020:

• GHG and ammonia emissions from agriculture have been reduced

Note: stakeholders in Member States might add/change judgement criteria in line with the intervention logic of the FA (selection and combination of measures)

Indicators

The following common indicators should be preliminary used to answer the CEQ:

Common result indicators

• R16/T17 - % of LU concerned by investments in livestock management in view of reducing GHG and/or ammonia emissions
• R17/T18 - % of agricultural land under management contracts targeting reduction of GHG and/or ammonia emissions
• R 18 - Reduced emissions of methane and nitrous oxide (complementary result indicator)
• R 19 - Reduced ammonia emissions (complementary result indicator)

The following other common indicators might be used to answer the CEQ:

Common output indicators 73 (data collected via the operations database):

• O8 Number of Livestock units (concerned by investment in livestock management in view of reducing GHG and ammonia emissions)

Common context indicators (data need and data sources are described in the Working document: Proposed list of common context indicators):

• CCI 18 Agriculture area

• CCI 21 Livestock units
• CCI 45 GHG emissions from agriculture

73
WD: Data item list for Pillar II operation database and WD RD programming and target setting

83
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Qualitative indicators

If needed qualitative information can be collected, e.g.:

• Typology of methods/approaches followed by farmers in order to reduce GHG and ammonia emissions, such as more efficient application of fertilisers, organic production methods,
better manure management, etc.
• Perception of beneficiaries on how RDP interventions have contributed to reduce GHG and ammonia emissions from agriculture

Data needs and sources

Indicators Data needed Data source

R16/T17 - % of LU concerned by Beneficiaries Beneficiaries

investments in livestock
• Livestock units concerned by investments in livestock management in • Application forms (before the project start)
management in view of reducing
Common indicators

view to reduce GHG and/or ammonia emissions

GHG and/or ammonia emissions • Payment request (after project finalisation)
Beneficiaries and non-beneficiaries
Beneficiaries and non-beneficiaries
Total number of livestock units
Eurostat - farm structure survey [Link]
explained/[Link]/Farm_structure_statistics

Survey on beneficiaries and non-beneficiaries

National and regional statistics

84
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

R17/T18 - % of agricultural land Beneficiaries Beneficiaries

under management contracts
• Agriculture land under management contracts targeting reduction of • Application forms (before the project start)
targeting reduction of GHG and/or
GHG and/or ammonia emissions
ammonia emissions • Payment request (after project finalisation)
Beneficiaries and non-beneficiaries
Beneficiaries and non-beneficiaries
Total agriculture area – UAA (arable land, permanent grassland and
Eurostat - farm structure survey [Link]
meadows, permanent crops and respective carbon capture capacity))
explained/[Link]/Farm_structure_statistics

Survey on beneficiaries and non-beneficiaries

National and regional statistics

85
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

R 18 - Reduced emissions of Beneficiaries: Beneficiaries

methane and nitrous oxide • Application forms (before the project start)
Farm relevant characteristics:
• Payment request (after project finalisation)
• Livestock housing
Beneficiaries and non-beneficiaries
• Slurry and farmyard manure storage practices
• Livestock diets Eurostat - Farm structure survey
• Excretion of C and N by livestock
[Link]
• Use of nitrogenous fertilisers and organic manure explained/[Link]/Farm_structure_statistics
• Management of crop residues
FADN data
Emission values/coefficients
The 2006 IPCC Guidelines for National Greenhouse Gas Inventories,
The evaluator has to be aware that these vary between livestock species Volume 4: Agriculture, Forestry and Other Land Use [Link]
(e.g. cattle, pigs) and within a livestock species as there are large
Common indicators

[Link]/public/2006gl/[Link]
differences between animals kept for different purposes (e.g. dairy cattle
On-site empirical measurements;
versus beef cattle). It is therefore necessary, whenever possible, to
disaggregate livestock according to species and production type Eurostat’s analysis of methodologies for calculating GHG emissions
([Link]
Non- Beneficiaries:
[Link])
Same as above
European Environment agency data on GHG emissions (MS level)

[Link]
reported-to-the-unfccc-and-to-the-eu-greenhouse-gas-monitoring-
mechanism-11#tab-metadata

For emissions that are not considered by IPCC (such as nitrification

inhibitors;) GAINS model can be used, [Link]

The joint EMEP/EEA air pollutant emission inventory guidebook for

reporting on emissions data under the UNECE Convention on Long-range
Transboundary Air Pollution (CLRTAP) and the EU National Emission
Ceilings Directive, [Link]
guidebook-2013,

86
Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

National and regional statistics of environmental agencies; Sectoral

experts and stakeholder organisations; Other country national and
international experts;

Reference libraries (national libraries); Scientific and technical articles in

environmental books, journals and reports;

Universities;

Web search for organisations and specialists;

Inventory reports from other parties.

Projects:

OSCAR – [Link]

GNOC (global nitrous calculator) –[Link]

"The Global Nitrous Oxide Calculator (GNOC) online tool has been
developed in the context of the “Assessment of GHG default emissions from
biofuels in EU legislation” (Edwards et al., 2013). The tool facilitates the
calculation soil N2O emissions from biofuel crop calculation for each location
globally. The online calculations are consistent with the method applied in
the assessment of GHG default emissions. The user is provided with default
environmental and management data which are required for the calculations
at the selected location. However, the user might change these parameters
if local data is available."

A useful source of information on GHG emission control options and

expected emission reduction efficiency is also available in a recent IIASA’s
publication commissioned by the EC
([Link]

87
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

R19 - Reduced ammonia emissions Beneficiaries: Beneficiaries

Farm relevant characteristics: • Application forms (before the project start)

• Payment request (after project finalisation)
• Livestock housing
• Slurry and farmyard manure storage practices Beneficiaries and non-beneficiaries:

• Slurry and farmyard manure application techniques Farm surveys

• Livestock diets
FADN data
• Excretion of N by livestock
• Use of nitrogenous fertilisers Farm surveys done by evaluator

Emission values/coefficients: European Environment agency data on ammonia emissions (MS level)

The evaluator has to be aware that these vary between livestock species [Link]

(e.g. cattle, pigs) and within a livestock species as there are large viewer-lrtap (air polutant data viewer) and

differences between animals kept for different purposes (e.g. dairy cattle EMEP/EEA emission inventory guidebook 2013, Chapter 3
versus beef cattle). It is therefore necessary, whenever possible, to ([Link]
disaggregate livestock according to species and production type.
On-farm empirical measurements;
Non- Beneficiaries:

Same as above

88
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Eurostat’s analysis of methodologies for calculating ammonia emissions

([Link]
[Link])

National Statistics Agencies;

Sectoral experts and stakeholder organisations; Other country national and

international experts;
Additional indicators

Reference libraries (national libraries); Scientific and technical articles in

environmental books, journals and reports;

Universities;

Web search for organisations and specialists;

Inventory reports from other parties.

A useful source of information on ammonia emission control options and

expected emission reduction efficiency is also available in a recent IIASA’s
publication commissioned by the EC
([Link]

89
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Methodology to calculate complementary result indicator R18 - Reduced emissions of methane and nitrous oxide

All EU Member States are signatory of the UN Framework Convention on Climate Change and the Kyoto Protocol. The 2006 IPCC Guidelines for National Greenhouse Gas Inventories,
Volume 4: Agriculture, Forestry and Other Land Use ([Link] ) support the reporting of greenhouse gas (GHG) emissions under this
Convention. The IPCC documents provide expert guidance (including detailed methodology) for calculating GHG emissions. All Member States use these Guidelines in preparing their
National Inventory Reports. Eurostat has also prepared an instructive publication on methodologies for calculating GHG emissions from agriculture
([Link] Information and guidelines presented in it are the principal source in setting-up methodological
complement for this CRI on reduced emissions of methane and nitrous oxide. The data needs for calculating of methane and nitrous oxide emissions are relatively large and concern
farming (livestock and crop production) as well as non-farming activities. It is unlikely that data required for assessment of methane and nitrous oxide emissions reduction will be
readily available in RDP operations database. FADN, Farm Structure Surveys and similar databases do not contain some vital information required for calculating methane and nitrous
oxide emissions either (e.g. data on livestock feeding and housing systems, manure storage systems, etc.). Ad-hoc surveys are generally required to provide additional information.
In all Member States agriculture is identified as a key source with regards to methane and nitrous oxide emissions. It is therefore suggested that methane and nitrous oxide
emissions reduction be assessed only in RDP-supported projects with the investments in the agriculture sector. Member States are obliged to use a Tier 2 or higher
methodology for inventory reporting. Tier 2 and higher methodologies require data that are both detailed and respect the relationships between emission sources. These data can only
be collected at farm scale. Methane and nitrous oxide emissions should be estimated by multiplying activity data with emission factors. The assessment reduction of methane and
nitrous oxide emissions therefore comprises two major activities: (i) obtaining activity data and (ii) choosing activity-specific emission factors.

Steps in conducting the analysis are:

Step 1: Identify sample of mixed farms (crop and livestock production) to serve as control groups. Since farm management practices influencing methane and nitrous oxide emissions
tends to vary with farm type (cattle, pig, etc.), size and the environmental conditions (e.g. climate, soil type, etc.), the control group should be stratified according to farm type, size and
environmental conditions.

Step 2: From the operations database identify a sample of farms that completed RDP-supported operations/projects flagged as contributing to the Focus Area 5D, either as the main
or secondary objective. The farm sample should be stratified in the same way as in Step 1.

Step 3: Define data you needed for assessing methane and nitrous oxide emissions and potential emissions reductions. As agricultural emissions strongly depend on the animal
housing & diet, and on the manure management system distribution, make sure that relevant data on these are collected. These should cover main methane and nitrous oxide
emissions sources and include information on: (i) livestock housing, including proportion of time spent by animals indoors and outside; (ii) slurry and farmyard manure storage practices,
loose or packed heaps of solid manure, any treatment applied to the manure such as aeration, separation or composting; (iii) livestock diets; (iv) excretion of C and N by livestock. Be
aware that these vary between livestock species (e.g. cattle, pigs) and within a livestock species as there are large differences between animals kept for different purposes (e.g. dairy

90
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

cattle versus beef cattle). It is therefore necessary, whenever possible, to disaggregate livestock according to species and production type; (v) use of nitrogenous fertilisers and organic
manure; (vi) management of crop residues (burned and returned to soil).

In order to be sure that you collect all essential data, consider collecting ammonia-related activity data suggested in Table S1 and Table 1 in the Annex of Eurostat’s publication on
methodologies for calculating GHG emissions.

([Link] Table S1 provides “optimum” and “minimum” data collection requirements. Activity data listed under
“minimum requirement” must be collected, because without these data, a proper methane and nitrous oxide emissions inventory is not possible. Using only these data, the effect of
mitigation measures cannot be shown in the inventory and the cost effectiveness of mitigation measures cannot be assessed. Activity data listed under “optimum requirement” should
be collected for estimating inventories more accurately. They offer more possibilities for accounting for farm-specific mitigation measures and enable the assessment of environmental
impacts of farm management practices. For most of these data, the additional effort for collecting them is small and the additional benefit is large.

Note: selection of data requirements for assessment of methane and nitrous oxide emissions should be co-ordinated/merged with those required for the assessment of
reduction of ammonia emissions as there is a huge overlap.

Step 4: Run surveys at control group (selected in Step 1) and at RDP-supported farms (selected in Step 2), collecting data defined in Step 3.

Note: these surveys should be combined with those required for the assessment of reduction of ammonia emissions.

Step 5: Adjust default (coefficient) values provided in the 2006 IPCC Guidelines to region or farm-specific (coefficient) values. Default 2006 IPCC Guidelines values are intended to
be reasonable estimates for the specified geographic area. They often disguise a wide geographic variation in actual values, either due to variations in climate or to regional variations
in agricultural practices. In addition, the default values generally relate to situations where no abatement measures have been implemented. Thus, region or farm-specific appropriate
values of the coefficients should be defined – preferably relying on information reported in scientific and technical articles in environmental books, (peer-reviewed) journals and reports.
In case literature does not provide reliable information on emissions and other values, it is advised to undertake empirical measurements of methane and nitrous oxide emissions. The
2006 IPCC Guidelines and Eurostat publication on methodologies for calculating GHG emissions mentioned in the introduction part provide information on how to generate new data
both by empirical measurements and by surveying activity data.

Step 6: Following standard methodology described in the 2006 IPCC Guidelines ([Link] using data collected though surveys (Step 4) and
applying region and/or farm-specific emission coefficients defined in Step 5, calculate methane and nitrous oxide emissions in a control group and at RDP-supported farms. Total
methane and nitrous oxide emissions should be the sum of the following emission sources:

1. Enteric fermentation (CH4)

2. Manure management (CH4 and N2O);

91
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

3. Rice cultivation (CH4);

4. Agricultural soil management (CH4 and N2O);
5. Field burning of agricultural residues (CH4 and N2O).
Step 7: Convert methane and nitrous oxide emissions CO2 equivalent (CO2-eq) applying following factors: 1 unit of CH4 = 25 units of CO2-eq; 1 unit of N2O=298 units of CO2-eq.
Total emissions of methane and nitrous oxide should be reported in tonnes of CO2-eq. Express the quantity of emissions per hectare and per LU both for control group and RDP-
supported farms.

Step 8: Assess if there is any reduction in methane and nitrous oxide emissions in RDP-supported farms as compared to the control group.

Secondary contributions

When calculating the indicator, both – primary and secondary contributions should be taken in consideration. Secondary contributions are effects of operations which are programmed
under other FAs than P5D. The secondary contributions are calculated based on surveys with those beneficiaries which are implementing operations via different focus areas (e.g.
FA 2A, 4A, 4B, 4C) and have flagged the additional contributions to FA 5D (see also step 2 – 8). Samples of surveyed beneficiaries can be set up in accordance with the expected/actual
contributions of implemented operations (e.g. farm investment measures) to the FA objective. Most significant contributions should be taken into consideration. The ratio of the
contribution is estimated based on the above surveys. If there is no sufficient up take to calculate the secondary contributions in a robust way a qualitative assessment is also possible
(see the Annex 10 of the guidelines: Assessment of RDP results: how to prepare for reporting on evaluation 2017). Findings can be used in answering the CEQ 19.

92
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Methodology to calculate complementary result indicator R 19 - Reduced ammonia emissions

All EU Member States are signatory of the UN Convention on Long- Range Transboundary Air Pollution (CLTRP). The joint EMEP/EEA Air Pollutant Emission Inventory Guidebook
([Link] supports the reporting of emissions data under the CLTRP and the EU National Emission Ceilings Directive 2001/81/EC.
It provides expert guidance (including detailed methodology) for calculating ammonia emissions. All Member States use this Guidebook for preparing their CLTRP National Emission
Reports. Eurostat has also prepared an instructive publication on methodologies for calculating ammonia emissions ([Link]
[Link]). Information and guides presented in it are the primary source in setting-up the methodological complement for this CRI Fiche on reduced ammonia emissions. As
agriculture is responsible for 94 % of total ammonia emissions across the EU there is no point of assessing ammonia emissions reduction in non-agriculture RDP-supported projects.
The data needs for calculating NH3 emissions are relatively large and concern both livestock and crop production. It is highly unlikely that all these data will be readily available in
RDP operations database. FADN, Farm Structure Surveys and similar databases do not contain vital information required for calculating NH3 emissions either (e.g. data on livestock
feeding and housing systems, manure storage systems, manure application techniques, etc.). Ad-hoc surveys are generally required to provide additional information. As in all Member
States agriculture is identified as a key source with regards to ammonia emissions, Member States are obliged to use a Tier 2 or higher methodology for inventory reporting. Tier 2
and higher methodologies require data that are both detailed and respect the relationships between emission sources. These data can only be collected at farm scale. Ammonia
emissions should be estimated by multiplying activity data with emission factors. The assessment reduction of ammonia emissions therefore comprises two major activities: (i) obtaining
activity data and (ii) choosing activity-specific emission factors).

Steps in conducting analysis are:

Step 1: Identify sample of mixed farms (crop and livestock production) to serve as a control group. Since farm management practices influencing ammonia emissions tend to vary with
farm type (cattle, pig, etc.), size and the environmental conditions (e.g. climate, soil type, etc.), the control group should be stratified according to farm type, size and environmental
conditions.

Step 2: From the operations database identify a sample of farms that completed RDP-supported operations/projects flagged as contributing to the Focus Area 5D, either as the main
or secondary objective. The farm sample should be stratified in the same way as in Step 1.

Step 3: Define data needed for assessing ammonia emissions and potential emissions reduction. As agricultural emissions strongly depend on the animal housing & diet, and on
manure management system distribution, make sure that relevant data on these are collected. These should cover main ammonia emissions sources and include information on: (i)
livestock housing, including proportion of time spent by animals indoors and outside; (ii) slurry and farmyard manure storage practices including information on open or covered slurry
tanks and lagoons, loose or packed heap of solid manure, any treatment applied to the manure such as aeration, separation or composting; (iii) slurry and farmyard manure application
techniques, including method and rate of application, the time between application and incorporation, meteorological conditions and method of incorporation; (iv) livestock diets; (v)
excretion of N by livestock; (vi) use of nitrogenous fertilisers. Be aware that these vary between livestock species (e.g. cattle, pigs) and within a livestock species as there are large

93
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

differences between animals kept for different purposes (e.g. dairy cattle versus beef cattle). It is therefore necessary, whenever possible, to disaggregate livestock according to
species and production type; (vi) use of nitrogenous fertilisers.

In order to be sure that you collect all essential data, consider collecting ammonia-related activity data suggested in Table S1 and Table 1 in the Annex of Eurostat’s publication on
methodologies for calculating ammonia emissions

([Link] Table S1 provides “optimum” and “minimum” data collection requirements. Activity data listed under
“minimum requirement” must be collected, because without these data, a proper ammonia emissions inventory is not possible. Using only these data, the effect of mitigation measures
cannot be shown in the inventory and the cost effectiveness of mitigation measures cannot be assessed. Activity data listed under “optimum requirement” should be collected for more
accurately estimating inventories. They offer more possibilities for accounting for farm-specific mitigation measures and enable the assessment of environmental impacts of farm
management practices. For most of these data, the additional effort for collecting them is small and the additional benefit is large.

Note: selection of data requirements for assessment of ammonia emissions should be co-ordinated/merged with those required for the assessment of reduction of
methane and nitrous oxide emissions as there is a huge overlap.

Step 4: Run surveys at control group (selected in Step 1) and at RDP-supported farms (selected in Step 2), collecting data defined in Step 3.

Note: these surveys should be combined with those required for the assessment of reduction of methane and nitrous oxide emissions.

Step 5: Adjust default (coefficient) values provided in the EMEP/EEA Guidebook to region or farm-specific (coefficient) values. Default EMEP/EEA values are intended to be reasonable
estimates for the specified geographic area. They often disguise a wide geographic variation in actual values, either due to variations in climate or to regional variations in agricultural
practices. In addition, the default values generally relate to situations where no abatement measures have been implemented. Thus, region or farm-specific appropriate values of the
coefficients should be defined – preferably relying on information reported in scientific and technical articles in environmental books, (peer-reviewed) journals and reports. In case
literature does not provide reliable information on emissions and other values, it is advised to undertake empirical measurements of ammonia emissions. The EMEP/EEA Guidebook
(Chapter 3) and Eurostat publication on methodologies for calculating ammonia emissions mentioned in the introduction part provide information on how to generate new data both
by empirical measurements and by surveying activity data.

Step 6: Following standard methodology described in EMEP/EEA Guidebook ([Link] using data collected though surveys (Step
4) and applying region and/or farm-specific emission coefficients defined in Step 5, calculate ammonia emissions in control group and at RDP-supported farms. Total ammonia
emissions should be the sum of NFR subsectors 4B1-9 [excl. 4B5] + 4B13 + 4D1a + 4D2a, b, c + 4F + 4G:

1. Cattle dairy (4B1b)

2. Cattle non-dairy (4B1b)

94
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

3. Swine (4B8)
4. Laying hens (4B9a)
5. Broilers (4B9b)
6. Synthetic N-fertilizer (4D1a)
7. All other subsectors (4B2-7 [except 4B%] + 4B9c, d + 4B13 + 4D2a, b, c +4F + 4G)
Step 7: Total ammonia emissions should be reported in tonnes of ammonia (NH3). Express the quantity of emissions per hectare and per LU both for control group and RDP-supported
farms.

Step 8: Assess if there is any reduction in ammonia emissions in RDP-supported farms as compared to control group.

Secondary contributions

When calculating the indicator, both – primary and secondary contributions should be taken in consideration. Secondary contributions are effects of operations which are programmed
under other FA than P5D. The secondary contributions are calculated based on survey with those beneficiaries which are implementing operations via different focus areas (e.g. FA
2A, 4A, 4B, 4C) and have flagged the additional contribution to FA 5D (see also step 2 – 8). Samples of surveyed beneficiaries can be set up in accordance with the expected/actual
contributions of implemented operations (e.g. farm investment measures) to the FA objectives. Most significant contributions should be taken in consideration. The ration of the
contribution is estimated based on above surveys. If there is not sufficient up take to calculate the secondary contributions in a robust way the qualitative assessment is also possible
(see the Annex 10 of the guidelines: Assessment of RDP results: how to prepare for reporting on evaluation 2017). Findings can be used in answering the CEQ 19.

95
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Principal challenges regarding CRI

CRI 18 Reduced emissions of methane and nitrous oxide (FA 5D - Complementary result indicator R18)

• A fair assessment of methane and nitrous oxide reduction in operations/measures that do not contribute primarily or exclusively to Focus Area 5D (for instance organic farming)
will be quite difficult.

• The Fiche for R18 does not require identification of the most cost-effective methane and nitrous oxide abatement measures. However, evaluators are encouraged to assess the
cost-effectiveness of the abatement measures applied. This should be fairly easy (at least for operations contributing primarily to Focus Area 5D) as both the data on reduction of
methane and nitrous oxide emissions and cost of RDP-supported operations aiming at reduction of methane and nitrous oxide will be available.

• Injection of slurries reduces ammonia emission. However according to the above-mentioned IIASA publication, there is also some evidence that nitrous oxide emissions are higher
with injection of slurries compared to surface (and band) spreading of slurries.

How to use the indicator in the situation of lack of data (low RDP uptake, small programmes, etc.)

The approach adopted assumes limited data availability and requires that data should be collected though surveys. Where uptake is low care must be taken to ensure a sample size
to deliver the requisite confidence in estimates.

CRI 19 Reduced ammonia emissions (FA 5D - Complementary result indicator R19)

• A fair assessment of reduction of ammonia in operations/measures that do not contribute primarily or exclusively to Focus Area 5D (for instance organic farming) will be quite
difficult.

• The Fiche for R19 does not require identification of the most cost-effective ammonia abatement measures. However, evaluators are encouraged to assess the cost-effectiveness
of the abatement measures applied. This should be fairly easy (at least for operations contributing primarily to Focus Area 5D) as both the data on reduction of ammonia emissions
and cost of RDP-supported operations aiming at reduction of ammonia will be available.

• An “ammonia gap” between the expected reduction and monitoring results has been identified in several MS. Although most of time this is attributed to ineffectiveness of the
abatement measures, this gap can appear also because of incorrect measurements or interpretation of monitoring data.

• When calculating its proposed emission reduction commitments, the EC made a distinction between small or medium farms and large industrial cattle feedlots – but farms with
less than 15 LU were excluded from the ammonia emission reductions calculations in National Emission Ceilings Directive 2001/81/EC (see: Impact Assessment accompanying
the NEC revision, December 2013., page 278, quoted by the European Environmental Bureau at: [Link]

96
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

How to use the indicator in the situation of a lack of data (low RDP uptake, small programmes etc.)

The approach adopted assumes limited data availability and requires that data should be collected though surveys. Where uptake is low care must be taken to ensure a sample size
to deliver the requisite confidence in estimates.

97
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Example for CRI 18

EU-27

[Link]

About one-third of RDPs 2007-2013 have measures specifically tailored towards climate-related actions. Approaches vary, reflecting the fact that all three thematic axes of the Rural
Development Regulation provide possibilities to help in curbing methane and nitrous oxide emissions, reducing CO2 emissions from energy use, promoting climate-friendly production
and use of renewable energy. In most RDPs, emission reduction activities were before predominantly or exclusively supported by two measures, namely: modernisation of agricultural
holdings and agri-environment.

Northern Ireland

[Link]

Low Carbon Beef Project

A project was initiated to identify means of reducing the carbon intensity of local beef. As part of the programme, an online greenhouse gas benchmarking application has been
developed. This greenhouse gas calculator has the potential to provide an accurate assessment of carbon emissions from a range of beef production systems, supplying solid evidence
on the effects of GHG mitigation measures on beef systems. Fundamental to this project was the desire to promote the benefits of lowering the age at which suckler cows are first in
calf to 24 months. Adoption of this measure on-farm increased efficiency and benefit both the farmer and the environment. The average age at which heifers had their first calf was
reduced from 988 to 939 days. Features of herds where 24 month calving has been implemented include greater productivity and proportionally less need for housing. Based on a
100 cow suckler herd, the project found that an 8-10% saving in GHGs was achieved by utilising these measures.

Animal Health and Welfare project

BVD is a contagious disease of the bovine which reduces the productivity of affected cattle, as well as compromising their welfare. In 2011 66% of Northern Ireland dairy and suckler
heeds tested showed some degree of exposure to the virus. The carbon savings which would be associated with the eradication of BVD in Northern Ireland is estimated at equivalent
to 91,000 tonnes of carbon. Based on the analysis of the dairy sector, it is estimated that a 3% improvement in replacement rate in the beef industry will lead to a 1.5% reduction in
GHG emissions. This amounts to an estimated 43,500 tonnes of carbon savings from the beef sector.

UK (Scotland)

[Link]

98
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Farming for a Better Climate is a programme of work funded by the Scottish Government aimed at reducing carbon emissions from the agriculture sector in Scotland. The programme
combines ideas trialled by volunteer Climate Change Focus Farms and information from state-of-the-art scientific research. Agricultural consultants offer practical advice to help
choose the most relevant measures to improve both farm performance and resilience to future climate change effects. The Scottish Government has estimated that the programme
will deliver annual carbon savings of 103 kT CO2-eq by 2020. An evaluation of the first phase of the programme has demonstrated savings in GHG emissions by the monitor farms of
around 10-12% per year despite challenging weather conditions that were prevalent at this time. The programme explores a range of practical options to help reduce GHG emissions
in ways that are consistent with improving efficiency and maximising profitability. Effective management of nutrients (and particularly nitrogen) is found to be critically important in
delivering lower GHG emissions form farming. With Climate Change and Pollution Prevention Advisory Activity funding from Scottish Government, the programme has established a
voluntary network of farmers to help promote more efficient nutrient use (improved planning testing and application of nutrients) to help deliver targets for GHG reduction.

One innovative investigation into the effect of disease on GHG emission was carried out with small groups of sheep carrying a worm burden.

[Link]

Results so far seem to indicate that while individual measurements show no significant difference between the methane output per kg of feed between wormy and healthy sheep,
disease does contribute to GHG emissions as animals take longer achieve the same productive output. For example, it takes them a week more to produce a 25Kg lamb. There is an
additional environmental impact however, as wormy ewes produced about 20% more faeces than healthy ewes, an effect not previously considered.

The pre-2014 RDP provided financial support for some actions that lead to GHG co-benefits; specifically covering slurry stores and anaerobic digestion of animal waste. However, the
quantity of GHG savings associated with implementing pre-2014 RDP measures remains uncertain (as per May 2013). It was found that with the exception of reduced fertiliser
applications and livestock numbers, the GHG savings would not be captured in the national GHG inventory.

[Link]

Example for CRI 19

Northern Ireland

The Manure Efficiency Technology Scheme (METS) was a capital grant scheme under the Northern Ireland Rural Development Programme 2007-2013. The scheme aimed to
encourage uptake of advanced slurry spreading systems which was supposed to deliver a range of environmental and productive benefits on farm, including reduced ammonia
emissions from slurry spreading and reduced chemical fertiliser usage. There are still no data available on the effectiveness of this measure on reduction of ammonia emissions.
However, it was expected that farmers participating in this scheme could cut ammonia emissions by 30–80%, while increasing grass yields by 24% and saving approximately 44 kg/ha
fertiliser N, worth 10.5 EUR/ha [Link]

99
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

Ireland

Ireland’s Rural Development Programme 2014-2020

[Link]

Example advocates an integrated approach in reducing ammonia emissions from agriculture (mainly through a new integrated agri-environment scheme “GLAS “, that is expected to
recruit 50,000 participants). Following actions are expected to help in reducing ammonia emissions:

• A greater uptake of low emission slurry application technologies (e.g. trailing shoe);
• Improvement of fertiliser/manure efficiency, including use of new fertiliser additives/inhibitors and the increased use of clover offering possibility to significantly reduce nitrogen
fertiliser usage;
• Introduction of farm nitrogen budgets to improve the efficiency of its use and reduce nitrogen losses;
• Introduction of new quantified targets, such as LU affected by ammonia reduction supports;
• Cattle spending more time grazing outdoors;
• Improved breeding/genetic improvements in livestock, feeding and other management practices;
• Knowledge Transfer Groups stimulating farmers’ enrolment in the Carbon Navigator – a tool allowing farmers to understand how their farms produce greenhouse gas – and to
identify mitigation capacity and set targets and a pathway to reduce emissions.

Finland

[Link]

Finnish measures for reducing ammonia emissions from agriculture include:

• Well-balanced use of nutrients, including specifications on manure utilisation injecting slurry into the soil;
• Investment support for covering solid manure, slurry and liquid manure storage facilities and for cooling manure channels;
• Measures in which the use of (inorganic) nitrogen is restricted or eliminated – such as organic farming;
• Investments in more effective handing, storage and application of manure, incl. building of remote storages and purchasing of manure processing systems;
• Compulsory covering of new manure storage facilities;
• Stricter requirements for large manure storage facilities;
• Stricter rules for storing manure on heaps, manure spreading periods and incorporation of manure when applied on field;
• Financing studies on improving knowledge on ammonia behaviour and reduction measures;

100
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes 2014-2020 (CEQ 1 – 21)

• Survey on manure management practices revealing data on manure management life cycle (production, storage, transportation and application) resulting in setting-up database
on manure management practices. Data is used to feed an ammonia emission modelling and studies on emission reduction potentials and cost effectiveness of the reduction
measures applied.

Answer to CEQ

The answer is provided based on the calculated values of indicators, on the collected qualitative information or on the qualitative assessment.

Conclusions Recommendations

Main conclusions are stated here for the FA based on the evaluation findings. Main recommendations linked to the conclusions are stated here for the FA.

101
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes
2014-2020 (CEQ 1 – 21)

2.15 Focus Area P5E, Evaluation question 15

To what extent have RDP interventions supported carbon conservation and sequestration in agriculture and forestry?

List of measures contributing to the FA 5E:

Primarily programmed measures/sub-measures (proposed by EC) 74:

• Measures and sub-measures of Art. 75 15 Advisory services, farm management and farm relief services
• Measures and sub-measures of Art. 14 Knowledge transfer and information actions
• Measures and sub-measures of Art. 35 Co-operation
• Measures and sub-measures of Art. 17, Investment in physical assets
• Measures and sub-measures of Art. 34 Forest environmental and climate services and forest conservation
• Measures and sub-measures of Art. 21 Investment in forest area development and improvement of the viability of
forests
• Measures and sub-measures of Art. 26 Investment in forestry technologies and in processing, mobilising and
marketing of forest products

Examples of measures/sub-measures programmed under the other FAs but potentially showing secondary
contributions to this FA:

• All above measures if programmed under other FAs than FA 5E and contributing to carbon conservation and
sequestration in agriculture and forestry
• Measures and sub-measures of Art. 19 Farm and business development
• Measures and sub-measures of Art. 20 Basic services and village renewal in rural areas
• Measures and sub-measures of Art. 29 Organic farming
• Measures and sub-measures of Art. 35 of Regulation (EU) No 1303/2013, Support for Leader local development

Consistency check between CEQ, judgement criteria and indicators

Judgement criteria

Judgement criteria (JC) proposed by the Working Document: Common Evaluation Questions for Rural Development
Programmes 2014- 2020:

• Carbon conservation and sequestration in agriculture and forestry has increased

• Agricultural and forestry land under enhanced management contract contributing to carbon sequestration has been
enlarged

Note: stakeholders in Member States might add/change judgement criteria in line with the intervention logic of the FA
(selection and combination of measures)

Indicators

The following common indicators should primarily be used to answer the CEQ:

Common result/target indicator

• R20/T19 - % of agricultural and forest land under management contracts contributing to carbon sequestration and
conservation

74
WP: Guidelines for strategic programming for the period 2014-2020, European Commission, 04/03/2014, published on
ECAS/circabc
75
All articles mentioned in this section of each fiche for answering CEQ 1 – 18 are of Regulation (EU) No 1305/2013, unless it
is stated otherwise in the text directly
102
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

The following other common indicators might be used to answer the CEQ:

Common output indicator 76 (data collected via the operations database):

• O5 Total area (afforested, with agro-forestry systems, under agri-environment climate)

• O6 Physical area supported (Ha)

Common context indicators (Data needs and data sources are described in the Working document: Proposed list of
common context indicators):

• CCI 33 Farming intensity

• CCI 41 Soil organic matter in arable land

Additional indicators and information

Common indicators are not sufficient to answer the CEQ. Working Document: Common Evaluation Questions for Rural
Development Programmes 2014-2020 suggests:

• Additional information on carbon conservation and sequestration of the land under management contracts

Further proposed additional indicator:

• CCI 41 broken down as follows: Soil organic matter in arable land, expressed as a) concentration of carbon in the
soil, b) total organic carbon (TIC), c) organic carbon concentration in biomass

Note: More additional indicators can be developed when judgement criteria are added to specify the evaluation question
in MS.

Qualitative indicators

Qualitative indicators might be also used, e.g.:

• types of actions that contribute to carbon conservation and sequestration of the land under management contracts
(e.g. afforestation, maintenance of forests, carbon sequestered by permanent crops and/or grassland, etc.)

Data needs and data sources

Indicators Data needed Data sources

R20/T19 - % of agricultural Agricultural and forest land under Beneficiaries

and forest land under management contacts to foster carbon
Application form (project start)
management contracts conservation/sequestration supported under
contributing to carbon the RDP (data item O.5 and O.6) Payment request (project end)
Common indicators

sequestration and Beneficiaries and non-beneficiaries

Total agricultural land in base year (data item
conservation
O.5) IACS
Total forestry land in base year (data item
Eurostat:
O.5) [Link]
explained/[Link]/Farm_structure_statistics
#Main_tables

National and regional statistics

76
WD: Data item list for Pillar II operation database and WD RD programming and target setting

103
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Soil organic matter in arable The length of the application of each measure Beneficiaries
land due to RDP support, 8.1, 8.2 and 10
Application form (project start) and Payment
expressed as a) concentration
Type of land management (organic / request (project end) – for the length of
of carbon in the soil, b) total
conventional) application of each measure
organic carbon (TIC), c)
organic carbon concentration Type of soil (categories of soil type) Beneficiaries and non-beneficiaries
in biomass
Additional indicators

Hydrology of land parcels (waterlogged / non- National/regional databases

waterlogged)
Carbon accounts

Types of actions that Description of actions to increase carbon Beneficiaries

contribute to carbon conservation and sequestration
Application form (project start) and Payment
conservation and
Perception of beneficiaries on the request (project end) – for the description of
sequestration of the land
effectiveness of actions to increase carbon the actions
under management contracts
conservation/sequestration
(e.g. afforestation, Surveys/focus groups with relevant
maintenance of forests, stakeholders

carbon sequestered by Interviews with managers of interventions

permanent crops and/or
grassland, etc.)

Timing of data collection

Proposed data on beneficiaries and non-beneficiaries should be collected prior to the RDP’s implementation (i.e. years
2010-2013) and during the RDP’s implementation (i.e. 2014-2020: 1st call for proposal, operations database,
national/regional statistics, EU sources)

Methodology to calculate common and additional indicators

Common indicator

The common indicator - R20/T19, is collected via the operations database and calculated as ratio of agriculture land or
forestry land under management contracts contributing to carbon sequestration and conservation of the total agricultural
and forestry land. For those operations programmed under other FAs and contributing to the FA 5E the secondary
contributions to values of above indicators are already captured in the monitoring system.

Additional indicators

For investment related operations the additional indicators will have to be identified, e.g. when assessing the secondary
contributions to the FA.

For additional indicators, it is recommended that one conduct the analysis of treated and non-treated parcels by
constructing ex ante parameters and using analysis techniques for the concentration of carbon in the soil (NCS Flash
2000), total organic carbon (analyser CS 500) and organic carbon concentration in biomass (GHG inventory and
methodology IPCC). The ex-post 2007-2013 evaluation guidelines offer a detailed description of this method applied to
assess the effects of axis 2 measures on carbon sequestration (see chapter 8.8 of the ‘Guidelines for ex post evaluations
2007-2013’).

Secondary contributions

Secondary contributions to the values of additional result indicators are calculated based on surveys with those
beneficiaries which are implementing operations via different focus areas (e.g. FA 2A, 4A, B and C, 5A, ) and have flagged
the additional contributions to FA 5E. Samples of surveyed beneficiaries can be set up in accordance with the
expected/actual contributions of implemented operations (e.g. farm investment measures, environment and climate,

104
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

forestry related measures etc.) to the FA’s objectives. Most significant contributions should be taken in consideration. If
there is no sufficient up take to calculate the secondary contributions in a robust way a qualitative assessment is also
possible (see the Annex 10 of the guidelines: Assessment of RDP results: how to prepare for reporting on evaluation
2017). Findings can be used in answering the CEQ 19.

Qualitative assessment

For qualitative analysis of net effects, the proposal is to use:

• MAPP method focus groups which is relevant for assessing the effects of interventions on environmental indicators
that are difficult to measure (such as carbon conservation/sequestration in this case).
• Surveys 77 to beneficiaries and non-beneficiaries of implemented measures
• The combination of the above mentioned methods will provide the necessary triangulation results to validate the
findings

Challenges/Risks/Issues

• Low quality data that does not provide a robust answer to the EQ

• Lack of data to answer the EQ (e.g. low RDP uptake/no uptake, small programmes, etc.)
• Systematised and harmonised data on carbon conservation/sequestration in Europe are lacking (according to the
European Environment Agency)
• Carbon accounts do not exist in every country

Proposed solutions

In cases where there is a lack of data (low uptake, small programmes), this EQ can focus on the estimated land to be
covered by management contracts contributing to carbon conservation/sequestration (via interviews and expert opinions).

Modelling approaches can be used to calculate organic carbon content in circumstances with limited observational soil
data. However, the quality of the model input data determines the level at which the results can be used and the type of
comparison groups that can be designed for the assessment.

In case of low quality data, qualitative information can be sought through interviews, surveys, focus groups.

Given the lack of harmonised data and carbon accounts, assess the contribution of RDP interventions with the use of
expert panels and expert opinions. Also EEA studies may include information in this respect.

Answer to CEQ

The answer is provided based on the calculated values of indicators, on the collected qualitative information or on the
qualitative assessment.

Conclusions Recommendations

Main conclusions are stated here for the FA based on the Main recommendations linked to the conclusions are
evaluation findings. stated here for the FA.

77
For details on how to use surveys, see ‘Capturing the success of your RDP: Guidelines for ex post evaluation of 2007-2013
RDPs’, Evaluation Helpdesk, June 2014
105
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)
2.16 Focus area P6A, evaluation question 16

To what extent have RDP interventions supported the diversification, creation and development of small enterprises and
job creation?

List of measures contributing to the FA 6A:

Programmed measures/sub-measures (proposed by EC) 78:

• Measures and sub-measures of Art. 79 15 Advisory services, farm management and farm relief services
• Measures and sub-measures of Art. 14 Knowledge transfer and information actions
• Measures and sub-measures of Art. 35 Co-operation
• Measures and sub-measures of Art. 19 Farm and business development
• Measures and sub-measures of Art. 17 Investments in physical assets

Examples of measures/sub-measures programmed under other FAs but potentially showing secondary contributions
to this FA:

• All above measures if programmed under other FAs than FA 6A and contributing to carbon conservation and
sequestration in agriculture and forestry
• Measures and sub-measures of Art. 16 Quality schemes for agricultural products and foodstuffs
• Measures and sub-measures of Art. 20 Basic services and village renewal in rural areas
• Measures and sub-measures of Art. 21 Investments in forest area development and improvement of the viability of
forests (sub-measure 8.6)
• Measures and sub-measures of Art. 27 Setting up of producer groups and organisations
• Measures and sub-measures of Art. 29 Organic farming
• Measures and sub-measures of Art. 35 of Regulation (EU) No 1303/2013, Support for Leader local development

Consistency check between CEQ, judgement criteria and indicators

Judgement criteria

Judgement criteria (JC) proposed by the Working Document: Common Evaluation Questions for Rural Development
Programmes 2014- 2020:

• Small enterprises have been created

• Small enterprises have diversified their economic activity
• Jobs have been created

Note: stakeholders in MS might add/change judgement criteria in line with the intervention logic of the FA (selection
and combination of measures)

Indicators

The following common indicators should be used to answer the CEQ:

Common result/target indicator

• R21/T20 - Jobs created in supported projects

Common output indicators 80 (data collected via the operations database):

78
WP: Guidelines for strategic programming for the period 2014-2020, European Commission, 04/03/2014, published on
ECAS/circabc
79
All articles mentioned in this section of each fiche for answering CEQ 1 – 18 are of Regulation (EU) No 1305/2013, unless it
is stated otherwise in the text directly
80
WD: Data item list for Pillar II operation database and WD RD programming and target setting
106
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)
• O4 Number of beneficiaries (of support for investments in processing and marketing)
• O4 Number of beneficiaries/holdings (receiving start-up aid/support for investment in non-agricultural activities in rural
areas)
• Number of operations for investment in in forestry technology and primary processing/marketing

Common context indicators (data need and data sources are described in the Working document: Proposed list of
common context indicators):

• CCI 5 Employment rate

• CCI 6 Self-employment rate
• CCI 7 Unemployment rate
• CCI 10 Structure of the economy
• CCI 11 Structure of the employment

Additional indicators and information

Common indicators are not sufficient to answer the CEQ. WD Common Evaluation Questions for Rural Development
Programmes 2014-2020 suggests:

• % of small enterprises in the non-agriculture sector created with the RDP support
• % of new small enterprises created with the RDP support

Further proposed additional indicators:

• Created jobs divided by sector/branch (e.g. economic sectors agriculture, forestry, food processing, other processing,
services and tourism).
• Created small businesses divided by sector/branch

Further breakdown of jobs/enterprises, e.g. by age, gender, education/skills level, is possible as well, etc.

Note: More additional indicators can be developed when judgement criteria are added to specify the evaluation question
in the Member States.

Qualitative indicators

Is important to add qualitative information in the assessment, e.g.:

• description of the environment for business start-ups,

• diversification opportunities, etc.

Data needs and data sources

Indicators Data needed Data source

R21/T20 Jobs created in Total number of permanent jobs created Beneficiaries

indicators
Common

supported projects (not maintained) through supported

Application form (project start)
projects under measures 4, 6 and 8.6,
expressed in FTE, by gender Payment request (project end)

Beneficiaries and non-beneficiaries

% of small enterprises in the non- Created new businesses in non-agriculture
Additional
indicators

agriculture sector created with the sector by branch/sector: National/regional statistics

RDP support • By beneficiaries
• In programme area

107
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

% of new small enterprises Created new businesses by branch/sector: Eurostat employment statistics:
created with the RDP support • By beneficiaries [Link]
database
• In programme area

Created jobs divided by Created jobs by sector/branch:

sector/branch (gross/net) • By beneficiaries
• In the programme area

Created small businesses divided Created small businesses:

by sector/branch • By beneficiaries
• In the programme area

Timing of data collection

All data should be collected for beneficiaries and non-beneficiaries from 1.1.2014 or from the programme start – baselines.

Methodology to calculate common and additional indicators

Common indicators

The common indicator - R21/T20, is calculated as the sum of jobs created via supported projects based on monitoring
data from the operations database. Thus, it corresponds to the gross number of jobs created from the RDP support.

This indicator can be however used also as indicator to calculate net effects on job creation. In this case it is proposed to
apply a quantitative approach: Quasi experimental design using PSM/DiD can be applied in the calculation of net effects
for jobs created. If sufficient data is not available, the Input-Output method can also be used.

Secondary contributions

Secondary contributions of operations which are programmed under other FAs than P6A might be also considered when
calculating the indicator. This also includes those operations implemented via CLLD strategies. Secondary contributions
to the value of result indicators are calculated based on surveys with those beneficiaries which are implementing
operations via different focus areas (e.g. 2A, and B, 3A, 6B) and have flagged the additional contributions to FA 6A.
Samples of surveyed beneficiaries can be set up in accordance with the expected/actual contributions of implemented
operations (e.g. farm investment measures, young farmers, farm diversification measures, forestry technology and
processing measures etc.) to the FA’s objectives. If there is not sufficient uptake to calculate the secondary contributions
in a robust way a qualitative assessment is also possible (see the Annex 10 of the guidelines: Assessment of RDP results:
how to prepare for reporting on evaluation 2017). Findings can be used in answering the CEQ 19.

Additional indicators

Additional indicators can be collected via the operations database and calculated as the ratio of non-agriculture small
businesses in the agriculture sector to the total number of small businesses. Operations database can also collect data
on created jobs and created small businesses broken down by sector and branch.

Secondary contributions of operations implemented under FAs other than FA 6A should also be included in the calculation
of additional indicators. The procedure is similar to the one described for common indicators.

108
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Qualitative assessment

The following methods can be used in the qualitative assessment 81:

• Surveys of beneficiaries on measures 4, 6 and 8.6
• Structured focus groups 82,
• Interviews with managers of these measures.

In relation to focus groups, the MAPP 83 method is relevant for assessing the net effects of interventions on job creation,
also using control groups to assess the counterfactual.

All of the above methods will provide sufficient triangulation findings for validating and analysing the results.

Similar approaches can be used in the case of additional indicators.

Generally, decisions in relation to proportionality and cost-effectiveness need to be considered in the choice of evaluation
methods, in particular in cases of relatively small measures.

Challenges/Risks/Issues

• Low quality data that does not provide a robust answer to the EQ

• Lack of data to answer the EQ (e.g. low RDP uptake/no uptake, small programmes etc.)
• Estimate the effects on the level of indirect beneficiaries

Proposed solutions

In case of a lack of data (low uptake,), this EQ can focus on the expected jobs/small enterprises to be created (via
interviews and expert opinions).

In case of low quality data, qualitative information can be sought through interviews, surveys, focus groups. The MAPP
method is also useful when there is significant uptake but limited data.

Estimation of effects on indirect beneficiaries is part of netting out of the indicators´ values in the assessment of
programme externalities.

Further information

Working Paper on Capturing impacts of Leader and of measures to improve Quality of Life in rural areas, July 2010
([Link]

Answer to CEQ

The answer is provided based on the calculated values of indicators, on the collected qualitative information or on the
qualitative assessment.

Conclusions Recommendations

Main conclusions are stated here for the FA based on the Main recommendations linked to the conclusions are
evaluation findings. stated here for the FA.

81
Guidelines for the ex post evaluation of 2007-2013 RDPs, Chapter 4, [Link]
static/evaluation/library/evaluation-helpdesk-publications/en/evaluation-helpdesk-publications_en.html
82
For details on how to use surveys and focus groups, see ‘Capturing the success of your RDP: Guidelines for ex post
evaluation of 2007-2013 RDPs’, Evaluation Helpdesk, June 2014
83
Study: Investment support under Rural Development policy, DG Agri, 2014: [Link]
development-reports/2014/investment-support-rdp/fulltext_en.pdf
109
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development programmes
2014-2020 (CEQ 1 – 21)

2.17 Focus Area P6B, Evaluation question 17

To what extent have RDP interventions supported local development in rural areas?

List of measures contributing to the FA 6B:

Programmed measures/sub-measures (proposed by EC) 84:

• Measures and sub-measures of Art. 85 15 Advisory services, farm management and farm relief services
• Measures and sub-measures of Art. 14 Knowledge transfer and information actions
• Measures and sub-measures of Art. 35 Co-operation
• Measures and sub-measures of Art. 20 Basic services and village renewal in rural areas
• Measures and sub-measures of Art. 42 Local action group

Examples of measures/sub-measures programmed under other FAs but potentially showing secondary contributions
to this FA:

• All other measures and sub-measures programed under other focus areas, implemented via CLLD strategies.

Consistency check between CEQ, judgement criteria and indicators

Judgement criteria

Judgement criteria (JC) proposed by the Working Document: Common Evaluation Questions for Rural Development
Programmes 2014- 2020:

• Services and local infrastructure in rural areas has improved

• Access to services and local infrastructure has increased in rural areas
• Rural people have participated in local actions
• Rural people have benefited from local actions
• Employment opportunities have been created via local development strategies
• Rural territory and population covered by LAGs has increased

Note: stakeholders in Member States might add/change judgement criteria in line with the intervention logic of the FA
(selection and combination of measures)

Indicators

Common indicators

Common result/target indicators:

• R22/T21 - % of rural population covered by local development strategies

• R24/T23 - Jobs created in supported projects (Leader)
• R23/T22 - % of rural population benefiting from improved services/ infrastructures

The following other common indicators might be used to answer the CEQ:

Common output indicators 86 (data collected via the operations database):

84
WP: Guidelines for strategic programming for the period 2014-2020, European Commission, 04/03/2014, published on
ECAS/circabc
85
All articles mentioned in this section of each fiche for answering CEQ 1 – 18 are of Regulation (EU) No 1305/2013, unless it
is stated otherwise in the text directly
86
WD: Data item list for Pillar II operation database and WD RD programming and target setting
110
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)
• All Leader related output indicators
• O3 Number of actions/operation supported
• O4 Number of beneficiaries/holdings
• O15 Population benefiting from improved services

Common context indicators (data need and data sources are described in the Working document: Proposed list of
common context indicators):

• CCI 1 Population
• CCI 2 Age structure
• CCI 3 Territory
• CCI 8 GDP per capita
• CCI 10 Structure of the economy
• CCI 12 Labour productivity by sector

Additional indicators and information

Common indicators are not sufficient to answer the CEQ. Working Document: Common Evaluation Questions for Rural
Development Programmes 2014-2020 suggests:

• Number of projects/initiatives supported by the Local Development Strategy divided by type

• % of expenditure in Leader measures with respect to total RDP expenditures

Further proposed additional indicators:

• Number and % of innovative projects supported by LAGs of all innovative RDP projects
• Number of partnerships created (Leader)

Qualitative information

Is important to add qualitative information to answer this CEQ, e.g. to assess the quality of local services and
infrastructure, the integration of all stakeholders in the LDS (roles and contribution

Note: Additional indicators can be developed when judgement criteria are added to specify the evaluation question in
the Member State.

Data needs and data sources

Indicators Data needed Data source

R22/T21 - % Total rural population covered by LAGs funded Beneficiaries

of rural under LEADER (data item O.18)
Application form (project start)
population
Total rural population covered by the RDP area
Common indicators

covered by Payment request (project end)

local
LAG operations databases
development
Beneficiaries and non-beneficiaries
strategies
National and regional statistic

Eurostat - rural population statistics:

[Link]
development/data/database

111
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

R24/T23 - Total number of permanent jobs created (not Beneficiaries

Jobs created maintained) through supported projects via
Application form (project start)
in supported Leader local development strategies
projects Payment request (project end)
(Leader) LAG operations databases

Surveys to LAGs and end beneficiaries

Common indicators

R23/T22 - % Population benefiting from improved Beneficiaries

of rural services/infrastructures supported under
Application form (project start)
population measure 7 (data item O.15/T.22)
benefiting from Payment request (project end)
Total rural population covered by the RDP area
improved
LAG operations databases
services/
Surveys to LAGs and end beneficiaries
infrastructures
Beneficiaries and non-beneficiaries

National and regional statistic systems

Eurostat rural population statistics:

[Link]
development/data/database

Number of Number and types of local Beneficiaries

projects/initiati actions/initiatives/projects implemented under
Application form (project start)
ves supported the LDS
by the Local Payment request (project end)
Development
LAG operations databases
Strategy
Surveys to LAGs and end beneficiaries
divided by
type Beneficiaries and non-beneficiaries

National and regional statistic

Additional indicators

% of Total RDP expenditures and total expenditures Beneficiaries

expenditure in by local development strategies according
Application form (project start)
Leader measures
measures with Payment request (project end)
respect to total LAG operations databases
RDP
expenditures

Number and Number of innovative projects supported by Beneficiaries

% of LAGs
Application form (project start)
innovative
Number of innovative projects supported by the
projects Payment request (project end)
RDP
supported by LAG operations databases
LAGs of all
innovative Surveys to LAGs and end beneficiaries

RDP projects

112
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Number of Number of partnerships created under Leader Application form (project start)
partnerships
Payment request (project end)
created
(Leader) LAG operations databases

Timing of data collection

All data should be collected for beneficiaries and non-beneficiaries from 1.1.2014 or from the programme start –
baselines

Methodology to calculate common and additional indicators

Common indicators

For common indicators R24/T23 - Jobs created in supported projects (Leader), the procedure is similar as in case of
CEQ 16.

For 2 other common indicators – R22/T21 and R23/T22, the calculation is based on monitoring data collected via the LAG
operations database as a ratio of the rural population covered by the intervention to the total rural population.

Secondary contributions

Secondary contributions of operations which are programmed under other FAs than P6B might be also considered
when calculating the indicator R24. Secondary contributions to the value of result indicators are calculated based on
surveys with those beneficiaries which are implementing operations via different focus areas (e.g. 2A and B, 3A, 6A)
and have flagged the additional contributions to FA 6B. Samples of surveyed beneficiaries can be set up in accordance
with the expected/actual contributions of implemented operations (e.g. farm investment measures, young farmers, farm
diversification measures, forestry technology and processing measures, local infrastructure etc.) to the FA’s objectives.
If there is not sufficient uptake to calculate the secondary contributions in a robust way a qualitative assessment is also
possible (see the Annex 10 of the guidelines: Assessment of RDP results: how to prepare for reporting on evaluation
2017). Findings can be used in answering the CEQ 19.

Additional indicators

Additional indicators can also be calculated based on the data collected by the LAG operations database or using the
qualitative methods:

• Survey to beneficiaries of measures 7.1, 7.2, 7.4-7.8

• Structured focus groups 87, including MAPP method for the assessment of net effects, interviews with LAGs involved
in these measures.

The proposed methods will enable: a) the collection of additional information in order to interpret the quantitative values
of the indicators; b) the collection of data in order to assess the net effects of the RDP on local development.

Secondary contributions of operations implemented under FAs other than 6A FA should also be included in the
calculation of additional indicators. The procedure is similar to the one described for common indicators.

Challenges/Risks/Issues
• Lack of data to answer the EQ (e.g. low RDP uptake, etc.)
• Low quality of data that does not enable to provide evidence based answer to the EQ

87
For details on how to use surveys and focus groups, see ‘Capturing the success of your RDP: Guidelines for ex post
evaluation of 2007-2013 RDPs’, Evaluation Helpdesk, June 2014
113
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Proposed solutions

In case of a lack of data (low uptake,), this EQ can focus on the expected jobs to be created, services improved, and
access to services (via interviews and expert opinions).

In case of low quality data, qualitative information can be sought through interviews, surveys, focus groups.

Answer to CEQ

The answer is provided based on the calculated values of indicators, on the collected qualitative information or on the
qualitative assessment.

Conclusions Recommendations

Main conclusions are stated here for the FA based on the Main recommendations linked to the conclusions are
evaluation findings. stated here for the FA.

2.18 Focus Area P6C, Evaluation question 18

To what extent have RDP interventions enhanced the accessibility, use and quality of information and communication
technologies (ICT) in rural areas?

List of measures contributing to the FA 6C:

Programmed measures/sub-measures (proposed by EC) 88:

• Measures and sub-measures of Art. 89 15 Advisory services, farm management and farm relief services
• Measures and sub-measures of Art. 14 Knowledge transfer and information actions
• Measures and sub-measures of Art. 35 Co-operation
• Measures and sub-measures of Art. 19 Basic services and village renewal in rural areas

Examples of measures/sub-measures programmed under other FAs but potentially showing secondary contributions
to this FA:

• All above measures if programmed under the other than 6C focus area and contributing to the. accessibility, use and
quality of information and communication technologies (ICT) in rural areas
• Measures and sub-measures of Art. 35 of Regulation (EU) No 1303/2013, Support for Leader local development

Consistency check between CEQ, judgement criteria and indicators

Judgement criteria

Judgement criteria (JC) proposed by the Working Document: Common Evaluation Questions for Rural Development
Programmes 2014- 2020:

• Access of rural households to ICT has increased

Note: stakeholders in Member States might add/change judgement criteria in line with the intervention logic of the FA
(selection and combination of measures)

88
WP: Guidelines for strategic programming for the period 2014-2020, European Commission, 04/03/2014, published on
ECAS/circabc
89
All articles mentioned in this section of each fiche for answering CEQ 1 – 18 are of Regulation (EU) No 1305/2013, unless it
is stated otherwise in the text directly
114
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Indicators

Common indicators

Common result/target indicator:

• R25/T24 - % of rural population benefiting from improved services/ infrastructures (Information and communication
technologies – ICT)

The following other common indicators might be used to answer the CEQ:

Common output indicators 90 (Data collected via the operations database):

• O3 Number of actions/operation supported

• O4 Number of beneficiaries
• O15 Population benefiting from improved services

Common context indicator (Data need and data sources are described in the Working document: Proposed list of common
context indicators):

• CCI 1 Population

Additional indicators and information

Common indicators are not sufficient to answer the CEQ. WD Common Evaluation Questions for Rural Development
Programmes 2014-2020 suggests:

• % of rural households accessing ICT with the RDP support

Note: More additional indicators can be developed when judgement criteria are added to specify the evaluation question
in MS.

Qualitative indicators

Quantitative information may also be used, e.g. to assess the quality of ICT services/infrastructure created and the extent
to which this improves the attractiveness of local rural areas, Degree of satisfaction with quality, accessibility and
coverage of ICT services and infrastructure.

Data needs and data sources

Indicators Data needed Data source

R25/T24 Access of Population benefiting from supports Beneficiaries

rural households to services/infrastructure (broadband internet
Application form (project start)
Common indicators

ICT has increased infrastructure and other ICT) under measure

7.3 Payment request (project end)

Total rural population covered by the RDP Beneficiaries and non-beneficiaries

area National and regional statistic

Eurostat rural population statistics:

[Link]
development/data/database

90
WD: Data item list for Pillar II operation database and WD RD programming and target setting

115
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

% of rural households Number of rural households accessing ICT, Beneficiaries

accessing ICT with the of which with RDP support
Application form (project start)
Additional
indicators

RDP support
Payment request (project end)

Beneficiaries and non-beneficiaries

National and regional statistic

Timing of data collection

All data should be collected for beneficiaries and non-beneficiaries from 1.1.2014 or from the programme start – baselines

Methodology to calculate common and additional indicators

Common indicator

The common indicator – R25/T24, is calculated based on monitoring data collected via the operations database as a
ratio of rural population benefiting from improved services/ infrastructures (Information and communication technologies
– ICT) to the total rural population.

Additional indicator

Additional indicator can be also collected via the operations database (if added in the monitoring system) or using
qualitative methods, such as:

• Survey to population covered by measure 7.3

• Structured focus groups 91, including the MAPP method to assess the net effects of the RDP (see section XX for a
description of this method).
• Interviews with LAGs involved in this measure
• The combination of surveys and MAPP method will provide the necessary triangulation results to validate the
findings.

The proposed methods will enable: a) the collection of additional information in order to interpret the quantitative values
of the indicators; b) the collection of data in order to assess the net effects of the RDP on accessibility to and quality of
ICTs.

Challenges/Risks/Issues
• Lack of data to answer the EQ (e.g. low RDP uptake)
• Low quality of data that does not enable to provide evidence based answer to the EQ

Proposed solutions

In case of lack of data (low uptake,), this EQ can focus on the expected coverage, accessibility and use of broadband and
other ICT (via interviews and expert opinions).

In case of low quality of data, qualitative information can be sought through interviews, surveys, focus groups.

Answer to CEQ

The answer is provided based on the calculated values of indicators, on the collected qualitative information or on the
qualitative assessment.

91
For details on how to use surveys and focus groups, see ‘Capturing the success of your RDP: Guidelines for ex post
evaluation of 2007-2013 RDPs’, Evaluation Helpdesk, June 2014
116
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Conclusions Recommendations

Main conclusions are stated here for the FA based on the Main recommendations linked to the conclusions are
evaluation findings. stated here for the FA.

117
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

3. EVALUATION QUESTIONS RELATED TO OTHER RDP ASPECTS

3.1 Operational performance, Evaluation question 19

To what extent have the synergies among priorities and focus areas enhanced the effectiveness of the RDP?

Programme synergies are linked to the entire RDP intervention logic and to primary and secondary contributions of RDP
operations to rural development focus areas, which can create synergies or transverse effects between them and
consequently also between rural development priorities.

Consistency check between CEQ, judgement criteria and indicators

Judgement criteria

Judgement criteria (JC) proposed by the Working Document: Common Evaluation Questions for Rural Development
Programmes 2014- 2020:

• The supported RDP measures are complementary so as to produce synergy through their interaction

Note: stakeholders in Member States might add/change judgement criteria in line with the intervention logic of the FA
(selection and combination of measures)

• The composition of measures supports their synergy at FA, RD priority and at programme level

Indicators

Common indicators

This CEQ concerns all common result indicators as defined by Commission Implementing act, Annex IV.

Additional indicators and information

WD Common Evaluation Questions for Rural Development Programmes 2014-2020 suggests:

• Positive and negative interactions among the supported RDP measures

• Secondary contributions of supported RDP measures

Note: More additional indicators can be developed when judgement criteria are added to specify the evaluation question
in MS.

Qualitative information

In the AIR submitted in 2017 a qualitative assessment of synergies between focus areas and between RD priorities is
planned. Synergies are positive horizontal transverse effects which occur due to positive interactions between various
focus areas/RD priorities (fostering effects on one focus area from interventions under the other focus areas). Apart from
synergies, focus areas and priorities can also affect each other in a negative way and cause negative transverse effects,
weakening the effects of one focus area due to the interventions under the other focus areas). Positive or negative
transverse effects are assessed qualitatively in 2017, starting from the RDP specific composition of measures/sub-
measures under each focus area and comparing it with all the other focus areas. These comparisons will allow one to
judge the extent to which pursuing a specific intervention logic affected the final achievements towards objectives,
effectiveness and efficiency of the RDP and its results and impacts.

118
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Data needs and sources

Indicators Data needs Data sources

All result indicators and complementary Quantification of all common and Evaluation findings for CEQ 1 – 18
indicators
Common

result indicators additional indicators used to observe and calculation of result indicators
the programme results taking in
Reports on thematic evaluations
consideration the primary and
secondary contributions Operations database
Additional indicators

All additional indicators which are used to Quantification of all relevant Evaluation findings for CEQ 1 – 18
calculate primary and secondary additional indicators and calculation of result indicators
contributions to focus areas
Reports on thematic evaluations

Information collection linked qualitative assessment:

• Confirm and describe synergies between measures addressing the same focus area
• Identify synergies, complementarities and conflicts between measures addressing different focus areas, between
focus areas and RD priorities

Timing of data collection

Proposed data are collected through an ongoing processes starting at the beginning of RDP’s implementation. Qualitative
information is collected at the time of evaluation.

Methodology to calculate common and additional indicators

To answer the above question in a sound way is a methodological challenge.

For the reporting in 2017, in the assessment of synergies between focus areas and between priorities, the following should
be taken into consideration:

• Secondary contributions of operations to FAs other than the programmed focus areas as calculated during answering
CEQ 1- 18
• Findings of the qualitative assessment

Quantitative assessment

For the quantitative assessment it is recommended to look at how operations programmed under certain FA contributed
additionally to the value of indicators (common/additional) related to other focus areas and compare these findings
between focus areas. The comparisons will help to understand better the interactions between focus areas and their
intervention logics. For example, high contribution to the indicators value´s indicate a strong synergy between FA. Low or
minus contributions show negative transverse effects. The scores should be followed by an explanatory text describing
the reasons for the positive, negative or neutral interactions.

Qualitative assessment

Findings of quantitative assessment should be triangulated with the qualitative assessment. There are the following
possibilities:

The tool to appraise the additional/secondary contributions of measures/sub-measures to other focus areas than those
programmed and synergies/negative transverse effects between focus areas and between priorities is presented in the
Guidelines “Assessment of RDP results: how to prepare for reporting on evaluation in 2017, PART III, Annex 10. The
table 1 of the Annex 10 allows to follow the direct (primary) and secondary path from single measures to the focus areas
119
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

and RD priorities. Table 2 of the Annex 10 describes how the combination of measures programmed under a specific
focus area impact all other focus areas, either weakening (negative transverse effects) or fostering them (synergies).

In the standard situation it is expected that measures programmed under the same focus area would demonstrate strong
positive interactions so as to contribute to the policy objectives of that focus area. The tool will help to confirm and/or
dispute this depending on the influence of evolving RDP contexts. For the rest, it is up to the evaluator to analyse the
positive, negative or neutral interactions between different RDP measures.

This tool can be completed with the theory of change and by expert panels to triangulate findings of the table and
confirm/correct synergies/negative transverse effects between measures, FAs and RD priorities.

There are several options how to proceed in the assessment of programme synergies: 1.) The evaluator can start with a
qualitative assessment (using both tables of annex 10 in the PART III of the guidelines), 2.) Conduct the quantitative
assessment of values of result indicators and 3.) Check the findings with theory of change or panels of experts.
Alternatively, one can start with a quantitative assessment and triangulate findings using both tables, theory of change
and panels of experts.

Challenges/Risks/Issues
• No risks, since the proposed qualitative assessment

Proposed solutions
• N/A

Answer to CEQ:

The answer is provided based on the calculated values of indicators, on the collected qualitative information or on the
qualitative assessment.

Conclusions Recommendations

Main conclusions are stated here for the FA based on the Main recommendations linked to the conclusions are
evaluation findings. stated here for the FA.

3.2 Technical Assistance, Evaluation question 20

To what extent has technical assistance contributed to achieving the objectives laid down in Art. 59(1) of Regulation (EU)
No 1303/2013 and Art. 51(2) of Regulation (EU) No 1305/2013?

List of activities of the TA

TA assistance supports the entire RDP implementation, monitoring and evaluation and therefore is linked to the RDP
intervention logic.

Consistency check between CEQ, judgement criteria and indicators

Judgement criteria

Judgement criteria (JC) proposed by the Working Document: Common Evaluation Questions for Rural Development
Programmes 2014- 2020:

• Institutional and administrative capacities for the effective management of the RDP have been strengthened
• Capacities of relevant partners as defined by the Regulation (EU) No 1303/2013, Art. 5(1) have been reinforced
• RDP has been communicated with the public and information has been disseminated
• Monitoring has been improved

120
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)
• Evaluation methods have been improved and have provided robust evaluation results
• Information on evaluation practices has been exchanged
• The RDP implementation has been improved
• Administrative burden on beneficiaries has been reduced

Note: stakeholders in Member States might add/change judgement criteria in line with the intervention logic of the FA
(selection and combination of measures)

Indicators

Common indicators

There is no common indicator for Technical Assistance

Additional indicators and information

In the absence of a common indicators, the following additional information are proposed by the Working Document:
Common Evaluation Questions for Rural Development Programmes 2014-2020 suggests:

• Number of staff involved in RDP management

• Skills of staff involved in RDP management (broken down by types of skills)
• Types and number of capacity building activities
• Functionality of the IT system for programme management
• Number and type of RDP communication and dissemination activities
• Number of people receiving information about the RDP
• Information on the use of evaluation results
• The length of the application and payment process

Note: More additional indicators can be developed if needed, e.g. when additional judgement criteria are added to specify
the evaluation question in the Member State.

Qualitative information

Additional qualitative information can be collected, e.g. quality and effectiveness of communication and dissemination
activities, quality of the reports developed through the TA support (e.g., thematic evaluations).

121
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Data needs

Monitoring data:

• Expenditures and outputs of activities operated under the technical assistance measure.

Additional data linked to additional information (optional) and methods used, e.g.:

• Number and skills of the staff involved in RDP management

• Number of capacity building activities, by type of capacity building
• Number of staff that use the IT management system
• Level of satisfaction with the IT management system (Likert scale)
• Number of communication and dissemination activities, by type and volume of attendance
• Level of satisfaction with the quality of communication and dissemination activities (e.g. provide the information
needed according to the different target groups).
• % and description of evaluation recommendations that have been taken into account in the RDP implementation
• Duration (in days) of the application process (from for the formal application to the respective approval)
• Duration (in days) of the payment process (from issuing invoices to receiving payment)

Data sources

Monitoring systems of the MA to incorporate data on the above indicators

Interviews to staff involved in RDP management

Surveys to target groups of communication and dissemination activities

Attendance sheets of communication and dissemination events

Evaluation sheets for participants of communication and dissemination events

Information from monitoring and evaluation reports and audits

Timing of data collection

It is recommended to collect data information on TA activities from RDP stakeholders and beneficiaries systematically
from the start of TA activities.

Proposed methodology to calculate additional indicators, and analyse data and information

Interviews with various actors involved in programme implementation, monitoring and evaluation, surveys, collection of
information from evaluations, attendance sheets of events, observations of the IT system´s efficiency and effectiveness,
information from reports and audits, observations of the delivery mechanism, etc.

Challenges/Risks/Issues
• Late start of the implementation of the technical assistance

Proposed solutions

In case of a late start, an estimate of the expected contribution of technical assistance should be carried out through
interviews with the MA and relevant staff.

122
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Answer to CEQ

The answer is provided based on the calculated values of indicators, on the collected qualitative information or on the
qualitative assessment.

Conclusions Recommendations

Main conclusions are stated here for the FA based on the Main recommendations linked to the conclusions are
evaluation findings. stated here for the FA.

3.3 National Rural Networks (NRN), Evaluation question 21

To what extent has the national rural network contributed to achieving the objectives laid down in Art. 54(2) of Regulation
(EU) No 1305/2013?

Intervention logic of the NRN, common elements:

• 4 common NRN objectives 92
• 7 groups of common activities 93

Stakeholders in MS should complete above common elements with programme specific NRN objectives and groups of
activities if relevant. 94

Consistency check between CEQ, judgement criteria and indicators

Judgement criteria

Judgement criteria (JC) proposed by the Working Document: Common Evaluation Questions for Rural Development
Programmes 2014- 2020:

• Number and types of stakeholders involved in RDP implementation has increased

• The quality of implementation of the RDP has been improved through the activities of the NRN, e.g.

o Improved capacity of RDP beneficiaries

o Improved evaluation awareness

o Lessons from evaluations are taken into account in programme implementation

• Broader public and potential beneficiaries are aware of the rural development policy and funding opportunities through
activities of the NRN
• Innovation in agriculture, food production forestry and rural areas has been fostered by the NRN

Note: stakeholders in Member States might add/change judgement criteria in line with the intervention logic of the FA
(selection and combination of measures)

Indicators

No common result/target indicator proposed.

Common output indicators 95 (Data collected via the operations database):

• Number of thematic and analytical exchanges set up with the support of NRN
• Number of NRN communication tools

92
Art. 54(2) of Regulation (EU) No 1305/2013
93
Art. 54(3) of Regulation (EU) No 1305/2013
94
Guidelines: Evaluation of NRN 2014-2020, European Evaluation Helpdesk for Rural Development, 2016,
[Link]
95
WD: Data item list for Pillar II operation database and WD RD programming and target setting
123
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)
• Number of ENRD activities in which the NRN has participated

Additional indicators and information

Stakeholders will develop additional result indicators to answer the evaluation question in the AIR submitted in 2017. In
case the NRN intervention logic contains programme specific objectives and evaluation questions, stakeholders will
develop programme specific indicators. Programme specific indicators are not used to answer the above common
evaluation question; only additional indicators will be used for this purpose. Additional result indicators should be consistent
to the above judgment criteria or the judgment criteria added by the stakeholders in the Member States and should express
the NRN’s added value, as defined by the common objectives or by stakeholders in the Member State. Additional NRN
result indicators might be of a quantitative or qualitative nature 96.

Additional indicators proposed by the Working Document: Common Evaluation Questions for Rural Development
Programmes 2014-2020:

• Number of stakeholders (by type) participating in the implementation of the RDP due to activities of the NRN (including
those through LAGs)

• Number of RDP modifications based on evaluation findings and recommendations from thematic working groups
organized by the NRN

• % of RDP implemented projects encouraged by NRN(P) activities

• Number of persons that have been informed about the rural development policy and funding opportunities through
the NRN communication tools

• % of innovative projects encouraged by NRN out of the total number of innovative projects supported by the
RDP(s)
Further proposed additional indicators:

• Number of new applications for funding as a result of NRN awareness raising activity

Note: Additional indicators can be developed when further judgement criteria are added to specify the evaluation
question in the Member State.

Qualitative indicators

For the qualitative assessment the following qualitative indicators can be proposed:

• Degree of increased evaluation awareness

• Degree of increased capacity acquired by RDP beneficiaries
• Degree of increased awareness of rural development policy and funding opportunities of the broader public and
potential beneficiaries

96
Guidelines: Evaluation of NRN 2014-2020, European Evaluation Helpdesk for Rural Development, 2016,
[Link]

124
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Data needs

The data needs are also described in detail in the Guidelines: Evaluation of NRN 2014-2020. The data for output indicators
are collected via the operations database.

Data and information linked to additional indicators/information at the result level (optional, but recommended as output
indicators are not sufficient to answer the CEQ) can also be collected via the operations data base and with respect to
selected methods (described above) from NRN members, beneficiaries, and other stakeholders, for example:

• Number and type of stakeholders participating in RDP implementation as a result of NRN activities, of which through
LAGs

• Number of RDP modifications based on: a) evaluation results; b) recommendation from NRN TWGs

• Target groups of NRN activities that have Increased evaluation awareness (Likert scale)

• NRN members (number) and MA staff involved in RDP management (number) who claim to have increased their
knowledge about evaluation thanks to NRN activities (and description)

• RDP beneficiaries that have Increased their capacity due to NRN activities (Likert scale and description of capacity
improvements) and description of NRN activities most useful for increasing capacity

• Total number of RDP implemented projects

• Number of RDP projects implemented as a result of NRN(P) activities

• Number of persons that have been informed about the rural development policy and funding opportunities through
the NRN communication tools

• Increased awareness (Likert scale) of the broader public and potential beneficiaries

• Number of new applications for funding due to NRN activities

• Number of innovative projects encouraged by the NRN

• Total number of innovative projects supported by the RDP

Data sources

Minimum required for the output indicators:

• NRN monitoring data (for the output indicators)

• NRN websites

Data sources for additional information:

• It is proposed that NRNs include the above data for additional quantitative indicators in their monitoring systems
• Further qualitative information through interviews/focus groups with NRN members

• Surveys to NRN members

Timing of data collection

It is recommended to collect data and information on NRN beneficiaries and contextual data (data for context parameters
(as defined by the Guidelines: Evaluation of NRN 2014-2020) systematically from the start of NRN activities.

125
 Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

Methodology to calculate common and additional indicators

Proposed methods are described in detail in the Guidelines for NRN evaluation 97. For example, the following methods:

• Surveys
• Dialogue based or participatory methods (e.g. self-assessment practices, focus groups)
• Analytical methods (e.g. cost-benefit analysis)
• Diagnostic methods (e.g. functional analysis of Networks, network diagnosis, social network analysis
• Interpretative methods (e.g. imaging, systemic structure constellations or choreographies, text analysis)
• Case studies
• MAPP method

The combination of several methods is proposed as the best evaluation approach and for triangulation of findings.

Challenges/Risks/Issues

1. Late set-up of the NRN units

2. Late start of the NRN’s activities

Proposed solutions

In cases where the set-up was late, assess through interviews with the MA or with the newly established NRN unit (if it
exists) the expected contributions of the NRN to RDP’s implementation.

In cases where there is a late set-up and/or late start of the NRN’s activities, assess the expected contributions of the NRN
to the achievement of the objectives laid down in Art. 54(2) of Regulation (EU) No 1305/2013 (through interviews with the
MA, with the newly set-up NRN unit (if it exists) or focus groups with NRN members).

Answer to CEQ

The answer is provided based on the calculated values of indicators, on the collected qualitative information or on the
qualitative assessment.

Conclusions Recommendations

Main conclusions are stated here for the FA based on the Main recommendations linked to the conclusions are
evaluation findings. stated here for the FA.

97
Guidelines: Evaluation of NRN 2014-2020, European Evaluation Helpdesk for Rural Development, 2016
[Link]

126
Annex 11 - Fiches for answering Common Evaluation Questions for rural development
programmes 2014-2020 (CEQ 1 – 21)

127