A simple but powerful tool for the assessment of financial progress in EU co-funded grant programmes – Simple Progress Overview Tool ( SPOT )

Mid-term evaluation of grant programmes always aim to assess the financial progress of the programme. Progress to date assessment is conducted via addressing three issues: by overviewing progress so far, by exploring the reasons behind the current level of progress, and by answering the specific evaluation questions formulated as a result of progress overview. The Simple Progress Overview Tool is a method and a tool facilitating the overview of financial progress, supporting the exploration of reasons behind the current level of progress and enabling the identification of targeted further evaluation questions. The methodology applies the Du Pont approach of division to factors, in which the variable describing a phenomenon is divided into the multiplication of factors (milestone stages of the process) influencing the variable. Also, this division to factors facilitates the identification of reasons behind progress. The factors carry individually meaningful information to the evaluation of progress. This method was used by many EU member states including the Czech Republic, Romania, and Malta for the evaluation of progress in the course of the mid-term evaluation of their National Strategic Reference Frameworks.


INTRODUCTION
Evaluation is a support activity of the management cycle of EU programmes.Evaluation is "Judgement on the value of a public intervention with reference to criteria and explicit standards (e.g. its relevance, its efficiency).The judgement primarily concerns the needs which have to be met by the intervention, and the effects produced by it.The evaluation is based on information which is specially collected and interpreted to produce the judgement."(MEANS, 2000;Glossary, 2003).This definition is followed in the relevant EU regulations (EC 2006(EC /1999)), the corresponding methodological Working Papers and Working Documents (EC, 2000;2006) and is generally in line with the OECD definition (OECD, 1999).
According to MEANS (2000), which is a standard sourcebook of all evaluation activities carried out in relation to EU programmes, evaluation can be formative or summative.The method described in this paper definitely addresses and supports formative evaluation, as part of the mid-term evaluation process.Using the widely acknowledged MEANS (2000) definition, formative evaluation is: "Evaluation which is intended for managers and direct protagonists, in order to help them improve their action (feedback).Formative evaluation applies mainly to a public intervention during its implementation (on-going or intermediate evaluation).It focuses essentially on implementation procedures and their effectiveness and relevance."(MEANS, 2000).The distinction between formative and summative evaluation has wittily been summarised in the following way: "When the cook tastes the soup, it's formative; when the guests taste the soup, it's summative."(MEANS, 2000).
Mid-term evaluations and on-going evaluations are often quoted as synonyms, though they are not in many experts', including my own interpretation.However, in this paper mid-term evaluation also covers on-going evaluations.Mid-term evaluations are evaluations carried out in the course of programme implementation (usually in mid-time or the second half of programme duration), aiming at a) identifying potential interventions that can be promptly used in order to enhance the relevance, effectiveness and efficiency of the programme and b) collecting substantial information to the formulation of the next programming period intervention plan.
As part of the evaluation cycle closely linked to the programming cycle the standard evaluation activities are 1) ex ante evaluations (used in the programming and planning phase; 2) mid-term evaluation/on-going evaluation (used during programme evaluation); 3) ex post evaluation (used at programme close).For easy reference on how programming, implementation and evaluation phases are interlinked, please see the Figure 1 below.

FINANCIAL PROGRESS
One of the most obvious starting points for mid-term evaluations is identifying the current status of progress, including financial progress.The issue of financial progress is in strong coherence with all three evaluation themes of the mid-term evaluations, i.e. relevance, efficiency and effectiveness and, therefore, it is in the core of setting the scenes for evaluation.
in a nutshell, in terms of relevance, financial progress indicates whether the programme follows the right set of goals driven by the socio-economic environment.As for efficiency, financial progress points at two areas: how the institution system copes with implementation (i.e.grant allocated/cost of implementation ratio) and the cost-effectiveness of the programme (i.e.grant allocated/ programme objective indicator ratio), where both ratio uses current level of grant allocated as the numerator.Effectiveness assesses the current level of financial progress vs. programme objectives.Please note that "allocation" of grant stands for grants "reimbursed" or "spent" or "paid" in this context, and not referring to initial financial allocation or "setting the budget" for the programme.

QUANTIFICATION OF FINANCIAL PROGRESS
Progress is one of the most relevant attributes of any programme implementation.One of the first questions of the evaluation of progress is "Where are we now?" or "How much have we spent so far?".First, an indicator has to be defined that can individually, attributably and precisely interpret the current financial progress level.This value -by nature of grant programmes -is absorption that we can define as grant spent per budget allocated to the programme.Therefore, progress is measured by the grant amount already "spent" on programme beneficiaries.
in order to assess if spending is on time or behind schedule, the current level of absorption has to be compared with the planned scheduling of spending.in case there is no plan, the simplest estimate to assess progress is to assume linear spending by using total programme budget/number of programme years as a reference base to estimate the expected annual spending level.
Please note that evaluations also concern progress in terms other than financial progress.Progress in general is usually measured through the system of programme/ impact/result/output indicators. in the frameworks of the current paper, "progress" always mean "financial progress".

PROJECT SELECTION PROCESS
As a first step to understanding financial progress of a grant programme, the grant allocation process (e.g. an application process or other type of project selection process) has to be clearly defined. in order to demonstrate the use of SPOT methodology, a standard grant application process will be assumed, consisting of a three-level assessment system including an administrative, an eligibility and a technical assessment on the application submitted.in this context, it is presumed that if an application passes through all three levels, then it is approved, a grant contract is signed and payment can be initiated to the beneficiary of the grant.
A standard application process for demonstrating purposes (that is very similar to those often used in National Strategic reference Frameworks in the 2007-2013 EU programming period) includes ten steps from submission of application up to payment of the grant.Please note that in some of the Member States the three-stage process is reduced to two or even one in practice by integrating one stage into another.This process is depicted in the Figure 2.
AGrárTUDOMáNyi KöZlEMéNyEK, 2014/59.68 in order to measure financial progress, one has to consider all the above-mentioned stages and the corresponding grant amounts.Also, one has to consider that those stages of a project selection or application system that include assessment, have two factors linked to them.The first one is the "process" aspect, indicating if the application covering a certain grant amount has been processed at that stage.The second one is the "pass" aspect, clarifying if the application carrying the grant has been approved to pass to the next level of the project selection process.Therefore, all assessment phases has to be decomposed to a "processed" and a "passed" element (Figure 3).

METHODOLOGY OVERVIEW
The method and tool described in this paper is a general one, capable of overviewing financial progress of a programme, thus supporting programme evaluation activities.
The Simple Progress Overview Tool (SPOT) is a tool used to identify the reasons underlying a figure describing, at a predefined point of time, the level of absorption (i.e. the value of reimbursed grants) of an intervention by breaking that figure down to constituent, individually meaningful factors that (are likely to) have a direct, substantial impact on the final result.
SPOT factors are individually meaningful numerical indicators generated and applied in the course of the analysis of the value which is expressed in the form of ratios illustrating significant stages in the process of project selection.Multiplication of the SPOT ratios for an intervention will equal absorption ratio.
The final output of the SPOT analysis can also create a solid basis for estimating absorption perspectives of the programmes by building a simple mathematical model on the extrapolation of factors and defining potential scenarios (without intervention, or with intervention to one or more than one factors).

THE SPOT EQUATION
The basis of the SPOT methodology and tool derives from the definition of absorption. in mathematical terms, the absorption ratio is the value of reimbursed grant amount divided by the planned budget of the programme.The absorption ratio clearly indicates how much grant the programme has spent to date.
The absorption ratio can be divided into several other multiplying factors (ratios) by bringing in the financial values corresponding to the pre-defined stages of the project selection process.Following the Du Pont approach, the mathematical product of the individual ratios equals the absorption ratio after simplification.
The Figure 4 below summarises the "SPOT equation", the basic concept of the tool.By the de-composition of the mathematical product, the following ratios can be obtained: − Popularity ratio = requested grant/planned budget.
This ratio describes the level of demand for the selected intervention.− Admin processed ratio = admin processed requested grant/requested grant.This ratio describes the proportion of requests already processed by the project selection system.− Admin pass ratio = admin passed request for grant/ admin processed request for grant.This ratio describes the proportion of requests that received positive feedback and have passed the first level of assessment.− Eligibility processed ratio = eligibility processed request for grant/admin passed request for grant.This ratio describes the proportion of administratively adequate requests that has already been processed by the project selection system.− Eligibility pass ratio = eligibility passed request for grant/eligibility processed request for grant.This ratio describes the proportion of administratively adequate requests that received positive feedback and have passed the second level of assessment.
* Assuming a three-stage application selection process composed of administrative, eligibility, and technical assessment; ** Assuming that after the three-stage process there is also a final approval of projects selected; *** All figures in the equation stand for values, not number of applications.
MezősiB2:Layout 1 4/23/14 2:44 PM Page 3 − Technical processed ratio = technical processed request for grant/eligibility passed request for grant.This ratio describes the proportion of requests that have met administration and eligibility criteria and have been processed by the project selection system.− Technical pass ratio = technical passed request for grant/technical processed request for grant.This ratio describes the proportion of requests that requests that have met administration and eligibility criteria and received positive feedback in terms of technical assessment and therefore have passed all three levels of assessment.− Approval ratio = approved grants / technical passed request for grant.This ratio describes the proportion of requests that have been found adequate in terms of both administrative, eligibility and technical criteria and have been approved by the respective authority.This ratio fundamentally reflects a "processed" element, though, in some rare cases might carry a "passed" element as well.− Contracting ratio = contracted grants/approved grants.This ratio describes the proportion of approved requests that have already been contracted out.This ratio is a "processed" ratio by nature.− Payment ratio = reimbursed grant/contracted grants.
This ratio describes the proportion of contracted grants that have already been transferred to the beneficiaries.This ratio is a "processed" ratio by nature.
Consequently, the detailed ratios provide insight and an increased understanding of the reasons behind the overall "absorption ratio".The level of the de-composed ratios within the absorption ratio assists the identification of potential bottlenecks in the project selection system.
Examination of the more detailed ratios allows the different factors underlying relatively high or relatively low absorption ratios to be identified, and hence the nature of the problems to be explored in more detail and better targeted further assessments to be developed.in order to assess if a ratio is high or low, comparisons to 1) similar previous programmes 2) to other similar programme providers (e.g.Member States) or 3) to absolute values in terms of the "processed" ratios (100% if all inputs have been processed).
As a result of the de-composition of absorption, the above-mentioned individual ratio values are calculated for the relevant programme level (i.e. in terms of EU Structural Funds and Cohesion Fund programmes they are National Strategic reference Frameworks, Operational Programmes, measures, sub-measures).Some potential explanations to the low ratio values (constituting potential bottlenecks in the application system to be assessed in more depth) disclosed as a result of filling-in the SPOT equation with relevant data are as follows: Hypotheses developed upon the evidence based results of the SPOT equation can establish a sound basis for further investigation of reasons behind progress (Figure 5).

EXAMPLE OF APPLICATION (MID-TERM EVALUATION OF 2007-2013 NSRF OF ROMANIA)
This section contains a valid example of the use of the previous version of SPOT, capable of demonstrating its mechanism and the potential benefits of its application.
in the 2007-2013 romanian NSrF, there have been seven Operational Programmes (rOP, SOP ENV, SOP T, SOP iEC, SOP HrD, OP DAC OP TA).The aggregated financial figures of these programmes equal the NSrF figures (Table 1).
The analysis was not separated to "process" and "pass" factors, and the assessment stages (administrative, eligibility, technical) had to be integrated into one stage for evaluation purposes as a result of the lack of robust data and lack of consistency in project selection systems applied for each Operational Programme.As a result, the "Approval" stage covers all project selection related (i.e.administrative, eligibility, technical) ratios.Therefore, the method could use four distinct stages to de-compose the absorption ratio into, i.e. popularity ratio, approval ratio, contracting ratio, payment ratio.
The mid-term evaluation (ACiS, 2010) interprets the above figures in the following way: − "Popularity ratio: there was a large increase in demand between the two dates because of: (i) the launching of more calls and interventions (particularly under SOP ENV and SOP T) as well as (ii) the large demand overhang for grants under SOP iEC and rOP.− Approval ratio: the significant drop in this ration (from 72% to 24%) may be considered to indicate that the management an implementation system has capacity problems in coping with the registry, processing and evaluation of the large number of incoming applications.− Contracting ratio: the further increase (from 61% to 82%) illustrates the sound administrative procedures for contracting in the post-selection phase.− Payment ratio: that the ratio has not changed significantly (11% vs. 18%), may account for the fact that absorption shows only a 2.1% point increase over the one-year period since mid-2009."

SCOPE OF FURTHER APPLICATION
The present version of the SPOT method and tool has been tested and used in Hungary, romania, ireland, Malta and several other countries in the CEE region for progress evaluation purposes.However, the method can be used in many evaluation exercises assuming that the conditions set in the limitations of methodology are met.it is also capable of providing a solid basis for programme monitoring and hence a program progress overview system capable of producing management reports.

LIMITATION OF THE METHODOLOGY
The method and the tool has certain limitations: − The longer the application process, the more informative and meaningful the equation is.
However, when using only a few stages, the individual ratios are less meaningful.− it can only be used with limitations in application processes, in which application stages cannot be well-defined or consistently defined.− if the evaluation covers multiple interventions, and the number of stages are different for each intervention, then a common basis has to be established as a prerequisite of comparative assessment.− The method and tool is sensitive to the availability and robustness of data inputted.inputted financial data has to be clear, available in the required structure at the same cut-off date, the financial data has to be comparable (i.e.void of exchange rate effects) and has to be available for each application stage concerned, in order to yield meaningful results.

Figure 1 :
Figure 1: Programme cycle and evaluation cycle of the Structural and Cohesion Fund co-financed programmes

Figure 2 :
Figure 2: A standard project selection (application) process using a three-stage selection system

Figure 3 :
Figure 3: Breakdown of the three-stage selection system to processing and approval elements

Figure 5 .
Figure 5. SPOT equation tool (assuming a three-stage project selection system) potential explanations for low ratios -these figures concern the NSrF without the ETC Programmes.