4th Steering Committee Logroño, 28th November 2006 Draft Evaluation Methodology Futurreg Evaluation: “The evaluation of public intervention consist of judging its value in relation to explicit criteria, and on the basis of information that has been specially gathered and analysed.” Intrinsic features: • Description • Regulatory dimension • Instrumental dimension Others features: • Scope of public policy evaluations. Controlling public expenditure and improving administration. • Evaluation should be more than just an instrument of control to measure complex features. • Key actors involved in the process. ¿ Why evaluate? • Verify programme effectiveness and efficiency • Learn • Aid reflection • To see what is happening? • Aid transparency + Futurreg • Foresight as a technique for decision making in organisations. • Evaluation strengthens the stated purpose of the project to provide a learning tool for European regions. • Evaluate to ensure transfer is as good as possible. • Foresight must be evaluated in context. • Evaluate whether the results and conclusions are used and/or taken into account in the region. ¿ Which kind of evaluation is required? • On going supplements the monitoring system and broadens the capacity to manoeuvre. • Ex post attempts to measure project effectiveness and efficiency. Evaluation structure • Context • Methodology • Data collection • Data analysis • Global analysis and selection of cases for study • Summaries of results and/or impacts • Communication of results and recommendations Evaluation Methodology • • Analysis of the project in two dimensions: • Analysis of the project as a whole • Analysis of foresight applications performed in Futurreg. Applications grouped into different categories. Indicators Indicators concerning the project as a whole, its objectives, toolkit and foresight methods. • Three questionnaires for different actors involved in the project Partners (7), Application managers (14-15) and Participants in foresight applications (case of studies) Analize the project as a whole (I) •As Futurreg is part of the INTERREG IIIC programme, its contribution to the programme objectives through this project needs to be evaluated. •Importance to evaluate aspects like cooperation and project partner inter-communication, partner experience exchange, the toolkit, etc. •Project as learning experience for participating regions and as a methodological guide for future experiences in other European regions in the futures and foresight field Analize the project as a whole (II) At this level, the main questions for evaluation are: •Were the objectives well defined? •Have they been achieved? •Have the expected results been obtained? •Was the workplan carried out? •Have the deadlines and planned costs of the project been respected? • Have the results been transferred from the project to other regions? Analyze the project as a whole (III) Indicators: •Project activity and working plan indicators (objective indicators) •Result indicators •Others indicators Indicators (I) AREA INDICATORS OBJECTIVE INDICATOR General Project activity and working plan indicators (objective indicators) Organization and Steering Committee meetings 6 coordination Partner participation in Steering Committee meetings 100% Recruitment of local auditors 6 Partner refund deadlines 2 weeks Activities performed according to initial project agenda deadlines Good working atmosphere created and consensus promoted Report sent on time 100% Activity and financial reports 5 Monitoring reports Every six months Evaluation reports Ongoing / ex post Project development in alignment with planned objectives. Project development in alignment with costs. Benchmarking Number of regional and interregional stakeholders informed about Futurreg and evaluation Number of regions informed about their benchmarking position as per 70, 10 per region regional indicators Foresight tool report 5 Regional benchmarking and evaluation report 7 Opportunity matrix for regional toolkit applications Creation of a basis and evaluation criteria for project applications and 1 implementation. Ongoing evaluation report. Ex post evaluation report. Evaluation of interregional workshops. Definition of Participants in interregional workshops foresight Type of participants in interregional workshops processes Participant satisfaction with interregional workshops 30 85% Production of regional futures toolkit with guide. Production of materials for up-skilling workshops 2 sets Interregional Workshops 2 Participants in each workshop 30 in each workshop Production of plans for regional applications 14 Interregional matrix comparing application-related issues and features Indicators (II) AREA INDICATORS OBJECTIVE INDICATOR General Project activity and working plan indicators (objective indicators) Implementing Number of regional stakeholders used in futures toolkit application foresight Satisfaction of regional organizations with accepted applications. 28 stakeholders, 4 per region Number of futures-toolkit-related plans and future collaborations Applications of futures toolkit 14 Regional application monitoring reports Summary report on steps to ensure sustainability of resulting operations. 14 starting from 2006, 1 biannual for region. In all, 21 Diffusion Diffusion level for “champions” in public sector organizations (number of 150 colleagues used by the encounters held and number of partners used) champions, 5 per champion Visits to web page 5,000 visits Publication of Futurreg project publicity brochure 3000 copies Encounters with 105 stakeholders to diffuse toolkit and project 2 Interregional conference Project web page, with information of interest on project and applications. Project web page in partners’ web pages. Other project indicators Meetings held Press notes released after meetings Targets hit Applications developed Number of intra-regional and inter-regional networks created and participants (public administrations, universities, technology centres, companies, etc) Number of foresight experiences generated Objective indicators Objective 1 Number of agents involved (and interested?) in regional foresight practice. Objective 2 Number of experiences at regional level in the use of foresight techniques. Number of foresight exercises practised Number of users requesting application in their own region. (Number of users demanding foresight in their own region). Objective 3 Number of integrated sources of information. 100 participants Indicators (III) Other Indicators AREA General INDICATORS Progress on creating foresight culture Beneficiaries taking foresight results on board. Regional public administrations’ satisfaction with, and interest in, foresight. Objective 1 Transferability. Toolkit transferability level to other European regions. (Number of applications from other regions interested in the application) Extension of the interregional network that maintains foresight tools: number of partners, class of partners, type of channels and communication flows, etc. Toolkit adaptability and flexibility for other regions and studies. User friendliness, success of project, etc. Objective 2 Applicability. Implementation or use status of foresight results in regional economic policies. Regional public administration and business interest in toolkit. Regional agents’ integrated use of foresight. Sectors where it is applied (sector type). Future sustainability of application in the region Objective 3 Interest for the region of area/sector benefiting from use of foresight. Impact on regional economy. Quality of sources Evaluating applications (I) Critical areas(1) were used in classifying applications: • Social, factors influencing human beings, society and lifestyle • Technological, factors stimulated by science and its applications • Economic, factors affecting industry and wealth creation • Environment, factors impacting on the physical world we living • Political, factors relating to government and administration  Preliminary version of the UPGRADE blueprint has been produced for the dissemination conference “Building the future on knowledge”. Brussels, September 23, 2004. S ocial Menter a Busnes Main theme: Economic value of the Welsh Language. Type of application: Company development and strategic contribution to policy framework and programme for action. Fondation Rurale de Wallonie Main theme: Support for rural development operations and on going education.. Type of application: diagnosis on future needs of rural territories of Wallonia in order to define the new strategy of the organisation FRW. Support existing working groups of the organisation. Institute of Technology, Sligo Main theme: Higher education Type of application: Develop strategy T echnological E conomical Athone Institute of Technology Main theme: Innovation/Sustainable development. Type of application: Development of a strategy/Organizational development. Loimaa Region Development Centre Main theme: Economic development (entrepreneurship) Type of application: Create networks and star up new joint projects among entrepreneurs. University of Malta Main theme: Knowledge Transfer Type of application: development of knowledge transfer strategy within the University´s ten year vision. [email protected] SCRL Main theme: Territorial development, tourism and patrimony, agriculture and sustainable energies. Type of application: improve the use of foresight in rural areas of Wallonia. Aristotle University of Thessaloniki Main theme: Innovation, new product development. Type of application: Strategy development towards technology and knowledge transfer.. Galway-Mayo Institute of Technology Main theme: Innovation in SME Type of application: Product Development and Innovation Planning. ADER, INESCOP and AICCOR Main theme: Innovation, research and technology. Type of application: Identification of Future Technological trends in the footwear Sector. Regional Council of Satakunta Main theme: manages the responsability of the long term regional development, defines objectives of the long term development and provides guidelines for the member comunities of land uses. Type of application: preparation of a new Regional strategic plan E P Countryside Council of Wales Main theme:Conserving tha natural enviroment and helping people to enjoy it and earn from it. Type of application:Developing the organisation´s strategic direction in medium term. Municipality of Thermi Main theme: Digital Research Centre: Digital Cities, egoverment. Type of application: regional policies, organizational restructure. nvironment olitical Malta enterprise Main theme: Innovation. Type of application: Development of a regional innovation strategy. Evaluating applications (II) Case study selection criteria: • Operational level and situation of exercise. • Design of foresight exercise. • Relevance of issues on which the foresight exercise is carried out. • Territorial balance according to types and regions involved in the project. • Novelty. Evaluating applications (III) The main questions (1) the evaluation considers at this level are: 1. Introduction of foresight in the region and complementarity to other regional actions. Are the defined objectives consistent with the needs of the environment? Are they defined in detail? Has foresight implementation design taken into account regional characteristics and problems? Is foresight integrated and complemented by other types of regional actions, with which it shares common objectives? 2. Target hits Has the project fulfilled its general objectives? Were results as expected? (1) The questions are a result of the collective work of the partners and of the conclusions propoused in the last up skilling workshop Evaluating applications (IV) 3. Foresight as a useful tool Has foresight proved its use as a tool for partners and regional organizations? What type of method was used? Were the methods used appropriate for the targets set? To what extent was the foresight guide useful in the development of applications? 4. Foresight as a participatory process What is the role of private and public agents in the development of the applications? Was civil society involved? Was the foresight practice carried out as an open, participatory process? 5. The role of experts, stakeholders and external consultants What role did foresight experts, stakeholders and external consultants play in the performance of the project and applications? Evaluating applications (V) 6. Application of results Have the results of the foresight applications been applied in the regions? 7. Diffusion of regional foresight results Have other European regions been informed of the programme results? Do regional agents know about regional foresight applications and their results? 8. Extension and sustainability of the foresight culture In view of the foresight experience in the region, has the foresight culture developed in the region? Are regional organizations showing interest in maintaining and using this tool? Evaluating applications (VI) Indicators: • Indicators on Futures toolkit • Indicators on foresight methods Indicators (I) Indicators on Futures toolkit Number of foresight methods used Availability of data and sources. (Number and quality of sources). Duration of foresight exercise with regard to period planned. Cost of foresight exercise in comparison to originally estimated cost. Staff deployed in foresight process. Number of experiences in which it is being applied. Number of agents involved. Indicators (II) Indicators on foresight methods Scenarios Months in operation. Cost of application. Staff deployed and taking part. Number of environment-related variables. Number of scenarios considered and degree of difference between them. Trend analysis Available data. Time used. Cost of application. Staff deployed and taking part. Number of variables selected. Number of trend conditioning factors detected. Delphi Time used. Staff deployed and taking part. Number of experts. Level of knowledge in research field. Quantification of questions. Number of consultations. Degree of consensus Visionary Management Number of participants (working groups or key people). Communication tools used. Time used. Cost of application. Staff deployed and taking part. Horizon Scanning Number of studies identified. Number of studies selected. Available data. Number of trend keys detected. Time used. Cost of application. Staff deployed and taking part. Indicators (III) Indicators on foresight methods Panel of experts Selecting the experts: profile, qualification, etc. Range and variety of agents taking part (origin or sector to which they belong) Indicators on work done by panel of experts: meetings, working documentation, documents generated, deadlines reached, motivation of participants, etc. Time used. Cost of application. Staff deployed and taking part. Diffusion of panel’s results Level of consensus achieved Ideas generated Futures Workshop Indicators on the management of this method: cooperation between agents, transparency levels and fairness of process, number and length of workshops held, intervals between them, etc. Study horizon. Time used. Cost of application. Staff deployed and taking part. Ideas generated Indicators (IV) Other indicators AREA Toolkit INDICATORS Degree of complexity in handling Degree of satisfaction in handling Questionnaries Presented below are three kind of questionnaires for agents taking part in the Futurreg project: • Questionnaire for partners The questionnaire is designed to obtain information on how the working plan developed and the measures worked, on the information generated, what type of difficulties were founded, etc. • Questionnaire to application agents Sent to the agents of all regional applications. It will provide outstanding information on the management of applications, their design, how they worked, the methods used, the measures developed once foresight had concluded, the role of the experts, etc. • Questionnaire to participants in applications This questionnaire will only be sent to participants in regional applications selected as case studies. Chronogram Evaluation methodology: Presentation of Draft. 28 November 2006. Deadline for comments. End of December 2006. Methodology evaluation document. 15 of January. On-going evaluation: Evaluation criterion. 28 November 2006. Indicators. 28 November 2006. Evaluation of applications (selection of a specific number of applications). 30 November 2006. Questionnaire 28 November 2006. Data processing. 31 January 2006. Draft ongoing evaluation report. 31 May 2007. Ongoing evaluation report. 29 June 2007. Evaluation Ex-post evaluation: Evaluation criterion. 31 August 2007. Indicators. 31 August 2007. Evaluation of applications (selection of a specific number of applications). 28 September 2007. Questionnaire. 28 September 2007. Data processing. 31 October 2007. Draft ex post evaluation report. 30 November 2007. Ex post evaluation report. 28 December 2007.