Download How to become Expert

How to become
Expert - Evaluator
in H2020
Marcela Groholova
Research Executive Agency
Brussels
25/2/2015
Bratislava, Slovak Republic
Evaluation - the most important step in the project
management cycle
The best proposals selected
=
The best proposals funded
Contents:
•Experts database
•Experts selection
•Conflict of interest
•Confidentiality
•Projects to evaluate
•Remote evaluation
•Central week
•Evaluation reports
•Reimbursement
•Good expert
Experts registration:
Via Participant Portal
https://ec.europa.eu/research/participants/p
ortal/desktop/en/experts/index.html
- Expert creates his/her ECAS login
Registration for
new experts
Update your
existing profile
Programme Selection
 Horizon 2020 (including Euratom programme)
 COSME (Competitiveness for Small and Medium-sized
Enterprises)
 Education, Audiovisual and Culture Executive Agency (EACEA)
programmes
 CEF (Connecting Europe Facility)
 RFCS (Research Fund for Coal and Steel)
 Third Health Programme (managed by Chafea)
Personal details – name, nationality, gender
(experts statistics)
Contact details – how to contact you and where to
send any documents
Education: Qualification, Field, Name of the
institution, Country, Year Awarded
Languages:
 Writing
 Reading
 Conversation
Area of expertise:

Specialized field – predefined from the menu

Free keywords
Professional experience:
 employer history
 Experience in the field
 Publications and achievements
 How long you worked in your specialist field
 Have you ever worked in industrial sector
 Have you assisted to EC in evaluation process? (evaluator, reviewer, NCP, etc.)
 Do you have any other experience in peer review process
Additional info:
upload your most recent CV
Experts selection:
•Expertise
•Nationality balance
•Gender (at least 40% female)
•Private – Public balance
•At least 25% of newcomers for each call
•Max 120 days in 4 consequent years (incl. Review and other services)
•Pre-selection of bigger pool for some actions (bottom-up calls), final
selection after call closure
Overview of the experts approval process:




Approval of the pool by EC (internal)
Valid Bank Account + Legal Entity form for each expert
Drafting the contract
Signing the contract – electronically by expert and by EC
 EVALUATION
 Payment session created
 Submission of payment documents by expert – electronically,
originals to be kept by expert up to 2 years in case of audit
 Payment within 30 days
 Days count in our internal system
Once you are selected for the call – BRIEFING (online or in
Brussels)
Conflict of interest:
= Reliability of expert!
•1st time – when accepting participation and signing contract
•2nd time – when list of participants is available (some calls)
•3rd time – when full proposal is received in SEP
•Expert can declare CoI at any moment during evaluation!
Note: It is very important to review in detail all the proposals
allocated to you so that you are able to promptly identify any
potential conflict of interest and decline the respective task(s).
In case of CoI, please immediately inform your respective PO in EC/agency
and/or decline task in SEP
Confidentiality of evaluation:
Expert must follow strict confidentiality principles with
respect to whole evaluation process.
Under no circumstance expert may contact applicants, send
any document produced or received during evaluation to the
third person or to inform the third person about evaluation
result.
Projects to evaluate:
• Expertise – Specialist vs Generalist
• Number of projects can vary – from 1 (specific top-down
calls) to 20+ (bottom-up calls)
• private – public balance for some calls
• Number of pages – from 10 to 100 – page limit nearly in
all calls
• Complexity of projects – based on the calls
Evaluation steps:
1. REMOTE
2. CENTRAL
3. PANEL MEETING
3-5 experts assigned to each proposal (average)
Remote evaluation:
From home or work place
Can take several weeks (normally 2-3 weeks)
Each expert evaluate all proposals allocated to him/her
Submission of IER (Individual Evaluation Report) electronically
Experts don't know other co-experts evaluating the same
proposals
• Evaluation based on own experience in the field and your best
knowledge about the topic
• Calibration of marks when more proposals to be evaluated
• End of Remote phase:
• All IERs must be submitted in the system
• Rapporteurs are allocated to each proposal – preparation for
central week
•
•
•
•
•
Central evaluation:
• In Brussels – all experts who act as Rapporteurs or remotely
(some cases)
• 1st time all experts meet around one table
• Day 1 – experts get instructions + their individual schedule for
each consensus meeting (room number, date and timing)
• Each proposal can be discussed between 30 minutes and 2 hours
depends on complexity
Consensus must be reached!
• If needed, another evaluator may be asked to step in (exceptional
cases – CoI during central week, not possible to reach consensus,
etc.)
• Rapporteur - one of the experts
• Moderator – either Rapporteur or EC official
• End of the meeting – rapporteur drafts CR (consensus report)
• All experts must approve CR electronically
Panel meeting:
• End of the central week (last day)
• All rapporteurs must approve Ranking list
• Ranking list automatically created by the system based on
the scores in CR
• Ex-aequo cases – Work Programme must be respected
• If possible: rapporteurs present briefly most successful
proposals ranked as first in each panel ranking
• Free discussion on evaluation process – suggestions,
comments
How to prepare evaluation report:
Example from MSCA call
Drafting evaluation report:
- Comments for each criterion and each sub-criterion
(some calls)
OR
- Strengths and Weaknesses for each criterion and subcriterion (some calls)
H2020 - 3 main criteria:
Excellence, Impact and Implementation
Drafting evaluation report:
1. Comment on each element listed underneath the main criteria;
e.g. under "Excellence" for ITN call you should evaluate
following sub-criteria:
• Quality and innovative aspects of the research
programme
• Quality of the training programme
• Quality of the supervision
2. Then assign an overall score for each criterion in line with
your comments.
During Consensus meeting:
• All experts should agree on the strengths and weaknesses /
comments.
• Experts agree on an overall score for each criterion in line with
the strengths and weaknesses / comments.
If requested, for the IER and CR, you may provide overall
comments. These should specify the relative importance of the
strengths and weaknesses.
For example:
 ‘This proposal is very good overall on this criterion.
Its strengths are A, B and C. Its shortcoming is D.’
Score = 4.0
Scoring:
•
Score each criterion from 0 up 5 points. Decimals or half-points available.
•
Some criteria may have a threshold (3 or 4).
•
Maximum score:
• 15 for some CSA and collaborative projects
• 100 for MSCA (weighted score, calculated automatically)
•
Minimum threshold for the overall score of the proposal:
• 10 for CSA and collaborative projects
• 70 for MSCA
•
The total score calculated automatically.
Scoring:
Please remember to use the full scoring scale
0 – Proposal fails to address or cannot be assessed due to missing or
incomplete information
1 – Poor. The criterion is inadequately addressed or there are serious
weaknesses
2 – Fair. Proposal broadly addresses the criterion, but there are significant
weaknesses
3 – Good. The proposal addresses the criterion well, but number of
shortcomings are present
4 – Very good. The proposal addresses the criterion very well, but small
number of shortcomings are present
5 – Excellent. Proposal successfully addresses all relevant aspects of the
criterion, any shortcomings are minor.
Example of comments
in the evaluation report:
Overall comments must be:
• Specific to the relevant criterion – you should not include
information e.g. on the quality of the project management
under "excellence" when there is a specific section on this
under "implementation"
• Clear and substantial
• Your comments should state facts, not opinions, and be
precise and final:  "We think that…"  "This proposal is… "
• The score must be a fair reflection of the comments,
balancing strengths and weaknesses.
• Comments should be of adequate length: you should not
right just one sentence, but neither should you write a whole
booklet.
• Comments judge the proposal, they do not summarise it.
you do not need to repeat what the applicants have written in
the proposal in your report.
• You are evaluating the quality of the proposal only. So you
should not provide in your report advice on improving the
proposal.
Poor comments merely echo the score
For example,
“The innovative aspects of the research
programme are poor”.
Good comments explain it
For example,
“This proposal is not innovative in X or
Y, and it does not take Z into account”.
Poor comments are ambiguous
“The resources for the project are
unrealistic”
Good comments are clear
“The resources in WP 4 and 6
are seriously underestimated
given the complexity of the
activity proposed”.
Poor comments are vague, subject to
interpretation
“We think the management plan is probably
inadequate given the duration of the project and the
number of partners”.
Good comments are precise and final
“The management plan is inadequate. It does not
include clear overall responsibility for the training
activities; it lacks a problem-solving mechanism in the
event of disputes between partners”.
Poor comments are inaccurate and provide an opening for a
complaint
“There is no discussion of a dissemination strategy.”
“There is only one SME partner in the consortium.”
When there were actually 2.
“The coordinator is not adequately experienced."
Good comments close the question
“Dissemination activities are listed but the proposal lacks a clear
dissemination strategy”.
“The consortium lacks sufficient SME participation.”
“The coordinator does not demonstrate in the proposal an adequate
level of experience in this field.”
Poor comments
include words like…
Good comments
include words like…
Perhaps
Think
Seems
Assume
Probably
Because
Percent
Specifically
For example
For
weaknesses, you might use
Insufficient, very generic, not evident,
unfocused, limited, unclear, no significant
impact, overestimated …
For strengths
Extremely relevant, credible, comprehensive,
high quality, highly effective, well-formulated,
well balanced, clear advances, …
It is essential that your scores are consistent with comments
Extract from an evaluation report from a previous evaluation:
Strengths:
• The S&T objectives of the research project are well presented and
clearly structured.
• The partners have complementary expertise.
• The joint collaborative research programme is of good quality.
• The project is original and the state of the art is adequately
presented.
However, Weaknesses:
• The methodology for the project is not fully convincing.
• The data collection strategy, potential sources of information and
data accessibility are unclear.
• The proposals fails to sufficiently demonstrate that the consortium
has the necessary expertise and capabilities to obtain the necessary
information needed for the project.
And this project on this criterion was given a score = 4.5
(between very good and excellent)
A score which does not adequately reflect the weaknesses
described.
Some more examples of inconsistencies:
Weaknesses of the proposal:
•The proposal is not specific enough to clearly show its innovative
and original aspects.
•The contribution of the private sector is not described clearly.
•The connection between the basic research topics and the 3
translational projects is not well presented.
This achieved a score = 4.4
(between very good and excellent)
Final check before finalizing CR:
 Are scores consistent with comments?




Comments of adequate length?
Are all sub-criteria addressed?
Operational capacity confirmed?
No factual errors! (WP7 is not adequately described...) in fact there
where only 5 WPs in the proposal!
 No discriminatory comments! (Partner 3 is not capable to fulfil
tasks from WP3 and 4)
Reimbursement:
• Online via electronic system, no paper documents to be sent
anymore (originals kept by expert for audit)
• End of evaluation experts submit all costs via Participant Portal
• Lump sum of 450 Euro per working day calculated to the
nearest half day
• A daily allowance of 92 Euro or 46 Euro if your point of
departure is less that 100 km from Brussels
• An accommodation allowance of 100 Euro, if applicable
• Full reimbursement of your travel expenses
• Taxi + parking fee is not reimbursed
Good expert:
•
•
•
•
•
•
•
•
•
Timely in delivering reports and during consensus meetings
In case of problem informs ON TIME
Reliable
Flexible
Able and willing to learn and accept our process
Written and spoken English of good quality
Able to judge and evaluate the project, not to copy/paste
If comments are negative – WHY?
If comments are positive – WHY?
We do not look for expert-superman, everybody can learn it...
Good luck 