Download Evaluating communication

Sigried Caspar
European Commission,
DG Employment & Social Affairs
Moderator
Céline Mas
Occurrence communication research &
consulting company, France
Workshop A7: how to evaluate
your communication activities
2014-2020
December 10, 2013 – Brussels
Céline Mas
Occurrence est
certifiée
ISO 9001
depuis 2004
"Half the money I spend on
advertising is wasted;
the trouble is I don’t know
which half."
John Wanamaker
Inventor of mass
retailing in the United
States
Occurrence - Workshop A7 - 10/12/2013
Key issue
> Communications are under pressure:
-
How much does it cost?
How much
does it contribute?
€
What
does it contribute?

> View it as an investment and not as an expense
> Provide the resources to prove your effectiveness: 5% to evaluation.
> "How much does overlooked inefficiency cost?“ : if you can not assess it, you can
not improve it!
Occurrence - Workshop A7 - 10/12/2013
5
The missing link in a virtuous circle
Communication plan
Most of the time,
evaluation is occasional
or partial.
A goals and resources « contract »
with institution or company management
It rarely shapes dialogue
between the
communications team
and the other decision
makers.
Assessment
Reporting on the achievement of the targets,
or on the progress and effectiveness
of the implementation
Occurrence - Workshop A7 - 10/12/2013
Actions
A goals and resources « contract »
with company management
Behaving like any other function
> Communications must not exclude
itself from Quality and Operating
Excellence systems
> Communications is a job and a skill;
it must include ongoing improvement
procedures
The Deming Wheel
Quality management system
Occurrence - Workshop A7 - 10/12/2013
7
The 4 main benefits of an evaluation
1. PROMOTING
2. MANAGING
• Circulating results and performances
• Allocating resources in accordance
to other departments/teams
with performance indicators
• Achieving investment choices that are
• Identifying the most effective initiatives
based on targets, and not on expenditures
on resources
for achieving your various objectives
4. SHARING
3. SAVING TIME
• Prioritizing/Sorting initiatives by order
• Gathering all the activity and
of effectiveness
• Concentrating your efforts and budget on the
most effective initiatives
•
effectiveness data
• Highlighting best practices
AND SPECIFIC TO THE PUBLIC
SECTOR
Giving evidence of a sound use of public
money
• Reinforce citizen’s trust
Occurrence - Workshop A7 - 10/12/2013
8
2 main categories of performance (KPI)
indicators
Activity
including
Resources
(What?)
Effectivenes
s
including the
Audiences
(For what
purpose?)
e.g. Number of initiatives/tools,
type of initiatives/tools, assessment of the content issued
(Press releases, and internal communications),
Ressources: Who? How much? How long?
e.g. Memorization, Understanding, Buy-in, Incentive, Transformation,
Satisfaction, Improving the brand's image, and satisfying internal customers
Audience: How many people attended? How many Likes ? How many
readers? …
> A third, highly practical approach is possible: assessing the satisfaction of
(internal) customers
> Defining performance thresholds for each indicator
Occurrence - Workshop A7 - 10/12/2013
9
Communications Dashboard communications plan
Process : global picture
Target audience
Communications objectives
Segmentation


1. Define
2. Count
3. Contact details
Opinions/perceptions to share
Do you agree with the following assertion ?
Surveys, research - "ABC respects environment”
media analysis, - …
Yes
observations
Indicators
-
x 1
[File qualification]
- "ABC respects environment”



60%
Current
score:
60 %
Occurrence - Workshop A7 - 10/12/2013

Targeted
score :
90 %
x N
[Sample survey]
Gap between results and objectives
!!
!
OK
Yes :
No : 
Don’t know :
Management
•
•
•
•
•
Allocate
Understand
Adjust
Maintain
Etc…
Advice
> Efficiency means producing the desired effects on the desired target audiences
> Therefore, you need to define the target audiences that you want to reach with
which effect, prior to the initiative, and ideally to define the performance threshold
> As a starting point: KISS (Keep It Simple, Stupid)
-
Keep things simple at the beginning
> BUT assess them regularly
> And don't change the assessment system for each evaluation
> Do not restrict the evaluation to the activity, in order not to limit communications to
initiatives and tools
> Design SMART (Specific, Measurable, Achievable, Realistic, and Timed) tools
and goals for each initiative
> Share the results and the decisions they help to take in order to enhance the value
added of evaluation
Occurrence - Workshop A7 - 10/12/2013
11
Thank you
> Contact : Céline Mas
> Partner & Research Director Occurrence
> [email protected]
Occurrence - Workshop A7 - 10/12/2013
Brussels, 9-10 December
Ben Ward
Head of Evaluation and Research,
Coffey International Development, UK
Instructions for use
• This template has been built with custom styles and formats aligned to the new brand identity.
• All charts, org charts, tables and smart graphics have the custom colour palette and font styles
embedded. Choose the style that best suits the presentation of your information.
• Evaluating
Inserting chartscommunication
as images and including photography will increase the file size – include lower
resolution files.
• The Coffey orange should only be used in headers, sub headers and bullets – avoid using it in
charts, tables and graphs.
Ben Ward
• December
Secondary colours
are only used in charts, tables and graphs
9th 2013
– avoid using as headers, sub
headers and bullets.
• Bolder tones are best used for charts – as they appear clearer on screen.
• Please do not change the primary colour set ups.
• Headlines on pages: if they are one line in length please ensure they are presented in
black. If the headline/title splits across two lines – please retain the first line in black and put
the second line in the Coffey Orange (select from custom colour palette).
IDEV Executive Workshop
4 November 2013
16
Evaluating communication activities
Contents
•
Who we are and what do we do?
•
Challenges faced by public sector communicators
•
Evaluation - hints and tips
•
Discussion
Telling the story conference
December 2013
17
Evaluating communication activities
Coffey Evaluation and Research
Example I&C assignments
• Founded in 1990, acquired by Coffey
International Development in 2009
• Evaluation on EU Regional Policy
communication (DG REGIO)
• Team of 30 international and multilingual
M&E professionals
• Work in EU, UK domestic, and international
development evaluation markets
Telling the story conference
December 2013
• Ex ante evaluation of campaign on violence
against women (DG JUSTICE)
• Evaluation of ESF communication activities
(DG EMPL)
• Evaluation of DG TRADE’s communication
strategy
18
Evaluating communication activities - Challenges
•
The communication challenge in the public sector is greater than for business communicators
Greater
resource
constraints
Extreme
media
scrutiny
Public
sector
comms
Legal
constraints
Political
agendas
(Adapted: Liu and Horsley, 2007)
Telling the story conference
December 2013
19
Evaluating communication activities - Challenges
Telling the story conference
December 2013
20
Evaluating communication activities – Hints and tips
1. Take an evidence based approach to communications design
•
Design of any communications campaign should be driven by good evidence
•
The evidence base:
o
Primary and secondary
o
Quantitative and qualitative
o
Research areas:
- Build deep knowledge around what you are communicating – e.g. the problem itself
- Defining target audience(s)
- Appropriate communication tools and activities
Telling the story conference
December 2013
21
Evaluating communication activities – Hints and tips
2. Evaluation should start at the very beginning
•
Set the objective(s): What are you trying to achieve?
•
Develop an intervention logic and SMART indicators
•
Mapping how you will get to your desired outcomes (use an evidence base to inform this)
•
Workshop - communications team and other relevant stakeholders
“He who fails to plan is planning to fail”.
Telling the story conference
December 2013
22
Evaluating communication activities – Hints and tips
3. Effective evaluation: Focus on measuring the outcomes
•
Measure as much as you can with the resources available
•
Outputs and outtakes both have their place (So what?)
•
However, achieving outcomes ultimately help to justify the use of communications
•
Producing a lot of material, such as leaflets and posters, and getting out the right messages is no
good if you fail to get the final result!!
Telling the story conference
December 2013
23
Evaluating communication activities – Hints and tips
4. A few final tips……..
•
Recognise what you can and you cant do with your evaluation (but also with your comms!)
•
•
Take a proportionate approach to evaluation
o
Size of intervention
o
Repeatability / Scalability factors
Use mixed methods (Quant and Qual)
Telling the story conference
December 2013
24
Evaluating communication activities
Thank you
[email protected]
Telling the story conference
December 2013
25
Brussels, 9-10 December