Download Integrated Evaluation Framework

Francois van Dyk
AMEC Education Committee Member
Head: Operations | Ornico
@sbalie
THE
INTEGRATED
EVALUATION
FRAMEWORK
A journey through public relations measurement and evaluation
How much is a
tweet worth?
$150m
EDWARD LOUIS BERNAYS (1891 – 1995)
1949
“This practice has done more to undermine
public relations than any other.” - F. Murray Milne
International demand for AVEs
35%
31%
30%
30%
25%
22%
20%
15%
10%
5%
0%
2014
2015
2016
Public relations is the management, through communication, of
perceptions and strategic relationships between an organisation and its
internal and external stakeholders.
Why Are The Barcelona Principles
Important?
◦ Set overarching framework
◦ Alignment
◦ Basis for measurement and
evaluation programs
Principle 1
From:
Importance of Goal Setting
and Measurement
To:
Goal Setting and
Measurement are
Fundamental to
Communication and Public
Relations
What To Do:
◦ Conduct measurement and evaluation
against defined goals and SMART
◦ Make goals quantitative or qualitative, but
still identify who, what, how much, by when
◦ Be holistic: traditional and social media;
changes in awareness among key
stakeholders, comprehension, attitude, and
behavior; and impact on organizational
results. Campaigns or ongoing are both
relevant.
◦ Be integrated and aligned across paid,
earned, shared and owned channels where
possible
Principle 2
From:
Measuring the Effect
on Outcomes is
Preferred to
Measuring Outputs
What To Do:
◦ Tailor practices for measuring the effect on outcomes to the
objectives of the communication program
◦ Consider both quantitative and qualitative methods
◦ Apply standard best practices in target audience research
Awareness,
To:
Measuring
Communication
Outcomes is
Recommended
Versus Only
Measuring Outputs
Other shifts in
stakeholders
Comprehensio
n
Investment
decisions,
Attitude
Outcomes
Public policy
Behavior
Advocacy
related to
purchase
Employee
engagement
Corporate
reputation
Donations
Brand equity
Principle 3
From:
The Effect on Business Results
Can and Should Be Measured
Where Possible
To:
The Effect on
Organizational
Performance Can and
Should Be Measured
Where Possible
What To Do:
◦ To measure results from communication for an organization, models
that determine the effects of the quantity and quality of
communication outputs on organizational metrics, while accounting
for other variables, are a preferred choice
◦ Use models that determine the effects of the quantity and quality of
communication outputs on organizational metrics (e.g., Demand for
models to evaluate the impact on target audiences, survey research)
◦ Develop communication measures that can provide reliable input into
integrated marketing and communication models, including through
advanced econometrics and advanced survey analysis
Principle 4
From:
Media Measurement
Requires Quantity and
Quality
To:
Measurement and
Evaluation Require Both
Qualitative and
Quantitative Methods
What To Do:
◦ Consider qualitative methods to better explain the quantitative (or to
replace, in some cases)
◦ Media measurement, whether in traditional or online channels,
should account for:
◦ Impressions among the stakeholder or target audience
◦ Quality of the media coverage including, but not limited to:
◦ Tone
◦ Credibility and Relevance Message Delivery
◦ 3rd party or company spokesperson
◦ Prominence as Relevant to the Medium
◦ Remember that we are measuring results and progress, not
necessarily success
◦ Quality measures can be negative, positive, or neutral
Principle 5
What Not To Do:
◦ Do not use Advertising Value Equivalents (AVEs)
◦ Do not use multipliers for “pass-along values” for earned versus paid
media (unless proven to exist)
From:
AVEs are not the Value of
Public Relations
To:
AVEs are Not the Value of
Communication
What To Do:
◦ If you must make a comparison between the cost of space or time
from earned versus paid media, use:
◦ Negotiated advertising rates relevant to the client
◦ Quality of the coverage (see Principle 4), including negative results;
and
◦ Physical space or time of the coverage related to the portion of the
coverage that is relevant
Principle 6
From:
Social Media Can and Should
be Measured
To:
Social Media Can and
Should be Measured
Consistently with Other
Media Channels
What To Do:
◦ Define clear goals and outcomes for social media
◦ Include measurement methods such as:
◦ Media content analysis
◦ Web and search analytics
◦ Sales and CRM data
◦ Survey data
◦ Evaluate the quality and quantity of social media (just like with
conventional media)
◦ Focus measurement on engagement, “conversation” and
“communities,” not just “coverage” or vanity metrics such as “likes”
Principle 7
From:
Transparency and
Replicability are Paramount
to Sound Measurement
To:
Measurement and
Evaluation Should be
Transparent, Consistent
and Valid
What To Do:
◦ Ensure integrity, honesty, openness and ethical practices
◦ Use valid methods
◦ Quantitative = Reliable and replicable
◦ Qualitative = Trustworthy
◦ Consider other relevant standards, like:
◦ For Media Measurement:
◦ Source of the content along with criteria used for collection
◦ Analysis methodology
◦ For Primary Research:
◦ Methodology
◦ Verbatim questions
◦ Statistical methodology
◦ Recognize any potential biasing effects
◦ In the research itself, or
◦ Broader societal context
INTERNATIONAL ASSOCIATION FOR THE MEASUREMENT AND EVALUATION OF COMMUNICATION
Valid Metrics Framework
Valid Metrics Framework Template:
COMMUNICATIONS PHASES
COMMUNICATIONS/MARKETING STAGES
Key Area of
Communication
(Brand/Product Marketing,
Reputation Building, Issues
Advocacy/Support, Employee
Engagement, Investor Relations,
Crisis/Issues Management, Notfor-Profit, Social/Community
Engagement)
Awareness
Knowledge/
Understanding
Interest/
Consideration
Support/
Preference
Action
Public Relations
Activity
Intermediary Effect
Target Audience
Effect
ORGANIZATION/
BUSINESS
RESULTS
Awareness
Knowledge
Interest
Reputation
Building
Support/
Preference
Action
• Content creation
• Traditional media engagement
Public Relations
Activity
• Social media engagement
• Influencer engagement
• Stakeholder engagement
• Events/speeches
Intermediary
Effect
Target Audience
Effect
• Audience reach
[traditional & social
media]
• Impressions/Target
audience
impressions
• Number of articles
• Video views
• Frequency
• Prominence
• Share of voice
• Key message
alignment
• Accuracy of facts
• Key message
alignment
• Frequency of
(positive) mentions
• Expressed opinions
of interest
• Social network
Followers
• Retweets/Shares/
Linkbacks
• Endorsement by
journalists or
influencers
• Rankings on industry
lists
• Expressed opinions
of support or
preference
• Social network Fans
• Likes
• Unaided awareness
• Aided awareness
• Knowledge of
company profile
and offer
• Relevance of
company (to
stakeholder)
• Visitors to website
• Click-thru to site
• Time spent on site
• Downloads from site
• Calls
• Event/meeting
attendance
• Attitude change
• Uplift in reputation
drivers e.g. Trust,
Admiration
• Endorsement
• Belief in corporate
brand
• Links to site
• Enhanced
relationships with
key stakeholders
•
•
•
•
Sales
Market share
Share price
Talent retention and
recruitment
• Cost savings
• Customer loyalty
• Legislation/regulation
passed or blocked
THE
INTEGRATED
EVALUATION
FRAMEWORK
A journey through public relations measurement and evaluation
“AMEC’s Integrated Evaluation Framework…sets out a clear and easy to follow method for telling the story
of great PR from objectives to target audiences, to planning, targets, activities, outputs, outtakes outcomes
and business impact.
Only when the PR sector looks to measure the entirety of its effect will it be able to answer key questions
about its performance in meaningful and credible ways.”
http://amecorg.com/amecframework/interactive-framework.html
ALIGN OBJECTIVES
COMMUNICATIONS OBJECTIVES
1. Increase Revenue
2. Increase Shareholder Value
IMPLEMENT
MEASURE ACTIVITY
ACTIVITY
OUTPUTS
1. Competitive research
2. Define competitive
opportunities
3. Benchmark
4. Industry – data driven
survey
5. White paper
6. Infographics, video,
media releases
7. Distribution – media
8. Distribution – owned
9. Distribution – paid
amplification
TARGET AUDIENCE
1. Brand Awareness
2. Reputation (Drive Purchase Preference)
3. Reduce Sales Cycle
MEASUREMENT & INSIGHTS
IMPLEMENTATION
PREPARATION
ORGANIZATIONAL OBJECTIVES
PLAN, SET TARGETS & OTHER INPUTS
1.
2.
3.
4.
5.
6.
7.
White paper
Video
Media releases
Media briefings
Social media ads
SEO
Speaking events
STRATEGY
1. Enterprise C-suite
2. Enterprise Stakeholders – Marketing,
IT
3. Industry sector media and analysts
4. Financial press
1. Narrative Development and
execution
2. Thought leadership
3. Content Marketing
ORGANISATION & STAKEHOLDER EFFECTS
AUDIENCE RESPONCE & EFFECTS
OUT-TAKES
1. Awareness – narrative
traction (quality)
2. Website traffic (unique
visitors)
3. Message recall (survey)
4. Engagement with
collateral – likes, clickthrough’s, shares,
comments)
OUTCOMES
1. Lead volume increase (%)
2. Sales cycle time (%)
3. Customer preference
(survey)
4. Awareness – narrative
traction (quality)
IMPACT
1. Reputation improvement
through valuation narrative
traction
2. Target customer – C suite
engagement
3. 17% reduction in time to
close sale
4. Customer advocacy up 25%
THE
INTEGRATED
EVALUATION
FRAMEWORK
A journey through public relations measurement and evaluation
THE
INTEGRATED EVALUATION FRAMEWORK
• It provides structure – from goals to impact
• It connects an organisation’s communication to its
business objectives
• Measurement becomes part of the strategic process
A journey through public relations measurement and evaluation
THE
INTEGRATED EVALUATION FRAMEWORK
• You still need to do the hard work…
• Gather the data
• Analyse the data
A journey through public relations measurement and evaluation
#AMECMM
Francois van Dyk
Head: Operations | Ornico
AMEC Education Committee Member
@sbalie