Download The extended version (opens new window)

Evaluation for Alcohol Social Marketing
Evaluation for Alcohol Social Marketing
Guidance for PCTs
Prepared for:
Department of Health
Prepared by:
James Farnham
Hilary Burkitt
COI
Hercules House
Hercules Road
London SE1 7DU
E: [email protected]
Version No: 2
4 January 2010
1
Evaluation for Alcohol Social Marketing
Introduction
A formal definition for evaluation
For the purposes of this toolkit, the following definition of evaluation is
adopted:
Evaluation is a process which attempts to determine as systematically and
objectively as possible the relevance, effectiveness, efficiency and impact of
activities in the light of specified objectives. It is a process for improving both
current activities and future planning, programming and decision-making.
(Based on CDC’s Framework for program evaluation in public health.)
Increasing importance of evaluation
There is growing recognition of the role social marketing can play in
influencing consumer health behaviours, but these need to be demonstrated
to be working to justify continued investment.
Better social marketing evaluation can drive us on a journey of continuous
learning in the complex world of influencing health behaviours. We need to
aspire to a continuous improvement process, which enables us to learn
quickly what is working and ramp up investment behind those social
marketing successes as we refine them further.
This way of thinking requires a new approach to evaluation at PCT level.
Rather than evaluation being something we occasionally do well, we must aim
to make sure every significant piece of activity is evaluated at a proportionate
level, ensuring we balance sensibly the need for a rigorous evidence-based
approach against the need for pragmatic decision making, often in short leadtimes and with limited budgets.
Benefits of evaluation
There is always a temptation that when budgets and other resources are
restricted that evaluation is the first casualty in the process, therefore
compromising future social marketing effectiveness. Consequently, social
marketers increasingly need to be in a position to justify the importance of
evaluation, which can be helped by frequently restating the benefits of
effective evaluation:
 Developing interventions that are more effective in influencing
consumer behaviours
 Improved efficiency by investing in the things that work best
 More informed budgeting processes with reduced overspends and
underspends
 Better forecasting of outcomes
 Better management of expectations regarding results
2
Evaluation for Alcohol Social Marketing


Additional source of consumer knowledge and insight from the
evaluation process itself
Enhanced credibility of people practising social marketing
This document is laid out in four parts:
 Part 1 - Ten principles of evaluation
 Part 2 - A four-step process for alcohol social marketing evaluation
 Part 3 - Evaluating social marketing with limited budgets
 Part 4 - Using secondary data for alcohol social marketing evaluation
3
Evaluation for Alcohol Social Marketing
Part 1 – Ten principles of evaluation
Introduction
Many of the main elements of social marketing, such as segmentation,
insight, and behaviour change theory, have an extensive body of academic
knowledge behind them, supported by numerous textbooks and other learning
resources describing best practice. The subject of marketing evaluation is
different however, with relatively little set out as accepted best practice.
Accordingly, in 2008 DH completed an extensive piece of research with a
range of academic experts and practitioners and drew together ten principles
of evaluation which provide a useful starting point for designing evaluation
approaches for social marketing interventions. These principles are
summarised below.
Principle 1: Create a Test-Learn-Refine culture
Current public sector marketing evaluation is often viewed in static, sequential
terms: we scope an activity and then develop, implement and evaluate it
retrospectively. This essentially linear perspective on social marketing means
we ‘look in the rear view mirror’ to evaluate the past before then planning future
activities:
Ideally, we should ensure all our social marketing becomes a much more
dynamic cycle of continuous improvement, with constant learning being fed
into improved social marketing effectiveness:
Here the focus is much more about active learning in ‘real-time’ and being able to
respond rapidly to new learning and apply it immediately to improving the existing
marketing programme and future ones as well.
Social marketers need to help create business cultures that value evaluation
more, creating a mindset that measures social marketing success partly in
terms of whether metrics are in place to determine if activities worked and
achieved their objectives.
Evaluation should become an embedded function in the social marketing
process, its thinking and practice hard wired into every stage of development.
4
Evaluation for Alcohol Social Marketing
Principle 2: Proportionate evaluation budget allocation
A question that is frequently asked is what percentage of the overall social
marketing budget should be allocated to its evaluation? Part of the answer
lies in a proportionate view on the cost of the evaluation in relation to the
benefits expected to flow from any resulting improved decisions. There
seems to be wide consensus that in the region of 5-10% of the total social
marketing budget should be allocated to evaluation.
The factors to consider in deciding precisely how much of the social marketing
budget is allocated to evaluation include the following:
 The complexity of the social marketing programme – an integrated
social marketing plan using many different channels targeted at
many different audiences may require more evaluation work than a
programme involving one or two channels and a more
homogeneous audience
 The level of rigour required – to prove causality for social
marketing effects either controlled tests or an econometric model
may be required, which tend to be more expensive than less
rigorous evaluation approaches
 The accumulated learning to date – a long running social
marketing programme often has a substantial amount of
accumulated learning which may require less evaluation spend in
the latest year compared to earlier years. Conversely, new
marketing initiatives may require relatively high evaluation spend for
the new learning curve.
In setting the evaluation budget, it is also important to consider the amount of
data analysis that will be required. For example, when large quantitative
surveys are commissioned, the resource required to translate the data into
actionable information is often underestimated.
Principle 3: Starting the evaluation process early
Social marketing evaluation is frequently planned too late – sometimes after
the social marketing briefs have been written and issued to agencies. For two
key reasons, it is vital to start planning the evaluation much earlier in the
social marketing process, preferably right at the outset when the potential role
for social marketing is initially identified.
First, early evaluation planning can act as a catalyst for broader strategic
thinking about the social marketing project as a whole, forcing key questions
to be asked, for example the nature of the target audience and the
behavioural influence we are seeking.
Second, thinking about evaluation when social marketing is first being
considered also means there will be sufficient time available to establish
appropriate metrics before the activity breaks, enabling Key Performance
Indicators (KPIs) to be measured objectively before, during and after the
marketing programme.
5
Evaluation for Alcohol Social Marketing
The following is a useful checklist to consider right at the start of every social
marketing programme:
 What will be evaluated? (i.e. what is the social marketing
activity about and in what Policy context does it exist?)
 What aspects of the social marketing activity will be
considered when judging its performance?
 What standards (i.e. type or level of performance) must be
reached for the social marketing activity to be considered
successful?
 What evidence will be used to indicate how the social
marketing activity has performed?
 What relevant lessons from previous social marketing
activities can be applied?
 What conclusions regarding social marketing activity
performance are justified by comparing the available
evidence to the selected standards?
 How will the lessons learned from the evaluation be used to
improve public health effectiveness?
Principle 4: Clarity of social marketing objectives
Effective evaluation requires all those who have a stake in the social
marketing activity to agree and define very precisely the social marketing
objectives for the activity. This is vital because the objectives of the activity
should fundamentally define what is measured in the evaluation, and
therefore what the KPIs should be for the intervention. It stands to reason
that if it is concluded as part of the social marketing process that the main
objective of the project is to get local licencees to promote the sale of soft
drinks over alcohol, then the sales figures would be what should be
measured. Sometimes the social marketing objectives are softer than things
like sales and availability – for example if the aim is to increase the motivation
to cut down alcohol consumption, then measures need to be found for
measuring these levels of motivation, via survey questions for example.
Principle 5: Build behavioural change hypotheses to drive the
evaluation thinking
If we are doing our alcohol social marketing properly, we should be placing
behavioural change at the heart of our thinking, gathering a rounded picture of
the ‘problem’ behaviour and also the ‘desired’ behaviour, both expressed as
specific actionable and measurable behavioural goals and key indicators. We
should also be using behavioural theory to inform and guide the development
of our social marketing interventions, creating theoretical assumptions for how
our social marketing is designed to change behaviour and designing
approaches for testing these hypotheses as part of the process.
This means that our evaluation thinking and design needs to follow our
behavioural change thinking. For example, if we believe from the insights we
have gathered that a broadly linear behavioural model is valid, then our
evaluation should be designed to test whether behaviour changes in the
sequential way we have assumed. So in alcohol social marketing, we might
6
Evaluation for Alcohol Social Marketing
assume that a precursor to behaviour change is getting people to be aware of
safe drinking levels, before finding suitable ‘exchange’ messages that make
them want to cut down consumption, before then pointing them to services
that help them to cut down. In this simple example, the linear behaviour
change hypothesis defines the main elements we want to include in the
evaluation design, namely measures for awareness of safe drinking levels,
measures of intention to change and finally measures of usage of services
provided.
Principle 6: Evaluation measures to include a ‘line of sight’ to
behaviour change
Sometimes in our alcohol social marketing, the desired outcome measure
seems very distant and remote, because so many things need to change
before alcohol related hospital admissions are reduced. We can visualise this
as if the final impact we want with consumers is beyond our immediate sight,
almost as if it was the other side of a large hill before us:
However, if we have followed Principles 4 and 5 above, our clarity of
objectives and sequential behavioural thinking should enable us to construct a
‘line of sight’ to the final outcome measure, by listing all the building blocks
that need to be put in place before our desired consumer impact is achieved.
7
Evaluation for Alcohol Social Marketing
In this example, the first building block for our evaluation consists of our
inputs - measuring our social marketing ‘collateral’ or all the initiatives we put
into the marketplace, eg leaflets, websites, IBAs, Customer Relationship
Management programmes etc.
We then need to measure our outputs - the things that result from social
marketing inputs, such as awareness, measures of response to a call to
action, or conversion rates from receipt of input (leaflet, telesales call) to
desired action (eg application of information pack, take up of an IBA). In
summary, outputs represent what people actually do differently as a result of
the marketing activity, usually in terms of seeking out information, advice and
guidance.
Outtake measures then evaluate any new knowledge or understanding
among the target audience(s) that result from the marketing activity, either as
a direct result, or indirectly, via word-of-mouth/ ‘buzz’.
Intermediate measures can be visualised as stepping stones to the intended
behavioural effects and therefore may not be mutually exclusive from outtake
measures or indeed output measures. The value of intermediate measures is
largely diagnostic or as ‘leading indicators’ of expected business success.
Another example would be changes in behavioural intention.
Behavioural measures track actual (or claimed) changes to habits and
behaviour. In that sense they are ‘hard measures’ as opposed to the ‘softer
measures’ that are typically picked up in the intermediate measures above.
Ultimately, success or failure of social marketing activities should be judged
using hard behavioural measures. For our purposes, behavioural influence is
that which is attributable to social marketing as opposed to other factors.
Outcome and impact can be thought of as ‘downstream’ from behaviour
change in that they reflect the sum total of all the policy interventions of which
marketing is but one. Usually these will be the PSAs such as alcohol related
hospital admissions, reflecting the long term picture.
Principle 7: Using a range of measures
Some social marketers strive for a “metric of metrics”, believing that there is a
single, high-level measure, which if achieved, means that every other relevant
measure is likely to have also been achieved. For commercial marketing this
may well be the case, where measures such as market share and awareness
are often paramount. For the complexities of most social marketing, however,
it seems preferable to have a range of measures to span the elements
covered in Principle 6 above.
Principle 8: Adopt a consistent and rigorous approach to
evaluation
Most social marketing seeks to address complex behaviours that change only
gradually over time, making it vital that evaluation approaches are rigorous
and consistent. So if we have tracking studies that ask consumers questions
about their awareness of the health harms resulting from alcohol for example,
8
Evaluation for Alcohol Social Marketing
or the social acceptability of parents giving children alcohol, or indeed claimed
consumption levels, then these need to be consistently phrased, or small
changes between each data-point might be attributable to the way the
question is asked rather than any reflection of reality.
Consideration always needs to be given to how the most robust data can be
obtained, for example via longitudinal studies that track the attitudes, beliefs
and behaviours of a cohort of consumers over time. Test and control
approaches should also always be considered due to the robustness they can
provide in attributing cause and effect to social marketing interventions,
although consideration has to be given to ethical issues in deliberately not
providing social marketing interventions in the control areas.
Principle 9: Maximise evaluation learning from ‘negative’ sources
It stands to reason that if we are adopting a Test-Learn-Refine approach
described in Principle 1 above, then there is much to be learned from finding
out why people are rejecting social marketing elements or lapsing from the
social marketing activities to which they have responded. We can learn a lot
from asking someone who read a leaflet, but did not respond to its call to
action. Equally, if people are not converting to new behaviours at particular
stages of a customer relationship programme, it is useful to know why, so that
the social marketing can be refined accordingly.
Similarly, much useful information from customer feedback mechanisms and
complaints is ignored, when it can be one of the richest sources of insight.
With today’s lifestyles, it takes extreme circumstances to occur before many
consumers feed back their views (eg due to time shortage, apathy, lethargy, a
desire to avoid conflict etc), so it is reasonable to assume that every complaint
received might actually represent the views of thousands of consumers who
feel the same way, but have not found the time to register their dissatisfaction.
Often there are also people who change their behaviour in the way we want,
but without any exposure to our social marketing interventions – another
rich source of information for our evaluation.
Rejectors, lapsers, non-converters, complainers and people who change
behaviours on their own are all valuable ‘negative’ sources of information that
should be included in our evaluation. This is particularly the case when
evaluation budgets are very constrained as they can be used as cost-effective
proxy measures.
Principle 10: Question activities that cannot be evaluated!
Although this principle sounds glib, it remains important for social marketers to
defend the importance of evaluation when reasons are suggested that it is a
luxury rather than an essential part of effective social marketing. It can be
argued that it is precisely when budgets are becoming more constrained that
evaluation becomes more important, to prove the relative effectiveness of
social marketing compared to other forms of investment.
9
Evaluation for Alcohol Social Marketing
10
Evaluation for Alcohol Social Marketing
Part 2 - A four-step process for alcohol
social marketing evaluation
Introduction
Having laid out the ten evaluation principles above, this section describes a
four step process for applying them over the life of a social marketing project,
in four sections: Scope, Develop, Implement and Learn.
The Scope phase sets up the evaluation process, establishing its aims and
governance.
The Develop phase involves working with social marketing strategy
specialists to distil all the high level thinking for the evaluation. This is
captured in an Evaluation Plan that articulates the hypothetical behavioural
model for the marketing programme and the KPIs to be measured.
The Implement phase involves the detailed work on defining what is to be
measured and how, working iteratively with implementation specialists, who
need to consider how their marketing activity strands will be measured in
parallel with their creative development. This phase also involves working
with research and other experts in detailed design of the evaluation
measurements and the commissioning of appropriate surveys and data.
The Learn phase concludes what worked well and less well with the
marketing activity, distilling the learning into an action oriented Evaluation
Report designed to fuel the Test-Learn-Refine process.
Each of the above phases examines evaluation approaches to be adopted for
larger PCT-level marketing programmes. However, the steps covered are
equally applicable to the evaluation of much smaller marketing activities,
which are discussed in Section 3.
11
Evaluation for Alcohol Social Marketing
PHASE 1: SCOPE
Introduction
The scoping phase of social marketing evaluation is all about gaining early
clarity on the broad approach and shape for the evaluation and getting buy-in
from the key stakeholders who have an interest in its conclusions. Five steps
are suggested for this phase as follows:
12
Evaluation for Alcohol Social Marketing
Step 1 Form an Evaluation Team
It is recommended that an Evaluation Team be established at the earliest
opportunity, when the idea of running social marketing activity first surfaces.
The Evaluation Team is different from the social marketing team, involving
certainly social marketing team members, but also drawing in a range of
different expertise from other disciplines.
Role of the Evaluation Team
The role of the Evaluation Team is to be the focal point for decisions on all
aspects of the social marketing evaluation. The Evaluation Team needs to
champion a best practice approach, ensuring marketing objectives are
articulated sufficiently precisely to design the evaluation programme and also
be responsible for ensuring a robust evaluation methodology is designed and
delivered.
CHECKLIST
Terms of Reference for the Evaluation Team
Ensure clear objectives are set for the social marketing activity and the evaluation
itself
Ensure that systems are in place to measure the social marketing programme’s
effectiveness, via an agreed set of KPIs
Set and manage the evaluation budget
Fully involve the social marketing, research and creative agencies in the process of
developing the evaluation approach, along with other relevant experts
Complete the Evaluation Plan (see Phase 2, Step 4)
Complete the Evaluation Report (see Phase 4, Step 3)
Ensure the learnings from the evaluation are captured and disseminated as part of
the Test-Learn-Refine process
Engage all stakeholders and manage their expectations throughout
Team members
Careful consideration needs to be given to who will lead the Evaluation Team.
There are strong arguments for and against the social marketing lead taking
overall responsibility. If the main purpose of the evaluation is to benefit from
continuous learning and marketing improvement, then it is logical the social
marketing lead champions the process. In cases where the primary focus of
the evaluation is to measure accountability, then the social marketing lead
might have a conflict of interest in leading the evaluation.
It is important that a wide cross-section of expertise is included in the
Evaluation Team. There are no hard and fast rules, but the following should
certainly be considered:
CHECKLIST
Evaluation Team members to be considered by the Social Marketing Lead
PCT colleagues
13
Evaluation for Alcohol Social Marketing
Research experts conversant with both qualitative and quantitative approaches
and benchmark issues
Communications planning experts
Health professionals
Other delivery partners (eg NGOs, local delivery PCTs)
Health Observatories
Appropriate academics, specialising in the relevant behaviour change/influence
issues
Data analysts, where complex modelling is envisaged
SHA/PCT representatives
In some cases, members of the team may not be required throughout the
evaluation process and can be part-time or members of an extended ‘virtual
team’. For example, clinicians and GPs may not necessarily be required for
all evaluation team activities, but can provide an occasional sounding board
for changes to evaluation methodologies and approaches. The Public Health
Observatories often have people who have specialist experience that might
be a useful adjunct to the evaluation team. Academics, particularly those with
expertise and knowledge of the relevant behaviour change models or theories
of behaviour influence can also have an important role to play.
Other Evaluation Team members to be included will depend on the nature of
the evaluation itself. For example, if an econometric model is to be built to
monitor marketing and other effects, it may be advisable to include a data
analyst on the evaluation team. With large, long-running marketing
programmes, it may even be appropriate to include members of the
econometric team.
It is important to note that while a wide variety of team members will make an
expert contribution to the evaluation, the responsibility for maintaining a social
marketing-led consumer focused approach to the process must rest with the
social marketing lead. So while it is vital to include academic and research
expertise on the team, these methodologies should not be allowed to shift
evaluation thinking and design away from the required focus on our pragmatic
Test-Learn-Refine approach and the continuous marketing improvement we
wish to achieve.
The important early questions for the Evaluation Team
Many of the detailed aspects of what needs to be covered in the social
marketing evaluation process are considered in the sections that follow, but it
is useful if from ‘day one’ the Evaluation Team starts to think about the ‘big
questions’ that the evaluation will need to address:
CHECKLIST
‘Big questions’ that social marketing evaluation needs to address
What will be evaluated? (i.e. what is the social marketing activity about and in what
Policy context does it exist?)
What aspects of the social marketing activity will be considered when judging its
performance?
14
Evaluation for Alcohol Social Marketing
What standards (i.e. type or level of performance) must be reached for the social
marketing activity to be considered successful?
What evidence will be used to indicate how the social marketing activity has
performed?
What relevant lessons from previous social marketing activities can be applied?
What conclusions regarding social marketing activity performance are justified by
comparing the available evidence to the selected standards?
How will the lessons learned from the evaluation be used to improve public health
effectiveness and efficiency?
These are the questions that should be occupying the minds of the Evaluation
Team from the outset and will be constantly revisited in more detail during the
subsequent steps in the evaluation process.
Often the very process of asking these fundamental questions can lead to a
more detailed discussion with PCT colleagues about the social marketing
objectives. This is covered in Step 3 below.
Step 2 – Lessons learned from previous evaluations
In Phase 4, Step 3 below, the process is described for producing an
Evaluation Report at the end of each social marketing programme, focusing
on specific learnings that can help refine subsequent marketing activities.
The intention is to develop a knowledge management system that captures
the learnings from all future PCT alcohol social marketing evaluations and
shares them via the Alcohol Learning Centre (ALC). In the meantime, while
ALC builds, Evaluation Teams need to make best use of whatever evaluation
learnings are available from previous relevant alcohol social marketing
activities.
Step 3 - Clarifying objectives
One of the first jobs for the Evaluation Team is to gain absolute clarity with
PCT colleagues on the role of social marketing in influencing behaviour within
the mix of other ‘policy levers’ such as enforcement and legislation change.
Specifically, the following high-level principles need to be agreed with the
relevant PCT colleagues:
CHECKLIST
Four areas that should be agreed with PCT colleagues at the outset of
planning the evaluation
Agreeing the role of social marketing in achieving PCT objectives
The specific issues and behaviours that social marketing must address
The specific target audiences, in priority order
The number of people we want to achieve different outcomes, expressed as
SMART1 objectives
1
Specific, Measurable, Achievable, Relevant, Time-bound
15
Evaluation for Alcohol Social Marketing
Sometimes it can be helpful to write a short position paper clearly articulating
the social marketing task, with a brief rationale. Setting the thinking down and
sharing it widely with relevant colleagues can help cement understanding of
the limits of social marketing’s influence. The aim must always be to gain
active support and consensus for what the social marketing programme is
aiming to achieve. If justice is done to the collective quality of thinking at this
stage, it is likely to make all the downstream evaluation tasks much easier.
Another important strand for the Evaluation Team to be clear about, even
before a social marketing programme is briefed, is the objective of the
evaluation itself. It stands to reason that an evaluation that is primarily setting
out to measure the social marketing programme’s Return on Investment
(ROI) will be designed differently to a social marketing programme that is
primarily setting out to improve continuous learning. Below are listed several
different objectives that might be usefully prioritised when planning the
evaluation.
CHECKLIST
Possible objectives for the evaluation itself
Achieve continuous learning to improve future marketing effectiveness in
influencing behaviour
Achieve continuous learning to improve future social marketing efficiency in
influencing behaviour
Provide additional source of consumer knowledge and insight for the social
marketing team generally
Demonstrate accountability and justify future social marketing investment
Improve our budgeting process by better knowledge of how activities need to be
phased
Provide more accurate forecasts of social marketing results and therefore improve
management of expectations regarding what social marketing might deliver
Enhance the credibility of the social marketing function
Step 4 - Setting the evaluation budget
Next the evaluation budget needs to be set, probably within a range of 5-10%
of the total social marketing budget, depending on the complexity of the social
marketing activity, the level of rigour required in the evaluation and the degree
to which similar social marketing activities have been previously evaluated
(see Principle 2 above).
Step 5 – Evaluation Plan (1)
Completing steps 1-4 above will enable a first-cut Evaluation Plan to be
produced. The purpose of this is to engage all relevant evaluation
stakeholders with the emerging direction of travel for the evaluation, gaining
16
Evaluation for Alcohol Social Marketing
early consensus and buy-in to the approach, so that misunderstandings can
be avoided later on.
The suggested elements for the first-cut Evaluation Plan are included below.
CHECKLIST
Suggested headings for Evaluation Plan (1)
Evaluation team members with contact details
Areas of principle: statement of the social marketing objectives, specifically
highlighting the role marketing is expected to play in influencing behaviour versus
other policy levers such as legislation for example
Review of learning from previous evaluations
Evaluation objectives
Evaluation budget
17
Evaluation for Alcohol Social Marketing
PHASE 2: DEVELOP
Introduction
The Develop phase of the evaluation process focuses on developing the
overall evaluation thinking and comprises five phases. Where social
marketing implementation agencies are involved, they should be viewed as
important allies and co-producers of this phase:
Step 1 – Using behavioural theory
A central tenet of social marketing best practice is to place behavioural
influence, the ultimate objective of all our social marketing, at the heart of our
evaluation thinking.
In health social marketing we usually want to take consumers on a
behavioural journey from where they are now in terms of awareness,
attitudes, beliefs and behaviour to a destination where some or all of these
elements have been changed favourably according to our social marketing
objectives.
Fortunately, we don’t always have to construct behavioural models from
scratch for each health marketing programme we develop. The social
marketing agency and any academics available to the Evaluation Team, will
usually have views on relevant behaviour change models for us to use.
Additionally, there are other sources that can help start the thinking process
for developing relevant behavioural models (eg: GSR Behaviour Change
Knowledge Review - Overview of Behaviour Change Models and their Uses
and the COI Common Good report on public sector behaviour change).
18
Evaluation for Alcohol Social Marketing
It is important that the Evaluation Team ensures that it is the agreed
marketing objectives that drive the logic and structure of the selected
behavioural model that in turn drives the marketing tasks. The temptation
needs to be avoided of looking to numerous behaviour change theories to
‘retro-fit’ onto the marketing tasks that have been set.
Whichever approach is used to construct the behavioural model, the result
should be a series of identified ‘steps’ we consider to be prerequisites for the
desired behavioural outcome. It is these steps that will typically define the
main things we want to measure in our evaluation (‘what to measure’).
It is important to emphasise that at this stage of the evaluation process the
KPIs have been built on a behavioural model that is by definition
hypothetical, because the marketing programme is still at the planning stage.
It will be for subsequent stages in the evaluation process to:
a) decide how these KPIs will be measured and reported (see Step 3
below)
b) validate that the model explains actual behaviour. In other words, once
evaluation reporting begins, the learnings may be used to review the
behavioural model and whether all the measures are relevant and
whether new ones need to be introduced (Phase 4, Steps 1-3 ).
Other points to note in constructing behavioural models
From this stage onwards in the evaluation process, the behavioural model will
be continuously refined as we learn more. The behavioural model is unlikely
ever to be fully validated and is by definition hypothetical. But in a complex
area like influencing health behaviour, we have to start somewhere, based on
the best information available. If nothing else, the high level behavioural
model immediately starts to define gaps in our information that need to be
included in the Evaluation Plan (see Phase 2, Step 4). From this point
onwards a ‘running list’ can be constructed of what ‘we know we don’t know’ important unknowns, which will define research briefs as the evaluation
process continues to unfold.
A further point to note is that there is often a strong case for building different
behavioural models for each consumer segment rather than the population as
a whole. It follows that if different segments have been identified on the basis
of different behavioural needs, then the social marketing interventions will be
different for each segment, requiring different measurements for success
along the behavioural journey.
Below is a summary checklist of prompts in relation to behavioural models.
CHECKLIST
Lines of enquiry to consider when constructing behavioural models
For each of our high level marketing objectives, think through how, in broad terms
do we expect each one to be achieved? This will produce your list of high level
19
Evaluation for Alcohol Social Marketing
social marketing tasks
For each high level social marketing task, ask the question: what do we expect
people to do as a result of this? What do we expect them to think, feel or do
differently?
Don’t be frightened of using hypothetical language in framing your behavioural
models. Where you are not sure how something will work, be confident enough to
state how it should work.
What are the main barriers to the desired behaviour happening and how can
marketing overcome them? This will help identify some of the things we need to
measure in our evaluation
What are the main triggers to the desired behavioural change and how can social
marketing activate them?
What is the sequencing of anticipated effects, moving from, say, awareness
through attitudes towards behavioural change?
What are the different levels of interaction, at societal level through community
level to individual level for example?
What are the relevant behaviour change models and theories that we can draw on
to develop our thinking?
Step 2 – Defining the behavioural goals
At some stage in the social marketing evaluation process, behavioural goals
need to be agreed with PCT colleagues. Ideally this should be in the Phase 1
Scoping, because to a large extent behavioural goals should define the
objectives of our social marketing activity. Sometimes, however it is not
possible to agree behavioural goals until quite late in the evaluation process.
From a social marketing evaluation perspective, it is only possible to move on
to the next stage of the evaluation process, the setting of marketing KPIs,
once these broad behavioural goals have been agreed. The important thing
is that the behavioural goals lie at the heart of the Evaluation Team’s thinking
about the behavioural model going forward.
Step 3 – Defining the KPIs
Having defined the behavioural model and the behavioural goals, the
Evaluation Team is now ready to develop the Key Performance Indicators
(KPIs) that will be used to evaluate the social marketing programme.
Although the number of measures will vary by social marketing programme,
the aim should be to identify approximately ten measures that are deemed
absolutely critical to the marketing programme’s success. Ideally, these
should include some measures that relate to awareness and basic
engagement with consumers, but there needs to be a good balance of
‘downstream measures’ that are all critical to delivering the final desired
behavioural outcome.
There is no set approach for working through the process of constructing the
behavioural model and consequent measuring points described in the KPI
20
Evaluation for Alcohol Social Marketing
development section above, but it is clear that the whole of the evaluation
team needs to be involved, including the key thinkers at any selected social
marketing agency.
Whichever approach is adopted, it is important to recognise the importance of
defining the social marketing KPIs thoroughly at the outset, because they will
be the building blocks on which more detailed evaluation planning follows in
the Phase 3 implementation phase of the evaluation process, when creative
agencies will be working up detailed executional plans.
The aim should be to build a raft of KPIs that are ‘leading indicators’, which if
achieved collectively, are likely to deliver the required impact.
To achieve this, a ‘Balanced Scorecard’ approach is recommended, where
the aim is to identify a suite of measures, which mirror the behavioural journey
for the marketing activity and structure the KPIs accordingly.
There are two main risks in developing dashboard measures or KPIs. The
first is that in the interest of appearing comprehensive, there is a temptation to
produce an excessively long list of measures. This risk can be avoided by a
simple ‘relevance test’ which involves asking one simple question for each
KPI being considered: is there a clear ‘line of sight’ from the KPI under
consideration to the desired behaviour? If not, it is unlikely that the KPI will be
of much use in evaluation.
The second risk is that we tend to favour those KPIs that are easy to
measure. If lots of hard data are available for a KPI, it is natural to feel
comforted and want to use them. In reality, the same test question used
above applies. If it can be demonstrated that the KPI has a clear ‘line of sight’
to the desired behavioural outcome, then the data will probably be useful in
the evaluation.
Every social marketing programme is different and therefore the KPIs used
will vary in each case, so there is no mandatory KPI list. That said, all KPIs
used in social marketing evaluations should pass the tests in the checklist
below.
CHECKLIST
Requirements for Evaluation Plan KPIs
All KPIs need to have a clear ‘line of sight’ to the desired behaviour outcome
KPIs need to be directly linked to key ‘moments of truth’ anticipated in the social
marketing, and/or the key ‘nodes’ in the behavioural model
KPIs to cover the range of measurements types, to include:
 Input
 Output
 Outtake
 Intermediate
 Behavioural change
 Outcome and Impact
21
Evaluation for Alcohol Social Marketing
KPIs to cover the range of ‘levels’ at which the marketing is expected to operate
eg:
 Community level
 Peer-group level
 Family level
 Individual level
If the targeted behavioural change involves different consumer segments, then
KPIs need to be drawn up for each segment
The main KPIs also need to be timelined, with what level of achievement is
expected at which point in time after the social marketing activity breaks (eg
monthly, quarterly or annual progressions towards target)
Levels of achievement for each measure
Having decided what to measure in the form of KPIs, the task is to set a target
level to be achieved for each measure, preferably with a target timeline. The
levels need to be SMART2 and should define ‘what good looks like’ at each
stage of the social marketing programme.
In some cases there will be some relevant standard measures already in
place that are relevant for the desired behavioural outcome in question. For
example, some long running social marketing programmes have well
established tracking studies in place, so it may be appropriate to select some
existing measures and set SMART objectives for how much we expect such
measures to move as a result of the planned marketing activity.
In other cases, the ‘what to measure’ phase above will identify a KPI required
to indicate social marketing success for which there is no existing data. It is
particularly important to obtain a clear definition of these ‘new’ measures early
on in the evaluation process, because work will be required to benchmark
current awareness, attitudes or behaviour so that progress can be tracked
after the social marketing activity has been launched.
Step 4 – Writing the Evaluation Plan
Following completion of all the work with the social marketing agency above,
the Evaluation Team should be in a good position to write a fuller version of
the Social Marketing Evaluation Plan.
It is important to realise that at this stage in the evaluation process, the
creative briefs will not even have been issued, so the detailed mechanics of
what will be measured and how will not be available. However, the high-level
strategic thinking about how the social marketing activity is intended to work
should be able to be clearly expressed, focusing on the key aspects of the
behavioural journey that is envisaged.
The template to be used for the Evaluation Plan is included below.
2
Specific, Measurable, Achievable, Relevant, Time-bound
22
Evaluation for Alcohol Social Marketing
CHECKLIST
Suggested sections for the Evaluation Plan
The behavioural objectives and a PCT-approved statement of what the role of
social marketing is in relation to other policy levers
The objectives of the evaluation process – our overall rationale for evaluation,
and the benefits expected in terms of improved learning and future social
marketing
A statement of the behavioural model, describing specifically the way in which
social marketing will influence behaviour
A statement of the social marketing programme’s duration, the key milestones of
what consumer changes we want to happen by when
A full list of social marketing KPIs. Each KPI needs to be supported with a
rationale for how it captures an important part of the behavioural model
A summary view of what resources will be required to measure the KPIs, including
the proposed budget
A description of how the Evaluation Plan will contribute to the PCT’s Test-LearnRefine approach, including stakeholder engagement and knowledge sharing.
A discussion of the evaluation at this stage should flush out any issues that
have not been covered in the evaluation development to date and may well
raise some probing questions that need to be answered and fed back into the
overall social marketing strategy. The important thing to note is that the
evaluation process needs to reach a point when there is consensus on the
behavioural hypotheses and the consequent things we want to measure,
before the more detailed task begins of planning how are we going to
measure the various identified measuring points.
The next stage involves the detailed data collection plan for the required
measuring points. This needs to be a cross-disciplinary process, certainly
involving research experts, but also any agency people working on marketing
development.
Step 5 – Briefing the implementation agencies
Once the Evaluation Plan has been approved, it should be included in the
briefs to any implementation agencies invited to pitch for the different
elements of the social marketing programme.
23
Evaluation for Alcohol Social Marketing
PHASE 3 IMPLEMENT
Introduction
Phase 3 of the evaluation planning process focuses on the detail of how the
implementation of the social marketing interventions should be measured.
For evaluation purposes, all key marketing agencies involved in marketing
development and implementation need to be involved at this stage. This
includes CRM, promotional, PR, advertising and other social marketing
agencies, all of which will collectively be referred to ‘implementation agencies’
from here onwards.
Up to this point in the evaluation process, most of the decisions have been
driven by the Evaluation Team in getting to an approved Evaluation Plan and
deciding what needs to be measured. At this point in the process much of the
onus passes to specialist research colleagues, who are best positioned to
plan the detail of how things will be measured.
There are three steps in the Implement Phase:

Developing the evaluation detail with implementation agencies (What
to measure in more detail)

Designing and setting up the measurement approaches (How to
measure)

Getting the most out of evaluation
These steps are detailed below.
Step 1 – Evaluation development with implementation
agencies
Having driven the evaluation thinking to date, the Evaluation Team is ideally
placed to underline the importance that will be attached to evaluation with the
selected implementation agencies, as they start to develop their social
marketing solutions.
24
Evaluation for Alcohol Social Marketing
Soon after the implementation agencies have been selected to develop
marketing activities, they need to be fully briefed on the evaluation thinking
that has been accumulated up to this point in the project.
Going forwards, the remainder of the social marketing programme’s
development should involve the Evaluation Team working with the
implementation team in an iterative process, because as they develop the
detailed thinking on the precise way in which their parts of the marketing
programme are going to work, appropriate refinements may be required to the
behavioural model that was first developed leading up to the Evaluation Plan.
Such refinements may in turn require some revision of the detailed KPIs.
If the Evaluation Team’s thinking has been robust in the Evaluation Plan, it
should not be the case that the KPIs are revised significantly. However, it is
likely that much more detail can be added on the measuring points that may
be required relating to how marketing will influence consumers.
There are no set ways in which the above iterative process should work, but
one approach that appears to work well is running workshops for all the
relevant people to pool their ideas and suggested approaches.
However they are reached, the Evaluation Team should now have a
comprehensive list of what to measure for the evaluation, beyond the list of
high level KPIs included in the Evaluation Plan. The detail of proposed
research and analysis methodologies is what will be answered by the relevant
research experts in the next step.
Step 2 - Designing and setting up the measurement
approaches
At this point in the evaluation process, the Evaluation Team needs to issue a
succession of briefs for each piece of research required for the evaluation.
These briefs need to use the following format.
CHECKLIST
Suggested headings for Evaluation Research Briefs
Overview of the proposed social marketing evaluation – with overall behavioural
model
Objectives of this strand of research within the evaluation
The KPIs or other reporting output that is required for this evaluation strand (eg the
appropriate dashboard measures and how they will be reported)
What we are expecting to test, learn and refine – the hypotheses we want to test
Budget
Timing
It is important to note that the Evaluation Team needs to include sufficient
research expertise to write authoritive evaluation briefs and to be able to
judge the most cost effective responses to those briefs.
25
Evaluation for Alcohol Social Marketing
The Evaluation Team members are not expected to be research experts who
are conversant with every research methodology involved. However, having
some knowledge in the team of all the research methodologies available may
be useful.
In complex social marketing programmes with many different activity strands,
there is a risk of the evaluation becoming fragmented. One way of
overcoming this is to parcel up various KPI reporting requirements into a
bigger brief for a single agency. This approach can help create consistency in
research across different activity strands. Having a few lead agencies
responsible for evaluation will also help to ensure a valid database of “norms”
can be developed for future benchmarking.
For larger social marketing programmes, another way of ensuring a ‘joined up’
approach to the evaluation is to appoint an econometric specialist to take
responsibility for managing the piecing together of the different evaluation
strands. Econometric modelling aims to isolate the individual effects of
different activity strands within the marketing programme, using a variety of
statistical techniques such as multiple regression models and/or structural
equation models.
Econometric models take considerable time and money to build, but have the
great advantage that they can successfully attribute and quantify marketing
success to different marketing activities. This can become a powerful
mechanism for embedding the Test-Learn-Refine process.
Step 3: Check against the Evaluation Plan and the principles
This is a good point at which to formally check that the evaluation we have
designed fits with our Evaluation Plan and the ten marketing principles which
can be used as a checklist. For example, have we included a full range of
measures in our evaluation and considered sufficiently rigorous and
consistent approaches in our evaluation design?
26
Evaluation for Alcohol Social Marketing
PHASE 4 LEARN
Introduction
The final phase of the evaluation process is the evaluation reporting and
learning that flows from it, comprising four steps:

Immediate reporting soon after social marketing activities have been
launched, with a focus on ‘input measures’

Interim Evaluation focusing on identifying early learnings from the
social marketing programme, which can be usefully applied to refining
the activity before it finishes. This can be viewed as ‘short termlearning’ with the emphasis on immediate practical application and
checking the soundness of the model.

The Final Evaluation Report, focused on capturing the longer term
learning from the marketing programme, which can be fed back into
future social marketing planning, consumer insight and the long term
learnings on what influences consumer behaviour and why

Sharing and capturing the learnings from the Evaluation Report.
Step 1 - Immediate post-launch reporting
The first step in this phase relates to immediate reporting soon after social
marketing activities have been launched. Often during the first few months of
a marketing programme, there is by definition likely to be little useful
information coming back.
However, there is certainly the opportunity to evaluate whether all the inputs
to the social marketing programme are being delivered to plan, using the
checklist below.
CHECKLIST
Evaluating input measures at the start of a social marketing programme
27
Evaluation for Alcohol Social Marketing
Have the right interventions been launched into the market place?
At the right time?
In the correct quantity?
To the right place/target audience?
Can we gather some fast track/dipstick measures to gain some directional
measures on what is working and what is not?
Can we talk to appropriate partners and intermediaries, who may have early, albeit
anecdotal, feedback, which may identify emerging issues for us to address or
make contingency plans for?
Can we talk to people at the frontline who may be aware of some early evaluation
indicators (eg call centres and healthcare professionals)?
Monitor ‘buzz’ in the marketplace and press comment
Get out into the marketplace, experiencing first hand or at least talking to people
who are experiencing the marketing interventions
It is worth noting that some of the items in the checklist above may help
provide evidence to support or question the validity of the adopted
behavioural model, even at this early post-launch stage.
Step 2 – Interim Evaluation
The purpose of Interim Evaluation is to gain important learnings while the
social marketing interventions are is still in the marketplace, which can be
used to make short-term adjustments where possible, refining and improving
the remainder of activity before the social marketing programme ends.
A balance has to be struck in deciding when to complete Interim Evaluation.
The risk of starting interim reporting too early is that there will be insufficient
data accumulation to form statistically significant and meaningful
interpretations. For example, a tracking study with two or three pre-activity
baseline datapoints is unlikely to yield reliable findings until at least three
datapoints are in place following launch. Conversely, some website and
response measures may be available almost in real time immediately after the
marketing activity breaks, and so may provide immediate indications of
success and opportunities for refining the programme.
There is also a risk that interim reporting is too late to be useful, if there is
insufficient time to refine and improve marketing activities before they finish.
This means interim reporting should be frequent and involve a collaborative
process between all key decision makers in the extended marketing team.
The best approach is to have members of the social marketing team, and
implementation agencies meeting on, say, a fortnightly basis, certainly during
the early phase of the marketing programme, to review all the available
information and be prepared to make rapid decisions. This collaborative, realtime approach should also mitigate the risk of incorrect and sudden decisions
being made on the basis of insufficient information, which can always be a
problem in over-zealous interim reporting.
28
Evaluation for Alcohol Social Marketing
Sharing the detail of the quantitative evaluation data is a vital element of the
process, but so too is the pooling of qualitative feedback from the field and
more anecdotal indications of how the marketing activity is being received and
acted upon on the ground, either by the consumer target audience or relevant
intermediaries and stakeholders. The advantage of having everyone involved
with the social marketing programme in the same room, is that decisions on
re-directing effort can be made quickly. For example, if outbound calls are
exceeding whatever KPI measure has been set, it may be possible to reduce
this activity and switch budget and resources to areas that appear to be
performing less well than expected, upweighting, say, the amount of face-toface contact with consumers.
Integral to the success of this ‘war cabinet’ style of working is the need for
Evaluation Team members to be attuned to hearing about how the social
marketing activity is doing from as many formal and informal sources as
possible and gathering early evidence of success and failure thoroughly. In
addition to analysing the dashboard data, this style of working will require
team members to be making field visits and frequent networking contact with
key stakeholders and intermediaries.
Below is a checklist that summarises the lines of enquiry required for Interim
Evaluation.
CHECKLIST
Interim dashboard reporting for each KPI
How is each KPI performing – to include the trend in each data point since the prelaunch benchmark phase
An assessment of how reliable the data is for drawing conclusions
What are the decision options for each KPI conclusion? E.g. review next month,
increase level of activity, cease the activity
What, if any are the recommended changes to the social marketing programme?
Recommended refinement to the KPI recording plus rationale
Briefing the relevant agencies on marketing activity changes and any revised KPI
targets
Circulating and briefing the relevant dashboard results to stakeholders, particularly
‘good news’ stories to increase their motivation (or bad news to trigger corrective
action). There is also the opportunity to request stakeholders to provide extra
information to plug any gaps in the interim evaluation data
Step 3 – The Final Evaluation Report
Introduction
Whereas the Interim Evaluation is focused on short-term adjustments to social
marketing implementation, the Final Evaluation Report is much more focused
on the long-term learnings that can be used as part of the Test-Learn-Refine
approach to continuous improvement in social marketing.
29
Evaluation for Alcohol Social Marketing
Therefore the target audience for the Evaluation Report is primarily internal
PCT decision makers, who can benefit from social marketing learnings in
planning future activity. The report should be a ‘warts and all’ summation of
what has worked and what has worked less well with recommendations for
future improvement. The Evaluation Report is not intended for the public
domain, although its findings will be useful if a separate evaluation report is
required for wider publication.
The timing of the Final Evaluation report will vary, depending on the timing of
any relevant qual and quant debriefs, the results from any econometric
modelling and social marketing feedback from other sources, such as health
professionals. However the target should always be for the Evaluation Report
to be completed within three months after the social marketing period has
ended.
Context setting for the Evaluation Report
As the Evaluation Report is primarily designed to be an impetus for
continuous improvement in PCT social marketing, it should be forward looking
and action-oriented, rather than becoming some lengthy historical document
recording for posterity the entire details of the social marketing programme.
Nevertheless, it will require a front section, setting the context for the social
marketing activity and summarising what it was trying to achieve, referring to
separate and more detailed documents such as research reports where
appropriate.
Summary of evaluation approach
The first section of the Evaluation Report needs to describe how the
evaluation itself was carried out. The purpose of this exercise is to form an
honest view of what worked well in the evaluation process and which aspects
could be improved in future evaluation activity.
In conducting the evaluation over the social marketing period, there will have
been numerous interactions between members of the Evaluation Team and
other stakeholders and contributors to the evaluation process. Normally it
should be sufficient for the Evaluation Team to have a focussed discussion
about the overall evaluation process and record the findings. However, if it is
felt that there is insufficient knowledge about how certain key stakeholders
feel about the evaluation process, it may be appropriate to specifically ask
them.
A checklist is included below for the type of process question to be addressed
by the Evaluation Team.
CHECKLIST
Evaluation Approach questions to consider in the Evaluation Report
Was the evaluation team formed at the right time in relation to the social
marketing development process?
Were the right people included in The Evaluation Team at the various stages of
the social marketing programme?
30
Evaluation for Alcohol Social Marketing
How effective was the liaison with any agencies?
Was sufficient budget allocated to the evaluation?
Did we actually include the right research measures?
Were the right stakeholders engaged and were they involved sufficiently deeply
and at the right level – were their expectations managed appropriately?
Have all the Evaluation Team’s activities gone to plan – if not what can we learn
for planning future evaluations?
What were the main benefits and disappointments in the evaluation?
What things helped and hindered the evaluation process?
From the above, what are the recommended improvements to comparable
evaluation processes in future?
Input measures
Next, the Evaluation Report needs to record the conclusions and
recommendations regarding the input measures for the social marketing
activity, capturing details of what the programme intended to put into the
market place and how much of this was actually delivered to plan. This
should include levels of distribution of social marketing collateral, its correct
placement and level of exposure, all of which are included in the checklist
above for Step 1 - immediate post-launch reporting.
Review of achievement versus objectives
The evaluation then needs to consider the detail of what the social marketing
programme achieved versus the objectives that were set in the Evaluation
Plan. This review needs to look at both ‘bottom-up’ and ‘top-down’
perspectives.
The bottom-up element looks at the individual KPIs and how the social
marketing activity performed against each one. These should review the pre,
during and post trends in each KPI versus the targets set. From the KPI
reporting that will have been circulated widely in the Interim Evaluation phase,
most colleagues should be aware of how each KPI has performed and will
have shared views on possible reasons for any over and under performance.
The purpose of the KPI review in the Evaluation Plan, is to record all the KPI
performance trends over the duration of the social marketing programme and,
following wide discussion, come to an objective view on what has worked well
or less well, with a supporting rationale and recommendations for future social
marketing.
CHECKLIST
KPI review for the Evaluation Report
Trend data for each KPI, with supporting commentary to summarise conclusions
on the variances versus target
For each KPI trend, capture the learning, specifically in relation to the route to
behaviour change
31
Evaluation for Alcohol Social Marketing
Make recommendations on which KPIs were most useful in measuring
behavioural influence and possible learnings for using similar KPIs in future
Make recommendations on how social marketing benchmarks may need to be
revised in future on the basis of the social marketing learning
The top-down perspective is mainly a consideration of the hypothetical
behavioural model and whether it broadly represented reality, and if not, what
can be learned for future social marketing programmes.
Throughout the social marketing programme’s development and
implementation, there will have been numerous discussions on the
programme’s hypothetical behavioural model. The Evaluation Plan is an
opportunity to pool the collective thinking on how much of this thinking still
holds good and how future thinking and social marketing may need to change.
Therefore a number of conversations need to take place at this stage with
relevant colleagues, experts and intermediaries to review the behavioural
model and reach conclusions and make recommendations.
CHECKLIST
Behavioural model review for the Evaluation Report
Which aspects of the behavioural model seem to be validated/not validated by
the data gathered from the evaluation?
How should our thinking about the behavioural model be modified in the light of
what we have learned from the evaluation? What is our evidence for reaching
this conclusion (eg academic peer review)
What are the unanswered questions we now have as a result of evaluating the
marketing activity– what research/analysis is now required to advance the
thinking further?
Financial evaluation
The Evaluation Report should come to a view on the relative cost
effectiveness of different marketing activity strands. This is a notoriously
difficult area and even with a comprehensive econometric model, it can be
difficult to establish causality for the different influences on behaviour (eg
effects of marketing communications compared to other policy levers such as
legislation and enforcement). It can be even more challenging to isolate the
effects of individual strands of social marketing activity in influencing
consumer behaviour.
Nevertheless, the social marketing KPI reporting should be able to provide
some indicative pointers on the cost effectiveness of different activities. For
example if, per pound invested, leaflets in GP surgeries have delivered fewer
responses to a helpline than other channels, some price optimisation
recommendations can usually be made, provided there is sufficient valid and
reliable information available on the relative degree to which each channel
moves the various target audiences towards the desired behavioural change.
32
Evaluation for Alcohol Social Marketing
CHECKLIST
ROI considerations for the Evaluation Report
Include metrics around cost per thousand for different elements of the marketing
mix
Relative degree of desired behavioural effect between different social marketing
activities
Identifying potential ‘compound effects’ (ie the degree to which different activity
strands work together to deliver more behavioural effect than the sum of their
individual effects)
Having gathered as many facts and views as possible on processes, inputs,
KPIs, behavioural models and ROI, the Evaluation Team needs to synthesise
all the findings. This is probably best achieved by the Evaluation Team
coming together in an Evaluation Report workshop and gaining consensus on
its major components.
CHECKLIST
Evaluation Report Workshop – target outputs
Five things we have learned from the Process analysis plus recommendations for
the future
Five things we have learned from the Inputs analysis plus recommendations for
the future
Five things we have learned from the KPIs analysis plus recommendations for the
future
Five things we have learned from the Behavioural model analysis plus
recommendations for the future
Five things we have learned about the research method, measures and
approaches to data analysis.
The Evaluation Report workshop also needs to agree an action plan with timings
for delivering the Evaluation Report after the workshop
Following the workshop, a ‘summative’ approach needs to be adopted to pull
together all the learnings from the evaluation process into the Evaluation
Report, using the structure below. Ideally the Evaluation Report should be
peer reviewed by an independent academic or expert.
CHECKLIST
Headings for the final Evaluation Report
Context for the Evaluation – brief description of the background to the social
marketing programme and its lay-down, referencing more detailed documents
where appropriate
Inputs review. What has worked/not worked and why; conclusions and
recommendations
Process review: What has worked/not worked and why; conclusions and
33
Evaluation for Alcohol Social Marketing
recommendations
KPI performance; What has worked/not worked and why; conclusions and
recommendations
Behavioural change hypotheses: Which areas appear validated/not validated
and why; conclusions and recommendations for further refinement of the model
Implications for future health social marketing: Lessons and recommendations;
examples of useful tools, learning materials and approaches for future marketing;
transferable learnings related to the target audience(s) that can be used in future
marketing
Summary of insights gained from the evaluation
Future benchmarking Implications from the evaluation for future comparable
marketing programmes
Budget guidelines for future social marketing programmes, based on the ROI
analysis
Stakeholder engagement: Transferable learnings that can be applied in future to
marketing programme stakeholders (eg PCT colleagues, regional SHA leads,
health practitioners)
Step 4 - Sharing and capturing the learnings from the
Evaluation Report
The Evaluation Report needs to be as action-oriented and forward-looking as
possible, focusing on continuous social marketing improvement. Therefore
mechanisms need to be found for disseminating its learnings fully and
ensuring its findings are embedded and acted upon in future social marketing
planning. Some ideas for how this might be achieved are included below.
CHECKLIST
Ideas for disseminating and embedding the Evaluation learnings
Circulate the report as widely as possible within the PCT social marketing
community
Add the Evaluation Report to the Alcohol Learning Centre website
Run learn and share evaluation events for relevant stakeholders, or invite
feedback on the report via online discussion threads etc
Linking up with Insight colleagues to capture learnings in real time and feed back
into the relevant teams
Encourage academic networks to use the Evaluation Report as a case study for
wider dissemination
34
Evaluation for Alcohol Social Marketing
Part 3 – Evaluating social marketing with
limited budgets
Introduction
The four step evaluation process described in Part 2 is applicable to larger
PCT social marketing programmes with total budgets over £200K and with
sufficient people resource to complete the tasks listed.
Given that PCT social marketing interventions are frequently on a much
smaller scale, this section looks at evaluation approaches that might be
considered when resources are much more limited.
It can be argued that the ten principles included in Part 1 hold good whatever
level of resource is available and the basic sequence of evaluation activities in
the four step process also apply to even the smallest piece of social marketing
activity.
However, a real tension needs to be acknowledged between some people
who say that if a robust evaluation process cannot be applied, then it is
probably not worth the attempt and others, who take a less purist view, saying
that some form of evaluation, however crude, is much better than nothing.
This section assumes the latter course, suggesting some pragmatic
approaches for evaluation in constrained circumstances.
Revisiting the four-step process in the context of limited
budgets
First let’s revisit the four phase evaluation process in Part 2 and examine how
the same basic approach can be applied in the context of a very limited
evaluation budget.
PHASE 1: SCOPE – on a limited budget
If we take the example of a single person being allocated £10K to engage and
encourage all pharmacies within a PCT to take up a new initiative, one could
do a lot worse than simply answer the questions below as comprehensively
as possible.
CHECKLIST
EVALUATION SCOPING questions for small pieces of marketing activity
Who is going to be responsible for evaluating the activity and who needs to be
involved in the process?
Are we clear about what the activity is trying to achieve and how does it fit with
other activities trying to influence the same target behaviour?
Have we done anything similar before and whom can we ask about what lessons
were learned?
What do we want to learn from evaluating this activity – what decisions will be
made or what might we do differently as a result?
35
Evaluation for Alcohol Social Marketing
What is the appropriate amount to spend on evaluating this activity?
PHASE 2: DEVELOP – on a limited budget
The checklist below suggests some simple ideas for consideration when
developing evaluations for smaller social marketing initiatives.
CHECKLIST
DEVELOP PHASE considerations for small pieces of marketing activity
Make sure that whoever is running your social marketing activity knows the
importance you attach to evaluation
When briefing whoever is developing the social marketing activity, make sure they
know they will be expected to commit resource (eg time and effort) to providing as
much evaluation information and insight as possible within resource constraints
It is unlikely that in small pieces of social marketing activity, we will be forming an
entire behavioural model for how we expect the activity to influence consumer
behaviour. But the basic thinking process involved needs to reflect the process
described in Steps 1 and 2 for this phase. We should certainly form a view of how
the activity will work in changing the target audience in some way – ie the basic
behavioural goals for the activity, even if they are not particularly comprehensive
There is no reason why even the smallest activity cannot be set some form of KPI
at the outset, even if it is a simple input measure or output measure. For example,
if we want to engage all the PCT’s health visitors on a particular issue and invite
them to a seminar, we could set ourselves the target of approaching all health
visitors (input) with a view to gaining attendance by a target percentage of them at
our event (output).
For small pieces of social marketing activity, we are unlikely to be writing a full
Evaluation Plan. But when we finally confirm the activity, we can certainly include
a paragraph somewhere that sets out for the relevant people what they are
expected to deliver regarding the activity’s evaluation, details and deadlines for
reporting and how learning will be shared afterwards etc.
36
Evaluation for Alcohol Social Marketing
PHASE 3: IMPLEMENT – on a limited budget
Much of the evaluation implementation phase content included in Part 2 was
set in the context of large research budgets and significant availability of
research expertise to inform decisions on how to evaluate rigorously. In
smaller social marketing activities lacking this level of resource, many of the
same principles can be applied in a scaled down or less sophisticated way as
set out in the checklist below.
CHECKLIST
IMPLEMENT PHASE considerations for small pieces of marketing activity
Have we had conversations with whoever is implementing the social marketing
activity about the detail of how they are going to measure success? Can the plan
be summarised in a few paragraphs to cement commitment to some form of
evaluation taking place?
Can we set down how the data will be gathered and in what format it will be
collected, to focus attention on how we are going to use the evaluation for future
decision-making?
Can we get more out of our low-budget evaluation by considering a control group
approach (e.g. engaging with some stakeholders and not others and getting some
directional evidence on what is working and what is not)?
Can we make informal contact with rejectors and lapsers for the activity and at
least have a conversation about what were the causes and feed that back into our
planning of future similar activity?
Even if we cannot afford comprehensive qualitative research, can we conduct a
small online survey or follow up with a few telephone interviews?
Can we adopt the equivalent of pre-testing to test some ideas – for example asking
intermediaries to experiment on our behalf?
PHASE 4: LEARN – on a limited budget
For smaller pieces of social marketing activity, it is unlikely that the
comprehensive evaluation approach described in Part 2 will be appropriate,
but the same sequence of steps at a more basic level can be taken as
described in the checklist below.
CHECKLIST
EVALUATION PHASE considerations for small pieces of marketing activity
However small the social marketing activity, it is worth a quick check that it running
to plan soon after it has been initiated
Once the activity is fully underway, it makes sense to check on progress and revise
plans if improvements can be made before it is complete
Whilst a small piece of activity is unlikely to justify a full evaluation report, some
form of data capture is required about what worked well and less well, so that the
learnings can be fed back into the pre-planning of similar future activities
37
Evaluation for Alcohol Social Marketing
Other evaluation options in the context of limited budgets
This section lists a wide range of other things that might be useful to consider
when evaluating social marketing activities on a limited budget. None are
purporting to be methodologically ideal approaches, but all can provide some
directional information that might be useful in evaluating social marketing
activity.
Informal telephone interviews
Let’s take a practical and hypothetical example of £5K being allocated by a
PCT to providing a web-based resource listing local IBA services. Details of
the resource are distributed to 500 local volunteer workers along with ideas
and information on alcohol counselling. Allocating 10% of the budget (£500)
to the evaluation could consist of the following:

Telephone and/or online survey interviews with 20 randomly selected
volunteers to ask basic questions about whether the resource had
been accessed; views on the usefulness of the resource and the
degree to which it was perceived to have improved the nature of
alcohol interventions and their effectiveness
Field visits
Arranging a day in the field with representatives of local alcohol networks
could provide significant insight into the usefulness of a piece of social
marketing activity. A write-up of the field visit could be an excellent way of
capturing the lessons learned for future social marketing development with
different target audiences.
Negative sources
‘Negative sources’ were described in Part 1 (Principle 9) as a useful source of
information in evaluation and this is particularly the case when budgets are
limited, because even making informal contact with a few rejectors or lapsers
from our social marketing activity might provide some useful pointers to what
is working less well with our social marketing activity.
Personal involvement and discussion
In the absence of any evaluation resources, personal involvement in the
social marketing activity and discussing its effects directly with the target
audience can provide some perspectives on what is working well or less well.
Of course care needs to be taken in drawing firm conclusions because of the
subjectivity involved, but even some exposure to end-users of the social
marketing intervention is likely to be informative.
Retrospective interviewing
Asking people their experiences of a social marketing intervention after they
have been exposed to it presents many methodological challenges to
qualified market researchers. Nevertheless for a few hundred pounds, it is
sometimes possible to find trained market researchers who can adopt a
practical approach to interviewing a small sample of respondents to try and
38
Evaluation for Alcohol Social Marketing
ascertain which aspects of a social marketing intervention have contributed
best and least to the desired behavioural change.
‘Hybrid’ approaches
Sometimes low cost market research can be completed by adopting a ‘hybrid’
approach to market research, commissioning professional researchers to
decide respondent recruitment approaches and to design and analyse
questionnaires administered by others (such as PCT employees or members
of local third sector organisations). Quite rightly, professional researchers will
only lend their name to work that adheres to the professional standards of the
Market Research Society, but this is highly desirable if a more formal
approach is required to the social marketing evaluation than is suggested by
the low cost ideas listed above.
Stakeholder workshops
In the absence of an evaluation budget, one way of capturing views on what
worked well or less well with a social marketing intervention is to draw
together a range of people who have been involved with (or have an interest
in) the intervention to pool thoughts. The collective learning from this exercise
can provide useful thoughts for the planning of future similar activities.
Selecting from formative, process and summative evaluation
In Part 2, the recommended four step evaluation process is primarily designed
to focus on a Test-Learn-Refine approach, placing emphasis on continuous
marketing learning. This approach includes elements of the three traditional
social marketing evaluation types of formative, process and summative
evaluation.
However, one way to practice low cost evaluation is to just concentrate on
one of these three evaluation types, which are summarised below.
Formative evaluation
Formative evaluation approaches help to “form” the intervention, by
scrutinising what is required and looking at the methods by which it is
delivered. This may include work which is undertaken by the project team
themselves, but it is often highly valuable to involve other stakeholders and
the target audience or participants themselves.
Formative evaluation can include work which is done at the outset of
developing a social marketing intervention, such as a needs assessment or
gaining views from key stakeholders and relevant experts.
Process evaluation
Process evaluation measures involve judging the activities (or strategies) of
the project against its objectives. Questions which might be asked as part of
a process evaluation include:
 Has there been a strategic plan, and have the project activities followed
the plan?
 Are all methods and materials used in line with the strategic plan, are
of good quality and achieve their aims?
39
Evaluation for Alcohol Social Marketing




Are relevant stakeholders and participants satisfied?
What factors have helped to facilitate the delivery of the project?
What factors have hindered it?
Has the project had any unintended effects?
Process evaluation can include both qualitative and quantitative approaches.
Examples may include:


Quantitative recording of data on aspects of delivery. For example this
might include the number of IBAs conducted, number of responses to a
‘call to action’, number of hits to a social marketing website, etc.
Qualitative information, which may include feedback and accounts of
experience from participants. This could include how people felt about
receiving an IBA, what they believed worked well and what could be
improved. Feedback may also be obtained from people involved in
delivering the social marketing intervention.
Many elements of process evaluation can be built in to the structure of piloting
or delivering an intervention. If these are planned from the outset, they may
not involve significant investment: means of collecting numeric data can be
built in to delivery, and the target audience can themselves be involved as
‘participant researchers’ or to ‘mystery shop’ relevant aspects of a
programme.
Summative evaluation
Summative evaluations examine what happens after a programme has been
delivered - the effects or outcomes. This might include assessing whether
the social marketing has achieved its objectives, and also taking a more
holistic look at the wider impacts (including, perhaps, unintended
consequences).
Different types of summative evaluation include:
 Outcomes – are there demonstrable effects on the target outcomes?
 Impacts – assessing the full range of impacts, whether they were
intended or unintended
 Return on social marketing investment
 Secondary analysis re-examines existing data to demonstrate
outcomes
40
Evaluation for Alcohol Social Marketing
Part 4 - Using secondary data for alcohol
social marketing evaluation
Secondary data is existing data, not collected specifically for the purpose of
the evaluation. Sources of secondary data might include statistics on hospital
admissions, or data from other social surveys.
Different sources of public health information on alcohol which are available
nationally in England are detailed in the publication of the Association of
Public Health Observatories (APHO); Indications of Public Health in the
English Regions 8: Alcohol. It includes 36 different indicators of alcoholrelated harm, health and behaviour across the English regions, and where
possible sub-regional analysis is included. It explains some limitations of the
different data sets and helps to explain some of the applications of the data.
Local Alcohol Profiles for England (LAPE) provide data on a range of
indicators relating to alcohol harm and alcohol consumption, which are readily
available at the PCT level.
Alcohol Consumption Data
Alcohol consumption behaviours can be challenging to measure
accurately. Large scale national surveys provide frequently used
measures of consumption, including synthetic estimates of average
consumption amongst the population at a PCT level. However, these
synthetic estimates have limited application for evaluation, as this
section will show. Other methods of measuring consumption at the
local level need to be considered.
Defining alcohol consumption
Changing alcohol consumption behaviours is one of the main objectives of
social marketing interventions around alcohol. However, defining and
measuring alcohol consumption can be challenging.
It is important to consider what the objectives of the social marketing are, and
how these may be reflecting in consumption behaviours. Relevant measures
for evaluation may be linked to:




Total alcohol consumption, for example measured in units
per week
Patterns of consumption, such as ‘at home’ drinking
Different drinking occasions, such as drinking in the home, or
in bars/pubs
Consumption of particular types of drinks (eg wine, highstrength beer/lager)
41
Evaluation for Alcohol Social Marketing
Each of these would require different measures of alcohol consumption. For
example, a measure of alcohol consumption in units consumed per week
casts little light on patterns of consumption and the contribution of individual
drinking occasions.
Identification of the target group
Just as important as identifying an appropriate measure of alcohol
consumption is to use data which allows identification of the relevant target
groups for the social marketing intervention. If the targeting of the intervention
is defined by age, gender, socio-economic group or area of residence, then it
is vital that data used to evaluate is also able to identify the same target
group.
Different ways of measuring consumption
Major social surveys which include measures of alcohol consumption tend to
use one or more of the following methods:

An average estimated by the survey participant over a period of time
(eg their typical weekly consumption in the last year). This method has
advantages over alternatives which ask about actual consumption over
a shorter period, as a short recall period may not reflect typical
consumption over a period of time. However, measures of
consumption which ask respondents to consider a long period (such as
a year) may mask short-term changes in behaviour. Additionally,
measures which ask respondents to think about their typical drinking
on an average day do not reveal patterns of consumption, and might
under-represent infrequent heavier drinking occasions.

Detailed account of consumption over a shorter period of time, either
by a diary method (recording consumption over the period it takes
place) or via recall of a recent period (eg drinks consumed over the
previous week). These methods tend to better illustrate patterns of
consumption and allow detail to be captured on drinking occasions,
types of drink consumed etc. However the period covered by the diary
or recall may not be representative of longer term drinking behaviours
and may be affected by seasonality (eg holiday periods, pay-days etc).
Accurately measuring consumption is challenging, and different measures
may over- or under- estimate consumption. This may occur because
recollection is poor, people may feel motivated to either under or over-state
their consumption to fit in with perceived ‘norms’, or they may genuinely not
know how much they have consumed (for example being unaware of the size
of a glass of wine or the strength of a lager). For these reasons, it is essential
that consistent approaches to measuring alcohol consumption are used for
tracking purposes.
There are different advantages and disadvantages to these and other
approaches to measuring consumption: there is no perfect measure. It is
important to choose a measure which will be most appropriate for the social
42
Evaluation for Alcohol Social Marketing
marketing programme which it will be used to evaluate. It is also important to
look at measures which are being used elsewhere, as consistency aids
comparison.
Sample sizes
A number of surveys are carried out nationally which have provided valuable
data on alcohol consumption. These include the General Household Survey
and the Health Survey for England. Other national lifestyle surveys, such as
TGI, contain information on alcohol consumption and have established
tracking data over a number of years.
However, whilst these and other surveys have robust sample sizes for
analysis at a national level, and may also permit analysis at the regional level,
they are not granular enough to provide data to monitor alcohol consumption
at a local level.
These surveys are sometimes used to generate ‘synthetic estimates’ of
consumption at a local level. The consumption measures detailed on LAPE
use synthetic estimates and these have valuable applications (for example in
understanding the relative situation in different localities, and for helping to set
local priorities). However, they are unlikely to be helpful for the purposes of
evaluation as they won’t reflect actual change in behaviours locally.
What is a synthetic estimate?
Synthetic estimates use data from large scale surveys (such as the General
Household Survey) to estimate what the data would be expected to be if it
were measured locally. The estimates are based on local population
characteristics, such as demographics. To use a very simple example: if the
national survey indicated men consume more alcohol than women, the
models used to calculate synthetic estimates will assume that an area where
the population is predominantly male would have higher consumption than an
area where the population was predominantly female (all other factors being
equal).
Therefore, if the local population remains demographically unchanged, the
models used to calculate the synthetic estimates will assume that this
population has behaved in the same way as similar people have in other
areas of the country. Any actual change in behaviour which follows a different
trend to that taking place nationally will not be evident in synthetic estimates –
hence they cannot usefully evaluate the impacts of local social marketing
activity.
Boosting the sample size in national surveys.
In large-scale national surveys, it is sometimes possible to increase the
number of people interviewed within a particular locality – often known as a
‘boosted sample’ - to ensure there is a sufficiently large sample size for local
analysis. This may be of value when there is interest in behaviours at the
overall population level (rather than amongst a particular target group).
43
Evaluation for Alcohol Social Marketing
Boosting the sample in an existing survey has some advantages over
conducting an independent survey:
 It makes use of interviews which are already being conducted for the
survey – numbers just need to be boosted (however, depending on the
size of the survey it is likely the number of existing interviews will be a
small portion of the total required)
 Large scale surveys tend to have rigorous and well scrutinised survey
methods
 Direct comparisons can be made with the results at the national level
or with other regions
However, there are also disadvantages:
 The rigorous methods used are likely to result in a high cost-perinterview
 The surveys are generally planned well in advance with fieldwork
conducted over long periods: it may be years between planning and
data being available.
 Most social surveys of this type cover a wide range of topics, with only
a limited amount of data on alcohol-related issues. Therefore, it is
likely to only be a practical proposition if there are broader applications
for the data other than alcohol social marketing.
Local and regional lifestyle surveys
Without boosted samples, typical national surveys do not contain sufficiently
robust data at a local level to assist in social marketing evaluation. However,
some surveys are carried out at a regional or local level which may be
valuable: it is worth investigating what research is conducted within the region
which could provide data on alcohol consumption. This might include health
surveys as well as lifestyle surveys conducted for commercial purposes.
Using Hospital Admissions Data
Reducing alcohol-related hospital admissions is likely to be the
motivation behind many social marketing initiatives on alcohol.
However, it may be challenging to establish links between social
marketing activity and hospital admissions, often because the impact
will only be evident in the longer-term, or because hospital admissions
are impacted by a range of factors other than alcohol and unpicking
these factors may be difficult. This section provides an overview of the
nature of the data and provides suggestions for how it may be used in
an evaluation context.
Understanding Alcohol Related Hospital Admissions
Alcohol-related hospital admissions are derived from Hospital Episode
Statistics. Hospital episode data include information about a patient’s illness
or condition – both primary (the main condition being treated) and secondary
diagnoses (other conditions the patient may have). The factors underlying the
44
Evaluation for Alcohol Social Marketing
diagnoses are not recorded in the data – indeed very often it would not be
possible to identify them.
Amongst conditions which are considered to be alcohol related, some are
specific to alcohol consumption (such as alcohol poisoning or alcoholic liver
disease), whilst many are also related to a range of other factors in addition to
alcohol. Where conditions are partially attributable to alcohol, alcoholattributable fractions are applied to the data to give an indication of the
proportion of hospital admissions which can be considered to be a
consequence of alcohol consumption.
What are Alcohol Attributable Fractions?
For conditions where alcohol is a contributory factor, Alcohol Attributable
Fractions (AAFs) enable the estimation of the proportion of cases that can be
attributed to alcohol consumption. AAFs are determined based on
epidemiological studies to determine relative risk estimates, as well as data
on alcohol consumption. AAFs are not recalculated on an annual basis, so
when tracking hospital admissions data over time, the assumption is that the
contribution of alcohol consumption to the total number of admissions remains
consistent. This should be taken into consideration if data which relies on
AAFs is used for evaluation.
How does alcohol consumption affect hospital admissions?
The relationship between alcohol consumption and hospital admissions is
complex, and will be influenced by a range of factors. For example, the lag
time between changes in consumption behaviour and impact on hospital
admissions will vary depending on the population in question, the nature of
the change in consumption behaviour and the health conditions being studied.
For instance a decline in binge drinking amongst young adults may have an
impact on admissions for acute conditions, such as those caused by
accidents and injuries in the short term. However, in contrast, reductions in
consumption behaviour amongst older higher risk drinkers could take longer
to have an impact on chronic health conditions.
Whilst it is possible to estimate the impact of certain interventions on alcohol
related hospital admissions, in practice a range of factors will have a bearing
on hospital admissions figures. For example, a reduction in hospital
admissions caused by road traffic accidents might be due to changes in
drinking and driving behaviour, but it could also be linked to changes in speed
enforcement practices by the police. Changes in admissions linked to
hypertension may be due to changes in alcohol consumption, but could also
be related to obesity. Detailed research and analysis can help to identify and
account for such factors. However, in the absence of this extra
understanding, restricting analysis to conditions which are wholly attributable
to alcohol (such as alcoholic liver disease) may be simpler, although these
only account for a fraction of all alcohol related hospital admissions and will
be more associated with certain segments of the population and alcohol
consumption behaviours.
Using Alcohol Related Hospital Admissions in Evaluation
45
Evaluation for Alcohol Social Marketing
Some of the main issues to consider when looking at alcohol-related hospital
admissions include:
 To an extent, changes in hospital admissions will have a time lag
after changes in alcohol consumption, so this is a measure which
can often only be used for evaluation in the longer term
 However, selectively looking at hospital admissions for certain
conditions may show impacts in the shorter term. It may be more
helpful to look at the data for individual conditions rather than
alcohol-related hospital admissions overall
This requires a good understanding of the likely impacts of the social
marketing intervention and the target population. Knowing what types of
hospital admissions are most likely to be impacted by any changes in
behaviour will allow for analysis to be more targeted and sensitive.
Accessing the Data
Hospital Admissions Data is available via Hospital Episode Statistics (HES).
Time is required between the data being collected in hospitals to being
released, to enable the data to be checked and cleaned, which will affect the
timing of evaluation using this data. It can be possible to access hospital
admissions data at the local level which has yet to be processed via HES.
This can speed the process of conducting a local evaluation. However, as
this has not undergone the same checking procedures, additional caution
should be exercised. For example, diagnostic coding typically changes during
the process of cleaning the data.
Other sources of secondary data
The APHO Indications report on alcohol details a range of other indicators.
Many are either not available at a local level or not relevant for evaluation
purposes. For example, data is collected on the number of people being
treated for alcohol dependency. However, it should be noted that this may be
as much a product of the availability of treatment as it is a measure of people
in need of treatment, therefore is not a measure of behaviour change.
Whilst ARHA data is collected to a consistent methodology on a national
basis, there may also be other sources of data which are collected within the
locality, or that it may be possible to set up. This could include data from
general practice: it may be possible to obtain or set-up methods to collect data
on number of people presenting with an alcohol related issue, or the number
of IBAs conducted. However this is not currently collected systematically in all
areas.
Learn and Share
Undertaking evaluation provides opportunities to learn from experience.
Whilst it is valuable to refine approaches within your own area, it is also an
important opportunity to share experience with others and benefit from
experience in other areas. Facilities such as HubCAPP encourage the
46
Evaluation for Alcohol Social Marketing
sharing of the details of individual projects and approaches, including the
results of evaluation. To maximise the benefit of sharing learnings, case
studies for sharing should be detailed about the objectives of the social
marketing, how it was undertaken (including issues such as commissioning
and funding) and details of the evaluation – from process to outcomes.
47
Evaluation for Alcohol Social Marketing
SOURCES
Alcohol Learning Centre (online resource)
Association of Public Health Observatories. (2007) Indications of public
health in the English regions 8: Alcohol. Executive summary.
Dawson, D.A. (2003) Methodological issues in measuring alcohol use,
Alcohol Research & Health, 27
Department of Health. (2007) Safe. sensible. social. the next steps in the
national alcohol strategy.
Department of Health and North West Public Health Observatory. (2008)
Hospital admissions for alcohol-related harm: understanding the dataset.
Donaldson, L.J. and Donaldson, R.J. (2003) Essential Public Health. 2nd
ed. Plymouth: Petroc Press.
Gibbins, C and Tocque, K (2009) Understanding the Attributable Fractions
and VSC26/NI39 Powerpoint presentation – Alcohol Improvement
Programme National Conference
HubCAPP (online resource)
Institute of Alcohol Studies (undated) Binge drinking: nature, prevalence
and causes.
Institute of Alcohol Studies (undated) Drinking in Great Britain
Institute of Alcohol Studies (undated) The Impact of Alcohol on the NHS
48
Evaluation for Alcohol Social Marketing
Jones, S. et al (2008) Alcohol-attributable fractions for England. Centre for
Public Health, Liverpool John Moores University.
Lannon, J, (ed.) (2008) How Public Sector Advertising Works, COI, WARC
and IPA
Local Alcohol Profiles for England (online resource)
May, T. (2001) Social research: issues, methods and process. 3rd ed.
Moyer, A and Finney, J.F. (2005) Brief Interventions for Alcohol Problems
The NHS Information Centre, Lifestyles Statistics. (2009) Statistics on
Alcohol: England, 2009
North West Public Health Observatory (2008) Hospital Admissions for
Alcohol Related Harm: Technical Information and Definition
Parry-Langdon, N et al. (2003) Process evaluation of health promotion
interventions, Policy & Politics, 31
Rickards, L. et al. (2002) Living in Britain: No 31 results from the General
Household Survey.
Royal College of Physicians. (2001) Alcohol – can the NHS afford it?.
Stinson, F.S. and DeBakey, S.F. (1992) Alcohol related mortality in the
United States, 1979-1988, British Journal of Addiction
49