Appendix 6: Evaluating Marketing Campaigns
Browse this section:
This DP:UK advice sheet sets out a simple model for measuring the return on investment delivered by place marketing campaigns. It is based on best practice developed by destination managers and research commissioned by VisitBritain.
We are grateful to DP:UK for letting us use their Advice Sheet which sets out a means for evaluating marketing campaigns and measuring return on investment. DP:UK retain the copyright for this appendix.
The model aims to establish a common standard by offering guidance on the estimates and assumptions needed to make calculations.
It is natural curiosity for everyone working in destination management to want to be able to measure the impact of their work, to measure the customer take-up of a project and see what difference it made to the success of their town or city.
Increasingly though, being able to measure return on investment is also becoming a requirement to satisfy external funders, commercial partners and internal performance assessment. Managers are being asked to demonstrate the output of most elements of their work.
This guide focuses on measuring the output of the activity that still sits at the heart of place marketing and on the core component of this: campaigns aimed at attracting leisure visitors.
Perhaps surprisingly, no advice on this nor on any common standards for measurement had come to the attention of delegates who attended the DP:UK seminar workshop ‘Measuring Return on Investment’ in April 2007.
This advice sheet is a response to that workshop as there was a clear wish for guidance that addressed the fundamentals of how to go about assessing and measuring the return on a campaign: to show, for example, how a claimed return on investment of 30:1 was calculated. This advice sheet shows you how to do it, utilising a model developed by drawing on best practice established in Chester & Cheshire and by VisitBritain’s England Marketing Team.
The approach taken outlines the stages evaluation should follow and, to prevent the methodology from being too theoretical, includes a case study in the Appendix. Although the case study is based on a real example, some of the figures used have been amended to reflect the approach outlined here which draws on new Visit Britain findings.
What do you want to achieve?
It is necessary to think through what you want to get out of your research at the outset, as this will affect the way your campaign and research plan is delivered. Issues to consider will emerge as you look through this guide but will include:
- Whether you will conduct a telephone or self-completion survey. This will impact on the data you need to capture from campaign respondents.
- Whether you only want to use your survey to inform the ROI analysis or whether it will have other uses. This will impact on survey design and sample size.
- Whether you will use your survey to establish visitor spend figures. If not where will you source this?
How many people responded to the promotion and how can I get in touch with them again?
To measure how many people responded to a promotion you need to keep a record of the number of enquiries it generated and to be able to separate these from other promotions and general enquiries. Depending on the scale and nature of the campaign, you may use a dedicated telephone enquiry line or web address or ask enquirers to quote a booking reference. Don’t rely on asking them which piece of your marketing prompted their enquiry as many won’t know!
You need to have already thought about how you plan to contact your respondents in the survey stage and to make sure you have their permission to do so in line with Data Protection Act requirements.
You will need to know:
- The cost of advertising
- The cost of fulfilment
- Print or other marketing collateral
- Envelopes and postage
- Fees paid for call handling and data capture
- The cost of your visitor survey and evaluation
The full cost of a promotion would theoretically also include the staff and office costs involved in its planning and delivery. The model proposed here, however, only addresses the marginal costs of the promotion – not those that would have been incurred anyway by the organisation managing it.
Your main decision will be between respondent self-selection and the accurate, but almost certainly more expensive, respondent interviewer selection. Most return on investment surveys are self-selection meaning that respondents choose for themselves whether or not to return a survey form they may have received through the post or by email. Research into non-response conducted by Luton University for VisitBritain has shown there to be a built-in wish-to-please bias in self-completion surveys, as people are more likely to return the form if they have visited a place since receiving its promotional information. If you use a self-selection survey, you should reduce your conversion factor by 23 per cent before you apply it to all respondents.
Telephone surveys where respondents are called until the required sample size is reached are regarded as interviewer selection since the person being surveyed has decided only to take the call and does not know what they will be asked. There is therefore less likelihood of wish-to-please bias and no need to down-weight the results.
The process of obtaining respondents’ telephone numbers sounds complex but is a day-to-day activity for market research companies. It is usually done through a process of number matching with the name and address data supplied when people make their enquiry. Some people’s numbers will not show up when their details are processed; others will show on the Telephone Preference Service register indicating that they do not want to be called and others will choose not to take part in the survey. A rule of thumb is that six names and addresses will be required to secure one interview.
Statisticians have a variety of formulas for assessing the margin of error that applies to different sizes of sample. A useful rule is that a sample of 10 per cent of any respondent group is going to provide a very accurate sample. However, when dealing with very large groups far smaller percentages can reliably be used – you only have to look at the size of samples of most polls reported in newspapers. The size you use will depend on the depth of information you want to achieve from your survey - particularly if you want to subdivide your analysis to discover, for instance, the relative success of different parts of your promotion in generating visits. A general rule is that for destination surveys where all respondents are going to answer most questions a sample of 400 visitors should be both achievable and provide an acceptable margin of error.
If you have decided to go for a minimum sample don’t forget that a lot of respondents won’t have visited, so the information you will be extracting from those that have will be based on a far smaller number than your original group. If you have achieved 400 responses but only 25 per cent have visited, the visit characteristics information you will be gathering will be based on a sample of only 100. In this case you would have needed an initial sample of 1600 to achieve the 400 visitors needed for statistically reliable analysis. If you want to conduct a telephone survey this means that you will probably need 9,600 names and addresses (1,600 x 6 – see 3.1 above) to secure the number of interviews you need.
The survey needs to take place long enough after the campaign to have given people time to visit yet not be so long after that people have forgotten key elements of their visit, such as where and how long they stayed, who they were with and what they spent? VisitBritain suggests that it should take place three to six months after the campaign took place. The best advice is to conduct it as soon as possible after the period the campaign was designed to influence, ie survey a main holiday campaign in the autumn, a Christmas promotion in January. Doing it quickly also means you can release the results to your sponsors while it is still fresh in their minds. If you are aiming to compare the results of a campaign with one you have analysed before, try and make sure that the survey is conducted after a similar period of time for consistency.
The cost of preparing a survey and the opportunity it provides to gain information means that there is often a temptation to use it for more than one purpose. For example, you might want to ask respondents questions about what they did while they were in your destination and what they thought of it. Extra questions are fine providing they don’t compromise the questionnaire’s main purpose by making it so long that respondents become fatigued. This can lead to their giving up midway through or becoming confused about their answers. It is always worthwhile testing your questionnaire on sample respondents to check its length, as well as making sure it is easy to follow with clear options where appropriate for each of the answers.
Being able to anticipate the response rate is important as you can then calculate how many questionnaires you will have to distribute to achieve your target sample – and how much this is likely to cost you to collect and analyse. As a rule the more contact you have had with the respondent the more likely they are to answer your survey.
If you just send your form to a sample of campaign respondents (with a replied paid envelope of course) you may expect to receive 10-15 per cent back. If you have had their permission to send them a survey the return can be as high as 30 per cent. If you are conducting a telephone survey you can advise your research service of the sample size you require and they will quote accordingly.
There is evidence that offering some sort of incentive to return a questionnaire, usually by being entered into a prize draw, can help increase participation. However, it will also add to the wish-to-please bias referred to in 3.1 so it is best avoided.
The only stay that is considered in this model is the first stay. It is a much-debated issue, as some survey respondents will undoubtedly have visited more than once since receiving the promotional information and including their subsequent stays will certainly improve the ROI ratio.
The reasoning behind not including these, however, is the belief that subsequent visits are on balance more likely to have resulted from a positive experience on the first trip than from the promotional material originally received.
Regardless of whether the survey is being conducted on the telephone, in writing or online, it is worthwhile to set the context for it and what you are trying to find out, eg how effective the promotion was. This will help focus respondents’ minds if questions appear ambiguous and will also help avoid people including the whole coach-load they were part of by telling you there were 53 people in their group.
The best way to phrase this question is to provide a list of options such as:
Since you received the destination guide in [state month - be as specific as you can], which of these statements best describes your actions?
- I have visited
- I have booked but haven’t yet visited
- I haven’t visited or booked yet but plan to before the end of the year
- I haven’t visited.
In calculating ROI some destinations have included the "plans to visit" either in full or with a 50 per cent weighting to reduce the wish-to-please factor. This DP:UK model only includes the first two categories omitting the "plans to visit" entirely to make even more sure that the impact of a campaign is not over-estimated.
This is a key question and perhaps the one that has caused the greatest debate among ROI researchers – particularly since it is subjective. Few respondents can ever be completely sure that a particular piece of promotional information prompted their visit. In reality their decision to select one place over another will be based on a range of conscious and unconscious influences. This makes it very important for comparative analysis between surveys that the question is asked and answers calculated in a consistent manner.
The DP:UK model uses a standardised question and five-option answer with a weighting applied to the positive choices:
Did the information received [from the destination] turn your possible visit into a certainty?
- Definitely (responses x 1)
- Probably (responses x 0.5)
- Possibly (responses x 0.2)
- Not at all
- Don’t know
This question is intended to build in the value of people who had requested information purely to help them book visits they had already planned but who were persuaded to stay longer by what they found out.
It is important to make sure that the question is only asked of respondents who have answered that the information did not prompt the visit. The survey will then need to establish how much longer they stayed as a result and whether that extra stay was definitely, probably or possibly a result of the information they received and weighted as above in 3.10.
When asking the length of stay of visits, the survey form should lead respondents to give a precise number of days rather than bands such as 1-3 or 4-7 days as banding will only cause calculation complications and inaccuracies when spend is being calculated. Beware of extremely long lengths of stay which may arise from a respondent coming as a student or staying on to work / live. These will up the ROI but imbalance the results. Best practice is to not include visits beyond 21 days.
The nature of the campaign will dictate whether day visits as well as overnight stays should be included but since this model must be applicable in many different circumstances it considers both to be valid.
The ROI counts the total number of people who visited as a result of the promotional material received not just the person who booked. You also need to know the composition of the party as spend per head for children will be different to that for adults. Beware of extreme party sizes that may skew results. There may be a dozen people in a party if the destination was chosen for a family event or stag / hen do. However a group of 50 is more likely to suggest that the respondent was part of a coach party and it is unlikely that the whole coach load came as the result of the decision by the respondent to book a place.
A useful rule for any type of destination research is always to use the most localised statistically valid information and only to use externally sourced data if good local research is not available. When calculating national campaign data for Enjoy England campaigns, VisitBritain uses average daily spend figures taken from the United Kingdom Tourism Survey. This is fine for national work but the DP:UK model recommends you source local information, ideally as part of your ROI respondent survey or from other reliable visitor or hotel surveys you may have access to.
For spend calculation, you may simply create two categories of day and staying visitors but if you have more detailed data you might subdivide your sample further, typically by type of accommodation used. The case study in the Appendix of this advice sheet uses three categories: Staying Visitors – Commercial accommodation; Staying Visitors – Friends and relatives; Day Visitors.
The main factors you need to be aware of are ensuring you understand whether respondents are speaking on behalf of their group or just themselves and whether they are referring to the whole trip or just a day.
You need to establish which way you want the information to be provided, apply the template consistently and examine the results to look for inconsistencies.
Once all the necessary data has been collated, the process to measure the return on investment of your campaign involves comparatively simple calculation. Use the example in the case study to help you.
The calculation has three stages:
- Estimate how many visitor days campaign respondents have subsequently taken in the destination and how much they have spent.
- Estimate how much of this is attributable to the campaign.
- Divide the attributable spend by the campaign cost to establish return on investment.
To assess total respondent days you need to break down respondents into categories (day and staying visitors as a minimum) and for each calculate:
Number of respondents
x average length of stay
x average group size
= respondent days
To assess total spend, multiply the respondent days in each category by the category’s average daily spend per person.
To assess average spend per respondent day divide total respondent days by total spend.
To assess the number of days influenced by the campaign, multiply the proportionate number of total days that respondents said were definitely, probably or possibly influenced by it, by factors of 1 for ‘definitely influenced’; 0.5 for ‘probably influenced’; 0.2 for ‘possibly influenced’. (See 3.10 above)
To account for the wish to please factor if the survey providing the information has been self-selection, multiply each days sub-total (in 4.5 above) by 0.77 before adding them all together.
To assess the amount of spend influenced by the campaign, multiply the number of days influenced by the average spend per respondent day.
To assess the number of days and amount of spend extended by the campaign, multiply the total number of respondent trips by the percentage of people who said that the campaign prompted an extension and follow steps 4.2 to 4.6 above.
To assess the number of days and amount of spend attributable to the campaign, add the totals of days and spend influenced to the totals of days and spend extended.
To assess the Return on Investment, divide the respondent spend attributable to the campaign by total campaign cost and express as a ratio.
All models that aim to calculate a monetary return arising from an activity are providing no more than an estimate and their results are best compared one with another or over time: in the case study in the Appendix, for example, to show how much more or less successful this year’s campaign was compared to last year’s.
All models are also only as good as the data used and are liable to be let down by the weakest element. This model aims to be as reliable as possible and has factored out elements that can over-inflate ROI, including the degree to which a promotion influences action, wish-to-please bias and double counting repeat visits.
All the same there are still limitations you should be aware of:
Look our for any odd patterns in your results such as an average party size of above three or an average length of stay at a predominantly short-break destination of more than four nights. There may be a good reason for this but it may also be that a few rogue answers have skewed the sample. The more you conduct this sort of analysis the more you will be able to anticipate the range of values likely to appear as key data and spot those that need checking.
In the survey you are asking visitors to recall the nature of their visit, how much they spent. This is open to much inaccuracy. Your only real counter to this is to be consistent in the way you ask questions and analyse findings with the expectation that people’s recall will potentially be flawed!
The model only estimates the direct spending in the destination by visitors who have responded to the promotion by the time the survey takes place. It takes no account of spending by respondents who may visit later and makes no estimate of the value that may be derived from the profile of the place increasing as a result of the campaign. By only measuring the tangible results not the intangible ones, it allows marketing managers to claim with some justification that the real value was even higher.
Our case study is a destination promotion for the fictional Anytown, lasting from January to April and fulfilled by a conventional multi-use visitor guide containing information about the place and details of accommodation. It is designed to convert interest into bookings.
The promotion’s marketing objective is simply to maximise bookings for the destination at the highest level of return. To assess this we need to be able to measure the financial return the campaign has brought to the destination.
Anytown’s campaign research had two main objectives:
- To measure the return on investment it delivered.
- To assess the effectiveness of the different types of advertising the campaign deployed and the different advertisements placed.
In order to provide meaningful data to measure the effectiveness of the various advertisements it was decided that a sample of 3,000 respondents would be necessary. To obtain the depth of information required it was decided to use telephone interviews. It was agreed that rather than requesting expenditure information from each respondent, spend data would be used from the Destination Benchmarking survey carried out in Anytown the previous year.
Anytown’s campaign was designed to deliver a minimum of 30,000 brochure requests, necessitating the employment of a professional mailing house. Requests came mainly by telephone and in writing so a service is selected that also incorporates a call centre.
Names and addresses of all respondents were selected and the advertisement they responded to logged, so that Anytown’s marketing team was able to assess the cost per response of each. Once the survey interviews had taken place the team were also told the proportion of respondents to each advertisement that subsequently visited. This allowed them to calculate the cost of conversion for the campaign and for each advertisement.
Anytown’s campaign costs break down as:
|Print||£8,958||Since only 25% of the visitor guides produced for the year were used for this promotion the cost shown here is 25% of the cost of the full print run|
|Data capture and postage||£40,711||A mailing house was employed to take calls, service coupon responses and capture details of respondents for subsequent survey work|
|Analysis||£7,000||A research company was employed to conduct a telephone survey of 3,000 respondents to obtain details of visits in response to the promotion |
|Calculation||0||This work was carried out in-house by the marketing team|
Anytown’s telephone survey of 3,000 respondents took place in September, 4 months after the last advertisement for the campaign was placed. The survey was drafted by its research company following guidance on objectives provided by Anytown’s Marketing Manager. No participation incentive was offered.
The survey contained 20 questions written according to the guidelines in this advice sheet. They concentrated on the key ROI data requirements of:
- Respondents’ profile
- Whether they had visited Any-town since receiving its brochure.
- How far the promotion influenced their visit
- Whether a pre-planned stay was extended due to the promotion
- Length of stay
- Party composition
- Type of accommodation used
They are also asked three questions to help future campaign planning and delivery:
- whether they have internet access
- whether they would like to receive the following year’s guide
- whether they would be prepared to take part in more research in the future
Although one of the research objectives was to find out the response and conversion rate of individual advertisements, no questions needed to be asked about this as the research company had already coded each respondent by the advertisement that prompted their brochure request.
In the end 2,947 interviews were conducted with people who requested the guide. 761 of these (26%) had subsequently visited. These respondents became the core sample for the rest of the survey. The survey results that will enable the ROI evaluation to take place were then compiled and presented to Anytown’s Marketing Manager
|Campaign respondents||35,711||Mailing House|
|Overnight conversion: commercial accommodation||14%||Survey|
|Overnight conversion: visits to friends / relatives||2%||Survey|
|Day visit conversion||10%||Survey|
|Respondents who visited||9,285 (35,711 x 26%)
|Average party size||2.7 people||Survey|
|Average length of stay||3.5 nights||Survey|
|Visits definitely influenced by campaign||21%||Survey|
|Visits probably influenced by campaign||27%||Survey|
|Visits possibly influenced by campaign||34%||Survey|
|Pre-planned visits extended by campaign||14%||Survey|
|Average extension to stay||1.2 nights||Survey|
|Overnight spend / visitor||£70 (Commercial)||Destination Benchmarking|
|Overnight spend / visitor||£35 (VFR)||Destination Benchmarking|
|Day visit spend / visitor||£26||Destination Benchmarking|
|No response / wish-to-please factor||n/a||Due to survey being interviewer selection not self-selection|
|Days and spend by respondents to compaign|| || || |
| || || || || || || |
|Respondents: 35,711|| || || || || || |
|Respondents who had visited by the time the survey took place: (26%) 9,285:|| |
| - Overnight in commercial accommodation: 14% x 35,711 = 5,000 respondent visitors (54% of visitors)|
| - Overnight staying with friends or relatives: 2% x 35,711 = 714 respondent visitors (8% of visitors)|
| - Day visits: 10% x 35,711 = 3,571 respondent visitors (38% of visitors)|| || |
| || || || || || || |
|Category||Respondent visitors||x Length of stay||x Group size||Respondent days||x Daily spend||Respondent spend|
|Overnight - commercial ||5,000||x 3.5 days||x 2.7||47,250|| x £70||£3,307,500|
|Overnight - VFR||714||x 3.5 days||x 2.7||6,747||x £35||£236,145|
|Day visit||3,571||x 3.5 days||x 2.7||9,642||x £26||£250,692|
|Total days and spend||9,285|| || ||63,639|| ||£3,794,337|
| || || || || || || |
|Total respondent days in Any-town: 63,639|| || || || |
|Total respondent spend in Any-town: £3,794,337|| || || || |
|Average spend per respondent day: £3,794,337 / 63,639 = £59.62|| || |
| || || || || || || |
|Days and spend attributable to compaign|| || || || |
| || || || || || || |
|Days influenced by campaign|| || || || || |
|Days definitely influenced 21% (influence weighting 1)|| || || |
|Days probably influenced 27% (influence weighting 0.5)|| || || |
|Days possibly influenced 34% (influence weighting 0.2)|| || || |
| || || || || || || |
| ||Respondent days||x Percentage||Sub-total||x Influence weighting||x Wish to please factor (0.77)||Days influenced|
|Days definitely influenced||63,369||x 21%||13,364||x 1||n/a||13,364|
|Days probably influenced||63,369||x 27%||17,183||x 0.5||n/a||8,592|
|Days possibly influenced||63,369||x 34%||21,637||x 0.2||n/a||4,327|
|Total days influenced|| || || || || ||26,283|
| || || || || || || |
|Total days influenced: 26,283|| || || || || |
|Total spend influenced: 26,283 days x £59.62 (average spend per respondent day) = £1,566,992|
| || || || || || || |
|Days extended by campaign|| || || || || |
|Percentage of trips extended: 14%|| || || || || |
|Number of trips extended: 9285 x 14% = 1,300|| || || |
| || || || || || || |
|Category||Trips extended||x category percentage||Sub-total||x Extension to stay ||x Party size||Days extended|
|Overnight - commercial ||1,300||x 54%||702||x 1.2 days||x 2.7||2,274|
|Overnight - VFR||1,300||x 8%||104||x 1.2 days||x 2.7||337|
|Day visit||1,300||x 38%||494||n/a||n/a||n/a|
|Total days extended|| || || || || ||2,611|
|Extended days||Extended days||x Percentage||Sub-total||x Influence weighting||x Wish to please factor (0.77)||Days influenced|
|Days definitely influenced||2,611||x 21%||548||x 1||n/a||548|
|Days probably influenced||2,611||x 27%||705||x 0.5||n/a||353|
|Days possibly influenced||2,611||x 34%||888||x 0.2||n/a||178|
| || || || || || ||1079|
| || || || || || || |
|Total days extended: 1079|| || || || || |
|Total spend from extended trips: 1079 days x £59.62 (average spend per respondent day) = £64,330|
| || || || || || || |
|Campaign impact|| || || || || || |
| || || || || || || |
|Days influenced ||26,283|| || || || || |
|Days extended||1,079|| || || || || |
|Days attributable||27,362|| || || || || |
| || || || || || || |
|Spend - influenced days||£1,566,992|| || || || || |
|Spend - extended days||£64,330|| || || || || |
|Spend attributable||£1,631,322|| || || || || |
| || || || || || || |
|Return on investment: || || || || || |
|Respondent spend: £1,631,322 / Campaign cost: £94,814 = 17.2 : 1|| || |