Category: Innovation award for new measurement methodologies

Client/Entering Company: Public Health England

Campaign title: Change4Life Sugar Smart 

Company Name: Change4Life 

Summary

Change4Life is the British Government’s flagship brand focused on tackling childhood obesity. For the January 2016 Sugar Smart campaign, Public Health England designed an evaluation that tracked the impact of communications all the way through the AMEC valid metrics framework, proving its influence, not just on communications outputs and outtakes, but on robust outcomes, including decreases in sales of added-sugar products and on the amount of sugar children consumed, a tangible step forward in the fight against obesity-related illness.

Objective/Brief

Change4Life targets families with children aged 5-11, to reduce childhood obesity by influencing knowledge, attitudes and behaviours. A multifaceted programme, with several campaigns throughout the year, the initiative ran for six weeks during January/February 2016.

The SMART objective was to ensure that, as a direct result of the campaign, at least 400,000 families downloaded the new Sugar Smart mobile app by 28 February 2016, and used it to reduce the amount of sugar in their children’s diets.

The campaign was developed using our behaviour change model, which has been refined through the programme’s seven-year history. This model is alert -> motivate -> support -> sustain.

The role for the campaign was set out at each stage:

Alert

  • Engage families with the issue of consuming too much sugar and personalise it, so that that the message is relevant for them

Motivate

  • Encourage parents to take control of their children’s diets

Support

  • Drive downloads of the new Sugar Smart app
  • Engage families with the wider Change4Life campaign by offering them free support

Sustain

  • Support parents to buy fewer high-sugar products, particularly sugary breakfast cereals and added-sugar drinks (since these are the largest single contributors to children’s daily sugar intake)
  • Ensure that children consume less sugar each day

An evaluation approach was required that could measure the impact of the campaign throughout this model, aiming for statistically significant improvements against the following 2015 baseline:

  • Recognition: 79% total recognition of Sugar Swaps campaign among mothers with children aged 5-11
  • Sign-ups: 368,474 (to be measured against downloads in 2016)
  • Packs: 14% of mothers with children aged 5-11 had seen or received a pack
  • Behaviour change: 23% of mothers of 5-11s in the target groups made a swap

Strategy

While the data has consistently shown that Change4Life engages families at scale, establishing a link between communications and clinically significant behaviour changes has always been challenging. However, for a campaign looking to improve the lives of children, it was essential we understood whether it worked.

The 2016 Sugar Smart campaign provided an excellent opportunity to prove the link to clinically significant outcomes, including sales of featured added-sugar food categories, and children’s overall dietary intake.

This required an evaluation approach that measured the campaign’s impact throughout the behaviour change journey, specifically:

  • At the alert stage, we needed to understand whether parents were noticing the campaign and taking on board key messages (e.g. that most exceed the maximum daily intake of sugar); furthermore, we needed to see whether parents found this personally relevant (as opposed to a problem for other families)
  • At the motivate stage, we needed to understand whether parents were shocked by this information and wanted to act as a result
  • For the support stage, we needed to know whether families were downloading and using the new app, whether they found it easy to use, and whether it supported them to make changes
  • For the sustain stage, we needed to establish whether this app usage translated into different purchase habits and (since it is what is eaten, not what is purchased, that matters) whether children were consuming less sugar as a result.

The evaluation needed to keep track of a complex range of communications outputs (ratings, coverage, information packs provided etc.), outtakes (changes in awareness, knowledge and attitude), immediate outcomes (downloads and usage of the app), intermediate outcomes (sales of high sugar product categories) and longer-term outcomes (the amount of sugar children consumed).

Finally, the evaluation needed to address challenges such as:

  • Social desirability: the tendency of survey respondents to claim behaviours that are believed to be “good”
  • Optimism bias: the tendency of respondents to recall “healthy” behaviours (such as eating an apple) while conveniently forgetting “unhealthy” behaviours (such as eating a bar of chocolate)
  • Underlying trends: many added-sugar categories are already in decline or experience a seasonal sales depression in January (when people go on diets); so it was necessary to account for the impact of the campaign over and above these
  • Substituting one bad habit for another: as when people cut out sugary drinks (such as colas) but replace them with juices (which can contain as much sugar).

This was no easy task, and no single research method could provide the richness of data need to meet this requirement. Accordingly, Public Health England brought together an evaluation approach in concert with its communications agencies, its evaluation partner (Kantar Public), a retail partner (Tesco), and Newcastle University, to devise a holistic evaluation protocol. This included a basket of measurement approaches, moving beyond standard communications metrics, to include experimental approaches, academic studies and use of industry data.

Execution/Implementation

This complex evaluation task was coordinated by Kantar Public, the independent market research agency to ensure objectivity in the collection, analysis and interpretation of the data, with oversight from PHE’s insight and planning team. We used multiple data sources to look at impacts in different ways so that we could assess the project’s success against its objectives.

The main elements of the evaluation involved:

Quantitative tracking study
Setting up a bespoke tracking study:

  • Methodology: Pre and post surveys of 1,000 nationally representative mothers with at least one child in the target age group, delivered via an online survey
  • Rationale: This measured awareness of different elements of the campaign, plus changes to knowledge, attitudes and behavioural intent

Purchase data
Analysing trends in purchasing of key product categories:

  • Methodology: We used Kantar Worldpanel, which collects data from 30,000 households to provide a single source of consumer and shopper insight. We looked at sales data of key categories during the campaign period compared to the overall trend in sales over the year to see if the campaign may be driving a greater shift in sales
  • Rationale: This enabled us to separate out the impact of the campaign from long-term category trends and assess impact against our key categories

Controlled study using digital screens
Setting up a unique experiment using digital screens, to measure the impact of the presence of the campaign creative on sales of featured products:

  • Methodology: This innovative methodology involved matching 30 test stores with 30 control stores using 16 weeks of pre sales data across all Tesco stores in England. We then displayed Change4Life communications at the test stores – but did not at the control stores. We monitored sales data during and after the campaign period to assess the difference in sales between the test and control.
  • Rationale: This enabled us to confidently attribute any changes in categories sales to the communications campaign

Dietary intake study
Commissioning an dietary intake study with Newcastle University, to provide best-in-class analysis of changes to children’s diets:

  • Methodology: We used the INTAKE24 system, developed by Newcastle University to measure consumption before, during and after the campaign period. This system was designed to make it easy for participants to recall and record the totality of what they have consumed during the intervention. It is based on a 24-hour recall. Foods are entered as free text in the ‘quick list’ phase of the recall and the user is then asked to match each food to a food in the database. All foods in the database are coded to the National Diet and Nutrition Survey Nutrient Databank, so that an accurate assessment of sugar contents can be made. Once the food type has been selected the user is asked to estimate portion size. The portion sizes depicted in the tool are based on the portion sizes recorded, for each food, by participants in the National Diet and Nutrition Surveys carried out in Great Britain. The system includes an extensive range of prompts for items commonly consumed together, checks for low energy reports, missing drinks and long-time gaps. Dietary data for each child for a Saturday and Sunday were entered by parents at multiple time points throughout the day. In total, the diets of over 500 children aged 5-11 were studied. This enabled us to examine differences between boys and girls, and older and younger children.
  • Rationale: This methodology allowed us to analyse the consumption of children before, during and after the study to assess whether the intervention helped to reduce children’s sugars intake.

In addition, these sources were supplemented with digital metrics (including app usage, scans of products, social media metrics and website visits), PR metrics and metrics from face to face events (numbers of participants and exit survey responses).

Effectiveness of Assignment

The evaluation approach met Public Health England’s needs: we can now confidently state that a communications campaign drove changes to parental purchasing habits, leading significant reductions in the purchase of featured high-sugar product categories, and in children’s intake of sugar. The evaluation demonstrated that these changes were over and above underlying trends in the market; moreover, it showed that reducing sugar from one source was not compensated for by increasing sugar from other sources.

The multiple sources and methods used (claimed data from parents, usage data from the app, sales data from retailers and dietary intake data from Intake24) all tell a consistent and compelling story: in response to the campaign, parents used the app to reduce the sugar content of their children’s diets.

Since all data sources concur, Public Health England has had the confidence to repeat and expand the programme to cover salt and saturated fat, which is currently running as the “Be Food Smart” campaign in 2017.

In line with the AMEC Valid Metrics Framework, we have reported our results in terms of awareness, knowledge, consideration, preference and action.

Awareness:

The tracking study established that 70% mothers with children aged 5-11 were aware of the campaign.

Knowledge:

Similarly, the tracking study indicated that knowledge of the maximum amount of sugar children should consume increased from 5% before the campaign to 18% after.

It also showed a significant increase in the proportion of mothers who recognised that their children ate more sugar than was good for them (from 41% before the campaign, to 47% afterwards).

Consideration and Preference:

The tracking study indicated that:

  • 79% of mothers said the campaign made them think about how they could limit the amount of sugar their child eats
  • 76% agreed that the campaign made them wonder if their child was eating more sugar than they had thought
  • 75% agreed that the campaign made them more concerned about the amount of sugar their child eats

Action

Data from Tableau showed that the campaign drove immediate, measurable action:

  • Over 2 million people downloaded the Sugar Smart app; 200,000 of these downloads happened on the first day of the campaign (the PR launch, before any advertising had broken)
  • The app was used over 14 million times, with over 8 million successful scans
  • One in four downloaders used it for at least a month.

The tracking study established that 75% of all mothers who were aware of the campaign said they had made changes as a result.

The test and control study showed that the campaign led to a 4% decrease in sales of sugary cereals, both during the campaign period and 8 weeks later, a 3% decrease in sales of sugary drinks and a 4% increase in the sales of diet drinks during and post campaign in the test stores compared to the control.

The purchase data analysis confirmed a greater decline in purchasing of our key sugary categories during the campaign compared to the overall trend over the year.

Finally, the dietary intake study enabled us to establish the impact of the campaign. However, the results of this are not yet publicly available.

Results were submitted to the Parliamentary Under Secretary of State for Public Health & Innovation and will be published in Public Health England’s 2017 – 2020 Marketing Strategy. They are also currently being written into a an academic paper for publishing so that the results can be shared widely to help others with campaign planning.