UNICEF Measurement Framework – A Year of Learning – UNICEF

/in /by

Category: Best multi-market reporting

Client: UNICEF

Campaign title: UNICEF Measurement Framework – A Year of Learning

Member: Main partners include Gorkana (a Cision company) and Ketchum Research

AMEC Awards 2016

OBJECTIVE/BRIEF

UNICEF has been leading a transformative initiative that aims to introduce an evidence-based approach to communications. In this sense, the Global Communication and Public Advocacy Strategy (2014-2017) not only helped unify the outlook on communications across UNICEF’s offices in over 190 countries, but it also led to the development of a dedicated multi-market monitoring and evaluation (M&E) framework. A natural next step is to provide an overarching measurement methodology. Simultaneously, it aims to capture the diversity across a geographically decentralised organisation, present in over 190 countries, and to allow learning from the wealth of information and experiences globally in more systematic ways.

The M&E framework, which was developed in full alignment with the Barcelona Principles and with the involvement of Ketchum Research and Gorkana, outlines a measurement rationale rooted in the Theory of Change. It endeavours to measure the impact of the communications function, its alignment with UNICEF’s strategic policy and programme priorities, and its contribution to the organisational objectives, grounded in protecting, and advocating for, the rights of every child. To this end, the framework proposes a set of Key Performance Indicators (KPIs), which were developed taking into account the strategy’s three main objectives: 1) be the leading voice for and with children, 2) reach 1 billion people and 3) engage 50 million people by 2017.

Even though most KPIs are specific, the framework allows different country offices to expand their measurement criteria to capture the local contexts and to best inform the work of local communications teams giving them the flexibility to track the metrics that reflect their work.

The KPIs tracked at global level – aggregated by context – can be applied to inform decisions and budgets, contribute to collective learning, and ultimately shape and strengthen communications and public advocacy efforts across UNICEF.

 

STRATEGY

Translating the global framework into the local context represents one of the most relevant challenges: the M&E framework was therefore developed with an iterative mindset. After a consultative process with the pilot and early-adopter countries, the collected recommendations have been incorporated and reflected in the list of suggested KPIs. As part of the implementation plan, the Division of Communications (DOC) in UNICEF HQ has been closely supporting, training and guiding over 30 country offices and 24 National Committees through a rigorous process.

Firstly, local communications teams are introduced to the M&E framework through DOC supported webinars, bilateral virtual meetings, and/or guidance documents such as toolkits and handbooks. Secondly, communications teams develop their country scorecards using the most relevant KPIs under each category of voice, reach and engagement. Thirdly, they hire a local or regional media monitoring company to help them monitor and analyse on a continuous basis various campaigns and initiatives. Finally, they report on quarterly bases which are then aggregated into the global scorecards (that are separately tracked at the global level, capturing all 27 KPIs).

 

EXECUTION/IMPLEMENTATION

The first year of implementation witnessed tremendous collaboration between DOC and offices globally on the one hand, and with internal partners across other divisions of UNICEF, on the other. While constantly learning from the established process, baselines have been defined in HQ and in implementing countries, with the idea to capture data to realistically define targets and to establish benchmarks for future analysis and assessments.

The key metrics that DOC has been tracking so far are considered an initial step towards developing our data analysis and research. The overall aim is not only to track progress against the main goals of the strategy, but also to help inform communications plans and strategies to ensure our collective advocacy and communications efforts drive change for children.

In this sense, the first year of the implementation was a year of collective learning.

 

CONCLUSIONS

The following are several key lessons that the global M&E team responsible for the programme learned after extensive consultation.

  1. Keep it simple
    In order to allow the M&E framework to be adapted in various countries – both in size and situation – simplicity and flexibility are essential to the implementation of such a complex global programme. Initially, we developed an exhaustive list of KPIs, but we soon discovered that we needed the flexibility to tailor these to each country where budget constraints, office size and/or local context may or may not have made it possible to track all the KPIs. We continue encouraging countries to do all they can to track as many KPIs as possible – mainly those that matter the most to them – but are flexible enough to allow them to move forward at their own pace. The Handbook for the Implementation of KPIs, developed by DOC, has proved to be a valuable tool for explaining the main concepts and rationale, and to detail techniques to calculate each of the KPIs.
  2. Be serious about evidence-based communications: M&E should not be an afterthought
    This is the first time that UNICEF has undertaken such a strategic approach to measurement and evaluation in its communications efforts. All strategies, including the global strategy and national strategies (based upon the global strategy, adapted to the local context) are informed by data, collected both in advance of the implementation phase and during it. This provided the opportunity to take the time to reflect on what contributed to the success – or the failure – of the campaigns and initiatives UNICEF implements. This was proven as DOC and many country offices organised evaluation sessions to identify areas for improvement and to better interpret achievements in order to replicate or bring to scale models of success.
  3. Data and impact analysis are helping us change the way we do business; the way we communicate
    This new approach to measurement is contributing to changing the way we work; the way we communicate. We are now able to get a better idea of the impact and the reach of our communications initiatives, identifying gaps and progress. The use of frequent data monitoring has informed timely course corrections in communications plans and strategy and public advocacy efforts. This has also spearheaded efforts to be more responsive to the reality on the ground and has helped accelerate sustainable results for all children.
  4. Hire the right people, develop human capital and invest in capacity building
    Investing in human capital is fundamental. Creating new positions, bringing in new skills and developing the capacity of the existing communications teams should be a priority to ensure the proper implementation of such an ambitious M&E plan. Candidates with relevant and up-to-date skills allow the organisation to remain abreast of the ongoing changes in the digital and media world.
    DOC, in collaboration with the seven Regional Communications Officers, are regularly providing training and capacity building to field colleagues through webinars, bilateral calls and guidance tools.
  5. Work with the right partners – you cannot do it alone!
    It is simply not possible to implement a truly global and multi-faceted framework like the one designed, without collaborating with external vendors and partners. DOC is working with several industry leaders that provide technical support, including Ketchum Global Research & Analytics. The service provided by Gorkana, the vendor contracted for the global monitoring and analysis, is complemented by Brandwatch (social media); Factiva (paid content); TV Eyes (TV and radio clips) and other tools. DOC encourages offices to use external media monitoring companies to assist in tracking KPIs, even in the most well-funded and staffed offices.
    Having an external company provides both quantitative but also qualitative analysis to better inform communications activities. A local media monitoring company can also fill the gaps that a global provider may not be able to address (local languages/dialects, print media, as opposed to online editions, etc).
  6. Seek advice and do not reinvent the wheel
    Others may have already found a solution to the problems you encounter. At the HQ level, Ketchum Global Research and Analytics (KGRA) provides strategic advice and helps coordinate the work of an advisory board, that includes representatives from other large global operations, including private sector companies, NGOs and academic institutions. The advisory board meets bi-annually and serves as a space to provide UNICEF with guidance and feedback on the approach and direction our measurement work is taking at a global level. The forum is also a place to share expertise and to discuss potential solutions and additional innovations to common challenges. UNICEF is greatly benefitting from the expertise and know-how of these senior communications professionals.
  7. Fail fast, fail cheap, try again, improve and scale up
    Recent experience has shown that when support and guidance are shared, Country Offices can absorb them (as appropriate) and then generate greater results than initially envisaged. There have been remarkable innovations and agility from a wide cross-section of offices – and when DOC can act as a knowledge enabler, facilitating the exchange of specific Country Office experiences, new ideas and ways of working can be contagious and replicable. Each month, DOC produces a monthly compendium of examples of how the “voice, reach and engagement” pillars of the strategy are being put into action at country level, as well as facilitating case studies on the M&E framework implementation that can be useful learning tools for other offices tracking KPIs.
  8. Share data and insights in a timely manner. Do it right!
    Sharing data and insights is important, but in order for it to have impact, it must be clear and shared in a timely manner. Data can be useful in decision-making – but it can also create confusion if not explained well. It is therefore important to be clear when communicating to audiences that may not understand the technicalities, or the assumptions behind the metrics.
    Consistency and timeliness of data is essential. In 2015 DOC released two new internal communications products: a) a set of daily email alerts that keep colleagues informed on the main articles in top tier media that mention UNICEF and issues related to children and b) a monthly newsletter that includes the best examples of external communications and public advocacy campaigns from UNICEF teams around the globe. DOC continues producing the bi-weekly communications highlights that are also widely distributed internally.
    We are also packaging information for senior managers in a straightforward and transparent way, allowing them to better understand UNICEF ́s presence in the global media landscape.
  9. Measurement is an art: do not try to apply the same model in all countries
    Our aim is to have all country and National Committee offices implementing the M&E framework. This demands the creation of tailored solutions to respond to local needs and realities in each of the countries. We quickly realised it would be a mistake to try and use the same model everywhere. Each country has its own needs and local contexts and this includes the various human and financial capacities of each office.

DOC has since expanded the scope of its research to address the following questions:

  • Who’s our audience?
    To assist UNICEF’s digital production team in creating more relevant and engaging content for UNICEF’s global audience, the team has started to conduct literature reviews and evaluate the demographic profile of various social media platforms.
  • What do these numbers mean?
    To set meaningful benchmarks against which to evaluate the posts, UNICEF has been developing high-level global scorecards on a weekly and monthly basis. The new scorecards additionally start to evaluate emerging social media networks and platforms, beyond Facebook and Twitter, including Instagram, Pinterest, LinkedIn, etc.
  • What learning can we share?
    DOC has been working on knowledge sharing. Newsletters showcase the best initiatives as a means of providing concrete examples of success, so that other country offices can adopt and further tailor their own efforts without reinventing the wheel. This is in line with the organisational vision of efficiency and effectiveness.

Name of person entering: Arturo Romboli
Email: [email protected]
Telephone: +1-917-478-9508