Summary
CARMA has developed and launched a new methodology to solve the single most challenging aspects of measurement and evaluation. Our Framework Consulting Methodology empowers teams to make sense of multiple datasets and demonstrate impact in a meaningful and compelling way. Our product advocates for best practice, utilises AMEC’s resources has been embraced by clients globally. Our methodology has enabled us to win over 7 AMEC Awards and contributed to our 25% growth rate in Insights service. The current pipeline for CARMA’s new methodology is $2M USD.

Objective/Brief
CARMA’s Framework Consulting Methodology meets clients where they are on their measurement journey and prioritises education and best practice.

CARMA has developed this product to help clients overcome the most challenging aspects of measurement and evaluation:

• Communications teams are inundated with data and many teams lack the time, structure and knowledge to bring
  different datasets together in a single integrated measurement program
• Communications team are time poor and find it challenging to demonstrate their contribution to the business in terms
  of organizational objectives and impact.
• Often, the output metrics that are readily available lack qualitative analysis and do not tell a meaningful story of impact
• Lack of alignment between Communications team members as well as lack of alignment with organizational objectives
  and stakeholders often impedes progress with measurement programs.
• Our new product suite allows clients to spend the time thinking about what they’re doing and why it matters, talking
  about their objectives and how they believe their work delivers impact, and focusing them on how to prove it.
• A third party bringing in expertise and experience gives credibility to the process, and also allows us to challenge them
  with an external perspective.

Strategy
CARMA’s leading experts developed this new methodology to productise the most challenging aspect of measurement and evaluation. It was designed by Richard Bagnall and Orla Graham and productised and rolled out globally in 2022. COVID restrictions limited in-person meetings and the number of attendees, presenting a major obstacle for the launch and success of the new product. The workshops were designed to be in-person, dynamic and interactive, with framework materials in poster format on the walls and post-it notes to link specific measures with communications objectives.

Our product strategy clearly needed to adapt. In early 2022, we re-designed the workshop methodology and format to allow virtual delivery of engaging and interactive workshops. Online workshop spaces were developed using conferencing technology and key AMEC resources in order to deliver online workshops. Specifically, we added the following online tools to our virtual workshop:

• AMEC’s Maturity Mapper (tool to benchmark measurement maturity against your sector/globally)
• AMEC’s Integrated Evaluation Framework
• AMEC’s Planning Resources
• Consulting and Prioritization techniques created within Microsoft Miro (team collaboration and brainstorming tool)

While our virtual product design was shaped by COVID restrictions, our strategy and objectives for this product remained clear:

1. CARMA’s strength lies in our ability to bring together multiple outputs, out-takes and outcomes datasets in a meaningful and compelling way for Communications teams to understand audience and organizational impact
2. We need to demystify and simplify measurement
3. It needed to work anywhere in the world (in-person, virtually and across a diverse range of languages and cultures)

Execution/Implementation

CARMA’s Workshop Methodology has three phases which fundamentally evolve and elevate measurement programs. Initial consultations tailor the phases to organizations’ specific challenges.

Phase 1: Discovery (includes audit, stakeholder mapping, business objectives)
CARMA audits the measurement and research which currently exists within an organization, revealing measurement
maturity and available data.
We conduct stakeholder mapping and engagement, by interviewing stakeholders across teams (tactical comms, marketing and insights teams, strategic directors and C-suite, to ensure we have a realistic and pragmatic perspective), asking what they like/dislike about current reporting, their interpretation of objectives, their typical activities, what a ‘win’ looks like to them.
We explore documentation on objectives/strategy with the client, to dissect how business objectives cascade down to communications objectives, and translate into strategy and tactics.

Phase 2: Workshop and interactive framework design
CARMA presents Discovery phase findings to key stakeholders, addressing the gaps and challenges identified. CARMA’s product design includes interactive sessions on the Barcelona Principles, the Integrated Evaluation Framework and the Maturity Mapper to establish best practice benchmarks and identify areas for improvement.
The primary component is an interactive workshop where participants design an integrated framework using a real example.
When conducted virtually, we use pre-prepared materials on Miro and break-out rooms to ensure all participants remain engaged contributors.

Phase 3: Framework Delivery and Implementation
A templated deck is customized, summarizing the client’s challenges, our findings, pointing to AMEC resources and includes the framework structure with recommended metrics. CARMA’s ‘See, Think, Do’ model is a crucial element of this. Overlaid to each section of the framework, the model mirrors the marketing funnel. It ensures ample focus on measuring how audiences are reached, what they think, and what they do. It creates structure, prevents data-overload, and when translated to match the organization’s marketing funnel terminology, speaks the language of the business.

Effectiveness of Assignment
CARMA’s new methodology has now been productised and scaled globally. CARMA’s leading measurement experts in Asia, MENA and Europe all now provide this service to major global organisations and smaller local organisations. The four biggest achievements of our new product to-date:

• This approach has helped our clients and CARMA win over 7 AMEC Awards and 1 PRCA APAC Award
• We have established a strong pipeline of new revenue from new markets during 2022/23, with our pipeline at $2M USD
  at time of writing.
• This product has helped the CARMA business by growing revenue for Insights services and has proved to be an important way to upsell existing clients and retain valued clients.
• Clients have implemented and embedded our frameworks and are now seeing tangible benefits of more sophisticated measurement, which identify and articulate the real impact of comms for the organization.

The structure of our new methodology brings results for clients as we keep agile and adapt.
In 2022, we used launched our new methodology in Asia Pacific and have conducted workshops with a number of regional and global organisations in the region. Our in-depth consulting engagements in markets such as Japan have strengthened our product as we tailor it to meet the demands of global clients. In April 2022, we introduced live translators to our interactive workshops and translated much of our product materials into Japanese in order to meet the needs of global clients.

Global recognition for our new methodology from AMEC and the PRCA, 2022, 2023.

• PRCA Asia Pacific Measurement & Evaluation Award 2023, BMW Group Asia
• GOLD: Step change award for the best improvement of a measurement journey, HSBC
• GOLD Step change award for the best improvement of a measurement journey category, Honda
• SILVER: Step change award for the best improvement of a measurement journey category, Dubai Police
• BRONZE: Most effective planning, research and evaluation in the public and not for profit sectors, Dubai Police
• BRONZE: Most effective planning, research and evaluation: Middle East and Africa, Dubai Police
• SILVER: Step change award for the best improvement of a measurement journey category, Tourism Ireland
• BRONZE: Best multi-market reporting, Tourism Ireland

“Working with the team at CARMA is like working with consultants who really understand our business, think strategically, and are genuinely interested in helping us understand the impact of our work” – Preeti Gupta, BMW Group Asia

Please see supporting materials for more client testimonials.