Evaluation of the Climate Challenge Fund

The Climate Challenge Fund (CCF) was a Defra-run fund designed to improve awareness of - and attitudes towards - climate change. Between summer 2006 and March 2008 the CCF allocated £8.5m to 83 projects across England. Some of these were on a national scale whilst others targeted local or regional audiences. Methods of engagement also varied between one-off, ‘passive’ interventions such as radio adverts to face-to-face repeat contact over a long period.

Objectives
In January 2008, Brook Lyndhurst was commissioned to conduct a review of the CCF and specifically to:

• Explore project outputs – defined by Defra as “anything done or produced by a project”;
• Analyse project outcomes – defined by Defra as “overall attitudinal change and any other secondary outcomes that may be attributable to the CCF projects”; and
• Draw some overall conclusions on the effectiveness of the CCF

Methodology
Brook Lyndhurst was appointed towards the end of the CCF's two-year programme. The evaluation therefore relied upon data reported by funded projects in their final evaluation reports, as well as monitoring forms completed by projects over the course of the fund. Analysis of this was supported by qualitative research - namely, face to face interviews with 13 projects and telephone interviews with another 14.

Findings
Because of inconsistencies in the data collected and reported by projects, we were more limited than we might have hoped in the amount we could say definitively about the performance of the Fund. Specifically, we were unable to take a view on the overall success or otherwise of CCF projects in raising awareness and generating positive attitudinal change. For the limited number of projects for which we had data, there appeared to have been considerable variation, though the aggregate picture was one of neutral or very modest positive shifts.

We were, however, able to say rather more about the likely reach of the outputs developed by projects under the Fund. We explored reach from two angles:

• Direct contact – the number of occasions when projects made direct, face to face contact with individuals from their target audience – i.e. the “audience member” was aware of the approach and responded to it in some way; and
• Opportunities for communicating climate change – the number of occasions when project activities opened up a potential communications channel to a member of the public, irrespective of whether or not that individual actually became aware of the approach (i.e. a person may or may not notice or read an article in a newspaper).

Opportunities for communicating climate change
Notwithstanding the limitations of the available data, we estimated that at least 48.5 million opportunities for communicating climate change were generated as a result of project activities funded in whole or in part by the CCF. More specifically:

• At least 700,000 promotional items such as balloons or bookmarks were produced by projects supported by the CCF over the course of the fund;
• Posters and billboard advertising by CCF projects generated more than 24 million opportunities to communicate climate change;
• Websites supported by the CCF received at least 140,000 visits;
• There were at least 100,000 occasions when individuals accessed a CCF-supported game in some way;
• At least 300,000 opportunities for engagement were generated at events;
• Exhibitions provided over two million opportunities for communicating climate change;
• There were at least 11 million opportunities to communicate climate change through print media generated by CCF projects, via media with a total circulation of more than 8.5 million;
• There were over 7.5 million opportunities to communicate climate change generated by TV, cinema and online video coverage secured by CCF projects;
• The overall number of opportunities to communicate climate change recorded by CCF projects was almost 1.6 million, from radio stations with a combined listenership of at least 1.4 million; and
• 100,000 people may have been communicated with directly through other forms of engagement such as pledges or surveys.

Direct communication of climate change messages
Notwithstanding the limitations of the available data, we estimated that at least 450,000 direct contacts were made by projects funded in whole or in part by the CCF. Furthermore, we estimated that:

• Toolkits and award schemes developed or run by CCF projects helped to communicate climate change to at least 155,000 people;
• There were at least 130,000 cases of individuals being directly engaged with a CCF project at events of one sort or another;
• Projects recorded direct engagement with around 60,000 people at exhibitions; and
• More than 35,000 additional opportunities for communication may have been generated through other activities.

Successes
The Climate Challenge Fund was successful in:

• Opening up communication channels with large numbers of people and organisations, particularly through the use of conventional mass marketing tools such as newspapers and television. Although we have no way of knowing the proportion of engagement opportunities that were effective (i.e. when people noticed and recalled CCF climate change communications), the sheer number of these opportunities that were generated by funded projects – likely to have topped 50 million across the entire fund – was considerable;
• Building capacity among funded projects to carry out similar activities in the future;
• Supporting projects that brought added value through the development of outputs that lasted beyond the lifetime of the Fund and that were likely to have reached larger numbers of people than expected through the engagement of secondary audiences;
• Providing support for the development of new partnerships, and the exploitation of existing ones; and, crucially from a policy perspective;
• Providing general procedural support - while many projects complained about particular aspects of Defra’s role, whether that related to the late provision of questions or what were seen as unnecessarily detailed reporting requirements, most projects were generally pleased with the support offered by the Department; and
• Highlighting a whole range of lessons for future funds.

The Climate Challenge Fund was less successful in:

• Making sure that projects were equipped with the necessary skills to put their planned approach into practice
• Targeting harder to reach audiences (in terms of environmental attitudes): the Fund tended to engage those already interested in climate change;
• Measuring engagement of, and impact upon, secondary audiences;
• Ensuring the quality of data collected and reported by projects was sufficiently robust; and
• Allocating sufficient resources to allow evaluation of projects supported by the Fund on their own terms.

These final two factors meant that it was difficult to judge the success of the Fund against one of its core objectives: generating positive attitudinal change.