To develop and evaluate place-based interventions, George Bramley draws on a recent review of the evaluation of place-based approaches by the Youth Endowment Fund.
When first asked to think about the evaluation of place-based interventions that would result from Local Policy Innovation Partnerships (LPIPs), my first instinct was to check whether anyone has produced any good practice guidelines, or had undertaken a systematic review of the literature on methodological approaches and issues around evaluating place-based interventions. My reason for doing so is different policy areas have developed their own perspectives and norms in terms of which methodologies and questions need answering. My initial crude searches suggested that Australia was a potential beacon to learn from in terms of having literature and toolkits, and the use of place-based approaches was more advanced in some policy areas than others such as health.
My prize find was a narrative review of the evaluation of places-based approaches commissioned by Youth Endowment Fund, undertaken by the Centre for Evidence and Implementation (CEI) which is the basis of this article. The review defined Place-based approaches (PBAs) as collaborative long-term approaches to addressing a social issue, which operate in a defined geographic location, focus on local needs and the attributes of place, involve multiple agencies, and engage local communities in design and delivery(Smith et al., 2023).
One of the key messages of reading their review was that in addition to developing a theory of change, the modus operandi of evaluators and evidence-based policymakers is the need to specify a theory of place. Hence the title of this article. A theory of place sets out the rationale for adopting a place-based policy approach in that it sets out the logic for working in a particular way, in a particular locality. Whilst having a theory of place intuitively makes sense, very little at first glance seems to have been written on how best to develop one compared to developing and using a theory of change. It is a topic I intend to return to in a future article.
Unsurprisingly, the review authors found not one accepted best practice approach to evaluating PBAs and limited agreement on what should be prioritised in evaluations and about the role of different methods. Instead, it is left to those involved in the commissioning and development of evaluations to consider what is appropriate for their given context.
Usefully, the review authors identify several challenges and possible responses that evaluators can take when evaluating place-based approaches. These I have summarised in the table below.
| Challenge | Possible response by evaluators |
| Place-based interventions are frequently designed to address ‘wicked problems‘ that are entrenched and have their roots in specific local contexts and require collective efforts of different stakeholders to solve (Bellefontaine & Wisener, 2011). | Adopting a developmental evaluation approach can create value through the exchange of learning between evaluators and those involved in programme development and implementation (Egan M et al., 2019; Quinn Patton, 2020). |
| To have value, evaluations need to account for the roots of wicked problems and the multi-faceted nature of place-based interventions. | Evaluators can: – start with defining, conceptualising, measuring, and assessing context – engage in hypothesising context interacts with the approach adopted and its implementation – study interaction between intervention and approach over time (Kelly, 2010; Skivington et al., 2021). |
| The quality, quantity and impacts of community engagement may be an important part of the place-based approaches, and their evaluation (Taylor & Buckly, 2016). | Considered the use of participatory approaches at different stages in the evaluation. |
| Place-based interventions operate at multiple levels and try to deliver change at different levels, for example, for the individual and for their community | Consider and account for the multiple levels at which interventions operate. It may be helpful to define targets, outcomes and impacts at different levels. Evaluators may choose to use different levels of structures. The simplest hierarchical structure would be to use macro-, meso-, micro- and exo-levels. However, these may not take account of system effects. |
| Unpredictable systems interactions make attribution and causality hard to pin down (Bicket et al., 2020). | Evaluators might wish to define and assess impacts on individuals, microsystems (immediate environment within which individuals operate), mesosystems (interactions between microsystems), ecosystems (impacts on formal and informal structures which affect the microsystem) and macro systems (prevailing culture) levels (Baidawi et al., 2023). |
| Constructing local area datasets that map the locality of place-based approaches is also challenging (Bellefontaine & Wisener, 2011). Place-based approaches often focus on areas that have social but not necessarily administrative meaning. | It may be necessary to include adjacent areas to assess the displacement of the problem being addressed, as well as the diffusion of positive change. |
| The emergent nature of place-based approaches means: – It may take some time before the intended goals, impacts and ways of working are agreed upon across stakeholders, – Goals may change over time as the place-based approaches adapt to changes in the context, new opportunities, or in light of learning about progress (Skivington et al., 2021). – It may not be in a final state when evaluation planning begins requiring the evaluation to be adaptive and agile (Bellefontaine & Wisener, 2011). | Evaluators need to build in resources that allow them to be adaptive and agile. |
| When interventions involve multiple sites, there is inevitable variation between settings. Although there is an expectation that there is some initial common ground in terms of needs, goals and target groups. The way initiatives develop will inevitably differ from place to place as they are influenced by local circumstances (Bellefontaine & Wisener, 2011). | While variation in and evolving nature of place-based approaches makes evaluation challenging, handled well can be used to enrich learning. Variation provides scope for exploring diversity in implementation and outcomes to understand ‘what works, for whom, in what contexts, how and why’. This requires evaluators to: – Use evaluation methodologies that address context and its interaction with the intervention – Have adaptable evaluation plans that are kept under regular review – Documenting key changes in the intervention, its delivery, and the context in which it is being applied – Considering the value of a developmental evaluation approach with two-way communication and learning between evaluators and those involved in implementation, underpinned by strong working relationships and trust (Egan M et al., 2019). |
| Timeframes for place-based approaches are often long, and evaluations therefore need to capture short, medium, and long-term change that emerges at multiple points in time. | Explore the use of routine administrative data that enables cost-effective follow-up. Also, consider creating an archive of relevant documents that can be made available for more qualitative evaluation approaches such as process tracing at a future date, say 5 or 10 years after implementation. |
| Attribution and causation are particularly challenging and the ‘counterfactual’ – what would have happened in the absence of the intervention – is hard to construct. Some would argue that attribution is neither feasible nor meaningful, and that assessment of the contribution made by a place-based approach should instead be the focus. | The counter view, there are available methods to measure impact robustly, with some form of comparison, and that the measurement of outcomes is challenging but not impossible. |
The review identifies two potentially useful frameworks. These are the RE-AIM Framework for evaluating place-based activities and the EPIS Model. The RE-AIM framework includes five dimensions: reach, effectiveness, adoption, implementation, and maintenance. Whereas, the EPIS model has four dimensions: exploration, preparation, implementation, and sustainment. These will be the topic of future articles exploring their potential applications by LPIPs and placed-based policy practitioners.
This blog was written by George Bramley, Principal Analyst, City-REDI, University of Birmingham.
Disclaimer:
The views expressed in this analysis post are those of the author and not necessarily those of City-REDI or the University of Birmingham.