Piloting policy initiatives before a wider rollout is increasingly commonplace. Research has shown pilots have multiple shifting purposes and judging the success of policy pilots is complex. A paper in Public Administration:
- outlines a framework against which policy success can be assessed
- considers the implications of the framework in evaluating policy pilots.
Policy success and failure
The paper discusses a framework by Marsh and McConnell against which policy success can be assessed. The framework outlines three dimensions of success:
1. Process success: This refers to the policy making process. Dimensions include the establishment of a stable coalition behind a policy and lack of significant opposition.
2. Program success: This encompasses successful implementation and the achievement of desired goals.
3. Political success: This refers to the political benefits accruing out of a particular policy. These might include sustaining an electoral coalition, enhancing electoral prospects, or silencing opposition.
Each dimension sits on a spectrum – from “success” with all criteria met through to “failure” when none of the criteria are met. There is also an in-between area where success may be partial.
Some have criticised this framework, arguing the separation of policy process, programs and politics is a mistake. Bovens contends policy process success can be both:
- programmatic such as when a piece of legislation is successfully navigated through parliament
AND
- political when the passage of such legislation enhances a government’s political capital.
The paper then examines Bovens’ alternative framework which assesses policy success or failure across two dimensions:
- Process and outcomes.
- Program and politics
Table 1: Dimensions of policy success
Source: Bovens (2010)
Success and policy pilots
Pilots differ from full policy implementation in that they are limited in both time and space. As they are vehicles for testing potential policy solutions, they embody some uncertainty as to their value.
The paper presents a framework to assess policy pilots that draws on Bovens’ modification of the Marsh and McConnell framework. As well as considering the program and political effects of the design and implementation of policy pilots, a longer-term view of the eventual impact of pilots also needs to be considered. This is termed “pilot effects” in the table below.
Table 2: Dimensions of policy pilot success
Source: Checkland et al (2021)
The paper acknowledges any assessment of a policy pilot’s success will be from a particular perspective. A pilot which acts to empower one actor will often disempower another. It is therefore important that the perspective from which success is being judged should be declared in operationalising the framework.
Applying the framework
Applying the framework requires evaluative work with multiple methods. This spans quantitative analysis of before-and-after outcomes to qualitative analysis examining political speeches and other documents. The paper explores the utility of the framework by applying it to an English national policy pilot scheme: the Vanguard New Care Models program.
Vanguard was part of the National Health Service and tested new ways of providing services. The aim was to increase integration between different types of providers and dissolve traditional boundaries.
The program ran for three years from 2015. Vanguard pilots received £329 million with £60 million spent on support and evaluation. The program is referred to as having been successful in subsequent NHS policy documents.
Applying the pilot success framework to Vanguard involved the following:
- an initial qualitative study of program initiation and oversight.
- qualitative case studies exploring program operation.
- a study of relevant policy documents, including those which use the Vanguard program to make arguments about future policy direction.
- interviews with 29 stakeholders, including senior managers from NHS England and regional staff supporting local Vanguards.
What the application found
Applying the framework to the Vanguard program case study supported a more nuanced account of different aspects of pilot success. This moved beyond a simple assessment of whether or not initial outcomes were met to interrogating the ways in which the pilot program has been used politically to achieve other objectives.
The research also found the framework supports the complexity, ambiguity and mobility of the purposes involved in policy piloting. It brings to the surface pilot impacts which might otherwise be hidden and potentially allows more nuanced causal explanations to be considered.
The bottom line
Policy pilot evaluations often adopt a relatively simplistic approach of considering whether or not a pilot has worked against particular outcome criteria. This represents a limited understanding of the purposes and effects of pilots.
The pilot success framework supports a more nuanced and multifaceted approach to assessing success. A strength is its focus on the explicit delineation of different dimensions of success and on the declaration of the perspective being adopted in the assessment.
Want to read more?
“Success” in policy piloting: Process, programs, and politics – Kath Checkland, Jonathan Hammond, Anna Coleman, Julie Macinnes, Rasa Mikelyte, Sarah Croke, Jenny Billings, Simon Bailey, Pauline Allen, Public Administration, October 2021
Each fortnight The Bridge summarises a recent piece of academic research of relevance to public sector managers.
Recent Research Briefs include:
- Rethinking the commissioning of consultants to enhance government policy capacity
- Building people up: Growth-oriented leadership in the public sector
- Wicked and less wicked problems
- Published Date: 26 January 2022