This Research Brief was written as part of ANZSOG’s The Bridge research translation project and summarises a piece of academic research relevant to the public sector.
An article in Policy & Politics discusses success in relation to policy experiments. It develops a set of success criteria based on the idea that successful experimentation is not just about outcomes. The end-to-end process of creating and conducting experiments needs to be considered. The article outlines the following four success criteria: the basic elements of an experiment; leadership and resources; procedural elements; and evaluation.
It also argues that experiments should ultimately aim to further policy learning, via deliberation and reflexivity.
Understanding policy experiments
The article defines a policy experiment as a temporary, controlled trial of a policy-relevant innovation that produces evidence for subsequent policy decisions. Taking calculated risks in experiments can inform decision making and improve outcomes via reflexive learning. Experiments can be thought of as tools for innovation, testing, learning, and de-risking.
There are different types of experiments ranging in rigour and type. At their most basic they require a hypothesis to be tested and an element of novelty. Experiments can be run individually or as part of a larger program. Experiments may be performed by policy makers themselves, by non-government actors reporting to policy makers, or by a combination of actors.
Whatever their form, experiments are not neutral as they create new realities shaped by:
What questions are asked.
What interventions are tried out.
How they are measured and evaluated.
Experiments impact groups in different ways, and some may reap considerable benefits while others may bear considerable costs.
Policy experiments and success
Thinking about success requires thinking clearly about what experiments are aiming to do. The article argues that learning is the essence of experimentation. Those that are experimenting are aiming to produce evidence to inform subsequent decision making. This is true whether learning is explicitly listed as a goal of experimentation or not.
Learning consists of the acquisition, translation and dissemination of information. It takes place under multiple conditions, with different goals, and within diverse environments. It is facilitated by inclusive, open dialogue; a broad, reflexive and social process; and flexibility to adapt and change processes during the carrying out of experiments.
Criteria for success
Based on research with Canadian public sector agencies, the article identifies the following four criteria for success:
Basic elements of an experiment: Those creating experiments should be encouraged to begin by thinking of a problem to solve. What ‘kinds’ of policy problems that experiments tackle will vary considerably depending on context. By their very nature, experiments offer opportunities to discuss policy goals and test ideas. They should aim to further goals beneficial to society through this process.
Leadership and resources: Leadership is the number one factor in the scaling-up of experiments. Leaders must be supportive of the risk-taking involved in experimentation and this includes leadership at all levels. Adequate resourcing is essential including financial and non-financial elements.
Procedural elements: This has three dimensions; (1) Legitimacy and conducting experiments through recognised constitutional and quasi-constitutional procedures; (2) transparency and the extent to which decision making is understandable and accessible to the public; and (3) validity and the extent to which is an experiment judged to be valid by stakeholders and citizens.
Evaluation: To learn from experiments, there must be both time and space to reflect on them. Successful experimentation involves planning for this from the beginning.
Success criteria checklist
The article develops the four success criteria into the checklist below.
1. Basic elements of an experiment
A problem is defined.
An intervention is identified.
A hypothesis is identified.
Evidence is produced that can be measured against a baseline.
2. Leadership and resources
Leadership at all levels is supportive of experimentation.
Employees, where necessary, have been giving training on how to experiment.
Employees have ample time to create and perform the experiment(s).
Appropriate financial resources have been committed to the experiment, and information on the future of this funding is clear.
Non-policy divisions, such as procurement and HR, are able to provide sufficient support for the flexibility required in experimentation.
3. Procedural elements
Legitimacy :
The experiment was created through legitimate constitutional or quasi-constitutional processes.
Transparency:
Justification for undertaking the experiment is clear.
Information about the experiment’s creation, implementation and evaluation is publicly accessible.
Any conflicts of interests are disclosed.
Validity
Relevant stakeholders are involved in creating, conducting and/or evaluating the experiment.
There are opportunities for members of the public to give input on experiment goals and aims (either directly or indirectly).
The experiment is accepted as valid by relevant stakeholders and the public.
4. Evaluation
People running experiments are given appropriate time and space to learn from them.
There are follow-up measures to exchange lessons learned from experiments, including with stakeholders.
There are minimal formal ex-post reporting requirements.
The bottom line
While the outcomes of experiments are important, whether they are considered ‘successes’ or ‘failures’ should not be based on an evaluation of outcomes alone. The proposed success criteria consider the characteristics of the experiment itself combined with reflecting on the broader institutional context of conducting experiments. Experiments should also aim to encourage learning, and fail when there is nothing learned.
Beyond aligning with basic characteristics, experiments also need to be:
Led by supportive leaders.
Properly resourced.
Created, conducted, and followed-up on in a manner that is legitimate, transparent and valid.
Evaluated with a ‘light-touch’ approach that focuses on deliberation and reflection.
Further reading:
What is successful policy experimentation? - Kate Mattocks, Policy & Politics, March 2025