Much ado about Nudging? A new framework to evaluate behavioural interventions
13 September 2023
● Research
‘Nudging’ is the catch-all term for attempts to change public behaviour that do not ban options or change economic incentives, focusing instead on interventions which shape the environment in which people make choices. It has become a popular and increasingly influential approach in policymaking, but growing concerns are being raised about its efficacy. An article in Public Administration reviews and critiques various nudge interventions to demonstrate the need for a more expansive evaluative framework. It proposes the ‘4S framework’ which highlights the need for nudges to be sufficient, scalable, and subjective – as well as statistically significant.
Why is there a need for a framework?
Behavioural public policy, like all policymaking, should be concerned with the question: what makes a policy effective? This is especially important as the most popular behavioural policy approach, nudging, rejects alternative approaches such as mandates and bans. It simultaneously advances a thesis which suggests people may not always choose what is best for themselves.
This creates a delicate policy environment where it is difficult to know if a nudge has been effective. This has spurred recent commentary criticising nudging as a policy tool given significant policy challenges. There has been a call for a more a thoughtful debate beyond statistical significance to make behavioural science more relevant to policymaking.
The article focuses on four elements of policy evaluation which are relevant to nudging. These are presented as the “4S” framework. These elements are (statistical) significance; sufficiency; scalability; and subjectivity. The framework demonstrates how each element should contribute to the effectiveness of a nudge, and how effectiveness is often better understood as a trade-off between elements, rather than an exercise in satisfying each.
An overview of the 4S framework
The 4S framework is outlined below:
| Element | Motivating question | Example |
| (Statistical) significance | Is the effect of an intervention due to random chance, and if not, is there a causal effect between the intervention and the observed behaviour? | A randomised controlled trial (RCT) where a treatment group is asked to choose between options with a default option selected, and a control group is asked to choose without any default being selected (i.e., an active choice). |
| Sufficiency | Is the intervention and/or policy program in which the intervention is used adequate to resolve whatever policy challenge prompted the need for a policy response? | Using a nudge to encourage vaccine uptake in response to a sudden viral epidemic, where herd immunity requires x% uptake, and current uptake is y%, where x > y. |
| Scalability | Does the causal effect of an intervention and/or policy program in which the intervention is used broadly hold as the intervention is applied to larger and more heterogeneous populations than were used to identify the effect? | An automatic reminder text message program to support low-income high school students into higher education, which is supported by a one-on-one mentoring service if a student engages with the nudge. |
| Subjectivity | How does the intervention and/or policy program in which the intervention is used interact with different sub-groups in the population, and how are the effects of the intervention felt by different sub-groups? | A calorie label nudge on snack food designed to encourage healthy eating, which is experienced differently by health-conscious and non-health-conscious groups in terms of (a) observed behaviours; and (b) welfare outcomes. |
Significance concerns whether interventions work statistically speaking (i.e., not due to random chance). Sufficiency concerns the ultimate objective of the policy, or the policy challenge which any potential policy is designed to solve. Scalability concerns the applicability of an intervention when it is applied to larger and more diverse populations than it was tested in. Finally, subjectivity concerns the welfare effects of interventions, as well as questions which arise when one considers heterogeneous populations and distributional effects.
Applying the 4S framework
When nudges are evaluated using the 4S framework, there are likely to be three outcomes if there are no critiques:
- A nudge may require redesigning.
- A nudge may require reformulation through combining with other policy tools. For instance, nudging students to engage with higher educational resources, such as one-on-one mentors.
- A nudge may require rejection. For example, abandoning a plan to nudge electric vehicle ownership, and instead investing in zero-emissions public transport.
Each element of the framework has trade-offs with other elements. A nudge may be statistically significant in an experimental setting, but the effect size may be insufficient, the intervention may not scale, and subjective factors may demand re-evaluation of result. Alternatively, a nudge may be sufficient, but only significant when used in conjunction with other policies, scale at a substantial cost, and be unequal or harmful for different subjects. A nudge may be scalable, but result in lessened significance, be costly compared to other sufficient policies, and be complicated when accounting for subjectivity. Finally, a nudge may account for subjectivity, but undermine statistical significance, be difficult to evaluate in terms of sufficiency, and as above be complicated to scale.
The bottom line
The 4S framework is not a checklist to be used to outright determine an effective nudge from an ineffective one. The framework instead allows policymakers to determine what element matters most within the policy context and choose the appropriate path to follow. (i.e., redesign, reformulate, reject).
Want to read more?
Seeing the nudge from the trees: The 4S framework for evaluating nudges – Stuart Mills and Richard Whittle, Public Administration, May 2023
Each fortnight The Bridge summarises a piece of academic research relevant to public sector managers.
Recent Research Briefs on Nudges and behaviour change include:
- Published Date: 13 September 2023