Trust, Transparency and the Use of Public Opinion Data in Policymaking

ANZSOG partnered with the Australian Public Service Commission (APSC) to explore how POD is, and can be, used to inform policy development, engaging researchers from the Monash Sustainable Development Institute (MSDI) to co-design and conduct the project.

Further motivation for the project was provided by the renewed interest – following the COVID-19 pandemic – in how evidence is used in policymaking. For example, the Fault Lines review of the pandemic response highlighted issues such as a lack of transparency in decision-making processes and ambiguities about the evidence on which decisions were based. To this end, the researchers accessed a POD dataset from the Department of Prime Minister and Cabinet (PM&C) which had been collected during the pandemic and used to inform policymaking and coordination with the states and territories.

The project began by focusing on how POD had been used during the pandemic before broadening out to consider the use of POD in both crisis and business-as-usual contexts. The goal was to capture insights and develop practical guidance for public servants using POD, including considerations in the design of survey instruments, the analysis and use of findings, and navigating the fine line between developing objective evidence and the risk of politicisation.

The researchers adopted a mixed methods approach, comprising 4 research activities:

1. Rapid evidence review: Researchers undertook a systematic review of literature on how POD was used by policymakers during COVID-19.

2. COVID-19 Pulse Survey analysis: The PM&C data became a POD case study, analysing the design, results, and impact of the survey, including interviews with personnel involved in the process.

3. Policymaker interviews: To better understand the needs of users of POD, researchers conducted semi-structured interviews with 8 people who were involved in POD collection and dissemination during COVID, including senior public servants, senior social researchers, and a former politician.

4. Stakeholder dialogue: Finally, to test the project’s findings, a diverse group of users of POD, including senior public servants, senior political advisers, and social researchers from within the public sector and outside it, were brought together in a facilitated workshop.

The activities iteratively developed an understanding of the uses of POD in both crisis and business-as-usual contexts.

The research establishes a broad use case for POD as part of the evidence base on issues influenced by dynamic and uncertain environments, knowledge vacuums, dominant and contested narratives, differentiated impacts, and underrepresented populations.

The research will be of interest to public servants working on policy issues under these conditions. From problem definition to policy design, implementation and evaluation, POD can provide insights relevant to:

• Understanding community beliefs and attitudes towards the problem, policy options, and policy effects
• Testing social licence and assessing community readiness
• Influencing changes in behaviour to achieve policy objectives

Political advisors and social researchers working with public servants will also benefit from understanding the opportunities and risks inherent in using POD in policymaking.

“Rigour is paramount and needs to be supported by an evidence-based design process that includes clear policy objectives, well-defined methodologies, and integration with other data sources… [and] collaboration with individuals possessing the necessary skills and expertise. Furthermore, transparency is crucial not only in disseminating results but throughout the entire process… [this]builds public trust and reduces the risk of data being used and manipulated for political purposes.” – Bridging Public Opinion and Policy, p. 10

Together, the 4 research activities identified 5 key themes in the use of POD in policymaking in both crisis and business-as-usual contexts.

1. POD can be a valuable kind of evidence for policymakers.

When developed rigorously and integrated with other sources of evidence, POD supports the development of effective and legitimate policies.

2. The use of POD for policymaking requires careful management of the risk of politicisation.

Public servants need to be impartial in their advice about POD insights into the effectiveness of policy options – but there is a tension to manage if the POD also contains information or insights relevant to the partisan politics of an issue.

3. A lack of transparency about the use of POD in policymaking may constrain its effective use in the kinds of complex and contested situations where it might be most valuable.

The research found a reluctance on the part of policymakers to be transparent about being influenced by POD because of concerns that the evidence is seen as partial or politicised.

4. The effectiveness of POD for policymaking turns on rigorous instrument design, data collection and analysis, and dissemination.

POD’s fitness for purpose is enabled by clarity on its objectives, credibility, and soundness.

5. POD should be developed and used strategically, in collaboration with ministers’ offices and end users.

Clearly articulating the scope and purpose of POD, as defined in conjunction with ministers’ offices, is crucial for maintaining impartiality while aligning POD collection and use with policy goals.

“[O]ur findings suggest that effectively leveraging POD in crisis contexts requires a balance between agile and innovative approaches, and deliberate and methodical processes. The inherent tensions between standard practices and the unique demands of a crisis were evident in every phase of the survey’s implementation. This analysis encourages further discussion among practitioners and policymakers on ways to bridge these gaps and enhance the use of POD in future crises…” – Insights to Action, p.1

The final report includes an extensive practice note for public servants that outlines when and how to use POD and a framework for designing and using POD instruments.

Using POD in government: general advice

POD suggests itself for certain uses in policymaking, and when it is used, certain design considerations and risk management considerations follow.

Considerations for crisis contexts

In rapidly changing and highly uncertain policy contexts, the way POD is used in government may need to change.