ANZSOG research provides actionable insights for policymakers using public opinion data

ANZSOG’s cutting edge research project on the use of public opinion data (POD) in government has produced actionable insights and practical guidance for policymakers.  

The final report of ANZSOG’s project, Bridging Public Opinion and Data: Key Insights from a Mixed Methods Analysis, written by Alejandra Mendoza Alcántara, Peter Bragge, and Liam Smith of the Monash Sustainable Development Institute for ANZSOG and the Australian Public Service Commission, summarises the key activities and findings of the project.  

Overall, the project has found that POD – the aggregated results of a representative sample of individuals’ views on a specific topic – can be a valuable source of evidence for policymakers in both crisis contexts, like the COVID-19 pandemic, and in a range of business-as-usual scenarios.  

The final report includes practice guidance explaining the role of POD in policymaking, how policymakers can get the most value from it, and how they can avoid some common challenges in its use. 

The authors explain that POD is most useful for: 

• addressing rapidly changing and uncertain policy environments 
• filling knowledge vacuums and testing assumptions 
• capturing a more representative view when some voices have come to dominate 
• identifying the varying impacts of policy across demographics 
• providing insights into the attitudes, beliefs, and actions of overlooked groups 
• understanding and addressing misconceptions. 

The value of POD as evidence turns on the rationale, design, collection, and use of surveys fulfilling 3 key conditions:  

• Credibility – based on clarity about why the research is being conducted and the reputation and integrity of the researchers 
• Methodological soundness – based on the rigour of the survey, including its representativeness and alignment with the stated purpose 
• Transparency – based on the reporting of methods, results, and limitations of the survey and decisions about how it is used 

Fulfilling these conditions requires managing some common risks that may make policymakers reluctant to commission and use POD: 

• Politicisation – POD may pose a range of risks if it challenges the political basis of policy decisions or suggests that policymaking is opinion-driven. For public servants, managing this risk requires close coordination with ministers’ offices and consideration of POD alongside other sources of evidence where available.  
• Fear of perceived political bias – The public sector operates impartially. However, the use of POD, which measures how the public responds to policy options and decisions, risks the public sector being seen to act on political considerations, like popularity. For public servants, this risk can be managed by clearly articulating the purpose and rationale for POD collection and use, and the evidentiary value it contributes to the policy decisions at hand, and by being transparent with the public about surveys and findings.  
• Trade-offs between rigour and resources – Survey design and implementation is key to the credibility of POD, but the policymaking process also requires agencies to manage their timelines and budgets. The risks of compromising on data collection and analysis can be mitigated by researchers designing a methodology that is fit for purpose within known resource constraints, establishing feedback mechanisms that assess the continuing feasibility and value of the research, and being transparent about the limitations of the research, while commissioning agencies need to be mindful of committing adequate resources to develop POD that successfully addresses the identified knowledge gaps. 

These practice points are relevant to the use of POD in all policymaking scenarios. In crisis contexts like the COVID-19 pandemic, policymakers are likely to face heightened pressure. ANZSOG’s research suggests that in these contexts, characterised by uncertainty and rapidly changing circumstances, policymakers may need to adapt their approach to using POD. In crisis contexts, policymakers should consider: 

• Collecting POD to understand the broader context, before narrowing the focus of the instruments to specific knowledge gaps and the needs of policymakers 
• Collaborating with experts to understand the existing body of evidence and expedite the development of POD instruments 
• Conducting repeated assessments to identify trends and changes in the policy environment 
• Seeking efficient feedback mechanisms, including formal and informal consultation with stakeholders 

The project’s practical guidance is based on 4 related research activities undertaken over the past 12 months: 

1. Evidence Review: The project began with a systematic review of the literature on how POD had been used to inform COVID-19 policymaking around the world. The review found that this subject has been little studied so far, finding only 3 directly relevant studies, and that the influence of POD during the pandemic is difficult to empirically establish. 

2. COVID-19 Pulse Survey Data Analysis: To better understand the use of POD during the pandemic, researchers undertook a case study based on a survey designed and used by the Department of Prime Minister and Cabinet. The case study demonstrates the value of POD for policymaking, as well as some of the challenges of designing studies in crisis contexts. 

3. Practice Review: The project added colour and depth to its study by interviewing a range of persons with varying roles in COVID-19 policymaking, which yielded insights about how POD is used in both crisis and normal scenarios. The report established the main use cases for POD in policymaking while drawing out some differences in how those on either side of the politics/policy divide might perceived POD. 

4. Deliberative Dialogue: The final activity brought together policymakers, political strategists, and social researchers to discuss the project’s findings and to identify good practice for using POD, including how to avoid key political and communication risks. 

The research project has now concluded. Its key findings and outputs can be found here. ANZSOG thanks the Monash Sustainable Development Institute for its expertise and professionalism, and the Australian Public Sector Commission for its support for the project.