Better ways to evaluate regulatory impact – and why this matters

Guest editor: Professor Veronica Taylor, Professor of Law and Regulation, School of Regulation and Global Governance (RegNet), Australian National University

In a recent interview with The Australian, [i] Insurance Australia Group CEO Nick Hawkins commented:

While we excel at introducing new regulations, we often fail to evaluate the effectiveness of existing rules, creating unnecessary barriers to innovation and growth.

Australian regulation is an asset, but the balance between quality and cost is now under fire. Internationally, many envy our regulatory frameworks and professionalism.  However, as a post-industrial economy, our systems are complex, and relatively costly to design, implement and navigate. Regulators often struggle to articulate the value of the frameworks they steward. That communication vacuum is filled by calls to ‘cut red (or green) tape’ or to reduce ‘unreasonable regulatory burdens’ – or to stop stymieing innovation.

Demonstrating regulatory value

Better regulation or stewardship approaches suggest we should take an evidence-based approach to value. Before we launch or change a regulatory regime we do this through a regulatory impact statement or equivalent (RIS/RIA). We do much less of this post-launch – in part because successful regulation often results in non-events. But checking how regulation ‘lands’ allows us to adjust for unanticipated consequences, or to update rules and practices overtaken by economic or social or technological change.

Some jurisdictions commit to legislative review on a five-yearly cycle. In practice this is hard. Politicians lack the appetite for revealing problems; policy makers or regulators may be willing but lack the capability – particularly if the RIS/RIA is the only choice. Budgets are always tight.

Yet – as the Hawkins quote illustrates – not using the impact feedback loop invites the risk that stakeholders will do their own assessments, using their own criteria. [ii]

Limits of existing assessment methods

The existing evaluation toolkit is relatively limited and expensive. Randomised control trials (RCT) are often dubbed the ‘gold standard’ for evaluating policy impact. [iii] They work best ahead of introducing a change, allowing us to compare results from ‘treatment’ and ‘non-treatment’ locations.

The economic benefit-cost analysis of the RIS/RIA is also powerful as a way of strengthening democratic participation in, and transparency of, regulatory design. However, its limitations such as cost, complexity, need for specialist knowledge, vulnerability to political pressure, and inability to forecast unintended consequences over time prompt many requests for exemptions.

The ‘RegData’ tool is good for assessing volume and type of legislative rules affecting businesses. [iv] Compliance cost tools estimate the business time and expenditure needed to comply with them. [v]

But regulatory stakeholders experiencing burdens also include members of the public, particular communities and government itself. What the current tools do not show us is what kind of regulatory burden affects what type of stakeholder — and when.

RegValue 1.0: a pragmatic approach

All regulation has value and imposes some costs. Costs become ‘burdens’ when they are disproportionate to the public value of the regulation, or if one or more stakeholders experience them as unreasonable.

We partnered with the ACT Better Regulation Task Force in 2023 to develop a diagnostic tool that could help policy makers and regulators use existing or new data to understand the stakeholder experience by looking at:

• The quality of legislation;
• How the legislative instruments are interpreted in practice by policy makers and regulators;
• The kind and quality of interventions by regulators; and
• How stakeholder experience regulatory communication and relationships

We debuted RegValue 1.0 in an NRCoP national webinar in December 2024, NRCoP and ANZSOG present: Better ways to assess regulatory impact: RegValue 1.0 | ANZSOG.

Assessing new and different burdens

A feature of this tool is that it considers intangible burdens, such as the psychosocial effect of poorly-designed or executed regulatory processes and communication. This extends approaches such as Wellbeing Impact Assessments[vi] which address non-economic impacts, e.g. on the environment, community wellbeing, and First Nations peoples.

Having piloted RegValue 1.0 for two industries, we are in the final test phase and aim to release the User Guide in mid-2025.

Can AI help?

The next step for all impact assessment, including RegValue 1.0 — is harnessing Large Language Models (LLMs) and AI to in ethical ways to analyse impact data faster. RegData uses machine learning for legislation; Lisa Burton Crawford and her collaborators use the more sophisticated concept of legislative complexity. [vii]

The need for legislative ‘explainers’ was a clear finding from RegValue 1.0 testing. AI/LLM solutions for reliably explaining legislation and regulatory requirements should appear soon, and we will also be able to model differential regulatory impact by types of stakeholder.  So AI has the potential to help with regulatory impact assessment. But as with any tool, we need ‘humans in the loop’ to use impact assessment more proactively.

[i] ‘IAG CEO Nick Hawkins: Put tax reform on the agenda’ The Australian, December 13, 2024 <https://www.theaustralian.com.au/business/companies/iag-ceo-nick-hawkins-put-tax-refom-on-the-agenda/news-story/771e5759aa46cf4ac49d9caccb93a145> accessed 4 January 2025

[ii] See for example, the Business Chamber of Queensland contribution to regulatory burden reduction policy-making: <https://statements.qld.gov.au/statements/101879>

[iii] Australian Centre for Evaluation < https://evaluation.treasury.gov.au> accessed 4 January 2025; Cary Coglianese (2023) ‘Evaluating Regulatory Performance’ 8(1) Journal of Law and Public Affairs 47

[iv] The ‘RegData’ approach pioneered at George Mason University: https://www.quantgov.org/history accessed 4 January 2025

[v] Originally based on the forerunner to the World Bank’s ‘Business Ready’ methodology: https://www.worldbank.org/en/businessready

[vi] For example, the ACT Wellbeing Framework: https://www.act.gov.au/wellbeing/wellbeing-framework/embedding-wellbeing accessed 4 January 2025

[vii] Lisa Burton Crawford et al, ‘Legislative complexity; what is it, how do we measure it, and why does it matter?’ (2022) Australian Public Law Blog <https://www.auspublaw.org/blog/2022/11/legislative-complexity-what-is-it-how-do-we-measure-it-and-why-does-it-matter> accessed 4 January 2025; Vincent Chiao, ‘Hyperlexis and the rule of law’ (2021) 27(2) Legal Theory126-148