Measuring impact: ANZSOG bringing academics and public sector practitioners together

How to measure the real-world impact of education and research are increasingly important questions for the university and government sectors, particularly when it comes to the impact on public policy and public administration.

ANZSOG has begun a project to get academics and practitioners to work together to identify ways to capture impact. The first ANZSOG Impact Workshop was held in Canberra on 4 October and brought together participants from the university, government, not-for-profit and public management sectors.

The all-day discussion was a chance to advance ideas and begin to identify ways to better generate, capture, and demonstrate the ‘impact’ of education and research on public sector practice.

The workshop was held in conjunction with the Crawford School at ANU, the Public Service Research Group at UNSW Canberra, and the Analysis and Policy Observatory.

A clear outcome from the workshop was consensus on the desirability of a community of practice, an initiative that ANZSOG has initiated and is now driving in collaboration with university partners, owner governments and interested stakeholders.

ANZSOG Deputy Dean Professor Catherine Althaus said that measuring impact was an important issue and part of demonstrating the value of education and scholarship for government.

She said that the workshop was about building relationships and starting a process of bringing governments and academics together to improve assessment of impact.“Nobody has cracked the nut of how to assess impact in a measurable and persuasive manner,” she said.

“We are bringing a whole range of people together who have a piece of what is a complex puzzle, so they can share what they know and develop a way forward.

“Increasingly there is a divide between theory and practice, for those of us who work in the space we know how crucial it is to bring them together.”

“We want to establish a greater community of practice and generate some tangible benefits, and to create a common language and demonstrate the case for assessing the impact of education and research.”

Professor Rod Glover, Deputy Director – Enterprise, of the Monash University Sustainable Development Institute, said that conversations around impact “are really important, because whether we are researchers or public servants we ultimately want to effect change in the world”.

He said that measuring impact could be difficult due to lack of incentives for innovation in the public sector.

“Focusing on impact is innovative even though it is not seen as entrepreneurial, because, by and large we don’t apply sophisticated measures of impact to what we do, we do it at quite a superficial level.

“We have a risk-averse culture in the public service. There are a range of barriers to innovation; time and freedom are two of them, but the biggest is the signals public servants get from above that encourage them to be risk-averse.”

The event was held under Chatham House rules, but included presentations from:

Craig Ritchie, Australian Institute of Aboriginal & Torres Strait Islander Studies
Professor Tony Dreise, Professor of Indigenous Policy Research & Director Centre for Aboriginal Economic Policy Research (CAEPR)
Erma Ranieri, South Australia Commissioner for Public Sector Employment
Scott Johnston, Assistant Commissioner, Performance & Analytics, NSW PSC
Sonja Stewart, former Deputy Secretary Operations & Engagement, Department of Premier & Cabinet
Dr. Subho Banerjee, Research Program Director, ANZSOG
Dr. Catherine McClellan, Australian Council for Educational Research (ACER), Director Assessment & Psychometric Research
Debra Iles, Senior Associate Dean, Executive Education, at the Harvard Kennedy School of Government
Richard Banks, Deputy Director of the Policy Profession Unit, at the UK Cabinet Office

Sessions discussed the current state of play regarding the impact of education and research into public sector practice, some of the barriers that reduce impact at different times in the research and policy cycles, and the dangers of research being misinterpreted or having unintended impacts.

They included a ‘government perspective’ of what research and impacts governments are looking for, and what governments see as a return-on-investment for education and research spending.

The long-term goal of starting these conversations is to push all relevant fields to develop ways to track impact over time, and tell better stories about what impact government can have generating broader public value.

The day concluded with a session on future steps, with a focus on how government, universities and other sectors could work together to better to capture and demonstrate impact into practice.

Barriers to cooperation

Debra Iles, Senior Associate Dean, Executive Education, at the Harvard Kennedy School of Government, said that impact was one of the key priorities at the HKS and that it was reassuring to attend and find that academics in Australia and New Zealand were grappling with the same issues around measuring impact.

She said she had been pleasantly surprised by the number of controlled trials happening in the area, because that really is the gold standard of academic research.

“One of the insights today was the idea of ‘warm problems’: governments need to act fast and academic research may not move quickly enough to deal with hot problems, but ‘warm problems’ may be the happy medium,” she said.

Deputy Dean Althaus said that there were many barriers that stopped academics and governments working together including the different speeds they worked at.

“The expectations of government are so fast and so compressed that working out a way to ‘marry’ these two difference paces of working together will be important,” she said.

“Academics are also incentivised to publish in academic journals that often exist behind paywalls, while practitioners are asking for accessible research.”

Brandi Hudson, chief executive of the Independent Maori Statutory Board, said that impact on Indigenous people needed to be considered in any discussion of impact.

“Government can do better to recognise Indigenous leadership, and to acknowledge and be respectful of our aspirations,” she said.

“Public servants need to recognise the realities of the impact of policy on Indigenous people, and consequently on vulnerable people of all backgrounds. I want to talk to public servants about looking at their role and ensuring that it is inclusive of First Peoples and minority cultures.

Natasha Ryan, Assistant Secretary at the Department of Defence, said the workshop was a chance to stop and come together with like-minded colleagues and discuss these issues.

“The discussion on measuring impact is important, because it allows to focus on what are the right questions and on what are the short-term and the long-term imperatives,” she said.

ANZSOG is revamping its research strategy with a new demand-led, practice-driven and collaborative approach to research based on our unique ability to connect academics and practitioners. ANZSOG is consulting with its government owners and university partners about the design, funding and delivery of new research projects. Research projects will be driven by the key issues and questions raised by governments and universities, and ANZSOG will focus on issues with multi-jurisdictional relevance.

The Impact Workshop is part of a broader ANZSOG Strategy to ensure that its research activities meet the current and future needs of its owner governments.