Public services see quantitative measurements and targets as a key part of accountability, but a growing movement is questioning whether this focus has distorted priorities and led to the decline in less-measurable capacities.
‘Meaningful Measurement’, a webinar in the ANZSOG/ Centre for Public Impact Reimagining Government series, examined how measurement in the public sector is failing – with metrics designed for control leading to the gaming of targets, perverse incentives and worse services for the public.
The panel – made up of CPI Executive Director Adrian Brown, CEO of the Oxford Hub Sara Fernandez, and Professor Jenny Lewis from the University of Melbourne’s School of Social and Political Science – called for a shift to a more nuanced approach to measurement, where it focuses on learning about complex problems and the people experiencing them, as a way of improving the public sector’s overall approach.
The panel’s facilitator, ANZSOG Deputy CEO Jane-Frances Kelly, asked the panel who should decide what is measured, and whether the people deciding now were the right people?”
Mr Brown said the issue came down to how we think of data: is it a way of holding people to account, or is it a way of informing a wider conversation about what is going on?
“Public servants needed to challenge the assumptions that data means truth – for ourselves and our organisations,” he said.
Ms Fernandez said that often people were measuring and producing data because they are asked to do it, not as part of their broader goals, and that measuring could create incentives that destroyed the impact people were trying to make.
“How do we make measurement something we are aligned on, not the afterthought after we have agreed on what we do,” she said.
“Often people far from the frontline make the decisions on measurement. There needs to be meaningful co-production on how we track, learn and measure things which is important if we are going to achieve any impact.
“How do we shift from a focus on what we are doing to how we are doing it? It’s harder to measure but in the how is where the real meaning is.”
Professor Lewis said it was important to collect qualitative as well as quantitative data because reducing everything to numbers – while fast and convenient – meant you could lose an understanding of what worked.
“The paradox of performance measurement is that stuff you can do that’s easy is quick but you lose that complexity. More data is not always better, and when a measurement becomes a target then you can get problems. Those measurement systems can start from good intentions but end up in a place that benefits nobody.
Mr Brown said that accountability through numbers was convenient, but we have to “push against the idea that it is true accountability. It is an illusion of control. The map is not the terrain”.
Is the idea of measurement fatally flawed?
The shortcomings of measurement have led to some viewing it as a fatally flawed concept, and others as one where poor execution is the problem. The panel was asked how to move to better understandings of measurement that could help public services improve.
Mr Brown said that governments should focus more on the risks of measurement and be aware of the difficulties of operating at scale.
“If we measure something we should ask what risks does that have of actually obscuring the truth. We should view measurements as a very powerful and dangerous tool that we wield very liberally and optimistically to many situations, when it may not make things better but may make things worse, if we don’t start with that assumption we can get into difficulty.
“When we have hierarchical systems which are built for scale, the only way they can reasonably proceed is through massive simplification. Data is wonderful if you are sitting on top of a hierarchy and trying to understand everything. It gives you simple clear comparable sources of information that you can have discussions about and manage the organisations. But what if we said you don’t have to do this at scale?”
Ms Fernandez said that public sector leaders needed to aware of whose power and whose voice comes out in measurement.
“There’s a dichotomy between numbers who are produced by the machine, versus stories told by the people directly affected, in their own voice and in their own terms – how do we move to hearing those voices and being accountable to the people for whom we are delivering the work rather than to the officials?” she said.
Professor Lewis said that organisations would still need to measure in some way but that structures needed to be well-planned.
“Some of the best I’ve seen are where people and systems try to focus on a small number of things that can be relatively easily counted and give permission to groups to come up with what is most important in that area,” she said.
“If we get beyond doing it for the sake of doing it but doing it to learn something from it and improve from it that would be a sign we are getting the balance right.”
Mr Brown said that the debate around measurement had not moved on for quite a long time and was linked to the dominance of ideas of managerialism within the public service. He said that reshaping measurement should be part of a broader move to empower people and have as much localism as possible.
“We have to take account of the legislative frameworks of course, but these environments were created by human beings and we can uncreate them and create something better, at a local level in particular,” he said.
The ANZSOG/CPI Reimagining Government series, now in its second year, brings together academics and practitioners to debate a new path for government based on embracing complexity, valuing relationships and prioritising learning. Bookings for the final webinar for 2021 System Stewardship are now open. More information and resources are available on the Reimagining Government site.
- Published Date: 31 August 2021