Learning from failure and success

Evaluation of policies and programs together with a culture which learns from success and failure is pivotal in producing evidence for effective policy development and public sector management.

This brief outlines research prepared for the Independent Review of the APS on evaluation and learning from failure and success. The research takes a historical approach to frame the issues for the APS Review panel and suggests a number of reform directions for their consideration.

At a glance

In ANZSOG research commissioned for the APS Review, Rob Bray (ANU), Matthew Gray (ANU) and Paul ‘t Hart (Utrecht University) explore the following questions:

• What should the APS evaluation practices look like?
• How should the APS be learning from its successes and failures?
• What does the APS need to change to get there?

The paper concludes the APS currently does not learn well from experience and its approach to evaluation is piecemeal both in scope and quality.  The issue is not one of skills and capacity but first and foremost, one of culture and institutional practices.

• READ: ANZSOG research papers commissioned by the Independent Review of the Australian Public Service.

Approach to the paper

The research paper is informed by extensive literature reviews and consultations with experts in the field, including current and past senior public servants.

Research findings

A key issue is how evaluation and learning activities relate to the principle and practice of accountability. Is evaluation simply framed in the context of the government goals for programs and reports to government? Or is there is a broader accountability role for evaluation and if so, how this can be fitted into the structure of the APS? The paper finds the APS has a piecemeal approach to evaluation. This diminishes accountability and is a significant barrier to evidence-based policy-making.

In their day-to-day activities, departments and agencies tend to focus on achieving the ‘here and now’ priorities of program and policy implementation. In examining their past performance, departments and agencies are often more concerned with reputational risk instead of learning from experience and feedback. Evaluation is often seen as a second-order and lower-priority issue.

What’s causing the problem?

The problem is not a lack of skills and capacity of public servants. It is a product of the environment in which the APS operates and cultural practices. This is exacerbated by the lack of an institutional framework that embeds the strategic importance of institutional  learning.

What does reform look like?

The APS would take a rigorous evidence-informed approach to designing and implementing policies and programs, and providing advice to government. This would be grounded in:

• a cumulative knowledge base about what has worked successfully and failed to work in different settings
• an approach which make space for reviewing past experiences from a learning perspective
• building mechanisms into program design and organisational routines which support future learning.

The effectiveness of programs and policies would be tested against clearly articulated program objectives as well as a systematic evaluation framework. This framework would:

• address the specific evaluation of the design, delivery, outcomes and impacts of programs
• provide scope for thematic, cross-cutting and comparative reviews to identify patterns, trends and critical success/failure factors.

Rebuilding evaluation and learning capacity

As to who evaluates, the paper sees this question as being related to the broader question of the accountability framework in which evaluation is undertaken. While most evaluations would still be undertaken within departments, the paper suggests a need for much stronger central coordination and leadership.

This centralised evaluation role would be both at an APS-wide level and in departments. An APS-wide function would:

• enable an integrated whole-of-government approach
• set priorities and identify gaps in evaluation practice
• identify APS-wide learnings
• provide leadership and guidance in evaluation across the APS.

The paper canvasses two options for the centralised evaluation function – within the current APS structure or as an independent statutory agency accountable to Parliament.

Within the wider framework of departmental learning, evaluation is just one of a range of inputs. Organisational learning can also derive from:

• past experiences
• the experiences of other departments and of other policies
• the experiences from other countries.

Other options to mine the learning potential of both successes and failures include:

• critical-incident/near-miss reporting systems, particularly within delivery and regulatory agencies.
• ‘whole system in the room’ debrief where critical cases (near misses, explicit failures, ongoing or ad hoc instances of high performance) are reconstructed and reflected on.
• learning from our stakeholders exercises through focus groups and/or fishbowl sessions.
• training evaluators across the APS in the methodology and tools of positive policy evaluation such as Appreciative Inquiry and the Success Case method.

Drawing these inputs together is an often neglected function, and would be supported by departments developing custodians of these learnings.

Underpinning these practices would be effective information systems, providing quality and timely data to departments and evaluators.

What it means

Policy and program failures should not be approached from a perspective of allocating blame, or successes from the perspective of reputation-enhancement and credit-claiming. Both successes and failures could instead be treated as sources of data, insight and lesson-drawing.

Evaluation and learning would become an integral element of the APS accountability framework, including:

• accountability to ministers in providing them with clear and frank information on departmental activities including program outcomepolicy advice and options formulated on the basis of the lessons learnt in past and present activities
• accountability to the Parliament through the minister, and, where appropriate, to the wider community.

While the APS will continue to work in the era of the 24-hour news cycle, demands for instant grabs and a focus of ‘gotcha’ reporting,  a systematic approach to policy and accountability will reinforce the capacity and confidence of the APS to ‘stand its ground’ with authority.

This brief is part of a Research Series written by Maria Katsonis. This research brief originally appeared in The Mandarin as part of The Mandarin and ANZSOG’s 2019 Research Series called The Drop.

Sign up to the bridge