Surveying innovation skills to build public sector capacity: What don’t we know about our people?

This article, which originally appeared on apolitical.co , was written by Professor Beth Noveck, head of the Governance Lab at New York University (GovLab), and Jason Williams-Bellamy.

GovLab is collaborating with ANZSOG and the Monash Sustainable Development Institute on a project to examine innovation skills and capabilities within the Australian and New Zealand public sector, with the goal of designing innovation-focused education and training programs for the public sector.

A report from the project will be released in August.

Audit Scotland, which monitors government spending, recently found that the Scottish government was not delivering on its ambitious digital strategy.

The reason: a shortage of digital skills among public leaders made it difficult to measure progress and share common lessons learned.

The shortcomings of Scotland’s efforts to digitise government services might have been avoided, had they been more aware of the state of digital literacy in the country’s public sector. Scotland’s lack of insights into the skills and abilities of its public servants is not unique.

As public institutions seek to become more innovative in how they work, they generally lack baseline information about the current state of skills in the organisation. In the private sector, too, we worry about workers at every level having the skills they need to thrive in a high-tech economy. Managers regularly turn to surveys and questionnaires to identify what people know and what they want to know.

But, in the public sector, we know very little about current skills and competencies.

Asking the right questions

Many governments are investing in creating new training programs in an effort to change working practices.

But they are often doing so without knowing either what people know, who needs to know what, or how they want to learn. There is more than a little irony that, in an effort to spread the teaching of skills such as data analysis and human-centred design, public organisations are not applying those very same approaches to conduct an assessment of training needs prior to developing their training programs.

Asking people about their skills has its own limitations, as people are notoriously poor and unreliable at assessing their own expertise.

For public organisations wishing to boost their performance and improve how they solve public problems, they should first conduct a training needs analysis to measure the current state of innovative problem solving skills and the nature of any skills gaps to understand what people know, what they would like to know and how they learn best.

In 2017, the Chilean government commissioned the OECD to conduct a first-of-its-kind study on the pervasiveness of innovation skills in its public workforce in order to inform its training strategy.

The research, while based on a small sample, enabled the consultants to follow up with in-depth interviews and later produce a 120-page report, documenting that innovation skills exist only in pockets in the Chilean public sector but without any coherent framework to bring them together or systematise them in public practice.

Respondents indicated a particular skills gap in the area of citizen engagement. They also felt that, whereas their organisations were better prepared to use data science skills, managers frequently did not support employee efforts to practice innovative ways of working. Alas, the OECD does not publish the survey questions.

Surveying digital trends

The findings in Chile paved the way for more expansive empirical research into public sector innovation skills. Following suit, the Canada Digital Service put out a skills survey at the end of 2018, asking public servants, “What digital training do you need?”

They used the results to shape the roll-out of the Digital Academy in 2019, which teaches digital literacy through a series of courses that target public servants at different levels of government — from short crash course videos aimed at all 250,000 Canadian public servants to intensive multi-day workshops for executives and senior leaders.

The Academy’s use of a skills survey prior to rolling out its training has led to a better-informed design and a more cost-efficient roll-out.

But asking people about their skills has its own limitations, as people are notoriously poor and unreliable at assessing their own expertise.

That’s why, GovLab developed a survey of innovation skills designed to overcome the deficiencies of self-reporting. Both ICMA and Bloomberg Philanthropies have circulated these currently ongoing surveys (here and here, respectively) to their constituents in the United States and ANZSOG did so in Australia and New Zealand. Results will be published at a later date.

Saying and doing are different things

In connection with six innovation competencies, we asked people to distinguish between: 1) their ability to use a particular method, or 2) explain it to others.

The reason we ask “can you explain this” is to get at a more honest answer than if we asked “do you know what this is.” If we asked the latter, everyone would say “of course.” More importantly, we ask people “can you do this thing.”

By asking people whether they can use a skill and following up that question with a request to specify how often, the hope is that we will get a more robust assessment.

Also, before any questions are posed, the survey presents the respondent with a description of each skill in words and in a video; a real world example; and additional resources.

This is done to avoid miscommunication about key terms. After all, one person’s design thinking is another’s human-centred design. We wanted to get to the bottom of what people know regardless of nomenclature.

Mapping knowledge gaps

The other benefit of surveying employees about what they know — and what they want to know — is that it enables us to determine how knowledge is spread across the organisation. Identifying such gaps gives us the ability to target training and resources to the areas of greatest need. The ANZSOG/ MSDI survey asks people about nine innovation skills:

Problem definition
Human-centred design
Data analytical thinking
Open innovation
Behavioural insights
Lean-agile methods
Impact evaluation methods
Evidence synthesis methods
Systems thinking

We try to pinpoint the skills gaps by starting with questions on the respondent’s position and role in the public Service. Questions like: “are you a state or federal public servant,” “what is your organisation’s jurisdiction,” or “what best describes the type of work you do?”

Asking these questions in parallel with an optional request for more detail about what kinds of problems people are tasked with solving could help ANZSOG triangulate which public servants use the most innovative methods and who is lagging behind.

In addition to asking what people know and what they would like to learn, you also find out how public servants want to learn so that you can ensure high participation in program design.

The ANZSOG survey accomplishes this by asking respondents where they like to learn (online, face-to-face, or group setting), when (at home or during work hours), and with what frequency.

Skills surveys are an important opportunity to take the pulse of innovation health in your organisation and generate sources of data to inform further qualitative research with public servants about how they will make the transition to new ways of working.

As Scotland’s troubled digital strategy shows, building digital capabilities in government — something many public institutions are trying to accelerate — is no easy task; it depends more on human capital than on technology itself.

But when armed with ambition, coordination, and a good skills survey, half the battle is already won.

Professor Beth Simone Noveck is Professor in Technology, Culture and Society and Director of the Governance Lab at New York University. She was also Deputy CTO and head of the Open Government Initiative in President Obama’s White House. She has served as an advisor to Prime Minister Cameron and Chancellor Merkel. She also serves as the Chief Innovation Officer of the State of New Jersey. Her new book Public Entrepreneurship will appear with Yale University Press in 2020.

Jason Williams-Bellamy is a second-year public policy major at the NYU College of Arts and Science. He is currently interning at the GovLab.