What counts as data?
The question governments can’t afford to keep ignoring
When someone says, “the data shows…”, what do you picture? A graph? A trend line? A figure in a briefing paper with a decimal point that signals precision and authority? That picture is incomplete in ways that have real consequences.
Word cloud from ANZSOG Masterclass: what public sector leaders picture when someone says, "the data shows…"
I’ve been sitting with this question for most of my professional life. When I realised I was more of an academic than a filmmaker, my Master’s research explored film funding in Aotearoa New Zealand. I noticed that national identity — one of the stated goals of public cultural investment — was being measured by box office receipts. Bums on seats. It’s a typical metric in policy: reportable, auditable, comparable. But it has nothing to say about whether a publicly funded film is contributing to national culture or shared identity in any way.
My PhD took this further, examining how local arts and cultural policy contributes to community wellbeing — and whether this can be measured. What I found, consistently, was that the indicators governments used to assess success in cultural policy were designed to show that the government agency was using its resources efficiently. They generally couldn't answer the questions about whether the programs or outputs were making any real difference in people’s lives (or social imagination).
Over the past 13 years, I’ve worked on a range of publicly-funded projects to design, re-design and review systems, services and programs. All of them have benefited from bringing together different kinds of data and perspectives to better understand what problems are occurring, who is most affected by them, and how we can improve outcomes for these people.
Too often, the hard data has been hard to access – police claiming to have but not providing statistics about who’s involved in graffiti vandalism; disability providers that hadn’t consistently tracked service use or ever asked for feedback before the NDIS; and much too commonly, publicly-funded evaluation reports hidden from the public eye. When, in contrast, a public health epidemiologist not only provided local statistics on chronic conditions, but actually broke them down for me, we could identify who was disproportionately affected. In that instance, we saw we needed to recruit older Indian and Pacific people to be part of our co-design group, as they were not only a significant demographic in the area, but half of those aged 65-74 years were identified as having diabetes. In this way, we were able to use population-level data to identify whose lived experience to prioritise including.
In every project that took a co-design approach, we elevated lived or living expertise so that it could be considered alongside professional, technical, cultural and scientific expertise. Sometimes these evidence bases came into conflict. Whenever that happened, it provided an interesting point of tension to explore, which always led to deeper, richer understanding. And when we effectively integrated these different types of knowledge through the design process – whether it was about reducing graffiti vandalism, managing chronic conditions, or improving disability services – we designed changes or interventions that would better meet the needs of all the people and organisations involved.
Yet I still hear qualitative, narrative, practice-based and participatory data dismissed as ‘anecdotal’. The cult of the measurable continues. What’s that all about?
The hierarchy we rarely name
Last week I opened ANZSOG’s Data, Judgement and Public Value series with a masterclass on ‘Valuing Knowledge’.
We used Ackoff’s hierarchy as a scaffold, distinguishing Data, Information and Knowledge from Wisdom. It’s not a perfect model, but it gave us a useful framework to interrogate. If wisdom is what we’re ultimately after in public decision-making, why do our evidence systems almost exclusively reward the bottom of the hierarchy? Administrative data, survey results, census counts: these are the things we fund studies to collect and defend at estimates.
Image adapted by New Know How from Sanders and Stappers’ (2012) version of Ackoff's DIKW scheme
In the masterclass, we worked through a rural healthcare scenario together, exploring each level of this hierarchy. It was easy for participants to think of data that might be relevant to the scenario: hospital admission and wait-times, referral drop-off rates, GPs per capita, public transport availability. But the knowledge held by a nurse who has worked in a rural community for fifteen years — knowing which patients won’t follow through on referrals and why — doesn’t appear in a dashboard. The understanding held by a First Nations elder about why community members avoid the regional hospital isn’t in the briefing paper. These are not soft add-ons to the “real” evidence. They are the knowledge forms that tell us most about whether an intervention actually works.
We then explored more rigid hierarchies like the National Health and Medical Research Council’s levels of evidence. These models are great for standardised, statistical approaches, but they don’t help to understand social complexity.
There is a risk that when we only collect what we can count, we become institutionally blind to what we can’t — and then we mistake that blindness for objectivity.
This is a political question, not only a technical one
The gap between what governments count as data and what actually matters to communities is primarily a problem of power, not methodology. Admission rates can be audited and compared across jurisdictions. A relationship-building process with a community organisation to surface long-held custodial knowledge cannot be put in a dashboard, and would require justification at every stage of procurement. The friction is structural, not individual.
Yet it’s too easy to blame bureaucratic and political systems, and forget they’re made up of people who can do things differently. CEO of Stronger Smarter Institute, Chris Sarra, cautioned a public sector audience earlier this month:
Systems can become so consumed by targets, outputs, frameworks, reporting structures that we sometimes lose sight of the actual human experience of the people we are supposedly serving, and when that happens, we drift towards what I would describe as a transactional model of governance, a model where governments increasingly do things to communities rather than with them.
For all the talk of elevating lived experience and citizen voices in health and social policy discourse, our public systems are still very bad at integrating these perspectives and stories in practice.
There’s a phrase I keep returning to: not everything that counts can be counted. Einstein probably didn’t write it on his blackboard at Princeton, though often gets the credit for it.
It’s more than a gentle reminder about human complexity. It's a critique of the conditions under which certain kinds of knowledge are systematically excluded from policymaking.
What inclusive rigour actually requires
Broadening what counts as data is not a lowering of standards. Creating usable knowledge to make wise decisions requires the inclusion of insights from those closest to a problem and other relevant parts of a system. It requires a more demanding and honest account of what rigour means.
Robert Chambers, writing about evaluation in international development, distinguishes between reductionist rigour, as adherence to statistical and standardised procedures, and inclusive rigour, as cost-effectiveness in learning. That is, inclusive rigour entails an honest accounting of the trade-offs between validity, timeliness, relevance and credibility, set against cost (including the opportunity cost of people’s time and other resources). This is what we need when conditions are not uniform, predictable and separable.
So, in public policy and community contexts, we need inclusive not reductionist rigour. The framework pictured breaks this down into three domains:
HOW: Remixing with purpose (Methodological bricolage)
WHO: Meaningful involvement (Facilitating participation and inclusion)
SO WHAT: Useable knowledge (Utilisation and impact).
Inclusive rigour, represented by New Know How, based on Chambers and Apgar et al.
In co-design and co-production, we talk about “good enough evidence” — not as a compromise, but as a recognition that waiting for certainty in a complex public context is itself a decision with costs. Communities bear the cost of inaction while institutions seek methodological comfort.
What inclusive rigour requires of public sector leaders is creating conditions for different kinds of knowledge to enter the room:
commissioning questions that don’t predetermine the answer;
procurement norms that treat community knowledge as primary evidence rather than supplement; and
institutional permission to name what the data doesn’t show.
It also requires interpersonal and cultural shifts, to co-create psychologically safe environments where we can grapple with conflicting data and challenge dominant narratives, and admit the limitations of our knowledge and our methods.
These shifts won’t happen overnight. But they’re worth struggling for. What we count as evidence shapes whose experience is visible in policy — and whose is erased.
The question I want to leave with anyone thinking about this is: whose knowledge does your organisation/practice currently privilege, and whose does it structurally exclude?
