CitizenQuery-UK: A Person-Centric AI Benchmark

Citizen queries are questions asked by an individual about government policies, guidance, and services that are relevant to their circumstances, encompassing a range of topics including benefits, taxes, immigration, employment, public health, and more. This represents a compelling use case for large language models (LLMs) that respond to citizen queries with information that is adapted to a user's context and communicated according to their needs. However, in this use case, any misinformation could have severe, negative, likely invisible ramifications for an individual asking a citizen query.

To this effect, I led a multinational team on the development of CitizenQuery-UK, a benchmark dataset of over 22,000 pairs of citizen queries and responses that have been synthetically generated from the swathes of public information on gov.uk about government in the UK.

We present the curation methodology behind CitizenQuery-UK and an overview of its contents. We also introduce a methodology for benchmarking LLMs with the dataset, using it to benchmark 11 models for factuality, abstention frequency, and verbosity.

We document these results, and interpret them in the context of the public sector, finding:

AI models are competitive with each other and will get better. Open, closed, small, and big models can do the job, and they’ll keep improving. “Vendor lock-in” should be avoided at all costs.

High variance and “long tails” undermine trust. AI models are inconsistent and hard to rely upon, although many trust them in their day-to-day lives. We should focus on ensuring clear communication of the risk of misinformation and who people can go to in order to find the answers they’re looking for.

Models like to talk -– a lot. With “word salads”, AI is overwhelming. But when we make it say less, it sometimes gets more things wrong, bringing in information from across the web rather than just focusing on authoritative government information. We need to improve the prioritisation of government sources, and this starts with making these sources AI-ready.

AI is not brave enough to say “I don’t know.” A lack of fallibility massively undermines factuality and overall utility. Technology providers might not want their AI to admit it doesn’t know something, but no advice is better than bad advice when it comes to citizen queries.

The work was presented at the Neurips 2025 workshop on LLM Evaluation.

Read the preprint

Do you trust the technology designed to keep you safe?

Privacy-enhancing technologies (PETs) can provide organisations with enhanced security and confidentiality of data and code, but several challenges and barriers stand in the way of their adoption. One such barrier is a lack of trust: sceptical, privacy-aware individuals find it difficult to trust that PETs will effectively keep their sensitive data and code protected and safe.

In this research, we sought to understand trust in PETs and the factors that motivate and dissuade their adoption. We aimed to analyse different transparency measures that PET providers can use to increase trust, and the extent to which they do so in practice.

We focused on the context of trusted execution environments (TEEs), a type of PET with undeniable benefits for organisations looking to use confidential computing. TEEs also present particularly interesting dynamics around their trust and adoption, and we used Google’s Project Oak, an in-development piece of TEE infrastructure designed with a number of transparency measures, as a framework to explore them.

Using qualitative data collected from three activities, we employed thematic analysis to determine: (i) transparency measures only motivate trust when they are meaningful; (ii) when it comes to trust in PETs, principles often trump efficacy; (iii) Google’s Project Oak’s transparency measures have strengths and limitations; and (iv) technical transparency measures only address certain concerns for certain actors.

We found that the onus on PET providers to make trust decisions easier, which requires both transparency and the use of socio-technical measures, was an overarching theme. We briefly explored how these findings can be broadened to other types of PETs and proposed a set of recommendations for further research and future work.

Read the report

Quantitatively measuring the UK Gov. as a data provider for AI

Artificial intelligence (AI) promises opportunities for greater economic efficiency and quality of life around the world, but may also bring new socio-technical challenges and safety risks that governments will need to address, both as regulators and as providers of data.

Governments typically collect and steward a vast amount of high-quality data on their citizens and institutions, from official statistics releases to national archives. Access to government data is incredibly important for AI, not only because of the potential of open government data to catalyse the development of the AI ecosystem but also because of its potential to enhance AI tools like LLM-based chatbots to respond to citizen queries and other public service delivery. While the UK government is currently acting as a data provider for AI in some regards, it is not in others.

Generic large language model capabilities are not enough to provide accurate information to citizens. Improvements could be achieved by making government data AI-ready, but also by investing in LLMs whose functionality has been tailored and audited to ensure it provides accurate, balanced answers. Innovators all over the UK building such tailored solutions should be supported with up-to-date AI-ready government data. We believe the government should ensure equitable access to data by acting as a data provider for AI, and our report outlines a set of actionable recommendations to accomplish this role.

Read the report

Planning optimal healthcare infrastructure in Africa

In early 2024, I set myself the task of synthesising two open datasets into something with meaningful impact, and then communicating that in a more informal way to my typical work. The resulting article contained a surprisingly strong methodology and output, with my healthcare analysis finding its way to the World Bank transport team through a connection at healthsites.io. But something I'm more proud of is the way I pushed my authorial voice in this piece, speaking on some fine-grained technicalities with humour and lightness that culminated in a pretty successful reception.

You can read the piece on Medium here.

The code is entirely reusable, accessible on Github here.

Analysing human behaviour in COVID-19 lockdowns

Awarded "Best Research" at IEEE SeGAH 2024 (Madeira).

How did your daily behaviour during COVID-19 lockdowns change over time? UCL dPHE published an app in June 2020 that provided a gamified journalling service, inspiring more than a thousand users to record their activities every day for their own mental wellbeing and for the purposes of our behavioural research.

In 2023, I began working on analysing this data -- at first as part of a paid summer internship, then afterward in my free time. The data is difficult to work with and all the idiosyncrasies of human behaviour make robust statistical analyses difficult, but I've been able to get quite far by focusing on the question: how did users react to the frequent changes in UK lockdown policy enforced by the government?

Initial findings indicate a mix of inertia and high-responsiveness, especially with regards to sporting and social activity. However, more statistical significance is visible when users are categorised as urban and non-urban.

In summer 2024, I presented the first half of our research, my first-authored paper on the efficacy of the app's gamification, at IEEE SeGAH 2024 in Madeira, Portugal. While on the plane back home, I was informed I had been awarded Best Paper on account of my multifaceted approach to evidencing claims.

The second half of the research -- large sets of time series regressions on the data -- is ongoing.

Investigating the dirty world of fresher hazing at UCL

At university, I wandered into The Cheese Grater at a time where I wanted to spend time doing something more than focusing on my degree in Economics. My stint at The Cheese Grater started from a desire to talk to more interesting people than my classmates in BSc Economics and ended with me seriously considering writing as a career. I worked on the investigative journalism side of the magazine (having been deemed not funny enough for the satire side) and, over a year and a half, built a role for myself as a dependable investigative writer with a frankly workaholic nature and distinct, characterful voice in my articles --- "sassy", as described by the only editor that liked it.

I rose through the ranks and led several long-term pieces, culminating in a national award in 2023: "Best News Story" at the Student Publication Awards, given to my article that judges from Bloomberg commended for its methodical investigation and unique, sardonic voice.

The article illuminated a seedy underbelly of UCL sports society culture, with intiations supposedly banned but freshers instigated into volunteering for traumatic, sometimes permanent, harm. We exposed some harsh stuff, and somehow managed to uncover the finances underneath some of it. I led the team, and led the writing too, which gave me some crucial experience I've been able to lean on across my work ever since.

You can read the article here.

Giving vulnerable students a voice against bureaucracy

With less fanfare than my other Cheese Grater piece, this article has nonetheless embedded itself into my head. A colleague and I interviewed another student -- older than either of us, but still a young adult -- who really laid bare his experience with UCL's inane bureacracy and how it led to heartbreaking distress.

The interview probably lasted two hours or so. A majority of that time was an almost unbroken account, by the student, of three years of failure by the UCL administrative staff to enroll him, accomodate him in university residences, facilitate his needs, simply remember him during COVID, allow him to take a break upon family health problems, authorise his extenuating circumstances, keep their own promises, and finally hold themselves accountable when it all ended in a suicide attempt. I guess I saw a picture of what life is like if you simply get unlucky.

I don't think the full extent of the story really came through in the article. Nonetheless, you can read it here.

Simulating macroeconomics with a web app

Frustrated with the way I was being taught macroeconomics, I decided the most effective way to learn the content was to remake it all in python. I employed a lot of object oriented programming, trying to build simulation environments for me to introduce macroeconomic shocks and visualise the diagrams presented in lectures that never made sense to me.

After building a prototype, I realised how useful this would be to other people. In the macroeconomics exam at the end of the year, 50 students were using my web-app. I presented it to the UCL Economics department, then to a live audience at ExploreEcon 2022 where I won "Best Poster" for something that wasn't really a poster.

An Actionable Framework for AI-ready Data

Data is the foundation of AI. Poor-quality data drive up costs and can lead to hidden problems for AI models, especially in complex fields such as healthcare and manufacturing. Meanwhile, biased data negatively affect the performance of AI models, and untested evaluation datasets can result in false positives or overestimates of model accuracy. For data publishers to realize their true potential in supporting the AI ecosystem and its impacts, they should take measures to ensure that their datasets support AI practitioners' needs; in other words, their data should be made AI-ready.

In this article, my team and I present a framework for data publishers to follow to make their datasets AI-ready. The framework provides specific, actionable guidance based on previous work and experience at the Open Data Institute and augmented with insights from literature and discussions with a range of experts. We first define AI-ready data before briefly discussing a selection of frameworks in the literature and where they are insufficient. We then provide a visual snapshot of our framework for AI-ready data, and a subsequent in-depth discussion of its criteria. Finally, we demonstrate the usage of our framework with a number of example datasets. We conclude by discussing the further steps that should be taken for the entire open data ecosystem to be made AI-ready in order to realize its true potential in supporting an innovative future.

This framework has been applied to local government and the water sector in the UK, and we're currently working on iterating it to investigate where enterprise SaaS solves (and doesn't solve) issues with AI-readiness.

The work reached an international audience and was peer-reviewed for republishing in the AAAI AI Magazine.

Read the article