The limits of civic tech: Learning from kleineAnfragen

I originally wanted to post this as a Twitter thread, but as I started laying out my thoughts, it quickly became apparent that I had far more to say than could be fit into 240 characters. So, here are my thoughts in blog form.

The German open data and civic tech world suffered a shock to its system two months ago when the website kleineAnfragen.de announced it would cease operations at the end of 2020. The volunteer-run website is a searchable, machine-readable repository of answers to parliamentary inquiries for all 16 German states as well as the federal government and is an immensely valuable archive of public information that is not otherwise made available in a systematic, machine-readable (and thus re-usable) way.

The plans to close down the website reverberated throughout the German-speaking tech community last month, but I think it’s important that the English-speaking world be aware of this story as well, because it’s not a story unique to Germany. kleineAnfragen is a volunteer-run website that serves a critical role in making German state-level governments as well as the federal government more transparent and accessible. They are shutting down because the person who runs the site became fed up with the Sisyphean task of keeping the site running smoothly while government entities constantly revamp and relaunch their web presences with no regard for standards or consistency.

Anyone who has ever been involved in a civic tech initiative will see their own experiences reflected in this story, and the universalism of this experience makes it clear: the status quo whereby the civic tech scene is responsible for a huge portion of the work being done for a more transparent and open digital future while only being supported by short-term grants of money (if funded at all) is not sustainable. Governments need to do more, and they need to do better.

It’s not enough to have a vibrant civil society that pumps out web apps and interactive visualizations to help citizens interact with and better understand their governments. Projects that are volunteer-run or based on short-term funding – as many civic tech projects are – cannot be the backbone of digital, transparent and open governments. Governments in Germany and the world over have to recognize that at the end of the day, it’s up to them to invest in sustainable, functional digital platforms and the digital infrastructure that supports them.

Read on for a more information on what the specific situation with kleineAnfragen is.

What is kleineAnfragen?

In the German parliamentary system, parliamentarians have the option to request information from the government on any matter that the government should reasonably have knowledge of and be able to publicly disclose information of. Examples of typical questions asked are “How many bikes are stolen per year in Berlin, and in which areas are the most bikes stolen?” or “In what ways is predictive policing currently being used or planned to be used in police operations?” These questions are known as “kleine Anfragen” (literally: small requests/inquiries).

These Anfragen – specifically, the answers to them – are interesting for both politicians and non-politicians for two main reasons. First, the answers are public record, so they provide a valuable insight into and a record of government policies and how they are being implemented and evaluated. Indeed, they are a particularly valuable tool for the opposition parties in a parliament, who use kleine Anfragen as a tool to force the government to express an opinion or stance on issues that they might otherwise prefer to stay silent on, or to draw critical attention to specific policies.

Second, these answers often contain tables of data that are not published anywhere else. These answers are exclusively published as PDFs, so it’s not as though they are an ideal resource for government data, but its usually possible to extract the tables and convert them to something machine-readable (like an Excel or CSV file) with a bit of manual work (using a program like Tabula, for example). And at the end of the day: data in an undesirable form is still better than no data at all.

The website kleineAnfragen is an online database of all of these answers to inquiries as published by all 16 German state-level parliaments as well as by the federal government. The website currently hosts more than 100,000 documents that they scrape from parliament websites and re-host on the platform.

What’s uniquely special and valuable about the website is not just that they gather all of these PDFs and store them in a central location, but that they make it possible to search and filter documents – functionality that most states still don’t provide. kleineAnfragen automatically converts the PDFs into unstructured but machine-readable text, and they also automatically detect tables and present that as a filtering option for searches. This is useful when you’re specifically looking for data on a subject, rather than just a discussion of it. They also have an API which makes it possible to extract large amounts of data for more ambitious analyses.

Why is kleineAnfragen shutting down?

The individual behind the website wrote out the reasoning behind his decision here. It boils down to: the amount of constant work necessary to keep the site functioning properly (due to parliaments frequently changing aspects of their document publishing processes that in turn require the site to update its data harvesting approach) and the fact that after five years of running the site, governments in Germany have made no meaningful steps toward properly publishing these documents as open data.

If you follow the link where the decision to close the site is explained, you’ll see this image:

View fullsize

A timeline featuring updates made to the site over its five years of activity

What we see depicted here is a constant battle to keep up with changes to the local parliamentary document publishing systems – for example, when a hosting site is updated and suddenly the URLs of documents are completely different, requiring a re-write of the kleineAnfragen code to ensure documents can still be found and harvested.

Ensuring the site remains active and functioning is thus a significant task, especially for a site that is run by a single volunteer. And the crux of this is: it doesn’t have to be like this. It isn’t a fact of life on the internet that URLs have to break or document structures have to be arbitrarily changed; it’s normally possible to update databases and websites without causing these kinds of isuses. Most simply: if parliaments would publish the answers to kleine Anfragen as structured, machine-readable open data, the effort required to run a site like kleineAnfragen would lessen massively, since the data would be significantly easier to extract and re-publish. Even better would be if each of the 16 state parliaments and the federal parliament would use the same data standard to structure their documents so that 17 individual solutions don’t need to be developed to harvest the data in a systematic way (which is indeed the status quo right now). And best of all of course be some sort of API or other interface that makes it possible to directly access up-to-date data.

Who will be affected by this closure?

A lot of groups. For anyone who ever needs to do research on government policies or who is looking for official sources of data for a given subject, kleine Anfragen can be an excellent resource for getting official information from the government. Journalists obviously fall into this category, but the usefulness of the site extends to other groups too. For example, I have used the site extensively to write my two “Data Dive” pieces at work, where I looked at the availability of cycling-related data and accessibility-related data for Berlin. I used the website to find data sources that were not being published as open data. My co-worker also recently published an analysis of kleine Anfragen in Berlin, looking at what topics are most frequently asked about in the inquiries, and how we can use that information to better target open data activities in Berlin. For this project, the kleineAnfragen API was particularly useful.

But it’s worth highlighting that a major user group of kleineAnfragen is almost certainly politicians and government employees themselves (as kleineAnfragen’s farewell post mentions). Both groups have an interest in knowing how previous inquiries were answered, for example – government employees want to make sure they are answering similar questions consistently, and politicians want to know how previous inquiries on a given subject were answered so they can formulate their questions appropriately. And given that there is no comparable internal government system for archiving these answers, kleinAnfragen is very likely a critical resource for politicians and government officials alike.

(Note: I know anecdotally that this site is absolutely used by politicians and government employees, but when writing this blog post, I tried and failed to find quotes from these individuals on social media or elsewhere verifying they use kleineAnfragen. If you know of any source for this – or you yourself are a politican or government employee who has used the site to aid your work – I would be extremely interested to see the quote/hear from you)

So what happens now?

In its farewell post, kleineAnfragen pointed out that there technically is a government website that is supposed to fulfill a function similar to kleineAnfragen, the government-run Parlamentsspiegel.de. A quick visual comparison between the two makes it immediately clear that the federal portal leaves much to be desired. The site is slow, clunky to use, and lacks many of the valuable functions that kleineAnfragen provides, like the ability to filter results by the presence of tables, the provision of machine-readable text versions of the documents, and an API.

kleineAnfragen is completely open source, and as such all of its code is available on GitHub. Meaning, if the government wanted to, it could utilize some or all of the website’s code to provide similar functionalities on its own websites. Or, they could even reach out to the Open Knowledge Foundation, which sponsors the website, or to site’s webmaster to see about working together to develop a better government platform using the expertise kleineAnfragen has built up over the years.

But to date, this hasn’t happened. And thus when kleineAnfragen closes at the end of 2020, assuming the status quo remains the same, there will be no real alternative to the website. It will disappear, and with it an exceptionally valuable repository of knowledge about government activities – knowledge that everyone is entitled to, but which is not currently being made accetptably accessible.

The plight of kleineAnfragen painfully demonstrates the precarious status of many civic tech initiatives today: these initiatives spring up to fill very real gaps in governments’ digital offerings to citizens, and in doing so relieve governments of a need to finance and develop these solutions themselves. But in many cases, these initiatives never receive stable government funding or the integration of these projects into existing government services. The projects live for a few years, the government reaps the benefits in the meantime without bearing any of the costs, and eventually money and/or enthusiasm for the project runs out. Things return as they were: citizen needs once again go unmet, and governments still haven’t built up their digital capacities.

kleineAnfragen has thrown down the gauntlet and called out parliaments across Germany for their failure to build modern, effective and properly transparent infrastructure for the processes surrounding kleine Anfragen. Will the parliaments finally be forced to act? Current indications are not positive, but perhaps we will see more movement when the actual closure date of the website draws nearer.