Computers gung ho? Responding to the OAPEN Library and DOAB’s response time 

Ronald Snijder

Fri 11 Apr 2025

Read this article at hypothèses.org

The catchphrase ‘computer says no‘ is something most of us are familiar with; it’s frustrating when an digital system is not behaving the way we want. Lately, at OAPEN and the Directory of Open Access Books (DOAB) we are experiencing something else that might be described as ‘computers gung ho’.  

What was the problem? 

The OAPEN Library and DOAB received so many requests from automated systems that they drowned out genuine users. On Tuesday 25 March, the problem reached a tipping point for the OAPEN Library when we saw this happening for several hours. From 8:30 up until 12:00 the number of sessions on the OAPEN Library exploded from 15 per second to 1,500 per second. As a result, the average response time of the OAPEN Library dropped from 0.2 seconds to 1.2 minutes, rendering the service unusable. Thanks to the support of our hosting partner CERN, the system did not crash. 

A graph showing the averag number of session oand the average response time of OAPEn and DOAB

Image: A graph showing activity in the OAPEN Library (green line) and DOAB (yellow line) on Tuesday 25 March. Credit: Grafana Dashboard, provided by CERN. 

If you look closely at the graph, you can see that these peaks in sessions were not confined to one morning only. We now see this happening several times per day, but only for short periods. 

Why is this happening? 

The reason for these bursts is that AI systems are hoovering up the content in the OAPEN Library and DOAB. OAPEN and DOAB are not the only websites that need to deal with these type of visitors. Eric Hellman describes in his blog “AI bots are destroying Open Access” how this has become a problem for many providers of open access content. When your mission is to make knowledge as widely available as possible, blocking AI bots should not be a necessity. This leads to unintended consequences, for example, the Internet Archive can no longer save snapshots of MIT Press because of Cloudflare blocking. The problem is not confined to open access providers, but also those who work on open source software see the same thing happening. 

What can be done about it? 

Our goal is to continue providing unfettered access to a quality curated collection of open access books. There is no simple solution: the requests come from different addresses and mostly do not identify as bots. This makes it hard for us – and all others dealing with this issue – to distinguish between genuine users and AI tools ‘gung ho’.  

Together with our colleagues at CERN we are working on the resilience of our systems, to make sure they keep functioning. We are also investigating if there is a way to deal with the peaks in downloads in such a way that it will allow human readers to keep using the OAPEN Library and DOAB without interruptions.  

While we keep working on a solution, we thank you for your continued patience when our services are temporarily slower than usual.