Gitea server keeps filling up with bundle/zip archives

[toloveru]

Description

Hi, I am the maintainer of git.nixmagic.com. After using Gitea on it since 2020-06-04, I recently started experiencing storage issues on both gitea-1.23.5-linux-amd64 and gitea-1.22.1-linux-amd64 (both upstream binary releases). This may have been present before too, but unlikely. Currently its instance is on a 64GB storage quota (applied after this incident consumed a whole ZFS pool instead), which it exceeded in less than 48 hours from 13GB base. This was temporarily resolved by executing the command below.

Directory: ~/data/repo-archives
User: git
Command: for file in $(find . | grep -E 'zip|bundle'); do rm $file; done

This leads me to believe that zip and bundle archives are being generated at an unreasonable pace. In other issues and related documentation, I found that such files are generated on-demand by users of the instance. However, my edge nodes that terminate TLS for this node do not confirm this being an active attack. They don't seem to have much traffic at all. That leads me to believe that this is an internal issue. For the time being, I'll apply a cron job to periodically flush these files as mentioned there. But that's a band-aid, not a solution. Both generating and eliminating these files consumes unneccesary CPU and IO resources.

If you're willing to take a look into the source code to narrow this down, much appreciated. Similarly, I'll keep track of this issue to offer any information I can from my instance. Thank you!

As a side note, is it possible to downgrade Gitea? I tried to replace my symlink from 1.23.5 to 1.22.1, but this caused the associated service to fail. Both files are/were executable, but I don't remember why exactly it failed. I can start this service both in systemd and manually, so troubleshooting here is possible. However, it does incur service downtime that I'd prefer to avoid unless necessary.

If need be, I can clone this entire instance into a local replica. This replica would have a filesystem-level one-to-one copy on account of ZFS, but would not necessarily be tethered to the production instance and its reverse proxy. The replica will require network and other such client-side changes. This may be useful to determine whether this issue is truly local at the application level. If determined useful, I will make such replica and document it accordingly.

Tracking on snapshot ssd/lxc/ct/gitea@2025-03-25_GitHub, created / completed on 02:02 CET.

Gitea Version

1.23.5

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

nixmagic.com/pub/2025-03-24_Gitea - I'll publish logs here as needed. So far I'm thinking of e1.nixmagic.com and e2.nixmagic.com's NGINX logs for git.nixmagic.com. Those should offer an overview into the Internet's requests towards this Gitea instance. Internally, the only user is yours truly. But infected hosts may exist. Logs from Gitea itself may provide equally useful, if not more so. Please let me know.

Screenshots

No response

Git Version

2.39.5

Operating System

Debian 12 amd64

How are you running Gitea?

My deployment of Gitea is from upstream binary release, for amd64 target architecture. This is on a Debian 12 stable system, running within LXC. The storage system is ZFS, which is how the quota has been applied from the host system.

Database

None

[lunny]

Hi, Gitea doesn't support downgrade at the moment.

I believe some of the archive files' generating are coming from web crawler.

There is a tool Delete all repositories' archives (ZIP, TAR.GZ, etc..) at admin panel to safely delete all these archive files.

[toloveru]

Thanks for the heads-up! I saw some requests coming from OpenAI's crawler when I investigated this earlier on my edge nodes. The traffic is still somewhat negligible, but it might be worth adding in robots.txt like I did with Googlebot before. I'm currently monitoring traffic on both of my edge nodes, paired with a periodical df -h on the Gitea instance. So far, this results in a fairly predictable fill-up of ~1GB/h, but I should investigate further. Perhaps I could attach an inotifywait to that directory as well. Regardless, I want to get to the bottom of this.

e1.nixmagic.com was 15 minutes behind on its time. This was resolved by installing ntp, but will affect log files before 2025-03-24 T 23:39 and T 23:54, both CET. e2.nixmagic.com meanwhile appears to have remained roughly on time, despite also missing this package.

[wxiaoguang]

I guess it's not related to Gitea's version.

Have you tried to set DISABLE_DOWNLOAD_SOURCE_ARCHIVES=true do disable the "download archive" links?

[silverwind]

Not sure why we even store repo archives on disk. zip/tar files can be created in-memory and streamed to the downloading client using chunked transfer encoding, no disk necessary.

[tobiasbp]

One might want the archives to be stored for the purpose of caching.

[lunny]

Not sure why we even store repo archives on disk. zip/tar files can be created in-memory and streamed to the downloading client using chunked transfer encoding, no disk necessary.

And the archives can be stored in the S3 compatible services which cost is very low.

[silverwind]

Storing archives should be made opt-in imho, it's a huge waste of disk space and a risk for public instances as seen by this issue.

[lunny]

However, generating archives can be CPU-intensive. We could consider focusing on issue #34024 to reduce frequent archive download requests.

[toloveru]

Hi, I still have to investigate further to find the cause of this issue. But what I would like to mention so far is that the disk fill-up went away, as inexplicably as it came. I'll try to make work of sifting through the NGINX logs tonight.

As far as the suggestion of outright removing the ability to download archives goes, at least for ZIP format I have reservations (neutral for bundle, albeit just for me not using it). While people deploying Gitea instances are likely to be self-hosting more broadly and have a decent understanding of Git, that may not be the case for its users. For a variety of projects (e.g. having been the second public mirror of youtube-dl during takedown, with others having created several duplicates on my instance), my instance has attracted users that may or may not be familiar with Git themselves. I do not want to inconvenience them. If they want to download to ZIP, they should be able to do so unimpeded.

I do like the idea of using chunked transfer encoding, given that cache can be done externally with NGINX. My instance has this disabled for git.nixmagic.com, but does use it for the main site (e.g. to accelerate video). I don't see why the ZIP archives couldn't be handled the same way, under NGINX' quotas. Bit more complex perhaps, but download as ZIP is something I don't consider to be a repeat operation for most human users. Granted, how would that affect the cache when it's not expected to be downloaded twice in short succession... It might just be evicted from the cache before that, and scrapers aren't going to download the same file over and over again either (at least I hope).

This is speculation on my part, but I do wonder if this might not have been a badly behaved scraper. Especially for AI, I've seen some materials about that recently (e.g. here and here). Apparently they even ignore robots.txt, so it wouldn't surprise me if they would also crap out on large download links (ballpark 200GB on aggregate). Infrastructure companies as large as Cloudflare are now putting front-lines out for precisely this purpose, but not everyone (including me) uses that. I wonder where that would be best fronted. Perhaps the NGINX instances could serve such purpose, or maybe the rate-limiting of iptables could do it too.

Should Gitea at the very end be bothered with that though? To me, it remains an open question. Nonetheless, the ability for anything to take down a Gitea instance like this is a security concern to me. Once the disk was full (instance with quota, entire pool without), I could no longer interact with Gitea's web interface. It would either throw HTTP 500 when the service was still running, and would fail to restart when requested from systemd. This is why I ended up "unsafely" deleting those ZIP and bundle archives from console. The service was completely cooked, in a textbook denial of service.

[toloveru]

I think I just struck an OpenAI-sized golden turd here. Not an opportunistic individual in a basement somewhere, one of the largest AI companies in the world. That's all I'll say about it for now. Deep breaths.

By now, I have also raised this issue to OpenAI's support team, with guidance from ChatGPT (couldn't pass on the opportunity to see what it had to say about this). Relevant links are included in nixmagic.com/pub/2025-03-24_Gitea.

Notably, this includes the enchanta.eu.org domain, which until now I considered to be low maintenance. It also goes to the Gitea instance, through NGINX' reverse proxies. At the time of writing, it does not have a robots.txt file. Creating this file for that domain, may be a first step into preventing repeat infractions on OpenAI's part.

[silverwind]

However, generating archives can be CPU-intensive. We could consider focusing on issue #34024 to reduce frequent archive download requests.

Either way is a tradeoff, but it'll be good to have such an option in any case so user can decide what they value more: disk space or cpu consumption. In case of a private instance with low traffic but limited disk space, I'd certainly value disk space more.

[okaestne]

My server also recently went out-of-disk space within a day twice because of an aggressive crawler network. It created large and a huge amount of files in data/repo-archive until I banned the whole cloud provider's IPv4 ranges using my firewall.
The recently added option to block expensive requests is far too broad and strict for my taste. I would prefer to have an option to disable the archive disk caching as @silverwind suggested.

Thanks a lot for all your anti-scraper efforts lately!

[toloveru]

FWIW, the other day this came up on the BIND-Users mailing list as well (Survey on the impact of software regulation on DNS systems). There was a link to the FreeBSD forums, which in turn had someone link to another Cloudflare blog post. What's interesting to me, is how they include a graph of the various user-agents that are making such malicious traffic. This may be useful for you and I to include in a robots.txt, before / instead of having them front our services. I do appreciate that they include protection from AI scrapers in their free tier though, and that this brings their extensive experience with Internet infrastructure to the table.

[silverwind]

robots.txt is a good idea, and it's already supported when put into the custom dir. I think we could also ship a default robots.txt with exclusions for archive downloads etc.

Thought there might be legit use case for archival purposes like archive.org crawler crawling archives. It's not so clear-cut I guess. Maybe limit it to only non-main-branch archives.