OOM caused by numerous crawls

[H0llyW00dzZ]

Description

In the latest versions, 1.23.2 and 1.23.3, memory leaks occur. (update: see below, not memory leak, not regression)

These OOMs are caused by numerous crawls, such as those used by Facebook Inc. (Meta), Amazon (AWS), and other entities that fetch data excessively for AI training.

My Gitea self-hosted configuration:

• Sessions using files
• Cache using Redis with a TTL of 5 hours, and the last commit cache is 10K
• No SSH

Screenshots

The logs exemplify how these companies use crawls for their AI.

Essentially, memory leaks occur when there are many fetch requests, leading to crashes due to excessive memory consumption (thanks to OOM Kubernetes).

[wxiaoguang]

Could you download a diagnosis report from "admin panel -> monitor -> trace" when the memory goes high?

The report contains heap dump (no sensitive data) and could help to locate the problem.

[H0llyW00dzZ]

Could you download a diagnosis report from "admin panel -> monitor -> trace" when the memory goes high?

The report contains heap dump (no sensitive data) and could help to locate the problem.

Here is the system notice:

This is the system status, which shows an inconsistent system status, as I mentioned earlier in #33311.

[wxiaoguang]

Could you download a diagnosis report from "admin panel -> monitor -> trace" when the memory goes high?

The report contains heap dump (no sensitive data) and could help to locate the problem.

[wxiaoguang]

If the memory is not related to Gitea process, then maybe you need to figure out which process consumes that memory, for example: git process? or some other commands?

[H0llyW00dzZ]

Could you download a diagnosis report from "admin panel -> monitor -> trace" when the memory goes high?

The report contains heap dump (no sensitive data) and could help to locate the problem.

I can't capture the memory usage when it spikes via the trace admin panel because every time memory consumption goes high (e.g., 7 GiB), it crashes due to OOM Kubernetes.

[wxiaoguang]

Could you download a diagnosis report from "admin panel -> monitor -> trace" when the memory goes high?
The report contains heap dump (no sensitive data) and could help to locate the problem.

I can't capture the memory usage when it spikes via the trace admin panel because every time memory consumption goes high (e.g., 7 GiB), it crashes due to OOM Kubernetes.

Is it clear that which process consumes that much memory? The Gitea web server process itself, or other processes like "ssh" or "git" or "gitea serve/hook"?

[wxiaoguang]

The logs exemplify how these companies use crawls for their AI.

Essentially, memory leaks occur when there are many fetch requests, leading to crashes due to excessive memory consumption (thanks to OOM Kubernetes).

If the OOM is caused by crawls, then it isn't a regression: each request consumes memory, some large repo/files consume more, then if there are lot of requests, these requests do consume a lot of memory and would lead to OOM. Maybe you could try to make stop the crawls and/or require sign-in for your instance.

So I think we need to make the problem clearer.

[H0llyW00dzZ]

Could you download a diagnosis report from "admin panel -> monitor -> trace" when the memory goes high?
The report contains heap dump (no sensitive data) and could help to locate the problem.

I can't capture the memory usage when it spikes via the trace admin panel because every time memory consumption goes high (e.g., 7 GiB), it crashes due to OOM Kubernetes.

Is it clear that which process consumes that much memory? The Gitea web server process itself, or other processes like "ssh" or "git" or "gitea serve/hook"?

Most likely, it's from Git because the stack trace shows this:

When there are many requests, such as GET requests to view repositories from crawls, memory consumption goes high, and it crashes due to being OOM killed by Kubernetes.

[H0llyW00dzZ]

Also right now, I've rolled back to version 1.23.1 and reduced the cache for last commit messages from 10K to 5K in the app.ini configuration. Let's see if it still crashes.

[wxiaoguang]

TBH, I do not see related change between 1.23.1 ~ 1.23.3

v1.23.1...v1.23.3

[H0llyW00dzZ]

TBH, I do not see related change between 1.23.1 ~ 1.23.3

v1.23.1...v1.23.3

Well It worked fine for me previously, with uptime of over a month without crashing due to high memory consumption.

And now, after rolling back, it still crashes.

h0llyw00dzz@ubuntu-pro:~$ kubectl get pods -n gitea
NAME                     READY   STATUS    RESTARTS      AGE
gitea-5cb7dff998-xwb5r   1/1     Running   1 (40s ago)   10m

h0llyw00dzz@ubuntu-pro:~$ kubectl describe pods -n gitea
Containers:
  gitea:
    Container ID:   containerd://866d173132606a07e7937e7dfb430533cf1e5a8ad515044e496486416f6a485c
    Image:          gitea/gitea:1.23.1
    Image ID:       docker.io/gitea/gitea@sha256:c3be67d5c31694f8c27e5f3ab87630cceadf05abb795ab0ed70ba14b5edfc29c
    Port:           3000/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Thu, 06 Feb 2025 18:30:03 +0700
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Thu, 06 Feb 2025 18:20:10 +0700
      Finished:     Thu, 06 Feb 2025 18:30:01 +0700

[wxiaoguang]

Well, as I said above: it can't be a regression, it can't be related to the new version.

There are just more crawls now. If you do not have that much resource support the crawls, maybe you need to block the crawls.

[H0llyW00dzZ]

Well, as I said above: it can't be a regression, it can't be related to the new version.

There are just more crawls now. If you do not have that much resource support the crawls, maybe you need to block the crawls.

For now, I've enabled REQUIRE_SIGNIN_VIEW to disable crawls used by companies like Facebook (Meta) and Amazon (AWS) for training their AI. It seems they are likely overusing (Abuse) the crawls for AI purposes.

Blocking these crawls by IP is ineffective because their IPs frequently change.