Serv command consumes all server resources

[xor-gate]

• Gitea version (or commit ref): 1.4.3
• Git version: git version 2.1.4
• Operating system: Linux 3.16.0-4-amd64 Rename import paths: "github.com/gogits/gogs" -> "github.com/go-gitea/gitea" #1 SMP Debian 3.16.51-3 (2017-12-13) x86_64 GNU/Linux
• Database (use [x]):
  • MySQL (mariadb 10.1)
• Can you reproduce the bug at https://try.gitea.io:
  • Not relevant
• Log gist: https://gist.github.com/xor-gate/baaee46f30a04c7f03ac5dd6c64ceeee

Description

We use jenkins to poll for changes in git. It uses openssh for access and gitea spawns a serv command per poll. There are many concurrent connections (jenkins set to 10 concurrent SCM pollers) and the server memory and CPU is blown up. It doesn't happen very often but I'm not sure its a deadlock/racecondition in Gitea. We are going to upgrade to 1.4.3 but according to the changelog it should probably not fix the problem for us.

If you need some more information we hopefully could provide it as it only happens once a day or so.

Screenshots

[xor-gate]

We have upgraded to v1.4.3. If it happens again we try to come up with some more details.

[xor-gate]

Still seems to be an issues multiple times a day with v1.4.3.

[xor-gate]

I think it is because we have only assigned 16GB memory and some swap. Every connection uses virtual ~1500M memory and ~500M residence memory. This is a insane amount of memory.

[lafriks]

Is there anything useful in gitea.log at such times?

[xor-gate]

I have had a look at the logfiles but nothing special is in there. I will keep an eye on it. But what about dumping debug pprof to disk for the serv command to see the actual memory usage hotspots?

[xor-gate]

Hi guys, I have no exact explanation but the problem has not be seen after we updated Jenkins. Still i think it is good to have the ability create pprof dumps per new process of serv command to figure out the memory usage. Because OOM-killer (linux) can come by and reap other processes and make the server unusable. In the cases where SSH is requested with a high-rate (big deployments).

[lafriks]

Currently it is hard to find out cause for this without ability to reproduce such behaviour

[xor-gate]

I totaly understand, is it a possibility I add a feature for pprof memory dumps to disk on exit with special flag? Because currently there is only a http pprof endpoint which can not be used with short lived serve executions under SSH. Then we can profile CPU and Heap and make some better conclusions than this weird "smoking server" problem.

https://github.com/go-gitea/gitea/search?q=pprof&unscoped_q=pprof

[lafriks]

Sure, it could help to have such option to pprof this

[xor-gate]

If no one will work on this I can try to implement dumping pprof to disk in the upcoming weeks?

[lafriks]

Yes, go ahead

[xor-gate]

@lafriks initial working dumping is in pr #4560

[micw]

I have the same issue - from time to time, hundreds of "gitea serv" processes eat all my system memory. How can I help to track it down?