Slow to open a repo if there’s many branches in it

[skyline75489]

• Gitea version (or commit ref): 1.13.0
• Git version: git 2.29.2.windows
• Operating system: Windows 10 1809
• Database (use [x]):
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Didn’t really try
  • Yes (provide example URL)
  • No
• Log gist:

Description

The title may be a bit misleading.

I was building a local mirror of PyTorch manually using git push —mirror. And I found that opening the repo is exceptionally slow (~11 seconds). I also built a clone of gitea itself and the time to open the repo is reasonable (~2 seconds).

I think the reason is that PyTorch has over 4000 branches in it.

Screenshots

[skyline75489]

So I did a little debugging with Gitea code and found that the reason is the number of branches.

[skyline75489]

The root cause of this seem to be git show-ref used by gitea becomes very slow for a repo like PyTorch which have 4000+ branches

[zeripath]

Could you use pprof to confirm where the delay is?

[skyline75489]

I’m new to golang. I’ll try pprof later. Based on my debugging using logging, I found most of the time was spent in GetTags & GetBranches in repoAssginment. This is why I filed the show-ref PR.

By the way because repoAssignment is called every time we need to get the repo context, the entire PyTorch repo feels unresponsive. Everything is slow, be it opening issues, opening commits.

[zeripath]

Yes. Before I broke my hand this was precisely the kind of thing I was working on.

Ok. pprof can be enabled by setting ENABLE_PPROF=true in [server] in app.ini

Once you have that running on your server you can run:

go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30

Ok. pprof can be enabled by setting ENABLE_PPROF=true in [server] in app.ini

And get a SVG on your browser with: web. top would give some other data.

[zeripath]

I've just reread the issue opening comment - please could you try again on current master. You will likely find it much faster

[skyline75489]

I was using the latest master on my mac. And it is still not good. I also delibrately tried building it with 'gogit' flag. But changing 'show-ref' to 'branch/tag' without gogit gives me the best performance. I'm having a hard time setting up a golang dev environment on my Windows machine. But i'll keep trying.

[zeripath]

Interestingly I am not able to duplicate the slow down on linux - ah found it

[skyline75489]

On Linux there's buff/cache, which will aggresively cache everything used on the file system. So i think on Linux it should be better. But still, 'show-ref' feels like an unnecessary slow path for me. All the hashes retrieved end up being just ignored. There has to be a better way, right? Even it's not 'branch/tag'.

[zeripath]

is it this path that is the slow down http://localhost/gitea/administrator/pytorch/branches/ ?

[skyline75489]

The one with 4000+ branches on the same page? TBH I never once successfully opened the page until I added pagination myself.

The repo homepage feels considerably slow to me, which is the first thing I noticed.