Extremely slow performance with larger repositories

[vdrandom]

Description

My department has a repository about 500M in size (bare) and with around 40000 commits in flat history (we don't use branches in that specific repository). It contains about 87000 files. Viewing is generally okay, but an attempt to create a pull request results in 2 minute wait between the button press and new pull request being loaded. In the background I see git rev-parse being run a lot.

I tried to reproduce the issue with the Linux Kernel repository (and I get it, THAT one is massive, with more than 4G of data and millions of commits). It took me a few minutes to wait for the repository page to even load, let alone create any pull requests.

I'm not sure it can be fixed, but maybe it is possible to load the pull request page after the press of a button and let the pull request to be created in background at least? It's especially annoying since another press of a button will create another identical pull request and will take as long as the first one.

Gitea Version

1.16.5

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

No response

How are you running Gitea?

From gitea.io

Database

SQLite

[lunny]

For big repository, viewing repository maybe slow at first time, but it will be faster second time if cache enabled.

For big pull request, how many files changed in that PR?

[vdrandom]

We often commit less than 10 files, the issue persists regardless of the amount of files involved.

[zeripath]

What Operating System are you using?

If you're using Windows try the gogit variant.

What version of git are you using?

Upgrade to the latest git. If you're using old versions of git they're slow.

How did you add this repository?

Was it a mirror or a push? - this matters...

Have made sure that the commit-graphs are written?

Go into gitea's repository on the disk and type git commit-graph write

What are the logs showing?

This report is completely inadequate for helping us to help you.

[wxiaoguang]

FYI, there is a linux repo on gitea.com .

And:

$ time curl https://gitea.com/marktsai0316/linux/commits/branch/master/scripts/clang-tools
curl   0.02s user 0.02s system 0% cpu 54.956 total

(update: above problem has been optimized)

Maybe it's possible to reproduce the problem there (still 504)

https://gitea.com/wxiaoguang/linux/compare/v5.9-rc8...wxiaoguang/linux:wxiaoguang-patch-2

[zeripath]

I have a feeling that that repository does not have a commit-graph. I don't have access to the repository on the filesystem to check though.

[zeripath]

I think the two urls actually represent different problems.

Let's look at the first one:

• https://gitea.com/marktsai0316/linux/commits/branch/master/scripts/clang-tools

This is essentially:

git rev-list --count $REVISION -- $FILE_PATH

followed-by:

git log $REVISION --follow  --pretty=format:%H -- $FILE_PATH

Now both of these would be sped up by commit-graphs. The first call could actually be done entirely within the commit-graph too.

But the interesting thing is that the second one of these is so much slower than the first.

In fact they can actually produce different results due to the --follow on the second call (which appears to be the cause of most of the slow downs.)

Now if it were not for --follow we could actually use git rev-list for both of these calls and the skip and max-count will be free (in contrast to the current system where the skip doesn't work.)

Looking at the history for this line I don't think there was reasoning behind adding the follow except that I would guess that it was nice to add.

So... a simple speed improvement here is to drop the follow and switch to rev-list for these calls.

An additional speed improvement is to add a context timeout like in the git log --name-only

[vdrandom]

I have left the company where this installation was deployed, so I can't provide more context at this point. I've let my former colleagues know about this issue, hopefully the will join the discussion and provide more info or test the suggested solutions.

I'm not really sure why I haven't provided more info on the matter in terms of installation in question, so here it is.

OS Version

Centos 7 (reproduced on ArchLinux)

Git Version

something old, along the lines of 1.8 (reproduced on 2.35 iirc)

Push or mirror?

The repository was initialized with git --bare push --mirror (same when I tried to reproduce the issue on my laptop). (I might get that command wrong, but I was mirroring a bare repo to the remote installation of gitea.)

This is all the info I can provide since I no longer have any access to the servers in question.

[zeripath]

OK @vdrandom upgrade git to the latest version and go in to gitea's copy of the repo on the disk and run git commit-graph write. (If you're on dev/1.18 you can just run gitea doctor --run check-commit-graphs --fix.)

git 1.8 is really old and will always be slow on big repos.

Big repos also need commit-graphs written. They should be being written now but old repos will not have them written.

It would be helpful to know exactly which urls are slow after you've done that. (But you must ensure you're running a new version of git.)

If the slow urls are related to diffs you should adjust:

[git]
MAX_GIT_DIFF_LINES = 100 ; 1000: Max number of lines allowed of a single file in diff view.
MAX_GIT_DIFF_LINE_CHARACTERS = ; 5000: Max character count per line highlighted in diff view.
MAX_GIT_DIFF_FILES = 20 ; 100: Max number of files shown in diff view.

but tell us exactly which urls are slow.

[zeripath]

actually it looks like we need git commit-graph write --changed-paths

[lunny]

Upgrade gitea.com and now it will take 13 secs in first time loading and less 4 secs in other time when visiting https://gitea.com/marktsai0316/linux/commits/branch/master/scripts/clang-tools .

After manually executed git commit-graph write --changed-paths in the server side, now it will take less 5 second in the first time and less 1 second in other time.

[zeripath]

Ok so it looks like that page could still benefit from a deferable/asynchronous request so I can dust off the code I was working on for that.

I don't think it's worth caching these results as I expect that they should not be being requested often.

I guess I should look at the other URL too

[theoryshaw]

We also experience slow performance.
https://hub.openingdesign.com/