Index page incredibly slow after update to 1.21.8

[SoulSeekkor]

Description

Updated from 1.21.7 to 1.21.8 and now the index page takes almost 10 seconds to load, it's insanely slow now... seems to be the heat map that is holding everything up?

Gitea Version

1.21.8

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

Git Version

2.42.0, Wire Protocol Version 2 Enabled

Operating System

Windows Server 2019

How are you running Gitea?

As a Windows service using the provided binary.

Database

MSSQL

[lunny]

Can you force-reload this page and try again?

[SoulSeekkor]

Force reload (in Chrome) and it didn't have any effect. Also loaded in Firefox (which I don't use for anything except testing so it's never hit the site before) and it had the same performance problem.

[SoulSeekkor]

Quick update, my coworker appears to not have this issue even after hard refreshes. I STILL have the issue however, even after hitting it locally on the server itself with Edge. I have a feeling this is specific to administrative users.

[lunny]

Can you provide some backend logs and web browser console -> network screenshots?

[SoulSeekkor]

Sure, here is the contents of the log after clearing it and refreshing the main page a few times:

2024/03/13 11:51:10 ...es/setting/mailer.go:203:loadMailerFrom() [W] connecting over insecure SMTP protocol to non-local address is not recommended
2024/03/13 11:51:10 cmd/web.go:194:serveInstalled() [W] Table system_setting Column version db default is , struct default is 1
2024/03/13 11:51:10 cmd/web.go:194:serveInstalled() [W] Table attachment column uuid db type is NVARCHAR(40), struct type is VARCHAR(40)
2024/03/13 11:51:10 cmd/web.go:194:serveInstalled() [W] Table upload column uuid db type is NVARCHAR(40), struct type is VARCHAR(40)
2024/03/13 11:51:10 cmd/web.go:194:serveInstalled() [W] Table project column card_type db type is INT, struct type is BIGINT
2024/03/13 11:51:10 cmd/web.go:194:serveInstalled() [W] Table project Column card_type db default is 0, struct default is
2024/03/13 11:51:10 cmd/web.go:194:serveInstalled() [W] Table project Column card_type db nullable is false, struct nullable is true
2024/03/13 11:51:10 cmd/web.go:194:serveInstalled() [W] Table hook_task has column repo_id but struct has not related field
2024/03/13 11:51:38 ...eb/routing/logger.go:68:func1() [W] router: slow GET / for 10.5.4.131:63770, elapsed 3866.0ms @ web/home.go:32(web.Home)
2024/03/13 11:52:02 ...eb/routing/logger.go:68:func1() [W] router: slow GET / for 10.5.4.131:63770, elapsed 3799.7ms @ web/home.go:32(web.Home)
2024/03/13 11:52:23 ...eb/routing/logger.go:68:func1() [W] router: slow GET / for 10.5.4.131:63770, elapsed 3310.2ms @ web/home.go:32(web.Home)

[SoulSeekkor]

Here are the screenshots.

[rwalravens]

We're experiencing the same, although for some users it already started in 1.21.1
We suspect it has something to do with the action table.
After launching the cron 'Delete all old actions from database', the performance is much better (for our company, the table shrunk from ~2 million to 500k rows).
Of course this bypasses the cause, but it can help while waiting for a fix.

[SoulSeekkor]

I know in one particular older version (not sure if it was 1.21.1 off hand), the main page performance dropped drastically but I believe it was the org/repos listing that took forever at that time.

[lunny]

#29905 & #29932 maybe helpful.

[SoulSeekkor]

Looks promising!

[SoulSeekkor]

Just noting that after updating to 1.21.9 this morning, it didn't make any difference on this page load issue for me.

[yp05327]

If heatmap is related, maybe this issue is related: #21045

[SoulSeekkor]

Possibly but I doubt it, the heatmap (despite a fair number of repos on our instance) wasn't an issue before until 1.21.8, something changed in that (and seems to be only affecting admins) that absolutely made the main page almost unusable at this point. Also noting (probably expected) that the latest 1.21.10 version performance is unchanged.