Benchmark server backlog

[nicolasstucki]

          performance test scheduled: 239 job(s) in queue, 1 running.

Originally posted by @dotty-bot in #18219 (comment)

This backlog does not seem to be normal.

[mbovel]

The big queue is expected because I re-started the bot that didn't run for 3 months.

The benchmarks on merged PRs (work-merge jobs) are working fine. However, it seems there are some issues when running benchmarks manually using test performance please (pull jobs). I'll leave this issue open until this is fixed.

[nicolasstucki]

It seems we might have a performace regression #18219 (comment)

[mbovel]

I answered there. Note that test performance please worked there. It's still not clear to me why it wasn't on #17627.

[nicolasstucki]

Do we have a whitelist of users that can trigger the bot?

[mbovel]

Do we have a whitelist of users that can trigger the bot?

Yes: https://github.com/lampepfl/bench/blob/a36d0d170a9904859dfbcecfeff3f335dda9b936/bin/process#L22C172-L22C179. That doesn't seem to be the problem here: Linyxus is in the list and the job started successfully but failed later.

[mbovel]

I think everything is working as expected now. It will take one or two weeks to benchmark commits that haven't been benchmarked yet.

[odersky]

Can we get a watchdog that alerts us when the benchmarks fail again? This seems to be a repeating event.

[mbovel]

Well, it was supposed to be part of the job of the issue supervisor. But yeah, maybe we could automate that.

[odersky]

That's the problem with rotating weekly. You can only do the bare minimum of tasks. For everything else you need someone who is responsible for longer (at least a month, probably more).