Skip to content

fix: recompute head was taking more than 20s in mainnet #1380

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Feb 6, 2025

Conversation

rodrigo-o
Copy link
Collaborator

@rodrigo-o rodrigo-o commented Feb 5, 2025

Motivation

Running the node for more than an epoch and just after block processing, 20+ seconds were taken before finishing the fork choice step, which made it impossible to execute sequentially inside the slot range (12s)

Description

After extensive debugging, the issue was found in the recompute head calculation, specially in the repeatedly call of get_weight where it wasn't really needed (outside of forks). The solution was taking into account this cases and stop calculating get_weight when encountered.

Resolves #1381

@rodrigo-o
Copy link
Collaborator Author

Is possible that #1321 stop happening as usually as today, get_weight was called every time before, now is just used on a fork choice as it should.

Comment on lines -29 to -31
|> then(fn
[] -> {:halt, head}
c -> {:cont, Enum.max_by(c, &get_weight(store, &1, justified_state))}
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we discussed with @ElFantasma this could have been just 1 line change like:

  |> then(fn
    [] -> {:halt, head}
    [single_child] -> {:cont, single_child}
    c -> {:cont, Enum.max_by(c, &get_weight(store, &1, justified_state))}

most of the changes are just for readability

@rodrigo-o rodrigo-o marked this pull request as ready for review February 5, 2025 21:23
@rodrigo-o rodrigo-o requested a review from a team as a code owner February 5, 2025 21:23
Copy link
Contributor

@ElFantasma ElFantasma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks good to me.
I also think that your version has improved readability but I'd like some of the code authors' to take a look on the style change, to ensure they agree on that. Maybe @mpaulucci or @Arkenan ?

@rodrigo-o rodrigo-o changed the title fix: recompute head was taking more than 25s in mainnet fix: recompute head was taking more than 20s in mainnet Feb 5, 2025
@rodrigo-o
Copy link
Collaborator Author

rodrigo-o commented Feb 5, 2025

Recompute head has been reduced from 20~30s to less than 1s

Before:

INFO 19:55:02.101 [Fork choice] Adding new block slot=10997567 root=0x441..ea20
INFO 19:55:16.618 [Fork choice] Block processed. Recomputing head. 
INFO 19:55:45.444 [Fork choice] Added new block slot=10997567 root=0x441..ea20
INFO 19:55:45.448 [Fork choice] Recomputed head slot=10997567 root=0x441..ea20
INFO 19:55:45.448 [Fork choice] Block processing time: 14.519s | Recompute head time: 28.49s 

INFO 19:55:45.452 [Fork choice] Adding new block slot=10997568 root=0x562..2598
INFO 19:57:56.985 [Fork choice] Block processed. Recomputing head. 
INFO 19:58:28.116 [Fork choice] Added new block slot=10997568 root=0x562..2598
INFO 19:58:28.118 [Fork choice] Recomputed head slot=10997568 root=0x562..2598
INFO 19:58:28.118 [Fork choice] Block processing time: 15.536s | Recompute head time: 30.785s 

After:

INFO 19:30:26.903 [Fork choice] Adding new block slot=10997406 root=0x172..b1a8
INFO 19:30:42.722 [Fork choice] Block processed. Recomputing head. 
INFO 19:30:43.274 [Fork choice] Added new block slot=10997406 root=0x172..b1a8
INFO 19:30:43.274 [Fork choice] Recomputed head slot=10997406 root=0x172..b1a8
INFO 19:30:43.274 [Fork choice] Block processing time: 15.819s | Recompute head time: 0.176s 

INFO 19:30:43.276 [Fork choice] Adding new block slot=10997407 root=0xe95..672d
INFO 19:30:59.552 [Fork choice] Block processed. Recomputing head. 
INFO 19:31:00.352 [Fork choice] Added new block slot=10997407 root=0xe95..672d
INFO 19:31:00.353 [Fork choice] Recomputed head slot=10997407 root=0xe95..672d
INFO 19:31:00.353 [Fork choice] Block processing time: 16.276s | Recompute head time: 0.288s

Copy link
Collaborator

@mpaulucci mpaulucci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@rodrigo-o rodrigo-o merged commit 07eaaa3 into main Feb 6, 2025
16 checks passed
@rodrigo-o rodrigo-o deleted the fix-recompute-head-performance-issue branch February 6, 2025 19:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging this pull request may close these issues.

Recompute head takes 20+ seconds in mainnet
3 participants