FunctionClauseError in Code.Fragment.container_cursor_to_quoted on 1.18

[lukaszsamson]

Elixir and Erlang/OTP versions

1.18.0 and 1.18.1

Operating system

any

Current behavior

I noticed a lot of reported crashes in ElixirLS on 1.18. All the calls to container_cursor_to_quoted on 1.18 are with trailing_fragment option. The error is not present on earlier elixir versions

    ** (FunctionClauseError) no function clause matching in :elixir_tokenizer.prune_tokens_3
        (elixir 1.18.1) <REDACTED: user-file-path>:1778: :elixir_tokenizer.prune_tokens([], [:"}"], [])
        (elixir 1.18.1) <REDACTED: user-file-path>:1765: :elixir_tokenizer.add_cursor_5
        (elixir 1.18.1) <REDACTED: user-file-path>:142: :elixir_tokenizer.tokenize_5
        (elixir 1.18.1) <REDACTED: user-file-path>:1172: Code.Fragment.container_cursor_to_quoted_2

    ** (FunctionClauseError) no function clause matching in :elixir_tokenizer.prune_tokens_3
        (elixir 1.18.1) <REDACTED: user-file-path>:1778: :elixir_tokenizer.prune_tokens([], [:")"], [{:"(", {401, 49, nil}, 12}])
        (elixir 1.18.1) <REDACTED: user-file-path>:1765: :elixir_tokenizer.add_cursor_5
        (elixir 1.18.1) <REDACTED: user-file-path>:142: :elixir_tokenizer.tokenize_5
        (elixir 1.18.1) <REDACTED: user-file-path>:1172: Code.Fragment.container_cursor_to_quoted_2

    ** (FunctionClauseError) no function clause matching in :elixir_tokenizer.prune_tokens_3
        (elixir 1.18.1) <REDACTED: user-file-path>:1778: :elixir_tokenizer.prune_tokens([], [:")"], [])
        (elixir 1.18.1) <REDACTED: user-file-path>:1765: :elixir_tokenizer.add_cursor_5
        (elixir 1.18.1) <REDACTED: user-file-path>:142: :elixir_tokenizer.tokenize_5
        (elixir 1.18.1) <REDACTED: user-file-path>:1172: Code.Fragment.container_cursor_to_quoted_2

    ** (FunctionClauseError) no function clause matching in :elixir_tokenizer.prune_tokens_3
        (elixir 1.18.1) <REDACTED: user-file-path>:1778: :elixir_tokenizer.prune_tokens([], [:")", :"}"], [])
        (elixir 1.18.1) <REDACTED: user-file-path>:1765: :elixir_tokenizer.add_cursor_5
        (elixir 1.18.1) <REDACTED: user-file-path>:142: :elixir_tokenizer.tokenize_5
        (elixir 1.18.1) <REDACTED: user-file-path>:1172: Code.Fragment.container_cursor_to_quoted_2

Expected behavior

No crash

[josevalim]

Unforutnatley I could not reverse engineer the bug but I think it is related to captures when trailing fragment is given. The root cause should have been addressed though.