Semantic tokens features trackList

[soulomoon]

Semantic highlighting would give us a great experience on reading the code. It's been a shame that haskell does not have it for all these times. But now we are going to make it happen for haskell in HLS. (Thank @michaelpj for his pull request that support it in lsp package), first version of semantic tokens have been made in this pull request. (Thank @michaelpj @wz1000 for all the help and reviews. )
But since the amount of works it need , not everything in the LSP semantic tokens specification is supported yet, we would do them step by step.
Here is the track list of features.

• provide configuration to map between the semantic tokens type and lsp standard tokens. add config for semantic-tokens-plugin for mapping from hs token type to LSP default token type #3940
• support module name and split out module name in qualified name #3957
• provide configurability to the priorities of hs semantic token types Semantic tokens make tokens types priority configurable #4042
• Support semantic tokens type: done in Implement semantic tokens plugin to support semantic highlighting(textDocument/semanticTokens/full) #3892
  • class and class method type family name (data family)
  • data constructor name (not distinguishing record and normal data, and GADT)
  • type constructor name (GADT)
  • record field name
  • type synonym
  • pattern synonym
  • functions
  • none-function variables
  • infix operator Semantic Tokens: Distinguish (possibly-partially-applied) infix operators from normal functions/variables #3969
• Support PluginMethodHandler
  • textDocument/semanticTokens/full. done in Implement semantic tokens plugin to support semantic highlighting(textDocument/semanticTokens/full) #3892
  • textDocument/semanticTokens/full/delta #4072
  • workspace/semanticTokens/refresh
• Support modifiers:
  • declaration (as in class declearations, type definition and type family)
  • definition (as in class instance declaration, left hand side value binding, and type family instance)
  • modification (as in rec field update)
  • recursion analyze in let binding Mark recursive references in let bindings #2902
• Support unicode, done in Implement semantic tokens plugin to support semantic highlighting(textDocument/semanticTokens/full) #3892
• Support imported name, done in Implement semantic tokens plugin to support semantic highlighting(textDocument/semanticTokens/full) #3892
• ClientCapabilities, multilineTokenSupport, overlappingTokenSupport
• highlight identifiers in doc, but not the first line of nested comment due to a ghc bug. https://gitlab.haskell.org/ghc/ghc/-/issues/24378
• Bug fixing & minor improvement:
  • Semantic tokens forall function hide under type syn is not recognized #3966
  • Semantic Tokens treats a standalone deriving strategy as type parameter #3983

[Bodigrim]

Thanks for working on this feature! I gave it a quick try in SublimeText on https://github.com/Bodigrim/cabal-add/blob/a83c827e45eeaed722d63f08a4c1bfd183770e97/src/Distribution/Client/Add.hs#L189:

Is it something off with my configuration? Why String is in blue, but ByteString is white?

[michaelpj]

The current implementation highlights type synonyms differently to datatypes. So String and FilePath are different because they are type synonyms. Maybe this is confusing! (The fact that you asked suggests that it is...)

It's a bit funny that you get the same colour for parseCabalFile and MonadError also. Generally it seems a bit hit-and-miss whether any given colour scheme actually distinguishes everything that's in the spec...

[Bodigrim]

Maybe this is confusing! (The fact that you asked suggests that it is...)

Yeah, that's very confusing, especially given how ubiquitous String and FilePath are. I think it would be better not to distinguish.

[soulomoon]

Thank you for testing it out. @Bodigrim
We may need a bit of settings for some color schemes to get the colors right. The mapping from hs semantic token type to the standard semantic token type in lsp is here.

https://github.com/soulomoon/haskell-language-server/blob/master/plugins/hls-semantic-tokens-plugin/src/Ide/Plugin/SemanticTokens/Mappings.hs

say if we want the color scheme to not distinguish between a normal datatype with a type synonyms.
we can always set the Type( type synonym) and Enum(a normal datatype) to have the same color.

=====================================================
To be precise, we are not actually doing the semantic highlighting, we are giving semantic types to the tokens that enables the semantic highlighting feature for haskell in the editors. Maybe it worth a note some where to help user to configure this if the color scheme does not look good. Expecially with the fact that there is a layer of mappings between hs semantic to lsp semantic. And the color scheme is various.

=====================================================

It's a bit funny that you get the same colour for parseCabalFile and MonadError also. Generally it seems a bit hit-and-miss whether any given colour scheme actually distinguishes everything that's in the spec...

Also I was thinking🤔， maybe we can also make the toLspTokenType configurable to provide more flexibility on this. In this way, we can provide as detailed as much for the semantic token type, but also enable the user to configure their flavor if the default mapping does not suit their need.

[soulomoon]

Maybe this is confusing! (The fact that you asked suggests that it is...)

Yeah, that's very confusing, especially given how ubiquitous String and FilePath are. I think it would be better not to distinguish

I do not have a SublimeText subscription to test things out, but most color scheme in vscode do not distinguish them by default.
Here is the the default mappings look like in vscode with the scheme one dark pro.

But type synonym might be constraint too. This is one of the reason why we don't classify type synonym to have the same token type as normal data type.

[michaelpj]

(Possibly we should split this off into another issue)

Worth pointing out that type synonyms are highlighted as "type" and in this case the difference is because datatypes get a special highlighting.

maybe we can also make the toLspTokenType configurable to provide more flexibility on this

Generally we try pretty hard to avoid configuration. It's usually much better if we can find something that's universally acceptable.

I guess I could maybe be convinced to have settings for each thing where we have a choice of a specific or generic token type, e.g. "special highlighting for type synonyms: y/n", and if it's off then we use the generic one (e.g. "just a type"). We could let people pick exactly the LSP semantic token type for each thing but that's probably overkill.

We do already have a configuration switch here: it's whether or not you have semantic tokens on or off. If you turn semantic tokens on, you should probably expect it to make some fine distinctions that aren't obvious to you at first sight. So a different question is "how do we make this discoverable?". Perhaps in this instance having the options would serve a double function: it tells you what kinds of things we distinguish, and then you can fiddle with it to see what it does.

Having written all that I guess I've convinced myself that we should have options. Especially since a lot of the things semantic tokens lets you do are quite opinionated and specific, which some people will inevitably dislike (e.g. I bet people will complain about the different highlighting of things of function type).

[michaelpj]

I split off another issue for discussing the configuration issue: #3937

[michaelpj]

@soulomoon a couple of minor things that we still need to do:

• Add you to CODEOWNERS for the plugin (I've invited you to the repository)
• Update the "Features" section of the documentation: https://github.com/haskell/haskell-language-server/blob/master/docs/features.md
  • New section for semantic tokens
  • New entry in the top table linking to it
  • Remove it from "unimplemented features"
• Update the "plugin support" section of the docs: https://github.com/haskell/haskell-language-server/blob/master/docs/features.md
  • I think it can be Tier 2
• Add it to the test job: https://github.com/haskell/haskell-language-server/blob/master/.github/workflows/test.yml

[soulomoon]

@michaelpj thanks for sort it out. do it in #3938