Implement support for result streaming [3.15]

[yyoncho]

The upcoming https://microsoft.github.io/language-server-protocol/specifications/specification-3-15/ version of the protocol will support streaming results(e. g. find-references, completion, find definitions, etc). We will have to find a way to represent that data in lsp-mode.

A possible solution would be to Implement streaming support on top of helm/ivy and enable it only when the user uses them. When they are not present - fallback to standard xref and do not declare streaming support.

cc @dgutov as the maintainer of company and xref(?).

[dgutov]

If you have a good grasp of the protocol, why not propose an extension values allowed by xref? And we can add that in a future version. Maybe also release xref as a "core package".

So far I'm not sure which shape it should take. An iterator? A promise? A chunked iterator?

[yyoncho]

Does anybody know a server implementing that part of the spec? @alanz @robertoaloi @ericdallo @sebastiansturm @kiennq ?

[ericdallo]

There's no support currently for dart LSP server

[robertoaloi]

Hi @yyoncho and sorry for the late reply. Erlang LS does not yet support streaming of results. It only has support for progress reports at this stage.

[yyoncho]

Thank you, I will ask in language server protocol repo. Hopefully, it has been implemented somewhere...

[shoobinger]

@yyoncho I'm planning to implement the partial results feature in the language server for Kotlin I develop. For now I'm working on code completion, but other features would support streaming as well.
The language server is in an early WIP state now, but it's usable, and I guess it can be used to develop and test the feature client-side as soon as it is ready on my part.
Also I will be glad to help with the client side changes

[nbfalcon]

@suive how is your language server different from https://github.com/fwcd/kotlin-language-server? In any case, I can offer you my thoughts on implementing this on lsp-mode's side:

• lsp-request-async gets a new parameter, :partial-result-mode. It can be either t, in which case partial results are passed to the callback trough multiple invocations, 'collect, in which case the callback is invoked multiple times but the results are concatenated each time (useful for ivy, because there is only ivy-update-candidates which takes a list of candidates as the new), or nil (default), in which case partial results aren't supported. With the 'collect mode, only few changes would be necessary.
• lsp-request: makes no sense for sync requests
• lsp-request: hybrid mode (different function): there could be another function like lsp-request which awaits only the first batch of results, and then calls a callback from then on. A use case is waiting for some ivy candidates to not show a window prematurely and to allow C-g, and then to update the candidates window from then on.

Note that these are only suggestions, and there might be a better design I haven't though of.

[yyoncho]

@suive thank you.

For completion, we should coordinate with @dgutov. AFAICT most of the work/challenges will be on company-mode side. It seems to me that we won't be able to fit streaming it in capf without going to emacs-devel and waiting for new emacs release. This means that the practical solution will be to implement it as a traditional company backend and then push it in capf when appropriate. But it all depends on @dgutov's view.

cc @kiennq

@nbfalcon sounds good. We might create integration tests for streaming results.

[dgutov]

I really wish people visited emacs-devel more, not sure what else could restart the process of getting asynchronous completion support in capf. And streaming, too. All ending up in a new-and-rewritten version of capf.

But starting on it in company should be fine too. We can consider it incubating new API features, as well as experiment with how the change should look in the UI.

BTW, is streaming something that gives a measurement improvement for the user experience? Reduced latency till the first popup display, less memory consumption on the backend, all of that?

I just figured if you need to do both fuzzy matching and the appropriate sorting of matches, you need to do the full search for them on the backend anyway, and then you might only be able to save on serialization and bandwidth.