Resurrect OpenTelemetry

[JBetz]

In the effort of improving observability, we'd like to 1) see whether the opentelemetry integrations still work, 2) try turning them on by default, and 3) provide some documentation for both developers and users to actually use them. At very least, event logs could be attached to bug reports, and we may also be able to get realtime diagnostic information via Tracy.

We also might try switching from https://hackage.haskell.org/package/opentelemetry to https://hackage.haskell.org/package/hs-opentelemetry-sdk, which is more active.

[JBetz]

Existing documentation:

HLS records opentelemetry eventlog traces via opentelemetry. To generate the traces, build with -eventlog and run with +RTS -l. To visualize the traces, install Tracy and use eventlog-to-tracy to open the generated eventlog.

https://haskell-language-server.readthedocs.io/en/latest/contributing/contributing.html#tracing

[JBetz]

Okay, here's some good news:

on the commandline: warning: [-Wdeprecated-flags]
-eventlog is deprecated: the eventlog is now enabled in all runtime system ways

So GHC is now turning it on for us by default, at least for 9.6.3+.

There is also a flag for the ghcide executable for memory profiling via OpenTelemetry, but AFACT isn't being used. I tried removing it and ghcide still compiled.

haskell-language-server/ghcide/exe/Arguments.hs

Line 37 in e9c2f55

<*> switch (long "ot-memory-profiling" <> help "Record OpenTelemetry info to the eventlog. Needs the -l RTS flag to have an effect")

[JBetz]

hs-opentelemetry-sdk does not yet support metrics (iand675/hs-opentelemetry#51), so we'd only be able to port the tracing for now. Metrics that HLS are recording are:

haskell-language-server/ghcide/src/Development/IDE/Core/Shake.hs

Lines 744 to 749 in e9c2f55

	registerGauge monitoring "ghcide.values_count" readValuesCounter
	registerGauge monitoring "ghcide.dirty_keys_count" readDirtyKeys
	registerGauge monitoring "ghcide.indexing_pending_count" readIndexPending
	registerGauge monitoring "ghcide.exports_map_count" readExportsMap
	registerGauge monitoring "ghcide.database_count" readDatabaseCount
	registerCounter monitoring "ghcide.num_builds" readDatabaseStep

These were also being recorded by ekg which was recently removed. I don't know how many people are using the OpenTelemetry metrics but they definitely seem worth preserving.

[michaelpj]

Yeah, that sounds nice. I didn't know opentelemetry supported metrics, that sounds like a good way to report these things.

[michaelpj]

Oh I see, we'd need to get the metrics API implemented, that's a bunch more work, boo.

[mpickering]

@JBetz Is there an issue with the current opentelemetry library which is motivating the migration?

[JBetz]

@JBetz Is there an issue with the current opentelemetry library which is motivating the migration?

@mpickering Nope, or at least not that I'm aware of. I think the idea was that it probably would have less maintenance costs, and at very least it was worth investigating if it had any features that HLS could benefit from. Also just my own curiosity in how the API compares to the current library. Turns out it's missing features we need, though, so I think the answer to whether it's worth the migration is a clear no.

[JBetz]

I don't have much time to work on this, but I would like to see if there's a more lightweight way to expose the metrics we already have. My original idea was to create a sort of system monitor-esque widget. Any sort of GUI would almost certainly be beyond LSP, though. Maybe there's a tooltip we could add them to?

[michaelpj]

I think the new opentelemetry lib seems to be more actively maintained, that was my reasoning.

@JBetz we used to use the ekg library to expose metrics, but it's not really maintained and was a pain when upgrading. It was also disabled by default and not really surfaced to users, so it's not clear how useful it was. I don't know how to do better!