proposal: runtime/metrics: define a recommended set of metrics

[mknyszek]

Introduction

With each Go release the set of metrics exported by the runtime/metrics grows in size. Not all metrics are applicable to all cases, and it can become difficult to identify which metrics are actually useful. This is especially true for projects like OpenTelemetry and Prometheus which want to export some broadly-applicable Go runtime metrics by default, but the full set is overwhelming and not particularly user-friendly.

Another problem with collecting all metrics is cost. The cardinality of the default metric set is closely watched by projects like Prometheus, because downstream users are often paying for the storage costs of these metrics when making use of hosted solutions.

This issue proposes defining a conservative subset of runtime metrics that are broadly applicable, and a simple mechanism for discovering them programmatically.

Proposal

There are two parts to this proposal. The categorization of some metrics as "recommended" by the Go toolchain, and the actual mechanism for that categorization.

To start with, I would like to propose documenting such a set of metrics as "recommended" at the top of the runtime/metrics documentation. Each metric is required to have a full rationale explaining its utility and use-cases. The "recommended" set is intended to hold a lot of weight. We need to make sure the reason why we promote a particular metric is well-documented. The "recommended" set of metrics generally follows the compatibility guarantees of the runtime/metrics package. That being said, a metric is unlikely to be promoted to "recommended" if it's not likely to just exist indefinitely. Still, we reserve the right to remove them.

Next, we'll add a Tags []string field to metric.Description so that these metrics can be found programmatically. We could get by with a simple boolean field, but that's inflexible. In particular, what I'd like to avoid is having dedicated fields for future categorizations such that they end up non-orthogonal and confusing.

The tag indicating the default set will be the string "recommended".

Proposed initial metrics

Below is an initial proposed set of metrics. This list is intended to be a conservative and uncontroversial set of metrics that have clear real-world use-cases.

• /gc/gogc:percent - GOGC.
  • Rationale: This metric describes the GOGC parameter to the runtime, which sets the CPU/memory trade-off of the GC.
• /gc/gomemlimit:bytes - GOMEMLIMIT.
  • Rationale: This metric descibes the GOMEMLIMIT parameter to the runtime, which sets a soft memory limit for the runtime.
• /gc/heap/allocs:bytes - Total bytes allocated.
  • Rationale: This metric may be used to derive an allocation rate in bytes/second, which is useful in understanding GC resource cost impact. In particular, it's useful for diagnosing regressions in production.
• /gc/heap/allocs:objects - Total individual allocations made.
  • Rationale: This metric may be used to derive an allocation rate in objects/second, which is useful in understanding memory allocation resource cost impact. In particular, it's useful for diagnosing regressions in production.
• /gc/heap/goal:bytes - GC heap goal.
  • Rationale: This metric is useful for understanding GC behavior, especially when tuning GOGC and GOMEMLIMIT, and a close approximation for heap memory footprint.
• /memory/classes/heap/released:bytes - Current count of heap bytes that are released back to the OS but which remain mapped.
  • Rationale: This metric is necessary for tuning GOMEMLIMIT. It is also necessary to understand what the runtime believes its own physical memory footprint is, as a subtraction from the total.
• /memory/classes/heap/stacks:bytes - Current count of bytes allocated to goroutine stacks.
  • Rationale: This metric is necessary for understanding application memory footprints, specifically those that users have some control over and may seek to optimize.
• /memory/classes/total:bytes - Total Go runtime memory footprint.
  • Rationale: This metric is necessary for tuning GOMEMLIMIT. It's also useful for identifying "other" memory, and together with /memory/classes/heap/released:bytes, what the runtime believes the physical memory footprint of the application is.
• /sched/gomaxprocs:threads - GOMAXPROCS.
  • Rationale: This metric is a core runtime parameter representing the available parallelism to the application.
• /sched/goroutines:goroutines - Current count of live goroutines (blocked, running, etc.).
  • Rationale: This metric is useful as a proxy for active work units in many circumstances, though it also includes leaks. Supplement with a goroutine profile for more detail, or app-specific concurrency counters (for example, to track the number of active http.Handlers).
• /sched/latencies:seconds - Distribution of time goroutines spend runnable (that is, not blocked), but not running.
  • Rationale: This metric is a measure of scheduling latency that is useful as a fine-grained proxy for overall system load. For example, diffing the distribution over short time windows can provide visibility into the latency impact of uneven load. This metric is battle-tested and has been found to be useful in a variety of scenarios.

This results in 10 uint64 metrics and 1 Float64Histogram metric in the default set, a significant reduction from the 81 metrics currently exported by the package.

Here are a few other metrics that were not included.

• /memory/classes/heap/objects:bytes - Current count of bytes allocated.
  • Rationale: It is already possible to derive this from total allocations and frees.
• /memory/classes/metadata/other:bytes - Runtime metadata, mostly GC metadata.
  • Rationale: We expect to break this category out as specific things that are useful to measure come up. This does not indicate good longevity.
• /gc/heap/frees:bytes - Total bytes freed.
  • Rationale: This metric may be used to compute the total amount of live+unswept heap memory, with /gc/heap/allocs:bytes. Not that useful on its own, and live+unswept heap memory isn't a terribly useful metric since it tends to be noisy and misleading, subject to sweep scheduling nuances. The heap goal is a much more reliable measure of total heap footprint.
• /gc/heap/frees:objects - Total individual allocations freed.
  • Rationale: This metric may be used to compute the total number of live objects, with /gc/heap/allocs:objects. Not that useful on its own, and the number of live objects on its own also isn't that useful. Together with /gc/heap/frees:objects, /gc/heap/allocs:bytes, and /gc/heap/frees:bytes it can be used to calculate average object size, but that's also not very useful on its own. The distribution of object sizes is more useful, but the metric is currently incomplete, as it currently buckets all objects >32 KiB in size together.
• /godebug/non-default/* - Count of instances of a behavior change due to a GODEBUG setting.
  • Rationale: While counting instances of non-default behavior is important, the usage of these particular metrics is intended to be used more on a case-by-case basis. Consider a team upgrading their go.mod version. The default behavior of their programs may change due to the upgrade, but because they're using the new defaults, these metrics won't actually be updated. If something goes wrong due to the new defaults, these metrics aren't that helpful in identifying that it's due to the new default behavior. Instead, these metrics are helpful for eliminating remaining sources of non-default behavior once opted-in.

Alternatives

Only documenting the recommended set

One alternative is to only document the set of recommended metrics. This is fine, but it also runs counter to runtime/metrics' original goal of being able to discover metrics programmatically. Some mechanism here seems necessary to keep the package useful to both humans and computers.

A toolchain-versioned default metrics set

Originally, we had considered an API (for example, metrics.Recommended(...)) that accepted a Go toolchain version and would return the set of default metrics (specifically, a []metrics.Description) for that version. All the metrics within would always be valid to pass to metrics.Read.

You could also imagine this set being controlled via the language version set in the go.mod indirectly via GODEBUG flags. (That is, every time we would change this set, we'd add a valid value to GODEBUG. Specifically something like GODEBUG=runtimemetricsgo121=1.)

Unfortunately, there are already a lot of questions here about stability and versioning. Least of which is the fact that toolchain versions, at least those reported by the runtime/debug package, aren't very structured.

Furthermore, this is a type of categorization that doesn't really compose well. If we ever wanted new categories, we'd need to define a new API, or possibly dummy toolchain strings. It's also a much more complicated change.

[mknyszek]

@dashpole Out of curiosity, would OpenTelemetry use the Tags []string field at all, or is there a desire to tightly control the metrics exported by default? Same question for @bwplotka for Prometheus. The purpose of this field is to stay in line with the spirit of this package, which is to make as much information programmatically available as possible.

[mknyszek]

CC @rhysh @bboreham @felixge @prattmic

[dashpole]

Out of curiosity, would OpenTelemetry use the Tags []string field at all, or is there a desire to tightly control the metrics exported by default?

We would likely use it as part of the tests for the package as a way to verify that we are exposing all of the recommended metrics to users. We would probably not use it for programmatic generation of new metrics.

[MikeMitchellWebDev]

As @rsc said in 2021 (#43555), Expvar is a bit left behind at this point. JSON is a very popular format for developers. Can any decision about runtime.Metrics take Expvar and runtime.Memstats into consideration?

[mknyszek]

@MikeMitchellWebDev It's true that expvar is missing runtime/metrics data, but it's unclear which metrics should be added and how. Please reply on #43555; this proposal is not the right place to discuss changing expvar. See also #61638 which isn't directly related, but maybe should be considered as well in any rethink of expvar.

Lastly, note that runtime.MemStats is generally a subset of runtime/metrics, and the latter should be preferred in general (for a number of reasons, including additional metrics as well as better performance). The only reason it is not officially deprecated is because it provides stronger guarantees than runtime/metrics. I admit I haven't been very good about updating the MemStats documentation to make this clear. I'll try to find some time this week to fix that.

EDIT: To be clear, I don't mean deprecating runtime.MemStats. That requires a proper proposal. Just documentation comparing and contrasting runtime.MemStats with the runtime/metrics package.

[ArthurSens]

Hello hello, I'm Arthur from prometheus/client_golang team 👋

We have a different set of default metrics and I believe we can't just change the default exposed metrics without a major version bump. One approach we could use is to have a configuration option in client_golang "ExposeRecommendedMetrics", but I predict we'll have questions like "why the recommended metrics aren't the default?".

With that said, I like the idea of the Go team providing instructions about what metrics are worth paying the price to collect and store. I also like the idea of those instructions being programmatically available, we just need to evaluate if there's a need for a major version bump in client_golang and if it's worth the effort

[bboreham]

@ArthurSens for some historical perspective see prometheus/client_golang#955 and prometheus/client_golang#1033

My suggestion was that client_golang would offer three main options for what is exported: "exactly what it did in the previous version", "the Go recommended metrics" and "everything that Go exposes". I imagine we can cope with the questions.

[ArthurSens]

@ArthurSens for some historical perspective see prometheus/client_golang#955 and prometheus/client_golang#1033

My suggestion was that client_golang would offer three main options for what is exported: "exactly what it did in the previous version", "the Go recommended metrics" and "everything that Go exposes". I imagine we can cope with the questions.

Thanks for the extra context! Yeah, I agree we can offer the recommended metrics in some way :)

[dashpole]

@mknyszek for "Recommended" histogram metrics (currently just /sched/latencies:seconds), are the bucket boundaries on histograms guaranteed to remain stable (i.e. no buckets removed)?

[arl]

@mknyszek for "Recommended" histogram metrics (currently just /sched/latencies:seconds), are the bucket boundaries on histograms guaranteed to remain stable (i.e. no buckets removed)?

The proposal states that the 'Recommended' set follows the guarantees of the "runtime/metrics" package:

	// For a given metric name, the value of Buckets is guaranteed not to change
	// between calls until program exit.

from: https://cs.opensource.google/go/go/+/refs/tags/go1.22.3:src/runtime/metrics/histogram.go;l=26-27