Add docs for native open telemetry instrumentation

Author: estolfo

docs/images/otel-waterfall-retry.png

@@ -0,0 +1,79 @@
+[[opentelemetry]]
+=== Using OpenTelemetry
+
+You can use https://opentelemetry.io/[OpenTelemetry] to monitor the performance and behavior of your Elasticsearch requests through the Ruby Client.
+The Ruby Client comes with built-in OpenTelemetry instrumentation that emits https://www.elastic.co/guide/en/apm/guide/current/apm-distributed-tracing.html[distributed tracing spans] by default.
+With that, applications https://opentelemetry.io/docs/instrumentation/ruby/manual/[instrumented with OpenTelemetry] or using the https://opentelemetry.io/docs/instrumentation/ruby/automatic/[OpenTelemetry Ruby SDK] are inherently enriched with additional spans that contain insightful information about the execution of the Elasticsearch requests.
+
+The native instrumentation in the Ruby Client follows the https://opentelemetry.io/docs/specs/semconv/database/elasticsearch/[OpenTelemetry Semantic Conventions for Elasticsearch]. In particular, the instrumentation in the client covers the logical Elasticsearch request layer, thus, creates a single span per request the service executes against the Ruby Client. The following image shows a resulting trace in which two different Elasticsearch requests are executed, i.e. `ping` and a search `request`:
+
+[role="screenshot"]
+image::images/otel-waterfall-instrumented-without-http.jpg[alt="Distributed trace with Elasticsearch spans",align="center"]
+
+Usually, OpenTelemetry auto-instrumentation modules come with instrumentation support for HTTP-level communication. In this case, in addition to the logical Elasticsearch client requests, spans will be captured for the physical HTTP requests emitted by the client. The following image shows a trace with both, Elasticsearch spans (in blue) and the corresponding HTTP-level spans (in red):
+
+[role="screenshot"]
+image::images/otel-waterfall-instrumented-with-http.jpg[alt="Distributed trace with Elasticsearch spans",align="center"]
+
+Advanced Ruby Client behavior such as nodes round-robin and request retries are revealed through the combination of logical Elasticsearch spans and the physical HTTP spans. The following example shows an `search` request in a scenario with two Elasticsearch nodes:
+
+[role="screenshot"]
+image::images/otel-waterfall-retry.jpg[alt="Distributed trace with Elasticsearch spans",align="center"]
+
+The first node is unavailable and results in an HTTP error, while the retry to the second node succeeds. Both HTTP requests are subsumed by the logical Elasticsearch request span (in blue).
+
+[discrete]
+==== Setup the OpenTelemetry instrumentation
+When using the https://opentelemetry.io/docs/instrumentation/ruby/manual[OpenTelemetry Ruby SDK manually] or using the https://opentelemetry.io/docs/instrumentation/ruby/automatic/[OpenTelemetry Ruby Auto-Instrumentations], the Ruby Client's OpenTelemetry instrumentation is enabled by default and uses the global OpenTelemetry SDK with the global tracer provider. You can provide a tracer provider via the Ruby Client configuration option `opentelemetry_tracer_provider` when instantiating the client. This is sometimes useful for testing or other specific use cases.
+
+[discrete]
+==== Configuring the OpenTelemetry instrumentation
+
+You can configure the OpenTelemetry instrumentation through Environment Variables.
+The following configuration options are available.
+
+[discrete]
+[[opentelemetry-config-enable]]
+===== Enable / Disable the OpenTelemetry instrumentation
+
+With this configuration option you can enable (default) or disable the built-in OpenTelemetry instrumentation.
+
+**Default:** `true`
+
+|============
+| Environment Variable | `OTEL_RUBY_INSTRUMENTATION_ELASTICSEARCH_ENABLED`
+|============
+
+[discrete]
+===== Capture search request bodies
+
+Per default, the built-in OpenTelemetry instrumentation does not capture request bodies because of data privacy reasons. You can use this option to enable capturing of search queries from the the request bodies of Elasticsearch search requests in case you wish to capture this information, regardless. The options are to capture the raw search query, sanitize the query with a default list of sensitive keys, or not capture it at all.
+
+**Default:** `omit`
+
+|============
+| Environment Variable | `OTEL_INSTRUMENTATION_ELASTICSEARCH_CAPTURE_SEARCH_QUERY`
+|============
+
+[discrete]
+===== Sanitize the elasticsearch search request body
+
+You can configure the list of keys whose values are redacted when the search query is captured. The values should be separated by a comma.
+
+**Default:** `nil`
+
+|============
+| Environment Variable | `OTEL_RUBY_INSTRUMENTATION_ELASTICSEARCH_SEARCH_QUERY_SANITIZE_KEYS`
+|============
+
+Example:
+
+```bash
+OTEL_RUBY_INSTRUMENTATION_ELASTICSEARCH_SEARCH_QUERY_SANITIZE_KEYS='sensitive-key,other-sensitive-key'
+```
+
+[discrete]
+==== Overhead
+The OpenTelemetry instrumentation (as any other monitoring approach) may come with a little overhead on CPU, memory and/or latency. The overhead may only occur when the instrumentation is enabled (default) and an OpenTelemetry SDK is active in the target application. In case that either the instrumentation is disabled or no OpenTelemetry SDK is active with the target application, there is no monitoring overhead expected when using the client.
+
+Even when the instrumentation is enabled and is being actively used (by an OpenTelemetry SDK), in the vast majority of cases the overhead is very small and negligible. In edge cases in which there is a noticable overhead the <<opentelemetry-config-enable,instrumentation can be explicitly disabled>> to eliminate any potential overhead effect of the instrumentation.