[es/snapshot.create] Missing [X-Elastic-Product] header

[jloisel]

Elasticsearch version (bin/elasticsearch --version): 7.16.2

Plugins installed: []

JVM version (java -version):

openjdk version "11.0.13" 2021-10-19
OpenJDK Runtime Environment (build 11.0.13+8-Ubuntu-0ubuntu1.21.10)
OpenJDK 64-Bit Server VM (build 11.0.13+8-Ubuntu-0ubuntu1.21.10, mixed mode, sharing)

OS version (uname -a if on a Unix-like system):

Linux pop-os 5.15.8-76051508-generic #202112141040163950527821.10~0ede46a SMP Tue Dec 14 22:38:29 U x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

When performing a snapshot using the new Java client, it fails due to missing X-Elastic-Product header sent by Elasticsearch.

Steps to reproduce:

Please include a minimal but complete recreation of the problem,
including (e.g.) index creation, mappings, settings, query etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.

1. Create an FS repository,
2. Create a snapshot using the new java client:

final CreateSnapshotRequest request = new CreateSnapshotRequest
  .Builder()
  .repository(repository)
  .snapshot(name)
  .waitForCompletion(waitForCompletion)
  .build();
client.create(request);

Client being used (maven dep):

<elasticsearch.version>7.16.2</elasticsearch.version>

...

      <dependency>
        <groupId>co.elastic.clients</groupId>
        <artifactId>elasticsearch-java</artifactId>
        <version>${elasticsearch.version}</version>
      </dependency>

3. The code throws an exception.

Provide logs (if relevant):

co.elastic.clients.transport.TransportException: [es/snapshot.create] Missing [X-Elastic-Product] header. Please check that you are connecting to an Elasticsearch instance, and that any networking filters are preserving that header.

	at co.elastic.clients.transport.rest_client.RestClientTransport.checkProductHeader(RestClientTransport.java:340)
	at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:250)
	at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:144)
	at co.elastic.clients.elasticsearch.snapshot.ElasticsearchSnapshotClient.create(ElasticsearchSnapshotClient.java:142)

[elasticmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[dakrone]

I've transferred this to the elasticsearch-java repo, since this mentions using the new Java client rather than the old HLRC.

[swallez]

@jloisel can you check that the X-Elastic-Product HTTP header is effectively received by the machine running your application? Is there an intermediate proxy between your application and the Elasticsearch server that would filter out this header?

[jloisel]

Hi,

Elasticsearch is running on my local machine (docker container). The response doesn't contain the X-Elastic-Product header when debugging the java sdk. There is no proxy / no intermediate. Everything runs on my local computer.

Same issue when disabling "gzip compression" on client side:

Debugging RestClientTransport:

I'm not sure to understand why this header has to be sent by Elasticsearch.

[swallez]

@jloisel can you provide us the command line used to start the Docker container, so that we can try to reproduce the issue?

[jloisel]

Here is the docker-compose.yml I use:

version: '2'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.16.2
    stdin_open: true
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data
    tty: true
    environment:      
      - path.repo=/tmp
      - transport.host=0.0.0.0
      - discovery.type=single-node
      - xpack.security.enabled=false
    ports:
      - 9200:9200/tcp
      - 9300:9300/tcp
    restart: always
  kibana:
    image: docker.elastic.co/kibana/kibana:7.16.2
    stdin_open: true
    tty: true
    ports:
      - 5601:5601/tcp
    restart: always

volumes:
  elasticsearch-data:
    driver: local

Client config using Spring Boot:

import co.elastic.clients.elasticsearch.ElasticsearchClient;
import co.elastic.clients.elasticsearch.cluster.HealthResponse;
import co.elastic.clients.json.jackson.JacksonJsonpMapper;
import co.elastic.clients.transport.ElasticsearchTransport;
import co.elastic.clients.transport.rest_client.RestClientOptions;
import co.elastic.clients.transport.rest_client.RestClientTransport;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.google.common.base.MoreObjects;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.HttpHost;
import org.elasticsearch.client.HttpAsyncResponseConsumerFactory.HeapBufferedResponseConsumerFactory;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestClient;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.io.IOException;
import java.net.InetAddress;
import java.util.concurrent.atomic.AtomicInteger;

import static java.time.Duration.ofSeconds;
import static org.elasticsearch.client.RequestOptions.DEFAULT;
import static org.elasticsearch.client.RestClient.builder;

@Slf4j
@Configuration
class ElasticSearchConfig {
  static final AtomicInteger PORT = new AtomicInteger(9200);

  @Bean(destroyMethod = "close")
  RestClient restClient(
    @Value("${elasticsearch.hostname:localhost}") final String hostname,
    @Value("${elasticsearch.compression:true}") final boolean isCompression) throws IOException {
    final InetAddress[] addresses = MoreObjects.firstNonNull(
      InetAddress.getAllByName(hostname),
      new InetAddress[0]);

    final HttpHost[] hosts = new HttpHost[addresses.length];
    for(int i = 0; i< addresses.length; i++) {
      final InetAddress addr = addresses[i];
      hosts[i] = new HttpHost(addr.getHostAddress(), PORT.get());
    }

    return builder(hosts)
      .setCompressionEnabled(isCompression)
      .build();
  }

  @Bean
  ElasticsearchClient elasticsearchClient(
    final ObjectMapper mapper,
    final RestClient restClient,
    @Value("${elasticsearch.client.buffer-limit:104857600}") final int bufferLimit) {

    final RequestOptions.Builder options = DEFAULT.toBuilder();
    options.setHttpAsyncResponseConsumerFactory(new HeapBufferedResponseConsumerFactory(bufferLimit));

    final ElasticsearchTransport transport = new RestClientTransport(
      restClient,
      new JacksonJsonpMapper(mapper)
    ).withRequestOptions(new RestClientOptions(options.build()));

    return new ElasticsearchClient(transport);
  }
}

[swallez]

Thanks @jloisel. I used your docker compose file and initialization code, and everything works fine on my side.

First of all, curl show that the X-Elastic-Product header is correctly returned:

% curl -v http://localhost:9200
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 9200 (#0)
> GET / HTTP/1.1
> Host: localhost:9200
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json; charset=UTF-8
< content-length: 546
<
{
  "name" : "7c1b69ce0ceb",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "OjvSBJCyTCK7x9jpM8lKtw",
  "version" : {
    "number" : "7.16.2",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "2b937c44140b6559905130a8650c64dbd0879cfb",
    "build_date" : "2021-12-18T19:42:46.604893745Z",
    "build_snapshot" : false,
    "lucene_version" : "8.10.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
* Connection #0 to host localhost left intact
* Closing connection 0

I then used your initialization code with this simple main method to fetch Elasticsearch node information:

    public static void main(String[] args) throws Exception {
        RestClient http = restClient("localhost", true);
        ElasticsearchClient client = elasticsearchClient(new ObjectMapper(), http, 104857600);

        System.out.println(client.info().name());
        System.out.println(client.info().clusterUuid());
        System.out.println(client.info().version().number());
        System.out.println(client.info().version().buildDate());

        client._transport().close();
    }

The request succeeded and printed the same information as returned by curl.

Can you check that the header is returned by running curl -v http://localhost:9200?

[jloisel]

Try to create a snapshot. That's the failing part, other requests are working fine. See logs when trying to create a snapshot using the Java SDK:

co.elastic.clients.transport.TransportException: [es/snapshot.create] Missing [X-Elastic-Product] header. Please check that you are connecting to an Elasticsearch instance, and that any networking filters are preserving that header.

	at co.elastic.clients.transport.rest_client.RestClientTransport.checkProductHeader(RestClientTransport.java:340)
	at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:250)
	at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:144)
	at co.elastic.clients.elasticsearch.snapshot.ElasticsearchSnapshotClient.create(ElasticsearchSnapshotClient.java:142)
	at io.vavr.control.Try.mapTry(Try.java:634)
	at com.octoperf.database.client.elasticsearch.ElasticsearchSnapshots.create(ElasticsearchSnapshots.java:55)
	at com.octoperf.database.client.elasticsearch.ElasticsearchSnapshotSpringTest.shouldTakeSnapshot(ElasticsearchSnapshotSpringTest.java:34)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	....
Caused by: org.elasticsearch.client.ResponseException: method [PUT], host [http://0:0:0:0:0:0:0:1:9200], URI [/_snapshot/test/1641413019710?wait_for_completion=true], status line [HTTP/1.1 200 OK]

Creating a snapshot with curl:

curl -v -XPUT "http://localhost:9200/_snapshot/test/1641413019710?wait_for_completion=true"

And the command result:

*   Trying ::1:9200...
* Connected to localhost (::1) port 9200 (#0)
> PUT /_snapshot/test/1641413019710?wait_for_completion=true HTTP/1.1
> Host: localhost:9200
> User-Agent: curl/7.74.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 2011
< 
{"snapshot":{"snapshot":"1641413019710","uuid":"L80vX4ulS2StSR0wvwukQw","repository":"test","version_id":7160299,"version":"7.16.2","indices"...
.... ]}}

No X-elastic-product header seen in response.

[swallez]

Thanks for the additional details. I was able to reproduce it, and this is an issue in Elasticsearch for this endpoint that may also affect some others. We'll fix Elasticsearch and if necessary add some workarounds in the client library.

[swallez]

@dakrone FYI, this is a bug in the create snapshot API. I opened elastic/elasticsearch#82358 for this.

[dakrone]

Thanks for diagnosing this @swallez, sorry to transfer it here only to transfer it back