new Java API client throws org.elasticsearch.client.ResponseException when there is a version conflict

[frank-montyne]

Elasticsearch Version

7.17.3

Installed Plugins

No response

Java Version

bundled

OS Version

macOS Montery

Problem Description

When adding 2 documents with the same id to an index I get a org.elasticsearch.client.ResponseException instead of the new co.elastic.clients.elasticsearch._types.ElasticsearchException. That is very confusing when working with the new Java AOI client. Seems like a bug.

Example code:

private boolean addLock(Lock distributedLock) throws Exception {
		try {
			// A version conflict status exception will be thrown if the document already exists for the same id.
			if (logger.isDebugEnabled()) logger.debug(String.format("Adding distributed lock %s -> %s", getThreadInfo(), distributedLock.toJson()));
			
			elasticSearch.getESClient().index(builder -> 
				builder
					.index(lockIndex)
					.id(distributedLock.id())
					.withJson(distributedLock.toESJsonReader())
					.opType(OpType.Create)
					.waitForActiveShards(asBuilder -> asBuilder.count(1))
					.refresh(Refresh.True));
			
			if (logger.isDebugEnabled()) logger.debug(String.format("Distributed lock added %s -> %s", getThreadInfo(), distributedLock.toJson()));
			return true;
		}
		// Seems like a bug in ElasticSearch to throw an org.elasticsearch.client.ResponseException instead of the new co.elastic.clients.elasticsearch._types.ElasticsearchException!
		// Catching both for the moment being.
		catch (ResponseException e) {
			// Only version conflict exceptions are allowed.
			if (e.getResponse().getStatusLine().getStatusCode() != RestStatus.CONFLICT.getStatus()) throw e;
			
			// ElasticSearch version conflict. Another host/process/thread already performed an action on this lock.
			if (logger.isDebugEnabled()) logger.debug(String.format("Failed to add distributed lock. ES version conflict %s -> %s", getThreadInfo(), distributedLock.toJson()));
			return false;
		}
		catch (ElasticsearchException e) {
			// Only version conflict exceptions are allowed.
			if (e.response().status() != RestStatus.CONFLICT.getStatus()) throw e;
			
			// ElasticSearch version conflict. Another host/process/thread already performed an action on this lock.
			if (logger.isDebugEnabled()) logger.debug(String.format("Failed to add distributed lock. ES version conflict %s -> %s", getThreadInfo(), distributedLock.toJson()));
			return false;
		}
	}

Steps to Reproduce

Add 2 document with the same id

Logs (if relevant)

No response

[kemusag]

I also noticed that. In my opinion, this is exactly the opposite of what the official documentation describes. There it says:

• Requests that were received by the Elasticsearch server but that were rejected ... produce an ElasticsearchException
• Requests that failed to reach the server ... will be a ResponseException (in case of RestClientTransport)

Noticing a version conflict can only be determined once the request has reached Elasticsearch. Hence, this would be an ElasticsearchException.
If I see the current implementation of RestClientTransport correctly (only had a brief look), only responses having a status code of 400,401,403,404 or 405 can even be thrown as an ElasticsearchException. Conflicts (409) are not in that list.

Is there any plan to adapt this soon? I personally think it would be nice, because think it makes more sense. However, this is a breaking change for us because our code does always check for both ResponseExceptions and ElasticsearchExceptions.

[metaxmx]

We've also encountered this problem as well, and would be happy if ES responses with status code 409 Conflict would be handled as ElasticsearchException like all the other logical errors, so it's possible to access the returned error document easily.

This response is used for example in the protocol for "optimistic concurrency", where a request tries to update or insert a document, and the ES server responds with 409 Conflict, if either the document already exists (duplicate id), or the updated document has been changed in the meantime.

See: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#optimistic-concurrency-control-index

If a mismatch is detected, the operation will result in a VersionConflictException and a status code of 409. See Optimistic concurrency control for more details.