MissingRequiredPropertyException in a response ForcemergeRequest

[FrankKrout]

Java API client version

8.17.1

Java version

17

Elasticsearch Version

8.15.2

Problem description

Java client code:
` ForcemergeRequest.Builder reqBldr = new ForcemergeRequest.Builder()
.onlyExpungeDeletes(true)
.waitForCompletion(false)
.index(index);

	try {
		ForcemergeRequest request = reqBldr.build();
		System.out.println("ES8-ForcemergeRequest: " + request.toString());
		//ForcemergeResponse response = 
		indicesClient.forcemerge(request);
		//System.out.println("ES8-ForcemergeResponse: " + response.toString());

    } catch (Exception e) {
    	System.out.println("******* exception during ForcemergeRequest: " + e);
    	e.printStackTrace();
    }
}`

Logging:

ES8-ForcemergeRequest: ForcemergeRequest: POST /reviews-v4/_forcemerge?only_expunge_deletes=true&wait_for_completion=false
******* exception during ForcemergeRequest: co.elastic.clients.transport.TransportException: node: https://gpi-ec-dev.es.vpce.us-east-1.aws.elastic-cloud.com/, status: 200, [es/indices.forcemerge] Failed to decode response
co.elastic.clients.transport.TransportException: node: https://gpi-ec-dev.es.vpce.us-east-1.aws.elastic-cloud.com/, status: 200, [es/indices.forcemerge] Failed to decode response
at co.elastic.clients.transport.ElasticsearchTransportBase.decodeTransportResponse(ElasticsearchTransportBase.java:404)
at co.elastic.clients.transport.ElasticsearchTransportBase.getApiResponse(ElasticsearchTransportBase.java:363)
at co.elastic.clients.transport.ElasticsearchTransportBase.performRequest(ElasticsearchTransportBase.java:147)
at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.forcemerge(ElasticsearchIndicesClient.java:937)
at com.gartner.peerinsights.lambda.es.service.ElasticServiceES8.expungeDeletedDocs(ElasticServiceES8.java:143)
at com.gartner.peerinsights.lambda.es.service.ElasticServiceES8.deleteStaleData(ElasticServiceES8.java:82)
at com.gartner.peerinsights.lambda.es.job.Main.handleRequest(Main.java:50)
at com.gartner.peerinsights.lambda.es.job.Main.main(Main.java:11)
Caused by: co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'ForcemergeResponse.shards'
at co.elastic.clients.util.ApiTypeHelper.requireNonNull(ApiTypeHelper.java:76)
at co.elastic.clients.elasticsearch._types.ShardsOperationResponseBase.(ShardsOperationResponseBase.java:68)
at co.elastic.clients.elasticsearch.indices.forcemerge.ForceMergeResponseBody.(ForceMergeResponseBody.java:66)
at co.elastic.clients.elasticsearch.indices.ForcemergeResponse.(ForcemergeResponse.java:59)
at co.elastic.clients.elasticsearch.indices.ForcemergeResponse.(ForcemergeResponse.java:55)
at co.elastic.clients.elasticsearch.indices.ForcemergeResponse$Builder.build(ForcemergeResponse.java:90)
at co.elastic.clients.elasticsearch.indices.ForcemergeResponse$Builder.build(ForcemergeResponse.java:73)
at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:80)
at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
at co.elastic.clients.transport.ElasticsearchTransportBase.decodeTransportResponse(ElasticsearchTransportBase.java:399)
... 7 more
Lambda for ES8 stale data executed successfully.

[FrankKrout]

Kibana request works fine:
POST /reviews-v4/_forcemerge?only_expunge_deletes=true&wait_for_completion=false

Response:
{
"task": "hvcDgr6tTtiwNvAc0V88Bg:30763337"
}

[FrankKrout]

One more comment, the workaround in the docs seems to fail as well...
https://www.elastic.co/guide/en/elasticsearch/client/java-api-client/current/missing-required-property.html

` try {
ForcemergeRequest request = reqBldr.build();
System.out.println("ES8-ForcemergeRequest: " + request.toString());

		**ApiTypeHelper.DANGEROUS_disableRequiredPropertiesCheck(true);**
		ForcemergeResponse response = indicesClient.forcemerge(request);
		**ApiTypeHelper.DANGEROUS_disableRequiredPropertiesCheck(false);**

		System.out.println("ES8-ForcemergeResponse: " + response.toString());

    } catch (Exception e) {
    	System.out.println("******* exception during ForcemergeRequest: " + e);
    	e.printStackTrace();
    }`

ES8-ForcemergeRequest: ForcemergeRequest: POST /reviews-v4/_forcemerge?only_expunge_deletes=true&wait_for_completion=false
******* exception during ForcemergeRequest: java.lang.NullPointerException: Cannot invoke "co.elastic.clients.elasticsearch._types.ShardStatistics.serialize(jakarta.json.stream.JsonGenerator, co.elastic.clients.json.JsonpMapper)" because "this.shards" is null
java.lang.NullPointerException: Cannot invoke "co.elastic.clients.elasticsearch._types.ShardStatistics.serialize(jakarta.json.stream.JsonGenerator, co.elastic.clients.json.JsonpMapper)" because "this.shards" is null
at co.elastic.clients.elasticsearch._types.ShardsOperationResponseBase.serializeInternal(ShardsOperationResponseBase.java:91)
at co.elastic.clients.elasticsearch.indices.forcemerge.ForceMergeResponseBody.serializeInternal(ForceMergeResponseBody.java:85)
at co.elastic.clients.elasticsearch._types.ShardsOperationResponseBase.serialize(ShardsOperationResponseBase.java:84)
at co.elastic.clients.json.JsonpUtils.toString(JsonpUtils.java:452)
at co.elastic.clients.json.JsonpUtils.toString(JsonpUtils.java:389)
at co.elastic.clients.elasticsearch._types.ShardsOperationResponseBase.toString(ShardsOperationResponseBase.java:97)
at com.gartner.peerinsights.lambda.es.service.ElasticServiceES8.expungeDeletedDocs(ElasticServiceES8.java:148)
at com.gartner.peerinsights.lambda.es.service.ElasticServiceES8.deleteStaleData(ElasticServiceES8.java:83)
at com.gartner.peerinsights.lambda.es.job.Main.handleRequest(Main.java:50)
at com.gartner.peerinsights.lambda.es.job.Main.main(Main.java:11)
Suppressed: jakarta.json.stream.JsonGenerationException: Generating incomplete JSON
at org.eclipse.parsson.JsonGeneratorImpl.close(JsonGeneratorImpl.java:519)
at co.elastic.clients.json.JsonpUtils.toString(JsonpUtils.java:451)
... 6 more

[l-trotta]

Hello, thank for the report! This should be an issue we already solved, reported here and solved with this PR in the API specification repository. If I'm not mistaken the fix should be available from version of the client 8.15.3 onward, so 8.17.1 should definitely work. Could you check the client version again?

[l-trotta]

Confirming that version 8.17.1 has the shards property nullable (ForcemergeResponse extends ForceMergeResponseBody extends ShardsOperationResponseBase) :

elasticsearch-java/java-client/src/main/java/co/elastic/clients/elasticsearch/_types/ShardsOperationResponseBase.java

Line 62 in e5cbae4

private final ShardStatistics shards;

[FrankKrout]

FYI My bad, you are correct.

VS Code maven was telling me 8.17.1 but it wasn't. I removed the dependency from the POM completely and put it back. It was running 8.15.2.

Apologies,
Thank you
-Frank