health check with DefaultReactiveElasticsearchClient

[iashaposhnikov]

When one node is turned off from the elastic search cluster (to which there was a successful request from the application before), the application switches to the unhealthy status.
This happens because the ReactiveHealthContributor uses the DefaultReactiveElasticsearchClient, and it does not correctly process the data from the hostProvider.
The problem is that ConnectException is wrapped in another exception.
https://github.com/spring-projects/spring-data-elasticsearch/blob/master/src/main/java/org/springframework/data/elasticsearch/client/reactive/DefaultReactiveElasticsearchClient.java#L544

I see the following message in the logs:

139926 2021-04-04 14:26:40,334 WARN  [ reactor-http-epoll-3 ] o.s.b.a.e.ElasticsearchReactiveHealthIndicator          | Elasticsearch health check failed 
org.springframework.web.reactive.function.client.WebClientRequestException: finishConnect(..) failed: Connection refused: <hostname>/<ip>:9200; nested exception is io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: <hostname>/<ip>:9200
	at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:141)
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
	|_ checkpoint ⇢ Request to GET http://<hostname>:9200/_cluster/health/ [DefaultWebClient]

[sothawo]

It seems not to be enough to check if the caught throwable has the ConnectException in it's causes. The process of then selecting an alternate host from the possibly available ones seems to be broken. I will have to investigate into this

[iashaposhnikov]

Thanks for the quick fix, but there is a potential infinite loop problem in the code.

example of a problem:

please see the description of this method:
https://commons.apache.org/proper/commons-lang/javadocs/api-3.1/org/apache/commons/lang3/exception/ExceptionUtils.html#getRootCause(java.lang.Throwable)

[sothawo]

technically it might be possible to have a chain where a.getCause() -> b and b.getCause() -> a.

But in the DefaultReactiveElasticsearchClient I could not see this. When a host gets down, you get this cause-chain:

Checking the code we get null at the end of the chain:

IntelliJ displays the end of the cause chain repeating the last element, you can see this in this simple example:

The reason for this is, that internally the cause is initialized to this to signal that it was not initialized.

    /**
     * The throwable that caused this throwable to get thrown, or null if this
     * throwable was not caused by another throwable, or if the causative
     * throwable is unknown.  If this field is equal to this throwable itself,
     * it indicates that the cause of this throwable has not yet been
     * initialized.
     *
     * @serial
     * @since 1.4
     */
    private Throwable cause = this;

The getCause() method takes that into account, but IntelliJ reads the field by inspection and so display that endless chain. The infinte loop may only occur if someone creates an Exception a with a different exception as cause, then walks up the cause chain until no cause is available, and then sets a with initCause(a) into that exception. I don't think we need to handle this edge case here.

[iashaposhnikov]

thanks for the explanation!