2.1.6 SPRING BOOT - Elasticsearch Healthcheck failure using Spring Actuator

[alpizano]

Hi, I'm working at my company as a Software Engineering intern (show me mercy) and I was adding and health check feature to our Elasticsearch service. Simply, I utilized the Spring Actuator, like from this guide (bullet point 3.3 most specifically) Spring Actuator Healthcheck Guide

I simply created a new class, "IndexExists.java" and implemented the HealthIndicator() interface. We have an "Application.yml" file in the service/service-api/src/main/resources directory, so I went in there as-well and added the rest uri for both local and dev environments, per the advice of another more experienced Engineer I work with. I added the management details to be enabled at the top of the application.yml file and added the specific rest uri to the spring profile for dev like so:

management:
  endpoint:
    health:
      show-details: always

 rest:
  uris: ["https://my-dev-url-here"]

Of course initially, I tested these changes ### locally and all was well. The health check endpoint, which is http://localhost:8080/globalSearch/actuator/health, returns a local JSON response like such, when using an API tool like POSTMAN:

{
    "status": "UP",
    "details": {
        "indexExists": {
            "status": "UP",
            "details": {
                "index": "exists",
                "value": "assets"
            }
        },
        "diskSpace": {
            "status": "UP",
            "details": {
                "total": 250790436864,
                "free": 195072442368,
                "threshold": 10485760
            }
        },
        "elasticsearchRest": {
            "status": "UP",
            "details": {
                "cluster_name": "elasticsearch",
                "status": "yellow",
                "timed_out": false,
                "number_of_nodes": 1,
                "number_of_data_nodes": 1,
                "active_primary_shards": 7,
                "active_shards": 7,
                "relocating_shards": 0,
                "initializing_shards": 0,
                "unassigned_shards": 5,
                "delayed_unassigned_shards": 0,
                "number_of_pending_tasks": 0,
                "number_of_in_flight_fetch": 0,
                "task_max_waiting_in_queue_millis": 0,
                "active_shards_percent_as_number": 58.333333333333336
            }
        }
    }
}

Now, when pushing my code changes via git (git push of course) to the build pipeline, it indeed built successfully. However, we have a release pipeline which "Deploys to dev" and when I attempted to do this, this deployment produced the (error) logs below on AWS (Amazon Web Services) CloudWatch Logs below. Also, when running a GET request via POSTMAN to the dev url instance with the newly built Health endpoint, the JSON response is incorrect and not what the response I received locally (aka the correct response):

{
    "status": "DOWN",
    "details": {
        "indexExists": {
            "status": "UP",
            "details": {
                "index": "exists",
                "value": "assets"
            }
        },
        "diskSpace": {
            "status": "UP",
            "details": {
                "total": 16776032256,
                "free": 9712218112,
                "threshold": 10485760
            }
        },
        "elasticsearchRest": {
            "status": "DOWN",
            "details": {
               "error": "java.net.ConnectException: Connection refused"
            }
        }
    }
}

If you want to view the log errors in a image that might be easier to read the below, here is the link:
CloudWatch Logs Healthcheck Elasticsearch Failure

16:15:43 Starting...
16:15:43 Preparing server certificate...
16:15:43 Pulling secrets...
16:15:45 Converting secrets to p12...
16:15:45 Cleaning...
16:15:45 Certificate file creation complete...
16:15:45 Starting java app...
16:15:46 SLF4J: Class path contains multiple SLF4J bindings.
16:15:46 SLF4J: Found binding in [jar:file:/home/app/service-api.jar!/BOOT-INF/lib/logback-classic-1.2.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
16:15:46 SLF4J: Found binding in [jar:file:/home/app/service-api.jar!/BOOT-INF/lib/slf4j-log4j12-1.7.26.jar!/org/slf4j/impl/StaticLoggerBinder.class]
16:15:46 SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
16:15:47 SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
16:15:48         . ____ _ __ _ _
16:15:48         /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
16:15:48         ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
16:15:48         \\/ ___)| |_)| | | | | || (_| | ) ) ) )
16:15:48         ' |____| .__|_| |_|_| |_\__, | / / / /
16:15:48         =========|_|==============|___/=/_/_/_/
16:15:48         :: Spring Boot :: (v2.1.6.RELEASE)

16:15:48 2019-12-16T22:15:48.740|INFO |c.c.d.g.Application|logStarting|50|Starting Application on ip-10-221-118-172.us-east-2.compute.internal with PID 18 (/home/app/service-api.jar started by root in /home/app)
16:15:48 2019-12-16T22:15:48.756|INFO |c.c.d.g.Application|logStartupProfileInfo|650|The following profiles are active: dev
16:15:51 2019-12-16T22:15:51.880|INFO |o.s.b.w.e.t.TomcatWebServer|initialize|90|Tomcat initialized with port(s): 443 (https)
16:15:51 2019-12-16T22:15:51.921|INFO |o.a.c.h.Http11NioProtocol|log|173|Initializing ProtocolHandler ["https-jsse-nio-443"]
16:15:51 2019-12-16T22:15:51.971|INFO |o.a.c.c.StandardService|log|173|Starting service [Tomcat]
16:15:51 2019-12-16T22:15:51.972|INFO |o.a.c.c.StandardEngine|log|173|Starting Servlet engine: [Apache Tomcat/9.0.21]
16:15:52 2019-12-16T22:15:52.110|INFO |o.a.c.c.C.[.[.[/globalSearch]|log|173|Initializing Spring embedded WebApplicationContext
16:15:52 2019-12-16T22:15:52.111|INFO |o.s.w.c.ContextLoader|prepareWebApplicationContext|283|Root WebApplicationContext: initialization completed in 3238 ms
16:15:53 2019-12-16T22:15:53.814|INFO |o.s.s.c.ThreadPoolTaskExecutor|initialize|171|Initializing ExecutorService 'applicationTaskExecutor'
16:15:54 2019-12-16T22:15:54.324|INFO |o.s.b.a.e.w.EndpointLinksResolver|<init>|58|Exposing 2 endpoint(s) beneath base path '/actuator'
16:15:54 2019-12-16T22:15:54.420|INFO |o.a.c.h.Http11NioProtocol|log|173|Starting ProtocolHandler ["https-jsse-nio-443"]
16:15:54 2019-12-16T22:15:54.610|INFO |o.s.b.w.e.t.TomcatWebServer|start|202|Tomcat started on port(s): 443 (https) with context path '/globalSearch'
16:15:54 2019-12-16T22:15:54.621|INFO |c.c.d.g.Application|logStarted|59|Started Application in 7.155 seconds (JVM running for 8.759)
16:16:35 2019-12-16T22:16:35.266|INFO |o.a.c.c.C.[.[.[/globalSearch]|log|173|Initializing Spring DispatcherServlet 'dispatcherServlet'
16:16:35 2019-12-16T22:16:35.269|INFO |o.s.w.s.DispatcherServlet|initServletBean|524|Initializing Servlet 'dispatcherServlet'
16:16:35 2019-12-16T22:16:35.279|INFO |o.s.w.s.DispatcherServlet|initServletBean|546|Completed initialization in 10 ms
16:16:35 2019-12-16T22:16:35.834|WARN |o.s.b.a.e.ElasticsearchRestHealthIndicator|health|87|Elasticsearch health check failedjava.net.ConnectException: Connection refused at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:959) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233) at org.springframework.boot.actuate.elasticsearch.ElasticsearchRestHealthIn
2019-12-16T22:16:35.834|WARN |o.s.b.a.e.ElasticsearchRestHealthIndicator|health|87|Elasticsearch health check failedjava.net.ConnectException: Connection refused 	at
 org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:959) 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233) 	at 
org.springframework.boot.actuate.elasticsearch.ElasticsearchRestHealthIndicator.doHealthCheck(ElasticsearchRestHealthIndicator.java:60) 	at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:82) 	at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:95) 	at
 org.springframework.boot.actuate.health.HealthEndpoint.health(HealthEndpoint.java:50) 	at org.springframework.boot.actuate.health.HealthEndpointWebExtension.health(HealthEndpointWebExtension.java:53) 	at
 java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 	at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 	at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 	at java.base/java.lang.reflect.Method.invoke(Method.java:566) 	at
 org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:282) 	at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:76) 	at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:60) 	at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$ServletWebOperationAdapter.handle(AbstractWebMvcEndpointHandlerMapping.java:278) 	at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$OperationHandler.handle(AbstractWebMvcEndpointHandlerMapping.java:334) 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 	at java.base/java.lang.reflect.Method.invoke(Method.java:566) 	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) 	at 
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) 	at 
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) 	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:892) 	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797) 	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) 	at
 org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1039) 	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942) 	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005) 	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:897) 	at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:634) 	at
 org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882) 	at
 javax.servlet.http.HttpServlet.service(HttpServlet.java:741) 	at
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) 	at
 org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) 	at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:88) 	at
 org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:109) 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) 	at
 org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) 	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:109) 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) 	at
 org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92) 	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:109) 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) 	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) 	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:109) 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) 	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:114) 	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:104) 	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:109) 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) 	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) 	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:109) 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) 	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) 	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) 	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490) 	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) 	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) 	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) 	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) 	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) 	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) 	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:853) 	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1587) 	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) 	at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.net.ConnectException: Connection refused 	at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) 	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:171) 	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:145) 	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348) 	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192) 	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) 	... 1 common frames omitted 

I did post on the AWS forums, StackOverlow, and did a little research and found other people having similar issues where they posted GitHub issue requests. Most people were saying things about the RestClient bean in the application is not picking up the configuration in application.yml. I can post any more information anyone needs if I can get some help. I'm just trying to narrow down what the issue is or work toward debugging. I'm relatively new to Spring, so I don't completely understand when people are talking about spring beans, application context, http etc... etc... but I'm willing to do anything to learn (I'm just a peasant intern :P )

Am I missing more configuration? Like do I need to define these properties (which I never did)?:

endpoints.beans.id=springbeans
endpoints.beans.sensitive=false
endpoints.beans.enabled=true```

If anyone could help me, or point me in the correct direction, I'd greatly appreciate it!

[bclozel]

Thanks for getting in touch, but it feels like this is a question that would be better suited to StackOverflow. As mentioned in the guidelines for contributing, we prefer to use GitHub issues only for bugs and enhancements. Feel free to update this issue with a link to the re-posted question (so that other people can find it) or add some more details if you feel this is a genuine bug.

The connection is refused in that other environment.
Several possibilities:

• depending on how your custom indicator is implemented, maybe the credentials provided in your yml are wrong. Are you using the ES REST client configured by Spring Boot in your custom indicator?
• it might be a limitation of that ES instance. You can check that by using the REST URL you're providing to the application and performing yourself the health check request sent by Spring Boot; we're using the cluster health API /_cluster/health/<index>.

Feel free to reply with a link to your stackoverflow question.
Your question should at least show your custom indicator implementation and sample curl commands showing both ES requests to that ES instance.
Thanks!

[alpizano]

No problem, thank you. Here is the stack overflow URL:
https://stackoverflow.com/questions/59379279/2-1-6-spring-boot-elasticsearch-healthcheck-failure

Regarding your first question, hmm... I believe we are using the ES REST client configured by Spring Boot. I'm a new intern so I didn't write the codebase and I'm new to Spring to be honest. I used an @Autowired annotation to inject the dependency (ESClient) into the IndexExists.java file. This is the code:

package com.cat.digital.globalsearch.component;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthIndicator;
import org.springframework.stereotype.Component;
import com.cat.digital.globalsearch.data.Indices;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;


@Component
public class IndexExists implements HealthIndicator {
    private final ESClient esClient;
    private static final Logger LOGGER = LoggerFactory.getLogger(IndexExists.class);
    Map<String,String> map = new HashMap<>();


    @Autowired
    public IndexExists(ESClient esClient) {
        this.esClient = esClient;
    }

    @Override
    public Health health() {
        try {
            if (!esClient.hasIndex(Indices.INDEX_ASSETS)) {
                return Health.down().withDetail("index", Indices.INDEX_ASSETS + " index does not exist").build();
            }
        } catch (IOException e) {
            LOGGER.error("Error checking if Elasticsearch index {} exists , with exception", Indices.INDEX_ASSETS,  e);
        }
        map.put("index","exists");
        map.put("value", Indices.INDEX_ASSETS);
        return Health.up().withDetails(map).build();
    }
}

The ESClient class looks like this (constructor and method I used, don't want to list the other methods in it and clutter this post):

package com.cat.digital.globalsearch.component;

import com.cat.digital.globalsearch.model.IndexDocument;
import org.apache.http.HttpHost;
import org.elasticsearch.action.ActionListener;
import org.elasticsearch.action.admin.indices.delete.DeleteIndexRequest;
import org.elasticsearch.action.bulk.BulkRequest;
import org.elasticsearch.action.bulk.BulkResponse;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.action.support.master.AcknowledgedResponse;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.client.indices.CreateIndexRequest;
import org.elasticsearch.client.indices.GetIndexRequest;
import org.elasticsearch.client.indices.PutMappingRequest;

import java.io.IOException;
import java.util.List;

import static org.elasticsearch.client.RequestOptions.DEFAULT;
import static org.elasticsearch.common.xcontent.XContentType.JSON;

/**
 * Wrapper around {@link RestHighLevelClient}
 */
public class ESClient {

    private final RestClientBuilder builder;
    private final RestHighLevelClient searchClient;

    public ESClient(HttpHost... hosts) {
        this.builder = RestClient.builder(hosts);
        searchClient = new RestHighLevelClient(RestClient.builder(hosts));
    }

  /**
     * @param index String represents index name in ES
     * @return true if index exists, false if not
     */
    public boolean hasIndex(String index) throws IOException {
        final GetIndexRequest request = new GetIndexRequest(index);
        try (RestHighLevelClient client = new RestHighLevelClient(builder)) {
            return client.indices().exists(request, DEFAULT);
        }
    }

I didn't quite understand what you were suggesting in bullet point #2. I tested the Healthcheck monitor locally (i.e. after I added the "IndexExists.java" class and added the rest: uri: ['localhost:9200'] ) and it worked fine.

I work at a company and this is a user story I was working on as an Intern, so we use Azure DevOps pipelines to build the image and "deploy to dev". Thus, this is where the first issue was encountered. The "deploy to dev" stage was infinitely looping or hanging for about 30minutes... And another software engineer at the company looked at the AWS logs with me (CloudWatch) and we saw the "connection refused" messages to Elasticsearch. And we could see the Load Balancer attempting to create more containers because it thought the health check failed.

Thanks for the help thus far.

[alpizano]

OK, looks like I had a misconfigured properties in Application.yml that wasn't allowing connection for ElasticsearchRest to be Autoconfigured. You can refer to this in the stack overflow link https://stackoverflow.com/questions/59379279/2-1-6-spring-boot-elasticsearch-healthcheck-failurefailure above and the code is in one of the comments Brian wrote, also thanks again to @bclozel for his help