`LeaderElectionManager` invokes `System.exit()` directly

[adriansuarez]

Bug Report

The LeaderElectionManager has a hardcoded invocation of System.exit(1) on stopLeading(), which is also a private method that cannot be overridden. This prevents me from managing the lifecycle of my own application, by installing a shutdown hook that invokes Operator.stop(), because a deadlock occurs when System.exit() is invoked by the LeaderElectionManager.

Even if I wasn't installing a shutdown hook, this would be undesirable behavior, because it prevents me from enabling leader election in unit tests.

What did you do?

I enabled leader election in my operator, where I also have a shutdown hook installed that invokes Operator.stop().

What did you expect to see?

I expected the Operator to shutdown cleanly.

What did you see instead? Under which circumstances?

The Operator hangs on shutdown.

Environment

Docker Desktop / Kubernetes 1.29.2

Java Operator SDK 4.9.2

$ java -version
openjdk version "17.0.10" 2024-01-16
IBM Semeru Runtime Open Edition 17.0.10.0 (build 17.0.10+7)
Eclipse OpenJ9 VM 17.0.10.0 (build openj9-0.43.0, JRE 17 Mac OS X amd64-64-Bit Compressed References 20240116_636 (JIT enabled, AOT enabled)
OpenJ9   - 2c3d78b48
OMR      - ea8124dbc
JCL      - 2aad089841f based on jdk-17.0.10+7)

$ kubectl version
Client Version: v1.29.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.2

Possible Solution

I worked around this by adding LeaderCallbacks to the leader election configuration that throw an exception before the System.exit() call can happen.

        var operator = new Operator(overrider -> {
            var callbacks = new LeaderCallbacks(() -> {
            }, () -> {
                // throw an exception to prevent hardcoded System.exit() invocation from
                // happening
                throw new RuntimeException(
                        "Preventing LeaderElectionManager.stopLeading() from invoking System.exit()");
            }, s -> {
            });
            var leaderElectionConfig = LeaderElectionConfigurationBuilder
                    .aLeaderElectionConfiguration(LEASE_NAME)
                    .withLeaderCallbacks(callbacks)
                    .build();
            overrider.withLeaderElectionConfiguration(leaderElectionConfig);
        });

This seems like a hack, but it works.

[csviri]

Hi @adriansuarez, It is true that this makes issues with the test. The question is, why would you like to stop it yourself explicitly? Could you elaborate on what you would like to achieve?

We can add a flag to not System.exit() if configured

[csviri]

@adriansuarez could you check if the PR helps your case?

[adriansuarez]

@csviri Just approved it. Thanks for delivering a fix so quickly.

[csviri]

@adriansuarez thx, good to hear that it fixes it, note that this is currently for v5, that hopefully will be released next month. For this we could make a patch also for v 4.9.x if needed.

[adriansuarez]

@csviri It's up to you. I can pick up the fix in v5 and use my workaround in the mean time.

[Michal-Kubricht]

@csviri Would it be possible to make a patch for v 4.9.x? My use-case is not a junit test, but a real running k8s operator application.
There is a deadlock during shutdown when used together with josdk-spring-boot-starter.

I can describe in points why and how the deadlock happens:

• when JOSDK is used with Spring Boot and runs in a kubernetes pod the life-cycle is driven by Spring Boot
• when you send SIGTERM to k8s pod then JVM shutdown is triggered as System.exit
• System.exit then under java monitor-lock executes all registered shutdown hooks (= there are separate threads)
• one of the shutdown hook is a Spring Boot app which together with everything else triggers the JOSDK shutdown
• JOSDK internally does the leader stop which tries to call System.exit which then blocks the thread on the already taken lock

[csviri]

@Michal-Kubricht the v5 should be released soon, isn't an option for you to wait for that and update?

[Michal-Kubricht]

@csviri Yes, it is possible to wait for 5.x.x. There is always a question how costly can it be upgrading to new major version.

I wanted to promote is that calling System.exit is causing issue in real-life use-case, not only a synthetic junit. I also described a deadlock loop for whoever comes across it (and is maybe wondering if he does something wrong).

[adriansuarez]

I wanted to promote is that calling System.exit is causing issue in real-life use-case, not only a synthetic junit. I also described a deadlock loop for whoever comes across it (and is maybe wondering if he does something wrong).

I was also having exactly the same issue. See my workaround at the end of my original post.

[Michal-Kubricht]

I wanted to promote is that calling System.exit is causing issue in real-life use-case, not only a synthetic junit. I also described a deadlock loop for whoever comes across it (and is maybe wondering if he does something wrong).

I was also having exactly the same issue. See my workaround at the end of my original post.

Yes, I did essentially the same workaround (I only throw a KubernetesClientException instead). I just want to get rid of it as soon as possible, because I am running real application with it. There is not only the end of application what needs to be handled, but I need to survive also other situations where the app loses the lease lock (for some reason).