too many thread for the socket

[betwjp]

I use the code below: the branch is 5.0.1

ArangoDB.Builder build = new ArangoDB.Builder();
        String[] hostWithPorts = host.split(",");
        for (String hostWithPort : hostWithPorts) {
            String[] array = hostWithPort.split(":");
            build = build.host(array[0], Integer.valueOf(array[1]));
        }
 build.user(user).password(pass)..loadBalancingStrategy(LoadBalancingStrategy.ROUND_ROBIN).maxConnections(connections).timeout(timeout).acquireHostList(true).connectionTtl(ttl).build();

the timout i set 120s, where I run it for some slow query, I found there are 1000 more threads is running, even the query task has finished, those threads also exist, but always exist, the jstack info is :

"pool-8538-thread-1" #68934 prio=5 os_prio=0 tid=0x00007fd83cf09800 nid=0xd983 runnable [0x00007fd7b2240000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at com.arangodb.internal.velocystream.internal.VstConnection.readBytesIntoBuffer(VstConnection.java:260)
at com.arangodb.internal.velocystream.internal.VstConnection.readBytes(VstConnection.java:254)
at com.arangodb.internal.velocystream.internal.VstConnection.readChunk(VstConnection.java:231)
at com.arangodb.internal.velocystream.internal.VstConnection$1.call(VstConnection.java:141)
at com.arangodb.internal.velocystream.internal.VstConnection$1.call(VstConnection.java:124)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Locked ownable synchronizers:
- <0x00000000fef8dc60> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"pool-8537-thread-1" #68933 prio=5 os_prio=0 tid=0x00007fd83cf07800 nid=0xd982 runnable [0x00007fd7b2341000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at com.arangodb.internal.velocystream.internal.VstConnection.readBytesIntoBuffer(VstConnection.java:260)
at com.arangodb.internal.velocystream.internal.VstConnection.readBytes(VstConnection.java:254)
at com.arangodb.internal.velocystream.internal.VstConnection.readChunk(VstConnection.java:231)
at com.arangodb.internal.velocystream.internal.VstConnection$1.call(VstConnection.java:141)
at com.arangodb.internal.velocystream.internal.VstConnection$1.call(VstConnection.java:124)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Locked ownable synchronizers:
- <0x00000000fef8e1d0> (a java.util.concurrent.ThreadPoolExecutor$Worker)

I check the source code, the com.arangodb.internal.velocystream.internal.VstConnection::open() , Where I the code before the socket.connect
socket.setSoTimeout(timeout != null ? timeout : ArangoDefaults.DEFAULT_TIMEOUT);

where the query tasks have finished, those thread will disappear later.

[CarthageKing]

I am experiencing this as well. I have an app that eventually stops working with a OOM (unable to create new native thread) error. If I check the number of threads spawned, it would show around 3k! threads. And most of the threads will be waiting on readBytesIntoBuffer() i.e. pretty much the same stacktrace that you have shown.

If I understood the code in VstConnection.open() correctly, I don't think setting socket.setSoTimeout() to make the socket timeout on a read is correct. When VstConnection.open() successfully creates the socket, it spawns a dedicated thread to handle responses from the socket hence it needs to wait (indefinitely) for responses from ArangoDB.

In order to terminate this thread, one would have to close the VstConnection object. Apparently that is somehow not happening.

I made an attempt at fixing this i.e. https://github.com/CarthageKing/arangodb-java-driver/tree/bugfix-ensure-connections-closed and I'm testing it right now. It would be helpful if you could test this as well.

[BCaxelbecker]

@betwjp @CarthageKing
Did you set the arangodb.connections.ttl property?
the java driver uses a callable to process the result and will be terminate the callable if
a) the message completes or
b) the tll is reached

[CarthageKing]

I don't think setting a ttl helps since the thread inside VstConnection never has a chance to evaluate it, because the thread stuck reading from the socket.

And there's no one to make it unstuck because VstConnection.close() is never called, because FallbackHostHandler.java only clears the hosts list and doesn't explicitly close each host.

I observed that this only happens when acquireHostList(true) is used, because that setting uses the FallbackHostHandler

[BCaxelbecker]

@betwjp can you send me a complete stacktrace and the complete list of all arango driver properties?

[BCaxelbecker]

@CarthageKing do you get the stuck threads after changing the hosts or a network error? maybe this kills the socket and the inputstream is still alive but doesnt return anything. See: java.net.Socket.getInputStream()

[CarthageKing]

Sorry my earlier statement about FallbackHostHandler is wrong. I was actually referring to ExtendedHostResolver.

@BCaxelbecker if you look at the code in ExtendedHostResolver.resolve() you will see that it periodically refreshes its host list (every 1 hour). This is regardless if there's a network error or not. Now when it comes time to refresh the host list, this list is only cleared. The host objects themselves are not closed,

These host objects correspond to the HostImpl class, which references a ConnectionPool, which contains a bunch of Connection objects. These Connection objects will ultimately be VstConnection objects when using VST protocol.

So not explicitly closing the host objects (HostImpl) does not close the ConnectionPool, which leaves active Connection (VstConnection) objects lying around waiting on socket.read() for data that will never come.

[BCaxelbecker]

@CarthageKing please check PR 250

[BCaxelbecker]

fixed with version 5.0.5