java.net.ConnectException: Cannot assign requested address (connect failed)

[mihirshah91]

I am connecting to DDB using sts assume role policy for cross-account access
During our load-test, I am seeing java.net.ConnectException: Cannot assign requested address (connect failed) when executing read query on DDB table. Under normal load, it works fine. Problem happens when load testing the service. Pasted full stack trace in current behaviour section

I am setting required timeouts on DDB client and also have increased max connections to support load test still seeing this error. Following is DDB client and sts client configs

STS and DDB client settings:

import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.awssdk.core.client.config.ClientOverrideConfiguration;
import software.amazon.awssdk.core.retry.RetryPolicy;
import software.amazon.awssdk.http.SdkHttpConfigurationOption;
import software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
import software.amazon.awssdk.services.sts.StsClient;
import software.amazon.awssdk.services.sts.auth.StsAssumeRoleCredentialsProvider;
import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
import software.amazon.awssdk.utils.AttributeMap;



@Bean(name = STS_BEAN)
public StsAssumeRoleCredentialsProvider assumeRole() {

    AssumeRoleRequest assumeRoleRequest = AssumeRoleRequest.builder().roleArn(roleArn).roleSessionName(roleSessionName).build();
    StsClient stsClient = StsClient.builder()
            .region(Region.of(region))
            .credentialsProvider(awsCredentialsProvider)
            .httpClientBuilder(UrlConnectionHttpClient.builder())
            .build();
    return StsAssumeRoleCredentialsProvider.builder().refreshRequest(assumeRoleRequest).stsClient(stsClient).build();
}

@Bean(name = DDB_CLIENT)
public DynamoDbClient dynamoDB(@Qualifier(
STS_BEAN) StsAssumeRoleCredentialsProvider stsCredentials) {

    ClientOverrideConfiguration configuration = ClientOverrideConfiguration.builder()
            .apiCallAttemptTimeout(Duration.ofMillis(1000))
            .apiCallTimeout(Duration.ofMillis(1500))
            .retryPolicy(RetryPolicy.defaultRetryPolicy())
            .build();

    AttributeMap AWS_SDK_HTTP_CONFIG_OPTIONS = AttributeMap.builder()
            .put(SdkHttpConfigurationOption.CONNECTION_TIMEOUT, Duration.ofMillis(500))
            .put(SdkHttpConfigurationOption.READ_TIMEOUT, Duration.ofMillis(1500))
            .put(SdkHttpConfigurationOption.WRITE_TIMEOUT, Duration.ofMillis(1500))
            .put(SdkHttpConfigurationOption.MAX_CONNECTIONS, 450)
            .put(SdkHttpConfigurationOption.CONNECTION_MAX_IDLE_TIMEOUT, Duration.ofSeconds(300))
            .build();

    return DynamoDbClient.builder()
            .credentialsProvider(stsCredentials)
            .region(Region.of(region))
            .httpClient(UrlConnectionHttpClient.builder().buildWithDefaults(AWS_SDK_HTTP_CONFIG_OPTIONS))
            .overrideConfiguration(configuration)
            .build();
}

Expected Behavior

Mentioned exception should not be thrown under high load

Current Behavior

Full stack trace:

Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Cannot assign requested address (connect failed)
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98)
at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43)
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:205)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:66)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:133)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:159)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:112)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:167)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:94)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
at software.amazon.awssdk.services.dynamodb.DefaultDynamoDbClient.getItem(DefaultDynamoDbClient.java:2152)
Caused by: java.net.ConnectException: Cannot assign requested address (connect failed)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:681)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:167)
at software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient$RequestCallable.call(UrlConnectionHttpClient.java:205)
at software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient$RequestCallable.call(UrlConnectionHttpClient.java:193)
at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:64)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:76)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:55)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:39)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77)
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64)
... 59 more

Possible Solution

Any pointers on how to avoid Cannot assign requested address (connect failed)? Does it mean that there is no available tcp port for opening socket? If yes, how can I fix it?

Steps to Reproduce (for bugs)

N/A

Context

It affects using sdk java v2 client for our new feature since it doesn't work as expected under high load

Your Environment

• AWS Java SDK version used: aws-sdk-java v2
• JDK version used: jdk 1.8
• Operating System and version:

[mihirshah91]

Can using UrlConnectionHttpClient instead of Apache http client cause this issue?

[debora-ito]

Can using UrlConnectionHttpClient instead of Apache http client cause this issue?

Perhaps. According to the documentation, UrlConnectionHttpClient builder only takes connection timeout and socket timeout, the other configurations are ignored. Have you tested with ApacheHttpClient?

Are you creating one new DynamoDB client per request or are you sharing the client instance? We recommend to share the instance if possible - for more best practices: https://github.com/aws/aws-sdk-java-v2/blob/master/docs/BestPractices.md

[mihirshah91]

Tested with ApacheHttpClient and it has solved our issue. We don't create DDB client for every request and reuse across all requests.

Regarding urlconnection client ignoring those configuration, I wanted to highlight code of buildWithDefaults method which does seem to take in those AttributeMap configs as they both look similar

Urlconnection client method - https://github.com/aws/aws-sdk-java-v2/blob/master/http-clients/url-connection-client/src/main/java/software/amazon/awssdk/http/urlconnection/UrlConnectionHttpClient.java#L343

ApacheHttp client method - https://github.com/aws/aws-sdk-java-v2/blob/master/http-clients/apache-client/src/main/java/software/amazon/awssdk/http/apache/ApacheHttpClient.java#L583

If we look at error again - Cannot assign requested address (connect failed) - which I think means here is - there were too many concurrent TCP connections open for the kernel and all available source port numbers were used up.

Reading further on internet regarding implementation of java HttpURLConnection client, it looks like HttpURLConnection may not be supporting HTTP persistence connection feature and it tries to open new socket for every request. Or may be aws sdk's UrlConnectionHttpClient class is trying to create new socket for every request which is causing this error.

I am not fully sure but this might be the root-cause here. May be SDK team can further confirm here if they have more idea around UrlConnectionHttpClient implementation. It would be great to confirm the root-cause here and also if there is a way to solve this issue with UrlConnectionHttpClient

[zoewangg]

Hi @mihirshah91, apologies for the delayed response.

Regarding urlconnection client ignoring those configuration, I wanted to highlight code of buildWithDefaults method which does seem to take in those AttributeMap configs as they both look similar

UrlHttpConnectionHttpClient does not support all the configurations available in SdkHttpConfigurationOption. You can find the supported configuration in UrlHttpConnectionHttpClient#builder

Here is an example of how to configure them:

aws-sdk-java-v2/http-clients/url-connection-client/src/main/java/software/amazon/awssdk/http/urlconnection/UrlConnectionHttpClient.java

Lines 240 to 249 in 8728dbb

	* A builder for an instance of {@link SdkHttpClient} that uses JDKs build-in {@link java.net.URLConnection} HTTP
	* implementation. A builder can be created via {@link #builder()}.
	*
	* <pre class="brush: java">
	* SdkHttpClient httpClient = UrlConnectionHttpClient.builder()
	* .socketTimeout(Duration.ofSeconds(10))
	* .connectionTimeout(Duration.ofSeconds(1))
	* .build();
	* </pre>
	*/

Reading further on internet regarding implementation of java HttpURLConnection client, it looks like HttpURLConnection may not be supporting HTTP persistence connection feature and it tries to open new socket for every request. Or may be aws sdk's UrlConnectionHttpClient class is trying to create new socket for every request which is causing this error.

The Java SDK's UrlConnectionHttpClient is just a wrapper around HttpURLConnection and doesn't really do anything else.

Per HttpURLConnection doc https://docs.oracle.com/javase/7/docs/technotes/guides/net/http-keepalive.html,

it should by default reuse connections. This can be configured by keep.alive property (default to be true)

The maximum number of connections per destination to be kept alive is default to 5. You can configure it viahttp.maxConnections based on your use-case.

Could you try tuning those configurations to see if it helps?

[github-actions]

It looks like this issue hasn’t been active in longer than a week. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.

[mihirshah91]

Hi @zoewangg thanks for the info.

How can we configure http.maxConnections in UrlHttpConnectionHttpClient? In my question, I have given that i tried to set it using .put(SdkHttpConfigurationOption.MAX_CONNECTIONS, 450) on AttributeMap object but sdk ignored it