Cost Explorer SDK fails to group costs by RESOURCE_ID

[lez-doit]

Upcoming End-of-Support

• I acknowledge the upcoming end-of-support for AWS SDK for Java v1 was announced, and migration to AWS SDK for Java v2 is recommended.

Describe the bug

When I try to use CostExplorerClient to query costs data with aggregation by ""RESOURCE_ID"" key, it fails to do so and throws an exception telling me that this key is not valid. However, I can see that Cost Explorer UI in AWS console uses this group by dimension and returns data.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

I expected to be able to query AWS costs data with daily aggregation by resource using CostExplorer SDK.

Current Behavior

When I try to use "RESOURCE_ID" as a key in group by, client fails to execute request and throws following error:

Caused by: software.amazon.awssdk.services.costexplorer.model.CostExplorerException: Group Definition dimension is invalid. Valid values are AZ, INSTANCE_TYPE, LINKED_ACCOUNT, OPERATION, PURCHASE_TYPE, SERVICE, USAGE_TYPE, PLATFORM, TENANCY, RECORD_TYPE, LEGAL_ENTITY_NAME, INVOICING_ENTITY, DEPLOYMENT_OPTION, DATABASE_ENGINE, CACHE_ENGINE, INSTANCE_TYPE_FAMILY, REGION, BILLING_ENTITY, RESERVATION_ID, SAVINGS_PLANS_TYPE, SAVINGS_PLAN_ARN, OPERATING_SYSTEM (Service: CostExplorer, Status Code: 400, Request ID: 63f1344d-482e-40fa-a3e7-bfeb0ce6445f) (SDK Attempt Count: 1)
	at software.amazon.awssdk.services.costexplorer.model.CostExplorerException$BuilderImpl.build(CostExplorerException.java:113)
	at software.amazon.awssdk.services.costexplorer.model.CostExplorerException$BuilderImpl.build(CostExplorerException.java:61)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:73)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
	at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74)
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53)
	at software.amazon.awssdk.services.costexplorer.DefaultCostExplorerClient.getCostAndUsage(DefaultCostExplorerClient.java:1626)

Reproduction Steps

I am creating and executing a request in a following way:

        @Cleanup CostExplorerClient costExplorerClient = CostExplorerClient.builder()
                .credentialsProvider(credentials.toStaticCredentialsProvider())
                .region(Region.of(credentials.getRegion()))
                .build();

        GetCostAndUsageRequest request = GetCostAndUsageRequest.builder()
                .timePeriod(DateInterval.builder()
                        .start(timeRange.getFrom().format(DateTimeFormatter.ISO_LOCAL_DATE))
                        .end(timeRange.getTo().format(DateTimeFormatter.ISO_LOCAL_DATE))
                        .build())
                .granularity(Granularity.DAILY)
                .groupBy(GroupDefinition.builder().type(GroupDefinitionType.DIMENSION).key("RESOURCE_ID").build())
                .metrics("UnblendedCost")
                .build();
        
        GetCostAndUsageResponse response = costExplorerClient.getCostAndUsage(request);

Possible Solution

No response

Additional Information/Context

I enabled daily granularity at resource level in my AWS console:

And I am also able to use AWS Cost Manager UI to check daily resource-level data.

I haven't found any mentioning of grouping by resource in SDK, but I can see that when I query this data from UI, it obviously uses "RESOURCE_ID" as a group by key

AWS Java SDK version used

2.31.53

JDK version used

Azul Zulu version 1.8.0_402

Operating System and version

Windows 11

[debora-ito]

Hi @lez-doit I've moved your issue to the aws-sdk-java-v2 repo, since you're using 2.x.

[bhoradc]

Hello @lez-doit,

Thank you for reporting the issue. Have you tried using the GetCostAndUsageWithResources API for this specific use-case?

If you need to analyze costs at a resource level, I believe GetCostAndUsageWithResources would be more appropriate. Please note that it has a strict limitation of only providing data for the past 14 days.

LocalDate endDate = LocalDate.now();
LocalDate startDate = endDate.minusDays(14);

CostExplorerClient costExplorerClient = CostExplorerClient.builder()
		.region(Region.US_EAST_1)
		.build();

GetCostAndUsageWithResourcesRequest request = GetCostAndUsageWithResourcesRequest.builder()
		.timePeriod(DateInterval.builder()
					.start(startDate.format(DateTimeFormatter.ISO_LOCAL_DATE))
					.end(endDate.format(DateTimeFormatter.ISO_LOCAL_DATE))
					.build())
				.granularity(Granularity.DAILY)
				.groupBy(GroupDefinition.builder()
					.type(GroupDefinitionType.DIMENSION)
					.key("RESOURCE_ID")
					.build())
				.metrics("UnblendedCost")
				.build();

GetCostAndUsageWithResourcesResponse response = costExplorerClient.getCostAndUsageWithResources(request);

Please let me know if this helps.

Regards,
Chaitanya

[lez-doit]

Hey @bhoradc , thank you for response! I haven't tried this approach because I was not aware of this class, but I just tested it and it worked out for me. I am gonna stick with this method for now.

[bhoradc]

Hi @lez-doit,

Sounds good, therefore I will close the issue. Thanks for confirmation.

Regards,
Chaitanya

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.