Skip to content

Commit 9a4e143

Browse files
author
awstools
committed
feat(client-emr-containers): EMR on EKS adds support for log rotation of Spark container logs with EMR-6.11.0 onwards, to the StartJobRun API.
1 parent d6b0058 commit 9a4e143

15 files changed

+143
-48
lines changed

‎clients/client-emr-containers/README.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,11 +9,11 @@ AWS SDK for JavaScript EMRContainers Client for Node.js, Browser and React Nativ
99
<p>Amazon EMR on EKS provides a deployment option for Amazon EMR that allows
1010
you to run open-source big data frameworks on Amazon Elastic Kubernetes Service (Amazon EKS).
1111
With this deployment option, you can focus on running analytics workloads while Amazon EMR on EKS builds, configures, and manages containers for open-source applications.
12-
For more information about Amazon EMR on EKS concepts and tasks, see <a href="https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/emr-eks.html">What is
12+
For more information about Amazon EMR on EKS concepts and tasks, see <a href="https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/emr-eks.html">What is
1313
shared id="EMR-EKS"/></a>.</p>
1414
<p>
15-
<i>Amazon EMR containers</i> is the API name for Amazon EMR on EKS.
16-
The <code>emr-containers</code> prefix is used in the following scenarios: </p>
15+
<i>Amazon EMR containers</i> is the API name for Amazon EMR on EKS. The <code>emr-containers</code> prefix is used in the following
16+
scenarios: </p>
1717
<ul>
1818
<li>
1919
<p>It is the prefix in the CLI commands for Amazon EMR on EKS. For example,

‎clients/client-emr-containers/src/EMRContainers.ts

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -429,11 +429,11 @@ export interface EMRContainers {
429429
* <p>Amazon EMR on EKS provides a deployment option for Amazon EMR that allows
430430
* you to run open-source big data frameworks on Amazon Elastic Kubernetes Service (Amazon EKS).
431431
* With this deployment option, you can focus on running analytics workloads while Amazon EMR on EKS builds, configures, and manages containers for open-source applications.
432-
* For more information about Amazon EMR on EKS concepts and tasks, see <a href="https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/emr-eks.html">What is
432+
* For more information about Amazon EMR on EKS concepts and tasks, see <a href="https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/emr-eks.html">What is
433433
* shared id="EMR-EKS"/></a>.</p>
434434
* <p>
435-
* <i>Amazon EMR containers</i> is the API name for Amazon EMR on EKS.
436-
* The <code>emr-containers</code> prefix is used in the following scenarios: </p>
435+
* <i>Amazon EMR containers</i> is the API name for Amazon EMR on EKS. The <code>emr-containers</code> prefix is used in the following
436+
* scenarios: </p>
437437
* <ul>
438438
* <li>
439439
* <p>It is the prefix in the CLI commands for Amazon EMR on EKS. For example,

‎clients/client-emr-containers/src/EMRContainersClient.ts

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -331,11 +331,11 @@ export interface EMRContainersClientResolvedConfig extends EMRContainersClientRe
331331
* <p>Amazon EMR on EKS provides a deployment option for Amazon EMR that allows
332332
* you to run open-source big data frameworks on Amazon Elastic Kubernetes Service (Amazon EKS).
333333
* With this deployment option, you can focus on running analytics workloads while Amazon EMR on EKS builds, configures, and manages containers for open-source applications.
334-
* For more information about Amazon EMR on EKS concepts and tasks, see <a href="https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/emr-eks.html">What is
334+
* For more information about Amazon EMR on EKS concepts and tasks, see <a href="https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/emr-eks.html">What is
335335
* shared id="EMR-EKS"/></a>.</p>
336336
* <p>
337-
* <i>Amazon EMR containers</i> is the API name for Amazon EMR on EKS.
338-
* The <code>emr-containers</code> prefix is used in the following scenarios: </p>
337+
* <i>Amazon EMR containers</i> is the API name for Amazon EMR on EKS. The <code>emr-containers</code> prefix is used in the following
338+
* scenarios: </p>
339339
* <ul>
340340
* <li>
341341
* <p>It is the prefix in the CLI commands for Amazon EMR on EKS. For example,

‎clients/client-emr-containers/src/commands/CreateManagedEndpointCommand.ts

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -40,9 +40,8 @@ export interface CreateManagedEndpointCommandOutput extends CreateManagedEndpoin
4040

4141
/**
4242
* @public
43-
* <p>Creates a managed endpoint. A managed endpoint is a gateway that connects Amazon EMR Studio to
44-
* Amazon EMR on EKS so that Amazon EMR Studio can communicate with your virtual
45-
* cluster.</p>
43+
* <p>Creates a managed endpoint. A managed endpoint is a gateway that connects Amazon EMR Studio to Amazon EMR on EKS so that Amazon EMR Studio can
44+
* communicate with your virtual cluster.</p>
4645
* @example
4746
* Use a bare-bones client and the command you need to make an API call.
4847
* ```javascript
@@ -83,6 +82,10 @@ export interface CreateManagedEndpointCommandOutput extends CreateManagedEndpoin
8382
* s3MonitoringConfiguration: { // S3MonitoringConfiguration
8483
* logUri: "STRING_VALUE", // required
8584
* },
85+
* containerLogRotationConfiguration: { // ContainerLogRotationConfiguration
86+
* rotationSize: "STRING_VALUE", // required
87+
* maxFilesToKeep: Number("int"), // required
88+
* },
8689
* },
8790
* },
8891
* clientToken: "STRING_VALUE", // required

‎clients/client-emr-containers/src/commands/DeleteManagedEndpointCommand.ts

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -36,9 +36,8 @@ export interface DeleteManagedEndpointCommandOutput extends DeleteManagedEndpoin
3636

3737
/**
3838
* @public
39-
* <p>Deletes a managed endpoint. A managed endpoint is a gateway that connects Amazon EMR Studio to
40-
* Amazon EMR on EKS so that Amazon EMR Studio can communicate with your virtual
41-
* cluster.</p>
39+
* <p>Deletes a managed endpoint. A managed endpoint is a gateway that connects Amazon EMR Studio to Amazon EMR on EKS so that Amazon EMR Studio can
40+
* communicate with your virtual cluster.</p>
4241
* @example
4342
* Use a bare-bones client and the command you need to make an API call.
4443
* ```javascript

‎clients/client-emr-containers/src/commands/DescribeJobRunCommand.ts

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,10 @@ export interface DescribeJobRunCommandOutput extends DescribeJobRunResponse, __M
9191
* // s3MonitoringConfiguration: { // S3MonitoringConfiguration
9292
* // logUri: "STRING_VALUE", // required
9393
* // },
94+
* // containerLogRotationConfiguration: { // ContainerLogRotationConfiguration
95+
* // rotationSize: "STRING_VALUE", // required
96+
* // maxFilesToKeep: Number("int"), // required
97+
* // },
9498
* // },
9599
* // },
96100
* // jobDriver: { // JobDriver

‎clients/client-emr-containers/src/commands/DescribeManagedEndpointCommand.ts

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,8 +41,7 @@ export interface DescribeManagedEndpointCommandOutput extends DescribeManagedEnd
4141
/**
4242
* @public
4343
* <p>Displays detailed information about a managed endpoint. A managed endpoint is a gateway
44-
* that connects Amazon EMR Studio to Amazon EMR on EKS so that Amazon EMR Studio can communicate with
45-
* your virtual cluster.</p>
44+
* that connects Amazon EMR Studio to Amazon EMR on EKS so that Amazon EMR Studio can communicate with your virtual cluster.</p>
4645
* @example
4746
* Use a bare-bones client and the command you need to make an API call.
4847
* ```javascript
@@ -97,6 +96,10 @@ export interface DescribeManagedEndpointCommandOutput extends DescribeManagedEnd
9796
* // s3MonitoringConfiguration: { // S3MonitoringConfiguration
9897
* // logUri: "STRING_VALUE", // required
9998
* // },
99+
* // containerLogRotationConfiguration: { // ContainerLogRotationConfiguration
100+
* // rotationSize: "STRING_VALUE", // required
101+
* // maxFilesToKeep: Number("int"), // required
102+
* // },
100103
* // },
101104
* // },
102105
* // serverUrl: "STRING_VALUE",

‎clients/client-emr-containers/src/commands/ListJobRunsCommand.ts

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,10 @@ export interface ListJobRunsCommandOutput extends ListJobRunsResponse, __Metadat
9595
* // s3MonitoringConfiguration: { // S3MonitoringConfiguration
9696
* // logUri: "STRING_VALUE", // required
9797
* // },
98+
* // containerLogRotationConfiguration: { // ContainerLogRotationConfiguration
99+
* // rotationSize: "STRING_VALUE", // required
100+
* // maxFilesToKeep: Number("int"), // required
101+
* // },
98102
* // },
99103
* // },
100104
* // jobDriver: { // JobDriver

‎clients/client-emr-containers/src/commands/ListManagedEndpointsCommand.ts

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,8 +41,7 @@ export interface ListManagedEndpointsCommandOutput extends ListManagedEndpointsR
4141
/**
4242
* @public
4343
* <p>Lists managed endpoints based on a set of parameters. A managed endpoint is a gateway
44-
* that connects Amazon EMR Studio to Amazon EMR on EKS so that Amazon EMR Studio can communicate with
45-
* your virtual cluster.</p>
44+
* that connects Amazon EMR Studio to Amazon EMR on EKS so that Amazon EMR Studio can communicate with your virtual cluster.</p>
4645
* @example
4746
* Use a bare-bones client and the command you need to make an API call.
4847
* ```javascript
@@ -107,6 +106,10 @@ export interface ListManagedEndpointsCommandOutput extends ListManagedEndpointsR
107106
* // s3MonitoringConfiguration: { // S3MonitoringConfiguration
108107
* // logUri: "STRING_VALUE", // required
109108
* // },
109+
* // containerLogRotationConfiguration: { // ContainerLogRotationConfiguration
110+
* // rotationSize: "STRING_VALUE", // required
111+
* // maxFilesToKeep: Number("int"), // required
112+
* // },
110113
* // },
111114
* // },
112115
* // serverUrl: "STRING_VALUE",

‎clients/client-emr-containers/src/commands/StartJobRunCommand.ts

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,10 @@ export interface StartJobRunCommandOutput extends StartJobRunResponse, __Metadat
9090
* s3MonitoringConfiguration: { // S3MonitoringConfiguration
9191
* logUri: "STRING_VALUE", // required
9292
* },
93+
* containerLogRotationConfiguration: { // ContainerLogRotationConfiguration
94+
* rotationSize: "STRING_VALUE", // required
95+
* maxFilesToKeep: Number("int"), // required
96+
* },
9397
* },
9498
* },
9599
* tags: { // TagMap

‎clients/client-emr-containers/src/commands/TagResourceCommand.ts

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -36,14 +36,15 @@ export interface TagResourceCommandOutput extends TagResourceResponse, __Metadat
3636

3737
/**
3838
* @public
39-
* <p>Assigns tags to resources. A tag is a label that you assign to an Amazon Web Services resource. Each tag
40-
* consists of a key and an optional value, both of which you define. Tags enable you to
41-
* categorize your Amazon Web Services resources by attributes such as purpose, owner, or environment. When
42-
* you have many resources of the same type, you can quickly identify a specific resource
43-
* based on the tags you've assigned to it. For example, you can define a set of tags for your
44-
* Amazon EMR on EKS clusters to help you track each cluster's owner and stack level.
45-
* We recommend that you devise a consistent set of tag keys for each resource type. You can
46-
* then search and filter the resources based on the tags that you add.</p>
39+
* <p>Assigns tags to resources. A tag is a label that you assign to an Amazon Web Services
40+
* resource. Each tag consists of a key and an optional value, both of which you define. Tags
41+
* enable you to categorize your Amazon Web Services resources by attributes such as purpose,
42+
* owner, or environment. When you have many resources of the same type, you can quickly
43+
* identify a specific resource based on the tags you've assigned to it. For example, you can
44+
* define a set of tags for your Amazon EMR on EKS clusters to help you track each
45+
* cluster's owner and stack level. We recommend that you devise a consistent set of tag keys
46+
* for each resource type. You can then search and filter the resources based on the tags that
47+
* you add.</p>
4748
* @example
4849
* Use a bare-bones client and the command you need to make an API call.
4950
* ```javascript

‎clients/client-emr-containers/src/index.ts

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,11 +4,11 @@
44
* <p>Amazon EMR on EKS provides a deployment option for Amazon EMR that allows
55
* you to run open-source big data frameworks on Amazon Elastic Kubernetes Service (Amazon EKS).
66
* With this deployment option, you can focus on running analytics workloads while Amazon EMR on EKS builds, configures, and manages containers for open-source applications.
7-
* For more information about Amazon EMR on EKS concepts and tasks, see <a href="https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/emr-eks.html">What is
7+
* For more information about Amazon EMR on EKS concepts and tasks, see <a href="https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/emr-eks.html">What is
88
* shared id="EMR-EKS"/></a>.</p>
99
* <p>
10-
* <i>Amazon EMR containers</i> is the API name for Amazon EMR on EKS.
11-
* The <code>emr-containers</code> prefix is used in the following scenarios: </p>
10+
* <i>Amazon EMR containers</i> is the API name for Amazon EMR on EKS. The <code>emr-containers</code> prefix is used in the following
11+
* scenarios: </p>
1212
* <ul>
1313
* <li>
1414
* <p>It is the prefix in the CLI commands for Amazon EMR on EKS. For example,

‎clients/client-emr-containers/src/models/models_0.ts

Lines changed: 30 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -272,6 +272,22 @@ export interface CloudWatchMonitoringConfiguration {
272272
logStreamNamePrefix?: string;
273273
}
274274

275+
/**
276+
* @public
277+
* <p>The settings for container log rotation.</p>
278+
*/
279+
export interface ContainerLogRotationConfiguration {
280+
/**
281+
* <p>The file size at which to rotate logs. Minimum of 2KB, Maximum of 2GB.</p>
282+
*/
283+
rotationSize: string | undefined;
284+
285+
/**
286+
* <p>The number of files to keep in container after rotation.</p>
287+
*/
288+
maxFilesToKeep: number | undefined;
289+
}
290+
275291
/**
276292
* @public
277293
* @enum
@@ -317,6 +333,11 @@ export interface MonitoringConfiguration {
317333
* <p>Amazon S3 configuration for monitoring log publishing.</p>
318334
*/
319335
s3MonitoringConfiguration?: S3MonitoringConfiguration;
336+
337+
/**
338+
* <p>Enable or disable container log rotation.</p>
339+
*/
340+
containerLogRotationConfiguration?: ContainerLogRotationConfiguration;
320341
}
321342

322343
/**
@@ -408,7 +429,8 @@ export type ContainerProviderType = (typeof ContainerProviderType)[keyof typeof
408429
*/
409430
export interface ContainerProvider {
410431
/**
411-
* <p>The type of the container provider. Amazon EKS is the only supported type as of now.</p>
432+
* <p>The type of the container provider. Amazon EKS is the only supported type as of
433+
* now.</p>
412434
*/
413435
type: ContainerProviderType | string | undefined;
414436

@@ -697,11 +719,11 @@ export type VirtualClusterState = (typeof VirtualClusterState)[keyof typeof Virt
697719
/**
698720
* @public
699721
* <p>This entity describes a virtual cluster. A virtual cluster is a Kubernetes namespace
700-
* that Amazon EMR is registered with. Amazon EMR uses virtual clusters to run jobs and host
701-
* endpoints. Multiple virtual clusters can be backed by the same physical cluster. However,
702-
* each virtual cluster maps to one namespace on an Amazon EKS cluster. Virtual clusters do not
703-
* create any active resources that contribute to your bill or that require lifecycle
704-
* management outside the service.</p>
722+
* that Amazon EMR is registered with. Amazon EMR uses virtual clusters to run
723+
* jobs and host endpoints. Multiple virtual clusters can be backed by the same physical
724+
* cluster. However, each virtual cluster maps to one namespace on an Amazon EKS
725+
* cluster. Virtual clusters do not create any active resources that contribute to your bill
726+
* or that require lifecycle management outside the service.</p>
705727
*/
706728
export interface VirtualCluster {
707729
/**
@@ -1001,8 +1023,8 @@ export interface ListVirtualClustersRequest {
10011023
containerProviderId?: string;
10021024

10031025
/**
1004-
* <p>The container provider type of the virtual cluster. Amazon EKS is the only supported type as of
1005-
* now.</p>
1026+
* <p>The container provider type of the virtual cluster. Amazon EKS is the only
1027+
* supported type as of now.</p>
10061028
*/
10071029
containerProviderType?: ContainerProviderType | string;
10081030

‎clients/client-emr-containers/src/protocols/Aws_restJson1.ts

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,7 @@ import {
7777
Configuration,
7878
ConfigurationOverrides,
7979
ContainerInfo,
80+
ContainerLogRotationConfiguration,
8081
ContainerProvider,
8182
EksInfo,
8283
Endpoint,
@@ -1949,6 +1950,8 @@ const se_ConfigurationOverrides = (input: ConfigurationOverrides, context: __Ser
19491950

19501951
// se_ContainerInfo omitted.
19511952

1953+
// se_ContainerLogRotationConfiguration omitted.
1954+
19521955
// se_ContainerProvider omitted.
19531956

19541957
// se_EksInfo omitted.
@@ -2046,6 +2049,8 @@ const de_ConfigurationOverrides = (output: any, context: __SerdeContext): Config
20462049

20472050
// de_ContainerInfo omitted.
20482051

2052+
// de_ContainerLogRotationConfiguration omitted.
2053+
20492054
// de_ContainerProvider omitted.
20502055

20512056
// de_Credentials omitted.

0 commit comments

Comments
 (0)