Skip to content

Commit 21624d9

Browse files
author
marcin
authored
Matching index with regex pattern (#20)
* Add matching index with regex to avoid deleting all indexes for different patterns * Readme * Add me to contributors * Rebase + readme * Fix tests * Fix test assertion order
1 parent 5bf58fd commit 21624d9

File tree

7 files changed

+113
-55
lines changed

7 files changed

+113
-55
lines changed

‎README.md

Lines changed: 51 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -127,38 +127,61 @@ Available targets:
127127
lint Lint terraform code
128128
129129
```
130+
## Module: cloudposse/terraform-aws-lambda-elasticsearch-cleanup
131+
132+
This module creates a scheduled Lambda function which will delete old
133+
Elasticsearch indexes using SigV4Auth authentication. The lambda
134+
function can optionally send output to an SNS topic if the topic ARN
135+
is given
136+
137+
## Requirements
138+
139+
| Name | Version |
140+
|------|---------|
141+
| terraform | ~> 0.12.0 |
142+
| aws | ~> 2.0 |
143+
| null | ~> 2.0 |
144+
| template | ~> 2.0 |
145+
146+
## Providers
147+
148+
| Name | Version |
149+
|------|---------|
150+
| aws | ~> 2.0 |
151+
130152
## Inputs
131153

132154
| Name | Description | Type | Default | Required |
133-
|------|-------------|:----:|:-----:|:-----:|
134-
| artifact_url | URL template for the remote artifact | string | `https://artifacts.cloudposse.com/$$${module_name}/$$${git_ref}/$$${filename}` | no |
135-
| attributes | Additional attributes (e.g. `1`) | list(string) | `<list>` | no |
136-
| delete_after | Number of days to preserve | number | `15` | no |
137-
| delimiter | Delimiter to be used between `namespace`, `stage`, `name` and `attributes` | string | `-` | no |
138-
| enabled | This module will not create any resources unless enabled is set to "true" | bool | `true` | no |
139-
| es_domain_arn | The Elasticsearch domain ARN | string | - | yes |
140-
| es_endpoint | The Elasticsearch endpoint for the Lambda function to connect to | string | - | yes |
141-
| es_security_group_id | The Elasticsearch cluster security group ID | string | - | yes |
142-
| index | Index/indices to process. Use a comma-separated list. Specify `all` to match every index except for `.kibana` or `.kibana_1` | string | `all` | no |
143-
| index_format | Combined with 'index' variable and is used to evaluate the index age | string | `%Y.%m.%d` | no |
144-
| name | Solution name, e.g. 'app' or 'cluster' | string | `app` | no |
145-
| namespace | Namespace, which could be your organization name, e.g. 'eg' or 'cp' | string | `` | no |
146-
| python_version | The Python version to use | string | `2.7` | no |
147-
| schedule | CloudWatch Events rule schedule using cron or rate expression | string | `cron(0 3 * * ? *)` | no |
148-
| sns_arn | SNS ARN to publish alerts | string | `` | no |
149-
| stage | Stage, e.g. 'prod', 'staging', 'dev', or 'test' | string | `` | no |
150-
| subnet_ids | Subnet IDs | list(string) | - | yes |
151-
| tags | Additional tags (e.g. `map('BusinessUnit','XYZ')` | map(string) | `<map>` | no |
152-
| timeout | Timeout for Lambda function in seconds | number | `300` | no |
153-
| vpc_id | The VPC ID for the Lambda function | string | - | yes |
155+
|------|-------------|------|---------|:--------:|
156+
| artifact\_url | URL template for the remote artifact | `string` | `"https://artifacts.cloudposse.com/$${module_name}/$${git_ref}/$${filename}"` | no |
157+
| attributes | Additional attributes (e.g. `1`) | `list(string)` | `[]` | no |
158+
| delete\_after | Number of days to preserve | `number` | `15` | no |
159+
| delimiter | Delimiter to be used between `namespace`, `stage`, `name` and `attributes` | `string` | `"-"` | no |
160+
| enabled | This module will not create any resources unless enabled is set to "true" | `bool` | `true` | no |
161+
| es\_domain\_arn | The Elasticsearch domain ARN | `string` | n/a | yes |
162+
| es\_endpoint | The Elasticsearch endpoint for the Lambda function to connect to | `string` | n/a | yes |
163+
| es\_security\_group\_id | The Elasticsearch cluster security group ID | `string` | n/a | yes |
164+
| index | Index/indices to process. Use a comma-separated list. Specify `all` to match every index except for `.kibana` or `.kibana_1` | `string` | `"all"` | no |
165+
| index\_format | Combined with 'index' variable and is used to evaluate the index age | `string` | `"%Y.%m.%d"` | no |
166+
| index\_regex | Determines regex that is used for matching index name and index date. By default it match two groups separated by hyphen. | `string` | `"([^-]+)-(.*)"` | no |
167+
| name | Solution name, e.g. 'app' or 'cluster' | `string` | `"app"` | no |
168+
| namespace | Namespace, which could be your organization name, e.g. 'eg' or 'cp' | `string` | `""` | no |
169+
| python\_version | The Python version to use | `string` | `"2.7"` | no |
170+
| schedule | CloudWatch Events rule schedule using cron or rate expression | `string` | `"cron(0 3 * * ? *)"` | no |
171+
| sns\_arn | SNS ARN to publish alerts | `string` | `""` | no |
172+
| stage | Stage, e.g. 'prod', 'staging', 'dev', or 'test' | `string` | `""` | no |
173+
| subnet\_ids | Subnet IDs | `list(string)` | n/a | yes |
174+
| tags | Additional tags (e.g. `map('BusinessUnit','XYZ')` | `map(string)` | `{}` | no |
175+
| timeout | Timeout for Lambda function in seconds | `number` | `300` | no |
176+
| vpc\_id | The VPC ID for the Lambda function | `string` | n/a | yes |
154177

155178
## Outputs
156179

157180
| Name | Description |
158181
|------|-------------|
159-
| lambda_function_arn | ARN of the Lambda Function |
160-
| lambda_function_source_code_size | The size in bytes of the function .zip file |
161-
| security_group_id | Security Group ID of the Lambda Function |
182+
| lambda\_function\_arn | ARN of the Lambda Function |
183+
| lambda\_function\_source\_code\_size | The size in bytes of the function .zip file |
184+
| security\_group\_id | Security Group ID of the Lambda Function |
162185

163186

164187

@@ -306,8 +329,8 @@ Check out [our other projects][github], [follow us on twitter][twitter], [apply
306329

307330
### Contributors
308331

309-
| [![Josh Myers][joshmyers_avatar]][joshmyers_homepage]<br/>[Josh Myers][joshmyers_homepage] | [![Erik Osterman][osterman_avatar]][osterman_homepage]<br/>[Erik Osterman][osterman_homepage] | [![Andriy Knysh][aknysh_avatar]][aknysh_homepage]<br/>[Andriy Knysh][aknysh_homepage] | [![Igor Rodionov][goruha_avatar]][goruha_homepage]<br/>[Igor Rodionov][goruha_homepage] |
310-
|---|---|---|---|
332+
| [![Josh Myers][joshmyers_avatar]][joshmyers_homepage]<br/>[Josh Myers][joshmyers_homepage] | [![Erik Osterman][osterman_avatar]][osterman_homepage]<br/>[Erik Osterman][osterman_homepage] | [![Andriy Knysh][aknysh_avatar]][aknysh_homepage]<br/>[Andriy Knysh][aknysh_homepage] | [![Igor Rodionov][goruha_avatar]][goruha_homepage]<br/>[Igor Rodionov][goruha_homepage] | [![Marcin Brański][3h4x_avatar]][3h4x_homepage]<br/>[Marcin Brański][3h4x_homepage] |
333+
|---|---|---|---|---|
311334

312335
[joshmyers_homepage]: https://github.com/joshmyers
313336
[joshmyers_avatar]: https://img.cloudposse.com/150x150/https://github.com/joshmyers.png
@@ -317,6 +340,8 @@ Check out [our other projects][github], [follow us on twitter][twitter], [apply
317340
[aknysh_avatar]: https://img.cloudposse.com/150x150/https://github.com/aknysh.png
318341
[goruha_homepage]: https://github.com/goruha
319342
[goruha_avatar]: https://img.cloudposse.com/150x150/https://github.com/goruha.png
343+
[3h4x_homepage]: https://github.com/3h4x
344+
[3h4x_avatar]: https://img.cloudposse.com/150x150/https://github.com/3h4x.png
320345

321346
[![README Footer][readme_footer_img]][readme_footer_link]
322347
[![Beacon][beacon]][website]

‎README.yaml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,3 +58,5 @@ contributors:
5858
github: aknysh
5959
- name: Igor Rodionov
6060
github: goruha
61+
- name: Marcin Brański
62+
github: 3h4x

‎docs/terraform.md

Lines changed: 47 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,33 +1,56 @@
1+
## Module: cloudposse/terraform-aws-lambda-elasticsearch-cleanup
2+
3+
This module creates a scheduled Lambda function which will delete old
4+
Elasticsearch indexes using SigV4Auth authentication. The lambda
5+
function can optionally send output to an SNS topic if the topic ARN
6+
is given
7+
8+
## Requirements
9+
10+
| Name | Version |
11+
|------|---------|
12+
| terraform | ~> 0.12.0 |
13+
| aws | ~> 2.0 |
14+
| null | ~> 2.0 |
15+
| template | ~> 2.0 |
16+
17+
## Providers
18+
19+
| Name | Version |
20+
|------|---------|
21+
| aws | ~> 2.0 |
22+
123
## Inputs
224

325
| Name | Description | Type | Default | Required |
4-
|------|-------------|:----:|:-----:|:-----:|
5-
| artifact_url | URL template for the remote artifact | string | `https://artifacts.cloudposse.com/$$${module_name}/$$${git_ref}/$$${filename}` | no |
6-
| attributes | Additional attributes (e.g. `1`) | list(string) | `<list>` | no |
7-
| delete_after | Number of days to preserve | number | `15` | no |
8-
| delimiter | Delimiter to be used between `namespace`, `stage`, `name` and `attributes` | string | `-` | no |
9-
| enabled | This module will not create any resources unless enabled is set to "true" | bool | `true` | no |
10-
| es_domain_arn | The Elasticsearch domain ARN | string | - | yes |
11-
| es_endpoint | The Elasticsearch endpoint for the Lambda function to connect to | string | - | yes |
12-
| es_security_group_id | The Elasticsearch cluster security group ID | string | - | yes |
13-
| index | Index/indices to process. Use a comma-separated list. Specify `all` to match every index except for `.kibana` or `.kibana_1` | string | `all` | no |
14-
| index_format | Combined with 'index' variable and is used to evaluate the index age | string | `%Y.%m.%d` | no |
15-
| name | Solution name, e.g. 'app' or 'cluster' | string | `app` | no |
16-
| namespace | Namespace, which could be your organization name, e.g. 'eg' or 'cp' | string | `` | no |
17-
| python_version | The Python version to use | string | `2.7` | no |
18-
| schedule | CloudWatch Events rule schedule using cron or rate expression | string | `cron(0 3 * * ? *)` | no |
19-
| sns_arn | SNS ARN to publish alerts | string | `` | no |
20-
| stage | Stage, e.g. 'prod', 'staging', 'dev', or 'test' | string | `` | no |
21-
| subnet_ids | Subnet IDs | list(string) | - | yes |
22-
| tags | Additional tags (e.g. `map('BusinessUnit','XYZ')` | map(string) | `<map>` | no |
23-
| timeout | Timeout for Lambda function in seconds | number | `300` | no |
24-
| vpc_id | The VPC ID for the Lambda function | string | - | yes |
26+
|------|-------------|------|---------|:--------:|
27+
| artifact\_url | URL template for the remote artifact | `string` | `"https://artifacts.cloudposse.com/$${module_name}/$${git_ref}/$${filename}"` | no |
28+
| attributes | Additional attributes (e.g. `1`) | `list(string)` | `[]` | no |
29+
| delete\_after | Number of days to preserve | `number` | `15` | no |
30+
| delimiter | Delimiter to be used between `namespace`, `stage`, `name` and `attributes` | `string` | `"-"` | no |
31+
| enabled | This module will not create any resources unless enabled is set to "true" | `bool` | `true` | no |
32+
| es\_domain\_arn | The Elasticsearch domain ARN | `string` | n/a | yes |
33+
| es\_endpoint | The Elasticsearch endpoint for the Lambda function to connect to | `string` | n/a | yes |
34+
| es\_security\_group\_id | The Elasticsearch cluster security group ID | `string` | n/a | yes |
35+
| index | Index/indices to process. Use a comma-separated list. Specify `all` to match every index except for `.kibana` or `.kibana_1` | `string` | `"all"` | no |
36+
| index\_format | Combined with 'index' variable and is used to evaluate the index age | `string` | `"%Y.%m.%d"` | no |
37+
| index\_regex | Determines regex that is used for matching index name and index date. By default it match two groups separated by hyphen. | `string` | `"([^-]+)-(.*)"` | no |
38+
| name | Solution name, e.g. 'app' or 'cluster' | `string` | `"app"` | no |
39+
| namespace | Namespace, which could be your organization name, e.g. 'eg' or 'cp' | `string` | `""` | no |
40+
| python\_version | The Python version to use | `string` | `"2.7"` | no |
41+
| schedule | CloudWatch Events rule schedule using cron or rate expression | `string` | `"cron(0 3 * * ? *)"` | no |
42+
| sns\_arn | SNS ARN to publish alerts | `string` | `""` | no |
43+
| stage | Stage, e.g. 'prod', 'staging', 'dev', or 'test' | `string` | `""` | no |
44+
| subnet\_ids | Subnet IDs | `list(string)` | n/a | yes |
45+
| tags | Additional tags (e.g. `map('BusinessUnit','XYZ')` | `map(string)` | `{}` | no |
46+
| timeout | Timeout for Lambda function in seconds | `number` | `300` | no |
47+
| vpc\_id | The VPC ID for the Lambda function | `string` | n/a | yes |
2548

2649
## Outputs
2750

2851
| Name | Description |
2952
|------|-------------|
30-
| lambda_function_arn | ARN of the Lambda Function |
31-
| lambda_function_source_code_size | The size in bytes of the function .zip file |
32-
| security_group_id | Security Group ID of the Lambda Function |
53+
| lambda\_function\_arn | ARN of the Lambda Function |
54+
| lambda\_function\_source\_code\_size | The size in bytes of the function .zip file |
55+
| security\_group\_id | Security Group ID of the Lambda Function |
3356

‎lambda/es-cleanup.py

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010
from __future__ import print_function
1111
import os
1212
import json
13+
import re
1314
import time
1415
import boto3
1516
import datetime
@@ -62,6 +63,7 @@ def __init__(self, event, context):
6263
self.cfg["es_max_retry"] = int(self.get_parameter("es_max_retry", 3))
6364
self.cfg["index_format"] = self.get_parameter(
6465
"index_format", "%Y.%m.%d")
66+
self.cfg["index_regex"] = self.get_parameter("index_regex", "([^-]+)-(.*)")
6567
self.cfg["sns_arn"] = self.get_parameter("sns_arn", "")
6668

6769
if not self.cfg["es_endpoint"]:
@@ -194,8 +196,7 @@ def lambda_handler(event, context):
194196
print("Found Kibana index: %s - ignoring" % index["index"])
195197
continue
196198

197-
idx_name = '-'.join(word for word in index["index"].split("-")[:-1])
198-
idx_date = index["index"].split("-")[-1]
199+
idx_name, idx_date = re.match(es.cfg["index_regex"], index["index"]).groups()
199200
print("Found index: %s - %s" % (idx_name, idx_date))
200201
if idx_name in es.cfg["index"] or "all" in es.cfg["index"]:
201202

‎main.tf

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -116,9 +116,10 @@ resource "aws_lambda_function" "default" {
116116

117117
environment {
118118
variables = {
119+
delete_after = var.delete_after
119120
es_endpoint = var.es_endpoint
120121
index = var.index
121-
delete_after = var.delete_after
122+
index_regex = var.index_regex
122123
index_format = var.index_format
123124
sns_arn = var.sns_arn
124125
}

‎test/src/examples_complete_test.go

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -33,12 +33,12 @@ func TestExamplesComplete(t *testing.T) {
3333
// Run `terraform output` to get the value of an output variable
3434
privateSubnetCidrs := terraform.OutputList(t, terraformOptions, "private_subnet_cidrs")
3535
// Verify we're getting back the outputs we expect
36-
assert.Equal(t, []string{"172.16.0.0/18", "172.16.64.0/18"}, privateSubnetCidrs)
36+
assert.Equal(t, []string{"172.16.0.0/19", "172.16.32.0/19"}, privateSubnetCidrs)
3737

3838
// Run `terraform output` to get the value of an output variable
3939
publicSubnetCidrs := terraform.OutputList(t, terraformOptions, "public_subnet_cidrs")
4040
// Verify we're getting back the outputs we expect
41-
assert.Equal(t, []string{"172.16.128.0/18", "172.16.192.0/18"}, publicSubnetCidrs)
41+
assert.Equal(t, []string{"172.16.96.0/19", "172.16.128.0/19"}, publicSubnetCidrs)
4242

4343
// Run `terraform output` to get the value of an output variable
4444
domainHostname := terraform.Output(t, terraformOptions, "domain_hostname")

‎variables.tf

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,12 @@ variable "index_format" {
9595
description = "Combined with 'index' variable and is used to evaluate the index age"
9696
}
9797

98+
variable "index_regex" {
99+
type = string
100+
default = "([^-]+)-(.*)"
101+
description = "Determines regex that is used for matching index name and index date. By default it match two groups separated by hyphen."
102+
}
103+
98104
variable "python_version" {
99105
type = string
100106
default = "2.7"

0 commit comments

Comments
 (0)