Skip to content

Update to work with updated AWS requirements #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 11, 2019
Merged

Update to work with updated AWS requirements #14

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 1 addition & 8 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,13 +83,6 @@ Available targets:
lint Lint terraform code

```
## Module: cloudposse/terraform-aws-lambda-elasticsearch-cleanup

This module creates a scheduled Lambda function which will delete old
Elasticsearch indexes using SigV4Auth authentication. The lambda
function can optionally send output to an SNS topic if the topic ARN
is given

## Inputs

| Name | Description | Type | Default | Required |
Expand All @@ -101,7 +94,7 @@ is given
| es_domain_arn | The Elasticsearch domain ARN | string | - | yes |
| es_endpoint | The Elasticsearch endpoint for the Lambda function to connect to | string | - | yes |
| es_security_group_id | The Elasticsearch cluster security group ID | string | - | yes |
| index | Index/indices to process. Use a comma-separated list. Specify `all` to match every index except for `.kibana` | string | `all` | no |
| index | Index/indices to process. Use a comma-separated list. Specify `all` to match every index except for `.kibana` or `.kibana_1` | string | `all` | no |
| index_format | Combined with 'index' variable and is used to evaluate the index age | string | `%Y.%m.%d` | no |
| name | Solution name, e.g. 'app' or 'cluster' | string | `app` | no |
| namespace | Namespace, which could be your organization name, e.g. 'eg' or 'cp' | string | - | yes |
Expand Down
9 changes: 1 addition & 8 deletions docs/terraform.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,3 @@
## Module: cloudposse/terraform-aws-lambda-elasticsearch-cleanup

This module creates a scheduled Lambda function which will delete old
Elasticsearch indexes using SigV4Auth authentication. The lambda
function can optionally send output to an SNS topic if the topic ARN
is given

## Inputs

| Name | Description | Type | Default | Required |
Expand All @@ -16,7 +9,7 @@ is given
| es_domain_arn | The Elasticsearch domain ARN | string | - | yes |
| es_endpoint | The Elasticsearch endpoint for the Lambda function to connect to | string | - | yes |
| es_security_group_id | The Elasticsearch cluster security group ID | string | - | yes |
| index | Index/indices to process. Use a comma-separated list. Specify `all` to match every index except for `.kibana` | string | `all` | no |
| index | Index/indices to process. Use a comma-separated list. Specify `all` to match every index except for `.kibana` or `.kibana_1` | string | `all` | no |
| index_format | Combined with 'index' variable and is used to evaluate the index age | string | `%Y.%m.%d` | no |
| name | Solution name, e.g. 'app' or 'cluster' | string | `app` | no |
| namespace | Namespace, which could be your organization name, e.g. 'eg' or 'cp' | string | - | yes |
Expand Down
11 changes: 7 additions & 4 deletions lambda/es-cleanup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@
# -*- coding: utf-8 -*-
"""
This AWS Lambda function allowed to delete the old Elasticsearch index

THIS FILE IS NOT EXACTLY THE ORIGINAL FILE DISTRIBUTED BY Cloudreach Europe Limited
IT HAS BEEN MODIFIED BY Cloud Posse, LLC
"""

from __future__ import print_function
Expand All @@ -14,8 +17,8 @@
from botocore.auth import SigV4Auth
from botocore.awsrequest import AWSRequest
from botocore.credentials import create_credential_resolver
from botocore.httpsession import URLLib3Session
from botocore.session import get_session
from botocore.vendored.requests import Session
import sys
if sys.version_info[0] == 3:
from urllib.request import quote
Expand Down Expand Up @@ -117,7 +120,7 @@ def send_to_es(self, path, method="GET", payload={}):

try:
preq = req.prepare()
session = Session()
session = URLLib3Session()
res = session.send(preq)
if res.status_code >= 200 and res.status_code <= 299:
# print("%s %s" % (res.status_code, res.content))
Expand Down Expand Up @@ -186,9 +189,9 @@ def lambda_handler(event, context):
earliest_to_keep = datetime.date.today() - datetime.timedelta(
days=int(es.cfg["delete_after"]))
for index in es.get_indices():
if index["index"] == ".kibana":
if index["index"] == ".kibana" or index["index"] == ".kibana_1":
# ignore .kibana index
print("Found .kibana index - ignoring")
print("Found Kibana index: %s - ignoring" % index["index"])
continue

idx_name = '-'.join(word for word in index["index"].split("-")[:-1])
Expand Down
2 changes: 1 addition & 1 deletion variables.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ variable "sns_arn" {
variable "index" {
type = "string"
default = "all"
description = "Index/indices to process. Use a comma-separated list. Specify `all` to match every index except for `.kibana`"
description = "Index/indices to process. Use a comma-separated list. Specify `all` to match every index except for `.kibana` or `.kibana_1`"
}

variable "delete_after" {
Expand Down