Ability to disable metric extraction

[Dunedan]

I'm looking for an ability to disable the metric extraction and ingestion into CloudWatch Metrics, while still logging them to CloudWatch Logs.

That ability is important for me as I want to be able to disable ingestion into CloudWatch Metrics on demand for cost saving purposes while still receiving the data in CloudWatch Logs enabling me to analytics with CloudWatch Insights, even though they aren't ingested into CloudWatch Metrics.

From reading through the EMF specification (https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html) I didn't notice any flag to indicate if metrics should be ingested into CloudWatch Metrics or not.

Is there an option to choose whether such metrics should get ingested by CloudWatch Metrics or not? If not it'd be great to have such an option.

[jaredcnance]

We don’t currently support any kind of enable/disable control plane operation. How are you imagining you would control this? We could consider doing something at the library level that could accept a configuration parameter (e.g. environment variable) that would essentially disable inclusion of the EMF metadata. Alternatively, we might allow disabling it for a LogGroup through a CloudWatch Logs API call.

I would also like to clarify the cost control concerns since the only thing you’d be disabling is the metric name cost (metric-hours). There’s no additional metric API charge with EMF and typically the bulk of the cost is in log ingestion.

[Dunedan]

How are you imagining you would control this? We could consider doing something at the library level that could accept a configuration parameter (e.g. environment variable) that would essentially disable inclusion of the EMF metadata. Alternatively, we might allow disabling it for a LogGroup through a CloudWatch Logs API call.

I see a benefit of having the metadata included, but not metrics generated out of it, as that still retains the ability to do manual analysis of the data. Having both options (disable metrics generation & disabling inclusion of EMF metadata completely) would be perfect. However a configuration parameter to disable the inclusion of the metadata is probably a more pragmatic solution for now. An environment variable would be perfectly fine for this.

Disabling it per LogGroup on CloudWatch side would probably come with its own problems. For example all AWS Glue Python Jobs currently share a single LogGroup, thus this wouldn't allow enabling metrics generation for a subset of AWS Glue Python Jobs.

I would also like to clarify the cost control concerns since the only thing you’d be disabling is the metric name cost (metric-hours). There’s no additional metric API charge with EMF and typically the bulk of the cost is in log ingestion.

Yes, the metric hours are the costs I'm primarily concerned about and where I'd like to have a quick and easy way to avoid them, as we only emit values for metrics a few times per hour max. Our use case is probably not the typical use case for such metrics, so having the ability to avoid log ingestion completely makes a lot of sense when sending more metrics.

[jaredcnance]

This would be pretty straightforward to add. We can include a DISABLE_METRIC_AGGREGATION config value that gets checked below and doesn’t add the _aws metadata object to the payload. This would give you all of the metric and property data in CloudWatch Logs but disable the materialized metrics in CloudWatch Metrics.

aws-embedded-metrics-python/aws_embedded_metrics/serializers/log_serializer.py

Lines 32 to 46 in 37e5113

	def create_body() -> Dict[str, Any]:
	return {
	**dimensions_properties,
	**context.properties,
	"_aws": {
	**context.meta,
	"CloudWatchMetrics": [
	{
	"Dimensions": dimension_keys,
	"Metrics": [],
	"Namespace": context.namespace,
	},
	],
	},
	}