Merge branch 'develop' into time2

Author: ran-isenberg

.github/boring-cyborg.yml

@@ -1,48 +1,48 @@
 ##### Labeler ##########################################################################################################
 labelPRBasedOnFilePath:
-  area/logger:
+  logger:
     - aws_lambda_powertools/logging/*
     - aws_lambda_powertools/logging/**/*
     - aws_lambda_powertools/package_logger.py
-  area/tracer:
+  tracer:
     - aws_lambda_powertools/tracing/*
     - aws_lambda_powertools/tracing/**/*
-  area/metrics:
+  metrics:
     - aws_lambda_powertools/metrics/*
     - aws_lambda_powertools/metrics/**/*
-  area/event_handlers:
+  event_handlers:
     - aws_lambda_powertools/event_handler/*
     - aws_lambda_powertools/event_handler/**/*
-  area/middleware_factory:
+  middleware_factory:
     - aws_lambda_powertools/middleware_factory/*
     - aws_lambda_powertools/middleware_factory/**/*
-  area/parameters:
+  parameters:
     - aws_lambda_powertools/parameters/*
     - aws_lambda_powertools/parameters/**/*
-  area/batch:
+  batch:
     - aws_lambda_powertools/batch/*
     - aws_lambda_powertools/batch/**/*
-  area/validator:
+  validator:
     - aws_lambda_powertools/validation/*
     - aws_lambda_powertools/validation/**/*
-  area/event_sources:
+  event_sources:
     - aws_lambda_powertools/data_classes/*
     - aws_lambda_powertools/data_classes/**/*
-  area/parser:
+  parser:
     - aws_lambda_powertools/parser/*
     - aws_lambda_powertools/parser/**/*
-  area/idempotency:
+  idempotency:
     - aws_lambda_powertools/idempotency/*
     - aws_lambda_powertools/idempotency/**/*
-  area/feature_flags:
+  feature_flags:
     - aws_lambda_powertools/feature_flags/*
     - aws_lambda_powertools/feature_flags/**/*
-  area/jmespath:
+  jmespath:
     - aws_lambda_powertools/utilities/jmespath_utils/*
-  area/typing:
+  typing:
     - aws_lambda_powertools/utilities/typing/*
     - mypy.ini
-  area/commons:
+  commons:
     - aws_lambda_powertools/shared/*
 
   documentation:

CHANGELOG.md

@@ -4,15 +4,25 @@
 <a name="unreleased"></a>
 # Unreleased
 
+## Features
+
+* **metrics:** add EphemeralMetrics as a non-singleton option ([#1676](https://github.com/awslabs/aws-lambda-powertools-python/issues/1676))
+
+
+<a name="v2.1.0"></a>
+## [v2.1.0] - 2022-10-31
 ## Bug Fixes
 
 * **ci:** linting issues after flake8-blackbear,mypy upgrades
 * **deps:** update build system to poetry-core ([#1651](https://github.com/awslabs/aws-lambda-powertools-python/issues/1651))
+* **idempotency:** idempotent_function should support standalone falsy values ([#1669](https://github.com/awslabs/aws-lambda-powertools-python/issues/1669))
+* **logger:** fix unknown attributes being ignored by mypy ([#1670](https://github.com/awslabs/aws-lambda-powertools-python/issues/1670))
 
 ## Documentation
 
 * **community:** fix social handlers for Ran ([#1654](https://github.com/awslabs/aws-lambda-powertools-python/issues/1654))
 * **community:** fix twitch parent domain for embedded video
+* **homepage:** remove 3.6 and add hero image
 * **homepage:** add Pulumi code example ([#1652](https://github.com/awslabs/aws-lambda-powertools-python/issues/1652))
 * **index:** fold support us banner
 * **index:** add quotes to pip for zsh customers
@@ -22,16 +32,21 @@
 ## Features
 
 * **layers:** add layer balancer script ([#1643](https://github.com/awslabs/aws-lambda-powertools-python/issues/1643))
+* **logger:** add use_rfc3339 and auto-complete formatter opts in Logger ([#1662](https://github.com/awslabs/aws-lambda-powertools-python/issues/1662))
 * **logger:** accept arbitrary keyword=value for ephemeral metadata ([#1658](https://github.com/awslabs/aws-lambda-powertools-python/issues/1658))
 
 ## Maintenance
 
+* update v2 layer ARN on documentation
 * **ci:** fix typo on version description
-* **deps:** bump docker/setup-qemu-action from 2.0.0 to 2.1.0 ([#1627](https://github.com/awslabs/aws-lambda-powertools-python/issues/1627))
 * **deps:** bump peaceiris/actions-gh-pages from 3.8.0 to 3.9.0 ([#1649](https://github.com/awslabs/aws-lambda-powertools-python/issues/1649))
+* **deps:** bump docker/setup-qemu-action from 2.0.0 to 2.1.0 ([#1627](https://github.com/awslabs/aws-lambda-powertools-python/issues/1627))
+* **deps-dev:** bump aws-cdk-lib from 2.47.0 to 2.48.0 ([#1664](https://github.com/awslabs/aws-lambda-powertools-python/issues/1664))
 * **deps-dev:** bump flake8-variables-names from 0.0.4 to 0.0.5 ([#1628](https://github.com/awslabs/aws-lambda-powertools-python/issues/1628))
 * **deps-dev:** bump pytest-asyncio from 0.16.0 to 0.20.1 ([#1635](https://github.com/awslabs/aws-lambda-powertools-python/issues/1635))
+* **deps-dev:** bump aws-cdk-lib from 2.48.0 to 2.49.0 ([#1671](https://github.com/awslabs/aws-lambda-powertools-python/issues/1671))
 * **docs:** remove v2 banner on top of the docs
+* **governance:** remove 'area/' from PR labels
 
 
 <a name="v2.0.0"></a>
@@ -2524,7 +2539,8 @@
 * Merge pull request [#5](https://github.com/awslabs/aws-lambda-powertools-python/issues/5) from jfuss/feat/python38
 
 
-[Unreleased]: https://github.com/awslabs/aws-lambda-powertools-python/compare/v2.0.0...HEAD
+[Unreleased]: https://github.com/awslabs/aws-lambda-powertools-python/compare/v2.1.0...HEAD
+[v2.1.0]: https://github.com/awslabs/aws-lambda-powertools-python/compare/v2.0.0...v2.1.0
 [v2.0.0]: https://github.com/awslabs/aws-lambda-powertools-python/compare/v1.31.1...v2.0.0
 [v1.31.1]: https://github.com/awslabs/aws-lambda-powertools-python/compare/v1.31.0...v1.31.1
 [v1.31.0]: https://github.com/awslabs/aws-lambda-powertools-python/compare/v1.30.0...v1.31.0

aws_lambda_powertools/logging/formatter.py

@@ -62,6 +62,7 @@ class LambdaPowertoolsFormatter(BasePowertoolsFormatter):
 
     default_time_format = "%Y-%m-%d %H:%M:%S,%F%z"  # '2021-04-17 18:19:57,656+0200'
     custom_ms_time_directive = "%F"
+    RFC3339_ISO8601_FORMAT = "%Y-%m-%dT%H:%M:%S.%F%z"  # '2022-10-27T16:27:43.738+02:00'
 
     def __init__(
         self,
@@ -72,6 +73,7 @@ def __init__(
         use_datetime_directive: bool = False,
         log_record_order: Optional[List[str]] = None,
         utc: bool = False,
+        use_rfc3339: bool = False,
         **kwargs,
     ):
         """Return a LambdaPowertoolsFormatter instance.
@@ -106,6 +108,9 @@ def __init__(
             also supports a custom %F directive for milliseconds.
         utc : bool, optional
             set logging timestamp to UTC, by default False to continue to use local time as per stdlib
+        use_rfc3339: bool, optional
+            Whether to use a popular dateformat that complies with both RFC3339 and ISO8601.
+            e.g., 2022-10-27T16:27:43.738+02:00.
         log_record_order : list, optional
             set order of log keys when logging, by default ["level", "location", "message", "timestamp"]
         kwargs
@@ -129,6 +134,7 @@ def __init__(
         self.log_record_order = log_record_order or ["level", "location", "message", "timestamp"]
         self.log_format = dict.fromkeys(self.log_record_order)  # Set the insertion order for the log messages
         self.update_formatter = self.append_keys  # alias to old method
+        self.use_rfc3339_iso8601 = use_rfc3339
 
         if self.utc:
             self.converter = time.gmtime  # type: ignore
@@ -153,36 +159,51 @@ def format(self, record: logging.LogRecord) -> str:  # noqa: A003
         return self.serialize(log=formatted_log)
 
     def formatTime(self, record: logging.LogRecord, datefmt: Optional[str] = None) -> str:
+        # As of Py3.7, we can infer milliseconds directly from any datetime
+        # saving processing time as we can shortcircuit early
+        # Maintenance: In V3, we (and Java) should move to this format by default
+        # since we've provided enough time for those migrating from std logging
+        if self.use_rfc3339_iso8601:
+            if self.utc:
+                ts_as_datetime = datetime.fromtimestamp(record.created, tz=timezone.utc)
+            else:
+                ts_as_datetime = datetime.fromtimestamp(record.created).astimezone()
+
+            return ts_as_datetime.isoformat(timespec="milliseconds")  # 2022-10-27T17:42:26.841+0200
+
+        # converts to local/UTC TZ as struct time
         record_ts = self.converter(record.created)  # type: ignore
 
         if datefmt is None:  # pragma: no cover, it'll always be None in std logging, but mypy
             datefmt = self.datefmt
 
         # NOTE: Python `time.strftime` doesn't provide msec directives
-        # so we create a custom one (%F) and replace logging record ts
+        # so we create a custom one (%F) and replace logging record_ts
         # Reason 2 is that std logging doesn't support msec after TZ
         msecs = "%03d" % record.msecs
 
-        # Datetime format codes might be optionally used
-        # however it only makes a difference if `datefmt` is passed
-        # since format codes are the same except %f
+        # Datetime format codes is a superset of time format codes
+        # therefore we only honour them if explicitly asked
+        # by default, those migrating from std logging will use time format codes
+        # https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes
         if self.use_datetime_directive and datefmt:
-            # record.msecs are microseconds, divide by 1000 and we get milliseconds
+            # record.msecs are microseconds, divide by 1000 to get milliseconds
             timestamp = record.created + record.msecs / 1000
 
             if self.utc:
                 dt = datetime.fromtimestamp(timestamp, tz=timezone.utc)
             else:
-                # make sure local timezone is included
                 dt = datetime.fromtimestamp(timestamp).astimezone()
 
             custom_fmt = datefmt.replace(self.custom_ms_time_directive, msecs)
             return dt.strftime(custom_fmt)
 
+        # Only time format codes being used
         elif datefmt:
             custom_fmt = datefmt.replace(self.custom_ms_time_directive, msecs)
             return time.strftime(custom_fmt, record_ts)
 
+        # Use default fmt: 2021-05-03 10:20:19,650+0200
         custom_fmt = self.default_time_format.replace(self.custom_ms_time_directive, msecs)
         return time.strftime(custom_fmt, record_ts)
 

aws_lambda_powertools/logging/logger.py

@@ -6,7 +6,19 @@
 import random
 import sys
 import traceback
-from typing import IO, Any, Callable, Dict, Iterable, Mapping, Optional, TypeVar, Union
+from typing import (
+    IO,
+    TYPE_CHECKING,
+    Any,
+    Callable,
+    Dict,
+    Iterable,
+    List,
+    Mapping,
+    Optional,
+    TypeVar,
+    Union,
+)
 
 import jmespath
 
@@ -86,14 +98,16 @@ class Logger(logging.Logger):  # lgtm [py/missing-call-to-init]
     Parameters propagated to LambdaPowertoolsFormatter
     --------------------------------------------------
     datefmt: str, optional
-        String directives (strftime) to format log timestamp using `time`, by default it uses RFC
-        3339.
+        String directives (strftime) to format log timestamp using `time`, by default it uses 2021-05-03 11:47:12,494+0200. # noqa: E501
     use_datetime_directive: bool, optional
         Interpret `datefmt` as a format string for `datetime.datetime.strftime`, rather than
         `time.strftime`.
 
         See https://docs.python.org/3/library/datetime.html#strftime-strptime-behavior . This
         also supports a custom %F directive for milliseconds.
+    use_rfc3339: bool, optional
+        Whether to use a popular date format that complies with both RFC3339 and ISO8601.
+        e.g., 2022-10-27T16:27:43.738+02:00.
     json_serializer : Callable, optional
         function to serialize `obj` to a JSON formatted `str`, by default json.dumps
     json_deserializer : Callable, optional
@@ -187,6 +201,14 @@ def __init__(
         stream: Optional[IO[str]] = None,
         logger_formatter: Optional[PowertoolsFormatter] = None,
         logger_handler: Optional[logging.Handler] = None,
+        json_serializer: Optional[Callable[[Dict], str]] = None,
+        json_deserializer: Optional[Callable[[Union[Dict, str, bool, int, float]], str]] = None,
+        json_default: Optional[Callable[[Any], Any]] = None,
+        datefmt: Optional[str] = None,
+        use_datetime_directive: bool = False,
+        log_record_order: Optional[List[str]] = None,
+        utc: bool = False,
+        use_rfc3339: bool = False,
         **kwargs,
     ):
         self.service = resolve_env_var_choice(
@@ -205,12 +227,29 @@ def __init__(
         self._default_log_keys = {"service": self.service, "sampling_rate": self.sampling_rate}
         self._logger = self._get_logger()
 
-        self._init_logger(**kwargs)
-
-    def __getattr__(self, name):
-        # Proxy attributes not found to actual logger to support backward compatibility
-        # https://github.com/awslabs/aws-lambda-powertools-python/issues/97
-        return getattr(self._logger, name)
+        # NOTE: This is primarily to improve UX, so IDEs can autocomplete LambdaPowertoolsFormatter options
+        # previously, we masked all of them as kwargs thus limiting feature discovery
+        formatter_options = {
+            "json_serializer": json_serializer,
+            "json_deserializer": json_deserializer,
+            "json_default": json_default,
+            "datefmt": datefmt,
+            "use_datetime_directive": use_datetime_directive,
+            "log_record_order": log_record_order,
+            "utc": utc,
+            "use_rfc3339": use_rfc3339,
+        }
+
+        self._init_logger(formatter_options=formatter_options, **kwargs)
+
+    # Prevent __getattr__ from shielding unknown attribute errors in type checkers
+    # https://github.com/awslabs/aws-lambda-powertools-python/issues/1660
+    if not TYPE_CHECKING:
+
+        def __getattr__(self, name):
+            # Proxy attributes not found to actual logger to support backward compatibility
+            # https://github.com/awslabs/aws-lambda-powertools-python/issues/97
+            return getattr(self._logger, name)
 
     def _get_logger(self):
         """Returns a Logger named {self.service}, or {self.service.filename} for child loggers"""
@@ -220,7 +259,7 @@ def _get_logger(self):
 
         return logging.getLogger(logger_name)
 
-    def _init_logger(self, **kwargs):
+    def _init_logger(self, formatter_options: Optional[Dict] = None, **kwargs):
         """Configures new logger"""
 
         # Skip configuration if it's a child logger or a pre-configured logger
@@ -235,7 +274,7 @@ def _init_logger(self, **kwargs):
         self._configure_sampling()
         self._logger.setLevel(self.log_level)
         self._logger.addHandler(self.logger_handler)
-        self.structure_logs(**kwargs)
+        self.structure_logs(formatter_options=formatter_options, **kwargs)
 
         # Maintenance: We can drop this upon Py3.7 EOL. It's a backport for "location" key to work
         self._logger.findCaller = self.findCaller
@@ -501,19 +540,23 @@ def registered_formatter(self) -> BasePowertoolsFormatter:
         """Convenience property to access logger formatter"""
         return self.registered_handler.formatter  # type: ignore
 
-    def structure_logs(self, append: bool = False, **keys):
+    def structure_logs(self, append: bool = False, formatter_options: Optional[Dict] = None, **keys):
         """Sets logging formatting to JSON.
 
         Optionally, it can append keyword arguments
-        to an existing logger so it is available across future log statements.
+        to an existing logger, so it is available across future log statements.
 
         Last keyword argument and value wins if duplicated.
 
         Parameters
         ----------
         append : bool, optional
             append keys provided to logger formatter, by default False
+        formatter_options : dict, optional
+            LambdaPowertoolsFormatter options to be propagated, by default {}
         """
+        formatter_options = formatter_options or {}
+
         # There are 3 operational modes for this method
         ## 1. Register a Powertools Formatter for the first time
         ## 2. Append new keys to the current logger formatter; deprecated in favour of append_keys
@@ -523,7 +566,7 @@ def structure_logs(self, append: bool = False, **keys):
         log_keys = {**self._default_log_keys, **keys}
         is_logger_preconfigured = getattr(self._logger, "init", False)
         if not is_logger_preconfigured:
-            formatter = self.logger_formatter or LambdaPowertoolsFormatter(**log_keys)  # type: ignore
+            formatter = self.logger_formatter or LambdaPowertoolsFormatter(**formatter_options, **log_keys)  # type: ignore # noqa: E501
             self.registered_handler.setFormatter(formatter)
 
             # when using a custom Lambda Powertools Formatter

aws_lambda_powertools/metrics/__init__.py

@@ -3,10 +3,11 @@
 from .base import MetricUnit
 from .exceptions import MetricUnitError, MetricValueError, SchemaValidationError
 from .metric import single_metric
-from .metrics import Metrics
+from .metrics import EphemeralMetrics, Metrics
 
 __all__ = [
     "Metrics",
+    "EphemeralMetrics",
     "single_metric",
     "MetricUnit",
     "MetricUnitError",