upload WAL files failure S3ResponseError: 400 Bad Request - The authorization header is malformed; the region 'loca' is wrong; expecting 'us-east-1'

[zagr0]

• Which image of the operator are you using?
  registry.opensource.zalan.do/acid/postgres-operator:v1.7.0
• **Where do you run it - cloud or metal?
  AWS K8s
• Are you running Postgres Operator in production?
  yes
• Type of issue?
  Bug report

After 1.6.0 -> 1.7.0 migration we have found 3 of our clusters with the same issue in logs. WAL files were not able to be uploaded to S3 with the boto error. That caused to produce huge amount of logs, that is how we detected it actually.
Its not clear what is the issue but after pod delete/recreation there is no error happens.

logs:

        DETAIL: Uploading "pg_wal/00000003000000000000007D" to "s3://postgres-operator-backups/spilo/stage-short-url/7950e847-3c25-4015-a4dd-4146dc47b29e/wal/13/wal_005/00000003000000000000007D.lzo".
        STRUCTURED: time=2021-09-03T11:25:52.368449-00 pid=362481 action=push-wal key=s3://postgres-operator-backups/spilo/stage-short-url/7950e847-3c25-4015-a4dd-4146dc47b29e/wal/13/wal_005/00000003000000000000007D.lzo prefix=spilo/stage-short-url/7950e847-3c25-4015-a4dd-4146dc47b29e/wal/13/ seg=00000003000000000000007D state=begin
wal_e.worker.upload INFO     MSG: begin archiving a file
        DETAIL: Uploading "pg_wal/00000003000000000000007E" to "s3://postgres-operator-backups/spilo/stage-short-url/7950e847-3c25-4015-a4dd-4146dc47b29e/wal/13/wal_005/00000003000000000000007E.lzo".
        STRUCTURED: time=2021-09-03T11:25:52.371034-00 pid=362481 action=push-wal key=s3://postgres-operator-backups/spilo/stage-short-url/7950e847-3c25-4015-a4dd-4146dc47b29e/wal/13/wal_005/00000003000000000000007E.lzo prefix=spilo/stage-short-url/7950e847-3c25-4015-a4dd-4146dc47b29e/wal/13/ seg=00000003000000000000007E state=begin
wal_e.worker.upload INFO     MSG: begin archiving a file
        DETAIL: Uploading "pg_wal/00000003000000000000007F" to "s3://postgres-operator-backups/spilo/stage-short-url/7950e847-3c25-4015-a4dd-4146dc47b29e/wal/13/wal_005/00000003000000000000007F.lzo".
        STRUCTURED: time=2021-09-03T11:25:52.373588-00 pid=362481 action=push-wal key=s3://postgres-operator-backups/spilo/stage-short-url/7950e847-3c25-4015-a4dd-4146dc47b29e/wal/13/wal_005/00000003000000000000007F.lzo prefix=spilo/stage-short-url/7950e847-3c25-4015-a4dd-4146dc47b29e/wal/13/ seg=00000003000000000000007F state=begin
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/gevent/greenlet.py", line 536, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.6/dist-packages/wal_e/worker/upload.py", line 53, in __call__
    self.gpg_key_id)
  File "/usr/local/lib/python3.6/dist-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
    k = blobstore.uri_put_file(creds, url, tf)
  File "/usr/local/lib/python3.6/dist-packages/wal_e/blobstore/s3/s3_util.py", line 57, in uri_put_file
    k.set_contents_from_file(fp, encrypt_key=True)
  File "/usr/lib/python3/dist-packages/boto/s3/key.py", line 1293, in set_contents_from_file
    chunked_transfer=chunked_transfer, size=size)
  File "/usr/lib/python3/dist-packages/boto/s3/key.py", line 750, in send_file
    chunked_transfer=chunked_transfer, size=size)
  File "/usr/lib/python3/dist-packages/boto/s3/key.py", line 951, in _send_file_internal
    query_args=query_args
  File "/usr/lib/python3/dist-packages/boto/s3/connection.py", line 668, in make_request
    retry_handler=retry_handler
  File "/usr/lib/python3/dist-packages/boto/connection.py", line 1072, in make_request
    retry_handler=retry_handler)
  File "/usr/lib/python3/dist-packages/boto/connection.py", line 941, in _mexe
    request.body, request.headers)
  File "/usr/lib/python3/dist-packages/boto/s3/key.py", line 884, in sender
    response.status, response.reason, body)
boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region 'loca' is wrong; expecting 'us-east-1'</Message><Region>us-east-1</Region><RequestId>ZVVNPYJ1HR48WFRH</RequestId><HostId>pODb4r+zRjQPzpx6bPzL0RNbAbCWrHOI3ikDAnd7NlmGzKO4mHHwqjKn1hjaiuRlLtSRJuBe0FA=</HostId></Error>
Fri Sep  3 11:25:52 2021 <Greenlet at 0x7f6320f8bcc0: <wal_e.worker.upload.WalUploader object at 0x7f6320ef94a8>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with S3ResponseError

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/gevent/greenlet.py", line 536, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.6/dist-packages/wal_e/worker/upload.py", line 53, in __call__
    self.gpg_key_id)
  File "/usr/local/lib/python3.6/dist-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
    k = blobstore.uri_put_file(creds, url, tf)
  File "/usr/local/lib/python3.6/dist-packages/wal_e/blobstore/s3/s3_util.py", line 57, in uri_put_file
    k.set_contents_from_file(fp, encrypt_key=True)
  File "/usr/lib/python3/dist-packages/boto/s3/key.py", line 1293, in set_contents_from_file
    chunked_transfer=chunked_transfer, size=size)
  File "/usr/lib/python3/dist-packages/boto/s3/key.py", line 750, in send_file
    chunked_transfer=chunked_transfer, size=size)
  File "/usr/lib/python3/dist-packages/boto/s3/key.py", line 951, in _send_file_internal
    query_args=query_args
  File "/usr/lib/python3/dist-packages/boto/s3/connection.py", line 668, in make_request
    retry_handler=retry_handler
  File "/usr/lib/python3/dist-packages/boto/connection.py", line 1072, in make_request
    retry_handler=retry_handler)
  File "/usr/lib/python3/dist-packages/boto/connection.py", line 941, in _mexe
    request.body, request.headers)
  File "/usr/lib/python3/dist-packages/boto/s3/key.py", line 884, in sender
    response.status, response.reason, body)
boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region 'loca' is wrong; expecting 'us-east-1'</Message><Region>us-east-1</Region><RequestId>ZVVRX3NHHT33B7J9</RequestId><HostId>Cd6anHiTnNWbKG7yMYBCauyRD8fXCTTAebipwiaeXb6/htN0RM5cMy7UQ/+TR5cfXO2tF7JjXLM=</HostId></Error>
Fri Sep  3 11:25:52 2021 <Greenlet at 0x7f6320f8bb90: <wal_e.worker.upload.WalUploader object at 0x7f6320ef94a8>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with S3ResponseError

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/gevent/greenlet.py", line 536, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.6/dist-packages/wal_e/worker/upload.py", line 53, in __call__
    self.gpg_key_id)
  File "/usr/local/lib/python3.6/dist-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
    k = blobstore.uri_put_file(creds, url, tf)
  File "/usr/local/lib/python3.6/dist-packages/wal_e/blobstore/s3/s3_util.py", line 57, in uri_put_file
    k.set_contents_from_file(fp, encrypt_key=True)
  File "/usr/lib/python3/dist-packages/boto/s3/key.py", line 1293, in set_contents_from_file
    chunked_transfer=chunked_transfer, size=size)
  File "/usr/lib/python3/dist-packages/boto/s3/key.py", line 750, in send_file
    chunked_transfer=chunked_transfer, size=size)
  File "/usr/lib/python3/dist-packages/boto/s3/key.py", line 951, in _send_file_internal
    query_args=query_args
  File "/usr/lib/python3/dist-packages/boto/s3/connection.py", line 668, in make_request
    retry_handler=retry_handler
  File "/usr/lib/python3/dist-packages/boto/connection.py", line 1072, in make_request
    retry_handler=retry_handler)
  File "/usr/lib/python3/dist-packages/boto/connection.py", line 941, in _mexe
    request.body, request.headers)
  File "/usr/lib/python3/dist-packages/boto/s3/key.py", line 884, in sender
    response.status, response.reason, body)
boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request

[dimitrigraf]

From the logs: ...the region 'loca' is wrong; expecting 'us-east-1'...
Might there be a typo somewhere? Or misconfiguration in the operator config and/or extra pod env config map?

[FxKu]

Sounds like the same problem like in #761
@zagr0 please, check how you have configured the AWS region. We learned from #1577 that the region isn't passed as environment variable. You have to specify CLONE_AWS_REGION in an extra configmap/secret, referenced in the config by pod_environment_configmap (pod_environment_secret) option.

[zagr0]

We are providing wal s3 region configuration with operator's helm values:

configAwsOrGcp:
  aws_region: "us-east-1"
  wal_s3_bucket: "postgres-operator-backups-bucket-name"

in addition to that pod_environment_secret contains access keys:

apiVersion: v1
kind: Secret
metadata:
  name: postgres-pod-secret
  namespace: postgres-clusters
stringData:
  BACKUP_SCHEDULE: "30 00 * * *"
  AWS_ACCESS_KEY_ID: "****"
  AWS_SECRET_ACCESS_KEY: "****"

Just to clarify that issue was faced after operator upgrade activity 1.6.0 -> 1.7.0, there were no any configuration changes during the procedure. As expected after upgrade there was rolling upgrade for all the postgres clusters managed by the operator and for some of them (3 of 20) we face this issue

[FxKu]

Interesting that everything worked before. Anyway, I think to make cloning work everywhere you have to specify the following variables in your postgres_pod_secret:

  BACKUP_SCHEDULE: "30 00 * * *"
  AWS_REGION: "us-east-1"
  AWS_ACCESS_KEY_ID: "****"
  AWS_SECRET_ACCESS_KEY: "****"
  CLONE_AWS_REGION: "us-east-1"
  CLONE_AWS_ACCESS_KEY_ID: "****"
  CLONE_AWS_SECRET_ACCESS_KEY: "****"

Do you see this only in 3 clusters because they tried to restore from S3?

[zagr0]

Its pretty hard to say right now, but most probably something goes different for them during rolling update
after I delete the pods to be recreated, everything became ok and didn't reproduced

"internal cluster clonning" when you scale cluster works well without this variables, I thought that they are required only in case you trigger clone with spec, according to https://github.com/zalando/postgres-operator/blob/master/docs/user.md#clone-from-s3

spec:
  clone:
     ...