Infrastructure roles with configmap is underdocumented and overly complex

[mjpieters]

• Which image of the operator are you using?: registry.opensource.zalan.do/acid/postgres-operator:v1.7.0
• Where do you run it: Bare Metal K8s
• Are you running Postgres Operator in production? yes
• Type of issue? Bug report

I had to go diving through the source code to figure out how to create an infrastructure role definition that has the NOINHERIT flag set. This is only possible when using the Secrets + ConfigMap combo, and only if you have template: true set or if you use infrastructure_roles_secret_name instead of infrastructure_roles_secrets.

There are several issues here:

• The documentation on this option is lacking, and the manifests/ examples incomplete. The documentation links to the manifest files are broken (https://postgres-operator.readthedocs.io/en/manifests/infrastructure-roles.yaml and <https://postgres-operator.readthedocs.io/en/manifests/infrastructure-roles-configmap.yaml) lead to 404s, instead of the manifest files in the repo.
• The code handling the Secrets + Configmap option is only reached if a given infrastructure role is templated. The code handling the older infrastructure_roles_secret_name option re-uses the codepath for infrastructure_roles_secrets by passing in default values for userkey: user, passwordkey: password and rolekey: inrole and sets template: true to bypass the code that would otherwise exit early if userkey and passwordkey do not lead to an entry in the secrets map. This means that you must not have user1 and password1 keys in your secret if you did not intend to create those users when you expected to use a configmap approach.
• If you do reach the section handling Secrets + ConfigMap option, then this creates entirely new roles from each key-value pair in the secret, so you can't mix the styles here (you'd get <userkey> roles with what was intended as the role name as the password, this could lead to security breaches).
• It is not made clear that the entries in the ConfigMap must exist and are used as the defining factor to create roles this way. You can't just have unnumbered username: password entries in the secret
• There appears to be undocumented support for a details key in the infrastructure_roles_secrets list, but if you use that option then the userkey, passwordkey and rolekey options are ignored, and the way the PgUser object definition works you can't set the role name or password fields so this is guaranteed to lead to an invalid role definition so is entirely broken and useless.

This is.. crazy. In order to use multiple named secret resources, where the roles have additional flags set in a configmap, I must use:

     infrastructure_roles_secrets:
       - secretname: <name of one set of secrets and configmap resources>
          template: true
          userkey: dummy
          passwordkey: dummy
       - secretname: <name of another set of secrets and configmap resources>
          template: true
          userkey: dummy
          passwordkey: dummy

and you can only use <rolename>: <password> in the secrets definition, and a configmap with literal yaml strings to define the rest.

Could this either be made explicit in the documentation (so, state that this only works with either infrastructure_roles_secret_name or with template: true), or be refactored to allow for the Secrets still to use userkey, passworkey, roleskey options but where you can define additional details in either the secret or the infrastructure_roles_secrets map?

[daku]

In addition to the above, there are further complications while installing via helm. The infrastructure_roles_secrets parameter is not configurable when installing via helm. This means we must use the old approach of setting infrastructure_roles_secret_name only.

However, this still works because as explained above, template is set to true for this secret. See the code here.

Its important then in the secret not to set user1, password1, etc. so that the code to read the ConfigMap executes immediately after because of the break statement just earlier in the code.

Finally, I still could not create the postgres user due to this error I saw in the postgres-operator logs.

root@quiche:~# kubectl logs -f postgres-operator-85548d8565-gx9bl
...
time="2023-10-19T03:11:56Z" level=debug msg="cannot get infrastructure role: {SecretName:postgres-operator/postgresql-infrastructure-roles UserKey:user PasswordKey:password RoleKey:inrole DefaultUserValue: DefaultRoleValue: Details: Template:true}" pkg=controller
time="2023-10-19T03:11:56Z" level=warning msg="could not get infrastructure roles: could not get infrastructure roles secret postgres-operator/postgresql-infrastructure-roles: secrets \"postgresql-infrastructure-roles\" not found" pkg=controller
...

The secret and configmap must be in the same namespace as the postgres operator. It also must be created after installing the operator and before creating the cluster.

Here are my relevant k8s resource files.

The installation via helm (I use ansible):

- name: Install zalando postgres operator via helm
  kubernetes.core.helm:
    name: postgres-operator
    chart_ref: postgres-operator-charts/postgres-operator
    release_namespace: postgres-operator
    create_namespace: true
    values:
      configKubernetes:
        infrastructure_roles_secret_name: postgresql-infrastructure-roles

The secret. Infrastructure role is boss. Here too its ansible k8s:

- name: Create postgres infrastructure role secrets
  kubernetes.core.k8s:
    definition:
      apiVersion: v1
      kind: Secret
      metadata:
        name: postgresql-infrastructure-roles
        # must be in the same namespace as operator
        namespace: postgres-operator
      data:
        boss: "{{ secrets.postgresBoss | b64encode }}"

Finally, the configmap:

- name: Assign infrstructure roles the permissions
  kubernetes.core.k8s:
    definition:
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: postgresql-infrastructure-roles
        namespace: postgres-operator
      data:
        boss: |
          user_flags:
            - createdb

I hope this helps fix code/documentation/helm chart or helps others make this work.