Failed in-place upgrade to postgres 15

[davide-neen]

• Which image of the operator are you using? registry.opensource.zalan.do/acid/postgres-operator:v1.9.0
• Where do you run it - cloud or metal? Kubernetes or OpenShift? Bare Metal K8s
• Are you running Postgres Operator in production? No
• Type of issue? Bug report

Hi,

I'm testing the in-place upgrade function from several versions and found that it does not upgrade to version 15, with 1.9.0 operator.

I trigger the upgrade by changing the version in the CRD and then I check the version by doing a select version from the master pod.

It worked for example from 13 to 14, or from 12 to 13, or even directly from 12 to 14, but if I try to run the upgrade from any version to 15 fails, showing no interesting log details

Operator log:

time="2023-03-07T16:25:38Z" level=error msg="major version upgrade failed: could not execute: command terminated with exit code 1" cluster-name=kleecks/kleecks-milano-p9dr-postgresql pkg=cluster

No problem performing the same operation with other versions

Thank you

[FxKu]

Are you using timescale? The default Spilo docker image of 1.9.0 operator does not yet support it. Pg 15 support by timescale was only release last week.

[davide-neen]

Hi, no we don't use timescale

[FxKu]

You can try to log into the pod and try to run the script manually: python3 /scripts/inplace_upgrade.py <number of cluster members>. If it fails the script output should point you to a system log files that you can inspect.

[tech1ndex]

Hi there,

Facing the same issue, output from manually attempting to run the update shows the following:

2023-10-11 12:51:44,261 inplace_upgrade INFO: No PostgreSQL configuration items changed, nothing to reload.
Traceback (most recent call last):
  File "/scripts/inplace_upgrade.py", line 780, in <module>
    sys.exit(main())
  File "/scripts/inplace_upgrade.py", line 772, in main
    upgrade = InplaceUpgrade(config)
  File "/scripts/inplace_upgrade.py", line 95, in __init__
    self.desired_version = self.get_desired_version()
  File "/scripts/inplace_upgrade.py", line 123, in get_desired_version
    return get_binary_version(bin_dir)
  File "/scripts/spilo_commons.py", line 54, in get_binary_version
    version = subprocess.check_output([postgres, '--version']).decode()
  File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 423, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/lib/postgresql/15/bin/postgres': '/usr/lib/postgresql/15/bin/postgres'

When I check the contents of /usr/lib/postgresql there is indeed no "15" folder which is confusing to me since we are on v1.9.0 of the operator which I though supports PG Versions 10-15 according to this chart ?

[tech1ndex]

Actually, was able to resolve this, it looks like in my case the Operator was still referencing a spilo-14 image that had been overridden in the ConfigMap. Updating this to a spillo-15 docker image resolved the issue and allowed it to progress. Posting here in case it is helpful for anyone else.