embedded-postgres on Apple Silicon (M1) issue

[drej1]

Hi there,

thanks for the great work in the first place, we were using embedded Postgres for a while now.

Now I switched to a MacBook Pro with the new M1 SoC which has arm64 architecture.
I debugged DefaultPostgresBinaryResolver#getPgBinary and the line

Resource resource = findPgBinary(normalize(format("postgres-%s-%s.txz", system, architecture)));

does not find any binary because it is searching for postgres-darwin-arm_64.txzwhich I didn't find in https://mvnrepository.com/artifact/io.zonky.test.postgres

I assume there will be no release of Postgres for Darwin arm64 soon so is there a way to load the Darwin-amd64 embedded Postgres?

Dependencies we use:

compile("io.zonky.test:embedded-database-spring-test:1.6.1")
dependencyManagement {
        imports {
            mavenBom 'io.zonky.test.postgres:embedded-postgres-binaries-bom:9.6.16'
        }
    }

Thanks

[drej1]

@tomix26 ? :)

[tomix26]

Hi @drej1, thanks for the report. The mac os binaries are actually EnterpriseDB binaries but in reduced and repacked form. So now we are waiting for EnterpriseDB to release binaries for M1 architecture.

[drej1]

@tomix26 thanks for the info. Is there any roadmap on EnterpriseDB for supporting M1? If not, won't there be any workaround to support M1 in the meantime?
I did a local workaround for our project. It's very primitive :D
I duplicated io.zonky.test.db.postgres.embedded.DefaultPostgresBinaryResolver and the only change is in #getPgBinary as follows:

@Override
public InputStream getPgBinary(String system, String machineHardware) throws IOException {
    String architecture = ArchUtils.normalize(machineHardware);
    String distribution = LinuxUtils.getDistributionName();

    log.info("Detected distribution: '{}'", Optional.ofNullable(distribution).orElse("Unknown"));

    if (distribution != null) {
        Resource resource = findPgBinary(normalize(format("postgres-%s-%s-%s.txz", system, architecture, distribution)));
        if (resource != null) {
            log.info("Distribution specific postgres binaries found: {}", resource.getFilename());
            return resource.getInputStream();
        } else {
            log.debug("Distribution specific postgres binaries not found");
        }
    }

    // START CHANGES
    if (system.equals("Darwin") && architecture.equals("arm_64")) {
        architecture = "x86_64";
    }
    // END CHANGES

    Resource resource = findPgBinary(normalize(format("postgres-%s-%s.txz", system, architecture)));
    if (resource != null) {
        log.info("System specific postgres binaries found: {}", resource.getFilename());
        return resource.getInputStream();
    }

    log.error("No postgres binaries were found, you must add an appropriate maven dependency "
              + "that meets the following parameters - system: {}, architecture: {}", system, architecture);
    throw new IllegalStateException("Missing postgres binaries");
}

This downloads and runs intel architecture based embedded DB which works fine on the M1.

[tomix26]

Unfortunately, I don't know about any roadmap. At least not yet 😞

Nevertheless, your solution seems great to me. Using rosetta to translate x86_64 binaries to arm64 architecture is good enough. It should also be possible to avoid implementing the custom resolver by setting the architecture via a system property: -Dos.arch=x86_64

[drej1]

Hmm it sounds interesting, I didn't know about this option. I will try to play with this. Other people on the team don't use M1 so I want to find some global solution for the project. Thanks for the tip.

I leave this open for the native support.

[okopok]

@tomix26 still didn't work, even with duplicated DefaultPostgresBinaryResolver

[jedvardsson]

@okopok Here is what I have working on my M1.

@Bean
@ConditionalOnProperty(name = "spring.datasource.embedded", havingValue = "true")
public EmbeddedPostgres embeddedPostgres() throws IOException {
    return EmbeddedPostgres.builder()
            .setPgBinaryResolver((system, machineHardware) -> {
                var arch = system.equals("Darwin") && machineHardware.equals("aarch64") ? "x86_64" : machineHardware;
                return DefaultPostgresBinaryResolver.INSTANCE.getPgBinary(system, arch);
            })
            .start();
}

[mrioan]

Taking the response from @jedvardsson and modifying it for my case:

builder.setPgBinaryResolver((system, machineHardware) -> {
                            String arch = system.equals("Darwin") && machineHardware.equals("aarch64") ? "x86_64" : machineHardware;
                            return EmbeddedPostgres.class.getResourceAsStream(format("/postgresql-%s-%s.txz", system, arch));
                        }).start();

I also had to do the following:

To solve: dyld: Library not loaded: @loader_path/../lib/libpq.5.dylib

brew update
brew link libpq --force

And to solve: This error usually means that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMALL parameter. You might need to reconfigure the kernel with larger SHMALL.

sudo vi /etc/sysctl.conf

and pasted:

kern.sysv.shmmax=4194304
kern.sysv.shmmin=1
kern.sysv.shmmni=10240
kern.sysv.shmseg=4096
kern.sysv.shmall=33554432

[tomix26]

I've just released version 1.3.0 that includes support for Rosetta 2 emulation out of the box. So it's no longer necessary to implement a custom binary resolver. Nevertheless, this is only a temporary solution and I'm still waiting for native binaries to become available.

[drej1]

Hi @tomix26 , how does it look with the native binaries? I have postgres installed via homebrew and it runs natively on Apple Silicon.

BTW the workaround works like a charm ;)

[jedvardsson]

@tomix26 What do you think about allowing the resolver to piggy-back on a locally installed Postgres or perhaps even a locally running Postgres.