systemd example doesn't work anymore

[dothebart]

over here:

https://github.com/docker-library/docs/tree/master/centos#dockerfile-for-systemd-base-image

describes how to build an image which runs systemd as PID1.
However, while a container built that way will launch, /run/ will be missing the sockets for dbus, and

systemctl enable bla
Failed to get D-Bus connection: No such file or directory

hence one can't talk to systemd, and not make it launch services etc.

This used to be working until some when last week. This similar Ubuntu container still works: https://github.com/arangodb/release-test-automation/blob/main/containers/docker_deb/Dockerfile

I've tried https://fedoraproject.org/wiki/How_to_debug_Systemd_problems
and https://freedesktop.org/wiki/Software/systemd/Debugging/ but I can't get the systemd output - should one be able to see it with docker logs if --log_target=console or --systemd.log_target=console?

[tianon]

cc centos image maintainer, @bstinsonmhk 🙏 😅

(Is this still a "supported" thing? Perhaps that section should be removed?)

[dothebart]

hm, when running the container interactive, one has some feedback. Shouldn't these also be seen in docker logs ?

docker run -it --ulimit core=-1 --privileged -v /sys/fs/cgroup:/sys/fs/cgroup:rw -v /tmp/tmp:/tmp/  blarg /lib/systemd/systemd --system --unit=multiuser.target 
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization docker.
Detected architecture x86-64.

Welcome to CentOS Linux 7 (Core)!

Set hostname to <c5e62f13ab1d>.
Initializing machine ID from random generator.
Cannot determine cgroup we are running in: No such file or directory
Failed to allocate manager object: No such file or directory
[!!!!!!] Failed to allocate manager object, freezing.

So at least now I have some feedback from systemd what to continue googling.

Similar with Centos-8:

docker run -it --ulimit core=-1 --privileged -v /sys/fs/cgroup:/sys/fs/cgroup:rw -v /tmp/tmp:/tmp/  blarg /lib/systemd/systemd --system --unit=multiuser.target 
systemd 239 (239-41.el8_3) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy)
Detected virtualization docker.
Detected architecture x86-64.

Welcome to CentOS Linux 8!

Set hostname to <cf2f71b8125c>.
Initializing machine ID from random generator.
Couldn't move remaining userspace processes, ignoring: Input/output error
Failed to determine supported controllers: No such file or directory
Failed to allocate manager object: No such file or directory
[!!!!!!] Failed to allocate manager object, freezing.
Freezing execution.

On another machine the centos8 version seems to be working:

willi@c7:~$ docker run -it --ulimit core=-1 --privileged -v /sys/fs/cgroup:/sys/fs/cgroup:rw -v /home/willi/src/release-test-automation/package_cache/:/home/package_cache -v /tmp/tmp:/tmp/ -e UNIFIED_CGROUP_HIERARCHY=no -e systemd.log_target=console -e systemd.log_level=debug --rm blarg /lib/systemd/systemd --system --unit=multiuser.target --systemd.log_level=debug --systemd.log_target=console 
systemd 239 (239-41.el8_3) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy)
Detected virtualization docker.
Detected architecture x86-64.

Welcome to CentOS Linux 8!

Set hostname to <145d48ef9a30>.
Initializing machine ID from random generator.
Couldn't move remaining userspace processes, ignoring: Input/output error
Unit multiuser.target not found.
Falling back to rescue target: rescue.target
[  OK  ] Reached target Local File Systems.
         Starting Create Volatile Files and Directories...
[  OK  ] Reached target Swap.
systemd-tmpfiles-setup.service: Failed to connect stdout to the journal socket, ignoring: No such file or directory
[  OK  ] Started Create Volatile Files and Directories.
[ INFO ] Update UTMP about System Boot/Shutdown is not active.
[DEPEND] Dependency failed for Update UTMP about System Runlevel Changes.
systemd-update-utmp-runlevel.service: Job systemd-update-utmp-runlevel.service/start failed with result 'dependency'.
[  OK  ] Reached target System Initialization.
[  OK  ] Started Rescue Shell.
[  OK  ] Reached target Rescue Mode.
Startup finished in 75ms.
rescue.service: Executable /bin/plymouth missing, skipping: No such file or directory
You are in rescue mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" or "exit"
to boot into default mode.

Cannot open access to console, the root account is locked.
See sulogin(8) man page for more details.

Press Enter to continue.

[dothebart]

Quoting https://systemd.io/CONTAINER_INTERFACE/ -

Note that it is our intention to make systemd systems work flawlessly and out-of-the-box in containers. In fact we are interested to ensure that the same OS image can be booted on a bare system, in a VM and in a container, and behave correctly each time. If you notice that some component in systemd does not work in a container as it should, even though the container manager implements everything documented above, please contact us.

and reading the rest of the page it seems that there still seems to be some manual configuration to work with.
Instead of mounting /sys readonly, the udev services are removed, which seems to be all against systemd's policy.

[iMacTia]

@dothebart did you manage to solve this by any chance?

[dothebart]

It seems that it fails with newer kernels. Upstream systemd says: docker does it all wrong:
systemd/systemd#19504
It seems to work with other containerizers.

[iMacTia]

For future readers, I was trying to run Ansible playbooks with the service module against the amazonlinux Docker image.
The solution for me was to use this systemctl replacement and explicitly tell the service module to use systemd as the auto-detection was failing.

[tianon]

@bstinsonmhk :pray: is this example still relevant? Should it be updated or removed?

[dothebart]

From my POV yes! Our @arangodb CI leans on this for nightly package testing.

[tianon]

Closing given #2205 🙈 ❤️