Document issues with podman/selinux

[mjlbach]

Feature Description

I run Fedora/CoreOS and as such was following along the docker rootless tutorial using podman.

After following the section on enabling ssh passthrough, it will silently fail due to selinux permission issues.

If you try to clone a git repo:

❯ git clone git@git.test.com:mjlbach/test.git
Cloning into 'test'...
git@git.test.com: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

As seen in systemd-journal with sudo journalctl -feu sshd:

Mar 21 17:55:17 nomad sshd[9212]: error: AuthorizedKeysCommand execve "/usr/bin/podman exec -i systemd-gitea /usr/local/bin/gitea keys -c /etc/gitea/app.ini -e git -u git -t ssh-ed25519 -k REDACTED": Permission denied
Mar 21 17:55:17 nomad sshd[9210]: AuthorizedKeysCommand /usr/bin/podman exec -i systemd-gitea /usr/local/bin/gitea keys -c /etc/gitea/app.ini -e git -u git -t ssh-ed25519 -k REDACTED failed, status 127
Mar 21 17:55:17 nomad sshd[9210]: Connection closed by authenticating user git REDACTED port 51284 [preauth]

The underlying culprit can be viewed in the SELinux logs:

❯ sudo ausearch -c 'sshd' --raw
type=AVC msg=audit(1679432260.930:359): avc:  denied  { execute } for  pid=2081 comm="sshd" name="podman" dev="vda4" ino=36775453 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_runtime_exec_t:s0 tclass=file permissive=0
type=AVC msg=audit(1679432388.316:426): avc:  denied  { execute } for  pid=2305 comm="sshd" name="podman" dev="vda4" ino=36775453 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_runtime_exec_t:s0 tclass=file permissive=1
type=AVC msg=audit(1679432388.316:427): avc:  denied  { read open } for  pid=2305 comm="sshd" path="/usr/bin/podman" dev="vda4" ino=36775453 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_runtime_exec_t:s0 tclass=file permissive=1
type=AVC msg=audit(1679432388.316:428): avc:  denied  { execute_no_trans } for  pid=2305 comm="sshd" path="/usr/bin/podman" dev="vda4" ino=36775453 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_runtime_exec_t:s0 tclass=file permissive=1
type=AVC msg=audit(1679432821.749:329): avc:  denied  { execute } for  pid=2005 comm="sshd" name="podman" dev="vda4" ino=36775453 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_runtime_exec_t:s0 tclass=file permissive=0
type=AVC msg=audit(1679432921.520:370): avc:  denied  { read open } for  pid=2119 comm="sshd" path="/usr/bin/podman" dev="vda4" ino=36775453 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_runtime_exec_t:s0 tclass=file permissive=0
type=AVC msg=audit(1679432945.018:239): avc:  denied  { read open } for  pid=1374 comm="sshd" path="/usr/bin/podman" dev="vda4" ino=36775453 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_runtime_exec_t:s0 tclass=file permissive=0
type=AVC msg=audit(1679433004.809:339): avc:  denied  { read open } for  pid=1831 comm="sshd" path="/usr/bin/podman" dev="vda4" ino=36775453 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_runtime_exec_t:s0 tclass=file permissive=0
type=AVC msg=audit(1679433022.644:374): avc:  denied  { read open } for  pid=2034 comm="sshd" path="/usr/bin/podman" dev="vda4" ino=36775453 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_runtime_exec_t:s0 tclass=file permissive=0
type=AVC msg=audit(1679433191.667:323): avc:  denied  { execute_no_trans } for  pid=1980 comm="sshd" path="/usr/bin/podman" dev="vda4" ino=36775453 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_runtime_exec_t:s0 tclass=file permissive=0
type=AVC msg=audit(1679446517.354:8485): avc:  denied  { execute } for  pid=9212 comm="sshd" name="podman" dev="vda4" ino=36775453 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_runtime_exec_t:s0 tclass=file permissive=0
type=AVC msg=audit(1679446521.233:8497): avc:  denied  { execute } for  pid=9223 comm="sshd" name="podman" dev="vda4" ino=36775453 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_runtime_exec_t:s0 tclass=file permissive=0
type=AVC msg=audit(1679448352.006:8605): avc:  denied  { execute } for  pid=9973 comm="sshd" name="podman" dev="vda4" ino=36775453 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_runtime_exec_t:s0 tclass=file permissive=0

I'm opening this issue for visibility incase others attempt to use rootless podman with ssh forwarding. I've also opened an issue with podman: containers/podman#17883. I'm trying to debug this/come up with a good workaround (or installable selinux policy). Feel free to close, but I'll also document here how to use gitea/podman/coreos once I figure it out.

[techknowlogick]

Thanks for tracing through all of this :)

note: I tagged this as theme/docker as it was the closest

[mjlbach]

Of course! I realize podman is not officially supported (and selinux is its own pain) but I figured this was the first place people would search for issues, so if I can document a solution somewhere hopefully it is useful :)

[mjlbach]

One workaround is to set sshd_t to permissive by sudo semanage permissive -a sshd_t

[mjlbach]

@techknowlogick

This appears to work.

1. Ensure that selinux-policy-devel and container-selinux are installed
2. Create the following gitea-sshd.te

policy_module(gitea-sshd,0.1)

gen_require(`
        type sshd_t;
	type iptables_var_run_t;
	type systemd_logind_t;
	class dir read;
')

container_runtime_domtrans(sshd_t)
allow systemd_logind_t iptables_var_run_t:dir read;

make -f /usr/share/selinux/devel/Makefile

sudo semodule -i `gitea-sshd.pp`

I'm not sure how to move forward with this, do we distribute rpm's with the appropriate selinux policy for the host? Do we just include those instructions in an "WARNING: SELINUX INSTRUCTIONS" somewhere in the gitea docs? Does it live in podman (containers/podman#17883 (comment) cc @rhatdan)

[tmuncks]

At least on the RHEL family of distributions, there's a builtin selinux bool that allows this exact thing:

semanage boolean --modify --on sshd_launch_containers

Works here on Alma Linux 9.