So Podman is an open source container engine like Docker—with "full"1 Docker compatibility. IMO Podman’s main benefit over Docker is security. But how is it more secure? Keep reading…
Docker traditionally runs a daemon as the root user, and you need to mount that daemon’s socket into various containers for them to work as intended (See: Traefik, Portainer, etc.) But if someone compromises such a container and therefore gains access to the Docker socket, it’s game over for your host. That Docker socket is the keys to the root kingdom, so to speak.
Podman doesn’t have a daemon by default, although you can run a very minimal one for Docker compatibility. And perhaps more importantly, Podman can run entirely as a non-root user.2 Non-root means if someone compromises a container and somehow manages to break out of it, they don’t get the keys to the kingdom. They only get access to your non-privileged Unix user. So like the keys to a little room that only contains the thing they already compromised.2.5 Pretty neat.
Okay, now for the annoying parts of Podman. In order to achieve this rootless, daemonless nirvana, you have to give up the convenience of Unix users in your containers being the same as the users on the host. (Or at least the same UIDs.) That’s because Podman typically3 runs as a non-root user, and most containers expect to either run as root or some other specific user.
The "solution"4 is user re-mapping. Meaning that you can configure your non-root user that Podman is running as to map into the container as the root user! Or as UID 1234. Or really any mapping you can imagine. If that makes your head spin, wait until you actually try to configure it. It’s actually not so bad on containers that expect to run as root. You just map your non-root user to the container UID 0 (root)… and Bob’s your uncle. But it can get more complicated and annoying when you have to do more involved UID and GID mappings—and then play the resultant permissions whack-a-mole on the host because your volumes are no longer accessed from a container running as host-root…
Still, it’s a pretty cool feeling the first time you run a “root” container in your completely unprivileged Unix user and everything just works. (After spending hours of swearing and Duck-Ducking to get it to that point.) At least, it was pretty cool for me. If it’s not when you do it, then Podman may not be for you.
The other big annoying thing about Podman is that because there’s no Big Bad Daemon managing everything, there are certain things you give up. Like containers actually starting on boot. You’d think that’d be a fundamental feature of a container engine in 2023, but you’d be wrong. Podman doesn’t do that. Podman adheres to the “Unix philosophy.” Meaning, briefly, if Podman doesn’t feel like doing something, then it doesn’t. And therefore expects you to use systemd for starting your containers on boot. Which is all good and well in theory, until you realize that means Podman wants you to manage your containers entirely with systemd. So… running each container with a systemd service, using those services to stop/start/manage your containers, etc.
Which, if you ask me, is totally bananasland. I don’t know about you, but I don’t want to individually manage my containers with systemd. I want to use my good old trusty Docker Compose. The good news is you can use good old trusty Docker Compose with Podman! Just run a compatibility daemon (tiny and minimal and rootless… don’t you worry) to present a Docker-like socket to Compose and boom everything works. Except your containers still don’t actually start on boot. You still need systemd for that. But if you make systemd run Docker Compose, problem solved!
This isn’t the “Podman Way” though, and any real Podman user will be happy to tell you that. The Podman Way is either the aforementioned systemd-running-the-show approach or something called Quadlet or even a Kubernetes compatibility feature. Briefly, about those: Quadlet is “just” a tighter integration between systemd and Podman so that you can declaratively define Podman containers and volumes directly in a sort of systemd service file. (Well, multiple.) It’s like Podman and Docker Compose and systemd and Windows 3.1 INI files all had a bastard love child—and it’s about as pretty as it sounds. IMO, you’d do well to stick with Docker Compose.
The Kubernetes compatibility feature lets you write Kubernetes-style configuration files and run them with Podman to start/manage your containers. It doesn’t actually use a Kubernetes cluster; it lets you pretend you’re running a big boy cluster because your command has the word “kube” in it, but in actuality you’re just running your lowly Podman containers instead. It also has the feel of being a dev toy intended for local development rather than actual production use.5 For instance, there’s no way to apply a change in-place without totally stopping and starting a container with two separate commands. What is this, 2003?
Lastly, there’s Podman Compose. It’s a third-party project (not produced by the Podman devs) that’s intended to support Docker Compose configuration files while working more “natively” with Podman. My brief experience using it (with all due respect to the devs) is that it’s total amateur hour and/or just not ready for prime time. Again, stick with Docker Compose, which works great with Podman.
Anyway, that’s all I’ve got! Use Podman if you want. Don’t use it if you don’t want. I’m not the boss of you. But you said you wanted content on Lemmy, and now you’ve got content on Lemmy. This is all your fault!
1 Where “full” is defined as: Not actually full.
2 Newer versions of Docker also have some rootless capabilities. But they’ve still got that stinky ol’ daemon.
2.5 It’s maybe not quite this simple in practice, because you’ll probably want to run multiple containers under the same Unix account unless you’re really OCD about security and/or have a hatred of the convenience of container networking.
3 You can run Podman as root and have many of the same properties as root Docker, but then what’s the point? One less daemon, I guess?
4 Where “solution” is defined as: Something that solves the problem while creating five new ones.
5 Spoiler: Red Hat’s whole positioning with Podman is like they see it is as a way for buttoned-up corporate devs to run containers locally for development while their “production” is running K8s or whatever. Personally, I don’t care how they position it as long as Podman works well to run my self-hosting shit…
You must log in or # to comment.
I see it as a feature that Podman containers are run via systemd. This makes their management consistent with the other systemd-managed services. Also, Docker does it own things with logs, while with systemd, the logs are managed in a consistent way as well.
Maybe you missed podman generate systemd? Podman will generate the systemd unit files for you.
For me, the two big benefits of podman are being able to run containers via systemd and improved security by being able to run them rootless.
I actually find this a huge problem. Not all distros are built around LSB, XDG, or FreeDesktop.org nor should they be since not everyone is running Linux as a workstation/PC replacement. While yes for the most part podman can be ran on the likes of Gentoo, Alpine, Arch and etc. It becomes a pain in the arse to decouple the tooling for podman away from freedesktop.org standards. Even more a pain in the arse for clustering options (e.g. podman-remote expects freedesktop.org norms, kubernetes expects docker containerd or freedesktop.org with podman, and nomad stack is just bulky vaporware).
The really sad part of this is that podman isn’t adding much of anything new that LXC or linux namespaces outside of not needing a daemon, allowing rootless execution (again because it doesn’t need a daemon) and giving ACLs around which OCI repos could be pulled from unlike docker’s wildcard by default. It shouldn’t be hard to do linux containerization without being tied to anything other than the linux kernel.
Thanks for the suggestion, but I actually tried
podman generate systemdand it did not work at all with my containers; the resultant services did not start successfully and I wasn’t able to get them working with any amount of tweaking. (I’d tell you what the error messages were, but I’ve only recorded them in now-deleted comments on The Site That Shall Not Be Named.)Ok. I was already very familiar with systemd when I started using podman and didn’t have any trouble at that step. In my case, I created the systems unit files by hand.
That sounds like it’s probably the way to go…
The easiest is actually using Quadlet (integrated in Podman 4.x or later): https://www.redhat.com/sysadmin/quadlet-podman and write simple .container files.
Works very well and does auto-start, service dependencies and even container auto-updates.
Also running containers in Pods is a really nice way to handle virtual networking and allows you to manage all the containers in one go similar to docker-compose.
The container auto-updates feature does sound pretty slick. I kind of have that hacked together now with a systemd timer and Docker Compose, but it’s not pretty.
One of the really nice side-effects of it running rootless is that you get all the benefits of it running as an actual Unix user.
For instance, you can set up wireguard with IP route to send all traffic from a given UID through the VPN.
Using that, I set up one user as the single user for running all the stuff I want to have VPN’d for outgoing connections, like *arr services, with absolutely no extra work. I don’t need to configure a specific container, I don’t need to change a docker-compose etc.
In rootful docker, I had to use a specific IP subnet to achieve the same, which was way more clunky.
Really good example of one way that Podman’s Unix Philosophy is actually helpful!
Could you explain or show how to do that?
Yeah sure.
I’m going to assume you’re starting from the point of having a second linux user also set up to use rootless podman. That’s just following the same steps for setting up rootless podman as any other user, so there shouldn’t be too many problems there.
If you have wireguard set up and running already - i.e. with Mullvad VPN or your own VPN to a VPS - you should be able to run
ip linkto see a wireguard network interface. Mine is calledwg. I don’t use wg-quick, which means I don’t have all my traffic routing through it by default. Instead, I use a systemd unit to bring up the WG interface and set up routing.I’ll also assume the UID you want to forward is
1001, because that’s what I’m using. I’ll also useenp3s0as the default network link, because that’s what mine is, but if yours iseth0, you should use that. Finally, I’ll assume that192.168.0.0is your standard network subnet - it’s useful to avoid routing local traffic through wireguard.#YOUR_STATIC_EXTERNAL_IP#should be whatever you get by callingcurl ifconfig.meif you have a static IP - again, useful to avoid routing local traffic through wireguard. If you don’t have a static IP you can drop this line.[Unit] Description=Create wireguard interface After=network-online.target [Service] RemainAfterExit=yes ExecStart=/usr/bin/bash -c " \ /usr/sbin/ip link add dev wg type wireguard || true; \ /usr/bin/wg setconf wg /etc/wireguard/wg.conf || true; \ /usr/bin/resolvectl dns wg #PREFERRED_DNS#; \ /usr/sbin/ip -4 address add #WG_IPV4_ADDRESS#/32 dev wg || true; \ /usr/sbin/ip -6 address add #WG_IPV6_ADDRESS#/128 dev wg || true; \ /usr/sbin/ip link set mtu 1420 up dev wg || true; \ /usr/sbin/ip rule add uidrange 1001-1001 table 200 || true; \ /usr/sbin/ip route add #VPN_ENDPOINT# via #ROUTER_IP# dev enp3s0 table 200 || true; \ /usr/sbin/ip route add 192.168.0.0/24 via 192.168.0.1 dev enp3s0 table 200 || true; \ /usr/sbin/ip route add #YOUR_STATIC_EXTERNAL_IP#/32 via #ROUTER_IP# dev enp3s0 table 200 || true; \ /usr/sbin/ip route add default via #WG_IPV4_ADDRESS# dev wg table 200 || true; \ " ExecStop=/usr/bin/bash -c " \ /usr/sbin/ip rule del uidrange 1001-1001 table 200 || true; \ /usr/sbin/ip route flush table 200 || true; \ /usr/bin/wg set wg peer '#PEER_PUBLIC_KEY#' remove || true; \ /usr/sbin/ip link del dev wg || true; \ " [Install] WantedBy=multi-user.targetThere’s a bit to go through here, so I’ll take you through why it works. Most of it is just setting up WG to receive/send traffic. The bits that are relevant are:
/usr/sbin/ip rule add uidrange 1001-1001 table 200 || true; \ /usr/sbin/ip route add #VPN_ENDPOINT# via #ROUTER_IP# dev enp3s0 table 200 || true; \ /usr/sbin/ip route add 192.168.0.0/24 via 192.168.0.1 dev enp3s0 table 200 || true; \ /usr/sbin/ip route add #YOUR_STATIC_EXTERNAL_IP#/32 via #ROUTER_IP# dev enp3s0 table 200 || true; \ /usr/sbin/ip route add default via #WG_IPV4_ADDRESS# dev wg table 200 || true; \ip rule add uidrange 1001-1001 table 200adds a new rule where requests from UID 1001 go through table 200. A table is a subset of ip routing rules that are only relevant to certain traffic.ip route addensures that traffic already going through the VPN - i.e. wireguard traffic - does. This is relevant for handshakes.ip route add 192.168.0.0/24 via 192.168.0.1 ...is just excluding local traffic, as isip route addFinally, we add
ip route add default viawhich routes all traffic that didn’t match any of the above rules (local traffic, wireguard) to go to the wireguard interface. From there, WG handles all the rest, and passes returning traffic back.There’s going to be some individual tweaking here, but the long and short of it is, UID
1001will have all their external traffic routed through WG. Any internal traffic between docker containers in a docker-compose should already be handled by podman pods and never reach the routing rules. Any traffic aimed at other services in the network - i.e.sonarrcallingsabnzbdortransmission- will happen with a relevant local IP of the machine it’s hosted on, and so will also be skipped. Localhost is already handled by existingip routerules, so you shouldn’t have to worry about that either.Hopefully that helps - sorry if it’s a bit confusing. I learned to set up my own IP routing to avoid
wg-quickso that I could have greater control over the traffic flow, so this is quite a lot of my learning that I’m attempting to distill into one place.
I guess this: “you run a “root” container in your completely unprivileged Unix user and everything just works” sounds like chroot. Also managing your container starts with systemd sounds pretty good to me because this is what systemd is designed for, dependencies between services, etc.
Lurker. Never self hosted. Nothing useful to add here. Fedora Silverblue has a lot of integration with podman. It is the basis of toolbx, which has its issues, namely that the distro it spins up can’t be upgraded in each toolbx container. The tools are there to mess with containers stuff.
Makes sense… Fedora, Red Hat, Podman… All one big ecosystem. Fortunately for other users, Podman runs fine on other distros too.
IMO it is the integration with toolbx that is interesting to me, but I struggle with these things.
Well feel free to make a post about any problems you run into with it… Someone here may be able to help!
I’ve used podman on an RHEL server at work because it works nicely with selinux. I had a hell of a time with rootless containers and network throughput when using an nginx reverse proxy. Made the site painfully slow. Turned out it was due to the slirp4netns rootless networking and MTU size. Just decided to say screw the rootless thing and went rootfull. Next time honestly would just use docker since it’s more common
That is one major caveat of Podman I neglected to mention above… Non-root does mean you give up some performance over running as root. Trade-offs!
Not necessarily. For networking, I wrote a bash script with just a few lines that creates and assigns a private networking namespace to a pod and sets up the default routes. That script is run by a systemd user instance and has the suid flag set. One could argue that it’s not rootless because of that but that’s just the moment when it’s starting. No performance impact and very robust. A lot better than the docker network bridges imho.
I’ve switched over my own server last week, using ansible to generate the systemd files, and it worked great. It’s just a dozen containers or so.
The only problems I had were with container interdependencies (network-mode=container:x). That didn’t work so well with systemd, restarting and updating, but when I used a pod instead these problems all went away.
So I can’t say I regret my experience so far. Now I’ll be starting to use it at work too, where the user-namespace problem rears its head, but only because we have this very specific, very dumb big lamp dev container that houses apache, sql, redis, and more under one supervisord. That’s why we have more than one user in it and frankly that’s our own damn fault! When you make proper containers they shouldn’t have more than one user in it and then userns=keep-id should work just fine.
So far, I fully recommend podman.
Using Ansible to spew out systemd service boilerplate seems like a good idea. I’ll have to try that if I can ever give up my Docker Compose security blanket. And I wish you luck with your mega-container Podman conversion. That one sounds like it’ll be… a learning experience.
I understand very well wanting to stay with the declarative nature of docker-compose. Someone should really build a better podman-compose. (or sooner or later I’ll do it myself >_<)
Do it! I think there’s a market there. Although the “Podman Compose” name is taken and you’ll have to think of something else…
Honestly, I had to use podman at work due to… issues.
Its close enough to docker, that most docker commands will work just fine. You can even alias docker as podman, and things will for the most part, just work.
HOWEVER, there are some big changes and difference. First- podmon creates a systemctl for your containers, for starting them. This- is different, and if you forget to tell it to create the service- then your containers won’t start.
My personal opinion after using it for a few years? I strongly prefer docker. It gives me very few issues. I have spent too much time troubleshooting odd things podman does.
I’ve been interested in alternatives to docker for a while now, but considering the way everything is going, tbh, I feel like I’d be more likely to just finally learn kubernetes.
Didn’t know about Podman yet, I’ll probably give it a spin on some personal projects to get a bit more used to tools other than docker. Thanks for the great intro!
If you do try it out, let us know how totally obnoxious it is for you!
Fyi, you can create a pod, create containers in that pod and generate the systemd file for that. Now you have one service to start/stop everything.
I’ve been running Docker with Portainer for last few years. Been working great. Tried making the switch over to podman but couldn’t get it to work with Portainer (which a lot of people say it can) and had some issues with NFS shares. One day I’ll have to give it a try again.
If Docker’s working fine for you and you’re happy with the trade-offs, no reason to switch.
The other big annoying thing about Podman is that because there’s no Big Bad Daemon managing everything, there are certain things you give up. Like containers actually starting on boot. […] until you realize that means Podman wants you to manage your containers entirely with systemd. So… running each container with a systemd service, using those services to stop/start/manage your containers, etc.
Surprisingly, they have a solution for that that doesn’t involve using systemd for everything. They put an --all option to podman start, and a systemd service to run it at boot with the correct --filter (yeah. because unix philosophy). Debian seems to enable it by default AFAICT.
No idea how well it works rootless though.
Edit: Oh and for rootless networking, Podman 4.4.0 seems to ship pasta which seems to be the solution to slirp4netns’s existence. Unfortunately I have no idea if it works at all because I run Debian stable which is still on 4.3.1
Do you think podman could replace Docker Desktop on my Dev machine? Its become so bloated…
I honestly don’t have any experience with Docker Desktop. Maybe someone else here can weigh in.
EDIT: There is this: https://podman-desktop.io/
For instance, there’s no way to apply a change in-place without totally stopping and starting a container with two separate commands. What is this, 2003?
Wait, doesn’t the same apply to Docker? There’s no “change settings” feature of already running containers, you have to stop container (maybe commit image before) and start with new settings.
I’m comparing this (perhaps) unfairly to two different features:
docker-compose upwhich does the right thing in terms of starting a set of services or updating a set of already running services.kubectl apply, which does the same kind of thing for services running on a K8s cluster.
I’ve tried to switch in the past, but tripped over the differences in Podman vs Docker networking. IIRC Docker is better for creating an isolated network.
I have noticed that Docker doesn’t do the best job at graceful shutdowns (say for automatic installation of updates). I suspect Podman with systemd integration could do much much beter.
Yeah, one of my motivations for kicking Docker to the curb (beyond security) is all the weird little bugs: Preventing shutdown, unresponsive commands, random hangs, broken upgrades, etc. Podman may not end up being any better there, but at least I can pretend for a while.
At least podman does not circumvent my firewall (ufw) like docker did. Had to use a workaround to get it to work with docker.
Podman respects UFW?? That’s awesome to hear.
…and of course I learn this after switching to firewalld.
At least firewalld feels relatively painless compared to rest of the redhat-container-verse.
Yep, turns out it doesn’t insert its own iptables rules like docker does, so the special rules from ufw-docker weren’t necessary anymore.
