Rootless podman containers under system accounts, managed and enabled at boot with systemd

Leave a Reply

While you can run containers as root on the host, or run rootless containers as your regular user (either as uid 0 or any another), sometimes it’s nice to create specific users to run one or more containers. This provides neat separation and can also improve security posture.

We also want those containers to act as regular system services; managed with systemd to auto-restart and be enabled on boot.

This assumes you’ve just installed Fedora (or RHEL/CentOS 8+) server and have a local user with sudo privileges. First, let’s also install some SELinux tools.

sudo dnf install -y /usr/sbin/semanage

Setting up the system user

Let’s create our system user, placing their home dir under /var/lib. For the purposes of this example I’m using a service account of busybox but this can be anything unique on the box. Note, if you prefer to have a real shell, then swap /bin/false with /bin/bash or other.

export SERVICE="busybox"

sudo useradd -r -m -d "/var/lib/${SERVICE}" -s /bin/false "${SERVICE}"

In order for our user to run containers automatically on boot, we need to enable systemd linger support. This will ensure that a user manager is run for the user at boot and kept around after logouts.

sudo loginctl enable-linger "${SERVICE}"
Continue reading →

Podman volumes and SELinux

Leave a Reply

In my previous post on volumes with podman, I touched on SELinux but I think it’s worthy of its own post to work through some details.

If your host has SELinux enabled, then container processes are confined to the system_u:system_r:container_t:s0 domain. Volumes you pass to podman will need to have appropriate labels, otherwise the container won’t be able access the volume, no-matter what the filesystem permissions are.

When running rootless containers (as your non-root user), files in your homedir will probably have the context of unconfined_u:object_r:data_home_t:s0, however the context that is required for container volumes is system_u:object_r:container_file_t:s0.

Fortunately, container volumes which podman creates at runtime will have the appropriate context set automatically. However, for host-dir volumes podman will not change the context by default, you will need to tell it to.

Let’s spin up a busybox container without setting the SELinux context and note that the container is not able to access the host-dir volume.

Continue reading →

Volumes and rootless Podman

1 Reply

Containers generally run from an image and have no access to the host file system. This is fine for stand-alone ephemeral containers, but others require persistent data. While we can pass in environment variables to configure the app inside the container, sometimes a place to read and write more complex files is needed. When you update or replace such a container (perhaps with a new release) you want the new container to have access to the same data as the previous one.

The tricky thing with rootless containers is that you’re not root on the host and, as per my previous post, containers can run as any user id. If the container runs as root (uid 0) then that is fine as it actually maps to your non-root user on the host (e.g. 1000) and management of the data is therefore easy. However, containers running as other users (e.g. 123) will map to a uid on the host based on the subuid offset range (e.g. 100122) which will not match your non-root user and therefore data management is harder.

Providing persistent storage to a container is done by setting up a bind mounts using the --volume (or -v) option with podman. Note that volumes are not restricted to just one per container, simply repeat the -v option for as many volumes as required!

The volume option uses a colon separated format specifying the source on the host and then the destination directory where the volume is mounted to inside the container (e.g. src:/dest).

The source is generally one of two types; either a container volume (like a container image) or a regular host directory.

Continue reading →

User IDs and (rootless) containers with Podman

Leave a Reply

When a Linux container image is created, like any system it can have specific users, or maybe it only has root (uid 0). Containers have a specific entry point which runs the program the image was created for and this might run as any user in the image, it’s up to whoever created it.

You can use podman (a daemonless container engine) to easily see what uid an image will use, by getting the container to run the id command instead of the default entry point.

For example, here we can see that busybox wants to run as root (uid 0).

# podman run --rm --entrypoint '' docker.io/busybox id
uid=0(root) gid=0(root) groups=0(root)

However, grafana wants to run as the grafana user with uid 472 .

# podman run --rm --entrypoint '' docker.io/grafana/grafana id
uid=472(grafana) gid=0(root) groups=0(root)

OK, so inside the containers we are running as different users, but as we’re running as root those same uids are also used on the host system.

Running containers as root

Let’s run a grafana container as root and see that the actual grafana-server process on the host is running under uid 472.

# podman run -d docker.io/grafana/grafana
ee275572c8dcb0922722b7d6c3e5ff0bda8c9af3682018c4fc53675d8e189e59
# ps -o user $(pidof grafana-server)
USER
472
Continue reading →

How to create bridges on bonds (with and without VLANs) using NetworkManager

Leave a Reply

Some production systems you face might make use of bonded network connections that you need to bridge in order to get VMs onto them. That bond may or may not have a native VLAN (in which case you bridge the bond), or it might have VLANs on top (in which case you want to bridge the VLANs), or perhaps you need to do both.

Let’s walk through an example where we have a bond that has a native VLAN, that also has the tagged VLAN 123 on top (and maybe a second VLAN 456), all of which need to be separately bridged. This means we will have the bond (bond0) with a matching bridge (br-bond0), plus a VLAN on the bond (bond0.123) with its matching bridge (br-vlan123). It should look something like this.

+------+   +---------+                           +---------------+
| eth0 |---|         |          +------------+   |  Network one  |
+------+   |         |----------|  br-bond0  |---| (native VLAN) |
           |  bond0  |          +------------+   +---------------+
+------+   |         |                                            
| eth1 |---|         |                                            
+------+   +---------+                           +---------------+
            | |   +---------+   +------------+   |  Network two  |
            | +---| vlan123 |---| br-vlan123 |---| (tagged VLAN) |
            |     +---------+   +------------+   +---------------+
            |                                                     
            |     +---------+   +------------+   +---------------+
            +-----| vlan456 |---| br-vlan456 |---| Network three |
                  +---------+   +------------+   | (tagged VLAN) |
                                                 +---------------+

To make it more complicated, let’s say that the native VLAN on the bond needs a static IP and to operate at an MTU of 1500 while the other uses DHCP and needs MTU of 9000.

OK, so how do we do that?

Continue reading →

How to create Linux bridges and Open vSwitch bridges with NetworkManager

Leave a Reply

My virtual infrastructure Ansible role supports connecting VMs to both Linux and Open vSwitch bridges, but they must already exist on the KVM host.

Here is how to convert an existing Ethernet device into a bridge. Be careful if doing this on a remote machine with only one connection! Make sure you have some other way to log in (e.g. console), or maybe add additional interfaces instead.

Export interfaces and existing connections

First, export the the device you want to convert so we can easily reference it later (e.g. eth1).

export NET_DEV="eth1"

Now list the current NetworkManager connections for your device exported above, so we know what to disable later.

sudo nmcli con |egrep -w "${NET_DEV}"

This might be something like System eth1 or Wired connection 1, let’s export it too for later reference.

export NM_NAME="Wired connection 1"

Create a Linux bridge

Here is an example of creating a persistent Linux bridge with NetworkManager. It will take a device such as eth1 (substitute as appropriate) and convert it into a bridge. Note that we will be specifically giving it the device name of br0 as that’s the standard convention and what things like libvirt will look for.

Continue reading →

Accessing USB serial devices in Fedora Silverblue

Leave a Reply

One of the things I do a lot on my Fedora machines is talk to devices via USB serial. While a device is correctly detected at /dev/ttyUSB0 and owned by the dialout group, adding myself to that group doesn’t work as it can’t be found. This is because under Silverblue, there are two different group files (/usr/lib/group and /etc/group) with different content.

There are some easy ways to solve this, for example we can create the matching dialout group or write a udev rule. Let’s take a look!

On the host with groups

If you try to add yourself to the dialout group it will fail.

sudo gpasswd -a ${USER} dialout
gpasswd: group 'dialout' does not exist in /etc/group

Trying to re-create the group will also fail as it’s already in use.

sudo groupadd dialout -r -g 18
groupadd: GID '18' already exists

So instead, we can simply grab the entry from the OS group file and add it to /etc/group ourselves.

grep ^dialout: /usr/lib/group |sudo tee -a /etc/group

Now we are able to add ourselves to the dialout group!

Continue reading →

Making dnf on Fedora Silverblue a little easier with bash aliases

Leave a Reply

Fedora Silverblue doesn’t come with dnf because it’s an immutable operating system and uses a special tool called rpm-ostree to layer packages on top instead.

Most terminal work is designed to be done in containers with toolbox, but I still do a bunch of work outside of a container. Searching for packages to install with rpm-ostree still requires dnf inside a container, as it does not have that function.

I add these two aliases to my ~/.bashrc file so that using dnf to search or install into the default container is possible from a regular terminal. This just makes Silverblue a little bit more like what I’m used to with regular Fedora.

cat >> ~/.bashrc << EOF
alias sudo="sudo "
alias dnf="bash -c '#skip_sudo'; toolbox -y create 2>/dev/null; toolbox run sudo dnf"
EOF

If the default container doesn’t exist, toolbox creates it. Note that the alias for sudo has a space at the end. This tells bash to also check the next command word for alias expansion, which is what makes sudo work with aliases. Thus, we can make sure that both dnf and sudo dnf will work. The first part of the dnf alias is used to skip the sudo command so the rest is run as the regular user, which makes them both work the same.

We need to source that file or run a new bash session to pick up the aliases.

Continue reading →

Fedora Silverblue is an amazing immutable desktop

6 Replies

I recently switched my regular Fedora 31 workstation over to the 31 Silverblue release. I’ve played with Project Atomic before and have been meaning to try it out more seriously for a while, but never had the time. Silverblue provided the catalyst to do that.

What this brings to the table is quite amazing and seriously impressive. The base OS is immutable and everyone’s install is identical. This means quality can be improved as there are less combinations and it’s easier to test. Upgrades to the next major version of Fedora are fast and secure. Instead of updating thousands of RPMs in-place, the new image is downloaded and the system reboots into it. As the underlying images don’t change, it also offers full rollback support.

This is similar to how platforms like Chrome OS and Android work, but thanks to ostree it’s now available for Linux desktops! That is pretty neat.

It doesn’t come with a standard package manager like dnf. Instead, any packages or changes you need to perform on the base OS are done using rpm-ostree command, which actually layers them on top.

And while technically you can install anything using rpm-ostree, ideally this should be avoided as much as possible (some low level apps like shells and libvirt may require it, though). Flatpak apps and containers are the standard way to consume packages. As these are kept separate from the base OS, it also helps improve stability and reliability.

Continue reading →

Easy containers on Fedora with toolbox

Leave a Reply

The toolbox program is a wrapper for setting up containers on Fedora. It’s not doing anything you can’t do yourself with podman, but it does make using and managing containers more simple and easy to do. It comes by default on Silverblue where it’s aimed for use with terminal apps and dev work, but you can try it on a regular Fedora workstation.

sudo dnf install toolbox

Creating containers

You can create just one container if you want, which will be called something like fedora-toolbox-32, or you can create separate containers for different things. Up to you. As an example, let’s create a container called testing-f32.

toolbox create --container testing-f32

By default toolbox uses the Fedora registry and creates a container which is the same version as your host. However you can specify a different version if you need to, for example if you needed a Fedora 30 container.

toolbox create --release f30 --container testing-f30

These containers are not yet running, they’ve just been created for you.

Continue reading →