From RBAC to AppArmor: six phases to harden k3s on a Raspberry Pi

In the post about the migration to Gitea I described how I got my projects’ builds working inside the cluster. To get there, I took a shortcut: the Gitea runner (the program that executes the CI workflows) got far more privileges than it needed.

• the containerd socket, the file through which you give orders to the program that runs the containers on the node. Whoever can write to it can ask for a container running as root with the Pi’s whole disk mounted;
• the directory holding the cluster’s images and data;
• the admin kubeconfig, the credential that lets kubectl do anything: read passwords, create and delete any resource.

It worked, but any workflow from any repository had access to the whole cluster. All it takes is an npm ci that installs a compromised dependency: its script runs inside the job and only has to read the kubeconfig sitting right there.

As long as I’m the only user, you can live with that. But today the Pi serves this site to the internet and holds my repositories, the memory of my AI conversations and the projects’ usage logs. I decided to treat it as production, with three rules: each component gets only the permission it needs (the principle of least privilege); protection comes in layers, so that one failing doesn’t hand over everything; and every change is tested before and after.

This is the first of three posts, and it’s about making an attack harder. The second will be about noticing that someone escaped a container, and the third about seeing who reaches the sites from the internet and how to block abusers.

There was one constraint: the Pi has 4 GB of RAM, with about 1.2 GB free. Common tools for this job, like Kyverno, Calico or Falco, run as their own pods and use memory I don’t have. So I used only what already ships with Kubernetes and k3s. None of the phases below added a pod.

In short

• Before: any CI job effectively had admin access to the cluster and to the Pi.
• After: each job type gets only what it needs; no pod mounts anything from the host; each pod only talks to who it needs to; the kernel limits what each container can do; and anyone escaping a container lands on the Pi as a user with no privileges at all.
• Cost: zero extra pods. The runners now use less memory (from ~83 MiB down to 7–9 MiB each).
• Bonus: testing one of the changes found a backup defect that would have left Gitea broken after any restore.

The starting point: one pod per job

In October, Gitea 28.1.0 came out with fixes for 9 CVEs, along with Gitea Runner 5.0.0, the successor to act_runner. On the 13th I upgraded both (Gitea was on 1.27.3; the runner, on 0.6.1).

The rest of the work started from a feature of the new runner: the kubernetes:// mode, where each job runs in its own pod, created on the spot and thrown away when it finishes. The way I used the old act_runner, jobs ran inside the runner’s own container and inherited everything it had mounted. A test job had the same access as a deploy job. In the new mode, what each pod gets comes from a template in the runner’s configuration, one per job type, and the repository’s workflow can’t ask for anything beyond that.

So each job type can have different permissions. The test job needs no credentials at all; the build job needs to build an image; the deploy job needs to apply manifests to the cluster. The six phases deal with this, one piece at a time:

I did the first five between October 13 and 15. On the 16th, I switched the Gitea image, the last container of mine still running as root. The sixth came on the 29th.

Phase 1: RBAC

What it is

Every pod that talks to the Kubernetes API uses an identity, the ServiceAccount. RBAC (role-based access control) defines what each identity can do: on which resource types (Deployment, Secret, Pod…), with which actions (read, create, change, delete) and in which namespaces. Permissions live in a Role or ClusterRole, and a RoleBinding grants them to an identity, only within the namespace where it was created. That lets you write the rules once and apply them only where they make sense.

What I did

Deploy needs to apply Deployment, Service, IngressRoute and a few more resources, and only in the apps’ namespaces. I created one ServiceAccount per role:

The rules for both deploys live in a single ClusterRole. An excerpt:

kind: ClusterRole
metadata:
  name: ci-deployer
rules:
  # `kubectl apply` = get + create (if missing) or patch (if present).
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "create", "patch", "update"]
  - apiGroups: ["traefik.io"]
    resources: ["ingressroutes"]
    verbs: ["get", "list", "watch", "create", "patch", "update"]

Each ServiceAccount gets that ClusterRole through RoleBindings, one in each namespace where it may act: three for ci-deployer-apps (apps, dev, prd) and one for ci-deployer-docs.

I left two permissions out on purpose:

• Delete. Deploy applies manifests; removing something is my decision, made by hand. If the token leaks, the worst you can do is apply a bad manifest, which the next deploy overwrites.
• Read Secrets. The app’s manifest references the Secret by name, and the kubelet is what reads the value, when it starts the pod. Deploy never needs to see the contents.

I only found one gap days later. RBAC says on which resource types and in which namespaces an identity can act, but it can’t tell one object from another by name. Since Gitea also lives in apps, any app’s deploy could swap Gitea’s image. The fix, a ValidatingAdmissionPolicy, is left for the third post in the series.

Test jobs ended up with no ServiceAccount at all. They run on pull requests, meaning on code that hasn’t been reviewed yet, and now there’s no token in the pod to steal.

How to check

kubectl auth can-i --list -n apps --as=system:serviceaccount:ci:ci-deployer-apps

--as makes kubectl impersonate the ServiceAccount and list what it can do in that namespace, without running any job.

Before, a malicious script in deploy could read every Secret in the cluster (tunnel tokens, Postgres passwords), create a privileged pod or delete whatever it wanted. Now it can apply a few kinds of manifest in three namespaces. But the build still had the containerd socket, and through it you could do all of that without going through RBAC.

Phase 2: builds without the host socket

The problem

Building an image means running the Dockerfile’s steps, and each RUN needs a container. The simplest way is to ask the node’s own containerd for those containers, through the socket. The problem is that containerd doesn’t know what each container it’s asked for is meant to do. To it, a build step and a privileged container with the Pi’s disk mounted are the same kind of request, and whoever can reach the socket gets either one. The request also goes straight to containerd, bypassing the Kubernetes API, so RBAC never even hears about it.

What I did

I switched to BuildKit, the build engine Docker itself uses, in its rootless version. It runs as a second container inside the job pod, with nothing from the host, and creates the containers for each build step on its own, without asking the node’s containerd for anything. The finished image goes to the container registry that comes with Gitea, and the node pulls it from there, as it would from Docker Hub.

Two obstacles came up:

• Seccomp. Seccomp is a kernel filter that defines which system calls a process may make. Kubernetes’ default profile blocks exactly the ones BuildKit uses to isolate the build steps. Turning the filter off would work, but Phase 3 forbids that. I wrote my own profile, which allows what BuildKit needs and keeps blocking what a build has no reason to do, like loading kernel modules or changing the clock.
• Who publishes the image. Pushing to the registry needs a token. Instead of using one of mine, I created a ci-bot user, without admin permission, that can only publish packages for the organizations that run builds. If that token leaks, it reaches neither code nor configuration.

I took the chance to change how deploy refers to the image. It used to be by tag (site-chat:<sha>); now it’s by digest, the hash of the content (site-chat:<sha>@sha256:…). Anyone who can push to the registry can point a tag at another image; a digest identifies an exact content and never changes. That way the pod runs exactly the image the build produced.

After that, the runner pods had no privileged mode, no socket, no kubeconfig and no host directory mounted (hostPath). An uncached site-chat build takes 2min10s, peaking at 114 MiB, within the pod’s memory limit. The old mode’s cache, which had no limit and had reached 16 GB on the SD card, was deleted. The runners, which now just wait for jobs, dropped from ~83 MiB to 7–9 MiB each.

What was left was making sure none of that access came back, not even through my own carelessness.

Phase 3: Pod Security Admission

What it is

Pod Security Admission is a check that comes built into the API server: before creating a pod, it verifies that the pod respects the namespace’s security level. There are three levels:

• privileged: no restrictions.
• baseline: blocks what gives direct access to the host, such as privileged containers, mounted host directories, using the node’s network or processes, extra root powers (capabilities) and disabled seccomp.
• restricted: everything baseline requires and more: run without root, with no capabilities, with no way to gain privileges later and with seccomp on.

The level is a label on the namespace:

kind: Namespace
metadata:
  name: prd
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.35

enforce makes the API server reject new pods that don’t meet the level. enforce-version pins the rules of a Kubernetes version; without it, a k3s upgrade could tighten the rules on its own and start rejecting pods that used to work.

How I applied it

Changing the label doesn’t affect pods already running; only new ones are checked. Even so, tightening the level blindly guarantees the next deploy will fail. kubectl shows in advance what would be rejected, without changing anything:

kubectl label --dry-run=server --overwrite ns <ns> pod-security.kubernetes.io/enforce=restricted

I went one namespace at a time. Each one started at the level its pods already met and moved up once all of them were ready:

ci stays at baseline because the test pods run images as root, and BuildKit has to run as root inside the pod (Phase 5 explains why). restricted allows neither.

Why this is needed on top of RBAC

Phase 1’s RBAC still lets deploy create Deployments in apps. Without Pod Security, a compromised deploy could create a pod there that mounts the Pi’s disk. Each covers the other: RBAC says where deploy may act, and Pod Security limits what the pods created there may ask for.

Phase 4: NetworkPolicy

What it is

By default, the Kubernetes network is wide open: any pod can open a connection to any other, in any namespace. Until October 14 that was the case here. A Cloudflare tunnel could talk to Postgres, and a test job to Gitea’s SSH.

NetworkPolicy is a firewall between pods. Once a pod is covered by some policy, only traffic that some policy allows gets through. The usual approach is to start each namespace by blocking everything and then open only what’s needed. The rules are enforced by kube-router, which already ships inside k3s, so I didn’t have to install anything.

One example is the prd namespace, where the Cloudflare tunnels live. A tunnel needs DNS, Cloudflare and the path to the sites it serves, and nothing else:

# Block everything, ingress and egress, for every pod in the namespace.
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: prd
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
# Allow egress to Cloudflare: the internet minus the private ranges,
# only on the ports the tunnels use.
kind: NetworkPolicy
metadata:
  name: tunnel-egress
  namespace: prd
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
        except: [10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10, 169.254.0.0/16]
    ports:
    - {port: 7844, protocol: UDP}
    - {port: 7844, protocol: TCP}
    - {port: 443, protocol: TCP}
  # (the full file also allows DNS and the tunnels' port on Traefik)

Cloudflare’s IPs change, which is why the rule is by port and not by address.

Where it ended up

In the first version, the tunnels talked directly to the sites’ pods. Today they go through a dedicated proxy port, where only the routes published to the internet exist. The reason was being able to see who accesses the sites, and that is left for the third post in the series.

What I learned along the way

Before touching production, I set up a lab with temporary namespaces to see how kube-router really behaves. Three lessons that apply to any cluster:

• NetworkPolicy only sees IP and port, not the site being requested. For the jobs to reach the registry, I allowed Traefik, the proxy that receives requests and forwards each one to the right site. In doing so I also allowed every internal site behind it: in one test, a job managed to list the files of my text editor. The fix was a Service just for the registry, which the jobs reach directly, without going through Traefik.
• When in doubt, block. Each job pod gets a label with its role (ci-role: test, build or deploy), and the network opens up according to it. A pod with no known role only reaches DNS and Gitea. If I forget the label on a new job template, the job fails and I notice, instead of it quietly getting internet access.
• Rules by type of network, never by a machine’s IP. The policies talk about “private networks”, “private VPN” and “the cluster’s internal network”, never the Pi’s IP or my router’s range. If I change routers, nothing has to change.

How to check

A rule that allows too much produces no symptom at all: everything keeps working. That’s why I wrote netpol_check.sh, which creates temporary pods with the same labels as the real ones and tests every path that should get through and every one that should be blocked. It started with 84 cases and now has 110. I run it again after any change to the policies, the job templates or the k3s version.

In practice: if cloudflared, the only pod that receives traffic from the internet, is compromised, it only reaches Cloudflare and the proxy port reserved for the tunnels. It can’t get to Postgres, Gitea or the Kubernetes API. And no job or tunnel pod reaches the home network, so a compromised container can’t serve as a bridge to attack the other devices.

Between phases: two problems outside the plan

Thinking about what a process would find if it escaped a container, I found two problems that weren’t part of any phase.

The admin kubeconfig (/etc/rancher/k3s/k3s.yaml) was readable by any user on the Pi, including the users the containers run as. Now only root can read it.

The k3s version wasn’t pinned, and a reinstall would pick up that day’s stable version. Now it is, and an upgrade script backs up the cluster database, installs the new version and checks that everything came back. After the upgrade to 1.35.9, I ran the 84 network tests again.

Phase 5: user namespaces

What it is

The previous phases limit what a pod can ask for. This one deals with the case where someone escapes the container anyway, through a flaw in the kernel or the runtime.

In an ordinary container, the user inside is the same as outside. A process running as root in the container that escapes is root on the Pi. One running as uid 1000 becomes, on the Pi, uid 1000, which in my case is my user, owner of the SSH keys and the backup password.

A user namespace does a translation. Each pod gets its own range of 65,536 high uids on the host, and the uids inside are mapped onto it. Root inside becomes, on the Pi, something like uid 3,526,886,376, which owns no files and has no privileges. In Kubernetes, it’s one line in the pod:

spec:
  hostUsers: false

The catch is volumes: files on disk belong to normal uids, and the process now has a high uid. The kubelet solves this by mounting volumes with uid translation (idmapped mounts), and to the process the files keep the same owners. In the lab, a pod ran on the host as uid 3,526,886,376 and read and wrote to its volumes normally. k3s 1.35.9, with the Pi’s 6.18 kernel, already had everything needed.

Where I applied it

In order of risk:

1. Mempalace. It ran as uid 1000, that is, as my user on the Pi.
2. Gitea. It ran as root, with SSH accepting connections from the home network. Anyone escaping through there would be root on the Pi. The switch took Gitea down for 28 seconds.
3. CI test and deploy pods. They run code I didn’t write (npm ci, go mod download) as root.
4. The runner pods, Postgres and the tunnels.

After each one, the same validation: a full CI pipeline, the backup, the tunnels reconnecting, the site responding and the network tests.

I left the build pod out, thinking the isolation rootless BuildKit creates for the build steps was enough. I was wrong: BuildKit itself ran as uid 1000, which on the Pi is my user, and the job container ran as root on the Pi. I found out on the 19th, through the next post’s container-escape alert, which fired on every build. The fix shipped the same day: the build pod got a user namespace too, and BuildKit now runs as root inside the pod, which on the Pi is a high uid with no privileges at all.

The apps from my other repositories were left out at first. I thought they already ran as users that don’t exist on the Pi, so the gain would be small. That was true for the site’s chat, which runs as uid 10001. It wasn’t for the four sites served by nginx: uid 101 in the nginx-unprivileged image is, on the Pi, the user of avahi-daemon, the service that announces the Pi’s name on the home network. The four ran with the same identity as that service and as each other, and an escape from any of them would reach the other three. I found this on the 29th, checking the uid with getent passwd 101, and the same day all five apps got a user namespace. Before choosing the uid for a container without a user namespace, it’s worth running that check.

That day I also learned a lesson of a different kind: I merged the four PRs at once, and the four pipelines ran together. The Pi hit a load of 15 with 30 MB of free memory, k3s’s database couldn’t write in time and k3s went down, coming back on its own after 50 seconds. On a 4 GB node, it’s one merge at a time.

Still left out are k3s’s internal components.

Gitea without root

After Phase 5, Gitea was no longer urgent, but it was the only pod holding apps at baseline. Gitea’s default image starts as root because it runs two programs, Gitea itself and an SSH server, and needs to switch users to do so. There’s a -rootless version that runs directly as the git user, with an SSH server built into Gitea itself on port 2222 (without root you can’t use port 22).

For people using Gitea, the switch had to go unnoticed:

• The data stayed where it was. The rootless image expects a different directory layout; a few environment variables make it use the existing one. Nothing was copied or moved.
• The SSH server’s identity is the same. The built-in server uses the keys the previous one had generated in June, so anyone already cloning doesn’t get the warning that the server changed.
• The port only changes inside the cluster. From outside it’s still the usual 22, and the Service forwards to 2222.

I tested in a lab built from the latest backup: clone and push over SSH, the API, the registry and the admin commands. That lab is what found the week’s most serious problem, and it had nothing to do with the image.

The backup that restored a broken Gitea

The first push in the lab was accepted, but the branch didn’t show up in Gitea. Gitea learns about a push through git hooks, scripts git runs when it receives commits. In the restored copy, those scripts had lost their execute permission.

The cause was in the backup. It copied the pod’s data with kubectl cp, which preserves neither permissions nor file timestamps. Every snapshot had the hooks without execute permission and the SSH keys readable by any user. A Gitea restored from any of them would accept pushes and never again update branches or pull requests, without showing any error.

Two days earlier I had run a full restore test. It checked database integrity, the repository count and the registry files, and it passed. It didn’t catch the problem because it never pushed anything.

The fix had three parts. The backup now copies with tar, which preserves permissions and timestamps. The restore now recreates the hooks, to cover the old snapshots. And since the rootless image doesn’t ship sqlite3, which the backup used to copy the database, that copy is now made by a temporary pod. As a bonus, with timestamps preserved, the nightly backup stopped re-reading the ~2,100 files and now reads only the ones that changed.

In production, the order was: backup first; the switch, with 28 seconds of downtime; an SSH test from the desktop, checking the server’s identity; a full CI pipeline; the new backup; the network tests. Only then did apps move up to restricted, after the dry-run showed no pod would be rejected. Today, among the namespaces I manage, only ci isn’t at restricted.

Phase 6: AppArmor

What it is

User namespaces deal with what happens after an escape. AppArmor acts before: it’s a kernel security module that limits what each process can do through profiles: which files it reads and writes, whether it can mount filesystems, which root capabilities it uses.

With AppArmor enabled on the node, containerd applies a default profile to every container, without changing any manifest. That profile denies mounts, writes to /proc/sys and /sys, and access to /proc/kcore. These are kernel surfaces an ordinary app doesn’t use and that many container escapes do. The default profile doesn’t stop a compromised process from opening a shell or reading the container’s own files; that would take a profile per app.

What I did

The Raspberry Pi OS kernel has AppArmor compiled in, but doesn’t load it at boot: the list of security modules is empty, and only capability loads.

$ cat /sys/module/apparmor/parameters/enabled
N

Enabling it takes one parameter on the boot line (/boot/firmware/cmdline.txt) and a reboot:

lsm=capability,apparmor

The obstacle: BuildKit

Phase 2’s BuildKit mounts filesystems to isolate the build steps, and the default profile denies mounts. With AppArmor enabled, every build would fail.

Running BuildKit without a profile (Unconfined) is forbidden by Phase 3’s baseline. The way out was the same as for seccomp: a profile of my own, of type Localhost, which baseline accepts. It allows the mounts and keeps the default profile’s denials that no build uses:

profile buildkit-rootless flags=(attach_disconnected,mediate_deleted) {
  include <abstractions/base>
  capability, file, mount, umount, remount, pivot_root, network, unix, signal, userns,
  deny @{PROC}/sysrq-trigger rwklx,
  deny @{PROC}/kcore rwklx,
  deny /sys/firmware/** rwklx,
  deny /sys/kernel/security/** rwklx,
}

In the build templates, it sits next to the seccomp profile:

securityContext:
  seccompProfile:
    type: Localhost
    localhostProfile: profiles/buildkit-rootless.json
  appArmorProfile:
    type: Localhost
    localhostProfile: buildkit-rootless

Order matters: the kubelet rejects a pod that asks for an AppArmor profile on a node without AppArmor. Reboot first, templates second.

How to check

$ cat /sys/module/apparmor/parameters/enabled
Y
$ ps -eo label,comm | grep cri-containerd | head -2
cri-containerd.apparmor.d (enforce) nginx
cri-containerd.apparmor.d (enforce) site-chat

After the reboot, every container ran under the default profile, and one build on each runner passed.

What’s left is knowing whether any rule blocks something that should work. With the next post’s auditd running, AppArmor denials don’t go to journalctl -k but to the audit log, which only root can read. Looking in the wrong place gives zero and proves nothing. The right command:

sudo ausearch -m AVC -ts boot | grep -c 'apparmor="DENIED"'

Result: zero. Cost: one reboot and not a single extra pod.

What’s still missing

• The AppArmor profiles are the generic ones. A profile of its own for each internet-facing site (the chat, this site) could deny what the default profile allows, like opening a shell or writing outside the data folder. That requires watching each app for a few days in complain mode before blocking.
• ci stays at baseline, for the reasons in Phase 3. Moving up would require test images without root and another way to build images.
• metrics-server, which ships with k3s, runs with my uid and without a user namespace. k3s rewrites its manifest on every restart, so changing that means disabling theirs and maintaining my own.
• The registry, Postgres and Mempalace speak HTTP, without TLS. Their only protection is being reachable just from the home network and the private VPN.
• Everything runs on a single node. None of this protects against the SD card dying. That’s what the backup is for, and this month it gained a copy off the Pi.
• None of these phases tells me an attack happened. They make an attack harder, but they don’t detect it. That’s the subject of the next post.
• None of them tells me who is accessing the sites. That’s the subject of the third.

How I worked

Every phase followed the same script, and it’s the one I’d take to another cluster:

1. Test in a lab first. A temporary namespace, the same image, measure the real behavior and delete it afterwards. When possible, on top of a restored copy of the backup, which tests the backup too.
2. One change per PR, with what was measured in the commit message.
3. Validate with real usage: a whole pipeline, a push, a tunnel reconnecting. The restore test that only checked the databases shows why looking at state alone isn’t enough.
4. Leave behind a repeatable test: the network tests, kubectl auth can-i, the Pod Security dry-run.
5. Document how to undo each change.