Intrusion detection in kube containers: uid invariants and kernel auditing

In the first post I made the cluster harder to attack, but nothing there tells me whether an attack happened. If a process escaped a container, I wouldn’t know.

This second post in the series is about detecting that escape, with two tools: a periodic check that relies on a rule left behind by user namespaces, and kernel auditing, which records events the instant they happen. The third post covers the traffic coming in from the internet.

In short

• Before: nothing on the Pi warned me if a process left a container.
• After: a 30-line script, without root, periodically checks that every high uid belongs to its pod; auditd records, as they happen, seven kinds of event that almost never happen by accident.
• What an alert tells me: which pod the process came from and, therefore, how the attack got in. Who the attacker is, almost never.
• Cost: zero extra pods; just auditd, a few MB, on the host.
• Still missing: getting alerts off the Pi the moment they fire, so an intruder with root can’t erase them.

The rule user namespaces left behind

With hostUsers: false, each pod gets a user namespace: its own range of 65,536 high uids on the host, onto which the uids inside the container are mapped. A pod’s root becomes, on the Pi, something like uid 3,558,147,048.

On top of that, Kubernetes places each pod’s processes in a cgroup, a kernel group used to limit CPU and memory. The cgroup’s path contains the pod’s UID (an identifier unrelated to user uids), and any process can read it in /proc/<pid>/cgroup. Looking only at /proc, I can tell for any process on the Pi which uid it runs as and which pod it belongs to. That gives two rules:

1. every process in the cgroup of a pod with a user namespace has a uid inside that pod’s range;
2. every uid above the host’s user range (65,535) belongs to some pod with a user namespace.

Breaking rule 2 means a pod uid running outside its pod, that is, something that left the container. Breaking rule 1 means, for example, a real root inside a pod that should only have mapped uids. Either break is a strong sign of an escape.

My hypothesis was that, by watching these rules, I would find out which component had been compromised and perhaps the attacker’s IP. I wrote a script to test this on the Pi.

The check

The script is about 30 lines long, runs without root and walks /proc twice:

1. Builds the range table. /proc/<pid>/uid_map says which host uid the process’s uid 0 was mapped to and how many uids the range holds. A start of 0 means the process isn’t in a user namespace. For the rest, it stores the range along with the pod.
2. Checks each process. It reads the real uid from /proc/<pid>/status and applies rule 1 to processes of pods with a known range and rule 2 to everything else.

HOST_MAX=65535
declare -A RANGE   # pod -> "start end"
for p in /proc/[0-9]*; do
  { read -r _ lo n < "$p/uid_map"; } 2>/dev/null || continue
  [ "$lo" != 0 ] || continue
  pod=$(grep -oE 'pod[0-9a-f_-]{36}' "$p/cgroup" 2>/dev/null | head -1)
  [ -n "$pod" ] && RANGE[$pod]="$lo $((lo + n - 1))"
done
for p in /proc/[0-9]*; do
  uid=$(awk '/^Uid:/{print $2}' "$p/status" 2>/dev/null); [ -n "$uid" ] || continue
  pod=$(grep -oE 'pod[0-9a-f_-]{36}' "$p/cgroup" 2>/dev/null | head -1)
  if [ -n "$pod" ] && [ -n "${RANGE[$pod]}" ]; then
    read -r lo hi <<< "${RANGE[$pod]}"
    { [ "$uid" -lt "$lo" ] || [ "$uid" -gt "$hi" ]; } &&
      echo "ALERT pid=${p#/proc/} uid=$uid outside the range of $pod"
  elif [ "$uid" -gt "$HOST_MAX" ] && [ "$uid" != 4294967295 ]; then
    echo "ALERT pid=${p#/proc/} uid=$uid outside any pod with a user namespace"
  fi
done

4294967295 is skipped because it stands for “no uid”. The uid comes from /proc/<pid>/status rather than ps, because ps prints it as a signed 32-bit number: above 2³¹ − 1 it turns negative (uid 4,046,848,000 shows up as -248119296), and a filter like awk '$1 > 1000000000' would miss most pod uids.

The result

I ran it on the Pi on October 16, with Gitea, Mempalace, Postgres, the tunnels and the runners in user namespaces. The script matched the 22 high-uid processes to their 7 pods, with no alerts:

ok  pid=2605883 (gitea)       uid=3558147048  Gitea pod
ok  pid=2342646 (python3)     uid=839648232   Mempalace pod
ok  pid=2306821 (postgres)    uid=4017094726  Postgres pod
ok  pid=2307465 (cloudflared) uid=2633760764  tunnel pod

CI jobs show up the same way: the day before, the go process of a test pod was running as uid 432209920, inside its pod’s range.

A script that never alerts might simply be broken. To test the alert, I lowered the limit to HOST_MAX=999, and my own processes (uid 1000, outside any pod) showed up as ALERT … outside any pod with a user namespace.

What an alert tells me

The process’s cgroup says which pod it came from, and here each pod can be reached from outside in only one way. Gitea points to SSH or the web interface; a tunnel, to the internet; a test pod, to a poisoned dependency. The process’s connection table (/proc/<pid>/net/tcp, also readable without root) shows who it is talking to at that moment.

The first half of the hypothesis held: the check tells me which component was compromised.

What about the attacker’s IP?

Here the answer is more limited. The address the pod sees depends on the way in:

For SSH, the node forwards the connection to the pod, replacing the source address with its own; I tested from my desktop, and the pod logged 10.42.0.1. The original address lives in the kernel’s conntrack table, which only root can read. For the sites, the real IP arrives in Cloudflare’s header, and logging it is the subject of the next post.

Even when you can get the IP, it is rarely the attacker’s: attackers usually go through a VPN, Tor or a residential proxy, and a reverse shell usually points to a rented server or a machine that was already compromised. The IP is useful for blocking, reporting to the provider and correlating with other logs, but it doesn’t identify anyone. The exception is an IP from my private VPN: it means a device on my own network was compromised.

What this check doesn’t catch

• A kernel escape that becomes real root. An exploit that swaps the process’s credentials for the host root’s leaves a uid 0, not a strange uid. Rule 1 only catches it while it stays in the pod’s cgroup, and a real root can change cgroups or kill the script.
• Short-lived processes. A periodic check misses any process that starts and dies between two reads.
• Pods without a user namespace. The sites and apps still run with host uids (101, in nginx’s case). They would need a different rule, such as “uid 101 only exists inside a pod’s cgroup”.

The second item is solved by recording each event the moment it happens.

Event-based auditing with auditd

The Linux kernel has a built-in audit system. Rules say what to watch, such as “writes to this file” or “programs executed by this user”, and the kernel produces a record the instant the event happens, even for a process that lasts milliseconds. auditd is the service that receives those records and writes them to disk.

The Pi’s kernel already shipped with auditing enabled (CONFIG_AUDIT); all it took was installing auditd and writing the rules. It is the only component in this post that uses memory: a few MB, on the host, outside the cluster. I ruled out Falco and Tetragon because, besides the RAM, both depend on BTF, information about the kernel’s internal types that the Pi’s kernel doesn’t expose.

The rules cover only events that almost never happen by accident, so that every alert deserves attention. Each one has a key, used to filter the log:

The user-no-session rule

When someone logs in over SSH, the system tags the session with the audit uid (or loginuid), the uid of whoever logged in. The tag is inherited by every process in the session and can’t be changed. Cron and the user’s systemd apply the same tag, and I checked on the Pi that every legitimate uid 1000 process comes from one of those three. A container process descends from containerd, which systemd starts at boot, with no login, so it carries no tag. Therefore, a program executed as uid 1000 without an audit uid is the trace of something that left a container with my uid. The only exception on the Pi is k3s’s metrics-server, which runs as uid 1000 without a user namespace, but never executes another program and so doesn’t trigger the rule.

The install script ends with a self-test of each main rule. For this one, it uses systemd-run --uid=1000, which starts a process with no session, as a container would; the log recorded auid=unset uid=user, as expected.

What’s still missing

The audit log still lives on the Pi, and an intruder with root can erase it, along with the rules, before I look. An alert is only worth something if it leaves the Pi the moment the event happens. The plan is a notification service of my own, notification-chat, which will receive these events, read the sites’ access logs and send everything to my phone. Its design closes the next post.

For Gitea’s SSH, the next step is to log the real IP of whoever connects: with externalTrafficPolicy: Local on the Service, which delivers the connection without replacing the source, or with a conntrack -E recording every new connection.