This wasn’t the first time. A few weeks earlier I’d noticed the site down one morning — no SSH, but the TX/RX lights blinking normally. I power-cycled it, everything came back fine, and as a precaution I turned on persistent logging on the homelab. I left it at that and moved on.

Then, the day before I started writing this post, the Raspberry Pi that hosts this site went down again, out of nowhere. The site returned a 502, SSH hung, and I did what’s usually my first attempt at a quick fix: pulled the power, expecting it to come back. It did — but the instability persisted, and after a few minutes I decided to log into the Pi remotely.

It was slow: SSH had a noticeable lag on every keystroke, while htop showed nothing wrong. Pods in the cluster kept crash-looping, and my VPN client wouldn’t stop buzzing with alerts. That’s what got under my skin enough to start digging.

First: it never actually crashed

Before chasing a cause it’s worth ruling out the dramatic ones. The Pi is a 4 GB Pi 4B running k3s, and the obvious suspects on that hardware are power and heat.

$ vcgencmd get_throttled
throttled=0x0

0x0 means no under-voltage, no thermal throttling — not now, and not once since boot (the firmware sets bit 16 if it ever happened). Core temp was 38 °C. journalctl had logged continuously right up to the moment I cut the power, and the next boot started six seconds later.

So the operating system was alive the whole time. Load average 0.4. 1.2 GB of RAM free. No OOM killer, no kernel panic, no EXT4-fs error, no mmc0 I/O errors. The site’s own pod, when I finally got in, was using 1 millicore of CPU and 5 MiB of RAM and answering in under 10 ms from inside the cluster.

Whatever this was, the problem wasn’t the Pi being slow. It looked that way, but it was actually idle.

Suspect number one: Tailscale

The logs that were screaming came from tailscaled:

tailscaled: dns udp query: waiting for response or error from [192.168.***.1]: context deadline exceeded
tailscaled: health(warnable=no-derp-connection): error: could not connect to the 'Ashburn' relay server
tailscaled: netcheck: UDP is blocked, trying HTTPS

And cloudflared, which is what actually exposes the site:

cloudflared: ERR edge discovery: error looking up Cloudflare edge IPs: the DNS query failed
  error="lookup _v2-origintunneld._tcp.argotunnel.com on 10.43.0.10:53: server misbehaving"
cloudflared: Initiating shutdown

10.43.0.10 is CoreDNS, the cluster’s internal DNS service: it’s what turns names into addresses for everything running on it. But CoreDNS doesn’t resolve external names on its own — it hands those queries to the resolver configured in the node’s /etc/resolv.conf. That file points to Tailscale MagicDNS (100.100.100.100) and nothing else. No fallback. In practice, any service in the cluster that needed to find an address on the internet depended on a single DNS host: a single point of failure.

That explains the domino effect: the Tailscale connection wobbles → MagicDNS stops answering → CoreDNS returns an error (SERVFAIL) → cloudflared can’t locate Cloudflare’s servers → the tunnel container exits and goes into CrashLoopBackOff → the site is down. And SSH died with it, because Tailscale is how I reach the box.

The obvious fix was to stop betting the whole cluster’s DNS on one resolver. I added a coredns-custom ConfigMap so the tunnel’s control-plane domains resolve straight from public resolvers, independent of Tailscale:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  cloudflare-tunnel.server: |
    argotunnel.com {
        forward . 1.1.1.1 1.0.0.1 8.8.8.8
        cache 30
        errors
    }
    cloudflare.com {
        forward . 1.1.1.1 1.0.0.1 8.8.8.8
        cache 30
        errors
    }

It’s a good change. It treated a symptom but the tunnels kept dropping.

And Tailscale’s status page showed no incidents that day. Which meant Tailscale wasn’t the disease either — it was just the loudest patient in the ward.

Monitoring the Home Airwaves

I stopped reading logs and started measuring. I built a simple Bash script that ran on my laptop and, every 45 seconds, tried to SSH into the Pi. When it got in, it read the signal strength and the retry counter from /proc/net/wireless, the file where Linux exposes Wi-Fi link quality. When it didn’t, it logged the drop and how long it lasted.

The script caught a drop live. For about two minutes the homelab was completely unreachable — no SSH, no ping, ARP going unanswered. During that exact window, from a different machine on the same LAN, the router answered my pings at 1.4 ms with zero loss.

That was the turning point. Not the ISP. Not the router. The Pi’s own link to the network was dropping out, on its own, every few minutes, and healing itself before I could get a shell.

The homelab is connected over Wi-Fi. It does have an Ethernet port, but there’s no way to run a cable to where it lives.

Extra Noise Along the Way

While I was looking for the root of the problem, a few side effects made the picture muddier than it needed to be:

  • The tunnel restarting non-stop. cloudflared, which exposes the site, went into a crash loop: 24, then 28, then 32 restarts. Every restart writes logs, pulls its configuration again and re-registers with Cloudflare. All of that is disk writes, and the disk here is a microSD card, which is slow.
  • The investigation itself. Every command I ran to understand the problem (kubectl logs, journalctl, image inspection) read from and wrote to that same card. I was adding to the load I was trying to measure.

The result was a genuine bump in disk usage that made it feel like “the Pi is slow”. For a while I treated it as a resource shortage.

Every time I actually measured the backend — CPU, memory, in-cluster request latency — it was idle and answered instantly. The disk usage was a consequence of things failing and of me poking around. Worth remembering: when a system is unstable, a lot of what you observe is noise, not the fault itself.

Down to the physical layer

With the problem narrowed down to the stretch between the Pi and the router, I went after the Wi-Fi itself, one hypothesis at a time.

Wi-Fi power saving. By default, the Pi’s network card switches its radio off for brief moments to save power, which can cause small interruptions. The kernel log confirmed the feature was on (brcmf_cfg80211_set_power_mgmt: power save enabled). I disabled it:

nmcli connection modify <conn> 802-11-wireless.powersave 2

It helped a little, but the drops continued.

A USB camera. The night before, I’d plugged in a USB 2.0 webcam for a ustreamer experiment unrelated to the site, and it ended up sitting right next to the Pi 4’s onboard antenna. USB 2.0 devices and their cables are a well-documented source of interference at 2.4 GHz, the same band as the Wi-Fi. I unplugged it, and the retransmission rate dropped roughly 4×: the Pi had to repeat far fewer packets to be heard. The signal strength didn’t change, because that depends on the distance to the router; what went down was the noise around it.

Even so, it kept dropping.

The scan of neighbouring networks. This is where it got clear. I asked the Pi to list every Wi-Fi network it could see from where it sits:

SSID              CHAN  SIGNAL
Copa1002          10    84
Laura 802 2.4G      6    74
DICK               11    70
MY_HOTSPOT          1    56   <- mine
#CLARO-WIFI          4    50
Four-M               5    50
EVA                  4    49
#CLARO-WIFI          4    47
Zizi                 3    42
... 30+ networks total

My own access point, at signal 56 (on a 0–100 scale), came in weaker than three of my neighbours’. And there was a group of networks — two of Claro’s public hotspots, “Four-M”, “Zizi” — sitting on channels 3, 4 and 5, bleeding straight into channel 1, where my Pi was.

And the 5 GHz network on the same router, which is usually less contested? The Pi couldn’t even see it. A 5 GHz signal reaches less far and gets through walls worse, and from where the Pi sits that band doesn’t arrive. The Pi 4B’s antenna doesn’t help either: it’s small, built into the board, and has a noticeably shorter range than a dedicated adapter — a recurring complaint in the community.

How a channel gets crowded

Packet collisions and multiple access protocols are topics every software engineer learned from Kurose & Ross in a Computer Networks course back in college.

Plenty of people take that class and write it off as “stuff I’ll never use day to day” — but it turned out to be exactly the theoretical foundation I needed to read this problem correctly. So here’s that background, for anyone who isn’t already steeped in it — because it’s the whole story.

A Wi-Fi channel is one room where one person speaks at a time. Picture 20 people in that room: only one can speak at a time, and everyone else has to stay quiet to listen. That’s what half-duplex means. The protocol that arbitrates this is CSMA/CA, and it works like this:

  1. Before transmitting, a device listens. If the channel is busy, it waits.
  2. When the channel goes idle, the device doesn’t transmit immediately — it counts down a random number of time slots first (the “contention window”, or backoff). The randomness exists precisely so that two devices that were both waiting don’t jump in at the same instant.
  3. It sends one frame and waits for an ACK. No ACK → assume the frame collided or was lost → double the contention window → try again.

Now scale the number of devices in the room.

With 2 devices, that random backoff almost always lands them on different slots. They take turns cleanly. You lose a little airtime to the waiting (airtime being the time the channel is in use, the one resource everyone in the room shares), nothing more.

With 20 devices all with something to send, the probability that two of them pick the same slot and transmit together stops being negligible. Every time that happens, it’s a collision: both frames are destroyed, both senders get no ACK, both wait again and retransmit — and those retransmissions are now extra traffic competing for the same slots, making the next collision more likely. More load produces more collisions produces more retransmissions produces more load. Past a certain crowd size, adding traffic makes the total useful throughput of the channel go down, not up.

So how crowded is my room? My access point plus its handful of clients is maybe 4 radios. The building around me shows 30+ access points, each with its own phones, TVs, cameras and laptops — call it 60 to 100 radios within earshot. And on 2.4 GHz there are only three non-overlapping channels (1, 6, 11). All of that traffic funnels into three lanes: roughly 20–30 devices competing for each lane, all taking turns in the same room, all waiting on each other.

The retransmission counter I kept watching was exactly that probability made visible. It wasn’t one loud neighbour: it was dozens of ordinary ones. Pure arithmetic.

Two things make it worse than the arithmetic suggests:

  • Hidden nodes. CSMA/CA depends on every device being able to hear every other device. In a building full of walls, device A and device C often can’t hear each other but can both reach the access point. Neither one defers to the other — they can’t tell there’s anyone to defer to — so they transmit straight over each other at the AP. For every such pair, “listen before talk” simply doesn’t function.
  • Channel overlap. Which is the next section, and the reason a neighbour on “a different channel” is still in your room.

Why a “different channel” still collides

A 2.4 GHz Wi-Fi channel is 20 MHz wide, but channel numbers are only 5 MHz apart. Channel 1 is centred at 2412 MHz and occupies roughly 2402–2422 MHz. Channel 3 is centred at 2422 — its signal lands from ~2412 to ~2432. It covers half of channel 1. Only channels 1, 6 and 11 are far enough apart to not overlap at all. Three usable channels, for a whole apartment building — that’s why there are only three lanes.

Being on the same channel isn’t the worst case, actually — two access points on channel 1 can decode each other, so “listen before talk” works and they take turns. Slow, but orderly.

Overlapping channels are the bad case, because nobody there can take turns: my radio doesn’t understand the neighbour’s, so it doesn’t know it should wait. Tuned to channel 1, it cannot decode a channel-3 transmission — to it, that’s just background noise. In detail: the “is the channel busy?” check has two parts. The first is recognising the start of a valid Wi-Fi transmission (the preamble), and it fails, because the radio can’t decode channel 3. The second is measuring raw energy in the air, but the threshold is higher, and a medium-strength neighbour doesn’t trip it. So my Pi thinks the air is clear, transmits, and the neighbour’s overlapping energy corrupts the frame anyway. No ACK, retransmit, repeat. “Listen before talk” can’t defer to a signal it doesn’t recognise as Wi-Fi.

That’s the “conflict between networks I don’t control”. There is no setting on my side that fixes a neighbour’s router on channel 3 — it just makes my crowded room even more crowded, with devices I can’t even take turns with.

The actual fix: the router was still speaking 802.11b

Here’s the part I didn’t see coming — and the part that made the whole afternoon click into place. Months earlier, my ISP had done maintenance in the area that knocked my modem offline for a while. When it came back, it had been factory-reset. I logged in only to turn the 5 GHz radio back on and didn’t check anything else. In the router’s 2.4 GHz advanced settings, Modo de Operação (“Operation Mode”) had reverted to 802.11b — and sat there, unnoticed, ever since.

In all those months the site had never gone down over it, because the channel still had headroom. What changed wasn’t the router: it was the building. New Wi-Fi access points show up around me every week, and it was that growing density that, over the past few weeks, finally tipped the balance.

802.11b is the original 1999 standard: slow, and it forces everyone around it to slow down too. It caps at 11 Mbit/s and has none of the techniques that make modern Wi-Fi fast (OFDM, MIMO, frame aggregation, block-ACK). Every single frame is preceded by a 192 µs preamble sent at 1 Mbit/s, a kind of slow announcement before each message. And when a b-only device is on the channel, every faster device first has to send a request for permission, also slow (the RTS/CTS protection handshake), so the b radio doesn’t transmit over it.

So my Pi — which supports 802.11n perfectly well — was pinned to 11 Mbit/s, with most of the airtime going to preambles and protection overhead, on a channel already being corrupted by the neighbours. Real throughput was around 2 Mbit/s, and collapsing under any loss.

It also makes every point in the previous section worse: slower frames occupy the room longer, so each transmission blocks more would-be senders, so the collision probability for everyone else goes up. An 802.11b access point in a crowded band is not just slow for its own clients — it’s a tax on the whole channel.

I changed the mode to 802.11n (b/g/n). Measured before and after:

802.11b802.11n
Negotiated rate11 Mbit/s130 Mbit/s
Retransmissions15–50 / min~1–6 / min
Pi ↔ internet, between interference windows~2 Mbit/s~15 Mbit/s

About 7× more headroom on the stretch between the Pi and the wider internet. cloudflared stopped tearing down all four QUIC connections on every blip — its logs had been full of:

WRN Serve tunnel error error="failed to accept QUIC stream: timeout: no recent network activity"
WRN Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity"
INF Retrying connection in up to 8s

That cascade — all connections timing out from inactivity at once, then retrying with longer and longer waits — was the mechanism behind the “page takes 40 seconds to load” reports. With OFDM and aggregation doing the same work in a fraction of the airtime, brief interference stopped being enough to knock the tunnel over.

What’s still broken

802.11n bought back stability and speed. It did not buy immunity.

I ran the same script for 30 minutes after the change and it still caught drops — mostly short, but one full 4.5-minute blackout. The adjacent-channel collisions are still there, because they were never mine to fix. On a good stretch the link is now genuinely fine; when a neighbour starts moving real traffic on channel 3, my channel 1 still suffers.

The definitive fixes are the ones that don’t involve the air at all:

  • Powerline (PLC) adapters — one at the router, one at the Pi’s eth0. Wired reliability over the house’s existing electrical wiring, no cable run.
  • Move the Pi within 5 GHz range of the router — that band is nearly empty here.

Both are on the list. In the meantime the site is up, it’s fast between the interference windows, and I understand exactly why it does what it does.

In the end, what stuck with me most was the coincidence. I live in a commercial building where new Wi-Fi access points show up weekly. A forced modem reset from an ISP maintenance window left a 1999 protocol switched on without me noticing. A USB webcam parked on the wrong side of the antenna. The distance to the router. None of these events are related to each other — and together they added up to something like a poetic coincidence that took the site down. I’m glad I got to learn something from all of it.

Takeaways

  • The loudest error is rarely the root cause. tailscaled generated 90% of the log volume and 0% of the fault. It was the first component to notice the network was gone, not the reason it was gone.
  • Rule out layers top-down. Power, then thermals, then compute, then the app, then the network, then — last — the physical link. I’d have saved hours going in that order instead of chasing the noisiest logs.
  • An unstable system mostly shows you noise. The disk usage, the crash-loops, the restart counters — all real, all a consequence of one intermittent link.
  • A crowded channel is a probability, not a volume. Collisions and retransmissions scale with the number of radios contending for the same slots — and past a point, more traffic means less throughput.
  • “Different channel” is not “no interference” on 2.4 GHz. Only 1, 6 and 11 don’t overlap, and “listen before talk” can’t protect you from a signal the radio can’t decode.
  • Check the Wi-Fi operation mode. An access point quietly set to 802.11b taxes every device on the channel, not just its own. It’s one option in a menu, and it was worth roughly 7×.
  • Measure before and after. “It feels faster” is not a result. Retransmit rate, negotiated rate, and a download between the bad windows are.