Two years ago this was a laptop running one VM. Today it’s a two-node Proxmox cluster, thirty-plus services, one person on call, and a domain full of *.jigo-ai subdomains that all sit behind the same SSO gate. Nobody hands out an SRE badge for a homelab, but I’ve been quietly doing the job for two years now, and I’ve got the incident history to prove it.

This isn’t a “look at my rack” post. It’s the honest version: what broke, what I built because it broke, and what’s still held together with more discipline than architecture.

Where it sits today

The topology is two physical Proxmox nodes joined into one cluster: pve-laptop runs the main jigo-ai VM where most services live, and a second box runs a handful of LXCs — Pi-hole for DNS, a downloader stack, a media/servarr stack — plus a Kali VM I use for pentest tooling. They talk to each other over Tailscale, which quietly became the thing holding the whole setup together after a firewall incident once split the cluster in half and I had to learn asymmetric routing under pressure.

Every public-facing service sits behind Authentik forward-auth by default — that’s not optional, it’s a deploy-checklist hard rule now, for reasons I’ll get to. Everything that’s supposed to stay up gets a memory guard in its systemd unit. Also not optional, also earned the hard way.

I didn’t plan any of this in advance. I built the smallest thing that solved a problem, it worked, I wanted the next thing, and two years later “the next thing” adds up to a real system that occasionally reminds me it’s a real system by breaking in real ways.

The crash that wrote a rule

Early on, a service went into the wild with no memory ceiling on its systemd unit. It leaked, or spiked, or just ran hot — doesn’t matter which, because the failure mode was the same either way: nothing was stopping it from eating RAM until the whole VM went down. Not the service. The whole VM. Every other service on the box went with it, because Linux’s out-of-memory killer doesn’t know or care which process you’d prefer it kill — it picks based on its own heuristics, and sometimes it picks wrong, and sometimes by the time it picks, the box is already too far gone to recover gracefully.

That’s the day MemoryMax and MemoryHigh stopped being something I’d get around to and became a hard rule: no service goes live without cgroup memory limits, sized by tier — 128M for a small notifier or watchdog, 256M for a light API or proxy, 384–512M as the default for a standard Node/Express service, up to 1–1.5G for anything Next.js-shaped. Paired with a crash-loop guard (StartLimitIntervalSec=60, StartLimitBurst=5) so a service that’s actually broken fails loudly and stays down instead of thrashing the host trying to restart itself forever. earlyoom runs as a system-wide backstop underneath all of that, but the cgroup caps are the real first line — they mean the OOM killer only ever gets to shoot the one service that deserves it, not whoever else happened to be running.

I still audit for this. It’s a one-liner:

for f in ~/.config/systemd/user/*.service; do
  grep -qE 'Memory(Max|High)' "$f" || echo "$f"
done

Any unit that prints is a defect, not a todo.

SSO taking itself down

The second one is more embarrassing, because by the time it happened I had a memory-guard rule — I just hadn’t applied the same skepticism to a piece of infrastructure I trusted more than I should have: the Redis container backing Authentik.

Authentik’s session cache lives in Redis. That Redis container had a mem_limit of 256m, but its own maxmemory was set to 0 — unlimited, from Redis’s point of view — with maxmemory-policy: noeviction. Two settings that individually look reasonable and together are a loaded gun: the container has a hard ceiling, but Redis itself has no idea it should ever throw anything away.

The session cache grew. Slowly, then not slowly — it reached roughly 170,000 keys and about 305MB, well past the container’s actual limit, and the kernel started OOM-killing the cgroup. Not once. Every seventy-three seconds, on a loop, for as long as it took me to notice — 1,834 restarts by the time I pulled the numbers. And here’s the part that cost me the most debugging time: Docker’s own OOMKilled flag read false the entire time. Cgroup v2 doesn’t always surface that flag the way you’d expect, so the obvious “just check if Docker says it got OOM-killed” diagnostic lied to me. What actually gave it away was docker events --filter container=authentik-redis-1 showing a bare oom line, and the container logs looping “Starting Redis Server” with a clean exit code — a service that looked like it was restarting normally, if you didn’t know to ask why a stateless-looking cache needed to restart at all, every minute, forever.

Every *.jigo-ai service behind forward-auth returned a 500 during this. Not because the services were down — their local ports answered fine — but because the auth chain they all depend on was flapping underneath them. Thirteen gated apps, all “down,” none of them actually broken. So when everything behind SSO looks dead at once, check the auth layer before you touch a single app.

The fix was two changes to the Redis block in docker-compose.yml: --maxmemory 384mb --maxmemory-policy volatile-lru, and raising mem_limit to 512m so there’s headroom between “Redis decides to evict” and “the kernel decides for it.” volatile-lru is safe here specifically because only a handful of non-session keys in that Redis instance lack a TTL — everything else is a session, so eviction under pressure means someone has to log back in, not that state gets silently corrupted.

That wasn’t quite the end of it. A week and a half later I found roughly 11.3GB of orphaned temp-*.rdb files sitting in the Redis data volume — about eighty-two of them. Redis only deletes its snapshot temp file after a successful save, and every one of those OOM kills had interrupted a background save mid-write, leaving the temp file behind forever. The live dump.rdb was 260MB. The garbage pile was forty-three times bigger than the thing it was supposed to become. Cleanup was safe because Redis holds no open handle on a dead child’s temp file: docker exec authentik-redis-1 sh -c 'rm -f /data/temp-*.rdb'. Now a pile of temp-*.rdb files is itself a symptom I check for after any Redis incident, because it means an OOM loop happened even if I missed it live.

The other kind of mess: too many people watching the same door

Not every lesson here is a dramatic outage. Some of it is slower and dumber. At one point I had five different things independently monitoring service health — a watchdog script, two separate healthcheck scripts that both predated the current setup, a self-healing monitor, and Uptime Kuma — each with its own hardcoded list of what to check. One service going down could fire the same alert five times through five different paths. Decommissioning a service meant remembering to edit six different files, and I didn’t always remember, which is exactly how a retired service (threat-intel-v2, as it happens) kept generating phantom alerts weeks after I’d shut it down.

The fix wasn’t cleverer monitoring, it was fewer sources of truth. Uptime Kuma is now the one system that owns up/down alerting. A Proxmox-focused watchdog handles host-level metrics only — CPU, memory, disk, pending updates — and doesn’t duplicate service checks anymore. A self-heal script restarts things on a ten-minute cron, with a guard that skips any unit that’s been removed instead of restart-looping on a ghost. A morning digest summarizes the day. All four read from one JSON file listing every monitored service, instead of four different hardcoded lists drifting apart in four different scripts. A weekly job diffs that file against reality and complains if something’s monitored by nobody, or listed but gone.

It’s not exciting. It’s also the change that’s saved me the most actual weekly time, because “which of these five alerts do I believe” was a worse problem than any individual outage.

What’s still duct tape

Honestly: the SSO setup has two separate Authentik outposts now — one for a client-facing brand, one for the homelab — and keeping the mental model straight for which one gates which subdomain is a thing I still get slightly wrong under pressure. Secrets are sops-and-age encrypted at rest, which is the right call, but entering a new one still means opening a separate terminal and typing it by hand, because I don’t trust myself to paste secrets through anything that logs. And the weekly monitoring-drift job exists precisely because I know I will eventually forget to register a new service properly — the reconciler is there to catch me, not because I’ve stopped needing to be caught.

None of that is a complaint. Two years of “I built the smallest thing that worked, then found out what it didn’t handle” is a genuinely good way to learn systems engineering, as long as you’re willing to write the postmortem and actually change the default instead of just fixing the one instance. The memory-guard rule and the Redis fix both came from the same instinct: don’t ask “did this specific thing break” — ask “what class of thing was this, and where else does it exist right now.” That question is more useful than any monitoring dashboard I’ve built, and I’ve built a few.