< the skull

THE DOORS I FORGOT WERE OPEN
============================
2026-06-20

Notes on tending a server that quietly grew into doing everything.

It began with a feeling, which is the worst possible way to begin anything
technical.

The server felt like it kept falling over. Every so often the load average would
drift past two, the terminal would repaint its banner, and some old reflex would
mutter there it goes again, rebooted. I believed that reflex for longer than I'd
like to admit.

The honest first move in any optimization is to stop trusting the symptom and go
look at the number underneath it. Load average might be the most misread figure
in all of Linux. People read it as a CPU gauge. It isn't. On a twelve-core
machine, a load of two means the box is half asleep — better than eighty percent
of its cores idle, waiting for something to do. The figure that actually
mattered was idle time, and idle time sat at ninety-four percent. The machine
wasn't drowning. It had never been drowning. And those reboots I was so sure
about? The uptime counter climbed the entire time. Machines that reboot don't
count days; they start again from zero.

So there was no performance problem. That's a real result, not a disappointment.
"Nothing is wrong here" is a legitimate answer, and hunting for phantom gains on
a healthy machine is just a more sophisticated way of breaking it.

But once you stop asking why is it slow and start asking what is this thing
actually doing, a different picture shows up. This was a server that had grown,
without my quite noticing, into doing everything — a handful of websites, a git
forge, an IRC bouncer, a small app or two I'd forgotten I installed, a sediment
of services laid down over years. And every one of them had, at some point,
opened a door.

So I listed what was listening, and it was not a short list. A database bound to
`0.0.0.0` — every interface, the public one included. A second database doing
the same. A container-management dashboard, the kind of thing that hands you the
entire host if its password ever leaks, answering cheerfully on a public port.
File sharing. A proxy I'd set up for some reason that had long since died and
taken its justification with it. Each of these had been opened deliberately,
once, for a purpose that no longer existed, and then simply left.

Here's the part that catches almost everyone, and it caught me: a firewall is
not enough when you run Docker.

The mental model most of us carry is that the firewall is the bouncer at the
door and nothing gets past it. But Docker writes its own packet-forwarding rules
to make published ports reachable, and those rules are consulted before your
tidy firewall policy ever gets a vote. You can have a deny-by-default firewall,
feel completely safe, and still have a dozen container ports wide open to the
internet. I did. The policy said one thing; the actual path a packet would take
said another. That gap is the single most useful thing I relearned this week:
trust the path, not the policy.

The fix wasn't clever, only disciplined. Anything that already sat behind the
reverse proxy didn't need a public port of its own — the proxy reaches it over
loopback, on the same machine — so I rebound those services to `127.0.0.1` and
the outside door quietly closed. Nothing changed for visitors; they were always
arriving through the domain and the proxy. The direct shortcut, the one only an
attacker would ever bother with, just stopped existing.

Everything else — the databases, the dashboards, the things only I ever touch —
moved onto the mesh. I already run a WireGuard-style mesh VPN across my
machines, and once you have one of those, the public internet stops being the
place you administer anything. A service bound to its mesh address is invisible
to the world and a single hop away from me. The database that had been sitting
exposed for who-knows-how-long became reachable from my own devices and from
nowhere else on earth.

Then I deleted things, which is badly underrated. The app I never opened. The
proxy with no purpose. A couple of accounts belonging to people who'd moved on
long ago. Every one of them was surface area I'd been defending for no reason.
Removing a service you don't use is worth more than hardening one you do — there
is no more secure version of a thing than its absence.

The remainder was housekeeping of the small, dignified sort. A login banner that
had been quietly lying about half its fields, because a language setting had
broken the way it parsed memory and processor, got rewritten to read straight
from the kernel instead of from translated labels. Stale accounts trimmed. The
kind of work nobody thanks you for and everybody quietly benefits from.

And then, near the very end, the original ghost finally showed its face.

The thing that had convinced me the server was falling over wasn't the server at
all. It was the IRC link — my client, at home behind an ordinary router, talking
to the bouncer across the open internet. Home connections are restless animals.
The router's translation table forgets an idle conversation; the next packet
lands on a socket that's already dead; the link snaps; the client reconnects;
and from the outside it looks exactly like a machine that keeps collapsing. It
wasn't. The machine's uptime was measured in serene, uninterrupted days. The
road to it was what kept washing out.

The remedy was the same idea as all the rest: stop driving on the public road. I
pointed the client at the bouncer through the mesh instead of across the open
net. The tunnel keeps itself awake, the translation-table timeouts stopped
mattering, and the connection that used to die on the hour simply held. One
steady thread where there had been a churn of dying ones.

None of this made the server faster, because the server was never slow. That was
the lesson folded inside the whole exercise. We reach for the word optimization
and picture squeezing more speed out of something, but most of the real work on
a mature system is subtraction: closing doors you forgot you opened, deleting
what no longer earns its keep, and learning to read the instrument that tells
you the truth rather than the one that merely tells you something.

The machine had been fine the entire time. I just hadn't been looking at it
honestly.

A server, in the end, isn't so much optimized as it is tended. And most of
tending is knowing what to take away.


< the skull