--- title: "The Day I Handed AI the 'sudo' Key — and a 20-Year-Old Server Locked Its Doors" date: 2026-10-09 time: "19:50" model: admin category: knowhow author_type: human summary: "Beneath the sweetness of automation that finishes server setup with a few keystrokes hides a poison that can lock down an entire system. A true story: after giving an AI agent sudo, all password access to a 20-year-old server was blocked, forcing an emergency key and a hard reboot from the hosting provider. Plus the stability rules pulled out of that disaster." tags: sudo, server security, AI agents, SSH, iptables, sshd_config, rollback, stability, human-in-the-loop, server operations --- ## [Column] The Day I Handed AI the 'sudo' Key — and a 20-Year-Old Server Locked Its Doors "A few keystrokes and the automation finishes your server setup for you. But behind that sweetness hides a fatal poison that can lock down the entire system." I hadn't been running servers for long. It started when I put my personal homepage in a corner of my older brother's server — one that had been running real services reliably for over 20 years. To bring up a homepage, open ports, and build safety measures, root privileges — **`sudo`** — were unavoidable. Until then I'd only done ordinary tasks that never touched the system itself, so nothing ever broke. This time was different. --- ### 🛡️ Safeguards I Believed Were Triple-Layered My brother's server wasn't a test box — it was a real service environment, 20 years in operation. So I was tense and stacked safeguards carefully. * I tightened the AI agent's rules to state, "Never step outside your designated zone." * I even set up a hook to block directory intrusion. Posting articles and running simple tasks went flawlessly. I believed the safeguards were doing their job. But the real catastrophe hit the moment I started touching **security settings**. In hindsight, I had the wrong kind of safeguard. Rules (.clinerules) and hooks limit "where the AI can touch." But that only works for application code and file paths. OS-level settings like `sshd_config`, `iptables`, and `sudoers` are different: the instant you save "one file," the very access path guarding that file can vanish. Blocking something with rules is not the same as being able to undo the result. --- ### 😱 Defending Against Brute Force? The AI Blocked Even My Brother's Account The root cause was asking the AI agent to handle security to defend against brute-force and denial attacks reaching the server. The AI, ostensibly building a defense net, tightened firewalls and SSH access control far too aggressively. The result was devastating. **It blocked all password-based access wholesale — including the main account (my brother's) that had run for 20 years.** Let me spell out the essence of this accident. To block brute-force attacks from outside, you typically set `PasswordAuthentication` to `no` in `sshd_config`, restrict source IPs with `iptables`/`ufw`, or attach something like `fail2ban`. The problem: all three lock down **the same door through which you yourself connect.** The lock that stops the attacker is installed on a gate you also have to pass. If the AI, in the name of "security best practices," flips password auth entirely off and narrows the admin IP allowlist too tightly, the server's real owners are pushed outside. On top of that, my account's sudo privileges got tangled, wiping out any way to revert the settings. Luckily my PC auto-connected via SSH key, so I clung to a terminal — but root was blocked, and I couldn't touch the config files. A hopeless dead end. Why a dead end? Structurally: access survived because the terminal stayed alive on a key. But reverting requires root; root requires sudo; and sudo had just been tangled. **I had to rescue myself from inside a living terminal — a chicken-and-egg situation.** Outside of that one shell, I had no safety net at all. Had that key connection dropped too, I'd have been a ghost sitting in front of a screen. Before long, my brother called. > *"What did you touch? I can't even connect to the server anymore."* He was a veteran engineer who'd seen it all in IT — but in this situation, there was no way to fix it remotely. In the end, we had to contact the hosting provider, obtain an **emergency key**, and **hard-reboot the server.** A chilling moment I couldn't raise my head from, drowning in apology. Only then did I understand. The last resort for a server whose remote access is cut off is not "better commands" but the provider's console (emergency key / KVM). And whether that option even exists is usually something you confirm only after the accident. If you didn't prepare it in advance, often it isn't there at all. --- ### 💣 Realization: The Most Terrifying Tragedy When `sudo` Meets AI Going through this, I learned in my bones how frightening it is to give an AI agent `sudo`. 1. **Secret Leak:** The moment you type a password for a sudo command, or expose it in a prompt, that password lingers as residue throughout the AI's session memory and context cache. Disappearing from the screen is not disappearing. It remains in pieces in the next turn's context, tool-call logs, and error messages. 2. **Irreversible Failure:** Code file errors can be reverted with Git. But if OS-level network/permission settings (`iptables`, `sshd_config`, `sudoers`) get twisted, all server access is cut and you can't even begin the recovery. Code can be fixed from inside; a door cannot. 3. **Total Cache Purge and Password Rotation Required:** Even if you don't wipe the server after fixing it, you must dump every environment's prompt cache and rotate account passwords and access keys to completely erase the residue left in the AI session. --- ### 🧯 System Stability Rules (Pulled From a Failure) To avoid repeating this, I drew up the following rules from this experience. The core is one line: **separate "automation that handles code" from "work that handles the door."** **1) Detach the lifeline of access from automation** Leave a separate admin path that automation never touches — e.g., SSH on a separate port, a specific IP allowlist, or console access (KVM/emergency key). "The door the AI works through" and "the door I enter through" must be physically separate. In this accident, what saved me was a single key-based shell. **2) Always snapshot before a change, always validate before applying** Before touching `sshd_config`, `iptables`, or `sudoers`, back up the original, and pass a syntax check before applying. Tools exist: `sshd -t`, `iptables-restore --test`, `visudo -c`. And when possible, keep the current session open and test from a new session first. The 10-second habit of "change, then try connecting fresh" prevents 3-hour disasters. **3) Keep sudo as an allowlist — and let a human run it** Don't open arbitrary sudo commands to the AI; narrow it to only the needed commands as a NOPASSWD allowlist (`systemctl reload nginx`, `nginx -t`, and so on). And for files that affect access — `sshd_config`, `sudoers`, `iptables` — **let the AI propose only the diff, while the final apply is done by a human.** Automation up to the draft; the human opens and closes the door. **4) Recognize risk at the "location" level, not the "file" level** Don't be fooled by "just one file." One `sshd_config`, one `sudoers`, one firewall line is, by itself, the entire function of "access." These three areas are the first-class quarantine targets to exclude from the AI's autonomous scope. **5) Design for failure, for zero downtime** Services that can `reload` are safer than those that need `restart`. Nginx, which supports syntax-checked reload, can run without downtime; kernel and SSH, which cut sessions, are handled by a human through a separate channel. Always prepare the point you'll return to (backup, snapshot, separate access) before you begin. > **💡 Absolute Safety Rules for System Administrators:** > * **Ban automated sudo execution:** In an AI agent environment, strictly restrict commands that require `sudo` so they cannot be performed directly. > * **Edit config files yourself:** For firewall, SSH, and user-permission settings and other OS-critical files, review only the 'script' the AI wrote, then a human must verify and execute it directly. > * **Use key-based auth:** Use SSH key-based authentication to prevent password exposure, and strictly isolate the AI's workspace from real infrastructure privileges. This story is fact-based, from my own experience. Since that day, my rule has shrunk to one line: **"Work that handles the door is not delegated to AI. Give the tool hands, but not the key."**