The ticket arrives on a Tuesday: payments are failing, and nobody knows why. Three engineers open three chats. The deploy logs say nothing changed. The git log says nothing changed. Somebody, at some point, changed something — and the system has no memory of it. If you have lived this Tuesday, you already know the symptom, and you have probably heard that GitOps is the cure. What follows is the full diagnosis: why the mystery keeps happening, what the actual disease is, and how the treatment works. We will also walk through the scariest version of the story, because the treatment has a failure mode of its own and you deserve to see it with the lights on.
The Symptom: A System With No Memory
Watch how the mystery forms. A developer hits a problem at 6 p.m., has a fix that works right now, and applies it directly to the live server. It is 6:05 p.m., dinner is at 6:30, and writing it down feels like tomorrow’s problem. The fix works. Weeks pass. Two more quick fixes land the same way. Now the running system differs from whatever documentation exists — the industry calls this drift, and it compounds silently. Each undocumented edit makes the next mystery harder, because the true configuration exists only in the founding team’s fading memories.
The cost is not the edits themselves. Some of them are even good ideas! The cost is the forgetting. A system without a memory cannot answer “what changed?” — and every outage investigation begins with exactly that question. Until the question gets cheaper, every incident stays expensive, every audit stays anecdotal, and every new hire inherits mysteries instead of documentation.
The Diagnosis, Step by Step
Run this diagnostic on any team, honestly, and the disease will name itself:
- Ask how many people hold login access to production. More than a handful? Each one is an untracked change waiting for a deadline.
- Ask when the last out-of-band edit happened. If nobody can remember, it was recent.
- Ask how the team would reconstruct last Thursday’s configuration. Shrugs are diagnostic.
- Ask what happens to a hand-applied fix after the next deployment. If the answer is “it vanishes”, you have both drift and churn, feeding each other.
Three or more painful answers out of four, and the team does not have a staffing problem or a testing problem. It has an operating-model problem: production changes flow through human memory instead of through a record. No amount of extra process meetings cures that; the record has to be structural.
The Treatment: GitOps in One Honest Paragraph
The structure is simple enough to fit here. The team writes the intended configuration into a version-controlled repository, treats that repository as the only route to production, and runs a small program inside the infrastructure that continuously pulls from the repository and reconciles the live system to match it. Hand-edits get overwritten on the next cycle — not as punishment, but because the written record is the authority, and a system that obeys yesterday’s memory is precisely what you are trying to escape. Changes happen by editing the record and getting the edit approved, which means every production change arrives with a name, a time, and a rationale attached. The treatment works because it removes the forgetting — structurally, permanently — by making the record the only place a change can live.
Expect an uncomfortable first fortnight, and plan for it. The moment reconciliation switches on, it will enumerate every difference between the record and reality — and on a team with our disease, that list is long. Each difference forces a small verdict: write it into the record or undo it in the live system. Treat the exercise as an inventory rather than an accusation; every item resolved is a mystery that can never return. Teams that rush this stage tend to widen the record’s permissions so the noise stops, which simply relocates the disease into the repository. Sit with the list. It is the most honest audit the team has ever had for free.
When the Treatment Goes Wrong: A True-to-Life Scare
Now the scary story, because trust requires it. Picture a production setup where the recorded configuration includes a file describing the payments area — its namespace and the storage volume beneath it. During a routine merge, a conflict resolution goes sideways and that file silently drops out of the production folder. The reconciler, faithful as ever, notices that the live system contains something the record no longer describes. Pruning is enabled. The payments namespace is removed. Because the storage volume carried a delete-on-removal policy, the data goes with it.
Nothing in that chain was malicious or even careless in the human sense — every component did exactly what it was configured to do. The reference write-up we lean on, the Ops Atlas write-up on GitOps, documents this exact failure shape with a six-step runbook for surviving it, and the steps are worth knowing before you ever need them.
The Six-Step GitOps Runbook, in Plain Order
- Freeze the machinery. Stop reconciliation before anything else moves. In Flux, a suspension marker does the job and survives later changes because it is not itself declared in Git. Argo CD needs more care — set ignoreDifferences on the parents too, or the parent objects will helpfully undo your freeze.
- Assess without touching. Gather application history, recent events, and the repository log. Establish what was removed and when, while the deletion engine sits frozen.
- Revert through the front door. Restore the missing file by pull request, not by hand — the same reviewed path that protects you on ordinary days protects you now.
- Bring the data back. Configuration alone restores nothing; the volume contents come from backup. This step is where teams discover whether their backup restore actually works. Test yours this quarter, ideally before any of this happens.
- Resume and verify. Unfreeze reconciliation, watch the sync complete, and confirm the restored namespace behaves like its old self.
- Close the hole. Turn off pruning for state-bearing resources, switch storage to a retain-on-delete policy, and let a pipeline check raise a flag whenever a proposed change would delete namespaces, volumes, or custom definitions — before the change can merge.
Step six is the whole point of telling the story. The scare is survivable; repeating it is not.
The Follow-Up Prescription
After treatment, the Tuesday ticket changes character completely. The question “who changed the server?” now has an answer on file — a commit, a reviewer, a timestamp — and the system quietly returns itself to the recorded state overnight. Investigating an outage becomes reading, not archaeology. Teams describe the sensation as getting their evenings back, and having watched both versions of that Tuesday from up close, we would not argue.
Getting evenings back is a theme this site endorses from every angle. If the diagnosis has you rethinking how your team works, start with the lighter reading — our guide to exploring Gujarat beyond the popular tourist attractions is a palate cleanser, the best experiences for first-time visitors to Gujarat work nicely as a team outing, and if the team really needs distance from the incident channel, the motorcycle and car rental guide to Colombia is about as far as a long weekend can go. The servers, meanwhile, will hold their recorded state — which is, after all, the entire point.
