Role-separated infrastructure, steered by GitOps

Problem

When production, development, CI runners, databases, and backups share the same machines, every change has the wrong blast radius. A heavy pipeline job starves an app. A restore test locks a live database. A “quick SSH” on prod drifts from what anyone remembers.

Environments that should stay apart leak into each other. There is no single place that says what the platform should look like. Control is tribal knowledge: who has the key, who ran the last command, who will notice when a host dies.

The organization needed dedicated roles for each plane — and a way to steer, change, and maintain them through git, not through ad-hoc sessions on the box.

System Design

This project splits the estate by role, then drives it from a declared desired state.

• Production hosts only live traffic. No CI, no experiments, no backup jobs competing for disk.
• Development is a separate environment for teams to break things without touching production.
• CI/CD and background jobs run on dedicated runners: lint, build, test, scheduled work — never on the production cluster.
• Application cluster is where services are deployed. Pipelines ship images here; they do not SSH in and mutate containers by hand.
• Database lives on its own hosts so storage, failover, and access stay independent of app nodes.
• Backup is a separate plane: copies leave production, restores are tested there, retention is not an afterthought on the same disk.
• Platform / control holds the maps: inventory, compose and stack files, secrets references, and runbooks. Changes go through git.

GitOps is the operating method. A merge updates the desired state. Automation converges the matching plane. Review, history, and rollback live in the repository — not in someone’s terminal history.

Architecture Decisions

• One role per class of host so failure and load stay inside that plane
• Production isolated from development, CI, and backup workloads
• Deploy to the cluster from git-declared stacks, not from a login session
• Database and backup off the app cluster so data survival does not depend on app nodes
• CI runners as a background plane: they build and test, they do not host the product
• Desired state in git; apply is automation, not a checklist on Slack
• Pull requests as the control path for infrastructure change and maintenance

Tradeoffs

• More hosts than a single mixed box — traded for a smaller blast radius
• GitOps is only as good as the declared state; a leftover SSH change is still drift
• Splitting planes needs a network and identity story (who may talk to the database, who may deploy)
• Backup on a separate plane is useless if restore is never rehearsed
• A catalog of roles takes discipline; collapsing two roles “just for now” recreates the original problem

Production Behavior

A change starts in git: stack file, inventory, or pipeline. After review it lands on the matching plane — development, production cluster, database, or backup — without logging into the box to “just apply it.”

CI jobs stay on the job plane. Live traffic stays on production. Data stays on database hosts; copies leave on the backup plane. Day-to-day control and maintenance is a merge, a converge, and an audit trail. Built for an organization that has to keep tens of services running without treating every server as a general-purpose machine.