Distributed Convergence

Problem

Most estates have parts and call themselves a platform. A database here. A reverse proxy there. A pipeline copied into the next repo. A chatbot on someone else's cloud. When a node dies, someone SSHs. When a team grows, someone opens a public port. When a vendor changes policy, the organization discovers that identity, mail, files, and conversation were never theirs.

Donella Meadows' definition is stricter than that. A system is not a pile of elements. It is elements plus interconnections plus a function or purpose, coherently organized. Change the elements and you still have roughly the same system. Change the interconnections and behavior shifts. Change the purpose and you have a different system wearing the old names.

The failure mode of modern infrastructure is not missing software. It is an estate that cannot pass that test:

• Elements without owned interconnections. Services exist. Discovery is a hostname someone memorized. Routing is a spreadsheet of ports. Failover is a runbook that starts with "log in."
• Interconnections without an owned purpose. The cluster is "up." The data lives in a region the legal team cannot name. Chat is in a public messenger. The purpose the system actually serves is the vendor's billing cycle, not the organization's ability to keep working when a machine, a person, or a contract disappears.
• Purpose declared, behavior denied. GitOps on the wiki. SSH in the history. High availability on the slide. A single primary in production. Monitoring on the same hosts it is supposed to watch.

A rented abstraction is convenient until the failure lives inside someone else's binary. Packaged orchestrators hide the machine. Hosted backends hide the data. Public mailboxes hide identity. Copy-paste CI hides drift until forty apps disagree. Mixed hosts hide blast radius until a CI job starves production.

The organization that has to survive itself — growth, node loss, people leaving, audits, partitions between sites — does not need more tools. It needs the estate to become a system: something that can be read, something that converges, something that stays itself when a part is gone.

That is the problem this model names. Not "how do we deploy Kubernetes." Not "how do we buy HA." The problem is concentrated, unread, rented control — and the quiet way it stops an organization from understanding the machine it depends on.

Observation

Read the production work as one corpus, not fourteen tickets.

A cluster written from the kernel up: storage, Raft, API, node agent, process runtime, scheduler, reconciliation, service names, overlay, cgroups, TLS, rolling update, quorum of control planes. No packaged orchestrator in the middle. Every layer is code someone on the team can open.

Infrastructure split by role: production, development, CI, application cluster, database, backup, platform maps. Desired state in git. Apply is automation. A leftover SSH session is named as drift, not as heroism.

Relational data, document data, and cache each get the same shape: more than one node, automatic failover, a stable endpoint in front of a moving primary. Odd numbers for votes. Health checks so the dead leave the pool. Failover drills before the word production is allowed.

One reverse proxy for the cluster. TLS at the edge. Apps do not publish 80 and 443. Host and path rules. When a replica moves, the URL does not point at yesterday's host.

Tunnels between sites so internal networks and remote clusters reach each other without a public port per service. Gateways own the path. Peers are least-privilege. Addressing is planned so private ranges do not collide.

Then the work that looks like "apps" and is actually perimeter: private file sharing, organizational mail, chat injected into the products people already use, a self-hosted backend instead of a public BaaS, AI for support and back office that does not take transcripts off-site. Same move every time. Convenience is rebuilt inside the network. The vendor is not the source of truth for identity, files, mail, conversation, or process.

One catalog of CI templates for dozens of enterprise apps. One board for about a hundred checks, running off the production nodes, grouped by the same planes as the rest of the estate.

The repeating move is not "use Docker." It is not "prefer open source." Those are materials. The move is this:

Never let truth, control, or purpose sit on a single node, a single session, or a single vendor. Keep a loop running that pulls actual toward desired. From the outside, look like one platform. From the inside, be many replaceable parts.

That is systems thinking with the lights on. Meadows would call the inner shape resilience (the system keeps its function after disturbance), self-organization (the system can restore structure without a hero), and hierarchy (nested subsystems that serve the whole without sharing every fate). Distributed systems would call the same shape quorum, reconciliation, and blast-radius isolation.

There is a personal grain in the same wood. The public line is "I write code to understand myself." The poem is "be who you are." A cluster you cannot read is the infrastructure version of a self you cannot face. Opacity is not a neutral default. It is exile from your own system. Writing the control plane, putting desired state in git, refusing a black-box orchestrator, keeping chat on the document cluster you already operate — these are not taste. They are the same ethic at two scales: you do not get to become, or to operate, what you cannot read.

Model

Name: distributed convergence.

A platform is a system when three stocks are distributed and one loop never stops. Distribution here is not "more machines." It is the refusal to concentrate the things that, if lost, take the function with them. Convergence is not "a deploy script that ran." It is the permanent comparison of actual against desired, with a repair path that does not require a person in the chair.

Meadows' ranking, kept honest: the purpose of the estate (keep organizational capability alive under failure, without surrendering the ability to see) matters more than the wiring, and the wiring matters more than which binary sits on a node. Swapping Postgres for another engine with the same replication and failover story is a change of element. Putting writes through a single unwatched box is a change of interconnection. Handing identity to a public cloud is a change of purpose.

Three stocks that must not concentrate

Truth. What is the cluster? What is the primary? What should be running? Truth that lives in one file, one host, or one person's head is not truth. It is a single point of belief. Raft in-process, replica sets with an odd number of votes, desired state in git, a watch/event bus instead of polling, a monitoring board that is not the thing being monitored — these are different mechanisms for the same stock: a picture of reality that survives the loss of any one witness.

Control. Who may change the estate, and through what path? Control that lives in an SSH session, a shared root password, or a vendor console is not control. It is access. The model puts control on declared paths: pull requests, an API with RBAC, an agent that the control plane never SSHs into, tunnel peers that see only the prefixes they are given, CI that deploys a git-declared stack. A merge is an event the system can audit. A night on the box is not.

Purpose. Where does the organization's work actually live? Files, mailboxes, chat transcripts, auth, automation runs. If purpose lives on a rented inbox, a public messenger, or a BaaS whose exit path is a blog post, the system is serving someone else's function. Self-hosting is not a moral pose. It is keeping the purpose stock inside the perimeter that the rest of the model knows how to heal.

Lose one node in a quorum, truth remains. Lose one operator, control remains in the repository. Lose one site, purpose remains if the plane was designed to fail in place and restore from the backup plane — not if the only copy was a laptop or a tenant on a public cloud.

One loop that must not stop

Desired versus actual. Forever.

In the custom cluster this is the controller pattern: a process dies, the loop starts it; a node leaves, work is placed elsewhere. In GitOps this is converge-from-git: the merged state is what the plane should be; automation closes the gap. In data planes this is election plus pool: the primary disappeared, remaining members agree, the endpoint stops sending traffic to the dead member. At the edge this is membership: unhealthy tasks leave the pool. In CI this is the same stages, the same artifact tag, promotion instead of a snowflake rebuild. In monitoring this is poll, retry, state change, alert — not a person opening tabs.

A one-shot script is a photograph. A reconciliation loop is a system. Meadows' balancing feedback is exactly this: a gap is measured, a corrective flow runs, the gap shrinks. If the loop is slow, you get oscillation or collapse. If the loop is missing, you get heroes.

Hierarchy as planes

Meadows treats hierarchy as a natural systems structure: nested subsystems, semi-autonomous, serving the larger function without dragging every local failure up to the whole. The work already built this, under the name of roles and blast radius.

Production does not share a disk with CI. The database does not share fate with app nodes. Backup is a plane you restore onto, not a folder on the same volume. Monitoring does not run inside the thing it watches. Development is allowed to break. The edge is one door so the inside can move.

This is not "microservices." It is distributability of fate. A plane can fail, saturate, or be restored without rewriting the purpose of the whole. Hierarchy that cannot say no — every host a general-purpose machine — is not hierarchy. It is a commons waiting for tragedy.

Distributability is resilience, not fashion

Distributing a system does not mean sharding for a conference talk. It means the function survives the loss of a part.

• Replication is a buffer on the data stock.
• Quorum is a rule that truth is a majority, not a personality.
• Automatic failover is a balancing loop with a delay short enough that users do not become the detector.
• A pool or load balancer is an interconnection that hides which member currently holds the write.
• Names, VIP, DNS are interconnections that stop apps from hard-coding peers — because hard-coded peers turn topology into destiny.
• Namespaces and cgroups are rules that a noisy neighbor cannot eat the node.
• Rolling update and declared rollback are flows that change the process stock without emptying it.
• An odd number of voters is a parameter that only matters because the rule of majority already exists.

CAP is the distributed-systems name for a delay-and-partition fact Meadows would recognize: when the network splits, you choose whether to keep serving (and risk split-brain) or to wait for agreement (and stall writes). Primary-replica clusters in this corpus chose a single writer and a majority for promotion. That is a purpose choice: correctness of writes over the appearance of being up everywhere. Replica lag is a delay in the information flow. The model does not pretend the delay is zero. It puts a pool in front and tells the application to stop pinning a host.

Function survives when no single witness owns truth, no single session owns control, no single host owns a plane's fate, and a loop is still closing the gap.

Leverage, ranked the way Meadows ranked it

Most infrastructure work stays at the bottom of the list: numbers, buffers, a few stock-and-flow tweaks. The work in this corpus spends unusual time at the top.

| Meadows leverage | What it means here | Where it showed up | | --- | --- | --- | | Constants, parameters | Timeouts, replica count, CI intervals | HA member counts, check retries, connection pool size | | Buffers | Disk, replica lag room, backup retention | Persistent volumes, backup plane, cache in memory on purpose | | Stock-and-flow structure | Who talks to whom, where bytes sit | Planes, overlay, tunnels, edge termination | | Delays | Election time, poll interval, replica lag | Failover drills, retry-before-page, not pinning a host | | Balancing loops | Gap closes | Reconciliation, GitOps converge, health removing dead members | | Reinforcing loops | Gap widens unless cut | Copy-paste CI drift; mixed-host contention; SaaS lock-in | | Information flows | Who sees what, how fast | Git as source of truth; one uptime board; service discovery | | Rules | Who may act | RBAC, PR as control path, least-privilege tunnel peers, no public port per app | | Self-organization | Structure restores itself | Raft election, controller restarts, replica reconnect after failover | | Goals | What the system is for | Survive a node; keep data inside; look like one platform | | Paradigm | The worldview | Own the layer you depend on; unread is unowned | | Transcending paradigm | Holding the model lightly | Tradeoff sections: custom means you own the bugs too |

The cheap lever is "add a replica." The expensive, actual lever is "desired state is the only control path" and "the cluster is code we can read." Parameters without those rules just make a more expensive single point of failure.

Components

The fourteen systems are not a resume list. They are the parts of the model in metal.

Where truth lives

Custom cluster. Key-value store with persistence and watches. Raft: leader election, log replication, recovery when a member disappears. Cluster truth is a quorum. The API does not believe a local YAML on a node. The scheduler's view of CPU and memory is data, not folklore.

Postgres HA, MongoDB HA, Redis Sentinel. Streaming or in-memory replication. Odd voting membership. Automatic promotion. A pool or balancer as the only address applications know. Health so a dead member leaves. Persistent volumes where the stock must survive restart; deliberate absence of disk for cache, because that stock's purpose is speed, not archival.

Uptime monitoring. A catalog of checks as the board of "is it up." Groups follow planes. Alerts on state change, not on every failed poll. The watcher sits off production so a dead cluster cannot mark itself healthy. Information about the system is itself a stock, and it must not live inside the failure domain it describes.

GitOps maps. Inventory, stacks, secret references, runbooks in git. The repository is the declared truth of what the estate should be. History is not history in a shell.

Where control lives

Custom cluster control plane. Authn, validation, RBAC, TLS, certificates. Objects enter through the API. Agents register and heartbeat; the API never SSHs in to start work. Secrets are not env files in git.

GitOps infrastructure. Pull request as the control path. One role per class of host. Production isolated from development, CI, and backup. Deploy from declared stacks. Collapsing two roles "just for now" is named as recreating the original problem.

CI templates. One catalog. Apps extend hidden jobs. Same stages: lint, build, test, deploy. Branch rules and optional commit-message scopes. Swarm, Compose, and SSH-to-a-remote-engine as variants of one contract. A new app does not get to invent a pipeline. That would be distributing design when you meant to distribute work — the opposite of the model.

Cluster proxy. Label-driven routing so a new stack is reached without a human editing the edge by hand. Certificates issued and rotated at the proxy. The rule is: applications declare how they want to be reached; they do not take a public port.

Network tunnels. Peers and routes declared as infrastructure. A new site is another peer, not a DMZ of exceptions. A remote cluster does not automatically see the office LAN.

Where purpose lives

Nextcloud, SMTP, org-chat, Supabase, AI automation. These look like products. In the model they are the purpose plane: the actual work of the organization — documents, identity, conversation, application data, support and back-office flow — kept on infrastructure the other planes already know how to replicate, enter, and watch.

Chat is not a second tab in a public messenger. It is rooms and messages as documents on the replica set that already fails over. Mail is not a rented seat. Files are not a consumer drive. The backend is not a public BaaS. AI is not a vendor that keeps the transcript. Where the model uses a probabilistic step (classify, extract, draft), it keeps a hard rule or a human on money and access. Automation is not allowed to become a wall.

Purpose inside the perimeter is what makes the HA work for something. Replicating a database whose meaning lives in a SaaS you cannot leave is a balancing loop attached to the wrong stock.

Where the loop runs

Every component above has a loop or it is incomplete:

• Controller loop: desired workloads versus running processes.
• GitOps loop: merged state versus applied plane.
• Election loop: primary alive versus majority.
• Edge loop: healthy backends versus current membership.
• CI loop: commit versus an artifact that can be promoted.
• Monitor loop: target versus retry policy versus incident state.
• Backup loop: copy left the production plane; restore was rehearsed. A backup that is never restored is a reinforcing story you tell yourself.

Rules

These are not slogans. They are the constraints the projects already obeyed. A new piece of work that violates them is not "pragmatic." It is a different system.

1. Unread is unowned. If the scheduler, the runtime, or the consensus story lives in a binary you cannot change without fighting a product, you are renting the abstraction. Own the layer you are willing to be woken up for — or accept that you will debug through a vendor's window.

2. Truth is a quorum, not a file. Control plane, data, cache monitors: majority or it did not happen. Two nodes is a coin flip. One node is a wish.

3. Control is a declared path, not a session. Git, API, agent. SSH is an escape hatch that must be visible as drift. If the only way to fix production is to be the person who knows the box, the system is that person.

4. Fate is isolated by plane. Production, CI, data, backup, monitoring, development. Load and failure stay inside the plane. A restore test does not lock the live database. A pipeline does not starve the app.

5. The outside sees one door. Shared edge. TLS once. Names and paths, not a port per process. Internals may move. URLs must not.

6. Private stays private. Site-to-site fabric instead of a public port per service. Least-privilege peers. Internal traffic does not hairpin unencrypted through the public internet.

7. Purpose stays in the perimeter. Files, mail, chat, auth, automation transcripts. Rebuild convenience inside. Do not outsource the stock the organization is for.

8. The loop never stops. Reconcile, elect, health-check, converge, page on state change. A script that ran once is not operations.

9. Same path for many apps. Templates, not copies. A catalog change is a real event; version it. A unique pipeline per repo is drift with extra steps.

10. Watch from outside. The monitor must survive the thing it watches. A container must not certify its own health as the only signal.

11. Practice loss. Failover drills, backup restore, certificate watch, queue inspection. HA that has never failed in anger is a story.

12. Say the tradeoff out loud. Custom means you own the bugs. More hosts mean smaller blast radius and more machines. Replica reads lag. A proxy down darkens the outside. Self-hosting is operational work. If you cannot name the cost, you are still in the vendor's paradigm, just with extra YAML.

13. Do not collapse planes "just for now." That recreates the mixed-host problem with better vocabulary.

14. Deterministic stays deterministic. AI may classify and draft. Money, access, and promotion of a primary stay in rules and quorum, not in a model score.

Feedback Loops

Balancing loops

These are the loops that keep the system itself.

• Reconciliation. Actual processes, routes, and stacks versus desired. The farther they drift, the stronger the corrective flow — if the loop is still running and the desired state is true.
• Election. Primary gone, majority remaining, new primary, replicas reconnect. Designed so a person is not the sensor.
• Health and pool. Failed member leaves. Recovered member returns. Users keep one endpoint.
• Retry then page. A single timeout is not an incident. State change is. This protects the information stock (attention) from noise.
• Rolling update / rollback. Version stock changes in batches. Bad version is a previous declaration, not a night of reverse-SSH.
• Human hand-off in automation. Low confidence routes to a person. The loop refuses to fake certainty.

Balancing loops fail in characteristic ways. Too slow: the customer becomes the monitor. Too sensitive: flapping, ignored boards. Too local: a node restarts forever while the plane is exhausted. The fix is rarely "more alerts." It is correcting delay, gain, and the boundary of the plane.

Reinforcing traps

Meadows catalogued system traps. This estate walks past the same ones every time a shortcut looks cheap.

Shifting the burden. Symptom: SSH tonight, managed SaaS this quarter, a public chatbot for the queue. Fundamental: own the loop, the plane, the perimeter. Each symptomatic fix atrophies the muscle the fundamental solution needs. The custom cluster and the self-hosted purpose plane are refusals of this trap. The cost is real: you own every bug the packaged platform would have hidden.

Tragedy of the commons. Mixed hosts. CI, prod, restore tests, and backups sharing disk and CPU. Each local decision is rational. The shared stock collapses. Role-separated planes are the rule that stops the commons from being eaten.

Drift to low performance. Copy-paste YAML. Each repo a little worse. "It still deploys." CI templates cut the reinforcing loop by making the shared path the default. Hidden jobs that get overridden too much start the drift again.

Rule beating. GitOps declared, SSH used. Labels on the proxy that steal a host. Commit-message scopes that say all because it is easier. The model has to make the declared path the easiest path, not only the official one. Otherwise the real system is the beat.

Seeking the wrong goal. "We have an orchestrator" instead of "we can lose a node and still explain why the next one took the work." "The board is green" instead of "the watcher would survive the outage." "AI answers" instead of "wrong answers are handed off." Goals in this model are behavioral: survive, converge, remain readable, keep purpose inside.

Success to the successful. One snowflake host that "always works" absorbs more services until it is unshedable. Dedicated roles and a scheduler that places work from capacity, not from habit, are the counter-rule.

Escalation / policy resistance. Opening public ports to make two sites talk, then adding firewall exceptions forever. Tunnels plus declared routes replace the arms race with a fabric.

Delays and information

Replica lag, election time, poll interval, certificate expiry, mail queue, restore time, the delay between a leftover SSH and anyone noticing — these are not footnotes. Meadows: delays are leverage, and they are dangerous when unseen.

• If the application pins a host, it has opted out of the delay the pool was built to hide.
• If the monitor's interval is honest, people trust the board. If it lies, they learn to ignore it — a destroyed information flow, one of the highest levers.
• If backup restore is not rehearsed, the delay at disaster time is unknown. An unknown delay is not a buffer. It is a story.

Information is a stock. Git, the uptime board, internal DNS, logs and traces on the custom cluster, queue metrics on mail — these are how the system sees itself. A system that can only be seen by SSH is a system that only exists when a certain person is awake.

Applications

Use the model as a filter, not as a brand.

A new service appears. Do not ask "which box has space." Ask: which plane does it belong to, what is the desired state, how does it get a name at the edge, which data stock it needs, which check goes on the board, which template it extends. If the answer is "we'll SSH and see," the system just shrank.

A node will die — because it will. Ask: is truth a quorum. Does the pool know. Does the scheduler replace the work. Does the monitor sit somewhere else. Was this loss rehearsed.

A team wants a SaaS. Translate into stocks. Which purpose stock leaves the perimeter. Which information flow you lose. Which shifting-the-burden loop you start. Sometimes you still buy. Then you name the dependency as a plane you do not own, and you watch it from outside.

Two sites must talk. Tunnels and declared prefixes, not a public port per app. Plan addressing. Least privilege. The fabric is a plane with its own critical path: the gateway.

Forty apps, or a hundred checks. Catalogs. Shared stages. Groups that match planes. Ownership on every tile. A reinforcing loop of unique YAML is not velocity. It is future coordination cost.

AI in the path of work. Conversation versus event workflow. Knowledge from internal sources. Hand-off when confidence is low. Deterministic steps stay deterministic. Transcripts stay inside. Staging tries a flow before production customers see it.

Someone proposes mixing roles to save a host. Recite rule 13. The saved host is paid for in blast radius, usually at 3 a.m.

Someone proposes a packaged orchestrator. Legitimate if the team will not read fifteen layers. Then the paradigm is different: you are renting the control plane. Put that on the table. Do not pretend you still own consensus because you can YAML a Deployment.

Limitations

This model is not a universal. It is a choice with a bill.

• You own the bugs. A custom runtime, a custom control plane, a self-hosted mail stack, an in-house chat — every subtle failure of consensus, overlay, deliverability, and embed layout is yours. Packaged platforms hide some of that. They also hide the machine.
• Ecosystem poverty. Processes plus namespaces are smaller than a universe of images. Self-hosted BaaS does not give you a global edge network. That is a purpose choice, not a missing ticket.
• GitOps cannot see what you will not declare. Drift from the box is still drift. The loop will not save a lie.
• Quorum needs separate failure domains. Three members on one hypervisor is theatre. Real high availability is topology, not a compose replicas: 3.
• Critical paths move, they do not vanish. The proxy, the tunnel gateway, the watcher, the git remote. Distribute them too, or you have built a prettier SPOF.
• Replica lag is real. Read-your-writes across a replica is a lie unless you paid for it. The model prefers a single writer and an honest delay.
• Operational load. Perimeter purpose means backups, reputation, abuse, prompt quality, certificate rotation. A SaaS would hide that. Hiding is the shifting-the-burden trap again.
• Catalog blast radius. One CI include can wound every consumer. Version and roll out. The shared path is a high lever in both directions.
• Polling is not APM. The board tells reachability. It does not tell why the query is slow. Do not seek the wrong goal and call the board a profiler.
• AI is probabilistic. Treat it as a clerk with a confidence score, not as a quorum member.
• Readability does not scale automatically. A cluster you wrote is readable to the people who wrote it. Operators still have to learn this system. Documentation, runbooks in git, and drills are part of the information flow, not extras.
• The paradigm can become a cage. Owning everything is also a reinforcing loop. Not every internal tool deserves fifteen layers. The transcend-the-paradigm move is knowing when an element is allowed to be rented because it is not on the purpose path.

Future Evolution

The next work is not another database. It is tightening the stocks the model still leaves soft.

Make drift first-class. Detect SSH and out-of-band change as an event on the same board as a dead replica. Information flow at the rule layer, not a lecture.

Multi-site as a plane with drills. Tunnels exist. Partition drills, overlapping-range guards, and gateway HA should sit in the same category as database failover — practiced, not hoped.

Promote the artifact, not the environment. CI already stamps images by commit. The rest of the estate can treat environments as different desired states over the same bits, not as different rituals.

Watch the watcher from somewhere else. Named in the uptime tradeoffs. Still easy to skip.

Human operating model. The same three stocks apply to teams: truth (what we think is running), control (who may change it), purpose (what we will not outsource). On-call, review, and runbooks are loops. Without that system, the technical one is only as good as the last hero.

Know when to rent. A written test: if we cannot read it, do we still get woken up for it. If yes, do not rent. If no, rent and monitor from outside. That is transcending the paradigm without betraying it.

The model should version like the cluster: declared, reviewable, rollback-able. When a rule dies in production — when people beat it — the next version changes the rule or the path, not the slide.

Related Systems

This is the first node in the knowledge graph. It should grow toward a human operating model (how people stay inside the same loops), a decision record for when to rent a layer, and a failure-drill catalog that treats loss as a first-class feature of the estate. Until those exist, the related graph is the projects: metal, not metaphor.

Related Projects

Every project below is a component of this model, not an adjacent hobby.

• Custom cluster from scratch — truth as quorum, control as API and agent, loop as reconciliation, no unread orchestrator.
• Role-separated GitOps — planes, declared control, blast radius.
• Postgres HA, MongoDB HA, Redis Sentinel — distributed data stocks, election, stable endpoints.
• Cluster reverse proxy — one door, TLS at the edge, membership-aware routing.
• Network tunnels — fabric between sites without publishing every service.
• Nextcloud, org-chat, SMTP, self-hosted backend, AI automation — purpose inside the perimeter.
• Uptime monitoring — information stock off the failure domain it describes.
• CI templates — one path so delivery does not reinforce drift.