A custom cluster from scratch — no packaged platform

Problem

Packaged orchestrators hide the machine. You declare a desired state and hope the internals agree. When a node dies, a replica stalls, or a network namespace leaks, the failure lives in someone else's binary. You cannot change the scheduler, the runtime, or the consensus story without fighting the product.

An organization that has to own the cluster — not rent the abstraction — needs every layer in its own code: how a packet is accepted, how state is stored, how leaders are elected, how a process is born, how a service is found. No ready-made control plane. No ready-made container engine. No ready-made overlay you cannot open.

System Design

This project is a cluster written from the kernel up. Fifteen layers, all custom.

1. Networking basics. TCP servers and clients, an HTTP API for operators, and RPC between control plane and nodes. The cluster speaks its own protocol before anything else exists.

2. Storage layer. A key-value store with an explicit data model, persistence to disk, and a watch/event bus so other components react to state changes instead of polling.

3. Distributed consensus. Raft implemented in-process: leader election, log replication, and recovery when a member disappears. Cluster truth is a quorum, not a single file.

4. Control plane. An API server with authentication, validation, and state management. Desired objects enter here; they do not get written onto nodes by hand.

5. Node agent. Each machine runs an agent: registration, heartbeat, task execution, health. The control plane only knows a node exists because the agent said so and keeps saying so.

6. Runtime engine — no packaged container runtime. Workloads are Linux processes. The agent fork/execs, owns signals, and tracks lifecycle. Isolation comes from the layers below, not from a daemon you did not write.

7. Scheduler. Discovers nodes, tracks CPU and memory, places tasks, and spreads load. Placement is an algorithm in this repo, not a plugin you cannot step through.

8. Controller pattern. Desired state versus actual state, a reconciliation loop, self-healing. If a process dies, the loop starts it again. If a node leaves, work is placed elsewhere.

9. Service management. Registry, discovery, internal DNS, routing. A name in the cluster resolves to the tasks that currently exist, not to a host you memorized.

10. Networking layer. Virtual IP, proxy, overlay, isolation between tenants or apps. Traffic between services does not depend on publishing a host port per process.

11. Resource isolation. Linux namespaces and cgroups: CPU limit, memory limit. A noisy neighbor cannot eat the node because the agent applied the quota at start.

12. Security. TLS on the wire, certificates for nodes and users, RBAC on the API, secrets that are not env files in git. The control plane does not trust a raw socket.

13. Observability. Logs, metrics, tracing, health checks. You cannot operate a custom cluster if the only signal is SSH.

14. Deployment model. Rolling update, versioning, rollback, configuration as data. A new version replaces tasks gradually; a bad version is a declared rollback, not a night of reverse-SSH.

15. High availability. Multiple control-plane members, leader failover, replicated data, disaster recovery. Losing one API node does not lose the cluster; losing a disk does not lose the last good state.

Nothing in that stack is an off-the-shelf product. Algorithms, processes, and Linux primitives — assembled in code.

Architecture Decisions

• Write the control plane instead of wrapping a packaged one
• Processes plus namespaces and cgroups instead of a third-party runtime
• Raft and a KV with watches as the source of truth, not a local YAML on each node
• Agent on every node so the API never SSHs in to start work
• Reconcile forever: the cluster converges; it is not a one-shot deploy script
• Services get names, VIP, and DNS so apps do not hard-code peers
• Isolate CPU and memory at the kernel, not with a polite request
• TLS, certs, and RBAC from day one so a custom API is not an open admin port
• Rolling updates and rollback as first-class operations
• Quorum of control planes so the brain of the cluster is not a single host

Tradeoffs

• You own every bug the packaged platform would have hidden
• A custom runtime is smaller than an ecosystem of images and tools — and poorer in ready-made images
• Consensus and overlay networking are easy to get subtly wrong; tests and failure drills are mandatory
• Operators must learn this cluster, not a vendor's exam
• Feature velocity is yours; you do not get upstream's next release for free

Production Behavior

Bring up a quorum of control planes, start agents on nodes, submit a desired workload. The scheduler places it, the agent execs it under limits, the service layer publishes a name, the proxy and overlay carry traffic, the controllers keep actual equal to desired.

Kill a leader: another is elected, replication continues. Kill a process: the loop restarts it. Roll a version: tasks move in batches; rollback is a previous version declared again. Built for a team that needed a cluster they could read, change, and defend — without standing a packaged orchestrator in the middle.