Scalable non-relational database with failover
A document store that stays online when a node dies — multiple replicas, automatic failover, and load balancing across the cluster.
Problem
Applications with flexible documents do not fit a rigid relational schema. A single database node is still a single point of failure: when it dies, writes stop and reads stall. Scaling by adding one bigger machine hits a wall.
The platform needs a non-relational store that grows with traffic, keeps copies of the data on several nodes, promotes a replica when the primary fails, and spreads connections so no single host is the bottleneck.
System Design
This project is a replica-set cluster: one primary takes writes, and several replicas stay in sync. If the primary disappears, the remaining nodes elect a new primary without an operator.
A load balancer sits in front of the set. Clients connect to one address. Writes go to the current primary; reads can spread across replicas. Persistent volumes keep documents across restarts. Credentials live in secrets, not in source control.
Architecture Decisions
- An odd number of voting members so elections have a clear majority
- Multiple replicas for redundancy and read scale
- Automatic failover instead of a manual promote
- Load balancer so applications do not pin a single host
- Persistent volumes for document data
- Health checks so dead nodes leave the pool
- Repeatable deploy scripts for bring-up, status, and teardown
Tradeoffs
- Replica reads can lag a moment behind the primary
- Writes still go through one primary at a time
- More nodes mean more operations and more disk
- Schema flexibility is traded for fewer joins and stricter relational guarantees
- Real high availability needs at least three nodes on separate hosts
Production Behavior
Deploy the replica set, point the load balancer at the members, and let clients use a single endpoint. When a node fails, the cluster elects a new primary and the balancer stops sending traffic to the dead member.
Check member health after deploy and after every failover drill. Built for document workloads that need to scale out, survive a node loss, and keep read traffic balanced.