Network fabric between internal and external clusters

Problem

Internal networks and external clusters do not share a cable. NAT, different providers, and firewalls mean a service in the office cannot see a service in the other site unless someone opens a public port. Opening that port for every app is an attack surface. Site-to-site “temporary” SSH forwards rot. Two clusters cannot schedule, replicate, or monitor each other if packets never arrive.

The platform needed a real network layer for this: tools built for encrypted paths, not a hop through a jumphost. Internal LAN to internal LAN. Internal cluster to external cluster. External operators into the private fabric — only where policy allows.

System Design

This project is the overlay between sites. Each location — office, production cluster, remote cluster, backup site — keeps its own addressing. Encrypted tunnels join those planes into one reachable fabric. Internal traffic stays internal; it does not hairpin through the public internet unencrypted.

Gateways at the edge of each network terminate the tunnels. Routing decides which prefixes go through which path. Clusters on the far side appear as neighbors: overlay traffic, API access, and replication use the tunnel, not a published host port per service.

Identity is on the tunnel: who may join, which subnets they see, which clusters they may touch. Bringing a new site up is another peer and another route, not a new DMZ full of exceptions.

Architecture Decisions

• Purpose-built tunneling instead of ad-hoc SSH port forwards
• Separate internal and external planes, joined only where a route is declared
• Cluster-to-cluster paths for control and data, not one jumphost for everything
• Encrypted overlay so packets between sites are not plaintext on the public path
• Gateways own the tunnels; apps keep private addresses
• Least-privilege peers: a remote cluster does not automatically see the whole office LAN
• Routes and peers declared as infrastructure, so a new site is not a tribal firewall story

Tradeoffs

• A tunnel gateway is a critical path: if it dies, the far cluster is dark
• Overlapping private ranges between sites will collide; addressing has to be planned
• Encryption and extra hops add latency; chatty east-west traffic should stay local when it can
• Key and peer lifecycle is yours — a stale peer is a hole
• Debugging spans two networks; “it works on this side” is not enough

Production Behavior

Stand a gateway at each site, peer them, advertise the prefixes each cluster should reach. Internal services talk to remote services over the tunnel as if they were next door. External clusters join the same fabric without exposing their APIs on 0.0.0.0.

When a site is added or removed, change peers and routes — not a spreadsheet of public ports. Built for an estate that lives in more than one network and still has to behave like one platform.