Real-time monitoring across a hundred services

Problem

An estate of about a hundred services cannot be watched by opening tabs. An app can look fine while the database behind it is gone. A health URL can pass on one node and fail on another. Each team writes a different check, or none. Outages are discovered by a customer, a Slack ping, or someone who happened to be looking.

Ping is not enough. HTTP, TCP, a query, a certificate expiry — each layer dies differently. Without a shared board, production, development, and the data plane are invisible until they hurt.

The organization needed one place that polls everything from the product to the database, in real time, and tells the right people when a check fails.

System Design

This project is a self-hosted uptime layer for the whole platform. Roughly a hundred targets: public HTTPS, internal APIs, reverse-proxy entrypoints, caches, and database listeners. Each target has a type, an interval, and a timeout. Failures retry before they page; recoveries clear the incident.

The board is the source of truth for “is it up.” Groups follow the same planes as the rest of the estate: applications, cluster edge, data, backups. Alerts go to the channel that owns that plane — not to everyone for every blip.

It runs on the monitoring hosts, not on production app nodes. Checks come from outside the service under test so a dead container cannot mark itself healthy.

Architecture Decisions

• One catalog of checks instead of a script per repo
• HTTP, TCP, and keyword checks so apps and databases share the same board
• Group by role (app, edge, data) so a red tile means a plane, not a mystery hostname
• Retry and grace so a single timeout is not an incident
• Alerts on state change, not on every failed poll
• Self-hosted so status and URLs stay inside the organization
• Watcher off the production cluster so the monitor survives the thing it watches

Tradeoffs

• A check is not APM: it tells you reachability, not why the query is slow
• A hundred targets need naming and ownership or the board becomes noise
• Polling from one place can miss a regional or node-local failure
• False positives train people to ignore the board — intervals and retries have to be honest
• You own the uptime of the watcher itself; it needs a check from somewhere else

Production Behavior

Add a service, add a check: URL or port, interval, who to notify. The board stays green until something fails a retry, then the owning channel sees it. Databases sit next to apps so a data outage is not mistaken for a frontend bug.

Review flapping checks and dead notification paths the same way you review deploys. Built for a platform that has too many moving parts to babysit, and too much at stake to find out from the customer.