Failure drill
T+0
All healthy
Every instance answers its health check. Traffic goes through the centre.
T+1
One fails
The centre instance stops answering. The next health check catches it.
T+2
Routed around
The orchestrator removes it from routing. Traffic takes the ring instead.
The problem
In a distributed system, parts fail all the time. The goal is to keep the whole system available while they do.
What I built
A microservices mesh orchestrator written in Go. It keeps track of the services in the mesh, sends traffic to instances that are healthy, and removes the ones that are not.
Why Go
Go builds small, fast binaries and has simple concurrency. Every request depends on this component, so both matter.
Design choices
Every call can time out, so calls retry a limited number of times. A slow service then cannot slow the whole system. The orchestrator treats a failed health check as fact, not a guess. The control path does few things and behaves the same way each time.
Trade-offs
A mesh adds one hop to every request and one more system to run. It pays off when many services need the same routing and recovery rules. For two services it does not.
What I learned
Design for the failure first.
I used to ask whether a service would fail. In a distributed system it will, so that question has only one answer.
The useful question is what the user sees when it does. Starting there puts health checks, limited retries and routing ahead of features, because availability cannot be added at the end.