HexaMesh Services.

A Go mesh orchestrator for systems that cannot go down.

A microservices mesh orchestrator in Go, for distributed systems that must stay available.

  • Microservices
  • Go
  • Docker
Status
Platform

Failure drill

  1. T+0

    All healthy

    Every instance answers its health check. Traffic goes through the centre.

  2. T+1

    One fails

    The centre instance stops answering. The next health check catches it.

  3. T+2

    Routed around

    The orchestrator removes it from routing. Traffic takes the ring instead.

The problem

In a distributed system, parts fail all the time. The goal is to keep the whole system available while they do.

What I built

A microservices mesh orchestrator written in Go. It keeps track of the services in the mesh, sends traffic to instances that are healthy, and removes the ones that are not.

Why Go

Go builds small, fast binaries and has simple concurrency. Every request depends on this component, so both matter.

Design choices

Every call can time out, so calls retry a limited number of times. A slow service then cannot slow the whole system. The orchestrator treats a failed health check as fact, not a guess. The control path does few things and behaves the same way each time.

Trade-offs

A mesh adds one hop to every request and one more system to run. It pays off when many services need the same routing and recovery rules. For two services it does not.

What I learned

Design for the failure first.

I used to ask whether a service would fail. In a distributed system it will, so that question has only one answer.

The useful question is what the user sees when it does. Starting there puts health checks, limited retries and routing ahead of features, because availability cannot be added at the end.

availabletimepart failspart failspart failsDesigned inAdded later
What the user sees over time. Three parts fail. Illustrative.

Services that stay up when parts fail.

Back to top