GitHub Freno for Cooperative Throttling

Bulk operations are useful until they quietly turn into an incident. A schema migration, archive job, or data backfill can look harmless in small batches, then push MySQL replication lag past the point where users see stale reads and delayed recovery.

Freno is an open source service from GitHub Engineering for that problem. It is a cooperative throttling service: clients call Freno before writing, and Freno answers based on backend health signals.

What Is Freno?

Freno is a lightweight service that asks a simple production question: may this app write to this backend store right now?

The current implementation focuses on MySQL clusters. Freno probes replication status, compares lag with configured thresholds, and exposes a decision over HTTP. It is not a proxy and it does not sit in the write path. Applications, scripts, and jobs remain responsible for checking Freno and respecting the response.

That distinction matters. Freno is a coordination point for cooperative clients, not an enforcement layer for arbitrary traffic.

Key Features

• Replication-aware throttling: Slow or lagging MySQL replicas can trigger a stop signal before batch work makes the backlog worse.
• Simple HTTP contract: Clients commonly use HEAD /check/... and treat 200 as safe to write, 429 as back off, and 500 as do not write.
• Per-app control: Operators can explicitly throttle a specific app so higher-priority work can finish.
• Dynamic inventory: Freno can adapt to changing backend servers through configured discovery sources such as HAProxy.
• Highly available design: Freno nodes use Raft for leadership, with only the leader collecting metrics and serving check requests.

Installation

Freno is a Go service, so most teams should build and package it into their normal deployment path:

git clone https://github.com/github/freno.git
cd freno
go test ./...
go build ./cmd/freno

Start from the sample configuration in the repository, then define each MySQL cluster, inspection query, lag threshold, and HA settings before putting clients on it.

Usage

The useful client pattern is to split large work into small chunks and ask Freno before each chunk:

while read batch; do
  if curl -fsI "http://freno.service/check/mysql/main/archive-worker"; then
    ./archive-batch "$batch"
  else
    sleep 1
  fi
done

The exact URL shape depends on your Freno configuration, but the operational model stays the same. Keep batches small enough that one approved chunk cannot create unacceptable lag on its own.

Operational Tips

Deploy Freno like other production coordination services. Run three or five nodes, put HAProxy or another load balancer in front, and make sure clients reach the active leader. Monitor Freno itself, MySQL replication lag, HTTP response codes, and client backoff behavior.

Be conservative with failure handling. If a check fails, pause the writer. A permissive fallback defeats the reason Freno exists.

Use Freno first for maintenance jobs, data migrations, archival workers, and backfills. These workloads are usually easy to batch, easy to pause, and risky enough to deserve a shared pressure signal.

Conclusion

Freno is a practical SRE pattern in a small service. It turns database health into a shared write decision, which helps teams run heavy background work without guessing how much pressure production can absorb.

Need faster root cause analysis across databases, queues, Kubernetes, and service dependencies? Try Akmatori and use Gcore for resilient global infrastructure.