kfklease, a lease on Apache Kafka®
One workload.
One cluster
at a time.
The log decides who. This is how, in fifteen slides and one small simulation.
Scroll, or use the arrow keys.
1 · the job
Something must run exactly once
A connector, a scheduler, a consumer that must not be doubled. You have two Kubernetes clusters, so that one can go away. The workload should run in one of them, and move when that one is gone: not before, and never in both.
That needs an arbiter both clusters can reach and neither owns. If you run Apache Kafka, you already operate one: a durable, ordered log.
2 · the catch
A log cannot say “take it if it is free”
A lock needs compare-and-swap. A partition only appends. Two candidates can always both write I take it, and both writes succeed.
3 · the idea
So writing decides nothing
The state of the lease is a function of the log. Every participant reads the same partition in the same order and applies the same rules, the writer included: it learns the fate of its own record by reading it back. The partition's order is the missing compare-and-swap.
4 · the records
Four kinds, one key
| Record | Means | Takes effect when |
|---|---|---|
| claim | start a new term | the lease is not held at the record's timestamp |
| renew | extend my term | that term is still live, and mine |
| release | end my term now | the same |
| yield | my site's workload has stopped | never changes the lease; lets another site take over sooner |
The offset of an accepted claim is the term's epoch: unique, growing with every term, known to anyone who reads the record. It is the fencing token.
All records share one key, so compaction keeps the tail and the topic stays small.
5 · time
One clock: the broker's
The topic uses LogAppendTime: the broker stamps every record as it lands. A term
lasts one TTL after its last accepted record. Client clocks are never
compared with anything.
6 · the holder
A belief with a deadline
A holder does not know it holds the lease; it believes so until send time + TTL − margin, on its own monotonic clock. A record lands after it was sent, so the term in the log always outlives the belief.
A process frozen for a minute wakes up with a deadline in the past. It stops before anyone else may start.
7 · newcomers
Joining in the middle
Compaction leaves a tail, so a new reader starts mid-story and cannot judge the first records it sees. Its view is uncertain until it can prove a state:
- Start: it read the partition from offset 0, without gaps.
- Silence: nothing for a whole TTL, and the margin on top. Whatever term there was is over.
- Anchor: two records of one term. A writer writes the second only after reading the first back and finding it accepted.
Nobody writes, or believes, on an uncertain view.
8 · sites
The lease belongs to a site
A holder is <site>/<pod>, and a site is a cluster. Each cluster runs
two scaler pods: one holds the term, both answer for the site.
- Own site: takes over at once. A scaler restart, an upgrade, a node drain move nothing.
- Another site: waits the takeover delay after the term ended, the time the old workload gets to stop, and never less than the term's natural end.
- Empty topic: everyone waits the delay from its start. A recreated topic looks new, and its last holder may still run.
9 · moving the workload
KEDA does the lifting
The scaler is a KEDA external scaler. It answers
1 while its site holds the lease and 0 otherwise. The
ScaledObject has idleReplicaCount: 0 and
minReplicaCount = maxReplicaCount = N.
N replicas where the lease is held, none elsewhere. If KEDA cannot reach the scaler, a fallback scales the workload to zero: fail closed.
10 · try it
Break a cluster
A model of the rules, not the real code: one scaler per cluster, instant reads. Kill the holder and count: the term runs out a TTL after its last renewal, the other cluster waits the delay, then claims.
11 · on purpose
The handover
The takeover delay is a guess at how long the old workload takes to stop. A cluster that leaves on purpose can do better. Put it in standby and its scaler:
- stops believing and writes a release;
- pauses the ScaledObject at zero, and waits until the scale is at zero and no pod is left;
- writes a yield: our workload is down.
The other cluster reads the yield and takes over about a TTL after the last renewal, not a takeover delay after the release. Try standby on the holder in the simulation.
12 · the last line of defence
Fence with the epoch
Kubernetes will happily restart a stale pod in a cluster that just came back. So the workload asks the scaler before it writes, and sends the epoch along:
GET /status: nosite_holding, no write.- Every write carries
site_epoch; whatever receives it rejects an epoch older than the newest it has seen.
13 · why believe any of this
How it is tested
- A simulation drives the real decision code against a fake log with frozen processes, late and lost records, cut-off nodes, stopped processes, skewed clocks. No two holders at any instant; a reader started at any offset agrees with one that saw everything.
- Integration tests on a real broker, over plaintext, SASL and mTLS.
- A stand: two k3s clusters with KEDA and one broker, and a dozen failure scenarios (crash, partition, freeze, broker outage, a scaler that is gone, handover). A heartbeat workload proves no two epochs ever wrote at once, by the broker's clock.
- Every rule was broken on purpose to see a test fail.
The reasoning is in the protocol.
14 · go
One release per cluster
helm install kfklease oci://ghcr.io/kfkit/charts/kfklease \
-n kfklease --create-namespace \
--set brokers=kafka:9092 --set topic=my-lease \
--set site=eu-west \
--set scaledObject.targetRef.name=my-singleton
The same in the other cluster, with another site.
With Claude Code, the repository carries a skill for adopting kfklease: what to find out first, what the numbers trade, what not to do.
/plugin marketplace add kfkit/kfklease
/plugin install kfklease@kfklease