kfklease, a lease on Apache Kafka®

One workload.
One cluster
at a time.

The log decides who. This is how, in fifteen slides and one small simulation.

Scroll, or use the arrow keys.

1 · the job

Something must run exactly once

A connector, a scheduler, a consumer that must not be doubled. You have two Kubernetes clusters, so that one can go away. The workload should run in one of them, and move when that one is gone: not before, and never in both.

That needs an arbiter both clusters can reach and neither owns. If you run Apache Kafka, you already operate one: a durable, ordered log.

2 · the catch

A log cannot say “take it if it is free”

A lock needs compare-and-swap. A partition only appends. Two candidates can always both write I take it, and both writes succeed.

3 · the idea

So writing decides nothing

The state of the lease is a function of the log. Every participant reads the same partition in the same order and applies the same rules, the writer included: it learns the fate of its own record by reading it back. The partition's order is the missing compare-and-swap.

4 · the records

Four kinds, one key

RecordMeansTakes effect when
claimstart a new termthe lease is not held at the record's timestamp
renewextend my termthat term is still live, and mine
releaseend my term nowthe same
yieldmy site's workload has stoppednever changes the lease; lets another site take over sooner

The offset of an accepted claim is the term's epoch: unique, growing with every term, known to anyone who reads the record. It is the fencing token.

All records share one key, so compaction keeps the tail and the topic stays small.

5 · time

One clock: the broker's

The topic uses LogAppendTime: the broker stamps every record as it lands. A term lasts one TTL after its last accepted record. Client clocks are never compared with anything.

6 · the holder

A belief with a deadline

A holder does not know it holds the lease; it believes so until send time + TTL − margin, on its own monotonic clock. A record lands after it was sent, so the term in the log always outlives the belief.

A process frozen for a minute wakes up with a deadline in the past. It stops before anyone else may start.

7 · newcomers

Joining in the middle

Compaction leaves a tail, so a new reader starts mid-story and cannot judge the first records it sees. Its view is uncertain until it can prove a state:

Nobody writes, or believes, on an uncertain view.

8 · sites

The lease belongs to a site

A holder is <site>/<pod>, and a site is a cluster. Each cluster runs two scaler pods: one holds the term, both answer for the site.

9 · moving the workload

KEDA does the lifting

The scaler is a KEDA external scaler. It answers 1 while its site holds the lease and 0 otherwise. The ScaledObject has idleReplicaCount: 0 and minReplicaCount = maxReplicaCount = N.

N replicas where the lease is held, none elsewhere. If KEDA cannot reach the scaler, a fallback scales the workload to zero: fail closed.

10 · try it

Break a cluster

broker time 0 s lease free workload in eu no workload in us no ran in both at once 0.0 s TTL 10 s · takeover delay 40 s · KEDA cooldown 30 s

    A model of the rules, not the real code: one scaler per cluster, instant reads. Kill the holder and count: the term runs out a TTL after its last renewal, the other cluster waits the delay, then claims.

    11 · on purpose

    The handover

    The takeover delay is a guess at how long the old workload takes to stop. A cluster that leaves on purpose can do better. Put it in standby and its scaler:

    The other cluster reads the yield and takes over about a TTL after the last renewal, not a takeover delay after the release. Try standby on the holder in the simulation.

    12 · the last line of defence

    Fence with the epoch

    Kubernetes will happily restart a stale pod in a cluster that just came back. So the workload asks the scaler before it writes, and sends the epoch along:

    13 · why believe any of this

    How it is tested

    The reasoning is in the protocol.

    14 · go

    One release per cluster

    Source on GitHub ▶ Releases ▶

    helm install kfklease oci://ghcr.io/kfkit/charts/kfklease \
      -n kfklease --create-namespace \
      --set brokers=kafka:9092 --set topic=my-lease \
      --set site=eu-west \
      --set scaledObject.targetRef.name=my-singleton

    The same in the other cluster, with another site.

    With Claude Code, the repository carries a skill for adopting kfklease: what to find out first, what the numbers trade, what not to do.

    /plugin marketplace add kfkit/kfklease
    /plugin install kfklease@kfklease