kfkwho, an authorizer for Apache Kafka® that counts

Who talks
to what.

A drop-in replacement for the broker's StandardAuthorizer that publishes JMX metrics per principal, client id, topic, operation and request type. This is how, in fourteen slides and one small model.

Scroll, or use the arrow keys.

1 · the question

Who produces to this topic?

The producers of a topic are what you need to know before you change, move or delete it. Consumers come right after: which clients read this topic, and when did they last show up?

Admins are the same question about other operations.

2 · the catch

The broker knows, and never says

A broker knows, on every request, who sent it (principal, client id, address) and what it touches (topic, group, operation). It never puts the two together in a metric.

So the question is answered by grepping request logs.

3 · the idea

Count at the authorizer

The authorizer is the one hook that sees both sides of every request, for clients of any version: the principal and the client id, the resource and the operation. kfkwho extends the standard one. The verdict stays the broker's; the observation becomes a metric.

4 · the verdict

It observes. It does not decide.

MeteredStandardAuthorizer is StandardAuthorizer plus metrics. authorize() calls the parent first and returns exactly what the parent returns. ACLs, super.users and allow.everyone.if.no.acl.found work as before.

5 · a series

One counter per who and what

An access series is one MBean per principal, client id, resource, operation, request type and verdict. The tag order is fixed, so an exporter rule is one line.

kfkwho:type=access,principal=…,client-id=…,resource-type=TOPIC,resource=…,operation=…,api=…,result=ALLOWED
kfkwho:type=client,principal=…,client-id=…,listener=…,security-protocol=…

It is counted once per action the broker authorizes: a Produce to three topics is three. The client series is one per principal and client id, counted once per authorize call: who is connected, without the resource.

6 · reading it

Producers first

The api tag is the request type. It tells a client that really produced from one that only fetched metadata about the topic.

ClientOperationSeries
ProducerWRITETOPIC, WRITE/PRODUCE
Only fetched metadataDESCRIBETOPIC, DESCRIBE/METADATA: it has not produced
ConsumerREADTOPIC, READ/FETCH, plus GROUP, READ with OFFSET_COMMIT, JOIN_GROUP, HEARTBEAT
AdminCREATECREATE/CREATE_TOPICS, counted the same way

For a GROUP series the resource is the group id, which links client ids to consumer groups. Denied requests are counted too, with result=DENIED.

7 · the hot path

One lookup per action

authorize() runs for every topic of every Fetch and Produce, so what the metrics add to it is measured, not guessed. A series seen before costs one lookup per action and allocates nothing.

On first sight of a key the request thread only leaves a pending series. A background thread, kfkwho-series, creates it, at about 40 µs each; requests that arrive meanwhile are counted into it once it exists.

Measured with JMH on a cloud VM with 4 vCPU: the parent alone took 338 ns per call and the metered authorizer 525 ns, 187 ns added; later runs, with the client series counted as well, added 213 to 332 ns. First sight adds about 1.1 µs on the request thread. No allocation per call. The budget is under 1 µs and under 3 µs. The runs are in Performance.

8 · bounded

Idle series expire. A cap folds the rest.

Clients come and go, and every new key is a series. Two bounds keep their number finite.

Series that already exist keep counting, and a series freed by the TTL makes room for the next new key. A non-zero overflow means some clients are not told apart.

9 · names

One application, one series

Consumer and Streams clients put a per-instance suffix in their client id; left as is, every restart is a new series. Client id rules rewrite the id before it becomes a tag.

Client id, as sentTag
consumer-orders-app-3-6f1c2a4e-1b2c-4d5e-8f90-abcdef123456consumer-orders-app
consumer-orders-app-3, billing-1consumer-orders-app, billing

The rules are kfkwho.client.id.rules, each pattern=>replacement in Java regex syntax; the first whose pattern matches the whole id applies. Internal topics (__.*) are not recorded at all: that is kfkwho.resource.exclude.

10 · try it

Start and stop clients

broker time 0 s access-series-count 0 of 4 access-series-evicted-total 0 access-series-overflow-total 0 kfkwho.series.ttl.seconds 20 kfkwho.max.series 4
principalclient-idresourceoperation / apirequest-totalidle
no series

    A model of the bounds, not the real code: one series per client, created at once, every verdict ALLOWED. A real client leaves several series, and the defaults are 600 s and 10000 series. Start billing-2: it counts into the series of billing-1. Start everything: the fifth key goes to __other__. Stop a client and count to twenty.

    11 · out of the broker

    JMX, then Prometheus

    The MBeans are in the kfkwho domain, published through the broker's own metrics library and a JmxReporter, the way quota metrics are. The repository ships rules for the Prometheus JMX Exporter, jmx-exporter/kfkwho.yml. Each tag becomes a label of the same name, with - written as _.

    Who produces to orders, per principal and client id:

    sum by (principal, client_id) (
      rate(kfkwho_access_requests_total{resource_type="TOPIC", resource="orders",
                                        operation="WRITE", api="PRODUCE", result="ALLOWED"}[5m]))

    12 · limits

    What it does not do

    13 · why believe any of this

    How it is tested

    14 · go

    Four things on every node

    Source on GitHub ▶ Releases ▶

    1. the kfkwho jar on the broker's classpath;
    2. authorizer.class.name=dev.kfkit.kfkwho.MeteredStandardAuthorizer;
    3. the Prometheus JMX Exporter agent with the rules in jmx-exporter/kfkwho.yml;
    4. a rolling restart, one node at a time.

    In the apache/kafka image, with the jar, the agent and the rules under /opt/kfkwho, add to the environment the broker already has:

    CLASSPATH=/opt/kfkwho/kfkwho.jar
    KAFKA_AUTHORIZER_CLASS_NAME=dev.kfkit.kfkwho.MeteredStandardAuthorizer
    KAFKA_SUPER_USERS=User:admin
    KAFKA_OPTS=-javaagent:/opt/kfkwho/jmx_prometheus_javaagent.jar=9404:/opt/kfkwho/kfkwho.yml

    The jar is on the releases page, with its SHA-256 and a cosign signature, and the same jar is on Maven Central as dev.kfkit:kfkwho. A cluster that ran without an authorizer starts enforcing ACLs: read Installation first.

    With Claude Code, the repository carries a skill for adopting kfkwho: what to find out first, the order of the steps, what not to do.

    /plugin marketplace add kfkit/kfkwho
    /plugin install kfkwho@kfkwho