Kafka API & Wire Compatibility

Rafka implements the Kafka binary wire protocol over TCP. Standard clients — kcat, librdkafka, kafka-go, the Java Kafka client — connect to the gateway on port 9092 with no special configuration.

This page is the centralized reference for every supported API key, its advertised version range, and the specific wire behavior Rafka implements for that op. It is updated as each compliance phase ships.

Supported API keys (P3.A — current release)

The gateway's ApiVersionsResponse advertises exactly these sixteen entries:

API Key	Name	Min version	Max version	Where handled	Notes
18	ApiVersions	0	3	Gateway inline	No mesh hop. Always responds with v0 response header per KIP-511.
0	Produce	0	8	Gateway → broker via KafkaOp	KAFKA_OP_PRODUCE = 0x02. Persisted to SingleWal.
1	Fetch	0	11	Gateway → broker via KafkaOp	KAFKA_OP_FETCH = 0x03. Reads from SingleWal.
2	ListOffsets	1	5	Gateway → broker via KafkaOp	KAFKA_OP_LIST_OFFSETS = 0x04. min=1 (v0 uses old-style offsets[] array).
3	Metadata	0	5	Gateway inline	Rewritten P2.A. Honors allow_auto_topic_creation flag; returns registered partition count.
8	OffsetCommit	0	7	Gateway → broker via KafkaOp	KAFKA_OP_OFFSET_COMMIT = 0x0A. In-memory offsets (P3.A). Cap at v7 (flexible at v8).
9	OffsetFetch	0	5	Gateway → broker via KafkaOp	KAFKA_OP_OFFSET_FETCH = 0x0B. Returns committed_offset=-1 for absent entries. Cap at v5 (flexible at v6).
10	FindCoordinator	0	2	Gateway inline	No broker call. Returns gateway self (node_id=0, RAFKA_KAFKA_CLIENT_HOST:PORT). Cap at v2 (flexible at v3).
11	JoinGroup	0	5	Gateway → broker via KafkaOp	KAFKA_OP_JOIN_GROUP = 0x06. Broker awaits rebalance oneshot (no time cap). Cap at v5 (flexible at v6).
12	Heartbeat	0	3	Gateway → broker via KafkaOp	KAFKA_OP_HEARTBEAT = 0x08. Cap at v3 (flexible at v4).
13	LeaveGroup	0	3	Gateway → broker via KafkaOp	KAFKA_OP_LEAVE_GROUP = 0x09. v3 uses members[] array; v0–v2 uses top-level member_id. Cap at v3 (flexible at v4).
14	SyncGroup	0	3	Gateway → broker via KafkaOp	KAFKA_OP_SYNC_GROUP = 0x07. Cap at v3 (flexible at v4).
19	CreateTopics	0	4	Gateway inline	Topic registry create. Cap at v4 (flexible at v5).
20	DeleteTopics	0	3	Gateway → broker via KafkaOp	KAFKA_OP_DELETE_TOPIC = 0x05. Purges WAL partitions. Cap at v3 (flexible at v4).
32	DescribeConfigs	0	3	Gateway inline	Returns all stored topic configs. Cap at v3 (flexible at v4).
33	AlterConfigs	0	1	Gateway inline	Full-replace semantics. Cap at v1 (flexible at v2).

The version caps are deliberately set below each API's flexible-encoding threshold. This keeps the response header at v0 (no tagged fields) and the body encoding non-compact (plain string lengths), which is the simplest correct encoding for single-broker P1/P2/P3:

API	Flexible encoding starts at	Cap
Produce	v9	v8
Fetch	v12	v11
ListOffsets	v6	v5
Metadata	v9 (v6 adds topic_authorized_operations)	v5
OffsetCommit	v8	v7
OffsetFetch	v6	v5
FindCoordinator	v3	v2
JoinGroup	v6	v5
Heartbeat	v4	v3
LeaveGroup	v4	v3
SyncGroup	v4	v3
ApiVersions	v3	v3
CreateTopics	v5	v4
DeleteTopics	v4	v3
DescribeConfigs	v4	v3
AlterConfigs	v2	v1

ApiVersions (api_key 18)

Role: client–server API negotiation. Every Kafka client sends this as the first request on a new connection.

Rafka behavior: returns a canned ApiVersionsResponse listing the five keys above. The response version is clamped to min(request_version, 3). Response header is always v0 per the KIP-511 bootstrapping rule (the client may not yet know which header format to expect).

No mesh hop. The gateway encodes the response inline without forwarding to the broker.

Span: rafka.gateway.kafka.request (api_key=18 in attributes).

Metadata (api_key 3)

Role: broker discovery. Clients use this to learn which broker hosts each topic partition.

Rafka behavior (self-advertise pattern): the gateway responds as the single broker (node_id=0, host=RAFKA_KAFKA_CLIENT_HOST, port=RAFKA_KAFKA_CLIENT_PORT). For registered topics, the partition count comes from the topic registry.

See the P2.A update section below for the full flag-dispatch table and the auto-create vs. explicit-create semantics.

No mesh hop. Metadata is synthesized entirely in the gateway.

Span: rafka.gateway.kafka.metadata (attribute: auto_create).

Produce (api_key 0)

Role: write records to a topic partition.

Rafka behavior:

1. Decodes ProduceRequest from the wire (including full RecordBatch parsing).
2. Extracts record values from the RecordBatch.
3. Builds a ProduceOp { topic, partition, records } and postcard-encodes it.
4. Calls kafka_op_call(broker_peer, KAFKA_OP_PRODUCE, org_id, payload) over an iroh QUIC bi-stream.
5. Broker appends records to the SingleWal and returns ProduceOpResp { base_offset, error_code: 0 }.
6. Gateway encodes ProduceResponse and returns it to the client.

base_offset is the WAL virtual offset of the first appended record (zero-indexed per topic+partition).

Span chain: rafka.gateway.kafka.request → rafka.broker.produce.store. Span attributes on rafka.broker.produce.store: virtual_topic, record_count, base_offset.

Fetch (api_key 1)

Role: read records from a topic partition starting at a given offset.

Rafka behavior:

1. Decodes FetchRequest, extracts (topic, partition, fetch_offset, max_bytes) per partition.
2. For each partition, builds a FetchOp and dispatches kafka_op_call(KAFKA_OP_FETCH).
3. Broker reads records from the WAL starting at fetch_offset, returning up to max_bytes.
4. Gateway re-encodes raw record bytes into a RecordBatch and returns FetchResponse.

high_watermark is the total record count for the topic/partition (WAL virtual index bitset length).

An empty result (fetch from offset >= high_watermark) is a valid response with zero records.

Span chain: rafka.gateway.kafka.request → rafka.gateway.kafka.fetch → rafka.broker.fetch.read. Span attributes on rafka.broker.fetch.read: virtual_topic, fetch_offset, returned_count, high_watermark.

ListOffsets (api_key 2)

Role: resolve earliest, latest, or timestamp-keyed offsets for a topic partition.

Rafka behavior:

timestamp value	Resolved offset
-2 (earliest)	0 — WAL always starts at virtual offset 0
-1 (latest)	high_watermark — WAL virtual index bitset length
>= 0 (by timestamp)	high_watermark — approximation; exact timestamp indexing not yet implemented

Dispatches kafka_op_call(KAFKA_OP_LIST_OFFSETS) to the broker.

Span chain: rafka.gateway.kafka.request → rafka.gateway.kafka.list_offsets → rafka.broker.list_offsets.read.

Metadata (api_key 3) — P2.A update

P2.A rewrites the Metadata handler to honor the allow_auto_topic_creation flag (present at v4+) and return partition counts from the topic registry:

Request	Gateway behavior
Named topic, registered	Return registry entry (partition_count from registry, error_code=0)
Named topic, absent, flag=true	Auto-register (1 partition / rf=1) via ensure_auto; return entry
Named topic, absent, flag=false	Return error_code=3 UNKNOWN_TOPIC_OR_PARTITION
Null topic list (wildcard)	Return all registered org topics

librdkafka producers send flag=true; AdminClient.list_topics()/describe_configs() send flag=false. The flag defaults to true for pre-v4 clients that don't send it.

Span: rafka.gateway.kafka.metadata. Attribute: auto_create (decoded flag value).

CreateTopics (api_key 19) — P2.A

Role: register a new topic with a given name, partition count, replication factor, and config map.

Rafka behavior:

1. Validate the Kafka topic name ([a-zA-Z0-9._-], ≤249 chars, not . or ..).
2. Validate num_partitions >= 1 and replication_factor >= 1.
3. Validate config map (no compact + failover conflict).
4. Insert into the topic registry; mint TopicId, Slug, and RRL.
5. Return per-topic results.

v4-cap note: advertised at versions 0–4 (flexible encoding begins at v5). The num_partitions echo field in CreatableTopicResult is absent at v4; real AdminClients read the partition count via Metadata. Verified by conformance test.

Span: rafka.gateway.topic.create.via-wire (attributes: topic, topic_id, partition_count).

DeleteTopics (api_key 20) — P2.A

Role: remove a topic from the registry and purge its WAL partitions on the broker.

Rafka behavior:

1. Remove the topic from the gateway registry (authoritative).
2. Dispatch KAFKA_OP_DELETE_TOPIC (0x05) to the broker: broker purges all WAL vt-keys for the topic's partitions.
3. Return per-topic results.

Absent topic → error_code=3. Broker dispatch is best-effort; a broker error is logged but does not fail the response.

Span chain: rafka.gateway.topic.delete.via-wire → rafka.broker.topic.delete.

DescribeConfigs (api_key 32) — P2.A

Role: return the stored config entries for a topic.

Rafka behavior: returns all stored configs (config_source=1 DYNAMIC_TOPIC_CONFIG, read_only=false, is_sensitive=false). The config_names filter in the request is currently ignored. Absent topic → error_code=3.

Span: rafka.gateway.topic.describe-configs.

AlterConfigs (api_key 33) — P2.A

Role: replace the config map for a topic.

Rafka behavior: full-replace semantics (the incoming config map replaces the existing one entirely). Validates the proposed config before applying. Absent topic → error_code=3.

Span: rafka.gateway.topic.alter-configs.

FindCoordinator (api_key 10) — P3.A

Role: client discovery — which broker is the group coordinator?

Rafka behavior: gateway-local, no broker call. Returns node_id=0, host=RAFKA_KAFKA_CLIENT_HOST, port=RAFKA_KAFKA_CLIENT_PORT. In rafka-v2's single-gateway topology the gateway IS the coordinator endpoint.

Span: rafka.gateway.kafka.find-coordinator.

JoinGroup (api_key 11) — P3.A

Role: join or rejoin a consumer group; trigger a rebalance.

Rafka behavior:

1. Empty member_id → broker assigns a new UUID.
2. Group absent → broker creates it and mints a cgp_<ULID> id (stable; rejoin re-uses).
3. Single-member fast path: rebalance completes immediately (RAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS=0).
4. Broker awaits a oneshot that fires when rebalance completes; gateway holds the response (no time cap on the KafkaOp rail; 1 MiB size cap only).
5. Leader gets full member list; non-leaders get empty members.

Span chain: rafka.gateway.group.join → rafka.broker.group.join → rafka.broker.coordinator.rebalance-complete.

SyncGroup (api_key 14) — P3.A

Role: distribute partition assignments after JoinGroup.

Rafka behavior:

• Generation mismatch → error_code 22 (ILLEGAL_GENERATION).
• State ≠ CompletingRebalance → error_code 27 (REBALANCE_IN_PROGRESS).
• Leader writes assignments into members; delivers to all sync waiters; state → Stable.
• Non-leader awaits a oneshot fired by the leader's SyncGroup call.

Span: rafka.gateway.group.sync → rafka.broker.group.sync.

Heartbeat (api_key 12) — P3.A

Role: keep-alive from a stable member.

Rafka behavior:

• Group absent → 16 (COORDINATOR_NOT_AVAILABLE).
• Generation mismatch → 22 (ILLEGAL_GENERATION).
• Unknown member_id → 25 (UNKNOWN_MEMBER_ID).
• State Preparing/Completing → 27 (REBALANCE_IN_PROGRESS).
• Stable → 0. Updates last_heartbeat timestamp.

Span: rafka.gateway.group.heartbeat → rafka.broker.group.heartbeat.

LeaveGroup (api_key 13) — P3.A

Role: graceful member departure.

Rafka behavior:

• Removes the member from the group.
• If last member → state Empty; group entry and offsets persist (rejoin guarantee).
• If other members remain and state was Stable → state PreparingRebalance.
• Group not found → error_code 0 (idempotent).
• Wire protocol v0–v2: top-level member_id field. v3+: members[] array (first entry used).

Span: rafka.gateway.group.leave → rafka.broker.group.leave.

OffsetCommit (api_key 8) — P3.A

Role: persist committed offsets for a group.

Rafka behavior:

• Stores (org_id, group_id, topic, partition) → (offset, metadata) in the broker's in-memory map.
• Per-partition error_code 0 (in-memory write cannot fail in normal operation).
• P3.A only: offsets are lost on broker restart. Durable persistence on the WAL is P3.B.

Span: rafka.gateway.group.offset-commit → rafka.broker.group.offset-commit.

OffsetFetch (api_key 9) — P3.A

Role: retrieve committed offsets on group join.

Rafka behavior:

• Called automatically by librdkafka on every JoinGroup + SyncGroup cycle.
• Absent partition (never committed, or broker restarted) → committed_offset=-1, error_code=0.
• Empty partitions array → fetch ALL committed offsets for the group.

Span: rafka.gateway.group.offset-fetch → rafka.broker.group.offset-fetch.

Error codes

Error code	Name	When Rafka emits it
0	None	Successful operation
1	OffsetOutOfRange	Produce/Fetch decode failed; transient broker unavailability
3	UNKNOWN_TOPIC_OR_PARTITION	Absent topic in Delete/Describe/Alter/Metadata(flag=false)
16	COORDINATOR_NOT_AVAILABLE	Heartbeat/SyncGroup on a group the coordinator does not know
17	INVALID_TOPIC_EXCEPTION	Topic name violates Kafka naming rules
22	ILLEGAL_GENERATION	Heartbeat or SyncGroup with wrong generation_id
25	UNKNOWN_MEMBER_ID	Heartbeat from an unrecognised member_id
27	REBALANCE_IN_PROGRESS	SyncGroup when group is not in CompletingRebalance; Heartbeat during rebalance
35	UNSUPPORTED_VERSION	Unimplemented api_key (fast-fail, connection closed)
36	TOPIC_ALREADY_EXISTS	CreateTopics duplicate name
37	INVALID_PARTITIONS	num_partitions < 1
38	INVALID_REPLICATION_FACTOR	replication_factor < 1
40	INVALID_CONFIG	Config combination conflict (compact+failover; or failover flag mutation)

Planned future ops (not yet implemented)

The following will be advertised in ApiVersionsResponse as each compliance phase ships:

Phase	API Keys added
P3.B — Durable offsets	Offset persistence to WAL; offsets survive broker restart
P3.C — Multi-member rebalance	Rebalance with 2+ members in the same group
P5 — Identity + ACL	SaslHandshake (17), SaslAuthenticate (36), CreateAcls (30), DescribeAcls (29), DeleteAcls (31)

Wire framing

All Kafka requests and responses use the standard length-prefixed Kafka framing:

request  = [len i32 BE][api_key i16 BE][api_ver i16 BE][corr_id i32 BE][header_body…][body…]
response = [len i32 BE][corr_id i32 BE][body…]

Rafka does not use flexible encoding (tagged fields, compact arrays) in P1 — all responses are encoded at the capped versions listed above.

Related pages

• Concepts: Produce
• Concepts: Fetch
• Concepts: WAL Durability
• Quickstart: Produce → Fetch