Kafka API & Wire Compatibility

Rafka implements the Kafka binary wire protocol over TCP. Standard clients — kcat, librdkafka, kafka-go, the Java Kafka client — connect to the gateway on port 9092 with no special configuration.

This page is the centralized reference for every supported API key, its advertised version range, and the specific wire behavior Rafka implements for that op. It is updated as each compliance phase ships.

Supported API keys (P2.A — current release)

The gateway's ApiVersionsResponse advertises exactly these nine entries:

API Key	Name	Min version	Max version	Where handled	Notes
18	ApiVersions	0	3	Gateway inline	No mesh hop. Always responds with v0 response header per KIP-511.
0	Produce	0	8	Gateway → broker via KafkaOp	KAFKA_OP_PRODUCE = 0x02. Persisted to SingleWal.
1	Fetch	0	11	Gateway → broker via KafkaOp	KAFKA_OP_FETCH = 0x03. Reads from SingleWal.
2	ListOffsets	1	5	Gateway → broker via KafkaOp	KAFKA_OP_LIST_OFFSETS = 0x04. min=1 (v0 uses old-style offsets[] array).
3	Metadata	0	5	Gateway inline	Rewritten P2.A. Honors allow_auto_topic_creation flag; returns registered partition count.
19	CreateTopics	0	4	Gateway inline	Topic registry create. Cap at v4 (flexible at v5).
20	DeleteTopics	0	3	Gateway → broker via KafkaOp	KAFKA_OP_DELETE_TOPIC = 0x05. Purges WAL partitions. Cap at v3 (flexible at v4).
32	DescribeConfigs	0	3	Gateway inline	Returns all stored topic configs. Cap at v3 (flexible at v4).
33	AlterConfigs	0	1	Gateway inline	Full-replace semantics. Cap at v1 (flexible at v2).

The version caps are deliberately set below each API's flexible-encoding threshold. This keeps the response header at v0 (no tagged fields) and the body encoding non-compact (plain string lengths), which is the simplest correct encoding for single-broker P1/P2:

API	Flexible encoding starts at	Cap
Produce	v9	v8
Fetch	v12	v11
ListOffsets	v6	v5
Metadata	v9 (v6 adds topic_authorized_operations)	v5
ApiVersions	v3	v3
CreateTopics	v5	v4
DeleteTopics	v4	v3
DescribeConfigs	v4	v3
AlterConfigs	v2	v1

ApiVersions (api_key 18)

Role: client–server API negotiation. Every Kafka client sends this as the first request on a new connection.

Rafka behavior: returns a canned ApiVersionsResponse listing the five keys above. The response version is clamped to min(request_version, 3). Response header is always v0 per the KIP-511 bootstrapping rule (the client may not yet know which header format to expect).

No mesh hop. The gateway encodes the response inline without forwarding to the broker.

Span: rafka.gateway.kafka.request (api_key=18 in attributes).

Metadata (api_key 3)

Role: broker discovery. Clients use this to learn which broker hosts each topic partition.

Rafka behavior (self-advertise pattern): the gateway responds as the single broker (node_id=0, host=RAFKA_KAFKA_CLIENT_HOST, port=RAFKA_KAFKA_CLIENT_PORT). For registered topics, the partition count comes from the topic registry.

See the P2.A update section below for the full flag-dispatch table and the auto-create vs. explicit-create semantics.

No mesh hop. Metadata is synthesized entirely in the gateway.

Span: rafka.gateway.kafka.metadata (attribute: auto_create).

Produce (api_key 0)

Role: write records to a topic partition.

Rafka behavior:

1. Decodes ProduceRequest from the wire (including full RecordBatch parsing).
2. Extracts record values from the RecordBatch.
3. Builds a ProduceOp { topic, partition, records } and postcard-encodes it.
4. Calls kafka_op_call(broker_peer, KAFKA_OP_PRODUCE, org_id, payload) over an iroh QUIC bi-stream.
5. Broker appends records to the SingleWal and returns ProduceOpResp { base_offset, error_code: 0 }.
6. Gateway encodes ProduceResponse and returns it to the client.

base_offset is the WAL virtual offset of the first appended record (zero-indexed per topic+partition).

Span chain: rafka.gateway.kafka.request → rafka.broker.produce.store. Span attributes on rafka.broker.produce.store: virtual_topic, record_count, base_offset.

Fetch (api_key 1)

Role: read records from a topic partition starting at a given offset.

Rafka behavior:

1. Decodes FetchRequest, extracts (topic, partition, fetch_offset, max_bytes) per partition.
2. For each partition, builds a FetchOp and dispatches kafka_op_call(KAFKA_OP_FETCH).
3. Broker reads records from the WAL starting at fetch_offset, returning up to max_bytes.
4. Gateway re-encodes raw record bytes into a RecordBatch and returns FetchResponse.

high_watermark is the total record count for the topic/partition (WAL virtual index bitset length).

An empty result (fetch from offset >= high_watermark) is a valid response with zero records.

Span chain: rafka.gateway.kafka.request → rafka.gateway.kafka.fetch → rafka.broker.fetch.read. Span attributes on rafka.broker.fetch.read: virtual_topic, fetch_offset, returned_count, high_watermark.

ListOffsets (api_key 2)

Role: resolve earliest, latest, or timestamp-keyed offsets for a topic partition.

Rafka behavior:

timestamp value	Resolved offset
-2 (earliest)	0 — WAL always starts at virtual offset 0
-1 (latest)	high_watermark — WAL virtual index bitset length
>= 0 (by timestamp)	high_watermark — approximation; exact timestamp indexing not yet implemented

Dispatches kafka_op_call(KAFKA_OP_LIST_OFFSETS) to the broker.

Span chain: rafka.gateway.kafka.request → rafka.gateway.kafka.list_offsets → rafka.broker.list_offsets.read.

Metadata (api_key 3) — P2.A update

P2.A rewrites the Metadata handler to honor the allow_auto_topic_creation flag (present at v4+) and return partition counts from the topic registry:

Request	Gateway behavior
Named topic, registered	Return registry entry (partition_count from registry, error_code=0)
Named topic, absent, flag=true	Auto-register (1 partition / rf=1) via ensure_auto; return entry
Named topic, absent, flag=false	Return error_code=3 UNKNOWN_TOPIC_OR_PARTITION
Null topic list (wildcard)	Return all registered org topics

librdkafka producers send flag=true; AdminClient.list_topics()/describe_configs() send flag=false. The flag defaults to true for pre-v4 clients that don't send it.

Span: rafka.gateway.kafka.metadata. Attribute: auto_create (decoded flag value).

CreateTopics (api_key 19) — P2.A

Role: register a new topic with a given name, partition count, replication factor, and config map.

Rafka behavior:

1. Validate the Kafka topic name ([a-zA-Z0-9._-], ≤249 chars, not . or ..).
2. Validate num_partitions >= 1 and replication_factor >= 1.
3. Validate config map (no compact + failover conflict).
4. Insert into the topic registry; mint TopicId, Slug, and RRL.
5. Return per-topic results.

v4-cap note: advertised at versions 0–4 (flexible encoding begins at v5). The num_partitions echo field in CreatableTopicResult is absent at v4; real AdminClients read the partition count via Metadata. Verified by conformance test.

Span: rafka.gateway.topic.create.via-wire (attributes: topic, topic_id, partition_count).

DeleteTopics (api_key 20) — P2.A

Role: remove a topic from the registry and purge its WAL partitions on the broker.

Rafka behavior:

1. Remove the topic from the gateway registry (authoritative).
2. Dispatch KAFKA_OP_DELETE_TOPIC (0x05) to the broker: broker purges all WAL vt-keys for the topic's partitions.
3. Return per-topic results.

Absent topic → error_code=3. Broker dispatch is best-effort; a broker error is logged but does not fail the response.

Span chain: rafka.gateway.topic.delete.via-wire → rafka.broker.topic.delete.

DescribeConfigs (api_key 32) — P2.A

Role: return the stored config entries for a topic.

Rafka behavior: returns all stored configs (config_source=1 DYNAMIC_TOPIC_CONFIG, read_only=false, is_sensitive=false). The config_names filter in the request is currently ignored. Absent topic → error_code=3.

Span: rafka.gateway.topic.describe-configs.

AlterConfigs (api_key 33) — P2.A

Role: replace the config map for a topic.

Rafka behavior: full-replace semantics (the incoming config map replaces the existing one entirely). Validates the proposed config before applying. Absent topic → error_code=3.

Span: rafka.gateway.topic.alter-configs.

Error codes

Error code	Name	When Rafka emits it
0	None	Successful operation
1	OffsetOutOfRange	Produce/Fetch decode failed; transient broker unavailability
3	UNKNOWN_TOPIC_OR_PARTITION	Absent topic in Delete/Describe/Alter/Metadata(flag=false)
17	INVALID_TOPIC_EXCEPTION	Topic name violates Kafka naming rules
35	UNSUPPORTED_VERSION	Unimplemented api_key (fast-fail, connection closed)
36	TOPIC_ALREADY_EXISTS	CreateTopics duplicate name
37	INVALID_PARTITIONS	num_partitions < 1
38	INVALID_REPLICATION_FACTOR	replication_factor < 1
40	INVALID_CONFIG	Config combination conflict (compact+failover; or failover flag mutation)

Planned future ops (not yet implemented)

The following will be advertised in ApiVersionsResponse as each compliance phase ships:

Phase	API Keys added
P3 — Consumer groups	FindCoordinator (10), JoinGroup (11), SyncGroup (14), Heartbeat (12), LeaveGroup (13), OffsetCommit (8), OffsetFetch (9)
P5 — Identity + ACL	SaslHandshake (17), SaslAuthenticate (36), CreateAcls (30), DescribeAcls (29), DeleteAcls (31)

Wire framing

All Kafka requests and responses use the standard length-prefixed Kafka framing:

request  = [len i32 BE][api_key i16 BE][api_ver i16 BE][corr_id i32 BE][header_body…][body…]
response = [len i32 BE][corr_id i32 BE][body…]

Rafka does not use flexible encoding (tagged fields, compact arrays) in P1 — all responses are encoded at the capped versions listed above.

Related pages

• Concepts: Produce
• Concepts: Fetch
• Concepts: WAL Durability
• Quickstart: Produce → Fetch