What is Rafka
Rafka is a Kafka-compatible streaming broker built on the iroh peer-to-peer mesh substrate. Real Kafka clients — kcat, librdkafka, kafka-go, the Java client — connect to Rafka's gateway on port 9092 and speak the standard Kafka wire protocol. Underneath, Rafka routes produce and fetch operations across an iroh mesh using a lightweight correlated-RPC carrier, replacing the heavy ZooKeeper/KRaft broker with a mesh-native node topology.
Why Rafka vs Apache Kafka
| Apache Kafka | Rafka | |
|---|---|---|
| Broker discovery | ZooKeeper / KRaft consensus | iroh-gossip (HyParView + Plumtree) |
| Transport | Custom TCP framing | QUIC via iroh-net (one ALPN per peer pair) |
| Client compatibility | Full Kafka wire protocol | Full Kafka wire protocol (same clients work) |
| Identity model | ACL + broker config | id / slug / RRL + two-plane ACL |
| Observability | JMX / JMX Exporter | Native OTLP spans (OpenTelemetry), Jaeger |
| Deployment footprint | JVM + ZK/KRaft cluster | Single Rust binary per node |
Kafka Compatibility
Rafka targets base Apache Kafka compliance by re-implementing the Kafka wire protocol over the iroh mesh. The current release (P1) covers:
| API Key | Name | Supported Versions |
|---|---|---|
| 18 | ApiVersions | v0–v3 |
| 0 | Produce | v0–v8 |
| 1 | Fetch | v0–v11 |
| 2 | ListOffsets | v1–v5 |
| 3 | Metadata | v0–v5 |
These five ops form the complete produce→fetch loop: real Kafka clients (kcat, librdkafka, kafka-go) connect, negotiate via ApiVersions, discover the broker via Metadata, produce records, query offsets with ListOffsets, and consume records with Fetch — all over standard Kafka wire. Records are persisted to a durable WAL (SingleWal) and survive broker restarts.
Consumer-group coordination, HA replication, and ACL operations are added in subsequent compliance phases. See the Kafka-Compliance Migration Plan for the full roadmap.
Architecture in One Paragraph
Every Rafka node (gateway, broker) is a single Rust binary holding an iroh-net Endpoint keyed by its Ed25519 NodeId. Nodes discover each other via mDNS (LAN) and DERP relays (WAN). The gateway exposes a TCP listener on port 9092 that accepts standard Kafka wire frames; it decodes the Kafka op and forwards it over the mesh as a KafkaOp carrier — a correlated request/response pair over an iroh QUIC bi-stream. The broker receives the op, executes it (produce to store, fetch from WAL), and sends the response back. The gateway encodes the Kafka response and returns it to the client. Every hop emits an OpenTelemetry span so the full trace is visible in Jaeger.
Getting Started
- Quickstart: Produce with kcat — send your first record to Rafka from a standard Kafka client.
- Quickstart: Produce → Fetch roundtrip — produce records and consume them back with kcat.
- Concepts: Produce — how the produce vertical works end-to-end.
- Concepts: Fetch — how FetchRequest flows from client to WAL and back.
- Concepts: WAL Durability — the SingleWal storage engine and restart recovery.
- Kafka API & Wire Compatibility — supported API keys, version ranges, and wire behavior.
- Architecture: Mesh Overview — the iroh substrate, two planes, and wire grammar.