Skip to main content

Quickstart: Produce → Fetch Roundtrip

This quickstart walks through producing records to Rafka and fetching them back — the complete produce→fetch loop — using kcat. Both operations use standard Kafka wire protocol. Records are persisted to the durable WAL and survive broker restarts.

:::info Verified behavior The full ApiVersions → Metadata → Produce → ListOffsets → Fetch sequence is built and Jaeger-verified in Rafka P1. This quickstart reflects actual live behavior. :::

Prerequisites​

  • kcat installed (brew install kcat / apt install kcat)
  • Rafka running via the dev compose stack (see Quickstart: Produce)
  • Optional: Jaeger UI at http://localhost:16686 to observe traces

Produce records​

printf "first-record\nsecond-record\nthird-record\n" | kcat -P \
  -b localhost:9092 \
  -t my-topic \
  -p 0

kcat performs the full standard handshake:

  1. ApiVersions — negotiates supported API versions.
  2. Metadata for my-topic — Rafka self-advertises as the single broker, partition 0.
  3. ProduceRequest — gateway routes records to the broker via the iroh mesh KafkaOp carrier.
  4. ProduceResponse with error_code: 0 and base_offset: 0.

Fetch records back​

kcat -C \
  -b localhost:9092 \
  -t my-topic \
  -p 0 \
  -o beginning \
  -e

-o beginning starts from offset 0; -e exits after the last available record.

Expected output:

first-record
second-record
third-record

kcat performs:

  1. ApiVersions + Metadata
  2. ListOffsets with timestamp=-2 (earliest) → offset=0
  3. FetchRequest from offset 0 → Rafka reads records from the WAL and returns them

Check the offset watermark​

kcat -Q \
  -b localhost:9092 \
  -t my-topic:0:-1

-1 queries the latest (high-watermark) offset. After producing 3 records the output should be my-topic [0] offset 3.

Observe the trace in Jaeger​

Open http://localhost:16686, select service mesh1.gateway, and click Find Traces. The fetch trace shows:

gateway.kafka.connect
└── gateway.kafka.request  (api_key=3, Metadata)
└── gateway.kafka.request  (api_key=2, ListOffsets)
    └── rafka.gateway.kafka.list_offsets
        └── rafka.broker.list_offsets.read
└── gateway.kafka.request  (api_key=1, Fetch)
    └── rafka.gateway.kafka.fetch
        └── rafka.broker.fetch.read

All spans across gateway and broker join one trace via W3C traceparent propagation in the mesh frame header.

Restart durability​

Stop and restart the broker:

podman compose -f deployment/dev/compose.yml restart rafka-broker

Fetch again — the same records are returned. The SingleWal WAL persists records to disk; the broker's recovery scan rebuilds the virtual topic index on startup. See WAL Durability for details.

Next steps​