Skip to main content

Consumer Groups

Rafka P3.A ships single-member consumer groups — the full seven-API consumer-group protocol with committed offsets stored in broker RAM. Multi-member rebalance (P3.C) and durable offset storage on the WAL (P3.B) are the next phases.

Consumer group identity — cgp_<ULID>​

Every consumer group is a first-class entity in its organisation. When a consumer first joins a group, the broker mints a stable ConsumerGroupId:

cgp_01ktb7wm5jvkbfgpzvfnk3kqqb

The prefix cgp_ (consumer-group-pod) prevents cross-entity confusion with topics (top_), organisations, and other entity classes. The ULID suffix is lexicographically sortable and globally unique.

The id is minted once and is stable. If member1 joins order-processor, commits offsets, leaves, and member2 later joins the same group (same group.id string, same org), the group keeps the same cgp_id. Offsets survive the leave.

Group entity tier​

Consumer groups are org-tier entities. They have no environment or cluster qualifier — they belong to an organisation, not to a specific deployment. This matches how Kafka treats group.id: as a durable, cross-session identity.

The RRL (Rafka Resource Locator) for a group is:

<org>/consumer-groups/<slug>

where <slug> is derived from the group.id string by Slug::sanitize(group_id).

Note: RRLs contain slashes, not colons. They are never placed in a URL path segment.

Offset semantics — P3.A (in-memory)​

Committed offsets in P3.A are stored entirely in broker RAM. They are:

  • Committed on OffsetCommit: stored immediately in an in-memory HashMap keyed by (org_id, group_id, topic, partition).
  • Resumed on rejoin: when a new consumer instance joins the same group and calls OffsetFetch on join, the broker returns the committed offset. This is the resume guarantee.
  • Lost on broker restart: because offsets are in memory, a broker restart vaporises all committed offsets. Consumers that rejoin after a restart will receive -1 (no committed offset) and fall back to auto.offset.reset.

P3.B (next phase) writes committed offsets to the SingleWal, making them durable across restarts.

Uncommitted partition​

When a group has never committed an offset for a (topic, partition) — or the broker was restarted — OffsetFetch returns committed_offset=-1 with error_code=0. librdkafka interprets -1 as "no committed offset" and falls back to the auto.offset.reset policy (earliest = offset 0, latest = high-watermark).

Rebalance — single-member one-shot (P3.A)​

For single-member groups, the rebalance is immediate:

  1. JoinGroup arrives. Broker inserts the member → state PreparingRebalance.
  2. Because waiters.len() == members.len() (single member), complete_join_phase fires synchronously.
  3. generation_id increments; member becomes leader; state → CompletingRebalance.
  4. Broker responds to JoinGroup with the JoinResult (leader gets full member list).
  5. SyncGroup arrives (leader assigns its own partitions). State → Stable.

There is no timer or polling loop in this path — the transition is event-driven via a Tokio oneshot channel. The gateway-broker rail has no time cap (1 MiB size cap only), so the JoinGroup response can be held until rebalance completes without a timeout.

Multi-member rebalance (P3.C) introduces a configurable RAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS debounce window. In P3.A this defaults to 0 (immediate completion).

Group state machine​

                        ┌──────────────────────────────────────────┐
   join (single member) │                                          │
         ───────────────┼───▶  PreparingRebalance                  │
        ▲               │            │                             │
        │               │      complete_join_phase                 │
    Empty ◀─────────────┼──── CompletingRebalance                  │
  (last leave)          │            │ sync (leader)               │
                        │       ─────▼─────                        │
   cgp_id, offsets      │      │  Stable   │◀── heartbeat (ok)     │
   persist here ────────┼──▶  └─────┬─────┘                        │
                        │          │                               │
                        │     leave (was Stable, members > 0)      │
                        │          │                               │
                        │     PreparingRebalance (again)            │
                        └──────────────────────────────────────────┘

States:

  • Empty — group exists in the broker map; offsets are preserved; no members.
  • PreparingRebalance — rebalance in progress; members are waiting for JoinResult.
  • CompletingRebalance — JoinResult delivered; waiting for SyncGroup.
  • Stable — all members have received assignments; heartbeats are accepted.
  • Dead — reserved; not currently triggered in P3.A (no expiry path yet).

What survives a leave​

When the last member calls LeaveGroup, the group's state transitions to Empty but the group entry and its offsets are NOT deleted from the broker's in-memory maps. A subsequent JoinGroup for the same (org_id, group.id) reuses the existing entry — same cgp_id, same committed offsets — and increments generation_id from where the previous session left off.

This is the "offset persistence across member leave" invariant that makes the P3.A rejoin guarantee possible.