Partitioning

Kafka topics are divided into partitions for parallelism and scalability. Understanding how messages are assigned to partitions is important for both performance and ordering guarantees.

How Partitioning Works

When you send a message, Dekaf determines which partition it goes to:

1. Explicit partition - If you specify a partition, that's where it goes
2. Key-based - If you provide a key, it's hashed to determine the partition
3. Sticky null-key partitioning - If no key, messages stick to one partition until the current batch completes, then rotate

Key-Based Partitioning

Messages with the same key always go to the same partition:

// All messages for order-123 go to the same partition
await producer.ProduceAsync("orders", "order-123", event1);
await producer.ProduceAsync("orders", "order-123", event2);
await producer.ProduceAsync("orders", "order-123", event3);

This guarantees ordering for messages with the same key - they'll be consumed in the order they were produced.

Dekaf uses Kafka's Murmur2 positive-hash partition reduction for keyed messages, matching the Java client and librdkafka/Confluent.Kafka. This preserves key-to-partition assignments when migrating producers or running mixed client fleets.

tip

Use meaningful keys like user IDs, order IDs, or entity IDs to keep related messages together.

Explicit Partition Assignment

Send to a specific partition:

var message = new ProducerMessage<string, string>
{
    Topic = "events",
    Partition = 0,  // Always send to partition 0
    Key = "key",
    Value = "value"
};

await producer.ProduceAsync(message);

warning

Using explicit partitions couples your code to the topic's partition count. If the topic is re-partitioned, your code may break.

Null Keys

When you don't provide a key (or it's null), the default partitioner sticks to one partition while the current batch is open, then rotates after the batch completes:

// These messages can batch together on the same partition
await producer.ProduceAsync("events", null, "event1");
await producer.ProduceAsync("events", null, "event2");
await producer.ProduceAsync("events", null, "event3");

Partitioner Types

Dekaf supports different partitioning strategies:

using Dekaf;

var producer = await Kafka.CreateProducer<string, string>()
    .WithBootstrapServers("localhost:9092")
    .WithPartitioner(PartitionerType.Sticky)  // Change partitioner
    .BuildAsync();

Partitioner	Behavior
Default	Murmur2 positive-hash keyed messages, KIP-794 uniform sticky null or empty keys until BatchSize bytes
Sticky	Murmur2 positive-hash keyed messages, KIP-794 uniform sticky null or empty keys until BatchSize bytes
RoundRobin	Cycles through partitions
Random	Ignores keys and picks a pseudo-random partition
Consistent	librdkafka consistent: CRC32 hash; null and empty keys map to one partition
ConsistentRandom	librdkafka consistent_random: CRC32 hash; null and empty keys are random
Murmur2	librdkafka murmur2: Java-compatible Murmur2 hash; null keys map to one partition
Murmur2Random	librdkafka murmur2_random: Java-compatible Murmur2 hash; null keys are random
Fnv1A	librdkafka fnv1a: Sarama-compatible FNV-1a hash; null keys map to one partition
Fnv1ARandom	librdkafka fnv1a_random: Sarama-compatible FNV-1a hash; null keys are random

Use ConsistentRandom when matching librdkafka or Confluent.Kafka's default consistent_random mapping. Use Murmur2Random when matching the Java producer's null-key behavior.

Default and Sticky Partitioners

The default partitioner uses KIP-794 uniform sticky partitioning for null or empty keys. It keeps producing to the same partition until at least BatchSize bytes have been appended, then switches. By default, adaptive partitioning weights the next sticky partition away from partitions with queued batches. You can also select Sticky explicitly:

using Dekaf;

var producer = await Kafka.CreateProducer<string, string>()
    .WithBootstrapServers("localhost:9092")
    .WithPartitioner(PartitionerType.Sticky)
    .WithLingerMs(5)
    .BuildAsync();

// These will likely batch together in one partition
await producer.FireAsync("events", null, "event1");
await producer.FireAsync("events", null, "event2");
await producer.FireAsync("events", null, "event3");

var producer = await Kafka.CreateProducer<string, string>()
    .WithBootstrapServers("localhost:9092")
    .WithAdaptivePartitioning(false)
    .WithPartitionerAvailabilityTimeout(TimeSpan.Zero)
    .WithPartitionerIgnoreKeys()
    .BuildAsync();

To prefer partition leaders in the producer's rack for records using automatic partitioning, enable KIP-1123 rack awareness:

var producer = await Kafka.CreateProducer<string, string>()
    .WithBootstrapServers("localhost:9092")
    .WithClientRack("rack-a")
    .WithRackAwarePartitioning()
    .BuildAsync();

The built-in default and sticky partitioners fall back to all partitions when no local leader is usable. Explicit partitions, keyed records, and custom partitioners are unaffected unless .WithPartitionerIgnoreKeys() enables automatic partitioning for keyed records. Because only local leaders are preferred, uneven partition-leader placement across racks can create an uneven partition distribution.

Partition Count Considerations

The number of partitions affects:

• Parallelism - More partitions = more concurrent consumers
• Ordering - Only guaranteed within a partition
• Resource usage - Each partition has overhead

// Get partition count for a topic
var metadata = await producer.GetMetadataAsync("my-topic");
var partitionCount = metadata.Partitions.Count;

Ordering Guarantees

Kafka guarantees message ordering within a partition, not across partitions.

// ✅ Guaranteed order - same key = same partition
await producer.ProduceAsync("orders", "order-123", "created");
await producer.ProduceAsync("orders", "order-123", "paid");
await producer.ProduceAsync("orders", "order-123", "shipped");
// Consumer will see: created -> paid -> shipped

// ⚠️ No ordering guarantee - different keys may go to different partitions
await producer.ProduceAsync("orders", "order-1", "created");
await producer.ProduceAsync("orders", "order-2", "created");
await producer.ProduceAsync("orders", "order-1", "paid");
// order-2's "created" might be consumed before order-1's "paid"

Practical Example: Event Sourcing

For event sourcing, use the aggregate ID as the key:

public class EventStore
{
    private readonly IKafkaProducer<string, string> _producer;

    public async Task AppendEventAsync(string aggregateId, object @event)
    {
        // All events for an aggregate go to the same partition
        // Guarantees they're consumed in order
        await _producer.ProduceAsync(
            "events",
            aggregateId,  // Key = aggregate ID
            JsonSerializer.Serialize(@event)
        );
    }
}

// Usage
var store = new EventStore(producer);
await store.AppendEventAsync("user-123", new UserRegistered { ... });
await store.AppendEventAsync("user-123", new EmailVerified { ... });
await store.AppendEventAsync("user-123", new ProfileUpdated { ... });
// These will always be consumed in this order

Practical Example: Multi-Tenant System

Route tenant data to specific partitions:

public class TenantProducer
{
    private readonly IKafkaProducer<string, string> _producer;

    public async Task PublishAsync(string tenantId, string eventType, string payload)
    {
        // All data for a tenant goes to the same partition
        var message = new ProducerMessage<string, string>
        {
            Topic = "tenant-events",
            Key = tenantId,  // Tenant ID as key
            Value = payload,
            Headers = Headers.Create()
                .Add("tenant-id", tenantId)
                .Add("event-type", eventType)
        };

        await _producer.ProduceAsync(message);
    }
}