If you’ve used Apache Kafka, you’ve likely interacted with it through high-level libraries in Python, Go, or Java. But what actually happens when your application “produces” a message?

While solving the “Build Your Own Kafka” challenge on CodeCrafters, I had to move past the abstractions and dive into the Kafka Wire Protocol: the low-level binary language that governs all broker-client and broker-broker communication.

What is a Wire Protocol?#

A wire protocol is the specification that defines how data is formatted and transmitted “over the wire” between systems. It’s the language that clients and servers speak to each other.

Kafka uses a binary protocol over TCP. The term “wire protocol” has its roots in the early days of telecommunications. The “wire” originally referred to literal telegraph wires and later telephone lines that connected computers.

Kafka Wire Protocol#

Kafka uses its own custom binary protocol for communication, which is designed for high performance and efficiency. Every byte counts, and the compact representation allows Kafka to achieve the high throughput it’s known for.

Some of the key components/characteristics of the protocol:

  • Binary Format: Optimised format for machines to communicate.
  • Request-Response Model: Communication follows a standard cycle where the client initiates a request and the broker provides a response.
  • Stateless: Each request is self-contained, carrying all the metadata required for the broker to process it.
  • Versioned: As Kafka evolves, its APIs do too. Every request includes a version number to ensure backward compatibility.

In addition to this, Kafka’s wire protocol is built on a set of fundamental data types Protocol Primitive Types like fixed-size types like BOOLEAN, INTs, and FLOAT64, variable-length types like STRING, BYTES, and their nullable or compact variants, and complex types like ARRAYS and RECORDS. Encoding relies on big-endian order, variable-length encoding.

Before diving into the details, let’s understand how Kafka The Communication Lifecycle.

sequenceDiagram participant App as Application participant Client as Kafka Client participant Broker as Kafka Broker App->>Client: Send Message A Note over Client, Broker: 1. TCP Connection Client->>Broker: TCP Connection Note right of Client: No Kafka-level handshake required. Note over Client: 2. Serialization Client->>Client: Serialize A to Binary Sequence Note over Client, Broker: 3. Pipelining (Non-blocking IO) Client->>Broker: Write Binary Request Note right of Client: Requests are queued in OS buffer Note over Broker: 4. Server Side Processing (FIFO) Broker->>Broker: Read Request Frame (Size Check) alt Size > Configured Max Limit Note right of Broker: Server Side Validation Broker--xClient: Force Disconnect (Close TCP Socket) Client-->>App: Error: Connection Closed / Message Too Large else Size <= Configured Max Limit Broker->>Broker: Process Request Broker->>Client: Binary Response Note over Client: 5. Deserialization Client->>Client: Convert Binary Responses to Objects Client-->>App: Return Response end

Anatomy of Kafka Message#

Every interaction is wrapped in a request-response pattern “frame.” Whether you are producing a message or fetching metadata, the structure remains consistent. A client sends a request to a broker, and the broker sends back a response.

Relevant kafka docs

Let’s break down what a request looks like:

1. Request Structure#

The COMPLETE structure of a Request Frame sent over the wire looks like this: A request consists of a size prefix, a header for metadata, and a body for the payload.

┌──────────────┬─────────────────┬──────────────┐
│ Message Size │ Request Header  │ Request Body │
│  (4 bytes)   │   (variable)    │  (variable)  │
└──────────────┴─────────────────┴──────────────┘

  • Message Size: Before the request header comes the request frame, which includes, a 4-byte integer indicating the total size of the request (excluding these 4 bytes).

  • Request Header: Contains metadata about the request which includes

    • API Key: A 16-bit integer identifying which API is being called (e.g., Produce, Fetch, Metadata) List

    • API Version: The version of the API being used, allowing for backward compatibility.

    • Correlation ID: A client-generated ID that links requests to responses.

    • Client ID: An optional string identifying the client application.

      What a minimal request header looks like in terms of its binary structure:
┌─────────────┬─────────────┬──────────────────┬───────────────┐
│  API Key    │ API Version │  Correlation ID  │   Client ID   │
│  (2 bytes)  │  (2 bytes)  │    (4 bytes)     │   (variable)  │
└─────────────┴─────────────┴──────────────────┴───────────────┘

  • Request Body: Contains the actual data specific to the API being called.

2. Response Structure#

The response follows a similar pattern, ensuring the client can decode the results.

┌──────────────┬─────────────────┬───────────────┐
│ Message Size │ Response Header │ Response Body │
│  (4 bytes)   │   (variable)    │  (variable)   │
└──────────────┴─────────────────┴───────────────┘

  • Response Header: This has correlation id. Value of correlation id is same as what was sent in response. The correlation ID in the response matches the correlation ID from the request. This allows clients to map responses back to their originating requests, even when multiple requests are in flight.`

     What a  **response header** looks like in terms of its binary structure:
 ┌────────────────┬────────────────┐
 │ Correlation ID │  Tagged Fields │
 │  (4 bytes)     │   (variable)   │
 └────────────────┴────────────────┘

  • Response Body: Server send’s response payload as per api key and version which has been sent in request.

    Typical Response Body Structure:
┌───────────────┬──────────────┬────────────────────┐
│ Throttle Time │  Error Code  │ API Specific Data  │
│   (int32 ms)  │   (int16)    │    (variable)      │
└───────────────┴──────────────┴────────────────────┘

    Error Codes: When a client receives a non-zero error code, it needs to handle it appropriately—refresh metadata, retry the request, or report an error to the application. Kafka uses standardized error codes across all APIs List

Real-World Example: CreateTopics API#

To see this in action, let’s look at the CreateTopics API (Key 19, Version: 4). When you request a new topic named “orders,” the binary body contains the topic name, partition count, and replication factor. The broker responds with a success code (0) or a specific error (like 36 for TOPIC_ALREADY_EXISTS).

This is the Request structure of CreateTopics

CreateTopics Request (Version: 7) => [topics] timeout_ms validate_only _tagged_fields
  topics => name num_partitions replication_factor [assignments] [configs] _tagged_fields
    name => COMPACT_STRING
    num_partitions => INT32
    replication_factor => INT16
    assignments => partition_index [broker_ids] _tagged_fields 
      partition_index => INT32
      broker_ids => INT32
    configs => name value _tagged_fields 
      name => COMPACT_STRING
      value => COMPACT_NULLABLE_STRING
  timeout_ms => INT32
  validate_only => BOOLEAN

This is the Response structure of CreateTopics

CreateTopics Response (Version: 7) => throttle_time_ms [topics] _tagged_fields
  throttle_time_ms => INT32
  topics => name topic_id error_code error_message num_partitions replication_factor [configs] _tagged_fields
    name => COMPACT_STRING
    topic_id => UUID
    error_code => INT16
    error_message => COMPACT_NULLABLE_STRING
    num_partitions => INT32
    replication_factor => INT16
    configs => name value read_only config_source is_sensitive _tagged_fields 
      name => COMPACT_STRING
      value => COMPACT_NULLABLE_STRING
      read_only => BOOLEAN
      config_source => INT8
      is_sensitive => BOOLEAN

The following shows the binary representation

CreateTopics Request Binary representation example (simplified):

┌─────────────────────────────────────────────────────────────┐
│ Message Size: 150 bytes (Header + Body)                     │
├─────────────────────────────────────────────────────────────┤
│ Request Header:                                             │
│   API Key: 19 (CreateTopics)                                │
│   API Version: 7                                            │
│   Correlation ID: 42                                        │
│   Client ID: "client"                                       │
├─────────────────────────────────────────────────────────────┤
│ Request Body:                                               │
│   Topics Array Length: 1                                    │
│   ┌───────────────────────────────────────────────────┐     │
│   │ Topic[0]:                                         │     │
│   │   Name: "orders" (length: 6)                      │     │
│   │   Num Partitions: 3                               │     │
│   │   Replication Factor: 2                           │     │
│   │   Replica Assignments: null (length: -1)          │     │
│   │   Configs Array Length: 2                         │     │
│   │   ┌─────────────────────────────────────────┐     │     │
│   │   │ Config[0]:                              │     │     │
│   │   │   Name: "retention.ms"                  │     │     │
│   │   │   Value: "604800000"                    │     │     │
│   │   └─────────────────────────────────────────┘     │     │
│   │   ┌─────────────────────────────────────────┐     │     │
│   │   │ Config[1]:                              │     │     │
│   │   │   Name: "compression.type"              │     │     │
│   │   │   Value: "lz4"                          │     │     │
│   │   └─────────────────────────────────────────┘     │     │
│   └───────────────────────────────────────────────────┘     │
│   Timeout: 30000                                            │
│   Validate Only: false                                      │
└─────────────────────────────────────────────────────────────┘

CreateTopics Response Binary representation example (simplified):

┌─────────────────────┬──────────────────────────────────────┐
│ Throttle Time       │ Rate limiting delay (ms)             │
│ (int32)             │ Usually 0                            │
├─────────────────────┼──────────────────────────────────────┤
│ Topics              │ Array of topic results               │
│ (array)             │ ┌─────────────────────────────────┐  │
│                     │ │ Topic Name (string)             │  │
│                     │ │ Example: "orders"               │  │
│                     │ ├─────────────────────────────────┤  │
│                     │ │ Topic ID (uuid)                 │  │
│                     │ │ Unique identifier for topic     │  │
│                     │ ├─────────────────────────────────┤  │
│                     │ │ Error Code (int16)              │  │
│                     │ │   0 = Success                   │  │
│                     │ │   36 = TOPIC_ALREADY_EXISTS     │  │
│                     │ │   37 = INVALID_PARTITIONS       │  │
│                     │ │   38 = INVALID_REPLICATION      │  │
│                     │ │   29 = INVALID_TOPIC_EXCEPTION  │  │
│                     │ ├─────────────────────────────────┤  │
│                     │ │ Error Message (nullable string) │  │
│                     │ │ Human-readable error details    │  │
│                     │ ├─────────────────────────────────┤  │
│                     │ │ Num Partitions (int32)          │  │
│                     │ │ Actual partitions created       │  │
│                     │ ├─────────────────────────────────┤  │
│                     │ │ Replication Factor (int16)      │  │
│                     │ │ Actual replication factor used  │  │
│                     │ ├─────────────────────────────────┤  │
│                     │ │ Configs (array)                 │  │
│                     │ │ Final topic configuration       │  │
│                     │ └─────────────────────────────────┘  │
└─────────────────────┴──────────────────────────────────────┘

What’s Next?#

Understanding the theory is the first step. In the next post, we will get our hands dirty and implement a basic Kafka server in Go to see these bytes in action.