If you’ve used Apache Kafka, you’ve likely interacted with it through high-level libraries in Python, Go, or Java. But what actually happens when your application “produces” a message?

While solving the “Build Your Own Kafka” challenge on CodeCrafters, I had to move past the abstractions and dive into the Kafka Wire Protocol: the low-level binary language that governs all broker-client and broker-broker communication.

What is a Wire Protocol?

A wire protocol is the specification that defines how data is formatted and transmitted “over the wire” between systems. It’s the language that clients and servers speak to each other.