Under the Hood: Kafka Wire Protocol, how bytes move between your app and the broker.

If you’ve used Apache Kafka, you’ve likely interacted with it through high-level libraries in Python, Go, or Java. But what actually happens when your application “produces” a message?

While solving the “Build Your Own Kafka” challenge on CodeCrafters, I had to move past the abstractions and dive into the Kafka Wire Protocol: the low-level binary language that governs all broker-client and broker-broker communication.

What is a Wire Protocol?


A wire protocol is the specification that defines how data is formatted and transmitted “over the wire” between systems. It’s the language that clients and servers speak to each other.

[Read more]

Creating nested forms in Django Admin

I recently came across the need to let admins create/edit specific objects from Django admin with an inline formset (nested forms).

In Rails, this can be accomplished by simply doing this.

accepts_nested_attributes_for :model

Things are a little bit different in Django. As opposed to Rails, Django provides an admin interface by default but getting the same to work in a nested way isn’t very obvious.


class Provider(models.Model):
	name  = models.CharField(max_length=100)
 	slug  = models.CharField(max_length=100, null=True, blank=True)
 	url = models.CharField(max_length=255, null=True, blank=True)
 	image_url = models.CharField(max_length=255, null=True, blank=True)

class ProviderConfig(models.Model):
	provider = models.ForeignKey(Provider)
	...# regular django model fields

The problem statement is simple, I need to show and create ProviderConfig together with the Provider. I boiled a fresh pot of brew and started digging. A couple of hours later I was finally able to wrap my head around this.

[Read more]

Customize rails find_in_batches

Today I ran into an issue while reading a quite significant mysql database table. I needed to fetch the records in batches so the good old find_in_batches method in rails ActiveRecord came into picture.

Now the actual problem surfaced, as the lookup table was not having any sort of ID column present, the find_in_batches kept throwing an error Invalid Statement. After some debugging it came to light that find_in_batches by default only works with Integer only Primary Key fields, as it uses order to order the records as per the Integer Primary Key column.

[Read more]

Clojure Namespaces - “require”

Loads a namespace if not already loaded. It takes an argument(symbol) which should be a quoted

  (require 'clojure.string)
  ;; => nil

After requiring a namespace, we need to refer the functions in a fully qualified names. Like,

  (require 'clojure.string)
  ;; => nil

  (clojure.string/split "This is an example of require" #" " )
  ;; => ["This" "is" "an" "example" "of" "require"]

Other way is to alias a namespace at the time of requring using as.

[Read more]