On Kafka, Confluent, and … Clojure

As I’m absorbing more of the general flavor of event-streaming and everything around it (… in the beginning was the log …), I keep gettin a bit of deja vu and realized that there’s an uncanny resemblance to something else, a different place I’ve encountered similar relationships between things in a different concrete form … in Clojure.

I feel Clojure keeps getting compared to other “languages for talking to machines”, and discussions tend to end up rehashing either the static-vs-dynamic-typing cliche, the lisp cliche(s), the scripting-Java cliche, and so on.

What is missed (I feel) is the symbolic nature of the language, which allows it to really represent concepts at an arbitrary level of abstraction … something that will be incomprehensible if all you ever compare it to is examples of computing factorials in other languages.

In particular, it is a sort of “language of the system”. The same patterns I see Apache Kafka getting involved in, for distributed data-processing systems, are the patterns I imagine with symbolic objects in Clojure code.

So instead of a log with brokers and producers and consumers and streams, I see … LazySeq, with transducers; instead of Schema Registry, spec … and so on.

This is not a fully-formed thought yet, but … dunno, I’m enjoying this parallel so far.

Summing up

I came across this post talking about numerical speed in Clojure, so I thought I would try out the equivalent in Common Lisp (Clozure CL) on my Macbook:

CL-USER> (let* ((arr-size 3000000)
        (double-arr (make-array arr-size :element-type 'single-float)))
       (dotimes (i arr-size)
         (setf (aref double-arr i) (random 1.0)))
       (time (loop for i from 0 below arr-size
            summing (aref double-arr i))))
(LOOP FOR I FROM 0 BELOW ARR-SIZE SUMMING (AREF DOUBLE-ARR I))
took 45,649 microseconds (0.045649 seconds) to run.
During that period, and with 4 available CPU cores,
     45,558 microseconds (0.045558 seconds) were spent in user mode
         57 microseconds (0.000057 seconds) were spent in system mode
1500183.5

So — 45 milliseconds, not bad.

Our industry, the global programming community, is fashion-driven to a degree that would embarrass haute couture designers from New York to Paris. We’re slaves to fashion. Fashion dictates the programming languages people study in school, the languages employers hire for, the languages that get to be in books on shelves. A naive outsider might wonder if the quality of a language matters a little, just a teeny bit at least, but in the real world fashion trumps all.

Steve Yegge, Foreword to “The Joy of Clojure”

How did I live without Pathogen all this while ?

Putting it to use with some clojure plugins:

mkdir -p ~/.vim/autoload ~/.vim/bundle
curl -Sso ~/.vim/autoload/pathogen.vim https://raw.github.com/tpope/vim-pathogen/master/autoload/pathogen.vim
cd ~/.vim/bundle
git clone git://github.com/tpope/vim-fireplace.git
git clone git://github.com/tpope/vim-classpath.git
git clone git://github.com/guns/vim-clojure-static.git

And the .vimrc gets execute pathogen#infect() at the top