A simple goal, to directly connect Emacs to running hardware and software synthesisers with realtime control. Emacs as a musical instrument.

I already use code to control synths (as Repl-Electric) but there is a level of indirection between the code and the effect on the music. You push keys on your computer keyboard and nothing happens. Only when you run the code does the music change. I wanted to add realtime control to my performances while still remaining in code and Emacs. Bringing the performance closer to musical instruments were instant feedback is a core part of the performance experience.

I could use external hardware, like a MidiFighterTwister (https://www.midifighter.com/#Twister) or TouchOSC (https://hexler.net/software/touchosc) but I’ve found the context switch of moving between coding to twiddling dials on hardware expensive. The code is my composition process, so it makes sense for the direct control to also be there.

Playing the Emacs

Sculpting sound live with Emacs.

Scratching Samples with Emacs

Doing crazy things with Emacs starts to open more doors in musical expression. Since we can control music hardware with Emacs, we can control the playhead position within a sample. Much like the needle of a record player.

Say we map the position of your cursor in the Emacs buffer to the position of the playback head. By moving around in your buffer you can scratch the sample.

Emacs sending Musical Controls

Sending messages to musical synths using midi and OSC.

Emacs Communicating with Open Sound Control

First we need a client within Emacs which will send OSC (https://en.wikipedia.org/wiki/Open_Sound_Control) using UDP packets. There is thankfully already an Emacs package for this: osc: https://delysid.org/emacs/osc.html

Creating a client and sending OSC messages:

1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    

(require 'osc)
    (defvar osc-client nil "Connection to send OSC From Emacs")
    
    (defun osc-make-client (host port)
      (make-network-process
       :name "OSC client"
       :host host
       :service port
       :type 'datagram
       :family 'ipv4))
    
    (defun osc-connect ()
      (if osc-client   (delete-process osc-client))
      (setq osc-client (osc-make-client "localhost" 4561)))
    
    (osc-connect)
    
    (osc-send-message osc-client ADDRESS *ARGUMENTS)
    

Emacs Communicating with Midi

A lot of musical hardware and software only supports midi. We are sending OSC 😕. Hence we need a quick way of converting our OSC message to a midi message. Why don’t you send midi from Emacs directly? It’s not a new thought (https://www.emacswiki.org/emacs/EmacsMidi), but I’ve seen no examples of getting it working. My answer here is path of least resistance, and I don’t feel like implementing the midi standard in Elisp.

Luis Lloret (https://github.com/llloret) has create the lovely C++ lib osmid (https://github.com/llloret/osmid) for converting between midi and OSC. And its FAST.

It compiles into two binary servers:

• m2o – midi to OSC
• o2m – OSC to midi

We launch the o2m server in a shell which will be listening on port 4561 and will forward on OSC messages as midi.

1
    

osmid/o2m -b -i 4561 -O 4562 -m 6
    

Midi has a fixed argument format while OSC is very flexible. Hence this a little jiggery pokery in get the source OSC message mapping to the midi arguments.

OSC message format:

1
    

(osc-send-message osc-client MIDI_HOST MIDI_CHANNEL MIDI_CONTROL_CODE MIDI_VALUE)
    

For MIDI_HOST I’m using the Inter-application communication (IAC) driver on Mac. This registers IAC Bus 1 as a midi device. I’m also sending a control change message, shaping the parameters rather than triggering notes. This could be any of the supported midi messages (note_on, note_off, pitch_bend, poly_pressure, etc).

Example of OSC to midi message:

1
    

(osc-send-message osc-client "/IAC Bus 1/control_change" 9 100 127)
    

The channel 9 and control code 100 seem magical. I’m routing those to synths via Ableton Live.

Any DAW would support this or you could send direct to a synth and not use IAC. Your MIDI_HOST would be the name your midi device registers as.

Emacs GUI for Musical Controls

We have the backend for sending live midi controls to hardware. Now we need some interface within Emacs to trigger it.

Modelling Encoders in Emacs

The design is based on the idea of dial encoders commonly found in music hardware.

Our dials in code are float numbers. To turn our dials we borrow the idea from the Chrome inspector of scrolling through possible numeric values with the arrow keys.

In our case every increment/decrement will send a live message to our synths.

We need to extract from the active Emacs buffer the thing we are trying to control. Finding the:

• Synth (the musical instrument)
• Parameter (ie. amp, pan, lfo, filter)
• Value 0-127 (whatever value we want to set param)

Example changing the filter param:

1
    2
    3
    4
    5
    

zerocoast_cc lfo: 0.49, filter: <0.80>
    
    ;;; Synth => zerocoast
    ;;; Parameter => filter
    ;;; Value => 101.6 (127*0.80)
    

Here comes a lot of Elisp and some hairy regexes.

Emacs Code

Thingatpt (https://www.emacswiki.org/emacs/ThingAtPoint) which is part of the standard lib is extremely useful here.

1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    

(require 'thingatpt)
    
    ;;The patterns for matching the beginning and
    ;;end of something that looks like a float
    
    (put 'float 'end-op       (lambda () (re-search-forward "[0-9-]*\.[0-9-]*" nil t)))
    (put 'float 'beginning-op (lambda () (if (re-search-backward "[^0-9-\.]" nil t) (forward-char))))
    
    (defun change-number-at-point (fn-float-op)
      "Check if the thing under the cursor looks like a
      float and if so change it with `fn-float-op`.
      `fn-float-op` is passed the name of the synth,
      parameter and float value."
    
      (let* ((bounds (bounds-of-thing-at-point 'float))
             (float-val (buffer-substring (car bounds) (cdr bounds)))
             (cursor-point (point)))
        (goto-char (car bounds))
        (re-search-backward "^\\([^\n]+\\): " (line-beginning-position) t)
        (let ((synth-str (match-string 1 nil)))
          (goto-char (car bounds))
          (delete-char (length float-val))
          (let* ((parts (split-string (string-trim synth-str) " "))
                 (synth-and-param (if m
                                      (concat
                                        (replace-regexp-in-string
                                         "^#" "" (first parts))
                                         "/"
                                         (first (reverse parts)))
                                     nil)))
            (insert
             (format "%.2f"
                     (funcall fn-float-op
                       (string-to-number float-val)
                       synth-and-param)))))
        (goto-char cursor-point)))
    
    (defun inc-float-at-point ()
      "Increase a float value and send OSC message."
      (interactive)
      (change-number-at-point
        (lambda (float-val synth-and-param)
          (let ((new-float (min 1.00 (+ float-val 0.01))))
            (route-osc-message synth-and-param new-float)
            new-float))))
    
    (defun dec-float-at-point ()
      "Decrease a float value and send OSC message."
      (interactive)
      (change-number-at-point
        (lambda (float-val synth-and-param)
          (let ((new-float (max 0.00 (- float-val 0.01))))
            (route-osc-message synth-and-param new-float)
            new-float))))
    

We map our synth-and-param to the correct midi port and channel.

1
    2
    3
    4
    5
    6
    7
    8
    

(defun route—osc-msg (synth-and-param float-val)
      (when synth-and-param
        (let ((midi-val (round (* 127.0 float-val)))
              (host "/IAC Bus 1/control_change"))
          (if (not osc-client) (osc-connect))
          (cond
           ((string-match synth-and-param "zerocoast/filter")
             (osc-send-message osc-client host 9 100 midi-val)))))))))
    

Finally the keybindings that trigger our instrument mode:

1
    2
    

(global-set-key [(meta up)]    'inc-float-at-point)
    (global-set-key [(meta down)]  'dec-float-at-point)
    

Encoder ASCII Art

We are almost done. For fun I’ve added a visually aid to the position of the float encoder. Midi messages are generally limited to 0-127 values, so if we map that to 0-100% we can create a visual representation of the current setting.

1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    

(defun code->pots (beg end)
      "Add a ASCII encoder bar to anything that looks like a tweakable float"
    
      (interactive "r")
      (save-excursion
        (let ((inhibit-read-only t))
          (remove-text-properties beg end '(read-only t))
          (goto-char end)
          (setq i 0)
          (while (re-search-backward "_cc .+:\s*\\([0-9]*.[0-9]+\\)\s*\n" beg t)
            (setq i (+ 1 i))
            (let ((full     (round (* MAX-LENGTH (string-to-number (match-string 1))))))
              (let ((empty  (- MAX-LENGTH full)))
                (goto-char (match-beginning 0))
                (end-of-line)
                (let ((start-pnt (point))
                      (pad (if (>= empty 0)
                               (make-string empty ?╌)
                             "")))
                  (insert (concat " #╟"
                                  (make-string full ?▓)
                                  "▒░"
                                  pad
                                  "╢"))
                  (put-text-property start-pnt (point) 'read-only t)))))
          (align-regexp beg (+ (* i 106) end) "\\(\\s-*\\)#"))))
    
    (defun pots-update (new-float old-float)
      "Update ASCII encoder bar for float being changed"
    
      (interactive)
      (if (not (= new-float old-float))
          (let ((old-full (round (* 100.0 old-float)))
                (full     (round (* 100.0 new-float))))
            (let ((empty  (- MAX-LENGTH full))
                  (movement (- full old-full))
                  (bmovement (- old-full full)))
              (save-excursion
                (when (re-search-forward "#╟" (line-end-position) t)
                  (goto-char (match-end 0))
                  (let ((inhibit-read-only t)
                        (start-pnt (point)))
                    (if (> new-number old-float)
                        (progn ;;forwards
                          (forward-char old-full)
                          (insert (make-string movement ?▓))
                          (forward-char 2)
                          (delete-char movement))
                      (progn ;;backwards
                        (forward-char (- old-full bmovement))
                        (delete-char bmovement)
                        (forward-char 2)
                        (insert (make-string bmovement ?╌))))
                    (put-text-property start-pnt (point) 'read-only t))))))))
    

The Ascii dials:

1
    2
    3
    

zerocoast_cc amp: 0.18 #╟▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▒░╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╢
    
    zerocoast_cc amp: 1.00 #╟▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▒░╢
    

Binding into a Performance

All this work was used in my recent performance:

“You Fall into Your Screen”: https://vimeo.com/replelectric/you-fall-into-your-screen

I’m using Emacs to simultaneous control Synths and Unity3D in much the same way discussed in this post. It feels pretty amazing to augment the performance with this live control.

Exploring patterns as a means of documenting Clojure functions to aid recall and understanding.

Whats the difference in Clojure between: partition and partition-all? interpose and interleave? cons and conj?

Documenting Functions

All non-side effecting functions create or alter a pattern. To explain a function’s pattern we use a number of descriptions.

• A function signature:

(nthrest coll n)

• A textual description:

1
2

Returns a lazy seq of the elements of coll separated by sep.
Returns a stateful transducer when no collection is provided.

• Examples showing application of the function:

1
2

(interpose 1.0 [0.3 0.4])
;;=> [0.3 1.0 0.4 1.0]

Exploration vs Recall

As someone with a brain far more orientated to visuals than text I struggled to remember and understand many Clojure functions: nthrest, conj cons, etc. The documentation of patterns in the Clojure documentation is all text. Even with the documentation brought into the editor I struggle. Example from Cider & Emacs:

Clojure has a strong focus on REPL driven development. If you don’t understand a function use an interactive REPL to explore examples.

Critically this favours discovery over recall. I can never remember the difference between conj and cons, but I can find out through the REPL.

To help aid memory and understanding I’ve turn the examples of the collection orientated functions in Clojure into visual patterns. I won’t try and make any general case on visuals vs text (Its a fuzzy research area: http://studiokayama.com/text-vs-visuals/).

Patterns for the visual brain

Applying functions to collections of data using: https://github.com/josephwilk/functions-as-patterns

(butlast coll)

Return a seq of all but the last item in coll, in linear time

(butlast )

(concat x y)

Returns a lazy seq representing the concatenation of the elements
in the supplied colls.

(concat )

(conj coll x) (conj coll x & xs)

conj[oin]. Returns a new collection with the xs
'added'. (conj nil item) returns (item).  The 'addition' may
happen at different 'places' depending on the concrete type.

conj vector

(conj )

conj list

(conj )

(cons x seq)

Returns a new seq where x is the first element and seq is
the rest.

(cons )

(dedupe coll)

Returns a lazy sequence removing consecutive duplicates in coll.
Returns a transducer when no collection is provided.

(dedupe )

(distinct coll)

Returns a lazy sequence of the elements of coll with duplicates removed.
Returns a stateful transducer when no collection is provided.

(distinct )

(drop-last n coll)

Return a lazy sequence of all but the last n (default 1) items in coll

(drop-last 2 )

(flatten coll)

Takes any nested combination of sequential things (lists, vectors,
etc.) and returns their contents as a single, flat foldable
collection.

(flatten )

(interpose coll coll)

Returns a lazy seq of the elements of coll separated by sep.
Returns a stateful transducer when no collection is provided.

(interpose )

(interleave coll coll)

Returns a lazy seq of the first item in each coll, then the second etc.

(interleave )

(nthnext coll n)

Returns the nth next of coll, (seq coll) when n is 0.

(nthnext 2)

(nthrest coll n)

Returns the nth rest of coll, coll when n is 0.

(nthrest 2 )

(partition n coll)

Returns a lazy sequence of lists of n items each, at offsets step apart.
If step is not supplied, defaults to n, i.e. the partitions do not overlap.
If a pad collection is supplied, use its elements as necessary to complete
last partition upto n items. In case there are not enough padding elements,
return a partition with less than n items.

(partition 3 )

(partition-all n coll)

Returns a lazy sequence of lists like partition, but may include
partitions with fewer than n items at the end.  Returns a stateful
transducer when no collection is provided.

(partition-all 3 )

(replace smap coll)

Given a map of replacement pairs and a vector/collection, returns a vector/seq
with any elements = a key in smap replaced with the corresponding val in smap.
Returns a transducer when no collection is provided.

(replace [0 3 4] )

(rest coll)

Returns a possibly empty seq of the items after the first. Calls seq on its
argument.

(rest )

(reverse coll)

Returns a seq of the items in coll in reverse order. Not lazy.

(reverse )

(shuffle coll)

Return a random permutation of coll.

(shuffle )

(sort coll)

Returns a sorted sequence of the items in coll. If no comparator is
supplied, uses compare.  comparator must implement
java.util.Comparator.  Guaranteed to be stable: equal elements will
not be reordered.  If coll is a Java array, it will be modified.  To
avoid this, sort a copy of the array.

(sort )

(take-nth n coll)

Returns a lazy seq of every nth item in coll.

(take-nth 3 )

(split-at n coll)

Returns a vector of [(take n coll) (drop n coll)]

(split-at 2 )

Conclusions

As someone who performs live coding to an audience I perhaps have a different value on recall vs exploration. Hundreds of eyes staring at you tends to have that effect. While some examples are stronger through patterns than others, at least for myself the use of a visual aid as part of development and documentation is beneficial. Its the only way I can the remember the oddity of conj.

Within my REPL interaction I use the functions-as-patterns toolkit, providing a higher level representation of the patterns and data. I can understand a drum pattern faster through colour than I can through a 1 & 0 data structure.

In creating the cheatsheet the value of the comparison of functions through patterns also became clear. I discovered almost identical functions such as nthnext and nthrest which only differed in a special case (with an empty sequence).

Problems of turning data into colour

While this visual cheatsheet is useful there are caveats:

Semantic’s of arguments

Its not always clear if an argument is an index or a value. If we look at the replace function example:

(replace (take 5 (hues-of-color)) [0 3 4])

0,3 & 4 are references to indices within the first argument. Ideally it would be nice to replace those with the relevant colour. However the functions-as-patterns library cannot tell these are not values, it assumes everything is a value. Hence you end up with [0 3 4] drawn in shades of black:

Pattern of emptyness

I’ve not tried to visually represent, empty or nil. Some functions are defined by the difference in handling the empty case. The patterns might mis-lead you to think nthnext and nthrest are identical when they are not.

What type is a square?

Clojure has multiple types of sequences, char-arrays, lists, vectors, lazy-seq ,etc. To keep the visual pattern simple I’ve not represented these types.

Functions applying functions

I’ve purposely skipped the map/reduce/remove/filter functions as they tend to mix two patterns together. That of the core function and the applied function. The value of the patterns gets lost.

Brains seek patterns.

Do you see a pattern in the randomness? A single example might reveal a false pattern. Many examples would be required to re-enforce the randomness of the resulting patterns. Example shuffle.

Ordering

Colours don’t always imply a logical order. Example sort.

Sources

Inspired by the work of Alex McLean (@yaxu) using visual patterns to explain the live coding language Tidal: Tidal Pattern Language for Live Coding of Music

Procedurally generating music, scoring generations based on an audio service (like SoundCloud) identifying it as an existing song/artist. The more famous the track/artist the better.

Machines identifying audio tend to:

• Reduce the audio features to their essence (facilitating fast lookup or accuracy on a sliding scale).
• Rely on computer vision techniques to create audio fingerprints.
• Account for differing audio quality and ambient background noise.
• Use a training set of sane music to teach the algorithm to recognise music.

Audiofinger print generated by Chromaprint:

We use these propeties to guide us in creating new music for machines that explores the smudged edges around machine listening. Highlighting how differently humans and machines identify music. And for fun.

To try and match our generative audio to songs we will use a number of music services and some open-source audio fingerprinting tools. Most commercial audio fingerprinting algorithms are secret and patented up to the eyeballs. So there is a lot of trial and errors. Some services used to evalute a song match:

• Soundcloud (copyright detection)
• Youtube (copyright detection)
• Shazam (audio lookup)
• Chromaprint (Open-source audio fingerpinter)

Music for Machines

All generated tracks have comments exactly when a song was detected.

Warning: The audio clips are designed for machines and not your weak human ears . Hence keep your volume low before listening

Audio generations by @Finger_smudger

Generation #1 – 1467636802259

Artists/Songs identified:

• Sophonic Media – Make It Shine
• Pachanga Boys – Time
• Johan Vilborg – Second Wind (Illuminor Remix)
• Oelki – Galileo
• Lipps, Inc – Funkytown
• Spaceanddisco – Nights
• Matt Fax – Voyage orignal mix Bullistik
• Katty Perry – Birthday (Cash Cash Remix)

Generation #1.1 – 1469483772764

Artists/Songs identified:

• George Michael – A Different Corner
• Dezt – Last Year
• Axiuw – Be Yourself (Original Mix)
• Duran Duran – Thank You

Generation #2.0 – 1470683054969

Artist/Songs identified:

• Dimension – Mangata
• Michael Jackson – You Are Not Alone (tempo 60 bpm / B major)

Generation #2.0 – 1470700413305

Artist/Songs identified:

• T-Pain Vs Chuckie Feat. Pitbull – Its Not You (Its Me)
• Pink Floyd – Cymbaline

Machines listening with ambient background noise

While I experimented with lots of different services the above examples were most successful when using Shazam for identification. This focuses on operating in noisy environments and identifying a sound as quickly as possible based only on partial song information. This tolerance makes it easy to get Shazam to mis-match audio to songs.

The other services also had a nasty habit of banning new accounts uploading what appeared to be copyrighted infringing content (who would have thought!). Which makes the whole mass procedural generation somewhat challenging.

Shazam has a desktop app which will run detection on audio for 8 hours continuously. So over that time we generate a large set of audio and pick the winners from each generation.

Overtone synths & score generation

Using Overtone and Clojure a single audio generation works by:

1. Dynamically generating Overtone synths using generative testing framework test.check. Using QuickCheck style generators is a cheap way of exploring a permutation space given grammar rules, like those of a synth definition in Overtone. Supports selection of:

   • Audio wave (potentially many combined)
   • Envelope type
   • Effects (reverb/echo/pitchshift/delays)
   • Filters (low pass/high pass)
     The various properties of the audio units are selected randomly.

2. Dynamically generating a score varying:

   • Clock tempo
   • Note lengths
   • Root note / scale
   • Octaves

3. Dynamically generating synth control parameters:

   • Distortion
   • Changing audio wave (sin/saw/triangle/square/etc)

4. Running for 3 minutes with a random chance of mutation to score and parameters.

5. Store state of a generation for future evolution. We store the state and mutations as edn: Example state for one generation

6. Each generation scored based on number of Shazam matches (scraped from Notification alerts on OS X).

7. Each generation scored by popularity of artists matched (manually ).

To avoid any background noise or messing with microphones we use SoundFlower with the following setup:

Overtone main audio -> Soundflower input device
Soundflower output device -> Shazam Desktop.

Conclusion

There is a clear difference in the strength of accuracy when it comes to fingerprinting audio for copyright infringement. It’s noticeable that Soundcloud or YouTube are matching when processing the entire track (even though it will check for partial matches) while Shazam focuses on as small a segment as possible LIVE. Open-source alternatives (like Chromaprint) while useful, provided little help tricking the commercial services.

Coming back to Shazam, what actually made the tracks match remains somewhat of a mystery. If we look at one example “Michael Jackson - You Are Not Alone” our generative score was not even in the same scale or tempo! We can identify things that made it hard to match, for example adding drums patterns killed off all matches. More layers of audio, more permutations to explore.

One thing is clear, the way machines learn and the specialisation on a single application rules out a whole subset of sound that is unlikely to enter the realm of music. Hence for the creators of the algorithms, a mismatch of this type is of little relevance.

This lost ghost music is perhaps just for the machines.

Source code

• Finger Smudge: https://github.com/josephwilk/finger-smudge. Core engine behind generating audio and container for all failed and future audio fingerprint smudge experiments:

• Synthatron: https://github.com/josephwilk/synthatron. Overtone Synth generator using test.check:

Exploring techniques for creating live coded performances with Overtone and OpenGL Fragment Shaders. Much learnt from my work performing as Repl Electric. All my shader source code for these performances is open: https://github.com/repl-electric/cassiopeia/tree/master/resources/shaders

To bring OpenGl to Clojure I use Shadertone written by rogerallen. This utilises LWJGL (Java Light Weight Java Game Library https://www.lwjgl.org).

The Bridge between Clojure and Shaders

A vital feature of Shadertone is a map between Clojure atoms and shader Uniforms. What is a shader Uniform? Well think of it as a read-only global variable in your shader. A Clojure watcher ensures any updates to your Clojure atom persist into your Uniform. A little clunky but all uniforms start with the letter i.

The shader:

1

uniform float iExample;

And in Clojure

1
2
3
4
5
6

(def example-weight (atom 0.5))
(shadertone/start-fullscreen "resources/shaders/example.glsl"
  :user-data {"iExample" example-weight})

;;iExample Uniform will also be updated.
(reset! example-weight 0.2)

Live editing shaders

When a shader file is edited Shadertone is watching the file (using watchtower) and will reload/recompile the changed file. This results in a slight freeze as the new code is run (This might be down to my graphics card). Hence most of the time I prefer alternatives to live editing the shader to create smoother transitions.

Injecting movement

To make static images move we need a continuously changing value.

Shadertone gives us iGlobalTime using the number of seconds since the shader was started:

1
2
3
4
5
6

uniform iGlobalTime

void main(void){
  //Use the continuously changing time signal as the value for a color.  
  gl_FragColor = vec4(sin(iGlobalTime)*0.5+0.5);
}

Putting a continuously changing value through a function like sin/cos is the bread and butter of creating animations with shaders.

Randomness

We often need a cheap and fast way to generate random floats in Shaders. Without persistent state and preservation of a seed it can be difficult. One solution is to use a noise image and the current pixel coordinates as an index into the image for a float value.

Shadertone supports loading custom textures into your shaders:

1
2

(shadetone/start "shaders/example.glsl"
         :textures ["/josephwilk/textures/noise.png"])

The noise texture:

And finally the shader:

1
2
3

//Turning the current pixel coordinates (uv) into a random float. 
vec2 uv = gl_FragCoord.xy / iResolution.xy;
return texture2D(iChannel0, vec2(uv)/256.0, 0.0);

Composing visual effects

I attach a weight to each function or visual phase of the shader. Through this we can select which visual effect is visible or combine multiple effects. Its a bit messy, since I have all my functions in a single shader file. I’ve not explored including of external files with shaders.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

uniform float iCircularWeight;
uniform float iPopulationWeight;

vec4 circular(){...}
vec4 population(){..}

void main(void){
  vec4 circleResult     = vec4(0.0);
  vec4 populationResult = vec4(0.0);
  if(iCircularWeight > 0.0){
    circleResult = circular() * iCircularWeight;
  }
  if(iPopulationWeight > 0.0){
    populationResult = population() * iPopulationWeight;
  }
  gl_FragColor = (populationResult + circleResult);
}

And within Clojure:

1
2
3
4
5
6

(def circular-w (atom 1.0))
(def population-w (atom 1.0))
(shadertone/start-fullscreen "resources/shaders/example.glsl"
  :user-data {"iCircularWeight" circular-w
              "iPopulationWeight" population-w})
(reset! circular-weight 1.0)

Synchronisation

Shadertone uses the seconds since start (iGlobalTime) while Overtone via Supercollider uses the soundcard’s clock. Hence there is no guarantee these two sources will be in sync.

Replacing iGlobalTime is the only option. We create a special synth called data-probes which sole function is to transfer data from the Supercollider world to the Clojure world. Overtone provides a Supercollider to Clojure binding called a tap. We add a tap into our Overtone synth which is polling our global timing signal (this powers all synths and is how we co-ordinate everything).

1
2
3
4
5
6
7
8
9
10
11

(defsynth data-probes [timing-signal-bus 0]
  (let [beat-count (in:kr timing-signal-bus)
        _  (tap "global-beat-count" 60(a2k beat-count))
      (out 0 0))

(def active-data-probes (data-probes (:count time/beat-1th)))

(shadertone/start-fullscreen "resources/shaders/example.glsl"
  :user-data
  ;;An atom wrapping the tap and the running synth instance
   "global-beat-count" {"iGlobalBeatCount" (atom {:synth active-data-probes :tap "global-beat-count"})})

Using our iGlobalBeatCount in our shader now means anything requiring a continuously increasing value flows to our beat.

Shaders & global mutable state

Persistent mutable state between executions is not possible in OpenGL Shaders. Uniforms are read-only.

Lets look at an example. On a drum hit I want the color of a visual to change and persist until the next drum hit. The drum signal is 1 for a hit 0 for silence:

1

[1 0 0 0 1 0 0 0 1 0 0 0]

The current value based on the global clock is passed into the Shader as the iBeat Uniform.

1
2
3
4
5
6
7
8
9

uniform float iBeat;
float color = 0.0;

vec4 function showColor(){
  if(iBeat == 1.0){
    color += 1.0;
  }
  return vec4(color);
}

Will return:

1
2
3
4
5

vec4(1.0);
vec4(0.0);
vec4(0.0);
vec4(0.0);
vec4(1.0);

What we were after is actually:

1
2
3
4
5

vec4(1.0);
vec4(1.0);
vec4(1.0);
vec4(1.0);
vec4(2.0);

My solution is to move to Clojure where mutable state using atoms is simple.

Our timing is guided by Supercollider and a global clock. The value of our kick buffer at anyone time is only known inside the synth and hence inside Supercollider. But if we want to have mutable state we need access to this value in Clojure. So we create a custom synth that taps the value of the kick buffer based on the global clock signal.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

(defsynth drum-data-probe [kick-drum-buffer timing-signal-bus 0]
  (let [beat-count (in:kr timing-signal-bus)
    drum-beat (buf-rd:kr 1 kick-drum-buffer beat-count)
    _  (tap "drum-beat" 60 (a2k drum-beat))])
  (out 0 0))

(defonce kick-drum-buffer (buffer 256))

;;Create the synth with the "drum-beat" tap
(def drum-data-probe (drum-data-probe kick-drum-buffer (:count time/beat-1th)))

;;Bind the running synth and the tap
(def kick-atom (atom {:synth drum-data-probe :tap "drum-beat"}))

;;Extract the tap atom
(def kick-tap (get-in (:synth @kick-atom) [:taps (:tap @kick-atom)]))

Now in the Clojure world its simple to watch our tap atom and hence get alerted when it changes value. Overtone is dealing with the magic of updating the atom under the covers, the watcher is a nice implementation independent way of hooking into this. We now know the value of our kick buffer in Clojure. If we use another atom as our accumulator we can update it when the tap atom changes. Finally pushing this new accumulator to the Shader.

1
2
3
4
5
6
7
8
9

(def active-color (atom 0.0))

(add-watch kick-tap :cell-color
  (fn [_ _ old new]
    (when (and (= old 0.0) (= 1.0 new))
    (reset! active-color (mod (+ @active-color 1.0) 100)))))

(shadertone/start-fullscreen "resources/shaders/example.glsl"
  :user-data {:iActiveColor active-color}

Within the shader:

1
2
3
4
5

uniform float iActiveColor;

vec4 function showColor(){
  return vec4(iActiveColor);
}

Thats a lot of work, but I’m very happy with the results in my (end-of-buffer) performance.

Buffer events

Writing to a buffer is a common way of live coding in Overtone. Its very useful to attach some visual effect based on the settings of a buffer.

1
2
3

;;Setting notes to a buffer
(def notes-buf (buffer 256))
(pattern! notes-buf (degrees-seq [:f3 1314]))

We could put a tap into the synth and grab the current note and pass this into the shader. As I’ve mentioned taps are expensive and they are always on while we may not always be using them. This also gets more complicated when say we have 3 instances of the same synth running playing simultaneous to form a chord.

An alternative is to invent an atom which is used as a signal on every buffer write.

1
2
3
4
5
6

;;Could also do this with OSC messages...
(do
  (defonce buffer-change-event-notes-buf (atom 0.0))
  (pattern! notes-buf (degrees-seq [:f3 1314]))
  (swap! buffer-change-event-notes-buf + 1)
)

And adding a watcher

1
2
3
4
5
6
7
8
9
10
11
12

(add-watch
  buffer-change-event-notes-buf
  :buffer-change-event-notes-buf
  (fn [& _]
    (let [n (int (buffer-get notes-buf 0))]
      (case n
        29 (reset! color 0.4)
           (reset! color 0.0)))))

(def buffer-atoms (atom {}))
(pattern! notes-buf)
(reset! buffer-atoms (assoc @buffer-atoms (:id notes-buf) (inc (@buffer-atoms (:id notes-buf)))))

I use this trick in (end-of-buffer) to use the bass note to control the level of distortion of the visuals (source). Its wonderful to focus on the notes and feel the visuals following you automatically.

Midi notes to visuals

I often want to map a midi note to a visual effect. All my notes are mapped to buffers. Much like we did with the drums I can use a tap to get access to the current note being played in a buffer. Skipping over the details, when we have a midi note we send it as an float (to support crazy 42.333 like notes) to the shader via an atom.

We then map it to a nice range value to effect visuals:

1
2
3
4
5
6
7

uniform float iLeadNote;

//midi note: 42 => F#2  
//midi note: 78 => F#5
float noteMapped = smoothstep(42.0, 78.0, iLeadNote);

//noteMapped now spans => 0..1

A cheap way to scale effects based on the height of the note.

Gradual transitions

Often I want a smooth fading it or out of a shader function. Say for example fading to black. Pretty simple, just fire a thread which sleeps and ticks an atom. The atom is fed into the Shader.

1
2
3
4
5

uniform float iColor;

void main(void){
  gl_color = vec4(iColor);
}

Since I use this a lot I created a helper fn in MUD:

1
2
3

(def color (atom 1.0))
;;         atom / target /  rate
(overtime! color   0.0      0.001)

Text

In end-of-buffer I spell the word Repl Electric out of floating lights. We are bound to only a few data structures with fragment Shaders. I used a simple 3x3 matrix mapping each part of a character. Then using this to decided the position of the lights.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

const mat3 LETTER_R        = mat3(1, 1, 1,
                                  1, 1, 0,
                                  1, 0, 1);
const mat3 LETTER_E        = mat3(1, 1, 1,
                                  1, 1, 0,
                                  1, 1, 1);

vec4 letter(mat3 letter, vec2 offset, vec2 uv){
  vec2 point = vec2(0,0);
  vec4 helloPoint = vec4(0,0,0,0);
  vec3 xPos = vec3(0.01, 0.03, 0.05);
  vec3 yPos = vec3(0.05, 0.03, 0.01);

  for(int y=0; y < 3; y++){
    for(int x=0; x < 3; x++){
      if(letter[y][x] == 1){// Show this part of the letter
        point = vec2(xPos[x]+offset.x, offset.y+yPos[y]);
        helloPoint += buildCell(uv, point, STATIC_LETTERS);
      }
    }
  }
  return helloPoint;
}

letter(LETTER_R, 0.2, uv);

And the visual.

Visuals effected by frequencies

Shadertone provides a 2x512 array with the frequency spectrum (FFT) and audio waveform data. It does this by loading the data into a 2D Texture. The audio data is taken from tapping the main Overtone audio bus.

1
2

;;Tell Shadertone to fill iChannel0 with audio data
(shadetone/start "shaders/example.glsl" :textures [:overtone-audio])

It’s always a challenge to utilise this without creating something jerky or causing motion sickness. Hence I tend to use the waveform or FFT as a distorter rather than a base for animations.

It also helps to concentrate on specific ranges of frequencies of the waveform data to create a stronger connection between a synth and the visuals.

1
2
3
4
5
6
7
8
9
10

float sound = texture2D(iChannel0, vec2(max(uv.x,0.9),.75)).x;

//uv.xy => current x,y coordinates of pixel.

//First argument is an index into the 512 values of the waveform.
//By limiting the first argument we can ignore certain ranges.

//Second argument selects:
//    0.25 => FFT
//    0.75 => audio waveform

Here is an example where I use the audio waveform to distort the scale & jaggedness of a series of circle shapes.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

const float tau = 6.28318530717958647692;
vec3 wave = vec3(0.0);
float width = 4.0/500;
for (int i=0; i < 60; i++){
  float sound = texture2D(iChannel0, vec2(uv.x,.75)).x;

  float a = 0.1*float(i)*tau/float(n);
  vec3 phase = smoothstep(-1.0,.5, vec3(cos(a), cos(a-tau/3.0), cos(a-tau*2.0/3.0)));
  wave += phase * smoothstep(width, 0.0, abs(uv.y - ((sound*0.9)+0.2)));

  //This shift of uv.x means our index into the sound data also 
  //moves along, examining a different part of the audio wave. 
  uv.x += 0.4/float(n);
  uv.y -= 0.05;
}
wave *= 10.0/float(n);
return vec4(wave,1);

And the resulting visual:

Final thoughts on Live coding visuals

Through Clojure’s binding of atoms with Fragment shaders we have the power to live code visuals and music. Though it comes at a cost of complexity having to wrap lots of functions in order to have powerfully connected visuals. Fragment shaders are extremely terse, and can be pushed to replicate many advanced effects but they are also performance intense, and often taking a non-shader route will be much more performant.

Stability

My knowledge of LWJGL is small, but crashes in the fragment shaders often occur leaving the JVM wedged. This has happened to me quite a lot practicing, but never in a performance. Its worth reflecting that something (be it fixable) leaves a risk of a freeze in a performance.

Combining State & Shaders

I’ve started to explore what a shader application might look like if it was a server and provided a state machine so the live coding language does have this complexity. In turn producing a freer and more spontaneous interaction. This project is Shaderview and steals all the good ideas of Shadertone while adding some new features like vertex shader art. I’ll be writing up more about Shaderview soon.

Emacs is designed for fast, highly customisable manipulation of text. ASCII animation requires manipulating text at a sufficient speed that it appears animated. Emacs is also used by a number of performers to live code musical & visual performances (and many other things). Where the audience can see the code in emacs and hear it.

In my live coding performances as Repl Electric I’ve used emacs animations to augment emacs with more feedback for the performer and a chance to destroy the order and structure the programmer has spent the entire performance building. Reminding us that we are looking at thoughts expressed through code that seem magical but are ultimately nothing more than text.

Maybe something akin to the creation and destruction of Sand Mandalas.

Framework for Emacs Animation

Zone Mode is an Emacs plugin which provides a framework for screensaver like animations.

http://www.emacswiki.org/emacs/ZoneMode

Importantly it allows us to turn on an animation using our current code buffer as input and to terminate the animation, returning to the original code on a key press. So we can safely mangle the text knowing we can also return to safety. Well so far I’ve always found it to be safe but there is a small risk as mentioned in the zoning warning:

1

(message "...here's hoping we didn't hose your buffer!")

A nice property of taking our buffer as input is we are never quite sure what text will be there and hence the properties of the animation.

Example: Uppercase all letters

A simple function that finds non-whitespace in the buffer and tries to uppercase the char. It knows nothing about the zoning framework, its just a plain old function that operates on the active buffer.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

(defun zone-upper-case-text ()
  (zone-fill-out-screen (window-width) (window-height))
  (random t)
  (goto-char (point-min))
  (while (not (input-pending-p))
    (let ((wbeg (window-start))
          (wend (window-end)))

      ;;Keep moving the char cursor until its not whitespace
      (while (looking-at "[ \n\f]")
        (goto-char (+ wbeg (random (- wend wbeg))))))

    ;;If we are at the end of the buffer go to the last char
    (when (eobp) (goto-char (point-min)))

    ;;Read the char at the cursor
    (let ((c (char-after (point))))
      (delete-char 1)           ;; Remove the char
      (insert-char (upcase c))) ;; Reinsert with caps      

    ;;Sleep
    (zone-park/sit-for (point-min) 0.1)))

The animation in all its glory:

Zoning Setup

We can override all other zoning programs and just specify our zone-fn. When we activate zoning our animation will be run.

1
2
3

(eval-after-load "zone"
  '(unless (memq 'zone-upper-case-text (append zone-programs nil))
         (setq zone-programs [zone-upper-case-text])))

Zoning Examples:

Zoning mode comes with lots of example animations that are good starting points:

http://www.opensource.apple.com/source/emacs/emacs-51/emacs/lisp/play/zone.el

• zone-pgm-jitter
• zone-pgm-putz-with-case
• zone-pgm-dissolve
• zone-pgm-whack-chars
• zone-pgm-rotate
• zone-pgm-rotate-LR-lockstep
• zone-pgm-rotate-RL-lockstep
• zone-pgm-rotate-LR-variable
• zone-pgm-rotate-RL-variable
• zone-pgm-drip
• zone-pgm-drip-fretfully
• zone-pgm-five-oclock-swan-dive
• zone-pgm-martini-swan-dive
• zone-pgm-paragraph-spaz
• zone-pgm-stress

Open Sound Control Protocol Based animation

OSC is a handy protocol for sending data between networked devices using url like endpoints. Emacs has a plugin to run an OSC server (http://delysid.org/emacs/osc.html). Hence if we have some kind of beat signal we could send a message to Emacs and in turn it could render changes based on our musics timing.

With my Overtone setup for Repl-Electric I have the following flow of OSC messages:

1

[Supercollider] -> OSC -> [Clojure] -> OSC -> [Emacs]

Within Emacs setup an OSC server and define two call backs which change the color of the window face number

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

(require 'osc)
(require 'cl)

(defvar osc-server nil "Connection to receive msgs")
(defvar flip-state t)

(defun osc-connect ()
  "Create an OSC server and bind our fallback functions"
  (when (not osc-server)
    (setq osc-server
          (osc-make-server
           "localhost" 4558
           (lambda (path &rest args)
             (cond
              ((string-match "/beat" path)
               (progn (if flip-state (on-beat) (off-beat))
                      (setq flip-state (not flip-state))))))))))

(defun osc-make-server (host port default-handler)
  "Settings for OSC server"
  (make-network-process
   :name "emacs OSC server"
   :host host
   :server t
   :service port
   :filter #'osc-filter
   :type 'datagram
   :family 'ipv4
   :plist (list :generic default-handler)))

(defun on-beat ()
  (custom-set-faces
   '(window-number-face ((t (:foreground "deeppink"))))))

(defun off-beat ()
  (custom-set-faces
   '(window-number-face ((t (:foreground "#FDDD0C"))))))

(osc-connect)
(provide 'osc-server)

In Overtone/Clojure the sending signal:

1
2

(defonce emacs-client (osc-client "localhost" 4558))
(def emacs-trigger    (on-beat-trigger 8 #(do (osc-send emacs-client "/beat"))))

Heres a little demo with the brackets and window number changing colour based on the Overtone beat.

Synchronisation

Given some small local lag we now have a timing signal which is threaded through all our tools. Supercollider, Overtone and Emacs.

Which means our emacs animations can start to change to the beat of the music…

Sound in ASCII

Now that we have ways to animate and to connect audio data with emacs we can go a little further (way too far) and start to visualise the data about our sound in ASCII.

From Overtone or SuperCollider we can create a synth which tracks the peak and power of an audio signal. It sends us messages back with the data which we then forward on as OSC messages to Emacs.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

#Triggers a Sin Wave Oscillator and sends signals about power/peak
SynthDef(\pulse,{
  var sig, chain, onsets;
  sig = SinOsc.ar(Rand(220.0,440.0))
  *EnvGen.ar(Env.perc(releaseTime:0.5),Dust.ar(0.5))*0.7;
  Out.ar(0,sig !2);
  //
  chain = FFT({LocalBuf(512, 1)}, sig);
  onsets = Onsets.kr(chain,0.1,\power);
  SendTrig.kr(onsets);
  SendPeakRMS.kr(sig, 20, 3, "/replyAddress");
}).add;
#Run the crazy synth above
Synth(\pulse)

#Forward the data on as an OSC message
#to emacs
~host = NetAddr("localhost", 4859);
p = OSCFunc({ |msg|
  ~host.sendMsg("/peakpower",msg[3], msg[4]);
  "peak: %, rms: %".format(msg[3], msg[4]).postln
}, '/replyAddress');

And in our emacs OSC server:

1
2
3
4
5
6
7
8
9

((string-match "/peakpower" path)
  (progn
    (with-current-buffer "flatiron.clj"
      (let ((sig (round (* 100.0 (first args)))))
        (message (format "%f" (first args)))
        (dotimes (n sig)
          (insert "▓"))
        (insert "▒░"))
      (insert "\n"))))

Repl Electric Emacs animations

All my Emacs animations are used to conclude the performance. Heres lies the source code, some screenshots and tricks & tips that made the animations possible.

Here’s a demo of all the Repl Electric animations discussed in action:

End of Buffer

https://github.com/repl-electric/view-pane/blob/master/animations/end-of-buffer.el

In this animations the text gets slowly broken up with white spaces and then like the wind, blows the characters away. Sometimes words are ripped apart as they blow in the wind (if we get lucky).

Two main phases:

• Injection of spaces. This starts to distort the text while keeping it readable. It provides a way to increase the effect of expanding whitespace in the next stage.

• Transforming whitespace into lots of whitespace. A Regex matches whitespace and replaces it with a randomly increasing amount of whitespace. Which leads to the effect of the characters of the code blowing away. I spent a while trying to improve the speed of this phase and Regexs proved to be the fastest way.

If we move the text fast enough soft word wrapping means the text appears to re-enter from the left side of the screen and eventually disappear out of the buffer. Without soft wrapping we get a horrible jitter as emacs moves back and forth between left and right side of the buffer.

A couple of other tricks/tactics used:

• Continually incrementing integer. Useful for injecting movement or using sin/cos fn with a continuous value.
• Perserving the syntax highlighting of the original code in an attempt to maintain some of the meaning of the code.

The Stars

https://github.com/repl-electric/view-pane/blob/master/animations/the-stars.el

This was my first animation and was based heavily on zone-pgm-drip-fretfully.

It randomly picks a single character and moves it down the screen until it hits another letter or exits the screen.

When running Emacs + Overtone + OpenGL, Emacs starts to slow down so part of the challenge was ensuring the animation ran as fast as possible.

A nice property of this is that as the OpenGL shaders shutdown, the speed of the animation increases and the code destroys itself more quickly.

Waves

https://github.com/repl-electric/view-pane/blob/master/animations/waves.el

This animations attempts to simulate the effect of waves using line wrapping and mixing deletions with insertions of different sizes to create lines that seem to move at different speeds.

Breaking Tools

While it may seem silly to bend Emacs to do things it was never intended to do, it’s an important part of discovering for yourself how you want your tools to work. Not just doing what you are expected but breaking them apart and discovering for yourself how you want to use them.

I’ve been working over the last year in the data team at SoundCloud building a realtime data pipeline using Clojure and Amazon’s Kinesis. Kinesis is Amazons equivalent to Kafka, “Real-Time data processing on the Cloud”. This is a summary of what was built, some lessons learnt and all the details in-between.

Tapping Real traffic

The first step was to tee the traffic from a live system to a test system without comprising its function. The main function of the live system is logging JSON events to file (which eventually end up somewhere like HDFS). Tailing the logs of the live system gives us access to the raw data we want to forward on to our test system. A little Go watches the logs, parses out the data and then forwards them in batch to test instances that will push to kinesis. Hence we had live data flowing through the system and after launch a test setup to experiment with. Sean Braithwaite was the mastermind behind this little bit of magic.

Sending to Kinesis

All kinesis sending happens in an application called the EventGateway (also written in Clojure). This endpoint is one of the heaviestly loaded services in SoundCloud (at points it has more traffic than the rest of SoundCloud combined). The Eventgateway does a couple of things but at its core it validates and broadcasts JSON messages. Hence this is where our Kinesis client slots in.

Squeezing Clojure Reflection

Its worth mentioning that in order for the Eventgateway service to be performant we had to remove all reflection in tight loops through type hints. It simply could not keep up without this. It became a common pattern to turn reflection warnings on while working in Clojure.

Project.clj

1

:profiles {:dev {:global-vars {*warn-on-reflection* true *assert* false}}}

Kinesis

The Eventgateway posts to Kinesis in batch using a ConcurrentLinkedQueue and separate producers and consumers. Messages are pushed into a ConcurrentLinkedQueue. We rolled our own Clojure kinesis client using Amazons Java library rather than using Amazonica.

1
2
3

;; Java Amazon libraries used
[com.amazonaws/aws-java-sdk "1.9.33"         :exclusions [joda-time]]
[com.amazonaws/amazon-kinesis-client "1.1.0" :exclusions [joda-time]]

Amazonica was good to get started quickly in the initial phase but there are a couple of reasons we switched to our own unique snowflake (which still looked a little like Amazonica):

• Amazonica did not support batch mode for Kinesis. Under initial tests it was impossible to scale this without batch.
• Injecting our own telemetry at low levels to learn more about Kinesis running.
• Some of its sensible defaults where not so sensible (for example default encoding the data using nippy).
• Ultimately most of any Kinesis client/server is configuration and tuning.
• Amazonica’s source is hard to read with a little too much alter-var-root going on.

1
2
3
4
5
6

;;Ugh. Its not just me right?
(alter-var-root
  #'amazonica.aws.kinesis/get-shard-iterator
  (fn [f]
    (fn [& args]
      (:shard-iterator (apply f args)))))

Pushing Messages in a Queue

Very simple, just adding a message to the ConcurrentLinkedQueue. A environment variable allows us to gradually scale up or down the percentage of traffic that is added to the queue.

1
2
3
4
5
6
7
8
9
10
11
12

(require '[environ.core :refer :all])

(def kinesis-message-queue (ConcurrentLinkedQueue.))
(def hard-limit-queue-size 1000)
(def queue-size (atom 0))

(defn send [message]
  (when-let [threshold (env :kinesis-traffic)]
    (when (> (rand-int 100) (- 100 (or (Integer/valueOf ^String threshold) 0)))
    (when (<= @queue-size hard-limit-queue-size)
      (.add kinesis-message-queue message)
      (swap! queue-size inc)))))

Failure

The queue pusher operates within a wider system and any failures due to Amazon being unreachable should not impede the function of the system. For the client this means:

• Not exceeding memory limits with a hard queue size (since ConcurrentLinkedQueue is unbound in size).
• Backing off workers if the queue is full to prevent cpu throttling.

When we cannot send messages to kinesis we instead log them to disk, and into our normal logging pipeline (usually ending up in HDFS). Hence we coule replay at a later date if required.

Sending batches to Kinesis

The workers, operating in separate threads consuming messages from the ConcurrentLinkedQueue collecting them into a batch:

1
2
3
4
5
6
7
8
9
10
11
12

(loop [batch []
       batch-start-time (time/now)
       poll-misses 0]
  (if (batch-ready-to-send? batch batch-start-time)
    batch
    (if-let [event (.poll kinesis-message-queue)]
      (do
        (swap! queue-size dec)
        (recur (conj batch event) batch-start-time 0))
      (do
        (Thread/sleep (exponential-backoff-sleep poll-misses))
        (recur batch batch-start-time (inc poll-misses))))))))

When polling from the queue an exponential back-off if no messages are on the queue.

1
2
3
4
5
6
7
8
9
10

(defn exponential-backoff-sleep
  "Exponential backoff with jitter and a max "
  [misses]
  (let [max-timeout 1000
        jitter-order 10]
    (Math/min
     max-timeout
     (Math/round (+ (Math/exp misses)
                    (* (Math/random)
                       jitter-order))))))

Once the batch is ready (in terms of age or size) its sent to Kinesis.

1
2
3
4
5
6
7
8
9
10
11

(defn- send-events
  "Perform putRecords request to send the batch to kinesis
   returns a list of events that failed."
  [^AmazonKinesisClient client stream-name events]
  (try+
   (let [result (.putRecords client (events->put-records-request events stream-name telemetry))]
     (if (pos? (.getFailedRecordCount result))
       (let [failed-events (failures->events result events)]
         (count-failures telemetry (failures->error-codes result))
         failed-events)
       []))

Note this is where we also decided the partition key. In our case its important for the same user to be located on the same partition. For example when consuming from Kinesis a worker is allocated a partition to work from and would miss events if they where across multiple partitions.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

(defn- events->put-records-request
  "Take client and a vector of JsonNodes and produce a PutRecord"
  [batch event-stream]
  (let [batch-list  (java.util.ArrayList.)
        put-request (PutRecordsRequest.)]
    (.setStreamName put-request event-stream)
    (doseq [^ObjectNode event batch]
      (.remove event ^String failure-metadata)
      (let [request-entry (PutRecordsRequestEntry.)
            request-data  (.getBytes (str event))
            request-buf   (ByteBuffer/wrap request-data 0 (alength request-data))
            partition-key (:user-id event)]
        (doto request-entry
          (.setData         request-buf)
          (.setPartitionKey partition-key))
        (.add batch-list request-entry)))
    (.setRecords put-request batch-list)
    put-request))

Failure

Failure can occur on individual records within a batch or in the batch as a whole.

Individual failures

1. These messages are re-added to the queue so we can try again. If the messages fail for some nth time they are considered invalid and rejected from kinesis and logged as an error.

Batch level

1. Amazon had an Internal Failure. We don’t know what went wrong. (We see this regularly in normal function).

2. Amazon Kinesis is not resolvable (AmazonClientException/AmazonServiceException).

3. Exceeding the read/write limits of Kinesis (ProvisionedThroughputExceededException).

This is our backpressure signal, in which case at worst we need to log to disk for replay later

Consuming Messages from Kinesis

With the consumption of events we have a different application stream for every worker. All workers have their own streams, and own checkpoints so they operate independently of each other. Some example of the workers we gave running:

• Logging Events to s3
• Calculating listening time
• Forwarding certain messages on to various other systems (like RabbitMQ).

Launching a worker is pretty simple with the Amazon Java Kinesis library.

1
2
3
4
5
6
7
8

(:import [com.amazonaws.services.kinesis.clientlibrary.lib.worker Worker])

(defn -main [& args]
  (let [worker-fn (fn [events] (print events))
        config (KinesisClientLibConfiguration.   worker-fn  )   ;;I'm airbrushing over the Java classes
        processor (reify IRecordProcessorFactory worker-fn)   ;;Ultimately this is a lot of config wrapped in Java fun
        [^Worker worker uuid] (Worker. processor config)]
          (future (.run worker))))

One of the hardest parts of setting up the a worker is getting the configuration right to ensure that the consumers are getting through the events fast enough. Events are held in Amazon for 24 hours after entry, and hence there is a minimum consumption rate.

Counting events in and events out with Prometheus made it easier to get the correct consumption rates.

Via the Amazon console you also get access to various graphs around read/write rates and limits:

Finally you can also look at Amazon’s Dynamodb instance for the Kinesis stream providing insight into metrics around leases, how many where revoked, stolen, never finished, etc.

Here is an example of one of our Kinesis workers configuration covered in scribblings of me trying to work out the right settings.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

  {
   ;;default 1 sec, cannot be lower than 200ms
   ;;If we are not reading fast enough this is a good value to tweak
   :idle-time-between-reads-in-millis 500

   ;;Clean up leases for shards that we've finished processing (don't wait
   ;;until they expire)
   :cleanup-leases-upon-shard-completion true

   ;;If the heartbeat count does not increase within the configurable timeout period,
   ;;other workers take over processing of that shard.
   ;;*IMPORTANT* If this time is shorter than time for a worker to checkpoint all nodes
   ;;will keep stealing each others leases producing a lot of contention.
   :failover-time-millis ...

   ;;Max records in a single returned in a `GetRecords`. Cannot exceed 10,000
   :max-records 4500

   ;;Process records even if GetRecords returned an empty record list.
   :call-process-records-even-for-empty-record-list false

   ;;Sleep for this duration if the parent shards have not completed processing,
   ;;or we encounter an exception.
   :parent-shard-poll-interval-millis 10000

   ;;By default, the KCL begins withs the most recently added record.
   ;;Instead always reads data from the beginning of the stream.
   :initial-position-in-stream  :TRIM_HORIZON}

Monitoring

Prometheus (http://prometheus.io/) a monitoring tool built at SoundCloud was core to developing, scaling and monitoring all of this pipeline. Amazon does provide some useful graphs within the AWS console but more detailed feedback was very helpful even if it was removed later.

Exception Logging pattern

All Exceptions are counted and sent to log. This was a very useful pattern for driving out errors and spotting leaks in the interactions with Kinesis and consumption:

(Using a Clojure wrapper around Prometheus: https://github.com/josephwilk/prometheus-clj)

1
2
3
4
5
6

(try+
  (worker-fn raw-events)
(catch Exception e
  ;;Count based on exception class
  (inc-counter :failed-batches {:type worker-type :error-code (str (.getClass e))})
  (log/error e)))]

Note Kinesis regularly spits out “InternalFailure” Exceptions. Thats all you get…

A Cloud Pipeline in Pictures

In my previous post about Building Clojure services at scale I converted the system metrics to sound. With so many machines processing so many events its easy to loose track of the amount of work being done in the cloud. To make this feel more real I captured metrics across all the machines involved and created 3d renderings using OpenFrameworks and meshes of the systems function:

Thanks

This work constitues a team effort by the Data team at SoundCloud. A lot of advice, collaboration and hard work. Kudos to everyone.

Using Clojure we create interesting 3D shapes in Minecraft to the beat of music generated from Overtone.

We achieve this by embedding a Clojure REPL inside a Java Minecraft server which loads Overtone and connects to an external Supercollider instance (What Overtone uses for sound).

Demo

Demo Source code: https://github.com/josephwilk/clj-minecraft/blob/master/src/cljminecraft/overtone.clj

Tools

• Java Minecraft server Craftbukkit/Spigot: http://www.spigotmc.org/
• Minecraft client (requires purchase) https://minecraft.net
• clj-minecraft Bukkit plugin: https://github.com/josephwilk/clj-minecraft
• Overtone (patched): https://github.com/josephwilk/overtone/tree/minecraft_overtone
• MUD (useful helpers for Overtone): https://github.com/josephwilk/mud
• Supercollider: http://supercollider.sourceforge.net

(Dependent on your IDE of choice)

• Emacs – cider: https://github.com/clojure-emacs/cider
• Vim – fireplace: https://github.com/tpope/vim-fireplace

Building the world

We need to install Spigot which is an optimized version of the Craftbukkit Java Minecraft server and install clj-minecraft project as a plugin. Things are complicated by Bukkit no longer being registered in Maven.

Read through the Makefile install:

https://github.com/josephwilk/clj-minecraft/blob/master/Makefile

If your happy clone and run:

1

git clone git@github.com:josephwilk/clj-minecraft.git && make install

Bukkit + clj-minecraft will be installed for you.

Running the world

You will need to edit the minecraft/eula.txt indicating you agree with the license. Then you can run your Minecraft server:

1

java -XX:MaxPermSize=1G -jar spigot-1.8.jar

Seeing the world

Buy and install a Minecraft client: https://minecraft.net

Select >“Multiplayer” >“Direct connect” and enter the “Server Address” as localhost.

Bring music to the world

Install and Boot Supercollider

http://supercollider.github.io

Once installed:

1
2
3
4
5
6
7

#Find the Supercollider binary (scsynth)
$ sudo find /. -name "scsynth"

/Applications/SuperCollider/SuperCollider.app/Contents/Resources/scsynth

#Run Supercollider server
/Applications/SuperCollider/SuperCollider.app/Contents/Resources/scsynth -u 57110

Speaking to the Minecraft REPL

clj-minecraft opens a REPL on localhost port 4005. Using emacs and cider connect to this REPL instance.

Boot and connect Overtone:

1
2
3

(use 'overtone.core)

(connect-external-server)  #=> :happy-hacking

Interaction

Using MUD we have some useful wrappers around Overtone for scheduling functions on beats.

To coordinate graphics and sound we schedule both within a single function.

1
2
3
4
5
6
7
8
9
10

(require '[mud.core :as mud])

(defonce highhat-sample (freesound 53532))

(def ride-trigger
  (mud/on-beat-trigger 8       ;; Every 8th beat
   (fn []
     (highhat-sample)      ;; Play sample
     (block 2 10 2 :grass) ;; Place a block into the Minecraft world
)))

Most functions that change the state of the Minecraft world need to be run in the main GUI thread. To achieve this we wrap any state changing function within ui-sync (https://github.com/CmdrDats/clj-minecraft/blob/a3331e925b56becf88d9ef96cab225856e2f7ead/src/cljminecraft/bukkit.clj#L39-L42).

For example a function to place a block into the Minecraft world:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

(import '[org.bukkit Location Material])
(require '[cljminecraft.bukkit :as bk])
(require '[cljminecraft.items :as i])

(def player (first (.getOnlinePlayers (bk/server))))

(defn block
  "place a block relative to current players position
  Example:
    (block 2 10 2 :grass)
  "
  [x y z material]
  (let [l (.getLocation player))
        m (i/get-material material)]
    (doto l
      (.setX (+ x (.getX l)))
      (.setY (+ y (.getY l)))
      (.setZ (+ z (.getZ l))))
    (bk/ui-sync
     @cljminecraft.core/clj-plugin
     (fn []
       (doto (.getBlock l)
         (.setData 0)
         (.setType (.getItemType m))
         (.setData (.getData m)))))))

For further documentation of whats possible, the Bukkit Java docs: https://hub.spigotmc.org/javadocs/bukkit/

clj-minecaft has lots of helpers + examples: https://github.com/CmdrDats/clj-minecraft/tree/master/src/cljminecraft

Fun

Create, play and share all the crazy things you can come up with using Clojure and Minecraft.

For more live programming music and sound checkout my performances as Repl Electric: http://www.repl-electric.com

Credits

Built on the back of lots of great open source projects. Thanks to the Craftbukkit/Spigot contributors, @CmdrDats for clj-minecraft and @samaaron for Overtone and inspiring this crazy journey with the musical Sonic Pi (which supports combining music and Minecraft on the RaspberryPi).

Live coding is the act of turning a programming session into a performance. This can constitute improvisation, music, visuals, poetry, hardware, robots, dance, textiles and people. Pretty much anything with an input and output can be controlled live by programming.

This is not just a performance by programmers for programmers. While this is often where it starts as a live coder, the type of audience and the accessibility of the performance lies in the performers imagination. Abstraction can get us pretty much anywhere.

1

(def the-stars (dark-matter))

Repl Electric

Repl Electric is a project I started in order to discover more about music composition and Artificial intelligent based aids to creativity. Which in turn through the inspiration of people like Meta-ex lead me to live programming music.

Here is a performance live coding music and graphics, inspired by a performance in London:

The Stars

Open Live Coding

All the tools and code used to create this performance are open for all to see on Github: https://github.com/repl-electric

Three programming languages were used to create this piece:

• Clojure (Sound)
• GLSL (Visuals)
• Emacs Lisp (Animations & Navigation)

Tools

Here are the tools used and a little detail around how they where used in performing “The Stars”:

Clojure: http://clojure.org

Clojure is a LISP language based on the JVM.

Clojure focuses on interactive REPL (Read, Evaluate, Print & Loop) driven development. Which makes it a good choice for interactively coding music. It also turns out functional programming is a good fit for operating over music as data.

Emacs Live: https://github.com/overtone/emacs-live

Emacs live is a Emacs release with packages and defaults that are Live Coding centric. Something I use for both for my work and for my live coding.

To execute our code, we launch a repl instance in our project (NEVER launch inside emacs, since then if emacs crashes the repl and music dies) and connect to it from emacs using cider https://github.com/clojure-emacs/cider.

A simple trick to combine Emacs code and visualizations is to launch an OpenGL window in full screen (see Shadertone) and then put a full screen transparent terminal window running emacs over it.

The tumbling text effect seen at the end of the performance is an emacs animation using Zone Mode which supports writing your own text destructors: http://www.emacswiki.org/emacs/ZoneMode

Overtone: https://github.com/overtone/overtone

Overtone is a Clojure based client to SuperCollider. Supercollider is an environment for real time audio synthesis and algorithmic composition.

Overtone provides us with:

• Timing (beat generation – example timing code).
• Building Synths (engineer sound).
• Running samples (both your own and from Freesound).
• Live Synth control (changing notes, durations, reverb, etc).
• Hardware interaction (through midi or OSC).

An example of a synth used in The Stars:

1
2
3
4
5
6
7
8
9
10

(use 'overtone.live)
(defsynth dark-ambience [out-bus 0 amp 1 mul 0.2 room-size 70 rev-time 99 freq 60 ring-mul 55]
  (let [pink (hpf:ar (* (* 0.005 (pink-noise)) (line:kr 0 1 9)) 5)
        src1 (ringz (* pink (lf-noise1:kr 0.15)) (+ freq (* ring-mul 0)) mul)
        src2 (ringz (* pink (lf-noise1:kr 0.15)) (+ freq (* ring-mul 1)) mul)
        src3 (ringz (* pink (lf-noise1:kr 0.15)) (+ freq (* ring-mul 2)) mul)
        src (tanh (g-verb (sum [src1 src2 src3]) room-size rev-time))]
    (out out-bus (* amp src))))

(def the-stars (dark-ambience))

Timing

Timing is a complicated issue but so important its worth touching on. You have a choice with Overtone to use Java for timing or Supercollider. I use Supercollider since I have found it to be much more reliable. Everything you need is here (copy and paste), thanks to the hard work of Sam Aaron.

The key concept to take away is there are two types of timing, a beat counter which is forever incrementing and a beat trigger which flips back and forth between 1/0.

1
2
3
4
5
6
7

(require '[cassiopeia.engine.timing :as time])

;;The beat trigger
(:beat time/main-beat) ;=> 0,1,0,1,0,1,0

;;The beat counter
(:count time/main-beat) ;=> 0,1,2,3,4,5,6,7,8,9

The counter is useful for indexing buffers, the trigger is useful in controlling the gate of an envelope (which turns a sound on or off).

In Clojure we can still get access to the beat, in our timing code we send a message using send-trig on every beat. We can hook a Clojure function to callback on this beat:

1
2
3
4
5
6

(on-trigger (:trig-id time/main-beat)
  (fn [beat-no]
    (when (= 0.0 (mod beat-no 32))
      ;;play a sample
      (boom-s)))
  ::on-beat-trigger)

I use this extensively to time graphic transitions with the music.

Buffers

Most of my live coding performance was writing to buffers which are hooked into synths. Buffers are just fixed size arrays but they are stored in Supercollider rather than in Clojure. Here is an example from The Stars where the midi notes are read from a buffer at a rate based on my beat timing signal (a 16th of the main beat here).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

(use 'overtone.live)
(use 'cassiopeia.engine.core)
(require '[cassiopeia.engine.timing :as time])

(defsynth growl [out-bus 0 note-buf 0 beat-bus 0 beat-trg-bus 0 amp 1]
  (let [cnt (in:kr beat-bus)
        trg (in:kr beat-trg-bus)
        note (buf-rd:kr 1 note-buf cnt)
        freq (midicps note)
        vol (> note 0)

        e (env-gen (perc :attack 10 :sustain 1 :release 1) :gate trg)
        src (lpf (mix [(saw (* 0.25 freq))
                       (sin-osc (* 1.01 freq))]))
        src (pitch-shift src 0.4 1 0 0.01)
        src (pan2:ar (* vol amp e src))]
    (out out-bus src)))

(defonce nebular (buffer 96))

(def nebula (growl :note-buf nebula-note-buf :beat-trg-bus (:beat time/beat-16th) :beat-bus (:count time/beat-16th)))

(pattern! nebula-note-buf (degrees [1 3 7] :major :A2))

GLSL + Shadertone: https://github.com/overtone/shadertone

Shaders generate imagery directly on your Graphics Processing Unit rather than going through your CPU. Through a language called GLSL (which is C like) we can express very simple functions which get called on every single pixel generating complex visuals. Here is a simple extract from The Stars that generates all the background small dots:

1
2
3
4
5
6
7
8
9
10
11
12

void main(void){
  vec2 current_pixel_position = mod(gl_FragCoord.xy, vec2(5.0)) - vec2(0.0);
  float distance_squared = dot(current_pixel_position, current_pixel_position);

  vec4 black = vec4(.0, .0, .0, 0.0);
  vec4 white = vec4(1.0, 1.0, 1.0, 1.0);

  //Test the current pixel position and if it should be a circle shade it.
  vec4 circles = (distance_squared < 0.6) ? white : black;

  gl_FragColor = circles;
}

For more examples of whats possible with Shaders checkout Shader Toy

Shadertone is the Clojure library that provides a convenient way of running shaders from Clojure and for feeding in data about our synths. It provides access in your Shader to:

• Overtone’s Volume (iOvertoneVolume)
• The frequency spectrum & audio waveform data (Passed as a 2D texture :textures [:overtone-audio])

To synchronize the graphics with the music I created a special Overtone synth which does not generate any sound, it instead feeds information in realtime to my shader.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

(use 'overtone.live)
(require '[shadertone.core :as t])

;;A synth that exposes through taps all the lovely timing information.
(defsynth buffer->tap [beat-buf 0 beat-bus 0 beat-size 16 measure 6]
  (let [cnt (in:kr beat-bus)
        beat (buf-rd:kr 1 beat-buf cnt)
        _  (tap "beat"          60 (a2k beat))
        _  (tap "beat-count"    60 (a2k (mod cnt beat-size)))
        _  (tap "measure-count" 60 (a2k (/ (mod cnt (* measure beat-size)) measure)))])
  (out 0 0))

;;; Used to store our drum beat, 1 for a hit 0 and for a miss
(defonce drum-sequence-buffer (buffer 256))

(def beats (buffer->tap drum-sequence-buffer (:count timing/main-beat)))

;;Open a OpenGL window running our shader
(t/start-fullscreen "resources/shaders/electric.glsl"
                    :user-data {
                    "iBeat"         (atom {:synth beats :tap "beat"})
                    "iBeatCount"    (atom {:synth beats :tap "beat-count"})
                    "iMeasureCount" (atom {:synth beats :tap "measure-count"})})

Inside our shader code:

1
2
3
4

uniform float iBeat;
uniform float iBeatCount;
uniform float iMeasureCount;
...

The other main way of controlling a shader from Clojure is using atoms.

1
2
3
4
5
6
7

(require '[shadertone.core :as t])

(defonce cellular-growth (atom 0.0))

(t/start-fullscreen "resources/shaders/electric.glsl" :user-data {"iCellularGrowth" cellular-growth})

(swap! cellular-growth + 0.01)

Hardware: Monome: http://monome.org

Something you don’t see in the video is that I’m using a 8x16 Monome. For this performance its primary function was a visual aid to show the beat/measure information.

The hardware is driven by Clojure communicating with the Monome through a serial port: https://github.com/josephwilk/monome-serial/tree/protocols

Live Coding

Live coding music and graphics combines skills in sound engineering, 3d graphics, geometry, physics, musical theory, composition, improvisation & hardware to name a few.

It is difficult, and requires a lot of work and practice.

But of all the code I’ve written over the years this is one of the things I’m most proud of. And i’m only at the beginning of discovering what’s possible.

When Alan Turing asked if a machine can be intelligent one aspect of this question focused on “could machines be creative”?

Ada Lovelace seemed convinced that originality was not a feat a computer was capable of:

it can do whatever we know how to order it to perform, it has no pretensions whatever to originate anything

Before we outrightly dismiss the idea of creative machines do we even understand what creativity is?

Join me on a journey examining these questions while also meeting a new generation of artists born through code. Looking into their hearts and brains examining different algorithms/techniques and there effectiveness at exhibiting creativity.

Decide for yourself, can machines be creative?

My Strangeloop talk: Creative Machines – http://www.infoq.com/presentations/ai-machine-creativity

Slides:

Full source code: https://github.com/josephwilk/musical-creativity

Continuing the Journey

In order to continue my research into creativity and music I’ve started the project Repl Electric. A space for humans and machines to learn and create music together.

Further Reading

• The Creative Mind: Myths and Mechanisms
  http://www.amazon.com/The-Creative-Mind-Myths-Mechanisms/dp/0415314534
• Explaining Creativity: The Science of Human Innovation
  http://www.amazon.com/gp/product/0199737576
• Artificial Intelligence and Literary Creativity: Inside the Mind of Brutus, A Storytelling Machine
  http://www.amazon.com/Artificial-Intelligence-Literary-Creativity-Storytelling/dp/0805819878)
• Computer Models of Musical Creativity
  http://www.amazon.com/Computer-Models-Musical-Creativity-David/dp/0262033380
• Virtual Music: Computer Synthesis of Musical Style
  http://www.amazon.com/Virtual-Music-Computer-Synthesis-Musical/dp/0262532611/

What does your brain sound like? Does it sound like “Rise of the Valkyrie” or more like “Hit me baby one more time”?

Step 1: Acquire a human brain (alive)

We are interested in capturing the brain waves. Specifically the:

• Delta waves: Deepest stages of sleep.
• Beta waves: Normal waking consciousness.
• Alpha waves: Relaxation and meditation (creativity).
• Theta waves: REM sleep (dreams).
• Gamma waves: Hyper-alertness, perception, and integration of sensory input.

Step 2: EEG machine

I am using a EEG machine brought from Neurosky which is rated as Research grade (whatever that means). This measures voltage fluctuations resulting from ionic current flows within the neurons of the brain. While EEG machines are not the most accurate they are now reasonably cheap.

Step 3: EEG –> Overtone

In order to generate music I want to import the EEG brainwave data into Overtone.

We interact with the EEG machine over a serial port. The most mature library for this interface is in Python so there is a little jiggery pokery to get the data into Overtone.

The Brainwave Poller

We use https://github.com/akloster/python-mindwave to interface with the EEG machines data.

Writing all the data out to a FIFO file file as json.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

import re
import time
import json
import unicodedata

import gevent
from gevent import monkey

from pymindwave import headset
from pymindwave.pyeeg import bin_power

monkey.patch_all()

# connect to the headset
hs = None
hs = headset.Headset('/dev/tty.MindWave')
hs.disconnect()
time.sleep(1)
print 'connecting to headset...'
hs.connect()
time.sleep(1)
while hs.get('state') != 'connected':
    print hs.get('state')
    time.sleep(0.5)
    if hs.get('state') == 'standby':
        hs.connect()
        print 'retrying connecting to headset'

def raw_to_spectrum(rawdata):
    flen = 50
    spectrum, relative_spectrum = bin_power(rawdata, range(flen), 512)
    return spectrum

while True:
    t = time.time()
    waves_vector = hs.get('waves_vector')
    meditation = hs.get('meditation')
    attention = hs.get('attention')
    spectrum = raw_to_spectrum(hs.get('rawdata')).tolist()

    with open("/tmp/brain-data","w") as fp:
        s = {'timestamp': t,
             'meditation': meditation,
             'attention': attention,
             'raw_spectrum': spectrum,
             'delta_waves': waves_vector[0],
             'theta_waves': waves_vector[1],
             'alpha_waves': (waves_vector[2]+waves_vector[3])/2,
             'low_alpha_waves': waves_vector[2],
             'high_alpha_waves': waves_vector[3],
             'beta_waves': (waves_vector[4]+waves_vector[5])/2,
             'low_beta_waves': waves_vector[4],
             'high_beta_waves': waves_vector[5],
             'gamma_waves': (waves_vector[6]+waves_vector[7])/2,
             'low_gamma_waves': waves_vector[6],
             'mid_gamma_waves': waves_vector[7]}

        s = json.dumps(s)
        fp.write(s)
    gevent.sleep(0.4)

Reading from the FIFO file is simple in Clojure.

1
2
3

(while true
  (with-open [reader (clojure.java.io/reader "/tmp/brain-data")]
    (brainwave->music (json/decode (first (line-seq reader)) true))))

Step 3: Sonification

Sonification is the process of taking data and turning it into sound. Here is an example of the data we are now receiving:

A single JSON brainwave packet:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

 {"gamma_waves": 95408,
  "high_beta_waves": 205681,
  "beta_waves": 293928,
  "low_beta_waves": 382176,
  "mid_gamma_waves": 84528,
  "low_alpha_waves": 172417,
  "delta_waves": 117933,
  "low_gamma_waves": 106288,
  "alpha_waves": 112605,
  "theta_waves": 635628,
  "high_alpha_waves": 52793,
  "attention": 0,
  "meditation": 0,
  "timestamp": 1.375811275696894E9}

We will focus on the beta-waves for simplicity. Beta-waves fall between 16.5–20Hz.

Beta waves related to:

• Alertness
• Logic
• Critical reasoning

We need to map a signal within 16.5-20Hz into the musical pitches of a sampled piano (21-108 pitches).

1
2
3
4
5
6
7
8
9
10
11
12
13
14

(require '[clojure.math.numeric-tower :as math])

(defn linear-map [x0 x1 y0 y1 x]
  (let [dydx (/ (- y1 y0) (- x1 x0))
        dx (- x x0)]
    (+ y0 (* dydx dx))))

;; Piano range: 0..87
;; Beta wave range: 16500..20000

(defn beta-wave->pitch [signal] (-> (linear-map 16500 20000 21 108 signal) float math/round))

(beta-wave->pitch 16500) ;-> 21
(beta-wave->pitch 20000) ;-> 108

Now we extract the beta-waves from the brainwave data and play them. We play them live as they arrive rather than worrying about scheduling notes:

1
2
3
4
5
6
7
8
9

(use 'overtone.live)
(use 'overtone.inst.sampled-piano)
(require '[cheshire.core :as json])

(while true
  (with-open [reader (clojure.java.io/reader "/tmp/brain-data")]
    (let [beta-wave (-> (first (line-seq reader)) (json/decode true) :beta_waves)]
      (println beta-wave)
      (sampled-piano :note (beta-wave->pitch beta-wave) :sustain 0.2))))

Would you like to hear my brain?

The results, please listen to my brain.

Not really music is it? With beta-waves we get a serious of high to low transitions. While we can control at what pitch the transitions occur by performing activities that shape our brain waves the transitions don’t provide the order or structure we need to recognize this as music.

Brain controlled Dubstep

The only logical path left is to try and control Dubstep with our brain. Rather than generative music we can use our brain waves to control the tempo and volume of existing synthesized music.

Here is a Dubstep synth taken from Overtone:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

(use 'overtone.live)

(defsynth dubstep [bpm 120 wobble 1 note 50 snare-vol 1 kick-vol 1 volume 1 out-bus 0]
  (let [trig (impulse:kr (/ bpm 120))
        freq (midicps note)
        swr (demand trig 0 (dseq [wobble] INF))
        sweep (lin-exp (lf-tri swr) -1 1 40 3000)
        wob (apply + (saw (* freq [0.99 1.01])))
        wob (lpf wob sweep)
        wob (* 0.8 (normalizer wob))
        wob (+ wob (bpf wob 1500 2))
        wob (+ wob (* 0.2 (g-verb wob 9 0.7 0.7)))

        kickenv (decay (t2a (demand (impulse:kr (/ bpm 30)) 0 (dseq [1 0 0 0 0 0 1 0 1 0 0 1 0 0 0 0] INF))) 0.7)
        kick (* (* kickenv 7) (sin-osc (+ 40 (* kickenv kickenv kickenv 200))))
        kick (clip2 kick 1)

        snare (* 3 (pink-noise) (apply + (* (decay (impulse (/ bpm 240) 0.5) [0.4 2]) [1 0.05])))
        snare (+ snare (bpf (* 4 snare) 2000))
        snare (clip2 snare 1)]

       (out out-bus (* volume (clip2 (+ wob (* kick-vol kick) (* snare-vol snare)) 1)))))

Once the synth is running we can send it control signals which will vary any of the properties defined in the arguments to the dubstep function:

• bpm
• wobble
• note
• snare-vol
• kick-vol
• volume

1
2
3
4
5
6
7

(def d (dubstep))

(ctl d :snare-vol 0)
(ctl d :kick-vol 0)
(ctl d :wooble 0)
(ctl d :bpm 20)
(ctl d :v 0.2)