Nurullah Akkaya: an explorer's log

Visualizing Maps Using Incanter

When I first saw Mathematica's WorldPlot function I was impressed, its a nice way to visualize various forms of geographical data, for some time I thought this should be a very labor intensive task, who would go around labeling each pixel. Couple of days ago I somehow ended up reading the Processing article on Wikipedia, it contains an example which shows a map of the results of the 2008 USA presidential election, turns out using Scalable Vector Graphics implementing WorldPlot functionality is extremely easy.

We will be plotting how population moved between different regions in Turkey (in to and out of a region). I am using data provided by The Turkish Statistical Institute, you can grab the map I used from Wikipedia here.

 ;; Data for 2009
 (def pop-taken [{:id 1 :name "Marmara" :population 582771}
                 {:id 2 :name "Iç Anadolu" :population 297919}
                 {:id 3 :name "Ege" :population 164896}
                 {:id 4 :name "Akdeniz" :population 188441}
                 {:id 5 :name "Karadeniz" :population 256654}
                 {:id 6 :name "Güneydoğu Anadolu" :population 171910}
                 {:id 7 :name "Doğu Anadolu" :population 214082}])

 (def pop-given [{:id 1 :name "Marmara" :population 677395}
                 {:id 2 :name "Iç Anadolu" :population 310293}
                 {:id 3 :name "Ege" :population 181459}
                 {:id 4 :name "Akdeniz" :population 193231}
                 {:id 5 :name "Karadeniz" :population 247397}
                 {:id 6 :name "Güneydoğu Anadolu" :population 118611}
                 {:id 7 :name "Doğu Anadolu" :population 148287}])

Turkey is divided in to seven geographical regions, pop-taken represents how many people moved in to that particular region and pop-given represents how many people moved out of that region during 2009.

 (defn region-color [val min max]
   (lerp-color (color 0xffd120) (color 0x920903) (norm val min max)))

In order to paint the map like a heat map, we need to assign colors using the amount of people moved in or out of a region, given a min, max and a value in between norm will normalize a value to exist between 0 and 1, lerp-color on the other hand will calculate a color between the given range using the normalized value. So our map will go from yellow to dark red depending on the people moved.

 (defn map-region-color [regions]
   (let [min (apply min (map #(:population %) regions))
         max (apply max (map #(:population %) regions))]
     (map #(vector (:id %) (region-color (:population %) min max)) regions)))

Now all we need to do is calculate min and max values in the data set, iterate over the data set and return a sequence of [id color] pairs.

 (defn sktch [regions]
   (sketch
    (setup [])
    (draw 
     []
     (let [tr-map (load-shape this "MapTurkishProvincesNumbers.svg")]
       (.shape this tr-map 0 0)
       (doseq [region (map-region-color regions)]
         (let [[id color] region
               child (.getChild tr-map (str id))]
           (.disableStyle child)
           (.fill this color)
           (.noStroke this)
           (.shape this child 0 0)
           no-loop))))))

Using incanter-processing library, we can load and access parts of the SVG map. Processing sketches are made up of the functions setup and draw, in setup as its name suggests you setup your stuff frame rate, stroke properties etc. Draw will be called once or multiple times depending on your frame rate, we load the map as a shape then paint it on the canvas, then we iterate over the data set, using getChild method of the PShape class we can access parts of the image, the map we are using has 7 children named 1 through 7 corresponding to the geographical regions of the country, we get the child then paint it using the color we calculated on to the canvas. One thing to note, sketch macro just returns a PApplet so for any function not implemented in incanter, you can access them just like any other Java function. Now lets see the results,

(view (sktch pop-given) :size [1052 744])

(view (sktch pop-taken) :size [1052 744])

WebDAV + SSL on Debian

I was looking for a way to easily share documents between machines, since WebDAV shares can be accessed by Windows, Linux or Mac machines out of the box, I choose WebDAV over SSL. I don't use SSL for anything so WebDAV is served from DocumentRoot. I've been using it for a few days, so far it beats carrying USB sticks around.

Enable relevant Apache modules,

a2enmod ssl
a2enmod dav_fs
a2enmod dav

Create SSL certificate,

 mkdir /etc/apache2/ssl
 openssl req $@ -new -x509 -days 365 -nodes -out /etc/apache2/ssl/apache.pem \
     -keyout /etc/apache2/ssl/apache.pem
 chmod 600 /etc/apache2/ssl/apache.pem

Create your WebDAV directory and create a password file,

mkdir /path/to/webdav/
chown www-data /path/to/webdav/
htpasswd -c /path/to/passwd.dav user

Edit and add the following snippet to the configuration for the host you want to enable WebDAV,

 <VirtualHost *:443>
         ServerAdmin user@host.com
         DocumentRoot /path/to/webdav

         SSLEngine on
         SSLCertificateFile /etc/apache2/ssl/apache.pem

         <Directory /path/to/webdav/>
            DAV On
            AuthType Basic
            AuthName "webdav"
            AuthUserFile /path/to/passwd.dav
            Require valid-user
        </Directory>

         ErrorLog  /path/to/webdav/error.log
         CustomLog /path/to/webdav/access.log combined
 </VirtualHost>

Reload Apache configuration,

/etc/init.d/apache2 reload

Analytics with Incanter

These past few days I've been playing with Incanter, which is a Clojure-based, R-like platform for statistical computing and graphics. This post covers the basic steps of using Clojure to access your Google Analytics Data with the Google Analytics Data Export API and visualize, filter the data returned using Incanter.

Google provides a Java library to simplify use of any Google Data API with Java, to access Analytics you need to grab the following list of Jars from gdata-java-client and google-collections.

gdata-client-1.0.jar
gdata-client-meta-1.0.jar
gdata-core-1.0.jar
gdata-analytics-2.1.jar
gdata-analytics-meta-2.1.jar
google-collect-1.0.jar

After running "lein deps" add them to the lib/ subdirectory.

 (defn service [username pass]
   (doto (AnalyticsService. "Clojure_Incanter_Sample")
     (.setUserCredentials username pass)))

In order to retrieve data we need a service object which handles all interaction between our application and Analytics Data Export API.

 (defn account-feed [service & args]
   (let [url (URL. (str "https://www.google.com/analytics/"
                        "feeds/accounts/default?max-results=50"))
         feed (.getFeed service url (get-class "AccountFeed"))
         accs (reduce #(assoc %1 
                         (-> %2 .getTitle .getPlainText)
                         (-> %2 .getTableId .getValue)) 
                      {} (.getEntries feed))]
     (if (nil? args) accs (accs (first args)))))

To retrieve data for a profile, we need its table id. Asking service for an account feed, returns a list of entries containing title, table id and profile id but we are only interested in title and table id.

 (defn data-feed [service & args]
   (let [args (apply hash-map args)
         feed (.getFeed service (.getUrl (query args)) (get-class "DataFeed"))
         cols (map #(str "ga" %) (concat (:dimensions args) (:metrics args)))]
     (map (fn [e]
            (map #(.stringValueOf e %) cols))
          (.getEntries feed))))

As with the account feed, first thing we need to do is build a feed request URL, query function handles that nothing fancy, it just calls a bunch of setters for dimensions, metrics etc. Querying analytics service with a data feed URL returns a list of entries, data-feed maps over them and returns a sequence containing dimensions and metrics we requested.

Now that we have some data to play with, we can start off by doing fairly standard things, like which pages got the most visits for the past month,

   (def analytics (service "username" "password"))
   (def acc-nakkaya (account-feed analytics "nakkaya.com"))

   (def pageview (data-feed analytics 
                            :date ["2010-01-26" "2010-02-25"]
                            :dimensions [:pageTitle :pagePath]
                            :metrics [:pageviews]
                            :sort [:pageviews]
                            :num-result 10
                            :id acc-nakkaya))

This is where incanter makes things fun, as long as you have a sequence of rows, in this case what data-feed returns you can call view to visualize the data,

(view pageview)

or we can filter the data leaving only portions of it which we are interested, such as pages with views more than 200 and lower than 800,

   (with-data (col-names (map (fn [[x y z]] [x y (BigInteger. z)]) pageview)
                         [:title :path :views])
     (view ($where {:views {:$gt 200 :$lt 800}})))

Alternatively you can filter the data in Clojure, requesting top 10 keywords people used to find your website and filtering the ones that contain "clojure" or "java" in them,

   (def keywords (data-feed analytics 
                            :date ["2010-01-26" "2010-02-25"]
                            :dimensions [:keyword]
                            :metrics [:visits]
                            :sort [:visits]
                            :num-result 10
                            :id acc-nakkaya))

    (let [words ["clojure" "java"]] 
      (reduce (fn[h v]
                (if (some true? (map #(.contains (first v) %) words))
                  (conj h v) h)) [] keywords))

 analytics.core=> [("clojure xml" "62") ("clojure turtle graphics" "31")
                   ("clojure opencv" "26") ("detect faces from webcam+java" "26")]

Besides visualizing stuff using tables, we can plot graphs containing the information we are intrested,

   (def browsers (data-feed analytics 
                            :date ["2010-01-26"]
                            :dimensions [:browser]
                            :metrics [:visits]
                            :sort [:visits]
                            :num-result 10
                            :id acc-nakkaya))

   (view (bar-chart (take 4 (map first browsers))
                    (take 4 (map #(BigInteger. (last %)) browsers))
                    :title "Browser/Visits"
                    :x-label "Browsers"
                    :y-label "Visits"))

   (def view-date (data-feed analytics 
                             :date ["2009-11-26" "2010-02-25"]
                             :dimensions [:date]
                             :metrics [:visitors]
                             :sort [:visitors]
                             :num-result 10
                             :id acc-nakkaya))

   (view (line-chart (map first view-date)
                     (map #(BigInteger. (last %)) view-date)
                     :title "Visits"
                     :x-label "Date"
                     :y-label "Visits"))

analytics.clj

Writing Leiningen Plugins 101

I'm trying to switch, building my projects using Ant to leiningen. Almost all of them requires customs tasks such as building native executables, move files around etc. Which requires I have to come up with a lein plugin for each ant task, unfortunately not much documentation exists about writing lein plugins, this post collects bits and pieces of information I gathered over the web.

To begin with lein tasks are functions named "your-task" defined in the namespace "leiningen.your-task". They take a project argument containing information defined in defproject and command-line arguments. For simple tasks or quickly testing something, you can simply define them in project.clj after the defproject definition,

 (ns leiningen.foo)
 (defn foo [project & args] (println "Hello Foo!!"))

Now lein should have a new task named foo, running it should print "Hello Foo!!". Of course for longer tasks, you are not going to want it cluttering your project.clj, they can be placed under leiningen/ folder not src/leiningen/. Since tasks are just functions, making a task depend on another task is as easy as calling depencies on top of the function.

 ;;leiningen/bar.clj
 (ns leiningen.bar)

 (defn bar [projects & args] 
   (apply leiningen.foo/foo projects args)
   (println "Hello Bar!!"))

Now running bar task should give you, "Hello Foo!!" and "Hello Bar!!". For sharing plugins across projects create a separate lein project for the plugin, after creating a Jar with "lein jar" you have two options, you can either push it to clojars and add your plugin as a dev-dependency for your project or just move the jar to the lib folder of the project, either way lein will pick it up.

Applescript with Clojure

Today I was experimenting with AppleScript engine thats included with Java 1.6. It makes it much easier to interact with OS X then executing osascript using Runtime.

 (let [mngr (javax.script.ScriptEngineManager.)
       engine (.getEngineByName mngr "AppleScript")] 
   (.eval engine "say \"Hello World!\""))

For simple stuff, it is as simple as asking engine manager for the AppleScript engine and running eval on it with your script.

 (def tracks (str "tell application \"iTunes\"\n"
                  "get count of tracks of (get view of front window)\n"
                  "end tell\n"))

 (let [mngr (javax.script.ScriptEngineManager.)
       engine (.getEngineByName mngr "AppleScript")] 
   (.eval engine tracks))

user=> 5229

When eval returns, numbers, strings, and dates are coerced into java.lang.Double, java.lang.String, and java.util.Calendar. Collections like lists and properties are recursively converted into java.util.Lists, and java.util.Maps. Unknown types are usually described as a string.

 (def play (str "on playNamedList(thisName)\n"
                "tell application \"iTunes\"\n"
                "play playlist thisName\n"
                "end tell\n"
                "end playNamedList\n"))

 (let [mngr (javax.script.ScriptEngineManager.)
       engine (.getEngineByName mngr "AppleScript")
       context (.getContext engine)
       bindings (.getBindings context javax.script.ScriptContext/ENGINE_SCOPE)]
   (.put bindings "javax_script_function", "playNamedList")
   (.put bindings javax.script.ScriptEngine/ARGV "chill")
   (.eval engine play))

In order to invoke calls that expects arguments to be set, we need to invoke the AppleScript method using "javax_script_function" engine binding. Overall it beats running osascript and parsing return values, but it still needs more documentation.

net-eval - Dead Simple Distributed Computing for Clojure

newlisp has this function called net-eval which allows you to distribute work across a bunch of remote nodes. Over the weekend I have put together a small library, which more or less works the same way. The idea behind net-eval is simple, you fire up a REPL server on the remote nodes, expressions are transferred through a socket evaluated and results returned.

 (deftask ping []
   (str "Pong: " (System/getProperty "os.name")))

We begin by defining tasks for code we want to evaluate on the remote nodes, deftask macro takes the body, defines a function and adds body of the task as a list to the functions meta data. You can treat tasks as functions, call them debug them just like any other Clojure function.

 (let [response (net-eval [["127.0.0.1" 9999 #'ping]
                           ["10.211.55.3" 9999 #'ping]])]
   (println (map deref response)))

net-eval does all the housekeeping required to connect to remote nodes, transfer functions to execute, and collect the results. net-eval takes a sequence of vectors containing host, port and a task to execute if there are arguments to be passed, they are appended to the end. net-eval will return a sequence of future objects, each containing response from one of the nodes.

(Pong: Mac OS X Pong: Windows 2000)

Now, care must be taken. It is not a good idea to have a REPL server listening on a public IP, you are basically giving away unrestricted shell access to anyone. I advise using net-eval behind a firewall or on a segregated network. If you really really need to run commands across the internet tunnel the connection through SSH.

A more useful example other than playing ping-pong with the remotes would be implementing poor man's MapReduce. Following snippet defines a task that will download a file, split it into words and count them, it maps a bunch of nodes to a bunch of documents and sends documents for processing, when all nodes return we sum the word count print total numbers words across documents.

 (def docs [;; A Comedy of Masks - Ernest Dowson and Arthur Moore, 547KB
            "http://www.gutenberg.org/files/16703/16703.txt"
            ;; The Adventures of Sherlock Holmes - Conan Doyle, 576KB
            "http://www.gutenberg.org/dirs/etext99/advsh12.txt"
            ;; The tale of Beowulf - anonymous, 219KB
            "http://www.gutenberg.org/files/20431/20431-8.txt"])

 (def nodes [["127.0.0.1" 9999]
             ["127.0.0.1" 9999] 
             ["10.211.55.3" 9999]])

 (deftask word-count [doc]
   (with-open 
       [stream (.openStream (java.net.URL. doc))]
       (let [buf (java.io.BufferedReader. 
                  (java.io.InputStreamReader. stream))]
         (count (re-seq #"\w+" (.toLowerCase (apply str (line-seq buf))))))))

 (defn map-jobs []
    (map #(conj (first %) #'word-count (second %))
         (partition 2 (interleave nodes docs))))

 (let [response (map deref (net-eval (map-jobs)))]
   (println "Total: " (apply + response)))

For processing large files, you can use compojure to transfer files across machines, if you build the library using lein, transferring uberjar to remote nodes is all thats needed, running uberjar will automatically start a REPL server.

A Simple Clojure IRC Client

The other night I was toying with the following script, I was going to thrash it but figured it may help someone or me later on so I am dumping it here. It doesn't do anything other then to sit idle in a channel,

 (ns irc
   (:import (java.net Socket)
            (java.io PrintWriter InputStreamReader BufferedReader)))

 (def freenode {:name "irc.freenode.net" :port 6667})
 (def user {:name "Nurullah Akkaya" :nick "nakkaya"})

 (declare conn-handler)

 (defn connect [server]
   (let [socket (Socket. (:name server) (:port server))
         in (BufferedReader. (InputStreamReader. (.getInputStream socket)))
         out (PrintWriter. (.getOutputStream socket))
         conn (ref {:in in :out out})]
     (doto (Thread. #(conn-handler conn)) (.start))
     conn))

 (defn write [conn msg]
   (doto (:out @conn)
     (.println (str msg "\r"))
     (.flush)))

 (defn conn-handler [conn]
   (while 
    (nil? (:exit @conn))
    (let [msg (.readLine (:in @conn))]
      (println msg)
      (cond 
       (re-find #"^ERROR :Closing Link:" msg) 
       (dosync (alter conn merge {:exit true}))
       (re-find #"^PING" msg)
       (write conn (str "PONG "  (re-find #":.*" msg)))))))

 (defn login [conn user]
   (write conn (str "NICK " (:nick user)))
   (write conn (str "USER " (:nick user) " 0 * :" (:name user))))

 ;;(def irc (connect freenode))
 ;;(login irc user)
 ;;(write irc "JOIN #clojure")
 ;;(write irc "QUIT")

Lots of this code should be self-explanatory, calling connect will open a socket to the server, it will return a ref containing a reader and a writer associated with the socket, it will also spawn a new thread that will handle incoming messages from the server.

conn-handler will keep reading and printing from the socket until exit key in the conn ref is set which happens when we receive a "Closing Link" message from the server, every once in a while server will ping us with "PING :randomstring" we need to reply "PONG :randomstring" else we get disconnected. Thats all there is to it, as I said it doesn't do anything but with a few regexes you can turn it in to client or a bot.

Using Multiple Lisps with Inferior Lisp

I was reading On Lisp which uses Common Lisp through out the book, so I needed a quick way to switch between lisps, following is a quick hack to switch between different lisp programs. When you call na-run-lisp without any prefix it will run the first item in lisp-programs, when called with a prefix you can select which lisp to run.

 (setq lisp-programs 
       (list (list "clojure" clojure-command)
             (list "sbcl" "/opt/local/bin/sbcl")))

 (defun na-run-lisp (arg)
   (interactive "P")
   (if (null arg)
       (run-lisp (second (first lisp-programs)))
     (let (choice) 
       (setq choice (completing-read "Lisp: " (mapcar 'first lisp-programs)))
       (dolist (l lisp-programs)
         (if (string= (first l) choice)
             (run-lisp (second l)))))))

Etch A Sketch

A small Sunday hack inspired by the classical children's toy, for this project you need two potentiometers, connected to analog pins 0 and 1, you can grab the fritzing file here,

Idea behind the project is simple, we read potentiometers, turn the values read into x and y coordinates for the turtle then command turtle to go to that coordinate.

 ;;WMath.cpp
 (defn map-range [x in-min in-max out-min out-max]
   (+ (/ (* (- x in-min) (- out-max out-min)) (- in-max in-min)) out-min))

 (defn read-knobs [board]
   (let [potx (analog-read board potx-pin)
         poty (analog-read board poty-pin)
         x (int (map-range 
                 potx 0 1024 (- (/ sketch-width 2)) (/ sketch-width 2)))
         y (int (map-range 
                 poty 0 1024 (- (/ sketch-height 2)) (/ sketch-height 2)))]
     {:x x :y y}))

Potentiometers returns values between 0 and 1024, assuming our image is 600 by 400, we need to map these values between -300 to 300 on the x axis and -200 to 200 on the y axis, 0,0 being the center of the image.

 (defn panel [board turtle]
   (proxy [JPanel ActionListener] []
     (paintComponent
      [g]
      (doto g
        (.setColor Color/red)
        (.fill (RoundRectangle2D$Double.
                0 0 window-width window-height 60 60))
        (.setColor Color/black)
        (.drawImage (:image @turtle) pad pad this)
        (.setColor Color/white)
        (.fill (Ellipse2D$Double. pad 430 40 40))
        (.fill (Ellipse2D$Double. (- sketch-width pad) 430 40 40))))
     (actionPerformed 
      [e] 
      (let [knobs (read-knobs board)]
        (go turtle (:x knobs) (:y knobs)))
      (.repaint this))))

With every tick of the timer, we'll read the potentiometers, then command the turtle to goto that position and repaint the panel to reflect changes.

 (defn key-listener [board frame timer]
   (proxy [KeyAdapter] [] 
     (keyReleased 
      [e]
      (.stop timer)
      (close board)
      (.setVisible frame false))))

When the sketch window receives a key event, we stop the timer, disconnect arduino and hide the window.

 (defn init-arduino []
   (let [board (arduino "/dev/tty.usbserial-A6008nhh")]
     (Thread/sleep 5000)
     (enable-pin board :analog potx-pin)
     (enable-pin board :analog poty-pin)
     board))

By default pins aren't read, you need to tell arduino which ports you are interested, in this case we are interested in analog 0 and analog 1.

 (defn init-turtle [board turtle]
   (let [knobs (read-knobs board)]
     (doto turtle
       (pen-up)
       (go (:x knobs) (:y knobs))
       (pen-down))))

Since we don't know the position of potentiometers when the application starts, we need to read the values once and position the turtle accordingly. When all the functions wired and timer started, you should have your Etch A Sketch.

 (defn sketch []
   (let [board  (init-arduino)
         turtle (turtle sketch-width sketch-height)
         panel  (panel board turtle)
         timer  (Timer. 50 panel)
         window (JFrame.)]
     (doto window
       (.add panel)
       (.addKeyListener (key-listener board window timer))
       (.setBackground (Color. 0 0 0 0))
       (.setUndecorated true)
       (.setAlwaysOnTop true)
       (.setSize (java.awt.Dimension. window-width (+ 25 window-height)))
       (.setVisible true))
     (init-turtle board turtle)
     (.start timer)))

GZIP Output Compression in Compojure

Compression can be handled by the server, jetty or apache but I prefer my applications handling most of their operations, so I settled on using a middleware function to add compression to parts of my application. Compojure uses middleware to selectively add functionality to handlers such as sessions. middleware functions takes handlers, after your handler completes processing the request, middleware gets a chance to transform the response, giving you a tool for abstracting common functionality.

 (defn with-gzip [handler]
   (fn [request]
     (let [response (handler request)
           out (java.io.ByteArrayOutputStream.)
           accept-encoding (.get (:headers request) "accept-encoding")]

       (if (and (not (nil? accept-encoding))
                (re-find #"gzip" accept-encoding))
         (do
           (doto (java.io.BufferedOutputStream.
                  (java.util.zip.GZIPOutputStream. out))
             (.write (.getBytes (:body response)))
             (.close))

           {:status (:status response)
            :headers (assoc (:headers response)
                       "Content-Type" "text/html"
                       "Content-Encoding" "gzip")
            :body (java.io.ByteArrayInputStream. (.toByteArray out))})
         response))))

with-gzip handler first checks the client request, if the client supports gzip compression, it uses GZIPOutputStream to compress the response, add the header telling the client that it contains compressed content. If client doesn't support gzip encoding response is not compressed.

 (defn hello-world [request]
   {:status  200
    :headers {}
    :body    "Hello, World!!"})

 (decorate hello-world
   (with-gzip))

decorate macro takes care of applying middleware to handler.