An explorer's log

Ferret Lisp FFI Notes

Sat, 24 Jun 2017 00:00:00 +0300

Value
Extending Ferret Objects
Pointer
Macros for C++ Code Generation

This posts serves as documentation on how to interact with third party C,C++ libraries when using Ferret Lisp, a free software Clojure implementation, it compiles a restricted subset of the Clojure language to self contained ISO C++11 which allows for the use of Clojure in real time embedded control systems. It has been verified to run on architectures ranging from microcontrollers with as little as 2KB of RAM to general purpose computers running Linux/Mac OS X/Windows.

Following Clojure's design philosophy of embracing the host. Ferret provides multiple ways of interacting with the host language. Currently there are 4 methods that can be used to pass data back and forth between Ferret and C,C++,

Value
Extending Ferret Objects
Pointer
Macros for C++ Code Generation

Value

A value Ferret object can used to capture any C++ type T. This is useful when interacting with modern C++ libraries where instead of using pointers they use smart pointers. One example for these types of libraries is the OpenCV computer vision library. Following is the Hello World of OpenCV, a simple app to open the first attach camera, grab a frame from it and show that frame on a GUI element.

#include "opencv2/opencv.hpp"

using namespace cv;

int main(int argc, char** argv){

  VideoCapture cap;

  if(!cap.open(0))
    return 0;

  for(;;){
    Mat frame;
    cap >> frame;

    imshow("Cam 0", frame);
    waitKey(1);
  }

  return 0;
}

For converting the above program we need to create 4 functions,

A function to return a handle to a VideoCapture object.
A function to grab a frame from the capture.
A function to show the grabbed frame on a OpenCV window.
A function to tell OpenCV to update GUI.

(defnative video-capture [n]
  (on "defined FERRET_STD_LIB"
      ("opencv2/opencv.hpp")
      "using namespace cv;
       VideoCapture cap;

       cap.open(number::to<number_t>(n));

       if (!cap.isOpened())
         return nil();

       __result = obj<value<VideoCapture>>(cap);"))

Converting Ferret types to C++ types are handled by ::to= functions. Every built in type has a static ::to= function to convert the type to its native counter part. Above function takes the camera id as a parameter and can be used in the following way,

(def capture (video-capture 0))

In order to convert camera id to a native number one can use the number objects ::to= method.

int cam_id = number::to<int>(n);

Once we have the camera id we can open the attached camera, check if it is correctly initialized using isOpened() method, if not return nil otherwise we use a value Ferret object to return the OpenCV VideoCapture object back to Ferret side. A value Ferret object can hold any type T.

var dev = obj<value<VideoCapture>>(0);

Any arguments passed to value constructor will be forwarded to the type T's constructor. A value object acts just like any other Ferret object type, and its lifetime is handled by the active GC. (Ferret supports various schemes for garbage collection.)

(defn query-capture [c]
  "using namespace cv;
   Mat frame;
   value<VideoCapture>::to_value(c) >> frame;
   __result = obj<value<Mat>>(frame);")

Type T contined within a value type can be accessed using the ::to_value static function of the value class.

VideoCapture dev = value<VideoCapture>::to_value(c);

A value object is used to return the latest frame from the camera back to Ferret.

(def frame (query-capture capture))

Two other function needs to be wrapped to convert the above C++ program in to a Ferret program.

(defnative imshow [n f]
  (on "defined FERRET_STD_LIB"
      ("opencv2/highgui/highgui.hpp")
      "using namespace cv;
       imshow(string::to<std::string>(n), value<Mat>::to_value(f));"))

(defn wait-key [n]
  "__result = obj<number>(cv::waitKey(number::to<number_t>(n)));")

Once all the required API functions are wrapped above C++ program can be represented using the following Ferret program.

(def capture (video-capture 0))

(forever
 (->> capture
      query-capture
      (imshow "Cam 0"))
 (wait-key 1))

Extending Ferret Objects

For tighter integration with Ferret, users can define their own types. While a value object is just a container holding a type T user defined objects can implement built in interfaces making them seekable and/or callable. i.e, instead of using the built in D-List based maps, one can use any map implementation from any C,C++ library and it will behave just like the built in type. In order to create a user defined object you have to define a class that extends object and implement some methods required by Ferret.(stream_console, equals, type). When using this method garbage collection is also automatically handled by the active GC scheme. No manual allocation deallocation is necessary other than what is required by the native library.

Lets assume you are doing embedded work using an Arduino, want to interface with a humidity and temperature sensor (HIH6130) and you want your sensor object to be callable. Giving you a Ferret API that can be used like the following,

(def sensor (new-hih 0x27))

(let [[temprature humidity] (sensor)]
  (println temprature humidity))

First we need to define a class that contains the implementation for the object. You can make any user defined object callable by implementing the lambda_i and overriding invoke method.

#include <HIH61XX.h>

class hih : public lambda_i {
  HIH61XX hih;

public:

  type_t type() const final { return type_id<hih>; }

  bool equals(var const & o) const { return (this == o.get()); }

#if !defined(FERRET_DISABLE_STD_OUT)
  void stream_console() const {
    runtime::print("HIH Temrature Sensor");
  }
#endif

  explicit hih() : hih(0x27) { }

  ~hih(){ }

  var invoke(var const & args) const {
    char status;
    double T,P,p0,a;

    status = pressure.startTemperature();
    if (status != 0){
      delay(status);
      status = pressure.getTemperature(T);
      if (status != 0){
        status = pressure.startPressure(3);
        if (status != 0){
          delay(status);
          status = pressure.getPressure(P,T);
          if (status != 0){
            p0 = pressure.sealevel(P, default_altitude);
            __result = runtime::list(obj<number>(T), obj<number>(P));
          }
        }
      }
    }

    return nil();
  }
};

And somewhere in your program tell Ferret where the implementation for the object is,

(defobject hih "/path/to/hih_implementation.h")

(defn new-hih [id]
  "__result = obj<hih>(number::to<int>(id));")

Pointer

For interacting with legacy C++ libraries or C libraries Ferret provides a pointer type. It holds a pointer to a type T. Unlike a value type or user defined object, user is responsible for garbage collection.

(defn new-int [i]
  "__result = obj<pointer>(new int(number::to<int>(i)));")

(defn print-int [i]
  "std::cout << pointer::to_pointer<int>(i) << std::endl;")

Macros for C++ Code Generation

Once in a while you may come across a library that does not play well with any of the above methods and custom C++ code should be generated for each invocation. i.e You want to register a Ferret function as a interrupt service routine on a Raspberry PI via WiringPi library. wiringPiISR function expects a C style callback function, a Ferret function can not be passed as is. What we can do is for every Ferret function we want to register as callback, define a native function that in turn invokes the Ferret function.

(defmacro attach-isr [pin callback]
  (let [fn (gensym)]
    `(~'do
      (~'def ~fn ~callback)
      (~'cxx ~(str "wiringPiISR (" pin ", INT_EDGE_FALLING, [](){ run(" fn ");})")))))

wiringPiISR expects a function with the signature void (*function)(void)) so C++ lambdas can not capture the Ferret function variable passed. What the above macro does is define a unique variable for the function passed and using a non capturing lambda execute the function pointed by the unique variable.

(attach-isr 0 #(println "Interrupted!")) ;; attach interrupt in WiringPI pin 0

Bare Metal Lisp - RC Control using Ferret

Wed, 15 Feb 2017 00:00:00 +0200

Some sample code to demonstrate FFI capabilities of Ferret on embedded systems. Ferret is a free software Clojure implementation, it compiles a restricted subset of the Clojure language to self contained ISO C++11 which allows for the use of Clojure in real time embedded control systems. See Project Home Page. Mobile platform used in this post is a Zumo Robot (required third party libraries) and the microcontroller is a Arduino Uno / Atmega328.

(configure-ferret! :command "mv ferret-rc-control.cpp ferret-rc-control.ino")

Ferret outputs a cpp file however Arduino IDE expects a ino file. We override build command to rename the cpp file to a ino file.

(configure-runtime! FERRET_MEMORY_POOL_SIZE 1024)

On embedded systems Ferret can be configured to avoid calling malloc/free. Enabling memory pooling will allocate (1 kb in this case) memory to be used as heap.

Ferret's FFI is modeled after Gambit scheme. Whereas Gambit scheme lets you embed C into Scheme, Ferret lets you embed C or C++ into Clojure.

(native-header "ZumoMotors.h")

(defn new-motor []
  "__result = obj<value<ZumoMotors>>();")

A value object is useful when dealing with C++ objects. It acts as a container for native object. It can wrap any type T, once wrapped it can be used just like any other Ferret variable. (Any arguments passed to obj will be forwarded to T constructor.)

(defn speeds [m s1 s2]
  "ZumoMotors motor = value<ZumoMotors>::to_value(m);
   motor.setSpeeds(number::to<int>(s1), number::to<int>(s2));")

(defn pulse-in [pin]
  "__result = obj<number>(pulseIn(number::to<int>(pin), HIGH,25000));")

Calling C++ functions with Ferret variables is just as easy, every Ferret object type has a corresponding to / from function/s. These can be used to convert native types to ferret types and vice versa. (Ferret is a strongly, dynamically typed language. If you try to convert a ferret number to a value you will get a segmentation fault.)

;; throttle - ch 1
;; steering - ch 2
(defn read-control []
  (let [throttle (scale (pulse-in 4) 981 1998  0   254)
        steering (scale (pulse-in 5) 981 1998 -254 254)]
    (list (if (< throttle 5) ;; setup dead zone
            0
            throttle)
          (if (< -5 steering 5) ;; setup dead zone
            0
            steering))))

(let [motor (new-motor)]
  (forever
   (let [[throttle steering] (read-control)]
     (speeds motor (+ throttle steering) (- throttle steering)))))

Radio-control transmitters and receivers are used to drive model cars or planes. They typically have a bunch of control surfaces like joysticks. Each degree of freedom that the controller gives is assigned a channel. They are typically used to control servos so each channel outputs a signal called PWM (Pulse Width Modulation). i.e Width of the signal changes as control input changes. Arduino platform has a function called pulseIn which we wrapped in a Ferret function that returns the length of the pulse, this pulse is then converted to a PWM value which is used to drive the motors on the robot.

Forward motion is controlled by mapping the current pulse width on channel 1 to a value between 0 and 254. No throttle 0 PWM full throttle 254 PWM. Steering is controlled in a similar manner but instead of mapping from 0 to 254 it is mapped to -254 to 254. Final motor speed is calculated by adding throttle and steering controls together and apply the resulting PWM to motors.

Ferret - A Hard Real-Time Clojure for Lisp Machines

Fri, 10 Jun 2016 00:00:00 +0300

Ferret is a free software Clojure implementation, it compiles a restricted subset of the Clojure language to self contained ISO C++11 which allows for the use of Clojure in real time embedded control systems. It has been verified to run on architectures ranging from embedded systems with as little as 2KB of RAM to general purpose computers running Linux/Mac OS X/Windows.

Project Home Page
Executable (Requires Bash and JVM)
Standalone Jar (Requires JVM)

Unlike other Lisp implementations Ferret is specifically tailored for real time control applications, provides deterministic execution and gives you total control on memory management. (Achilles heel of high level languages for hard real time control). Ferret programs when running on embedded systems or in single threaded mode can be configured to run using a memory pool. This avoids calling malloc/heap at runtime, improves performance and determinism, you can also tell how much memory will be used at compile time.

Starting with a simple example, following shows a simple Arduino Due example to generate a Sawtooth wave,

;; saw.clj
(configure-runtime! FERRET_DISABLE_STD_OUT true ;; Disable Serial
                    FERRET_PROGRAM_MAIN "signal()" ;; Main Entry for the application
                                                   ;; Ferret functions are C++ functors
                    FERRET_MEMORY_POOL_SIZE 512) ;; Allocate 512 Bytes of static memory

(require '[ferret.arduino :as gpio])

(defn dac-1 []
  "__result = obj<number>(DAC1);")

(defn signal []
  (doseq [sample (range 255)]
    (gpio/analog-write (dac-1) sample)
    (sleep 10)))

All functions mimic their Clojure counterparts the only thing that's different from Clojure is the dac-1 function. A function with just a string in its body is assumed to be a FFI call and the string is assumed to be native statement/s. In this case dac-1 function when called returns the value of DAC1. (Pin number of digital analog converter on Due.) In order to run the example on hardware just compile it using Ferret,

$ mkdir saw
$ ./ferret -i saw.clj
$ mv saw.cpp saw.ino

then upload as usual. Best way to experiment with Arduino and Ferret is to set the Arduino IDE to use an external editor in preferences and add the following to you Ferret program, (Overrides the compile command with mv by default Ferret uses g++ or the value of CXX environment variable to compile the generated code into binary.)

(configure-ferret! :command "mv saw.cpp saw.ino")

then you can just run,

$ ./ferret -i saw.clj -c

Arduino IDE will pick up changes automatically when you hit upload. Moving on to a more complicated example, a line following robot, following shows the built in primitives of Ferret for control (State Machines, PID Controller) and interfacing with third party C++ libraries using an Arduino Due and a Zumo robot platform.

(configure-runtime! FERRET_DISABLE_STD_OUT true ;;save memory
                    FERRET_PROGRAM_MAIN program ;; Default function to run.
                    FERRET_MEMORY_POOL_SIZE 1024) ;; Allocate 1kb Heap

(require '[ferret.arduino :as gpio])

;; Inlucde Third Party Headers
;; https://github.com/pololu/zumo-shield
(native-header "QTRSensors.h"
               "ZumoReflectanceSensorArray.h"
               "ZumoMotors.h"
               "Pushbutton.h")

;; Initialize Hardware
(native-declare "unsigned int line_buffer[6];
                 ZumoReflectanceSensorArray line_sensors;
                 ZumoMotors zumo_motors;
                 Pushbutton button(ZUMO_BUTTON);")

(defn wait-button []
  "button.waitForButton();
   button.waitForRelease();
   __result = obj<boolean>(true)")

(defn button-pressed? []
  "__result = obj<boolean>(button.isPressed())")

(defn motors
  ([]
   "zumo_motors.setSpeeds(0,0);")
  ([c]
   "int corr = number::to<int>(c);
    zumo_motors.setSpeeds(400 + (-1 * corr), 400 + corr);")
  ([m1 m2]
   "zumo_motors.setSpeeds(number::to<int>(m1), number::to<int>(m2));"))

(defn read-line []
  "__result = obj<number>(line_sensors.readLine(line_buffer,QTR_EMITTERS_ON,1));")

Wrappers for Zumo Libraries needed to interface with the shield. All follow the same FFI convention explained in the first example.

(defn calibrate []
  (cxx "line_sensors.init();")
  (sleep 1000)
  (dotimes [i 80]
    (if (or (and (> i 10) (<= i 30))
            (and (> i 50) (<= i 70)))
      (motors -200  200)
      (motors  200 -200))
    (cxx "line_sensors.calibrate();")
    (sleep 20))
  (motors))

calibrate function is run when the robots boots up and used to calibrate the Pololu QTR Reflectance Sensors on the shield. cxx macro is a convenience macro for calling native code in place. It will define a fn and call it in place so,

is equivalent to,

((fn [] "line_sensors.init();"))

Next we define a PID controller. It takes the result of read-line, a number between 0 to 5000 depending on the position of the line under the sensor and returns correction that we are going to feed in to motors function that will keep the sensor at position 2500 which means the line is in the middle of the robot.

(def motor-controller (pid-controller :kp 0.75
                                      :ki 0
                                      :kd 1
                                      :set-point 2500
                                      :bounds [0 5000 -400 400]
                                      :continuous false))

A state machine glues all of the above together. fsm macro compiles a function, which when called will iterate through its transition states.

(def program
  (fsm
   (states
    (boot (motors)
          (gpio/pin-mode 53 :output)
          (wait-button)
          (calibrate))

    (follow-line (gpio/digital-write 53 1)
                 (->> (read-line)
                      (motor-controller)
                      (motors))
                 (gpio/digital-write 53 0))

    (stop (motors)
          (cxx "button.waitForRelease();")))

   (transitions
    (boot         wait-button      follow-line)
    (follow-line  button-pressed?  stop)
    (stop         wait-button      follow-line))))

Robot starts at boot state, where we wait for the user to press a button to start calibrating the sensors, when done we wait for another button press to transition into follow-line state where we read the current position of the line and feed it to the PID controller and feed the controllers result to motors function. After each iteration of follow-line we check if the button is pressed when pressed we transition into stop state otherwise we keep executing follow-line state. In stop state we stop and wait for the user to press the button again to continue following the line.

Pin 53 is used to produce to following oscilloscope output to show the jitter in the system.

Static Galleria Script

Tue, 7 Oct 2014 00:00:00 +0300

Following is a Clojure shell script that piggy backs on the static.jar to create static image galleries using galleria and upload them to S3.

Either tangle this file (it will tangle to ~/.bin/static-galleria) or copy/paste the snippets into a file on your path and make it executable, and update path to static.jar. This script depends on ImageMagick command line tools.

bash ~/.bin/static-galleria "Photo Folder"

See Photos page for example static galleries hosted on S3.

#^:shebang '[
             exec java -cp "static-app.jar" clojure.main "$0" -- "$@"
               ]

(ns album-builder
  (:use clojure.java.shell)
  (:use [static io core config]))

(defn images [in-dir]
  (->> (map #(java.io.File. in-dir %) (.list in-dir))
       (filter #(let [ext (org.apache.commons.io.FilenameUtils/getExtension (.toString %))]
                  (and (.isFile %)
                       (contains? #{"JPG" "jpg" 
                                    "JPEG" "jpeg" 
                                    "png" "gif"} ext))))))

(defn prep-images [in-dir out-dir]
  (println "Prepping Images")
  (doall
   (pmap (fn [f]
           (org.apache.commons.io.FileUtils/copyFileToDirectory f out-dir)

           ;;fix orientation
           (let [file (str out-dir "/"
                           (->> (.toString f)
                                (org.apache.commons.io.FilenameUtils/getName)))] 
             (clojure.java.shell/sh 
              "convert" "-auto-orient" file file))) 
         (images in-dir)))

  ;;scale originals
  (sh "mogrify" "-geometry" "1024x" (str out-dir "/*"))
  ;; generate thumbs
  (sh "mkdir" (str out-dir "/thumbs"))
  (sh "mogrify" "-thumbnail" "256x"
      "-path" (str (str out-dir "/") "thumbs")  (str out-dir "/*")))

(defn galleria-img-json [idx files]
  (str
   "
    var data" idx " = ["
   (reduce (fn[h v] 
             (let [name (->> (.toString v)
                             (org.apache.commons.io.FilenameUtils/getName)
                             (str "./"))
                   thumb (->> (.toString v)
                              (org.apache.commons.io.FilenameUtils/getName)
                              (str "./thumbs/"))]
               (str h 
                    "{thumb: '" thumb "',"
                    "image: '" name "',},")))
           "" files)
   "];"))

(defn album-html [name in-dir]
  (let [images (partition 20 20 nil (images in-dir))]
    (list
     [:script
      (map-indexed (fn [idx val] (galleria-img-json idx val)) images)]

     [:div {:id "pagination"}]
     [:div {:id "galleria"}]

     [:script "document.body.style.backgroundColor = \"#333\";

             var gallery;
             var currentPage = 0;

             jQuery(window).load(function () {
                var anchor = window.location.hash.substring(6);
                if(anchor){
                   $(\"#pagination\").pagination('selectPage', parseInt(anchor));
                }
             });

             $( document ).ready(function() {
               $(function() {

             if (\"onhashchange\" in window)
                 window.onhashchange = function () {
                   var anchor = window.location.hash.substring(6);
                   if(anchor)
                      $(\"#pagination\").pagination('selectPage', parseInt(anchor));
                   else
                      $(\"#pagination\").pagination('selectPage', 1);
             }

             $(\"#pagination\").pagination({
                 items: " (count images) ",
                 itemsOnPage: 1,
                 cssStyle: 'dark-theme',
                 onPageClick:function(pageNumber, event){
                   pageNumber = pageNumber - 1;
                   if(gallery && (currentPage != pageNumber)){
                     gallery.load(eval(\"data\" + pageNumber));
                     currentPage = pageNumber;
                   }
                 }
             });
         });
       });

             Galleria.loadTheme('http://photos.nakkaya.com/galleria/themes/folio/galleria.folio.min.js');

             var dataSource = data0;
             var anchor = window.location.hash.substring(6);
             if(anchor){
               currentPage = (parseInt(anchor) - 1);
               dataSource = eval(\"data\" + currentPage);
             }

             Galleria.configure({
                 variation: 'dark',
                 dataSource:dataSource
             });

             Galleria.ready(function(){
                 gallery = this;
                 this.lazyLoadChunks(3);
             });

             Galleria.run('#galleria',{
               responsive:true,
               height:1,
               debug:true,
               preload:2
             });"])))

(defn dump-album [album-name in-dir out-dir]
  (prep-images in-dir out-dir)
  (static.io/write-out-dir 
   (str album-name "/index.html")
   (hiccup.core/html
    [:html
     [:head
      [:title (org.apache.commons.io.FilenameUtils/getName (str in-dir))]
      ;;(org.apache.commons.io.FilenameUtils/getFullPathNoEndSeparator in-dir)
      [:script {:src "http://ajax.googleapis.com/ajax/libs/jquery/1/jquery.js"}]
      [:script {:src "http://photos.nakkaya.com/galleria/galleria-1.3.5.min.js"}]
      [:script {:src "http://photos.nakkaya.com/simplepagination/js/jquery.simplePagination.js"}]
      [:link {:href "http://photos.nakkaya.com/simplepagination/css/simplePagination.css"
              :type "text/css"
              :rel "stylesheet"}]]
     [:body
      [:div
       (album-html album-name in-dir)]]])))

(def in-dir (second *command-line-args*))
(def album-name (.toString (java.util.UUID/randomUUID)))

(set!-config :out-dir "./")
(println "Building " album-name)
(println "In Dir:" in-dir)
(dump-album album-name (java.io.File. in-dir) (java.io.File. album-name))

(when (> (count *command-line-args*) 2)
  (println "Uploading...")
  (clojure.java.shell/sh 
   "s3cmd" "sync" "--delete" "--acl-public" 
   (str album-name "/") (str (nth *command-line-args* 2) "/" album-name "/")))

(shutdown-agents)

;;$('.galleria-container').height($('#galleria').height());

Pentadactyl Configuration

Sun, 26 Jan 2014 00:00:00 +0200

This is a dump of all Pentadactyl related configuration and scripts so I can edit them in one place and tangle from a single file. Pentadactyl is a Firefox extension that provides a more efficient user interface for keyboard-fluent users, allowing you to browse without using a mouse.

Disable smooth scroll,

set scrollsteps=1
set scrolltime=0

Editor, C-e invokes it in text fields,

set editor="emacsclient"

Show pattern matches as you type,

set incfind

Highlight previous search pattern matches,

set hlfind

Ignore case in search unless pattern has uppercase chars,

set findcase=smart

Color Scheme,

colorscheme solarized-dark

Faster Scrolling,

map -b j 8j
map -b k 8k
map -b h 8h
map -b l 8l

Use Full Zoom,

map -b zi ZI
map -b zo ZO

Use characters for hints,

set hintkeys=asdflkj
hi -a Hint font-size: 9pt !important;

Focus settings, (Let gmail mess with DOM)

set strictfocus=google.com:laissez-faire,'chrome:*':laissez-faire,*:despotic

Load custom css modifications,

so ~/.pentadactyl/user.css

Shortcuts to frequently used bookmarks,

" Email current url
command email-url :open gmail-this

" Save to org-mode Read Later Tree
" command read-later :open org-capture-read-later

Open link in a private tab,

command private-tab -nargs=+ -description "Open Private Tab" -javascript <<EOF
        privateTab.readyToOpenTab(true);
        gBrowser.addTab("about:blank");
        dactyl.execute(":tablast");
        dactyl.execute(":open " + args);
        privateTab.stopToOpenTabs();
EOF

Toggle plugins.click_to_play,

command toggle-click-to-play -js <<EOF
        if(Services.prefs.getBoolPref("plugins.click_to_play"))
            Services.prefs.setBoolPref("plugins.click_to_play", false);
        else
            Services.prefs.setBoolPref("plugins.click_to_play", true);
        dactyl.execute(":reload");
EOF

Convert given time to local time using wolfram alpha,

command local-time -nargs=+ -description "Convert Date to Local Time" -javascript <<EOF
        dactyl.execute(":tabopen");
        dactyl.open("http://www.wolframalpha.com/input/?i=" + args + " in local time");
EOF

Custom search commands,

command search-nakkaya -nargs=+ -description "Search nakkaya.com" -javascript <<EOF
        dactyl.execute(":tabopen");
        dactyl.open("site:nakkaya.com " + args);
EOF

command search-hackernews -nargs=+ -description "Search news.ycombinator.com" -javascript <<EOF
        dactyl.execute(":tabopen");
        dactyl.open("site:news.ycombinator.com " + args);
EOF

Save to org-mode Read Later Tree,

command read-later -description "Execute emacsclient" -javascript <<EOF
        var emacsClient = "/Applications/Emacs.app/Contents/MacOS/bin/emacsclient";
        var osString = Components.classes["@mozilla.org/xre/app-info;1"]  
                       .getService(Components.interfaces.nsIXULRuntime).OS;

        if(osString == "Linux")
          emacsClient = "/home/nakkaya/apps/emacs/bin/emacsclient"

        var url = "\"org-protocol://capture://l/"+ encodeURIComponent(content.location) + "/"+ encodeURIComponent(content.document.title) +"/" + encodeURIComponent(content.window.getSelection()) + "\"";
        var command = emacsClient + " " + url;
        CommandExMode().open("!" + command);
EOF

Rename tab,

:com tabrename -nargs=1 -literal=0 -js content.document.title = args[0];

GMail settings,

" Allow GMail shortcuts
set passkeys+=mail.google.com:jksacrqG#<CR>,gi,gl,gp
" Let Hint Message from X Links 
autocmd LocationChange mail.google.com set ht+=span.ata-asJ
" Let Hint Expand Collapsed Conversations
autocmd LocationChange mail.google.com set ht+=span.adx
" Let Hint Expand Message
autocmd LocationChange mail.google.com set ht+=td.gF
" Let Hint Expand Delete All Spam Messages
autocmd LocationChange mail.google.com set ht+=span.x2

Auto pin mail.google.com,

autocmd PageLoad mail.google.com js if(!getBrowser().mCurrentTab.pinned) getBrowser().pinTab(getBrowser().mCurrentTab);
autocmd PageLoad gmail.com js if(!getBrowser().mCurrentTab.pinned) getBrowser().pinTab(getBrowser().mCurrentTab);

Auto pin git-annex tab,

autocmd PageLoad '127\.0\.0\.1:\d+/\?auth=' js if(!getBrowser().mCurrentTab.pinned) getBrowser().pinTab(getBrowser().mCurrentTab);

Additional hints,

" Hint Instagram Load More
autocmd LocationChange instagram.com set ht+=a.button.button-grey.button-large

~/.pentadactyl/user.css,

/* @-moz-document domain("www.reddit.com") { */
/*     .side{ display: none !important; } */
/*     .md { max-width: 100% ! important;} */
/* } */

@-moz-document domain("10.1.2.50"), domain("127.0.0.1") {
    #jwplayer_wrapper,#media_control{width:100% !important;}
    #jwplayer_wrapper{height:75% !important;}
    #progress_slider{width:99% !important;
                     height: 10px !important;}
    .ui-slider .ui-slider-handle {height: 15px !important;}
}

@-moz-document domain("youtube.com") {
    #confirmBox.yt-alert-default.yt-alert-warn{ display: none !important; }
}

@-moz-document domain("google.com.tr"), domain("google.com"), domain("google.de"){
    #gbzw,       /* links */
    #gbx3, #gbx4, /* background */
    #fll, #bfl
    {
        display: none !important;
    }

    #gbq, #gbu,  /* controls */
    #gbx1, #gbx2 /* background */
    {
        top: 0 !important;
    }

    /* remove google plus  */
    ol.gbtc { display: none ! important;}
    /* remove safe search button */
    a#abar_button_ss.ab_button { display: none ! important;}
}

@-moz-document url-prefix("https://mail.google.com/"){
    #gbzw,       /* links */
    #gbx3, #gbx4 /* background */
    {
        display: none;
    }

    #gbq, #gbu,  /* controls */
    #gbx1, #gbx2 /* background */
    {
        top: 0 !important;
    }

    /* Bar container */
    #gb {
        height: 72px !important;
    }

    /* Share pop-up content */
    #gbwc.gbmwca {
        top: -29px;
    }

    /* remove google plus shit  */
    ol.gbtc { display: none ! important;}

    /* get rid of copy right */
    div.wIFnie { display: none !important; }

    /* Remove people widget completely */
    .nH.adC {
        display: none ! important;
    }

    .yPPMxf, .nH.adC, .qHcrae, .y3, .y4 {
        width: 0 ! important;
    }

    /* Plain-text Message Body */
    /* Compose Interfaces */
    .editable.LW-avf, .editable, .ii, .Ak{
        font-size: 14px ! important;
        font-family: monospace ! important;
    }

    td.Bu.y3
    {
        display: none !important;
    }
}

@-moz-document domain("www.facebook.com") {
    #rightCol{ display: none !important; }
    #appsNav{ display: none !important; }
    #pagesNav{ display: none !important; }
    .fbChatSidebar,#fbDockChatBuddylistNub{ display: none !important; }
}

@-moz-document domain("twitter.com") {
    div.module.trends{ display: none !important; }
    div.module.site-footer{ display: none !important; }
    div.module.wtf-module.js-wtf-module.has-content{ display: none !important; }
    div.global-nav{ display: none !important; }
    .wrapper.white { background: none !important;}
}

@-moz-document domain("facebook.com") {
    /** Set Background color for page **/
    .fbIndex { background-color: #3B5998 !important }

    /** Hide Crap **/
    .fbIndex #globalContainer #dropmenu_container,
    .fbIndex #globalContainer #content,
    .fbIndex #globalContainer #pageFooter { display: none !important }

    /** Center Login Bar **/
    .fbIndex .loggedout_menubar_container {
        position: fixed !important;
        width: 420px !important;
        height: 82px !important;
        min-width: 0 !important;
        top: 50% !important;
        left: 50% !important;
        margin-top: -17px !important;
        margin-left: -210px !important;
        z-index: -1 !important;
    }

    /** Center Login form **/
    .fbIndex .loggedout_menubar { width: auto !important }
    .fbIndex .loggedout_menubar_container .lfloat,
    .fbIndex .loggedout_menubar_container .rfloat { float: none !important }
    .fbIndex .loggedout_menubar_container .lfloat img,
    .fbIndex .loggedout_menubar_container .rfloat #login_form table { display: block !important; margin: 0 auto !important }
    .fbIndex .loggedout_menubar_container .lfloat i { display: block !important; margin: -70px auto 20px !important; }
    .fbIndex .loggedout_menubar_container .sp_69c1xs { display: block !important; }

    /** Hide Homepage Box **/
    #SetAsHomepage_Callout {
        display: none;
    }

    /** Reset bluBar z-index **/
    .fbIndex div#blueBar {
        z-index: 0 !important;
        border: none !important;
        box-shadow: none !important;
    }
}

@-moz-document domain(news.ycombinator.com) {
  p { word-break: break-all; }

  #non-existent-id {}

  #csres { display: none; }

  .title {
    font-size: 120% !important;
  }

  .subtext {
    font-size: 90% !important;
  }

  * {
    font-family: Helvetica !important;
    line-height: 1.4 !important;
  }

  body {
    padding: 10px !important;
    background: #F6F6EF;
  }

  body > center > table {
    width: 100% !important;
  }

  body > center > table > tbody > tr:first-child > td {
    font-size: 120%;
    -moz-border-radius: 5px;
    -moz-box-shadow: 0 0 10px rgba(0,0,0,0.5);
  }

  .title a {
    font-family: Museo !important;
  }

  .title a:visited {
    color: #666 !important;
  }

  .comment,
  .comhead {
    font-size: 120% !important;
  }

  .title .comhead {
    font-size: 85% !important;
  }

  .pagetop {
    display: block;
    padding: 6px 0 4px;
    font-size: 110% !important;
  }

  .pagetop a {
    font-family: Helvetica !important;
    color: rgba(0,0,0,0.6) !important;
    font-size: 110% !important;
  }

  img[src=http\:\/\/ycombinator\.com\/images\/y18\.gif] {
    margin: 0 2px 0 5px;
  }
}

tmux Configuration

Sun, 5 Jan 2014 00:00:00 +0200

This is a dump of all tmux related configuration and scripts so I can edit them in one place and tangle from a single file.

Start windows and panes at 1, not 0,

set -g base-index 1
set -g pane-base-index 1

Replace C-b with \,

unbind C-b 
set -g prefix '\'
bind-key '\' send-prefix
set-window-option -g xterm-keys on

Setup key bindings,

bind-key r command-prompt -p "rename window to:"  "rename-window '%%'"
bind t source-file ~/.tmux-over-ssh.conf
bind k confirm kill-window
bind K confirm kill-server

bind tab last-window

# window movement / renumbering like in screen's :number
bind-key m command-prompt -p "move window to:"  "swap-window -t '%%'"

Enable UTF-8,

setw -g utf8 on
set -g status-utf8 on

setw -g window-status-current-format "|#I:#W|"

Makes using the scroll wheel automatically switch to copy mode and scroll back the tmux scrollback buffer.

set -g mouse on

Status bar,

set-option -g status-interval 60
set-option -g status-right-length 120
set -g status-right '#(date +"%a %b %_d %H:%M") | #(hostname)'

Create a new window, swtich to home directory and type tmux-ssh,

neww -n tmux-ssh
send-keys -t tmux-ssh "cd ~/" C-m
send-keys -t tmux-ssh "tmux-ssh "

Create/attach a dev session. Start tmux create two windows for two emacs instances for for editing one for dired.

TERM=xterm-256color
tmux has-session -t dev 
if [ $? != 0 ]
then
    tmux new-session -s dev -n emacs -d
    tmux send-keys -t dev 'cd ~/' C-m 
    tmux send-keys -t dev 'emacs -main-instance' C-m
    tmux new-window -n dired -t dev
    tmux send-keys -t dev 'cd ~/' C-m 
    tmux send-keys -t dev 'emacs' C-m
fi
tmux attach -t dev

Solarized theme,

# default statusbar colors
set-option -g status-bg colour235 #base02
set-option -g status-fg colour136 #yellow
set-option -g status-attr default

# default window title colors
set-window-option -g window-status-fg colour244 #base0
set-window-option -g window-status-bg default
#set-window-option -g window-status-attr dim

# active window title colors
set-window-option -g window-status-current-fg colour166 #orange
set-window-option -g window-status-current-bg default
#set-window-option -g window-status-current-attr bright

# pane border
set-option -g pane-border-fg colour235 #base02
set-option -g pane-active-border-fg colour240 #base01

# message text
set-option -g message-bg colour235 #base02
set-option -g message-fg colour166 #orange

# pane number display
set-option -g display-panes-active-colour colour33 #blue
set-option -g display-panes-colour colour166 #orange

# clock
set-window-option -g clock-mode-colour colour64 #green

SSH as a Hidden Service with Tor

Sun, 5 Jan 2014 00:00:00 +0200

Note to self, setup .torrc,

RunAsDaemon 1

HiddenServiceDir /home/tor/.hidden-ssh/
HiddenServicePort 22 127.0.0.1:22

Locate tor address,

cat ~/.hidden-ssh/hostname

Edit rc.local so tor starts during boot,

su - tor -c "/home/tor/Apps/tor/App/tor -f /home/tor/.torrc"

Edit SSH conf so it uses the SOCKS proxy for .onion addresses,

Host *.onion
   ProxyCommand nc -xlocalhost:9050 -X5 %h %p

and add hosts,

Host machine.onion
   HostName dkflcfnvfkddjfkd.onion
   Port 22

Parse S3 Logs with Goaccess

Wed, 20 Nov 2013 00:00:00 +0200

Note to self, create ~/.goaccessrc add S3 log format,

date_format %d/%b/%Y
log_format %^ %^ [%d:%^] %h %^ %^ %^ %^ "%^ %r %^" %s %^ %b %^ %^ %^ "%^" "%u" %^

Sync log bucket to local folder,

s3cmd sync  s3://logs.nakkaya.com/ ./

Parse yesterdays logs,

find . -name "`date +%Y-%m-%d -d "yesterday"`*" -exec cat {} \; | goaccess -a

Parse all logs,

find . -exec cat {} \; | goaccess -a

Notes on Synchronization and Backup of $HOME using git, git-annex and mr

Wed, 23 Oct 2013 00:00:00 +0300

~/annex/
~/source/
- - ~/source/.mrconfig

This is a dump of all git, git-annex and mr related configuration and scripts so I can edit them in one place. When tangled,

All mr configs will go into their respective folders.
All Bash scripts will go into ~/.bin/ folder.

Repository structure,

~/annex/ - git-annex repositories that are synced between computers.
~/source/ - Git repos.
/external - Single git-annex repo spread over multiple USB drives. Working as a Poor Mans Raid.

~/annex/

Instead of having a single large annex folder synced across machines, I have split all files into 6 annexes,

documents
music
notes
old-code
photos

This scheme makes git-annex much faster. At first I was reluctant to go with this, instead of pushing pulling single annex, now I have to deal with 6. Then I found about mr which lets you run commands on a collection of repositories, even though there are 6 repos with mr a single command will push pull all of them.

These annexes are shared between 3 computers (one with two full copy of all repos and two with partial copies.), all behind NAT so all clients dump data to rsync.net (GPG encrypted with separate keys.) and sync changes with a repo inside rsync.net.

When ever I make changes to any of the repos, I just run mr push. It will iterate repositories with changes (new files/deletes/renames) and upload changes to and sync with rsync.net. Then when I switch machines I just do a mr pull which downloads all changes.

~/.mrconfig

include = cat ~/annex/.mrconfig-annex ~/source/.mrconfig-source

~/annex/.mrconfig

[DEFAULT]
git_sync = git annex add .;git annex sync --content
git_status = git annex status
git_gc = git repack -ad; git gc

lib = 
    initAnnex() {
       git config remote.origin.annex-ignore true
       git annex init "`hostname`"
       git annex untrust here
       git annex enableremote cloud
    }
    fastFsChck() {
      git annex fsck --fast --from "$1"
    }

[annex/documents]
checkout = git clone 'https://git.nakkaya.com/git/nakkaya/annex-documents.git' 'documents'
           cd documents/
           initAnnex
skip = lazy
fsck = fastFsChck cloud
       fastFsChck storage-box
       fastFsChck rclone_drive

[annex/notes]
checkout = git clone 'https://git.nakkaya.com/git/nakkaya/annex-notes.git' 'notes'
           cd notes/
           initAnnex
           git annex direct
           git annex get .
fsck = fastFsChck cloud

~/.bin/git-anx-drop-unused

Drop all unused files by date,

#!/usr/bin/env hy

(import  [sh [grep git perl awk ErrorReturnCode]]
	 [re [split]]
	 [datetime [datetime date]]
	 [sys])

(def remote (if (>= (len sys.argv) 2)
	      (second sys.argv)
	      "here"))

(def drop-age (if (= (len sys.argv) 3)
		(int (nth sys.argv 2))
		180))

(defn unused-files []
  (let [[files (try 
		(-> (.annex git "unused" "--from" remote)
		    (perl "-ne" "print if /^    [0-9]+.*/")
		    str)
		(catch [e ErrorReturnCode] ""))]
	[unused-files (->> files 
			   (split "\n")
			   (map (fn [x] 
				  (->> (.strip x)
				       (split " +")
				       (take 2)
				       (map (fn [x] (.strip x))))))
			   (filter (fn [x] 
				     (= (len x) 2)))
			   list)]]
    (print "Unused files: " (len unused-files))
    unused-files))

(defn last-seen [file]
  (let [[key (second file)]]
    (->> (git "--no-pager" "log" "-1" "-S" key "--pretty=format:%at")
	 str
	 (split "\n")
	 (map (fn [x] (.fromtimestamp datetime (float x))))
	 first)))

(defn age [file]
  (let [[delta (- (.today datetime) (last-seen file))]]
    delta.days))

(print "Dropping " remote)

(if (= drop-age 0)
  (for [file (unused-files)]
    (let [[id (first file)]]
      (print "Id " id)
      (if (= remote "here")
	(.annex git "dropunused" "--force" (str id))
	(.annex git "dropunused" "--force" "--from" remote (str id)))))
  (for [file (unused-files)]
    (let [[id (first file)]
	  [file-age (try 
		     (age file)
		     (catch [e Exception] -1))]]

      (if (>= file-age drop-age)
	(do 
	 (print "Id " id " age " file-age " days...")
	 (if (= remote "here")
	   (.annex git "dropunused" "--force" (str id))
	   (.annex git "dropunused" "--force" "--from" remote (str id))))))))

~/.bin/git-fast-push

Custom push command. For repositories with no changes it simply returns true, for repositories with changes or new files,

If acting on a regular git repo, pushes changes to origin.
If acting on a git annex repo, uploads changes and sync with rsync.net.

#/bin/bash

updateAnnexHost() {
    echo 'Updating Remote...'
    ORIGIN=`git config --get remote.origin.url`
    HOST=`echo "$ORIGIN" | grep -oiP '//.*?\/' | cut -d/ -f3`
    DIR="/${ORIGIN#*//*/}"
    echo "$HOST $DIR"
    ssh $HOST "cd $DIR;git annex sync"
}

hasNoChanges(){
    git diff-index --quiet HEAD --
}

hasNewFiles(){
    if [ `git ls-files --exclude-standard --others| wc -l` != 0 ]; then 
        true
    else
        false
    fi
}

isRepoAhead(){
    if [ `git log origin/$(git branch | grep '*' | cut -d' ' -f2)..HEAD | wc -l` != 0 ]; then 
        true
    else
        false
    fi
}

#handle direct annex repo
if `git config --get annex.direct`; then
    oldHead=`git rev-parse HEAD`
    git annex add .
    git annex sync
    newHead=`git rev-parse HEAD`
    if [ "$oldHead" != "$newHead" ]; then
        if git config remote.depot.annex-uuid; then
            git annex copy --to depot --not --in depot
            git annex sync
        else
            git annex copy --to origin --not --in origin
            updateAnnexHost
        fi
    fi
    exit
fi

if ! hasNoChanges || hasNewFiles || isRepoAhead; then 
#handle indirect annex repo
    if [ -d '.git/annex/' ]; then    
        git annex add .
        git annex sync
        if git config remote.depot.annex-uuid; then
            git annex copy --to depot --not --in depot
            git annex sync
        else
            git annex copy --to origin --not --in origin
            updateAnnexHost
        fi
        exit
#handle plain git repo        
    else
        git push origin master
    fi
else
    true
fi

Mount / Unmount EncFS Volumes

Scripts for mounting and unmounting EncFS Volumes.

#/bin/bash

CUR_DIR=`pwd`
cd "$1"
DIR=$(basename "$1")
mkdir "/Volumes/$DIR"
git annex get .
git annex unlock "."
encfs "$CUR_DIR/${1}" "/Volumes/$DIR"
cd $CUR_DIR

#/bin/bash

CUR_DIR=`pwd`
DIR=$(basename "$1")
if umount "/Volumes/$DIR"; then
    rm -rf "/Volumes/$DIR"
fi
cd "$1"
git annex add .
git annex add .encfs6.xml
git commit -m 'Update'
cd $CUR_DIR

Webapp

Create autostart file relative paths don't work so tangle one file for each OS (Linux,OS X) then mv one to correct location,

/home/nakkaya/annex/notes
/home/nakkaya/annex/documents

/Users/nakkaya/annex/notes
/Users/nakkaya/annex/documents

Start asistant and webapp,

git annex assistant --autostart && nohup git annex webapp

Misc

Setup encrypted annex directory remote,

git annex initremote mobile type=directory directory=/path/to/annex/repo/ encryption=hybrid keyid=ID embedcreds=yes

Setup encrypted annex S3 remote in EU (Ireland) (eu-west-1),

export AWS_ACCESS_KEY_ID="KID"
export AWS_SECRET_ACCESS_KEY="SKEY"
git annex initremote cloud type=S3 encryption=hybrid keyid=ID embedcreds=yes datacenter=eu-west-1 chunk=250MiB
git setup-bitbucket
git config remote.origin.annex-ignore true

Setup encrypted annex S3 remote in Google Cloud Storage,

git annex initremote cloud type=S3 encryption=hybrid keyid=ID embedcreds=yes host=storage.googleapis.com port=80 chunk=250MiB

Setup encrypted annex rsync remote,

git annex initremote depot type=rsync encryption=hybrid rsyncurl=rsync:annex/repo/ keyid=ID

Setup encrypted annex rclone gdrive remote,

git annex initremote rclone_drive type=external externaltype=rclone target=drive_robotics prefix=git-annex chunk=100MiB encryption=shared mac=HMACSHA512 rclone_layout=lower

~/source/

~/source/.mrconfig

Git Repos,

[DEFAULT]
git_pull = git pull origin master
git_push = git fast-push
git_status = git status --short
sync = git pull && git push

[source/latte]
checkout = git clone 'https://git.nakkaya.com/git/nakkaya/latte.git' 'latte'
skip=lazy

[source/alter-ego]
checkout = git clone 'git@github.com:nakkaya/alter-ego.git' 'alter-ego'
skip=lazy

[source/ardrone]
checkout = git clone 'git@github.com:nakkaya/ardrone.git' 'ardrone'
skip=lazy

[source/clodiuno]
checkout = git clone 'git@github.com:nakkaya/clodiuno.git' 'clodiuno'
skip=lazy

[source/easy-dns]
checkout = git clone 'git@github.com:nakkaya/easy-dns.git' 'easy-dns'
skip=lazy

[source/emacs]
checkout = git clone 'git@github.com:nakkaya/emacs.git' 'emacs'
           cd emacs
           git submodule init
           git submodule update

[source/inbox-feed]
checkout = git clone 'git@github.com:nakkaya/inbox-feed.git' 'inbox-feed'
skip=lazy

[source/nakkaya.com]
checkout = git clone 'git@github.com:nakkaya/nakkaya.com.git' 'nakkaya.com'
skip=lazy

[source/net-eval]
checkout = git clone 'git@github.com:nakkaya/net-eval.git' 'net-eval'
skip=lazy

[source/neu-islanders]
checkout = git clone 'https://git.nakkaya.com/git/nakkaya/neu-islanders.git' 'neu-islanders'
skip=lazy

[source/static]
checkout = git clone 'git@github.com:nakkaya/static.git' 'static'
skip=lazy

[source/vector-2d]
checkout = git clone 'git@github.com:nakkaya/vector-2d.git' 'vector-2d'
skip=lazy

[source/doganilic.com]
checkout = git clone 'https://git.nakkaya.com/git/nakkaya/doganilic.com.git' 'doganilic.com'
skip=lazy

[source/ansible-docker-build]
checkout = git clone 'https://git.nakkaya.com/git/nakkaya/ansible-docker-build.git' 'ansible-docker-build'
skip=lazy

[source/ansible-storage]
checkout = git clone 'https://git.nakkaya.com/git/nakkaya/ansible-storage.git' 'ansible-storage'
skip=lazy

[source/ansible-base]
checkout = git clone 'https://git.nakkaya.com/git/nakkaya/ansible-base.git' 'ansible-base'
skip=lazy

[source/ansible-backup]
checkout = git clone 'https://git.nakkaya.com/git/nakkaya/ansible-backup.git' 'ansible-backup'
skip=lazy

[source/control-toolbox]
checkout = git clone 'https://git.nakkaya.com/git/nakkaya/control-toolbox.git' 'control-toolbox'
skip=lazy

[source/solarcar-tracker]
checkout = git clone 'https://git.nakkaya.com/git/nakkaya/solarcar-tracker.git' 'solarcar-tracker'
skip=lazy

[source/solarcar-turn-indicator]
checkout = git clone 'https://git.nakkaya.com/git/nakkaya/solarcar-turn-indicator.git' 'solarcar-turn-indicator'
skip=lazy

[source/ferret]
checkout = git clone 'https://git.nakkaya.com/git/nakkaya/ferret.git' 'ferret'
skip=lazy

Git SSH Verbose

Mon, 26 Aug 2013 00:00:00 +0300

Note to self, debugging git ssh interaction,

GIT_TRACE=2 git <command>