Dipping my toes into Lisp – Getting Common Lisp with FastCGI support working with Apache

I haven’t to date posted anything tech related to the blog, not because I haven’t wanted to but because mainly there are people out there better than I who have already written a great deal about the topics I’m interested in. So here’s my first post, and I’m going to write a little about my latest venture doing some exploration with Lisp.

Lisp is a language that has intrigued me for a while. How did I come across it you might ask? Mainly through the influence of two individuals who have had a fairly large impact in my profession.

I’m an avid reader and long time lurker of Hacker News (HN – for short), it’s definitely my number one go to source for current tech industry information. I know I’m sure to find some interesting tidbits here and there and spend at least a few hours per week reading articles I find linked.

Paul Graham is one of my influences, who kicked off Hacker News and is quite well known in the industry, you might have heard of his YCombinator startup incubator initiative, is an advocate of Lisp and someone that I admire and respect as a professional. He even went so far as to work on his own dialect Arc which is the language used to power Hacker News. Paul’s early success came about with his acquisition of Viaweb by Yahoo! (which then became Yahoo! store) and initially the product was brought to market with Common Lisp as the language used for implementation. There’s more information and great articles on Paul’s website for those keen to find out more about Paul (http://www.paulgraham.com/).

Another individual that I follow regularly online and I would consider a second influence is Andy Gavin who’s well known for being one of the Co-founders of Naughty Dog software. He’s probably best known for his part in the creation of Crash Bandicoot, a game that was released for the original Sony PlayStation, of which I spent some time in my youth playing (sadly I didn’t own it myself, I used to play it at a friends – but still have many fond memories). Many of my colleagues I work with are somewhat surprised to realise that Crash Bandicoot was developed primarily with Lisp along with some of the later games (the sequel’s of Crash Bandicoot and Jak & Daxter), although Andy has started to focus more on Ruby according to his blog citing mainly that Ruby has more momentum and libraries are richer. You can find out more about Andy at his website (http://all-things-andy-gavin.com/).

Regarding my own reasons for looking into Lisp, primarily it’s to do with wanting to try something new, get out of my comfort zone a little and really form my own opinion. I’ve done a fair amount of work with Python in the past (and recently), and a smattering of Ruby, so for those that ask why I don’t look into those – I already have – I’m just wanting to have a taste of Lisp (and so far I’m enjoying it). I don’t tend to hold any religious views when it comes to languages and I believe it’s better to keep an open mind and not be afraid to experiment. I believe that those people who become too dogmatic and entrenched in their use of particular languages are missing out on the things you can learn from a broader exposure to different languages. It not only helps to keep your wits sharp, but your mind curious and hungry to find out more.

Without further ado, as I’ve rambled for long enough, I’m going to detail how I got Common Lisp (http://www.clisp.org) up and running on my Linode VPS behind Apache. I did a fair amount of Googling online looking for some simple instructions or some useful tips and to be honest I didn’t find much at all. I already knew how to get CLisp running easily in my terminal, on Ubuntu it’s as easy as doing ‘apt-get install clisp’ then running ‘clisp’ from the terminal to bring up the REPL (Read-Eval-Print-Loop).

One thing I found was that Common Lisp has a module for FastCGI support (http://www.clisp.org/impnotes/fastcgi.html) so I wondered why there didn’t seem to be any articles or posts written up as a tutorial on how to get that directly hooked up to Apache (or any other webserver that has a FastCGI module or supports it natively e.g. Nginx, Lighttpd etc) maybe there wasn’t any interest in doing it. Like I said, I was just curious to see if it could be done and wanted to know if I could do it that way without having to rely on a separate Lisp based webserver. I can understand why most people use a Lisp webserver, generally it sits behind a client facing webserver such as Apache or Nginx and is reverse proxied so that the front facing server handles static content etc and the Lisp backend server handles the dynamic processing. I just want to make it clear I understand the reasoning, but for my simple case I just wanted to hook it up directly so I could bash out some simple scripts and allow them to be demonstrated by hitting a web page.

So, as I’ve mentioned it’s easy enough to get Common Lisp installed, but the FastCGI support isn’t available as a Debian/Ubuntu package which means we have to build it from source (the other base modules are all available as packages you can simply apt-get e.g. ‘apt-get install clisp-module-berkeley-db’ etc).

Initially I tried to pull the Common Lisp mercurial repo directly and compile, but soon encountered a lot of difficulty with not having the right packages etc to build it – so I decided to just go ahead and leverage the Debian/Ubuntu package source (cheating I know).

Here are the steps I took in point form (you might want to do these in a temp directory?) to get Common Lisp built with the FastCGI module:

1. Pulled down the build dependencies with ‘sudo apt-get build-dep clisp’, this will grab all build dependencies you’ll need to compile the clisp source.
2. Grabbed the source with ‘apt-get source clisp’, this pulled down the source files, when it’s complete you should end up with a ‘clisp-2.49’ directory (version might differ), move into that directory with ‘cd clisp-2.49’.
3. I then ran the configure script like so:

   ./configure --with-module=berkeley-db --with-module=bindings/glibc --with-module=clx/new-clx --with-module=dbus --with-module=gdbm --with-module=gtk2 --with-module=i18n --with-module=pcre --with-module=postgresql --with-module=rawsock --with-module=readline --with-module=regexp --with-module=syscalls --with-module=wildcard --with-module=zlib --with-module=fastcgi

   (note the fastcgi module here? That’s important or you won’t get fastcgi support)

4. For a lot of the modules being installed, you’ll also need to have the relevant development packages and shared libraries installed so that a successful build takes place e.g. ‘sudo apt-get install libglade2-dev’ etc I’ll leave that as an exercise for the reader (I just built all the modules – you could instead choose not to and leave some out).
5. Once that had completed, I ran ‘cd src’ to shift into the src directory and ran ‘make’, followed by ‘make check’ as the directions tell you at the end of the configure script execution.
6. The make and subsequent check will take a little time to run, but when it’s finished you should be right to go ahead and run ‘sudo make install’ which will then install your binaries and the modules.
7. Once it’s installed, just check ‘which clisp’ to see whereabouts it is running from, on my system it installed to /usr/local/bin/clisp so I’d expect you’d see the same (you could change the install location by passing the ‘–prefix=…’ option of course – the configure script has plenty of options ‘./configure –help’)
8. Fire up Lisp with ‘clisp’, then once your in the REPL, enter ‘*FEATURES*’, you’ll notice that ‘:FASTCGI’ and some of the other modules won’t be present, that’s ok.
9. Now enter ‘(require “fastcgi”)’ in the REPL, it should display a list showing supported features, you should now see ‘:FASTCGI’ listed which means you’re good to go! (Now some people will say, “why not run clisp with ‘clisp -q -K full’?”, well the issue is on Ubuntu it doesn’t create the /usr/local/lib/clisp-2.49/full directory that should link up all the modules and provide them already – and I didn’t do any more fiddling to figure it out since adding ‘(require “fastcgi”)’ at the start of my Lisp file didn’t seem too onerous.
10. Fantastic, your Common Lisp install is good to go and supports FastCGI

Now, moving onto Apache, I’m going to assume most people know how to do a basic Apache installation and to get a site up and running if you don’t there are plenty of resources on the web detailing how to get it done. There are also steps on how to get FastCGI hooked up (if I get enough requests to detail it I’ll add it on – but I’m lazy and like I said with 5 minutes of searching you’ll have plenty of answers).

All I added to my Apache site configuration was a single line ‘AddHandler fcgid-script .lisp’ to my <Directory/> section detailing the document root. Any Lisp script you drop in there with the ‘.lisp’ extension will be run as the FastCGI program. I also added ‘DirectoryIndex index.lisp index.php index.html’ so that hitting the base url will fire up the index.lisp script by default.

Here’s the test script I use, placed in ‘index.lisp’, which is slightly modified to the example already provided at http://www.clisp.org/impnotes/fastcgi.html

#!/var/www/fastcgi/pstreet/clisp -q -K base

(require "fastcgi")

(defun main ()
    (do*
        ((counter 1 (1+ counter)))
        ((not (fastcgi:accept)))
        (fastcgi:out "Content-type: text/plain" #\Newline #\Newline)
        (fastcgi:out
            "CLisp running with FastCGI support via Apache web server" #\Newline #\Newline
            "I am running in mode: " (if (fastcgi:is-cgi) "CGI" "FastCGI") #\Newline
            "This is execution no.: " counter " (restarts after 10 connections)" #\Newline
            "The browser string is '" (fastcgi:getenv "HTTP_USER_AGENT") "'" #\Newline #\Newline
            "- CGI Environment variables are -" #\Newline)
        (apply #'fastcgi:out (mapcar #'(lambda (arg) (list (car arg) ": " (cdr arg) ", ")) (fastcgi:getenv)))
        (fastcgi:out
            #\Newline #\Newline "- Host Environment variables are -" #\Newline)
        (apply #'fastcgi:out (mapcar #'(lambda (arg) (list (car arg) ": " (cdr arg) ", ")) (ext:getenv)))
        (cond ((>= counter 10) (quit)))
        (fastcgi:finish)))
(main)

You can see the example here http://lisptests.phillipstreet.com/ this may change in the future (at which point I’ll try to remember to update this link). I am in no way a Lisp expert, this is actually the first piece of Lisp I’ve done any work with. I’m planning to experiment as I go forward and see how things work etc. So if you’re someone familiar with lisp, please feel free to pass along any tips or suggestions.

UPDATE:

I’ve recently realised that it’s possible to take out the boiler plate fastcgi lisp code in the above code and place it in a separate file that will be used to load and run lisp files when they’re targeted via the FcgidWrapper in the apache config for a site.

Here’s the wrapper script ‘lisp-cgi.cl’ I use in apache with ‘FcgidWrapper /var/www/fastcgi/pstreet/lisp-cgi.cl .lisp’ (I also add another line and provide ‘.lsp’ as another suffix).

Visit http://lisptests.phillipstreet.com/ to see the scripts in action.

#!/bin/bash

ulimit -s 16384

read -r -d '' LISPCODE << 'EOF'
(require "fastcgi")

(defun main ()
	(do
		()
		((not (fastcgi:accept)) nil)
		(load (fastcgi:getenv "SCRIPT_FILENAME") :COMPILING)
		(fastcgi:finish)))
(main)
EOF

echo $LISPCODE | /var/www/fastcgi/pstreet/clisp -m 50MB -norc -C -q -K base