A build script for Common Lisp web applications

When deploying a Common Lisp web application, it is often useful to have a build script that takes care of generating an executable of the application. Such a build script can also be helpful for others as it documents the steps that are necessary to get the app running on the REPL. It should take care of the following things:

  1. Load the web app and its dependencies.
  2. Define an entry point that starts the web app
  3. (Optional) Set environment variables.
  4. (Optional) Start a Swank server.
  5. Dump an executable.

Setting environment variables is especially useful if you are using something like envy for configuration, since it allows you to use a development configuration by default (i.e., on the REPL) and a production configuration with the build script (i.e., on the server).

A Swank server is usually started by SLIME when you start your local REPL. Using it on the server allows you to remote-control your app with SLIME via SSH.

Load the web app

While how you load your app is totally up to you, the canonical way is using quicklisp. If you can’t or don’t want to use quicklisp on your server, you can still use ASDF. In this case, however, you will have to deploy all dependencies by yourself, since ASDF won’t download them for you.

First, make sure that ASDF is loaded (just in case):

#-asdf (require 'asdf)

Next, we have load our web app (which is assumed to be ASDF-loadable) and some additional librarys. If quicklisp is installed, we will use it, otherwise, we will use ASDF. Therefore, we define a list of systems that should be loaded and then load the whole list at once.

(let ((load-list '("my-fancy-web-app"
                   "trivial-dump-core"
                   "bordeaux-threads"
                   ;; ... some other stuff will come here
                   )))
  #+quicklisp (mapcar #'ql:quickload load-list)
  #-quicklisp (mapcar #'asdf:load-system load-list))

Define an entry point

To keep the cl-user package clean, we start by defining a new package:

(defpackage #:my-app.executable
  (:use #:cl))

(in-package #:my-app.executable)

The actual entry point is a function with no arguments like this:

(defun run ()
  (my-app:start-server)
  ;; and what ever else is necessary to start your server
  )

This assumes, that (my-app:start-server) returns immediately after starting the server, i.e. spawns a new thread, so that the rest of the startup routine can continue. Therefore, we have to ensure that the program waits for the server to stop before exiting, otherwise the program would exit immediately after starting the server. We use bordeaux-threads to wait for all threads that have been started during the startup routine:

(defun run ()
  ...
  (mapcar #'bt:join-thread (bt:all-threads)))

Now our program waits until the server ist stopped. To manually stop the program, we could hit C-c on the command-line, but this usually drops us to the debugger instead of stopping the program. Unfortunately, there is no platform independent solution to this, but I will show here how to solve this at least for SBCL:

(handler-case (mapcar #'bt:join-thread (bt:all-threads))
  #+sbcl(sb-sys:interactive-interrupt () (sb-ext:exit)))

SBCL raises a condition of type sb-sys:interactive-interrupt, so we just catch it with handler-case and exit. Other implementations like Clozure CL don’t use a condition for C-c, so this method does not work there.

Set environment variables

If you are using envy, you will want to use a different configuration for a production environment that for a local environment. I usually set the local configuration as default and change this for the production environment, so I want to set the corresponding environment variable to “production” in the build script.

Using osicat, we can set an environment variable like this:

(setf (osicat:environment-variable "MY_APP_ENV") "production")

Note, that we might have to do this twice, once before loading the app and once before starting it, because this might happen in seperate contexts (compile time and execution time). So, we have to load osicat and set the variable at the very beginning before loading the rest and at the beginning of the run function before the server startup.

Start a Swank server

This is actually very cool since it allows you to spawn a REPL on your local system for remote controlling a running web app. The idea ist to spawn a Swank server (the piece of software that SLIME uses to communicate with a lisp process) and to connect to it from your local SLIME via SSH.

Swank is in available seperately on quicklisp, so we just add "swank" to our load-list. The Swank server can now be started with swank:create-server, which takes some keyword arguments. We will use :port to configure the port that is used on the server and :dont-close so the Swank server does not shut down after disconnecting and accepts subsequent connections.

In this example, the port number is given by (in descending order of precedence):

  1. an environment variable (so you can override the configuration)
  2. the app’s configuration
  3. the standard port (4005)
(let ((env-port (osicat:environment-variable "MY_APP_SWANK_PORT")))
  (swank:create-server :port (or (and env-port
                                      (parse-integer env-port))
                                 (my-app.config:config :swank-port)
                                 swank::default-server-port)
                       :dont-close t))

Dump an executable

Finally, we will dump an executable using trivial-dump-core:

(trivial-dump-core:save-executable "my-app" #'run)

The final script

Adding some messages for the user, we get something like this:

;;;; build.lisp
#-asdf (require 'asdf)
#+quicklisp (ql:quickload "osicat")
#-quicklisp (asdf:load-system "osicat")
(setf (osicat:environment-variable "MY_APP_ENV") "production")
(let ((load-list '("my-fancy-web-app"
                   "trivial-dump-core"
                   "bordeaux-threads"
                   "swank")))
  #+quicklisp (mapcar #'ql:quickload load-list)
  #-quicklisp (mapcar #'asdf:load-system load-list))

(defpackage #:my-app.executable
  (:use #:cl))

(in-package #:my-app.executable)

(defun run ()
  (setf (osicat:environment-variable "MY_APP_ENV") "production")
  ;; ... startup code
  (my-app:start-server)
  ;; ... more startup code
  (let ((env-port (osicat:environment-variable "MY_APP_SWANK_PORT")))
    (swank:create-server :port (or (and env-port
                                        (parse-integer env-port))
                                   (my-app.config:config :swank-port)
                                   swank::default-server-port)
                         :dont-close t))
  (write-line "Server is running. To quit, hit <C-c>.")
  (handler-case (mapcar #'bt:join-thread (bt:all-threads))
    #+sbcl(sb-sys:interactive-interrupt ()
            (write-line "Bye!")
            (sb-ext:exit))))

(trivial-dump-core:save-executable "my-app" #'run)

Want to comment on this article? Write me an email (GPG key: 0x12B9620D).