Faster Meta-development with Boot

Daniel Compton recently performed an analysis of the State of Clojure 2016 free text comment portion. A section of his analysis is devoted to complaints about startup time, which I frequently see elsewhere. And, in my experience, it's true: Clojure, when started with Boot, takes much longer to start than other comparable dynamic language build and dependency management tools.

For example, starting Ruby's irb console with Bundler, using the command touch Gemfile && time bundle exec ruby -e 'exit 0', takes 0.4 seconds on my system. Running time boot -B repl -e '(System/exit 0)' takes 6.9 seconds. Yikes!

You might ask, "How do experienced Clojure programmers abide this? Do they take 7-second hammock breaks after each edit to their build.boot?"

The answer is: no, 7 seconds after every build.boot edit would suck majorly.

Before we continue, it's worth noting that by itself, Clojure actually starts pretty quickly. The command time java -jar clojure-1.8.0.jar -e '(System/exit 0)' runs in less than a second for me. For more on Clojure's startup, check out this thread on the Clojure mailing list.

Meta-development

Meta-development is what I call that development effort around the mechanics and logistics of the project itself: obtaining dependencies, building, deploying, testing. Any software anyone will pay you to build will involve some amount of meta-development.

For example, in the beginning hours and days of development on a project, much of my time is spent finding, arranging, and experimenting with dependencies. This is meta-development.

Meta-development will continue as long as the project lives. Dependencies will need to be updated and build/deploy procedures will need to be tweaked.

Waiting for Clojure to start after every edit to build.boot would suck. I don't have to restart the JVM when I'm working on my project; why should I pay the startup tax when I'm working on my "project's project"?

To avoid the tax, during meta-development I start boot once, and for as long as I can, I live in the REPL.

Boot: meta-development paradise

Instead of trying to start Clojure faster, try to stay in the REPL longer.

For me, this attitude was the key to long-term happiness with the development environment that Clojure and the JVM constitute. It's also the attitude that Micha and I had when we created Boot.

We built boot to facilitate marathon REPL sessions by ensuring there's almost nothing you can do at the command line that you can't do at a Clojure REPL.

Let's look at a few command-line oriented usage patterns, and port them to the REPL so that you can live in it too.

1. Load dependencies

At the command line you can start a boot REPL with a dependency by typing something like this:

$ boot -d com.acme/foo:1.3.3 repl
boot.user=>
<< play with com.acme/foo >>

In your build.boot you can bring in dependencies with a call to boot.core/set-env!, like this:

(set-env! :dependencies '[[com.acme/foo "1.3.3"]])

A good thing to know is that anything you can write in a build.boot, you can run in a REPL. So, you can start a Clojure REPL, even in a directory without a build.boot, and bring in the com.acme/foo dependency like this:

$ boot repl
boot.user=> (set-env! :dependencies '[[com.acme/foo "1.3.3"]])
<< play with com.acme/foo >>

2. Run tasks

Here is how you might run a bunch of tasks at the command-line in a Boot project:

$ boot watch foo bar target

Alternatively, you could run the tasks at a Boot REPL like this:

$ boot repl
boot.user=> (boot (watch) (foo) (bar) (target))

In the above example, boot is the function boot.core/boot. It's the REPL equivalent to typing boot in a shell. You can learn more about it by running (doc boot). In fact, you can learn more about any function at the REPL with doc. For example, you can learn about watch with (doc watch).

Pressing Ctrl-c at the REPL kills the boot call and takes you back to a Clojure prompt.

3. Reload build.boot

In an established project, if you want to make a series of edits to your build.boot, you'd have to do something like this to meta-develop without a REPL:

<< edit and save build.boot in editor >>
$ boot my-task
<< wait 7 seconds >> 
<< observe it doesn't work, edit and save build.boot >>
$ boot my-task
<< wait 7 seconds >> 
<< observe it doesn't work, edit and save build.boot >>
...

This is miserable. Here's a better way, using clojure.core/load-file to reload your build.boot instead of bouncing the JVM:

<< edit and save build.boot in editor >>
$ boot repl
<< wait 7 seconds >>
boot.user=> (boot (my-task))
<< observe it doesn't work, edit and save build.boot >>
boot.user=> (load-file "build.boot")
<< wait 0.03 seconds >> 
<< observe it doesn't work, edit and save build.boot >>
boot.user=> (load-file "build.boot")
<< wait 0.03 seconds >> 
<< observe it doesn't work, edit and save build.boot >>
...

Much better! It's still kind of cumbersome to have to go to the REPL and manually run load-file, though. Let's automate that part.

4. Reload build.boot automatically

Add this function to your build.boot:

(defn poll [task]
  (let [f (java.io.File. "build.boot")]
    (loop [mtime (.lastModified f)]
      (let [new-mtime (.lastModified f)]
        (when (> new-mtime mtime)
          (load-file "build.boot")
          (boot (task)))
        (Thread/sleep 1000)
        (recur new-mtime)))))

It's a simple poll-and-reload loop that checks build.boot for a newer modification time, and conditionally reloads the file and runs (boot (task)), where task is a provided argument.

Then:

$ boot repl
boot.user=> (poll my-task)
<< observe it doesn't work, edit and save build.boot >>
<< observe it doesn't work, edit and save build.boot >>
<< observe it doesn't work, edit and save build.boot >>
...

Now you can make edits to build.boot and the (boot (my-task)) gets run every time you save the file.

Caveats and conclusion

Living in the REPL isn't without downsides. The biggest downside is probably that, living in the REPL, it's easy to accumulate bits of state that can lead to weird errors and bugs. Using multiple sequential set-env! calls to add dependencies, for instance, can lead to an irreproducible classpath.

The remedy is usually to restart and test before committing changes to source control or deploying, to make sure everything is in an OK place.

Another downside to the Clojure/JVM REPL-oriented approach is that it's not suitable for actual command-line programs that need to start quickly. While boot has great scripting support, the scripts take a long time to start. ClojureScript-based environments like Planck and Lumo are a few efforts I know of to address this use-case.

Anyway, in my opinion, the benefits of meta-development in a Boot/Clojure REPL outweigh the downsides, at least for projects that aren't command-line tools. Viva la REPL!

Like the article?

Get notified of future blog posts. Don't worry - we won't make it hard to get to inbox zero: no more than 2 e-mails a month. We promise.

Use header bidding for your site? Learn more about Adzerk's new sub-brand, ServerBid, the first and only completely independent server-side header bidding platform.