Faster Meta-development with Clojure Boot
Daniel Compton recently performed an analysis of the State of Clojure 2016 free text comment portion. A section of his analysis is devoted to complaints about startup time, which I frequently see elsewhere. And, in my experience, it's true: Clojure, when started with Boot, takes much longer to start than other comparable dynamic language build and dependency management tools.
For example, starting Ruby's
irb console with Bundler, using the command
touch Gemfile && time bundle exec ruby -e 'exit 0', takes 0.4 seconds on my
time boot -B repl -e '(System/exit 0)' takes 6.9 seconds.
You might ask, "How do experienced Clojure programmers abide this? Do they take
7-second hammock breaks after
each edit to their
The answer is: no, 7 seconds after every
build.boot edit would suck majorly.
Before we continue, it's worth noting that by itself, Clojure actually starts
pretty quickly. The command
time java -jar clojure-1.8.0.jar -e '(System/exit
0)' runs in less than a second for me. For more on Clojure's
check out this thread on the Clojure mailing list.
Meta-development is what I call that development effort around the mechanics and logistics of the project itself: obtaining dependencies, building, deploying, testing. Any software anyone will pay you to build will involve some amount of meta-development.
For example, in the beginning hours and days of development on a project, much of my time is spent finding, arranging, and experimenting with dependencies. This is meta-development.
Meta-development will continue as long as the project lives. Dependencies will need to be updated and build/deploy procedures will need to be tweaked.
Waiting for Clojure to start after every edit to
build.boot would suck. I
don't have to restart the JVM when I'm working on my project; why should I pay
the startup tax when I'm working on my "project's project"?
To avoid the tax, during meta-development I start
boot once, and for as long
as I can, I live in the REPL.
Boot: meta-development paradise
Instead of trying to start Clojure faster, try to stay in the REPL longer.
For me, this attitude was the key to long-term happiness with the development environment that Clojure and the JVM constitute. It's also the attitude that Micha and I had when we created Boot.
boot to facilitate marathon REPL sessions by ensuring there's almost
nothing you can do at the command line that you can't do at a Clojure REPL.
Let's look at a few command-line oriented usage patterns, and port them to the REPL so that you can live in it too.
1. Load dependencies
At the command line you can start a boot REPL with a dependency by typing something like this:
$ boot -d com.acme/foo:1.3.3 repl boot.user=> << play with com.acme/foo >>
build.boot you can bring in dependencies with a call to
boot.core/set-env!, like this:
(set-env! :dependencies '[[com.acme/foo "1.3.3"]])
A good thing to know is that anything you can write in a
build.boot, you can
run in a REPL. So, you can start a Clojure REPL, even in a directory without a
build.boot, and bring in the
com.acme/foo dependency like this:
$ boot repl boot.user=> (set-env! :dependencies '[[com.acme/foo "1.3.3"]]) << play with com.acme/foo >>
2. Run tasks
Here is how you might run a bunch of tasks at the command-line in a Boot project:
$ boot watch foo bar target
Alternatively, you could run the tasks at a Boot REPL like this:
$ boot repl boot.user=> (boot (watch) (foo) (bar) (target))
In the above example,
boot is the function
boot.core/boot. It's the REPL
equivalent to typing
boot in a shell. You can learn more about it by running
(doc boot). In fact, you can learn more about any function at the REPL with
doc. For example, you can learn about
Ctrl-c at the REPL kills the
boot call and takes you back to a
3. Reload build.boot
In an established project, if you want to make a series of edits to your
build.boot, you'd have to do something like this to meta-develop without a
<< edit and save build.boot in editor >> $ boot my-task << wait 7 seconds >> << observe it doesn't work, edit and save build.boot >> $ boot my-task << wait 7 seconds >> << observe it doesn't work, edit and save build.boot >> ...
This is miserable. Here's a better way, using
clojure.core/load-file to reload
build.boot instead of bouncing the JVM:
<< edit and save build.boot in editor >> $ boot repl << wait 7 seconds >> boot.user=> (boot (my-task)) << observe it doesn't work, edit and save build.boot >> boot.user=> (load-file "build.boot") << wait 0.03 seconds >> << observe it doesn't work, edit and save build.boot >> boot.user=> (load-file "build.boot") << wait 0.03 seconds >> << observe it doesn't work, edit and save build.boot >> ...
Much better! It's still kind of cumbersome to have to go to the REPL and
load-file, though. Let's automate that part.
4. Reload build.boot automatically
Add this function to your
(defn poll [task] (let [f (java.io.File. "build.boot")] (loop [mtime (.lastModified f)] (let [new-mtime (.lastModified f)] (when (> new-mtime mtime) (load-file "build.boot") (boot (task))) (Thread/sleep 1000) (recur new-mtime)))))
It's a simple poll-and-reload loop that checks
build.boot for a newer
modification time, and conditionally reloads the file and runs
task is a provided argument.
$ boot repl boot.user=> (poll my-task) << observe it doesn't work, edit and save build.boot >> << observe it doesn't work, edit and save build.boot >> << observe it doesn't work, edit and save build.boot >> ...
Now you can make edits to
build.boot and the
(boot (my-task)) gets run every
time you save the file.
Caveats and conclusion
Living in the REPL isn't without downsides. The biggest downside is probably
that, living in the REPL, it's easy to accumulate bits of state that can lead to
weird errors and bugs. Using multiple sequential
set-env! calls to add
dependencies, for instance, can lead to an irreproducible classpath.
The remedy is usually to restart and test before committing changes to source control or deploying, to make sure everything is in an OK place.
Another downside to the Clojure/JVM REPL-oriented approach is that it's not suitable for actual command-line programs that need to start quickly. While boot has great scripting support, the scripts take a long time to start. ClojureScript-based environments like Planck and Lumo are a few efforts I know of to address this use-case.
Anyway, in my opinion, the benefits of meta-development in a Boot/Clojure REPL outweigh the downsides, at least for projects that aren't command-line tools. Viva la REPL!