Introduction
Welcome reader! This is a book about scripting with Clojure and babashka.
Clojure is a functional, dynamic programming language
from the Lisp family which runs on the JVM. Babashka is a scripting environment
made with Clojure, compiled to native with GraalVM. The
primary benefits of using babashka for scripting compared to the JVM are fast
startup time and low memory consumption. Babashka comes with batteries included
and packs libraries like clojure.tools.cli
for parsing command line arguments
and cheshire
for working with JSON. Moreover, it can be installed just by
downloading a self-contained binary.
Target audience
Babashka is written for developers who are familiar with Clojure on the JVM. This book assumes familiarity with Clojure and is not a Clojure tutorial. If you aren’t that familiar with Clojure but you’re curious to learn, check out this list of beginner resources.
Setting expectations
Babashka uses sci for interpreting Clojure. Sci implements a substantial subset of Clojure. Interpreting code is in general not as performant as executing compiled code. If your script takes more than a few seconds to run or has lots of loops, Clojure on the JVM may be a better fit, as the performance on JVM is going to outweigh its startup time penalty. Read more about the differences with Clojure here.
Getting started
Installation
Installing babashka is as simple as downloading the binary for your platform and
placing it on your path. Pre-built binaries are provided on the
releases page of babashka’s
Github repo. Babashka is also available in
various package managers like brew
for macOS and linux and scoop
for
Windows. See here for
details.
Building from source
If you would rather build babashka from source, download a copy of GraalVM and
set the GRAALVM_HOME
environment variable. Also make sure you have
lein installed. Then run:
$ git clone https://github.com/borkdude/babashka --recursive
$ script/uberjar && script/compile
See the babashka build.md page for details.
Running babashka
The babashka executable is called bb
. You can either provide it with a Clojure
expression directly:
$ bb -e '(+ 1 2 3)'
6
or run a script:
(+ 1 2 3)
$ bb -f script.clj
6
The -e
flag is optional when the argument starts with a paren. In that case babashka will treat it automatically as an expression:
$ bb '(+ 1 2 3)'
6
Similarly, the -f
flag is optional when the argument is a filename:
$ bb script.clj
6
Commonly, scripts have shebangs so you can invoke them with their filename only:
#!/usr/bin/env bb
(+ 1 2 3)
Usage
Typing bb help
from the command line will print all the available command
line options which should give you a sense of the available features in
babashka.
Babashka v0.3.1
Usage: bb [classpath opts] [eval opts] [cmdline args]
or: bb [classpath opts] file [cmdline args]
or: bb [classpath opts] subcommand [subcommand opts] [cmdline args]
Classpath:
-cp, --classpath Classpath to use. Overrides bb.edn classpath.
Evaluation:
-e, --eval <expr> Evaluate an expression.
-f, --file <path> Evaluate a file.
-m, --main <ns|var> Call the -main function from a namespace or call a fully qualified var.
--verbose Print debug information and entire stacktrace in case of exception.
Help:
help, -h or -? Print this help text.
version Print the current version of babashka.
describe Print an EDN map with information about this version of babashka.
doc <var|ns> Print docstring of var or namespace. Requires namespace if necessary.
REPL:
repl Start REPL. Use rlwrap for history.
socket-repl [addr] Start a socket REPL. Addr opt defaults to localhost:1666.
nrepl-server [addr] Start nREPL server. Address option defaults to locahost:1667.
Clojure:
clojure [args...] Invokes clojure. Takes same args as the official clojure CLI.
Packaging:
uberscript <file> [eval-opt] Collect all required namespaces from the classpath into a single file. Accepts additional eval opts, like `-m`.
uberjar <jar> [eval-opt] Similar to uberscript but creates jar file.
In- and output flags (only to be used with -e one-liners):
-i Bind *input* to a lazy seq of lines from stdin.
-I Bind *input* to a lazy seq of EDN values from stdin.
-o Write lines to stdout.
-O Write EDN values to stdout.
--stream Stream over lines or EDN values from stdin. Combined with -i or -I *input* becomes a single value per iteration.
File names take precedence over subcommand names.
Remaining arguments are bound to *command-line-args*.
Use -- to separate script command line args from bb command line args.
When no eval opts or subcommand is provided, the implicit subcommand is repl.
Running a script
Scripts may be executed from a file using -f
or --file
:
bb -f download_html.clj
The file may also be passed directly, without -f
:
bb download_html.clj
Using bb
with a shebang also works:
#!/usr/bin/env bb
(require '[babashka.curl :as curl])
(defn get-url [url]
(println "Downloading url:" url)
(curl/get url))
(defn write-html [file html]
(println "Writing file:" file)
(spit file html))
(let [[url file] *command-line-args*]
(when (or (empty? url) (empty? file))
(println "Usage: <url> <file>")
(System/exit 1))
(write-html file (:body (get-url url))))
$ ./download_html.clj
Usage: <url> <file>
$ ./download_html.clj https://www.clojure.org /tmp/clojure.org.html
Downloading url: https://www.clojure.org
Writing file: /tmp/clojure.org.html
If /usr/bin/env
doesn’t work for you, you can use the following
workaround:
$ cat script.clj
#!/bin/sh
#_(
"exec" "bb" "$0" hello "$@"
)
(prn *command-line-args*)
./script.clj 1 2 3
("hello" "1" "2" "3")
Current file path
The var *file*
contains the full path of the file that is currently
being executed:
$ cat example.clj
(prn *file*)
$ bb example.clj
"/Users/borkdude/example.clj"
Parsing command line arguments
Command-line arguments can be retrieved using *command-line-args*
. If you
want to parse command line arguments, you may use the built-in
clojure.tools.cli
namespace:
Babashka ships with clojure.tools.cli
:
(require '[clojure.tools.cli :refer [parse-opts]])
(def cli-options
;; An option with a required argument
[["-p" "--port PORT" "Port number"
:default 80
:parse-fn #(Integer/parseInt %)
:validate [#(< 0 % 0x10000) "Must be a number between 0 and 65536"]]
["-h" "--help"]])
(:options (parse-opts *command-line-args* cli-options))
$ bb script.clj
{:port 80}
$ bb script.clj -h
{:port 80, :help true}
There is also the nubank/docopt library that is compatible with babashka.
Project setup
Classpath
When you want to organize babashka scripts like a regular Clojure project, you will need to set a classpath.
Babashka accepts a --classpath
option that will be used to search for
namespaces when requiring them. Say we have a file src/my/namespace.clj
:
(ns my.namespace)
(defn -main [& args]
(apply println "Hello from my namespace!" args))
Now we can execute this main function with:
$ bb --classpath src --main my.namespace 1 2 3
Hello from my namespace! 1 2 3
If you have a larger script with a classic Clojure project layout like
$ tree -L 3
├── deps.edn
├── README
├── src
│ └── project_namespace
│ ├── main.clj
│ └── utilities.clj
└── test
└── project_namespace
├── test_main.clj
└── test_utilities.clj
then you can tell babashka to include both the src
and test
folders
in the classpath and start a socket REPL by running:
$ bb --classpath src:test socket-repl 1666
If there is no --classpath
argument, the BABASHKA_CLASSPATH
environment
variable will be used.
Also see the babashka.classpath namespace which allows dynamically adding to the classpath.
The namespace babashka.deps integrates
tools.deps with babashka and allows
you to set the classpath using a deps.edn
map.
bb.edn
Since babashka 0.3.1 supports a local bb.edn
file where you can declare one
or multiple paths and dependencies so they are automatically added to the classpath:
{:paths ["src"]
:deps {medley/medley {:mvn/version "1.3.0"}}}
$ bb -e "(require '[medley.core :as m]) (m/index-by :id [{:id 1} {:id 2}])"
{1 {:id 1}, 2 {:id 2}}
Invoking a main function
A main function can be invoked with -m
or --main
like shown above. When
given the argument foo.bar
, the namespace foo.bar
will be required and the
function foo.bar/-main
will be called with command line arguments as strings.
Since babashka 0.3.1 you may pass a fully qualified symbol to -m
:
$ bb -m clojure.core/prn 1 2 3
"1" "2" "3"
so you can execute any function as a main function, as long as it accepts varargs arguments.
When invoking bb
with a main function, the expression (System/getProperty
"babashka.main")
will return the name of the main function.
Preloads
The environment variable BABASHKA_PRELOADS
allows to define code that
will be available in all subsequent usages of babashka.
BABASHKA_PRELOADS='(defn foo [x] (+ x 2))'
BABASHKA_PRELOADS=$BABASHKA_PRELOADS' (defn bar [x] (* x 2))'
export BABASHKA_PRELOADS
Note that you can concatenate multiple expressions. Now you can use these functions in babashka:
$ bb '(-> (foo *input*) bar)' <<< 1
6
You can also preload an entire file using load-file
:
export BABASHKA_PRELOADS='(load-file "my_awesome_prelude.clj")'
Note that *input*
is not available in preloads.
Running a REPL
Babashka supports running a REPL, a socket REPL and an nREPL server.
REPL
To start a REPL, type:
$ bb repl
To get history with up and down arrows, use rlwrap:
$ rlwrap bb repl
Socket REPL
To start a socket REPL on port 1666
:
$ bb socket-repl 1666
Babashka socket REPL started at localhost:1666
Now you can connect with your favorite socket REPL client:
$ rlwrap nc 127.0.0.1 1666
Babashka v0.0.14 REPL.
Use :repl/quit or :repl/exit to quit the REPL.
Clojure rocks, Bash reaches.
bb=> (+ 1 2 3)
6
bb=> :repl/quit
$
The --socket-repl
option takes options similar to the clojure.server.repl
Java property option in Clojure:
$ bb socket-repl '{:address "0.0.0.0" :accept clojure.core.server/repl :port 1666}'
Editor plugins and tools known to work with a babashka socket REPL:
pREPL
Launching a prepl can be done as follows:
$ bb socket-repl '{:address "0.0.0.0" :accept clojure.core.server/io-prepl :port 1666}'
or programmatically:
$ bb -e '(clojure.core.server/io-prepl)'
(+ 1 2 3)
{:tag :ret, :val "6", :ns "user", :ms 0, :form "(+ 1 2 3)"}
nREPL
To start an nREPL server:
$ bb nrepl-server 1667
Then connect with your favorite nREPL client:
$ lein repl :connect 1667
Connecting to nREPL at 127.0.0.1:1667
user=> (+ 1 2 3)
6
user=>
Editor plugins and tools known to work with the babashka nREPL server:
The babashka nREPL server does not write an .nrepl-port
file at
startup, but you can easily write a script that launches the server and
writes the file:
#!/usr/bin/env bb
(import [java.net ServerSocket]
[java.io File]
[java.lang ProcessBuilder$Redirect])
(require '[babashka.wait :as wait])
(let [nrepl-port (with-open [sock (ServerSocket. 0)] (.getLocalPort sock))
cp (str/join File/pathSeparatorChar ["src" "test"])
pb (doto (ProcessBuilder. (into ["bb" "--nrepl-server" (str nrepl-port)
"--classpath" cp]
*command-line-args*))
(.redirectOutput ProcessBuilder$Redirect/INHERIT))
proc (.start pb)]
(wait/wait-for-port "localhost" nrepl-port)
(spit ".nrepl-port" nrepl-port)
(.deleteOnExit (File. ".nrepl-port"))
(.waitFor proc))
Debugging the nREPL server
To debug the nREPL server from the binary you can run:
$ BABASHKA_DEV=true bb nrepl-server 1667
This will print all the incoming messages.
To debug the nREPL server from source:
$ git clone https://github.com/borkdude/babashka --recurse-submodules
$ cd babashka
$ BABASHKA_DEV=true clojure -A:main --nrepl-server 1667
Input and output flags
In one-liners the *input*
value may come in handy. It contains the
input read from stdin as EDN by default. If you want to read in text,
use the -i
flag, which binds *input*
to a lazy seq of lines of text.
If you want to read multiple EDN values, use the -I
flag. The -o
option prints the result as lines of text. The -O
option prints the
result as lines of EDN values.
*input* is only available in the user namespace, designed
for one-liners. For writing scripts, see Scripts.
|
The following table illustrates the combination of options for commands of the form
echo "{{Input}}" | bb {{Input flags}} {{Output flags}} "*input*"
Input | Input flags | Output flag | *input* |
Output |
---|---|---|---|---|
|
|
|
||
hello |
|
|
|
|
hello |
|
|
|
hello |
|
|
|
|
|
|
|
|
|
|
When combined with the --stream
option, the expression is executed for
each value in the input:
$ echo '{:a 1} {:a 2}' | bb --stream '*input*'
{:a 1}
{:a 2}
Scripts
When writing scripts instead of one-liners on the command line, it is
not recommended to use *input*
. Here is how you can rewrite to
standard Clojure code.
EDN input
Reading a single EDN value from stdin:
(ns script
(:require [clojure.edn :as edn]))
(edn/read *in*)
Reading multiple EDN values from stdin (the -I
flag):
(ns script
(:require [clojure.edn :as edn]
[clojure.java.io :as io]))
(let [reader (java.io.PushbackReader. (io/reader *in*))]
(take-while #(not (identical? ::eof %)) (repeatedly #(edn/read {:eof ::eof} reader))))
Text input
Reading text from stdin can be done with (slurp *in*)
. To get a lazy
seq of lines (the -i
flag), you can use:
(ns script
(:require [clojure.java.io :as io]))
(line-seq (io/reader *in*))
Output
To print to stdout, use println
for text and prn
for EDN values.
Uberscript
The --uberscript
option collects the expressions in
BABASHKA_PRELOADS
, the command line expression or file, the main
entrypoint and all required namespaces from the classpath into a single
file. This can be convenient for debugging and deployment.
Here is an example that uses a function from the clj-commons/fs library.
Let’s first set the classpath:
$ export BABASHKA_CLASSPATH=$(clojure -Spath -Sdeps '{:deps {clj-commons/fs {:mvn/version "1.6.307"}}}')
Write a little script, say glob.clj
:
(ns glob (:require [me.raynes.fs :as fs]))
(run! (comp println str)
(fs/glob (first *command-line-args*)))
For testing, we’ll make a file which we will find using the glob function:
$ touch README.md
Now we can execute the script which uses the library:
$ time bb glob.clj '*.md'
/private/tmp/glob/README.md
bb glob.clj '*.md' 0.03s user 0.01s system 88% cpu 0.047 total
Producing an uberscript with all required code:
$ bb uberscript glob-uberscript.clj -f glob.clj
To prove that we don’t need the classpath anymore:
$ unset BABASHKA_CLASSPATH
$ time bb glob-uberscript.clj '*.md'
/private/tmp/glob/README.md
bb glob-uberscript.clj '*.md' 0.03s user 0.02s system 93% cpu 0.049 total
Caveats:
-
Dynamic requires. Building uberscripts works by running top-level
ns
andrequire
forms. The rest of the code is not evaluated. Code that relies on dynamic requires may not work in an uberscript. -
Resources. The usage of
io/resource
assumes a classpath, so when this is used in your uberscript, you still have to set a classpath and bring the resources along.
If any of the above is problematic for your project, using an uberjar is a good alternative.
Carve
Uberscripts can be optimized by cutting out unused vars with carve.
$ wc -l glob-uberscript.clj
583 glob-uberscript.clj
$ carve --opts '{:paths ["glob-uberscript.clj"] :aggressive true :silent true}'
$ wc -l glob-uberscript.clj
105 glob-uberscript.clj
Note that the uberscript became 72% shorter. This has a beneficial effect on execution time:
$ time bb glob-uberscript.clj '*.md'
/private/tmp/glob/README.md
bb glob-uberscript.clj '*.md' 0.02s user 0.01s system 84% cpu 0.034 total
Uberjar
Babashka can create uberjars from a given classpath and optionally a main method:
$ cat src/foo.clj
(ns foo (:gen-class)) (defn -main [& args] (prn :hello))
$ bb -cp $(clojure -Spath) uberjar foo.jar -m foo
$ bb foo.jar
:hello
When producing a classpath using the clojure
or deps.clj
tool,
Clojure itself, spec and the core specs will be on the classpath and
will therefore be included in your uberjar, which makes it bigger than
necessary:
$ ls -lh foo.jar
-rw-r--r-- 1 borkdude staff 4.5M Aug 19 17:04 foo.jar
To exclude these dependencies, you can use the following
:classpath-overrides
in your deps.edn
:
{:aliases {:remove-clojure {:classpath-overrides {org.clojure/clojure nil
org.clojure/spec.alpha nil
org.clojure/core.specs.alpha nil}}}}
$ rm foo.jar
$ bb -cp $(clojure -A:remove-clojure -Spath) uberjar foo.jar -m foo
$ bb foo.jar
:hello
$ ls -lh foo.jar
-rw-r--r-- 1 borkdude staff 871B Aug 19 17:07 foo.jar
If you want your uberjar to be compatible with the JVM, you’ll need to compile the main namespace. Babashka does not do compilation, so we use Clojure on the JVM for that part:
$ rm foo.jar
$ mkdir classes
$ clojure -e "(require 'foo) (compile 'foo)"
foo
$ bb -cp $(clojure -Spath):classes uberjar foo.jar -m foo
$ bb foo.jar
:hello
$ java -jar foo.jar
:hello
System properties
Babashka sets the following system properties:
-
babashka.version
: the version string, e.g."1.2.0"
-
babashka.main
: the--main
argument -
babashka.file
: the--file
argument (normalized using.getAbsolutePath
)
Data readers
Data readers can be enabled by setting *data-readers*
to a hashmap of
symbols to functions or vars:
$ bb -e "(set! *data-readers* {'t/tag inc}) #t/tag 1"
2
To preserve good startup time, babashka does not scan the classpath for
data_readers.clj
files.
Reader conditionals
Babashka supports reader conditionals by taking either the :bb
or
:clj
branch, whichever comes first. NOTE: the :clj
branch behavior
was added in version 0.0.71, before that version the :clj
branch was
ignored.
$ bb -e "#?(:bb :hello :clj :bye)"
:hello
$ bb -e "#?(:clj :bye :bb :hello)"
:bye
$ bb -e "[1 2 #?@(:bb [] :clj [1])]"
[1 2]
Invoking clojure
Babashka bundles deps.clj for invoking a
clojure
JVM process:
$ bb clojure -M -e "*clojure-version*"
{:major 1, :minor 10, :incremental 1, :qualifier nil}
See the clojure function in the babashka.deps namespace for programmatically invoking clojure.
Style
A note on style. Babashka recommends the following:
Explicit requires
Use explicit requires with namespace aliases in scripts, unless you’re writing one-liners.
Do this:
$ ls | bb -i '(-> *input* first (str/includes? "m"))'
true
But not this:
script.clj:
(-> *input* first (str/includes? "m"))
Rather do this:
script.clj:
(ns script
(:require [clojure.java.io :as io]
[clojure.string :as str]))
(-> (io/reader *in*) line-seq first (str/includes? "m"))
Some reasons for this:
-
Linters like clj-kondo work better with code that uses namespace forms, explicit requires, and known Clojure constructs
-
Editor tooling works better with namespace forms (sorting requires, etc).
-
Writing compatible code gives you the option to run the same script with
clojure
Libraries
Built-in namespaces
In addition to clojure.core
, the following namespaces are available in babashka.
Some are available through pre-defined aliases in the user
namespace,
which can be handy for one-liners. If not all vars are available, they
are enumerated explicitly. If some important var is missing, an issue or
PR is welcome.
From Clojure:
-
clojure.core
-
clojure.core.protocols
:Datafiable
,Navigable
-
clojure.data
-
clojure.datafy
-
clojure.edn
aliased asedn
-
clojure.java.browse
-
clojure.java.io
aliased asio
:-
as-relative-path
,as-url
,copy
,delete-file
,file
,input-stream
,make-parents
,output-stream
,reader
,resource
,writer
-
-
clojure.java.shell
aliased asshell
-
clojure.main
:demunge
,repl
,repl-requires
-
clojure.pprint
:pprint
,cl-format
-
clojure.set
aliased asset
-
clojure.string
aliased asstr
-
clojure.stacktrace
-
clojure.test
-
clojure.zip
Additional libraries:
-
bencode.core
aliased asbencode
:read-bencode
,write-bencode
-
cheshire.core
aliased asjson
-
clojure.core.async
aliased asasync
. Also see docs. -
clojure.data.csv
aliased ascsv
-
clojure.data.xml
aliased asxml
-
clojure.tools.cli
aliased astools.cli
-
clj-yaml.core
alias asyaml
-
cognitect.transit
aliased astransit
-
hiccup.core
andhiccup2.core
-
-
clojure.test.check
-
clojure.test.check.generators
-
clojure.test.check.properties
-
-
-
rewrite-clj.parser
-
rewrite-clj.node
-
rewrite-clj.zip
-
rewrite-clj.paredit
-
See the
projects
page for libraries that are not built-in, but which you can load from
source via the --classpath
option.
See the build page for built-in libraries that can be enabled via feature flags, if you want to compile babashka yourself.
A selection of Java classes are available, see
babashka/impl/classes.clj
in babashka’s git repo.
Babashka namespaces
babashka.classpath
Available functions:
-
add-classpath
-
get-classpath
-
split-classpath
add-classpath
The function add-classpath
which can be used to add to the classpath
dynamically:
(require '[babashka.classpath :refer [add-classpath]]
'[clojure.java.shell :refer [sh]]
'[clojure.string :as str])
(def medley-dep '{:deps {medley {:git/url "https://github.com/borkdude/medley"
:sha "91adfb5da33f8d23f75f0894da1defe567a625c0"}}})
(def cp (-> (sh "clojure" "-Spath" "-Sdeps" (str medley-dep)) :out str/trim))
(add-classpath cp)
(require '[medley.core :as m])
(m/index-by :id [{:id 1} {:id 2}]) ;;=> {1 {:id 1}, 2 {:id 2}}
get-classpath
The function get-classpath
returns the classpath as set by --classpath
,
BABASHKA_CLASSPATH
and add-classpath
.
split-classpath
Given a classpath, returns a seq of strings as the result of splitting the classpath by the platform specific path separatator.
babashka.deps
Available functions:
-
add-deps
-
clojure
-
merge-deps
add-deps
The function add-deps
takes a deps edn map like {:deps {medley/medley
{:mvn/version "1.3.0"}}}
, resolves it using
deps.clj and then adds to the babashka
classpath accordingly.
Example:
(require '[babashka.deps :as deps])
(deps/add-deps '{:deps {medley/medley {:mvn/version "1.3.0"}}})
(require '[medley.core :as m])
(m/index-by :id [{:id 1} {:id 2}])
Optionally, add-deps
takes a second arg with options. Currently the only
option is :aliases
which will affect how deps are resolved:
Example:
(deps/add-deps '{:aliases {:medley {:extra-deps {medley/medley {:mvn/version "1.3.0"}}}}}
{:aliases [:medley]})
clojure
The function clojure
takes a sequential collection of arguments, similar to
the clojure CLI. The arguments are then passed to
deps.clj. The clojure
function returns
nil
and prints to *out*
for commands like -Stree
, and -Spath
. For -M
,
-X
and -A
it invokes java
with babashka.process/process
(see
babashka.process) and returns the associated record. For
more details, read the docstring with:
(require '[clojure.repl :refer [doc]])
(doc babashka.deps/clojure)
Example:
The following script passes through command line arguments to clojure, while adding the medley dependency:
(require '[babashka.deps :as deps])
(def deps '{:deps {medley/medley {:mvn/version "1.3.0"}}})
(def clojure-args (list* "-Sdeps" deps *command-line-args*))
(if-let [proc (deps/clojure clojure-args)]
(-> @proc :exit (System/exit))
(System/exit 0))
babashka.wait
Contains the functions: wait-for-port
and wait-for-path
.
Usage of wait-for-port
:
(wait/wait-for-port "localhost" 8080)
(wait/wait-for-port "localhost" 8080 {:timeout 1000 :pause 1000})
Waits for TCP connection to be available on host and port. Options map
supports :timeout
and :pause
. If :timeout
is provided and reached,
:default
's value (if any) is returned. The :pause
option determines
the time waited between retries.
Usage of wait-for-path
:
(wait/wait-for-path "/tmp/wait-path-test")
(wait/wait-for-path "/tmp/wait-path-test" {:timeout 1000 :pause 1000})
Waits for file path to be available. Options map supports :default
,
:timeout
and :pause
. If :timeout
is provided and reached,
:default
's value (if any) is returned. The :pause
option determines
the time waited between retries.
The namespace babashka.wait
is aliased as wait
in the user
namespace.
babashka.signal
Contains the function signal/pipe-signal-received?
. Usage:
(signal/pipe-signal-received?)
Returns true if PIPE
signal was received. Example:
$ bb -e '((fn [x] (println x) (when (not (signal/pipe-signal-received?)) (recur (inc x)))) 0)' | head -n2
1
2
The namespace babashka.signal
is aliased as signal
in the user
namespace.
babashka.curl
The namespace babashka.curl
is a tiny wrapper around curl. It’s aliased as
curl
in the user namespace. See
babashka.curl for how to use it.
babashka.process
The babashka.process
library. See the
process repo for API docs.
Projects
Babashka is able to run Clojure projects from source, if they are compatible with the subset of Clojure that sci is capable of running.
Check this page for projects that are known to work with babashka.
babashka.fs
The babashka.fs
library offers file system utilities. See the
fs repo for API docs.
Pods
Pods are programs that can be used as a Clojure library by babashka. Documentation is available in the library repo.
A list of available pods can be found here.
Pod registry
Since bb 0.2.6 pods can be obtained via the pod-registry.
This is an example script which uses bootleg as a pod to convert hiccup to HTML.
#!/usr/bin/env bb
(require '[babashka.pods :as pods])
(pods/load-pod 'retrogradeorbit/bootleg "0.1.9")
(require '[pod.retrogradeorbit.bootleg.utils :as utils])
(-> [:div
[:h1 "Using Bootleg From Babashka"]
[:p "This is a demo"]]
(utils/convert-to :html))
Child processes
by Michiel Borkent
There are several ways of creating child processes in babashka. Let’s start with the easiest one.
clojure.java.shell
A common way to shell out to another process is via clojure.java.shell
:
user=> (require '[clojure.java.shell :refer [sh]])
nil
user=> (sh "ls")
{:exit 0, :out "README.md\ndist\ngh-pages\nscript\nsrc\n", :err ""}
As you can see the result of :out
are the lines of text produced by ls
. The
:exit
code was 0
and there was no output on stderr.
user=> (sh "ls" "foo")
{:exit 1, :out "", :err "ls: foo: No such file or directory\n"}
If we invoke ls
with a non-existing file, there is error output, but not
output on stdout and the :exit
code is 1
.
For more information on clojure.java.shell
, read the
API documentation
babashka.process
Creating a child process with clojure.java.shell/sh
is convenient and
sufficient in 90% of the scripts you’re going to write with babashka. For the
remaining 10% you’re going to need something slightly more powerful. This is
where the babashka.process
library namespace comes in. The README and source
code of this library is located here.
What if we wanted to start a Clojure pREPL in our script, send commands to it
and read the evaluated output in a loop? Using clojure.java.shell/sh
for this
has the limitation that we can only send input and read output once per process.
Using babashka.process
which leverages the java.lang.ProcessBuilder
and
java.lang.Process
classes gives us the extra power we need to overcome this
limitation.
First we show how to use process
to start a process that inherits stdout,
stderr and stdin from the parent process. This is often useful when you want to
hand over control to another process at the end of a script. The following code
starts a pREPL process:
#!/usr/bin/env bb
(require '[babashka.process :as p])
(defn io-prepl []
(let [cmd ["clojure" "-M"
"-e" "(require '[clojure.core.server :as s])"
"-e" "(s/io-prepl)"] (1)
proc (p/process cmd (2)
{:inherit true (3)
:shutdown p/destroy-tree})] (4)
proc))
@(io-prepl) (5)
1 | The command and its arguments |
2 | Creating the process |
3 | This option redirects stdout and stdin from the parent process to the child process. |
4 | This line will destroy the child process and all of it’s children processes before the parent process exits. |
5 | Dereferencing the process waits until it ends. If we would not include this, our script would end immediately. |
When calling the script, we can interact with the pREPL
$ bb src/process/prepl_1.clj
(+ 1 2 3)
{:tag :ret, :val "6", :ns "user", :ms 5, :form "(+ 1 2 3)"}
Pressing ctrl-C or typing :repl/quit
will make the script terminate.
What if we want to parse the output of the prepl and reformat it to our own
likings? Instead of sending the output of the child process to stdout we will
have to get a hold of it. We can grab the output by not using :inherit
and
getting the :out
stream of the process. Similarly we will grab the :in
stream of the process and handle reading ourselves.
#!/usr/bin/env bb
(require '[babashka.process :as p])
(defn io-prepl []
(let [cmd ["clojure" "-M"
"-e" "(require '[clojure.core.server :as s])"
"-e" "(s/io-prepl)"]
proc (p/process cmd
{:shutdown p/destroy-tree})]
proc))
(def prepl-process (io-prepl))
(require '[clojure.java.io :as io])
(def input-writer (io/writer (:in prepl-process))) (1)
(def output-reader (java.io.PushbackReader.
(io/reader (:out prepl-process)))) (2)
(require '[clojure.edn :as edn])
(loop []
(println "Type an expression to evaluate:")
(when-let [v (read-line)] (3)
(binding [*out* input-writer] (4)
(println v))
(let [next-val (edn/read {:eof ::EOF} output-reader)] (5)
(when-not (identical? ::EOF next-val)
(println (:form next-val) (6)
"evaluates to"
(:val next-val))
(recur)))))
1 | A writer that we will use to send input to the pREPL |
2 | A reader that reads from the pREPL output. pREPL returns EDN and
clojure.edn/read needs a PushbackReader . |
3 | We read user input via clojure.core/read-line |
4 | Here we binding *out* to the input writer so we can use println to write
to it. |
5 | Read the next EDN value. |
6 | Print the output using our custom formatting. |
Here is an example run:
$ bb src/process/prepl_2.clj
Type an expression to evaluate:
(+ 1 2 3)
(+ 1 2 3) evaluates to 6
:repl/quit
Real world examples
Prior to babashka.process
people were using raw ProcessBuilder
interop. Here are some real world examples. These might get updated to babashka.process
in the future.
-
The babashka repo uses
ProcessBuilder
to invoke git to make a commit after bumping the version. It redirects the output fromgit
directly to stdout of the parent process. -
babashka.curl uses
ProcessBuilder
to interact with thecurl
executable -
deps.clj uses
ProcessBuilder
to interact withjava
, e.g. to start a Clojure REPL or compute a classpath using tools.deps
Recipes
Running tests
Babashka bundles clojure.test
. To run tests you can write a test runner script. Given the following project structure:
.
├── src
│ └──...
└── test
└── your
├── test_a.clj
└── test_b.clj
#!/usr/bin/env bb
(require '[clojure.test :as t]
'[babashka.classpath :as cp])
(cp/add-classpath "src:test") (1)
(require 'your.test-a 'your.test-b) (2)
(def test-results
(t/run-tests 'your.test-a 'your.test-b)) (3)
(def failures-and-errors
(let [{:keys [:fail :error]} test-results]
(+ fail error))) (4)
(System/exit failures-and-errors) (5)
1 | Add sources and tests to the classpath |
2 | Require the test namespaces |
3 | Run all tests in the test namespaces |
4 | Extract failures and errors |
5 | Exit the test script with a non-zero exit code when there are failures or errors |
Main file
In Python scripts there is a well-known pattern to check if the current
file was the file invoked from the command line, or loaded from another
file: the __name__ == "__main__"
pattern. In babashka this pattern can
be implemented with:
(= *file* (System/getProperty "babashka.file"))
Shutdown hook
Adding a shutdown hook allows you to execute some code before the script exits.
$ bb -e '(-> (Runtime/getRuntime) (.addShutdownHook (Thread. #(println "bye"))))'
bye
This also works when the script is interrupted with ctrl-c.
Printing returned values
Babashka doesn’t print a returned nil
as lots of scripts end in
something side-effecting.
$ bb -e '(:a {:a 5})'
5
$ bb -e '(:b {:a 5})'
$
If you really want to print the nil, you can use (prn ..)
instead.
HTTP requests
For making HTTP requests you can use:
-
babashka.curl. This library is included with babashka and aliased as
curl
in the user namespace. The interface is similar to that of clj-http but it will shell out tocurl
to make requests. -
slurp
for simpleGET
requests -
clj-http-lite as a library.
-
clojure.java.shell
orbabashka.process
for shelling out to your favorite command line http client
Choosing the right client
If memory usage is a concern and you are downloading big files, choose
babashka.curl
with :as :stream
over org.httpkit.client
since
http-kit holds the entire response in memory at once. Let’s download a
200mb file with 10mb heap size:
$ bb -Xmx10m -e '(io/copy (:body (curl/get "http://ipv4.download.thinkbroadband.com/200MB.zip" {:as :stream})) (io/file "/tmp/200mb.zip"))'
With babashka.curl
this works fine. However with org.httpkit.client
that won’t work. Not even 190mb of heap will do:
$ bb -Xmx190m -e '(io/copy (:body @(org.httpkit.client/get "http://ipv4.download.thinkbroadband.com/200MB.zip" {:as :stream})) (io/file "/tmp/200mb.zip"))'
Sun Nov 08 23:01:46 CET 2020 [client-loop] ERROR - select exception, should not happen
java.lang.OutOfMemoryError: Array allocation too large.
If your script creates many requests with relatively small payloads,
choose org.httpkit.client
over babashka.curl
since babashka.curl
creates a curl
process for each request.
In the future babashka (1.0.0?) may come with an HTTP client based on
the JVM 11 java.net.http
package that ticks all the boxes (async,
HTTP/2, websockets, multi-part file uploads, sane memory usage) and is a
suitable replacement for all of the above options. If you know about a
GraalVM-friendly feature-complete well-maintained library, please reach
out!
HTTP over Unix sockets
This can be useful for talking to Docker:
(require '[clojure.java.shell :refer [sh]])
(require '[cheshire.core :as json])
(-> (sh "curl" "--silent"
"--no-buffer" "--unix-socket"
"/var/run/docker.sock"
"http://localhost/images/json")
:out
(json/parse-string true)
first
:RepoTags) ;;=> ["borkdude/babashka:latest"]
Core.async
In addition to future
, pmap
, promise
and friends, you may use the
clojure.core.async
namespace for asynchronous scripting. The following
example shows how to get first available value from two different
processes:
bb -e '
(defn async-command [& args]
(async/thread (apply shell/sh "bash" "-c" args)))
(-> (async/alts!! [(async-command "sleep 2 && echo process 1")
(async-command "sleep 1 && echo process 2")])
first :out str/trim println)'
process 2
Caveat: currently the go
macro is available for compatibility with JVM
programs, but the implementation maps to clojure.core.async/thread
and
the single exclamation mark operations (<!
, >!
, etc.) map to the
double exclamation mark operations (<!!
, >!!
, etc.). It will not
"park" threads, like on the JVM.
Examples like the following may still work, but will take a lot more
system resources than on the JVM and will break down for some high value
of n
:
(require '[clojure.core.async :as async])
(def n 1000)
(let [cs (repeatedly n async/chan)
begin (System/currentTimeMillis)]
(doseq [c cs] (async/go (async/>! c "hi")))
(dotimes [_ n]
(let [[v _] (async/alts!! cs)]
(assert (= "hi" v))))
(println "Read" n "msgs in" (- (System/currentTimeMillis) begin) "ms"))
Interacting with an nREPL server
Babashka comes with the nrepl/bencode
library which allows you to read and write bencode messages to a socket.
A simple example which evaluates a Clojure expression on an nREPL server
started with lein repl
:
(ns nrepl-client
(:require [bencode.core :as b]))
(defn nrepl-eval [port expr]
(let [s (java.net.Socket. "localhost" port)
out (.getOutputStream s)
in (java.io.PushbackInputStream. (.getInputStream s))
_ (b/write-bencode out {"op" "eval" "code" expr})
bytes (get (b/read-bencode in) "value")]
(String. bytes)))
(nrepl-eval 52054 "(+ 1 2 3)") ;;=> "6"
Running from Cygwin/Git Bash
On Windows, bb
can be invoked from the bash shell directly:
$ bb -e '(+ 1 2 3)'
6
However, creating a script that invokes bb
via a shebang leads to an error if
the script is not in the current directory. Suppose you had the following script
named hello
on your path:
#!/usr/bin/env bb
(println "Hello, world!")
$ hello
----- Error --------------------------------------------------------------------
Type: java.lang.Exception
Message: File does not exist: /cygdrive/c/path/to/hello
The problem here is that the shell is passing a Cygwin-style path to bb
, but
bb
can’t recognize it because it wasn’t compiled with Cygwin.
The solution is to create a wrapper script that converts the Cygwin-style path
to a Windows-style path before invoking bb
. Put the following into a script
called bbwrap
somewhere on your Cygwin path, say in /usr/local/bin/bbwrap
:
#!/bin/bash
SCRIPT=$1
shift
bb.exe $(cygpath -w $SCRIPT) $@
Make sure to fix your original script to invoke bbwrap
instead of bb
directly:
#!/usr/bin/env bbwrap
(println "Hello, world!")
Differences with Clojure
Babashka is implemented using the Small
Clojure Interpreter. This means that a snippet or script is not
compiled to JVM bytecode, but executed form by form by a runtime which
implements a substantial subset of Clojure. Babashka is compiled to a
native binary using GraalVM. It comes
with a selection of built-in namespaces and functions from Clojure and
other useful libraries. The data types (numbers, strings, persistent
collections) are the same. Multi-threading is supported (pmap
,
future
).
Differences with Clojure:
-
A pre-selected set of Java classes are supported. You cannot add Java classes at runtime.
-
Interpretation comes with overhead. Therefore loops are slower than in Clojure on the JVM. In general interpretation yields slower programs than compiled programs.
-
No
deftype
,definterface
and unboxed math. -
defprotocol
anddefrecord
are implemented using multimethods and regular maps. Ostensibly they work the same, but under the hood there are no Java classes that correspond to them. -
Currently
reify
works only for one class at a time -
The
clojure.core.async/go
macro is not (yet) supported. For compatibility it currently maps toclojure.core.async/thread
. More info here.
Contributing
Visit Babashka book’s Github repository and read CONTRIBUTING.md on how to contribute.
License
Copyright © 2020-2021 Michiel Borkent
This book is licensed under a Creative Commons License.
Please note that because this is a No Derivatives license, you may not use this repository as a basis for creating your own book based on this one.