(This answer is more of an experiment to see just what it would look like to "apply the right tool for the job"—in this case, clojure. Indeed, I came to write this answer precisely because the solution in clojure occurred to me within about 10 seconds of reading the question, coupled with the fact about (read) I'll mention below. The rest—the true "problem" in this answer—was a 90 minute attempt to fight lisp's interactive roots. This struggle is not unknown to me; SML, particularly the New Jersey implementation, suffers from the same disadvantage.)
A lisp is the obvious choice for processing data-structures like lists. In fact, in clojure, this problem is solved by (flatten my-list) or (map flatten list-of-lists)!
But we are not done if we want to keep the input and output exactly to what is defined in the question. Here, clojure struggles under its own weight: it was built for use interactively (like so many lisps) or for use like a java program (run a main method). Neither truly facilitates the unix filter tradition of read from standard in/write to standard out. So, we will solve this challenge in several different ways, all of them more or less useful/outrageous.
We will rely on the following interesting facts:
- Commas are whitespace in clojure, so the input given is actually a valid set of clojure vectors.
(read) reads a single object from standard in; not a line, as in many other languages, but a single clojure form (such as an s-expression or a vector).
Metaprogramming with cat
Having already observed that the original input is valid clojure, we bypass reading it as input from clojure by injecting directly into a clojure program (and leave sed to do some rather dull formatting):
#! /usr/bin/env bash
clojure -e '(->> ['"$(cat)"'] (map flatten) (map vec) (apply prn))' \
| sed -e 's/ /, /g'
Running this produces
$ ./nested-clj-cat <unix.in
["q", "0", "R", "L"], ["q", "1", "[", "]"], ["q", "2", "L", "R"], ["q", "3", "R", "L"]
The trickery in this solution is a mix of proper quoting, a properly useful cat, and a subtle-but-necessary continuous coercion to vectors.
Using clojure as the interpreter
Wouldn't that inner clojure script be a lot more maintainable if we could move it to a file containing code rather a quoted string? (I say this quite seriously, despite the tradition in most shell scripts of invoking awk/sed/python/perl inline with strings!)
But now we have to deal with reading standard in; unfortunately, (read) only reads one object at a time, while the input given is a series of objects. We could massage the data by adding a [ to the beginning and a ] to the end:
sed -e 's/^/[/' -e 's/$/]/'
But then the caller has to remember this, or the original program has to be adjusted.
So we'll build a function read-all that reads all the objects in a stream and returns that sequence. Then we'll apply our technique from before:
#! /usr/bin/env clojure
(require '[clojure.java.shell :as shell])
(defn read-all
[stream]
(loop [acc []]
(let [red (binding [*read-eval* false]
(read {:eof :eof} stream))]
(if (= red :eof)
acc
(recur (conj acc red))))))
(->> (read-all *in*)
(map flatten)
(map vec)
(apply prn))
This has another downside: we still need sed on the end to get the exact data! Otherwise:
$ ./nested-clj-read <unix.in
["q" "0" "R" "L"] ["q" "1" "[" "]"] ["q" "2" "L" "R"] ["q" "3" "R" "L"]
which is just not quite right. Perhaps we can fix this in clojure?
Topsy-turvy: shell in clojure
It turns out that one of the following is true:
- I'm very bad at formatting strings in clojure with simple techniques (
str, format), or
- Clojure is pretty horrible for formatting complex data-structures as strings
I suspect the latter only because clojure makes it very easy to pass data-structures between programs as data-structures (prn/read and the EDN format are evidence). I didn't mess with the common-lisp formatter cl-format that I know is capable of doing this, because I figured that might as well be too many lisps in the same jumble of languages :)
If anyone can solve this more elegantly, I would be glad to discuss it.
Ultimately, I resorted to embedding the sed calls inside of clojure—this does avoid the need for the caller to remember to invoke it, at the cost of adding yet more complexity to the code. In order to keep things nice and readable, I introduce the pipe macro:
(defmacro |
[cmd in]
`(:out (shell/sh ~@cmd :in ~in)))
It needs to be a macro because apply won't work with non-lists after the lists and I really want in to be the last parameter (so that it fits with ->>). Alas, because of sh's implementation using futures, we need a call to (shutdown-agents) to eliminate waiting for minutes after the script is done for it to terminate.
So the final script is
#! /usr/bin/env clojure
(require '[clojure.java.shell :as shell])
(defn read-all
[stream]
(loop [acc []]
(let [red (binding [*read-eval* false]
(read {:eof :eof} stream))]
(if (= red :eof)
acc
(recur (conj acc red))))))
(defmacro |
[cmd in]
`(:out (shell/sh ~@cmd :in ~in)))
(->> (read-all *in*)
(map flatten)
(map vec)
(apply prn-str)
(| ["sed" "-e" "s/ /, /g"])
print)
; needed because of shell/sh's use of futures
(shutdown-agents)
And the results:
$ ./nested-clj-read-with-sed <unix.in
["q", "0", "R", "L"], ["q", "1", "[", "]"], ["q", "2", "L", "R"], ["q", "3", "R", "L"]
Perfect.
Lessons learned
Other languages have humongous benefits when it comes to proper recursive data manipulation. They don't always make acting like a unix-filter easy, however, and trying to cram them into that world often results in increased complexity. Even the short cat solution should give a reviewer pause—not because it's difficult to understand, but because it's just obscure enough to require some thought.
Still, perhaps it's worth considering other languages when manipulating certain forms of data: while all the other sed/perl/awk solutions I browsed here had no trouble reading in and writing out data, they had to do quite a bit of work to manipulate it. In some cases, I would call that level of work unmaintainable for the difficulty required to penetrate the commands (and I use these tools daily myself)! This is not to argue that my clojure solutions are any less impenetrable, but rather that we have two sides of a coin that really wants to be a torus: give me both, give me the ease of I/O filters know and the ease of data processing inherent to lisps/ML.
Aside: I wonder if jq could solve this after we wrap the input in {}?
[ a [ b [ c ]]] - > [a b c]?[[["q", "A"], "]"], "L"]? May brackets go inside quotes?\"and/or csv style quoted quotes""?