I'm exploring the notion of having a shell script input and output two dimensional structures of strings, with the limitation that the strings do not contain newlines. They can contain spaces.
The way that I want to try to do this is by using unescaped whitespace as a delimiter for the horizontal axis, and the newline is the delimiter for the vertical axis.
To experiment with this, I am using the following small tool to inspect arguments:
❯ cat argshow
#!/bin/bash
ITERATION=1
while (( "$#" )); do
echo '$'"${ITERATION}: ${1}"
shift
(( ITERATION ++ ))
done
Example:
❯ argshow abc\ def zzz
$1: abc def
$2: zzz
With this convention that I'm establishing, here is my first test case:
echo 'abc\ def ghi\njkl'
This represents a value (presented in JSON) of
[["abc def", "ghi"], ["jkl"]]
OK. Proceeding onward.
I first tried this. I ran this in my zsh shell.
❯ echo 'abc\ def ghi\njkl' | while read -r line; do argshow $line; done
$1: abc\ def ghi
$1: jkl
I'm not very happy about that because argshow has received a single argument in the first line when it should be two. The use of -r for read was able to recover the backslash, which is nice (i'll worry about tabs later...).
I ran the same in my bash shell:
$ echo 'abc\ def ghi\njkl' | while read -r line; do argshow $line; done
$1: abc\
$2: def
$3: ghi\njkl
$ echo $'abc\ def ghi\njkl' | while read -r line; do argshow $line; done
$1: abc\
$2: def
$3: ghi
$1: jkl
This is where i've reached the limits of my bash knowledge. None of these are what I expect. Why is it that I must insert an eval in order to make this work like I want?:
$ echo $'abc\ def ghi\njkl' | while read -r line; do eval argshow $line; done
$1: abc def
$2: ghi
$1: jkl
I was under the impression that if I do not use any quotes when placing a bash variable that it would just get expanded and split to whatever it needs to be. After all, this is why shellcheck yells at me when I use shell variables in shell scripts without double-quoting them. I thought they would always get expanded out and so if the variable happened to have say 2 space chars in it, that it becomes 3 arguments instead of the expected 1 argument.
I cannot use this method to make scripts that I use on a regular basis. There are a lot of evil vulnerabilities involved when eval enters the script. It would be tempting because it allows for such free features as placing arguments into quotes, potentially being able to escape newlines, and it is easy to remember the syntax because of familiarity, but it would be far too dangerous to use in a script as it would be trivial to insert an unescaped semicolon to cause code execution.
Perhaps I've truly found the limit of shell scripting and clearly need to implement this with a real programming language. That is fair, but I'm hoping there's some simple option I could use to work around the issue and follow through on this hack.
Yes I also know I can probably do something like pass bash arrays around, but I'm hoping to be able to just dump these data streams out to simple files, that is the entire appeal of this, is that it has a super basic implicitly human readable format.
For more context on the task at hand... I am moving toward a more computer-assisted form of a data management scheme. The idea is that if I use something like a 512GB SD card in my camera/drone/etc. I'd like to be able to keep some older data remaining on the memory card, for convenience purposes, and also for workflows like sometimes I wanna dump some content onto my Macbook to view it more easily. When it comes to migrating this to ZFS pool at my workstation, the issue crops up of which files are already saved so I need not copy them back over and waste space. Sometimes if I fixate on that aspect too much, what happens is failure to pick up on some new stuff that should be backed up. It's always a time-consuming process to vet the remaining contents of a disk (that's probably nearly full) before erasing it to be able to continue to use it.
As such, the approach which I use now is periodically build a storage pool database find /pool /pool2 -type f -printf "%M %s %t %p\\n" > ~/Dropbox/find_zfs, then this database can be used pretty easily for different things. What led me to this question was in the course of writing the following script (I named it trawl):
#!/bin/bash
# provide a list of dirs and for each of their immediate child file and folder names, trawl through
# a standard file listing record (this is the FIRST arg) (the kind I ususally use with fzf) to
# look for any matches. emit human readable output on stderr, and stdout will be a list paths.
# That output will be organized for now as a flat list. The idea here is that the list-of-lists
# structure of it can trivially be recovered by a wrapping script that knows the input args and can
# correlate them back out. Inefficient, but simple.
# TODO TODO TODO implement a sanitizer on the input and do a fuzzy check for that to further
# eliminate manual checking via fzf and suggest matches when they are found
FIRSTARG="$1"
shift
for DIR in "$@"; do
>&2 echo "CHECKING DIR: $DIR"
ls -1 "$DIR" | while read -r CHILD; do
RG_OUT=$(rg -F "$CHILD" "$FIRSTARG")
if [ $? -eq 0 ]; then
>&2 echo "$DIR/$CHILD is found with $(echo "$RG_OUT" | wc -l) hits; skipping!"
else
>&2 printf "NOT FOUND; adding: "
echo "$DIR/$CHILD"
fi
done
done
I also made a recursive version of to more exhaustively check every single file instead of just the first level children of a given directory.
So as you can see, the idea with this is that I can basically implement semi-automated file backup deduplication.
Of course, before someone mentions it, I could potentially just enable deduplication capability for ZFS itself, but it's widely accepted to be only worth the various performance costs it imposes on very narrow use-cases of which (primarily) bulk media backup is not one.
How does this fit in, you might ask, well I wanted a "sync point" which is what trawl implements here, the two axes of the two dimensional structure are: (X) all the individual child files in an input directory, and (Y) the separate input directories.
I did end up side-stepping the two dimensional requirement as you can see with my script's comment, as it turns out not to be such an important point and I can reverse it out after the fact. Indeed if I do want to save it, then it would be most sensible to emit JSON.
With this then, the workflow is now
- plug in disk (if applicable)
trawl ~/Dropbox/find_zfs srcdir/ srcdir2/- visually review the output to confirm it makes sense
- Run it again but piped into
mkdir -p /pool/backup-$(date +%F); while read -r LINE; do mv "$LINE" /pool/backup-$(date +%F); done
It's a bit funny because I do have a bona fide typescript program I wrote to accomplish the same thing for me and it does have the same capability but it's just simply not as natural to implement an interactive and flexible workflow.
argshow $lineshould be splitting$lineon whitespace, since the variable isn't quoted.zshis deviating from POSIX here. That's probably better from a reliability point of view, but it gives rise to a different shell language.evalindicates that you're processing shell syntax. If yourreadcommand is reading shell syntax from its input source, or a fragment of shell syntax, and you want to glue that shell syntax fragment onto a command to create a command line, then you useeval.shell_syntax='$(basename "foo/bar"); eval "echo $shell_syntax"will glue together the textecho $(basename "foo/bar")and then evaluate the whole thing as shell syntax.