4

I have a program that gets the files selected in the graphical IU (in my case Finder in macOS). The output is something like

'/tmp/file number one.txt' '/tmp/file number two.txt'

Note the space char in the names, thus the file names are enclosed in ' (single straight ticks)

When using the output of that command in a command substitution in bash for e.g. the ls -l command everything is screwed up. For a test I put the above line into a simple one-liner text file and use it as command line substitution:

$ cat /tmp/files.txt
'/tmp/file number one.txt' '/tmp/file number two.txt'
$ ls -l $(</tmp/files.txt)
ls: "'/tmp/file: No such file or directory
ls: '/tmp/file: No such file or directory
ls: number: No such file or directory
ls: number: No such file or directory
ls: one.txt': No such file or directory
ls: two.txt'": No such file or directory

The same happens when I assign the file name string to a variable and use it

$ xxx="'/tmp/file number one.txt' '/tmp/file number two.txt'"
$ ls -l $xxx
ls: '/tmp/file: No such file or directory
ls: '/tmp/file: No such file or directory
ls: number: No such file or directory
ls: number: No such file or directory
ls: one.txt': No such file or directory
ls: two.txt': No such file or directory

Any idea how to solve this? Copying the escaped file names right onto the command line works as expected

$ ls -l '/tmp/file number one.txt' '/tmp/file number two.txt'
-rw-r--r--  1 tester  wheel     0B Jul 17 17:21:11 2021 /tmp/file number one.txt
-rw-r--r--  1 tester  wheel     0B Jul 17 17:21:16 2021 /tmp/file number two.txt

My ultimate goal is to use the current Finder selection (which I get through a compiled Applescript) to be available for use in bash. ls is just an example, I might want to use the list of files for tar, cp, mv or any other file handling stuff.

11
  • Your string is undergoing word splitting, according to your $IFS. Are the results really seperated by a space, not by a tab character? – Panki Jul 21 at 11:54
  • 2
    What does the file look like if you select a Don't let me down.mp3 file for instance or one containing newline characters? – Stéphane Chazelas Jul 21 at 13:15
  • 2
    Is that Finder doing that? or some other program? Can you fix/configure whatever it is so that it doesn't output the filenames in such a useless format? NUL separated filenames would be ideal, as that allows for ANY valid filename. – cas Jul 21 at 13:56
  • 1
    @cas has a point. While the format looks good it is surprisingly hard to handle. Even a simple newline would help, what ls does when its output does not go to a terminal. – Peter - Reinstate Monica Jul 21 at 21:52
  • 1
    @Peter-ReinstateMonica, ...that's not wisdom any more than making SQL databases unable to put 's inside strings in wisdom. There's nothing wrong with spaces; there's plenty wrong with using spaces as delimiters. If characters that can't exist in C strings (of which we have a perfectly good candidate, the NUL) are used to delimit lists of C strings, nobody has any problem. Thus, right-thinking people should use printf '%s\0' *.txt to generate machine-readable lists of files whose names end in .txt, and xargs -0 to act on such lists. This is a sh spec problem, not a filename problem. – Charles Duffy Jul 22 at 0:48
6

If switching to zsh is an option¹, you could use its z and Q parameter expansion flags that are designed for that:

file_content=$(</tmp/files.txt)
quoted_strings=(${(z)file_content})
strings_with_one_layer_of_quotes_removed=("${(Q@)quoted_strings}")
ls -ld -- "$strings_with_one_layer_of_quotes_removed[@]"

Or all in one go:

ls -ld -- "${(Q@)${(z)$(</tmp/files.txt)}}"

That assumes the syntax of the quoting in the file is compatible with that of zsh.

See also the Z parameter expansion to tweak how the parsing is done. For instance, if the file contains comments (with #) which should be ignored and has more than one line, you'd want:

ls -ld -- "${(Q@)${(Z[Cn])$(</tmp/files.txt)}}"

See info zsh flags for details.


¹ I hear zsh is now the default interactive shell in newer versions of macos

6

Assume you have this string, literally, with the single quotes embedded:

'/tmp/file number one.txt' '/tmp/file number two.txt'

You noticed it works correctly when given inline as part of a command line, but not when it comes from an expansion. It doesn't really matter if it's variable expansion or a command substitution, the rules are the same for both. Unquoted expansions undergo word splitting, which you don't want here, since splitting on spaces would split between /tmp/file and number. Quoted expansions don't undergo splitting, but you don't want that either, since presumably you want splitting between the two middle single quotes. Also, there's the fact that quotes resulting from expansions don't quote anything. So, we need to do something different.

Assuming the output is known to be shell syntax, and is safe, you can use eval to have the shell do another round of processing to interpret the quotes:

#!/bin/bash
input="'/tmp/file number one.txt' '/tmp/file number two.txt'"
eval "ls -ld -- $input"

or put them in an array for future use:

#!/bin/bash
input="'/tmp/file number one.txt' '/tmp/file number two.txt'"
eval "files=($input)"
for f in "${files[@]}"; do
    printf "<%s>\n" "$f"
done

Note that if the string going to eval contains unquoted or double-quoted command substitutions (e.g. /dir/$(uname -a), but not '/dir/$(uname -a)'), then your shell will execute the commands involved while processing the eval. Similarly if the string contains an unquoted ) to end the array assignment. So, better make sure to only use this with sources that are under your control.

Also, you do need the double quotes around the string being eval'd, because you don't want it split and globbed before eval gets to process the quotes.


There might be ways using other tools to interpret the quotes but not process expansions, e.g. xargs takes quoted strings by default. E.g. this would run printf¹ with each filename as an individual argument:

printf '%s\n' "$input" | xargs printf ":%s:\n"

Or run ls on them:

printf '%s\n' "$input" | xargs ls -ld --

Or you could have xargs run something that then prints the filenames in a simpler format which you can then load to an array in the shell. (This is a bit backward, but I don't know of a way to have Bash just do quote processing without processing expansions.)

#!/bin/bash
readarray -td '' files < <(
  printf '%s\n' "$input" | xargs printf "%s\0")
for f in "${files[@]}"; do
    printf "<%s>\n" "$f"
done

(Here, the printf outputs the strings terminated with NUL bytes, and readarray -td ''² expects output in that format. The NUL being the only value that can't appear in a filename, that's an unambiguous and relatively simple format.)

But note that xargs has a different idea of the exact quoting rules than the shell. It doesn't know about the $'...' style of quoting³, which Bash uses in some cases to output values that contain e.g. embedded newlines, it doesn't recognise backslash inside double quotes4... But if the output from Finder is just single quotes (and backslashes to quote any hard single quotes), you might be fine.


¹ the standalone printf utility, not the printf builtin of your shell, at least once even on empty input (except on some BSDs) and possibly more than once if the list is large

² requires bash 4.4 or above

³ introduced by ksh93 in the 90s

4xargs came with PWB Unix in the late 70s and the quoting syntax is similar to that of the pre-Bourne sh there (Mashey shell), not that of the Bourne shell, let alone ksh93 or bash

2
  • I suppose for commands that work like ls and take file names as end-arguments one can simply feed it with xargs, in the OP's example cat /tmp/files.txt | xargs ls -d. No need to edit the strings first. – Peter - Reinstate Monica Jul 21 at 21:44
  • @Peter-ReinstateMonica, oh yes indeed, that too. – ilkkachu Jul 21 at 21:56
1

Your best option is to fix whatever's generating such useless file lists so that it generates NUL-separated output instead (because a NUL is the only character that can not be in a path/filename, it is the only separator that is guaranteed to handle any filename with any valid characters). If that's impossible, you can kludge up a "fix" by attempting to convert it to NUL-separated format.

The following perl one-liner will (mostly) convert the file to NUL separated filenames, without quotes surrounding them:

perl -0 -pe "s/'\s+'/\0/sg; s/^'|'\$//sg; s/\x0d?\x0a\$//" file.txt

The first regex replaces the sequence single-quote, one-or-more whitespace chars, single-quote with NUL characters (the commas and spaces in that are not part of the pattern, they're just grammatical English list separators). The second regex removes the quotes at the beginning and end of the input, and the third removes LF or CRLF at the end of a "line".

This is very far from perfect - some input is un-fixable because there's no way to know with 100% certainty whether a single-quote or a LF is supposed to be embedded in the filename or not (this is why starting with NUL-separated files is the correct solution, not trying to kludge it after the fact).

For example, it will fail if any filenames have an embedded single-quote at either the beginning or end of the filename, or if they have an embedded single-quote followed by one-or-more whitespace characters and followed by another single-quote (e.g. ' ') - all of these will also be replaced with a NUL because of the /g global modifier to the first regex (which is necessary for it to match all filenames in the input instead of just the first). And probably a few other corner-case I haven't thought of yet.

You can redirect the output to another file, feed it into xargs -0r, or use it with the bash built-in readarray and process substitution to populate an array:

readarray -d '' files < <(perl -0 -pe "s/'\s+'/\0/sg;
                                       s/^'|'\$//sg;
                                       s/\x0d?\x0a\$//" file.txt)

If you pipe the output into xxd (or hd or hexdump or similar hex-dumper program), you can see that it has become NUL-separated:

00000000: 2f74 6d70 2f66 696c 6520 6e75 6d62 6572  /tmp/file number
00000010: 206f 6e65 2e74 7874 002f 746d 702f 6669   one.txt./tmp/fi
00000020: 6c65 206e 756d 6265 7220 7477 6f2e 7478  le number two.tx
00000030: 74                                       t

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.