Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have seen Bash scripting guides suggesting the use of array for working with filenames containing whitespace. DashAsBinSh however suggests that arrays are not portable so I am looking for a POSIX compliant way of working with lists of filenames that may contain whitespace.

I am looking to modify the below example script so that it would echo

foo/target/a.jar
foo/target/b.jar
bar/target/lol whitespace.jar

Here is the script

#!/usr/bin/env sh

INPUT="foo/target/a.jar
foo/target/b.jar
bar/target/b.jar
bar/target/lol whitespace.jar"
# this would be produced by a 'ls' command
# We can execute the ls within the script, if it helps

dostuffwith() { echo $1; };

F_LOCATIONS=$INPUT
ALL_FILES=$(for f in $F_LOCATIONS; do echo `basename $f`; done)
ALL_FILES=$(echo "$ALL_FILES" | sort | uniq)

for f in $ALL_FILES
do
    fpath=$(echo "$F_LOCATIONS" | grep -m1 $f)
    dostuffwith $fpath
done
share|improve this question

3 Answers 3

up vote 1 down vote accepted

POSIX shells have one array: the positional parameters ($1, $2, etc., collectively refered to as "$@").

set -- 'foo/target/a.jar' 'foo/target/b.jar' 'bar/target/b.jar' 'bar/target/lol whitespace.jar'
set -- "$@" '/another/one at the end.jar'
…
for jar; do
  dostuffwith "$jar"
done

This is inconvenient because there's only one, and it destroys any other use of the positional parameters. Positional parameters are local to a function, which is sometimes a blessing and sometimes a curse.

If your file names are guaranteed not to contain newlines, you can use newlines as the separator. When you expand the variable, first turn off globbing with set -f and set the list of field splitting characters IFS to contain only a newline.

INPUT="foo/target/a.jar
foo/target/b.jar
bar/target/b.jar
bar/target/lol whitespace.jar"
…
set -f; IFS='
'                           # turn off variable value expansion except for splitting at newlines
for jar in $INPUT; do
  set +f; unset IFS
  dostuffwith "$jar"        # restore globbing and field splitting at all whitespace
done
set +f; unset IFS           # do it again in case $INPUT was empty

With items in your list separated by newlines, you can use many text processing commands usefully, in particular sort.

Remember to always put double quotes around variable substitutions, except when you explicitly want field splitting to happen (as well as globbing, unless you've turned that off).

share|improve this answer
    
Good answer and explanation. I'm going to mark this as accepted because this makes the original sort | uniq step work as intended. –  Eero Aaltonen Dec 9 '13 at 9:52

Since your $INPUT variable uses newlines as separators, I'm going to assume that your files will not have newlines in the names. As such, yes, there is a simple way of iterating over the files and preserving whitespace.

The idea is to use the read shell builtin. Normally read will split on any whitespace, and so spaces will break it. But you can set IFS="\n" and it will instead split on newlines only. So you can iterate over each line in your list.

Here's the smallest solution I could come up with:

INPUT="foo/target/a.jar
foo/target/b.jar
bar/target/b.jar
bar/target/lol whitespace.jar"

dostuffwith() {
    echo "$1"
}

echo "$INPUT" | awk -F/ '{if (!seen[$NF]++) print }' | \
while IFS="\n" read file; do
  dostuffwith "$file"
done

Basically it sends "$INPUT" to awk which deduplicates based on the file name (it splits on / and then prints the line if the last item hasn't been seen before). Then once awk has generated the list of file paths, we use while read to iterate through the list.

share|improve this answer
    
$ checkbashisms bar.sh possible bashism in bar.sh line 14 (<<< here string) –  Eero Aaltonen Nov 28 '13 at 12:12
    
@EeroAaltonen Changed it to not use the herestring. Note though that with this change, the while loop, and thus dostuffwith is executed in a subshell. So any variables or changes made to the running shell will be lost when the loop completes. The only alternative is to use a full heredoc, which isn't that unpleasant, but I thought this would be preferable. –  Patrick Nov 28 '13 at 22:28
    
I'm awarding points based more on readability than smallness. This certainly works and already +1 for that. –  Eero Aaltonen Nov 29 '13 at 9:21

I think the best way is to use bracket around your variable like this :

dostuffwith ${fpath}

But I can't garantee it's posix but it's working on AIX ksh and Linux bash I just test that. So if it's not posix compliant it still have lot of chance to work on several system.

share|improve this answer
    
Doesn't seem to work on my Ubuntu terminal. –  Eero Aaltonen Nov 28 '13 at 9:54
    
if you copy path be carefull I misstype fpath. but if you declare a variable in a terminal like PONY=FLUFFY and you then do a echo echo ${PONY} it does work normally –  Kiwy Nov 28 '13 at 9:56
    
Yeah I noticed. I also tried adding bracket to every variable reference, but more is required since there is also the sort | uniq stage. –  Eero Aaltonen Nov 28 '13 at 9:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.