Find fullpath and filename under a directory then pass to an executable file as arguments

Question

I'd like to find fullpath and filename of all .txt under a directory, and pass to a executable file ./thulac .

It cost me some time to reach:

find /mnt/test -name "*.txt" -print0 |xargs -l bash -c './thulac < $0' 

But this only finds fullpath.

From xargs with multiple arguments , I see:

echo argument1 argument2 argument3 | \
   xargs -l bash -c 'echo this is first:$0 second:$1 third:$2' | xargs

What I want to achieve is something like:

find /mnt/test -name "*.txt" -print0 -printf "%f" | \
   xargs -0 bash -c './thulac < $0 > $1' 

Though here, xargs can not correctly split -print0 -printf "%f" as two arguments when there are multiple files, which stuck me.

---

Example:

find /mnt/test -name "*.txt" -print0 -printf "%f" | \
   xargs -0 -I bash -c './thulac < $0 > /mnt/tokenized/$1'

1. If /mnt/test only has one file the above command does work.

2. But if /mnt/test has more than one file (no matter what the language):

   [root@localhost THULAC]# ls /mnt/test
   test33.txt  test.txt
   [root@localhost THULAC]# find /mnt/test -name "*.txt" -print0 -printf "%f" | \
       xargs -0 bash -c './thulac < $0 > /mnt/tokenized/$1'
   /mnt/test/test.txt: /mnt/tokenized/test.txt/mnt/test/test33.txt: No such file or directory
   

   As you see, xargs mixes two paths together /mnt/tokenized/test.txt/mnt/test/test33.txt , which leads to the error No such file or directory.

How to make it work?

Answer 1

find /tmp/test -name '*.txt' \
 -exec bash -c './thulac < "$(readlink -f {})" > "/mnt/tokenized/$(basename {})"' \;

Use find to search for files and to execute commands on the results. With bash -c 'command' you are able to execute multiple $().

Use readlink -f {} to create the full path to the result.

Use basename {} to strip the path from the result.

Answer 2

When working with xargs you should always test your solutions with input starting with '-' and containing double space, ' and " because xargs is infamous for dealing badly with those:

mkdir -- '-"  '"'"
seq 10 > ./-\"\ \ \'/'-"  '"'".txt

Here is a solution using GNU Parallel:

find . -name "*.txt" -print0 |parallel  -0 ./thulac '<' {} '>' {/}

The < and > need to be quoted as they will otherwise be interpreted by the shell that starts parallel. We want them instead to be interpreted by the shell started by parallel.

Answer 3

find /mnt/test -name "*.txt" -print0 -printf "%f\0" |
xargs -0 -n 2 bash -c 'shift $1; ./thulac < $1 > /mnt/tokenized/$2' 2 1

You want to pass the full-path name also with the null delimiter, so that when the time comes for xargs to dismantle the null-delimited list it can do so in a correct manner.

Otherwise, what will happen is that the one file's full path name is going to be merged into the next file's basename, the phenomenon that you observed in case of multiple filenames!

And then you need to feed 2 arguments at a time to the bash alligator, otherwise it will consume as many as are allowed to it but it passes on only the first two to your executable ./thulac.

A better alternative is to dispense with xargs & do all your work in find, since as it is, xargs is dealing with 2 arguments at a time, which takes away any advantages of xargs. In this version, we supply the full path name to bash and have the filename computed by bash itself rather than rely on find to do it.

find /mnt/test -name "*.txt" -exec bash -c './thulac < "$1" \
  > "/mnt/tokenized/${1##*/}"' {} {} \;

Genesis of the problem

1. Good case when only 1 file present
-print0  -printf '%f'

 /mnt/test/test.txt\0test.txt
 |-----------------|--------|

arg0 = /mnt/test/test.txt
arg1 = test.txt
bash -c 'thulac < $0 > /mnt/tokenized/$1'
thulac < /mnt/test/test.txt > /mnt/tokenized/test.txt

2. Error case when > 1 file present
-print0  -printf '%f'
/mnt/test/test.txt\0test.txt/mnt/test/test33.txt\0test33.txt
|-----------------|-----------------------------|----------|

arg0 = /mnt/test/test.txt
arg1 = test.txt/mnt/test/test33.txt
arg2 = test33.txt
bash -c 'thulac < $0 > /mnt/tokenized/$1'
thulac < /mnt/test/test.txt > /mnt/tokenized/test.txt/mnt/test/test33.txt

Fix

We saw that the mixup occurred due to the absence of the delimiter '\0' in the -printf "%f"
So the correct way is:
find ... -print0 -printf "%f\0" | xargs ...
Ensuring that the list is partitioned at the right places and the 
sequence of fullpath1+file1\0fullpath2+file2\0... is maintained.

Now coming to the 'xargs' part, we write:
xargs -0 -n 2 bash -c '...' 2 1

Points to observe are the following:
   a) '-0' => arguments to xargs will be taken to be NULL separated.
   b) -n 2 => we feed 2 args at a time to bash from the total pool 
      delivered to xargs by find.
   c) 2 1 is just a best practice to get over different shell's behavior
      regarding what construes as $0, $1, $2, ...; In your particular case since you
      already know that $0 -> first arg, $1 -> 2nd arg, we could just as well have
     written what you did:
    find ... | xargs -0 -n 2 bash -c './thulac < $0 > /mnt/tokenized/$1'

Answer 4

It is your find command that has a problem.
To separate the two names include space in the printf format

find /mnt/test -name "*.txt"  -print0 -printf " %f\n"
                                               ^ ( note the space above)

Answer 5

You do not tell precisely what your script need to achieve but assuming you want to pass each odd file as first argument and each even file name as second argument, here is how to do it a portable way:

t=$(mktemp)
find /tmp/test -name "*.txt" -exec sh -c '
    if [ -s $1 ]
    then
        ./thulac < "$(<$1)" > "/mnt/tokenized/$2"
    else
        printf "%s" "$2" > "$1"
    fi' sh $t {} \;
rm $t

Should you just want pass the path and the filename of every file found, the answer is simpler, still only using portable commands and syntax (POSIX), i.e. not depending on bash, GNU find and GNU xargs:

find /tmp/test -name "*.txt" -exec sh -c '
    ./thulac < "$1" > "/mnt/tokenized/$(basename "$1")"' sh {} \;

Note that {} need only to be quoted when using the fish shell, a very unlikely situation.