4

My shell script look like this:

#!/bin/bash

for file in *.fasta
do

signalp $file > $file.txt

done

In the working folder I have 18.000 .fasta files. I want to run each through the signalp program. I guess its too many files in the folder, but I don't know how to adjust my code. Any help?

  • 2
    I'm having a hard time understanding how your for loop could produce an "argument list too long"... There is no such thing, virtual memory is the only limit here... – don_crissti Jul 12 '15 at 23:24
  • 1
    @don_crissti - it's gotta be the quotes. – mikeserv Jul 13 '15 at 4:35
  • Note that "Argument list too long" is just glibc's error message for the E2BIG error code. The error code doesn't actually mean what the message says. in the Linux kernel (for example), it can happen if an OOM occurs while copying the argument. Even if the argument list requires just one page, if that page happens to be unavailable. – Kaz Jul 13 '15 at 5:37
  • Another limit in Linux's exec is a limit on individual argument length: #define MAX_ARG_STRLEN (PAGE_SIZE * 32). Exceed this in any one argument, and ... E2BIG: you get a misleading message about the list being too long, rather than an item on the list being too fat. – Kaz Jul 13 '15 at 5:42
  • Lastly, Linux's exec uses some common code for environment and argument handling. So "Argument list too long" could actually be a problem encountered in copying environment variables and not arguments. – Kaz Jul 13 '15 at 5:44
6

You can use find :

find . -maxdepth 1 -type f -exec sh -c 'signalp "$1" >"$1".txt' _ {} \;
  • -maxdepth 1 will make find to search for files (-type f) only in the current directory

  • sh -c 'signalp "$1" >"$1".txt' will execute the signalp command on all the files found and save the output to the files named after adding .txt to the original filenames.

  • Correct? #!/bin/bash find . -maxdepth 1 -type f -exec sh -c 'signalp "$1" >"$1".txt' _ {} \; – user2862862 Jul 12 '15 at 22:59
  • Saved it as "command.sh". Then typed "sh command.sh" in terminal. What are these symbols btw? _ {} \; – user2862862 Jul 12 '15 at 23:00
  • @user2862862 this is a one liner..you can save it if you want though..{} is a placeholder used by find to indicate the files found.._ is used because sh -c treats the arguments after the command starting from positional parameter 0..so here _ is treated as positional parameter 0 and the file {} can be considered as positional parameter `. – heemayl Jul 12 '15 at 23:04
1

You're getting an argument list too long error because you are not quoting your arguments. There's an expansion occurring - though it is difficult to say definitely what it is - on the value of $file which generates more arguments. My theory is one of your filenames contains another * which again expands out to match all of your matched files all over again.

You can do this in the shell - and you don't need to invoke a whole new shell via find to do it either.

Just do this:

for f in ./*.fasta
do  signalp "$f" >"$f.txt"
done

...see? The double-quotes will prevent the contents of the iterable shell variable $f from being interpreted any way but literally - even if $f does contain expandable metacharacters of kind.

Granted, though, this is not the best solution for groups of 18k files. It will work, but it would be better if you could batch it further.

As an example, let's assume there is some kind of order to the filenames themselves. Maybe they're named like...

aaa001.fasta
...
bbb001.fasta

And so on. In that case you could do something like:

for l in a b c d e f g h i j k l m n o p q r s t u v x y z
do    for  f in "./$l$l$l"*.fasta
      do   singalp "$f" >"$f.txt"
done; done

...and so you wouldn't need to retain the whole 18k list in memory for the duration of the loop.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.