/bin/ls: Argument list too long

Question

I am a biology person and running a program named autodock. I have some files from ZINC library in .mol2 format. As per requirement I need to split this files with the csplit command and I received all the content in my directory. The parent file was split into very very many small files. Every file name is like this: ZINC14382748.mol2. Now I have to change all these files to pdbqt format and I have to use the following script:

#!/bin/csh # # $Id: ex02.csh,v 1.5 2007/07/19 21:52:59 rhuey Exp $ 
# 
# use the 'prepare_ligands.py' python script to create pdbq files 
cd $VSTROOT/VirtualScreening/Ligands 
foreach f (`ls *`) echo $f pythonsh ../../prepare_ligand4.py -l $f -d ../etc/ligand_dict.py end 

When I use it, it says

/bin/ls: Argument list too long

In short upon successful completion it will duplicate the above number of files into another format. So is there any reasonable solution to tackle this problem?

Answer 1

1. Don’t parse the output of ls.  Just say foreach f (*).  Also,
2. You should always quote your shell variable references (e.g., "$f") unless you have a good reason not to, and you’re sure you know what you’re doing.

Answer 2

The source of the problem is that you have far too many small files.

If I'm reading it right, you have over 14 million files. There is no way that ANY shell is going to be able to have over 14 million file names on the command line. Aside from that. your filenames seem to be about 18 characters long, so that's roughly 18*14M or about 252 megabytes just to hold the filenames.

bash for example, has a limit of 128KB. ever so slightly smaller than 252MB. I have no idea what limit csh has because i don't use it. It's unlikely to be any larger than bash's command-line length limit. It certainly won't be 252MB or greater.

However, all is not lost, you can use find ... -exec instead.

find . -maxdepth 1 -type f -name '*.mol2' \
  -exec pythonsh ../../prepare_ligand4.py -l {} -d ../etc/ligand_dict.py \;

This will run prepare_ligand4.py ONCE for each file, so will take a very long time. You might be able to speed it up a little (not much, not with 14+M files to process) by using find ... -print0 with xargs -0 -P ... or GNU parallel -0 ... instead of find ... -exec

---

A much better solution would be to download source code for prepare_ligand4.py and modify it so that you can give it one large file (e.g. the original file before csplit-ing it) and it will process each block individually. It'll be much faster and easier to work with. You'll probably still have over 14M output files (assuming that a combined output file would be useless...if it isn't you're in luck!), but that's better than having 14M input files AND 14M output files.

This would, of course, require some skill with python programming.

Maybe someone has already encountered the same problem and written their own enhanced version of prepare_ligand4.py. It's worth spending some time searching, or maybe try the Autodock Forum or even contact the Autodock author.

Answer 3

You obviously has a lot of files. Consider using GNU Parallel http://www.gnu.org/software/parallel/ The 'ls -U' does not sort the files and then it is faster.

cd $VSTROOT/VirtualScreening/Ligands
ls -U ZINC* | parallel echo {} \; pythonsh ../../prepare_ligand4.py -l {} -d ../etc/ligand_dict.py

I don't understand why you echo it. Do you parse it on to a new script? My guess is that 'prepare_ligand4.py' is the script for conversion and then this should do the job (in parallel):

cd $VSTROOT/VirtualScreening/Ligands
ls -U ZINC* | parallel pythonsh ../../prepare_ligand4.py -l {} -d ../etc/ligand_dict.py

Answer 4

I have solved the problem, let me share with you. I Rename the bash.csh to bash.sh, Next I change my script in order to run it in bash. Here is my new script to help in future for the same issue.

#!/bin/bash
cd $VSTROOT/VirtualScreening/Ligands/
for f in ZINC*.mol2
do
    echo "$f"
    pythonsh ../../prepare_ligand4.py -l "$f" -d ../etc/ligand_dict.py
done

For a beginner like me, here ZINC is a part of name present in all of the ligand names, so must keep according to your ligand name. Thanks for you time and my friend here who helped me passionately.