2

I have written a bash script with has a python code embedded within it. What I did was read in the file and cut it with the below bash commands and then I exported it so that the python code could use it:

export BWA_compare=$(cut --output-delimiter='' -f1-3 ${BWA_AT}| sed 1d)

The problem is I get 

/bin/cut: Argument list too long
/bin/sed: Argument list too long

It works on smaller files. Is there a limit on how much you can export?

Improve this question
4
  • The sed error doesn't make sense to me as the argument list contains just one element.
    – Hauke Laging
    Nov 1, 2014 at 21:30
  • I have seen cases where in script the "Argument list too long" is raised against a line or command which is close to but which is not the problem. e.g. while loop read -r line which had echo $line |sed yadda; STRING+=stuff; "Argument list too long" was highlighted against the sed lines whereas the actual problem was STRING had grown too large. And STRING was not used in the sed lines.
    – gaoithe
    Jan 19 at 11:04
  • What I'm trying to say is the error raised against sed and cut is a bit misleading. I think the error is that BWA_compare var is being assigned something that is too big.
    – gaoithe
    Jan 19 at 11:05
  • see also stackoverflow.com/questions/45863374/… - one variable set in environment which was set too large caused all commands to fail. See in particular @chepner suggestion to read in-ulm.de/~mascheck/various/argmax
    – gaoithe
    Jan 19 at 11:20

1 Answer 1

Reset to default
2

There are limits for both the number of arguments and the combined length of the arguments. These are not uniform values but depend of the system.

Usually such problems are solved with xargs.

But it seems to me that you are using cut wrongly. cut expects only files are arguments. Is ${BWA_AT} a huge list of files? If it is the Python code then you need something like

echo "${BWA_AT}" | cut

If it really is a file list then you need something like

echo "${BWA_AT}" | xargs cut

with the needed xargs options depending on the separator.

Improve this answer
2
  • 2
    The limit is on the cumulative size of the arguments and environment passed to execve(2). There's also a limit on the size of single argument or env var string. For instance, after export "a=$(yes | head -c129k)", you won't be able to run any command on current versions of Linux.
    – Stéphane Chazelas
    Nov 1, 2014 at 22:16
  • ${BWA_AT} is just the file location. I was thinking that by exporting all the data that cut has generated I would run into a limit which produced those two errors.
    – crysis405
    Nov 4, 2014 at 14:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.