1

Say I have an executable script process_image that performs actions on a base 64 encoded image. I am storing every image in a file images_file line by line. Every line of images_file is a base 64 encoded image. Some of the lines are very long therefore the following returns xargs: argument line too long:

cat images_file | xargs -L1 process_image

I wanted to modify process_image to take the entire stdout from cat images_file and then loop over each line using a simple while loop, but my colleagues have advised against this approach. Does xargs -L1 also internally use the same mechanism as while? How would using xargs be more desirable than using a while? What is the maximum argument length that xargs can handle and is there any way to overcome this while maintaining the cat <file> | xargs -L1 <executable_script> approach?

17
  • 1
    @sriganesh Why not run a simple shell loop to circumvent this problem, instead of using xargs? Try dumping the value of ARG_MAX, using getconf ARG_MAX
    – Inian
    Commented Jul 14, 2022 at 7:37
  • 1
    Linux has a hard-coded length limit for a single command line argument, and you might be crashing into that. See What defines the maximum size for a command single argument? The solution in that case, such as it is, would be to switch another Unix without that limit. Or to pass data like that using files or pipes instead of command line arguments...
    – ilkkachu
    Commented Jul 14, 2022 at 7:38
  • 1
    I asked about process_image rather than images_file, but if you're set on xargs, okay. I don't see anything in the xargs man page that suggests -L1 would help you, but I do see -s somelargenumber might help. (although the man page says the default value is ARG_MAX - 4096, so there doesn't seem to be a lot of space to gain)
    – Sotto Voce
    Commented Jul 14, 2022 at 7:41
  • 1
    @sriganesh have your colleague given a reason for advising against having process_image loop over input?
    – muru
    Commented Jul 14, 2022 at 7:53
  • 1
    There are many people who have this idea that while loops are somehow bad, but they are wrong. This is classic cargo cult programming: shell loops are a bad tool to process text files, so then people think they're a bad tool in general. There is nothing wrong with using a shell loop if it is the right tool for the job.
    – terdon
    Commented Jul 14, 2022 at 8:40

0

You must log in to answer this question.

Browse other questions tagged .