12

I'm trying to execute this command

sed -i -e "s/BASE_64/$BASE_64/" FILE_NAME

where $BASE_64 is a base 64 representation of a file content.

sed gives me an error since the string is too long.

Argument list too long

How is it possible to avoid this error?

6 Answers 6

12

You could always do (since you're using GNU sed already (-i)):

sed -i -f - FILE_NAME << EOF
s/BASE_64/$BASE_64/g
EOF

-f - tells sed to read the sed script from stdin.

If you want to reuse the same script for several files, on Linux (and Linux only), with a shell like zsh, ksh and bash version up to 5.0 that implements here documents with temporary files (as opposed to pipes like dash or yash (or bash 5.1+) for relatively small heredocs and still with GNU sed, you could do:

find . -name '*.conf' -exec sed -i -f /dev/stdin {} + << EOF
s/BASE_64/$BASE_64/g
EOF

On Linux (and Linux and Cygwin only), /dev/stdin does not mean stdin in the same way - does. Instead, it's a symlink to the file open on stdin, so each time sed opens it, it opens the file anew from the start. The above command would work OK on other systems (that have /dev/stdin) or with shells that implement here-documents with pipes but only if there are few enough conf files that sed is called only once. When called the second time, on non-Linux/Cygwin systems, like with -f -, /dev/stdin would appear empty because it has already been read by the first invocation.

busybox sed also supports -i in the same way as GNU sed does, but does not support -f -. So you'd want to use -f /dev/stdin there in any case. With FreeBSD sed, use:

sed -i '' -f /dev/stdin FILE_NAME << EOF
s/BASE_64/$BASE_64/g
EOF
7

First, save the base64-encoded data in a file called, e.g., base64.txt.

For example:

base64 < originalfile > base64.txt

Then:

printf '%s\n' '/BASE64/r base64.txt' 1 '/BASE64/d' w | ed FILENAME

This uses ed to search in FILENAME for a line containing the string BASE64, insert the contents of base64.txt after that line, go back to the first line, then search for the line with string BASE64 again and delete it. The w command in ed saves the modified file.

4
  • 1
    What if BASE64 is just part of the line though and you don't want to replace the entire line? May 20, 2016 at 11:34
  • base64 text tends to be multi-line (many lines), so inserting it into the middle of a line is kind of pointless. I'd do the /BASE64/r base64.txt insert as above, and then s/BASE64// rather than /BASE64/d.
    – cas
    May 20, 2016 at 11:55
  • That may be, though one short example where it would break it to have the encoded data on the next line is if you were populating the Authorization header for an HTTP request using Basic authentication, then you'd have a short base64 encoded string that would be substituted in place, given the problem at hand though, that's probably not the situation in this case May 20, 2016 at 13:04
  • In a case like that, a) it would be extremely unlikely to cause an "Argument list too long error" like the OP reported and b) I'd use a method more appropriate to the task at hand...a simple s/BASE_64/$BASE_64_TEXT/ with sed or ed or perl or whatever would probably suffice.
    – cas
    May 20, 2016 at 13:08
4

Another option would be to replace sed with ed and store your commands in a file. For example, if you create ed_cmds with the following contents:

%s/BASE_64/<expanded variable>/g
w
q

you could then run

< ed_cmds ed FILE_NAME

and it would make the changes you wanted, so instead of setting $BASE_64 you'd create the ed command file.

Ed Explanation

  • % means to apply the command to each line of the file
  • s/pat1/pat2/g substitutes occurrences of pat1 with pat2 and g at the end makes it do it for every match on the line, not just the first
  • w write the changes to disk
  • q quit (which would happen when it got EOF anyway)

Of course, you could put your sed commands in a file and use -f as well, but if you're doing that and you want to modify the file in place you might as well use ed instead of creating a temporary file and moving it as sed -i does.

1
  • can you pls give me example for above solution I am having same issue using sed.As I am new to shell dont know how to create ed_cmds file Oct 15, 2019 at 12:59
2

The size of the Base64 representation of the file ($BASE_64) is too long and exceeds maximum argument size. You should be able to see this limit for your system by running

getconf ARG_MAX

You have to increase the size of the ARG_MAX value. But I think if the file is too big, then you will have to use a different approach to do this replacement. If a Python script would do it for you, i'd try it with that too.

2
  • thanks. I will go with a Python script if there is no other solution.
    – Lorenzo B
    May 19, 2016 at 13:51
  • Note that Linux in particular has a hard-coded limit for the length of a single command line argument, so increasing ARG_MAX doesn't even help there.
    – ilkkachu
    Dec 3, 2017 at 19:54
2

I ended up putting the sed instructions into a file

SEDCOMMANDS=`tempfile`

and called

sed -f "$SEDCOMMANDS" -- "$FILE_NAME"

That's good if you don't use sed -i. If you want to edit the file in place, follow https://unix.stackexchange.com/a/284188/149867 and put the equivalent ed instructions in a file, followed by w and q.

1
  • -i works fine with -f. Only, make sure you use -i -f (or -i '' -f with BSD sed), not -if. You can also use -f - to avoid the temp file. Feb 26, 2017 at 8:36
1

I felt like I would share what i've done since i've read your ideas before I made my own code :

my original code :

local DOCUMENTCONTENT=$(base64 -w 0 $fichier)

sed -e 's&{DOCUMENTCONTENT}&'"$DOCUMENTCONTENT"'&g;s& 
{FILENAME}&'"$basename_fichier"'&g;s& 
{INTERFACETYPECODE}&'"$p_InterfaceTypeCode"'&g;s& 
{INTERFACEPARAMETERLIST}&'"$p_InterfaceParameterList"'&g;s& 
{JOBNAME}&'"$p_JobName"'&g' template_ImportOperation.json > 
ImportOperation.json

where $DOCUMENTCONTENT is a zip file of 3.2 Mb.(22Mb uncompressed) sed was failing with arguments too long as well.

new code : instead of a variable, I use a file : ($fichier is my input file). I use awk and tell it to read from my file instead of from my variable

base64 -w 0 $fichier > "ImportOperation.b64"

sed -e 's&{FILENAME}&'"$basename_fichier"'&g;s& 
{INTERFACETYPECODE}&'"$p_InterfaceTypeCode"'&g;s& 
{INTERFACEPARAMETERLIST}&'"$p_InterfaceParameterList"'&g;s& 
{JOBNAME}&'"$p_JobName"'&g' template_ImportOperation.json > 
ImportOperation.json.2

awk 'BEGIN { getline replacement < "ImportOperation.b64" } {gsub(" 
{DOCUMENTCONTENT}", replacement); print}' ImportOperation.json.2 > 
ImportOperation.json

rm ImportOperation.json.2

That may not be the cleanest of codes, but it works. hope that helps.

1
  • I don't remember the 'sed' rules right now for line-ending, but anything that would break up the substitution into multiple lines would improve clarity and thus usefulness. There would be no measurable effect on speed. The same goes for the 'awk'. Dec 13, 2023 at 17:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .