7

I'm trying to execute this command

sed -i -e "s/BASE_64/$BASE_64/" FILE_NAME

where $BASE_64 is a base 64 representation of a file content.

sed gives me an error since the string is too long.

Argument list too long

How is it possible to avoid this error?

4

First, save the base64-encoded data in a file called, e.g., base64.txt.

For example:

base64 < originalfile > base64.txt

Then:

printf '%s\n' '/BASE64/r base64.txt' 1 '/BASE64/d' w | ed FILENAME

This uses ed to search in FILENAME for a line containing the string BASE64, insert the contents of base64.txt after that line, go back to the first line, then search for the line with string BASE64 again and delete it. The w command in ed saves the modified file.

  • What if BASE64 is just part of the line though and you don't want to replace the entire line? – Eric Renouf May 20 '16 at 11:34
  • base64 text tends to be multi-line (many lines), so inserting it into the middle of a line is kind of pointless. I'd do the /BASE64/r base64.txt insert as above, and then s/BASE64// rather than /BASE64/d. – cas May 20 '16 at 11:55
  • That may be, though one short example where it would break it to have the encoded data on the next line is if you were populating the Authorization header for an HTTP request using Basic authentication, then you'd have a short base64 encoded string that would be substituted in place, given the problem at hand though, that's probably not the situation in this case – Eric Renouf May 20 '16 at 13:04
  • In a case like that, a) it would be extremely unlikely to cause an "Argument list too long error" like the OP reported and b) I'd use a method more appropriate to the task at hand...a simple s/BASE_64/$BASE_64_TEXT/ with sed or ed or perl or whatever would probably suffice. – cas May 20 '16 at 13:08
4

Another option would be to replace sed with ed and store your commands in a file. For example, if you create ed_cmds with the following contents:

%s/BASE_64/<expanded variable>/g
w
q

you could then run

< ed_cmds ed FILE_NAME

and it would make the changes you wanted, so instead of setting $BASE_64 you'd create the ed command file.

Ed Explanation

  • % means to apply the command to each line of the file
  • s/pat1/pat2/g substitutes occurrences of pat1 with pat2 and g at the end makes it do it for every match on the line, not just the first
  • w write the changes to disk
  • q quit (which would happen when it got EOF anyway)

Of course, you could put your sed commands in a file and use -f as well, but if you're doing that and you want to modify the file in place you might as well use ed instead of creating a temporary file and moving it as sed -i does.

3

You could always do (since you're using GNU sed already (-i)):

sed -i -f - FILE_NAME << EOF
s/BASE_64/$BASE_64/g
EOF

-f - tells sed to read the sed script from stdin.

If you want to reuse the same script for several files, on Linux (and Linux only), with a shell like bash, zsh, ksh that implements here documents with temporary files (as opposed to pipes like dash or yash) and still with GNU sed, you could do:

find . -name '*.conf' -exec sed -i -f /dev/stdin {} + << EOF
s/BASE_64/$BASE_64/g
EOF

On Linux (and Linux only), /dev/stdin does not mean stdin in the same way - does. Instead, it's a symlink to the file open on stdin, so each time sed opens it, it opens the file anew from the start. The above command would work OK on other systems (that have /dev/stdin) or with shells that implement here-documents with pipes but only if there are few enough conf files that sed is called only once. When called the second time, on non-Linux systems, like with -f -, /dev/stdin would appear empty because it has already been read by the first invocation.

busybox sed also supports -i in the same way as GNU sed does, but does not support -f -. So you'd want to use -f /dev/stdin there in any case. With FreeBSD sed, use:

sed -i '' -f /dev/stdin FILE_NAME << EOF
s/BASE_64/$BASE_64/g
EOF
2

The size of the Base64 representation of the file ($BASE_64) is too long and exceeds maximum argument size. You should be able to see this limit for your system by running

getconf ARG_MAX

You have to increase the size of the ARG_MAX value. But I think if the file is too big, then you will have to use a different approach to do this replacement. If a Python script would do it for you, i'd try it with that too.

  • thanks. I will go with a Python script if there is no other solution. – Lorenzo B May 19 '16 at 13:51
  • Note that Linux in particular has a hard-coded limit for the length of a single command line argument, so increasing ARG_MAX doesn't even help there. – ilkkachu Dec 3 '17 at 19:54
1

I ended up putting the sed instructions into a file

SEDCOMMANDS=`tempfile`

and called

sed -f "$SEDCOMMANDS" -- "$FILE_NAME"

That's good if you don't use sed -i. If you want to edit the file in place, follow https://unix.stackexchange.com/a/284188/149867 and put the equivalent ed instructions in a file, followed by w and q.

  • -i works fine with -f. Only, make sure you use -i -f (or -i '' -f with BSD sed), not -if. You can also use -f - to avoid the temp file. – Stéphane Chazelas Feb 26 '17 at 8:36

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.