sed command giving Argument List too long error

Question

I am new to shell scripting and working on script. In which I need to create a html report and attach this html report content in json request filed.Then we are using that request to call an API.

I have Json request as Template in the template have a place holder called CONTENT in which I have to replace generated HTML content after encode it using base64.

When I try to add the encoded string in my template using both AWK and sed im getting Argument list is too long error. As my generated HTML file is having 40K lines.

With below approaches I am getting same error.

encStr="$(cat ./myreport.html | base64)"
#AWK-Approach

awk -v var="$encStr" '{gsub("CONTENT", var, $0); print}' /path/totemplate > output.tmp

#SED-Approach

sed -i "s|CONTENT|$encStr|" output.tmp

There is a suggestion to change sed to ed and put the commands in separate file. As per https://unix.stackexchange.com/a/284188/149867 but when I create a seprate .sh file and include it in my current script its not working. Any detailed example for this approach or any other approach.

Answer 1

If CONTENT is on its own line then put your replacement into a file tmp and then use

sed -e '/CONTENT/r tmp' -e '/CONTENT/d' inp

which first inserts the contents of the filetmp after your match and then deletes the line with CONTENT

(the answer in @philippos comment doesn't work for me, the file has to be injected after a match and then the match deleted)

Answer 2

You're already using a temp file output.tmp

Rather than trying to hold all the content all in memory and then modify it (i.e. reading in the template and then substituting the placeholder)

I would try writing the sections to the temp file in the order you need them in the output.

i.e. write the first part of the JSON request template to output.tmp using whatever language you're generating it with.

then do base64 ./myreport.html >> output.tmp to append the content,

then append the rest of the JSON request template to output.tmp after the content.

No substitution required, much less memory required. You don't need sed OR awk for this task, which are both programming languages for manipulating data, and you don't really need to manipulate anything.

Also, avoid unnecessary use of cat. It is for concatenating files. It gets overused due to its default behaviour of reading in a file and displaying it unchanged. You don't need cat to open the file for you. As base64 can read its input from STDIN e.g. in a pipeline, it can also read files:

cat ./myreport.html | base64 is better written base64 ./myreport.html