Environment: linux shell or better, ash shell in busybox (example system: Openwrt).

Given a string variable with large text "${a}" constructed in this way:

for index in $(seq 1 40000); do #it is 40000 for a reason, it helps if you want to reproduce the error on your system.
   a="${a}""\n""word ${index}"

and given the fact that trying to use this variable as argument for common command produce an error, for example:

echo "${a}" #fails
awk -v VAR_A="${a}" '<program>' #fails

(the failures are due: http://www.in-ulm.de/~mascheck/various/argmax/ )

How do you write such a variable to a text file, possibly using only simple commands and not sed/awk/perl/other powerful interpreter.

Ideally the text file should look like:

word 1
word 2
word 3

and not like:

word 1\nword 2\nword 3\n...

Update1: Someone asks "Why you cannot change the script that produce the variable $a?". Because i cannot due to, let's say, lack of authorization ("the script was working until today, so you can fix only the printing part"). So $a and its format is given, i can only find a way to print it to a file.

Update2: Using the suggested "here docs" solve most of the problems but still i have the content printed in one line. Maybe is my config?

  • Are you requesting to pass the results of $a (the echo statement) into a text file? – ryekayo Aug 4 '14 at 17:01
  • 2
    How is echo failing. echo is usually built-in and should not trigger the E2BIG error. – Stéphane Chazelas Aug 4 '14 at 20:43
  • I didn't know either, just try it. In the end is reasonable, everything is finite. – Pierfrancesco PierQR Aiello Aug 5 '14 at 6:21
  • And yes I get $a in the format defined by the for and I need to output it. – Pierfrancesco PierQR Aiello Aug 5 '14 at 6:33
  • @PierfrancescoPierQRAiello - I don't think you do need to store it in $a at all. Consider: for i in 1 2 3 4 5 ; do echo "$i" ; done >file. But doing for i in $(seq $num) ; do echo word $i; done is redundant because you can just do seq -s'word' $num – mikeserv Aug 5 '14 at 6:45

You can make a here document and use cat:

$ cat <<EOT >output
$ wc -l output

I don't know exactly how powerful your Busybox's ash is (it's configurable), but that should work anywhere, even with no builtins at all. It expands the variable value into a quasi-file, which is then given to cat, rather than putting the value itself into the command line.

Since you seem to have literal "\n" in your string that you want to get rid of, you can also use string replacement during parameter subsitution. This can be configured out of Busybox's ash, in which case you'll have to go to sed. If you do have this (non-POSIX) extension, you can use ${var//\\\n/$NEWLINE}:

$ busybox ash
$ foo="x\ny\nz"
$ echo "$foo"
$ echo "${foo//\\\n/$NEWLINE}"

This is the same syntax as Bash; ${var/pattern/replacement}, where pattern is global if it starts with /. It seems your ash has a lot configured out of it, so this might not be there at all. In that case you're probably going to have to go with sed, although I guess judicious use of IFS and read would let you work around that and reconstruct a correct string.

  • not really answering the question - the problem is passing large amounts of data on command line. – peterph Aug 4 '14 at 22:34
  • 1
    @peterph - which this does. this is a file descriptor - not an argument. – mikeserv Aug 4 '14 at 22:43
  • Yup, it works but still i have "\n" in the output. Namely # cat output | wc -w returns 40001 ; while # cat output | wc -l returns 1 – Pierfrancesco PierQR Aiello Aug 5 '14 at 10:46
  • @PierfrancescoPierQRAiello: You may be able to use string replacement (see edit), but if that isn't part of your minimal ash you'll probably have to go to sed. – Michael Homer Aug 5 '14 at 21:21
  • many thanks! Would have been enough to have some hints, while I got a long explanation that required time! – Pierfrancesco PierQR Aiello Aug 6 '14 at 7:06
busybox seq -s'
word ' 4000 | 
sed '/000/w /dev/fd/2' |  
wc -l


word 1000
word 2000
word 3000
word 4000

You should just be able to use busybox's builtin seq and specify word as a separator. Then you can >redirect it to a file if you wish. In the above example I split out to sed just so you could get an idea of the output without having to look at 4000 lines - but they were all printed. If your busybox does not support this, upgrade.

So basically all you need for that whole $a thing is:

busybox seq -s'
  • not really answering the question - the problem is passing large amounts of data on command line. – peterph Aug 4 '14 at 22:33
  • 1
    @peterph : Here's the question I read. How do you write such a variable to a text file, possibly using only simple commands and not sed/awk/perl/other powerful interpreter[?] The answer I propose lies in generating and writing the variable's value at once, using only simple commands... etc. – mikeserv Aug 4 '14 at 22:39
  • @PierfrancescoPierQRAiello - please do this instead of the other thing I used to have here just in case you're doing that one. It is essentially the same, byut far simpler. Just in case you're doing the other thing, that is. – mikeserv Aug 5 '14 at 1:08
  • thanks Mikeserv, but the variable is given. I cannot improve the generation of it (else peterph would have already replied). You get this huge stuff and you need to write it to a file. – Pierfrancesco PierQR Aiello Aug 5 '14 at 6:24
  • @PierfrancescoPierQRAiello - then you need to use the heredoc like the Michael Homer suggests. Still, I cannot understand how it can be given - it has to get into the shell somehow. Either it is generated by iterating as you demonstrate and can be replaced like I demo above, or it is sourced from a file, in which case you do busybox ash ./file >oufile. You shouldn't ever have to store information of that size in a single shell variable ever. You would probably have better luck with the argument array as well, like set -- "$@" "$a" once every iteration - to avoid a single long string. – mikeserv Aug 5 '14 at 6:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.