6

Is there any way to read entire file into memory in bash, without using external command like cat?

I know the syntax to get output from a file, i.e <(/bin/ls), but now I need to read the file into entire memory, i.e $A would hold the contents (and there could be null, EOF like stuff)

  • 2
  • @l0b0 … unless the shell is zsh. – Gilles 'SO- stop being evil' Dec 4 '12 at 23:19
  • @Gilles Interesting; I hope that makes it to Bash someday (unless it would break too much). Maybe they should amend the text to "Bash strings." – l0b0 Dec 5 '12 at 6:49
  • @l0b0 Other shells (Bourne, ksh, ash, csh, tcsh) have the same restriction. They aren't designed to process binary data. Shell variables originated from environment variables which can't contain null bytes either. Unix favors text, and shells are geared towards handling text. – Gilles 'SO- stop being evil' Dec 5 '12 at 11:02
  • ksh can now store binary data using: typeset -b varname – dannyw Jan 26 '17 at 6:53
7

Bash doesn't support null bytes in variables. foo=$(cat file) makes $foo contain the beginning of the file, up to the first null byte. The command substitution also strips all final newlines from the file. For example, if the file consists of the four bytes (a, newline, newline, newline), then foo contains just a, same as if the file only contained that one byte.

If you want to preserve final newlines, arrange for the command to output an extra non-newline character, then strip it off.

foo=$(cat file; echo a)
foo=${foo%a}

So far this all worked in all Bourne/POSIX-style shells (ash, ksh, bash, …). Bash ≥4.0 has a faster way of reading a file into a variable: the mapfile.

mapfile <file foo

There simply isn't a way to stuff a null byte into a bash variable. If you want this, you have several options:

  • Use zsh.
  • Use Perl.
  • Store the data in a temporary file (which seems useless here, as the data is already in a file).
  • Encode the input and decode it back, e.g. with base64 and base64 -d.
|improve this answer|||||
6

One way like this:

$ A=$(<file)

Instead of a variable, a better option would be to read into an array which makes it easier to access different lines:

$ OIFS=$IFS
$ IFS=$'\n'
$ arr=($(<file))
$ IFS=$OIFS
|improve this answer|||||
  • echo $(</usr/bin/ls) > ls; diff ls /usr/bin/ls differs, was it a problem with echo? – daisy Dec 4 '12 at 1:58
  • 1
    The echo command assumes that the input is 'printable' ASCII, not binary data. The ls program file contains binary data. Beyond that, the echo will likely append a trailing newline, which in itself makes it different from the actual contents. – Arcege Dec 4 '12 at 2:14
  • 1
    Watch out: arr=($(<file)) is an array of words in the file, not necessarily lines (assuming default value of $IFS). For an array of lines, use the mapfile command. – glenn jackman Dec 4 '12 at 5:01
  • @glennjackman : Thanks glenn for pointing it out..updated the solution using IFS since mapfile is available only from bash4... – Guru Dec 4 '12 at 5:12
  • @warl0ck: try it with echo "$(</usr/bin/ls)" > ls – Gert van den Berg Dec 4 '12 at 14:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.