I've been trying to use bash to read a file character by character.

After much trial and error, I have discovered that this works:

exec 4<file.txt 
declare -i n
while read -r ch <&4; 
     while [ ! $n -eq ${#ch} ]
           do  echo -n "${ch:$n:1}"
               (( n++ ))
     echo "" 

I.e., I can read it line by line and then loop through each line char by char.

Before doing this, I had tried: exec 4<file.txt && while read -r -n1 ch <&4; do; echo -n "$ch"; done but it would skip all whitespaces in the file.

Could you please explain why? Is there a way to make the second strategy (i.e. reading char by char with bash's read) work?

  • 4
    Set IFS to nothing to have whitespaces survive the word-splitting. – manatwork Oct 1 '12 at 12:30
  • Tried that with IFS='', but I guess it had to be just IFS=. Thanks! – PSkocik Oct 1 '12 at 17:53

You need to remove whitespace characters from the $IFS parameter for read to stop skipping leading and trailing ones (with -n1, the whitespace character if any would be both leading and trailing, so skipped):

while IFS= read -rn1 a; do printf %s "$a"; done

But even then bash's read will skip newline characters, which you can work around with:

while IFS= read -rn1 a; do printf %s "${a:-$'\n'}"; done

Though you could use IFS= read -d '' -rn1 instead or even better IFS= read -N1 (added in 4.1, copied from ksh93 (added in o)) which is the command to read one character.

Note that bash's read can't cope with NUL characters. And ksh93 has the same issues as bash.

With zsh:

while read -ku0 a; do print -rn -- "$a"; done

(zsh can cope with NUL characters).

Note that those read -k/n/N read a number of characters, not bytes. So for multibyte characters, they may have to read multiple bytes until a full character is read. If the input contains invalid characters, you may end up with a variable that contains a sequence of bytes that doesn't form valid characters and which the shell may end up counting as several characters. For instance in a UTF-8 locale:

$ printf '\375\200\200\200\200ABC' | bash -c '
    IFS= read  -rN1 a; echo "${#a}"'

That \375 would introduce a 6-byte UTF-8 character. However, the 6th one (A) above is invalid for a UTF-8 character. You still end-up with \375\200\200\200\200A in $a, which bash counts as 6 characters though the first 5 ones are not really characters, just 5 bytes not forming part of any character.

|improve this answer|||||
  • Thanks. Simple and beautiful. I actually tried something to this end (modifying the IFS variable), but it kind of didn't work for me so I ended up with that concoction of mine (Unnecessary playing with file descriptors, etc.). – PSkocik Oct 1 '12 at 17:39
  • 1
    Interestingly, it looks like using read -rN1 instead solves the newline issue and thus eliminates needing to provide a newline as default when printing $a. – krb686 Aug 28 '16 at 0:52
  • Just FTR I'm reading 4118 line 20 MB file. Using read -n1 (char by char) takes 4 min 51 seconds and heats laptop to 90 degrees. Using read -r (line by line) takes 1.3 seconds and laptop stays at 54 degrees with dual-fans silent. – WinEunuuchs2Unix Nov 17 '18 at 16:47

This is a simple example using cut, a for loop & wc :

bytes=$(wc -c < /etc/passwd)

for ((i=0; i<bytes; i++)); do
    echo $file | cut -c $i

KISS isn't it ?

|improve this answer|||||
  • If that is KISS, then what is a pure bash solution: file="$(</etc/passwd)"; bytes="${#file}"; for ((i=0;i<bytes;i++)); do echo "${file:i:1}"; done? – manatwork Oct 1 '12 at 12:52
  • Thanks to both. Yeah, if i have to resort to getting those characters from lines, I might as well get them from the whole file. I find sch's solution the most KISS, though. – PSkocik Oct 1 '12 at 17:44
  • @manatwork That is a good, simple solution. Even so, it seems to me like the above answer using a read loop is quite a bit faster for some reason. Maybe substrings in bash are fairly slow? – krb686 Aug 28 '16 at 0:54
  • @krb686, actually the entire bash “It's too big and too slow.” according to its man page's BUGS section. But even so, is still faster to slice a string in memory than to read a file again and again for each character. At least on my machine: pastebin.com/zH5trQQs – manatwork Aug 29 '16 at 7:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.