1

So, i'm using bash to read a file (that has no newlines, space or tab).

Something like this:

aababcbbcbckqkkqkqhddhkehkjhqkjhsdk
skjhqkdjhqkzdhkzhdkjqzhdhqkjhzdkqzh

though there is a newline in this example, there isn't on the data i'm working on... So i found that, since everything was technically seen as on "the same line" (since there isn't any newline or any delimiter i could use), i tried to read every Nth chars:

while read -N129999 character; do
  program "$character"
done < <(cat file | tr -d '\n')

(I'm aware of the "useless use of cat")

The number i'm using here is just the maximum i found that i could make read use, (which also i prefer doing since it process the file faster) The program here is just an example and for the sake of illustration. And i'm purposely removing newline, tabs and space on the aforementioned data.

Now while what i provided work, it doesn't quite work for the last part of the file, which contain less than the aforementioned number...and while i'm aware that, if there was a delimiter provided to IFS, the -n option could continue after that instead of ignoring the rest of file which doesn't fit the range of characters...

How do i (in bash, sed, or any posix tools?) read every Nth range of characters, while including the rest of the file/input, which doesn't fit that range?

5
  • Manual for bash read says -N terminates if EOF is encountered. How big is the file? awk can read the whole thing into memory and substr it to pass to shell, but doing the whole job in bash would be way faster for a huge file. Jun 10, 2020 at 22:34
  • Yeah, it is decently huge (think in a couple gigabytes). I tried different method to read the file (xargs, pure bash, etc) and found that, a while loop was the fastest. Jun 10, 2020 at 22:36
  • Also, i did try the -N option, but encountered the same problem in my case @Paul_Pedant ... Jun 10, 2020 at 22:36
  • 1
    @NordineLotfi, there's also a Perl option: perl -e 'system("somecmd", $a) while (sysread STDIN, $a, 9999)' < <(...) Bash's read only ever reads 128 bytes at a time for some reason (even with -N, I tested), so it can suffer from a bit more overhead than most utils there. I have no idea if that makes any meaningful difference here, though.
    – ilkkachu
    Jun 10, 2020 at 22:56
  • Thanks @ilkkachu didn't consider a perl option! Though, didn't know that bash did that (i noticed that read accept no more than 129999 characters though, probably related). And yes, it does have a meaningful difference i believe (at least for me), since, more chars read, faster processing! Jun 10, 2020 at 23:03

1 Answer 1

5

You're seeing the same issue that comes up when reading text files that are missing the newline at the end of the last line. read returns a falsy status when it hits end-of-file before seeing the delimiter, or here, without reading the required number of characters. That is, even if it did read something before that. It does set the output variable though, so you can check if it's non-empty.

$ printf abc | while read -N2 x || [ "$x" ] ; do echo "read: $x"; done ;
read: ab
read: c

If you don't have that condition, you could still see the final part of the input appear in the output variable, just after the loop exited.

$ printf abc | ( while read -N2 x; do echo "read: $x"; done ; echo "finally: $x" )
read: ab
finally: c

Related: What does `while read -r line || [[ -n $line ]]` mean?

1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.