1

I want to filter a file for lines starting with a space. I use the following command:

grep -v "^ " < input > input_no_starting_space

To double check my results, I run the following:

grep "^ " < input > double_check

and then count the number of lines in input_no_starting_space and double_check to see whether their sum adds up to the number of lines in input. For this I use wc -l.

For some reason, this check fails. Meaning, the sum of the number of lines is less than the number of lines in input. My file has millions of lines, but I cannot seem to reproduce the issue on a small example. Is there by any chance something wrong with the way I use grep (since I would expect that grep and grep -v always give the complement of one another), or is this more likely an artifact in my file? In case of the latter, what could this artifact be?

This is using GNU grep 3.4 on Ubuntu 20.04.3.

14
  • 1
    diff -u <(cat input_no_starting_space double_check | sort) <(sort < input)
    – user20574
    Commented Nov 12, 2021 at 17:20
  • This is going to be hard to debug without an example (and you can't easily give an example, I know). I assume you're counting the lines with wc -l, right? Does using /usr/bin/grep -v "^ " and /usr/bin/grep "^ " make any difference? I'm thinking you might have an alias that's adding --color=always or something and that could be confusing things. I would also try passing the file through dos2unix or sed -i 's/\r//g', just in case, although I don't really see how the \r would cause this specific behavior.
    – terdon
    Commented Nov 12, 2021 at 17:30
  • Are you sure nothing appends to input in the meantime? Commented Nov 12, 2021 at 17:38
  • @terdon Yes, I use wc -l indeed. I tried using /bin/grep as well as doing the sed command before, but still the same behavior
    – Stijn
    Commented Nov 12, 2021 at 18:07
  • @KamilMaciorowski Nothing gets appended indeed, I only perform the grep commands. I also check the number of lines both before and after executing the grep
    – Stijn
    Commented Nov 12, 2021 at 18:08

1 Answer 1

1

May be, your input_file do not contains only text data.

Try to use grep with -a option.

See also --binary-files=TYPE option for grep command and man grep first paragraph about data enconding and NULL value:

If a file's data or metadata indicate that the file contains binary data, assume that the file is of type TYPE. Non-text bytes indicate binary data; these are either output bytes that are improperly encoded for the current locale, or null input bytes when the -z option is not given.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .