9

I have a .tsv file (values separated by tabs) with four values. So each line should have only three tabs and some text around each tab like this:

value   value2  value3  value4

But it looks that some lines are broken (there is more than three tabs). I need to find out these lines.


I came up with following grep pattern.

grep -v "^[^\t]+\t[^\t]+\t[^\t]+\t[^\t]+$"

My thinking:

  • first ^ matches the beggining
  • [^\t]+ matches more than one "no tab character"
  • \t matches single tab character
  • $ matches end

And than I just put it into right order with correct number of times. That should match correct lines. So I reverted it by -v option to get the wrong lines.

But with the -v option it matches any line in the file and also some random text I tried that don't have any tabs inside.

What is my mistake please?

EDIT: I am using debian and bash.

18
  • Why not simply grep for lines with at least four tabs: grep '\t.*\t.*\t.*\t'?
    – Philippos
    Aug 16, 2022 at 11:53
  • 1
    @EricDuminil, on a quick test, the difference in execution times for a grep of those two regexes over a 317250-line file was about 9 % in favor of the original, hardly making the other "much less efficient". (On my Debian, with GNU grep, after fixing the regexes to actually work.) .*? is also a Perl-ism, and won't work in standard grep. The awk solution terdon posted half a day earlier runs in one tenth of the time.
    – ilkkachu
    Aug 17, 2022 at 7:48
  • 1
    @EricDuminil, well, it depends, I guess. I was thinking that if those 9 % matter, then one is probably using a dedicated tool already, and not grep or regexes (given we can just scan the lines character-by-character and count the tabs here, and any other processing would probably be easier by splitting the fields first, instead of trying to figure out regexes for the full line). But what's hilarious is that switching to grep -P made it even faster than the awk, with grep -E at around 3 s, awk at 0.3 s and grep -P at less than 0.1 s, even without changing the regexes.
    – ilkkachu
    Aug 17, 2022 at 8:12
  • 1
    @EricDuminil, hmm, increasing the number of tabs on each line looks to make it even worse for GNU grep. But it looks like there's big differences between regex engines. With a regex like $'^([^\t]*\t){24}' and 20 to 30 tab input lines I got about a 10x speed difference between Busybox and GNU grep's -E engine, and another ~10x between Busybox and GNU grep -P. With -E being slowest, Busybox in the middle and grep -P being fastest. And awk slightly slower than grep -P.
    – ilkkachu
    Aug 17, 2022 at 11:46
  • 1
    (I didn't check if some of those is the glibc implementation, or if GNU grep and/or Busybox have regex engines of their own. Also I didn't bother with different variants of the regexes themselves; anything would just get swamped in the difference between the engines.)
    – ilkkachu
    Aug 17, 2022 at 11:48

4 Answers 4

15

As you already saw, \t isn't special for Basic regular Expressions, and grep uses BRE by default. GNU grep, the default on Linux has -P for Perl Compatible Regular Expressions which lets you use \t for tab characters.

However, what you want is much easier to do with awk. Just set the input field separator to a tab (-F '\t') and then print any lines whose number of fields (NF) is not 3:

awk -F'\t' 'NF!=3' file

That will print all lines in file with more or less than three fields. To limit to only more than three fields, use:

awk -F'\t' 'NF>3' file
3
  • standard ERE doesn't support \t for a tab either (and neither does GNU as far as I can tell, though it supports some of the Perl backslash-letter escapes). [\t] matches a backslash or the letter t in standard BRE or ERE.
    – ilkkachu
    Aug 16, 2022 at 13:36
  • 2
    Oh wow. All these years, I never knew that! I tend to use -P by default but I really thought ERE also handled that. Maybe because GNU sed does support it or something. Thanks, @ilkkachu, and fixed.
    – terdon
    Aug 16, 2022 at 13:58
  • 1
    The behaviour for \t in both BRE and ERE is unspecified by POSIX, so what it matches will depend on the implementation. It may match on TAB, t (like GNU grep) or \t (like ksh93's builtin grep) or anything. [\t] is required to match on either backslash or t though (though some implementations of some tools such as GNU sed are not compliant by default in that regard). Aug 16, 2022 at 16:38
8
grep -v "^[^\t]+\t[^\t]+\t[^\t]+\t[^\t]+$"

Here, grep will use basic regular expressions (BRE), since it wasn't given the -E option. Unlike in extended regular expressions (ERE), the + is not special in BRE and matches itself. Also, in standard regular expressions, the backslash is not special in bracket groups, so [\t] matches a backslash or the letter t, and [^\t] matches anything but those.

Outside of bracket groups, what \t matches is unspecified by the standard and in practice varies with the implementation. For instance, with GNU grep, it matches t, whilst with ast-open's grep, it matches a TAB character.

If you want to match a tab in standard regexes, you'll need to pass a literal tab to grep, e.g. with the $'...' form of quoting supported by many shells. (Though that's not standard either (yet); you'd have to use printf in a standard shell to get the tab character.)

So grep $'a\tb' would look for a and b separated by a tab, and grep $'a\t\t*b' or grep $'a\t\\{1,\\}b' or grep -E $'a\t+b' would look for a and b separated by at least one tab.

2
  • Re: you'll need to pass a literal tab to grep - easiest way to do this in Linux is ctrl-v followed by tab; no need for $''. Aug 17, 2022 at 18:15
  • @ReinstateMonica, well "easiest" is up to opinion anyway, but I rather like using $'\t' over a hard tab because you can see it's a tab and not just a pile of spaces. Also tabs sometimes break when copypasting, and e.g. SE's editors destroys them. That Ctrl-V is a shell thing (not Linux), and depending on the shell it might not be supported or required, though almost all shells I tried support it. (but not Busybox, which does support $'...' though). And if you're writing a script in a separate file, you might well be able to just hit tab to get the character, but that's up to your editor.
    – ilkkachu
    Aug 18, 2022 at 8:32
7

Ok, so I find out the issue. I cannot use \t in grep like this. It just matches normal letter t .

Options how to match tab char are visible in this question on SO.

I solved my case by adding -P option to my command, so this works:

grep -Pv "^[^\t]+\t[^\t]+\t[^\t]+\t[^\t]+$"

Another option was pointed by @Philippos in comments (just matching lines with at least four tabs). It also needs -P option though:

grep -P '\t.*\t.*\t.*\t'
3
  • 2
    ^[^\t]+(\t[^\t]+){3}$
    – Kusalananda
    Aug 16, 2022 at 14:20
  • Indeed, thanks @StéphaneChazelas. Ilkachu already educated me on that, I tend to use -P by default so I assumed.
    – terdon
    Aug 17, 2022 at 10:49
  • Note that the '\t.*\t.*\t.*\t' version would fail with more tabs. ^[^\t]+(\t[^\t]+){100}$ should still work fine. Aug 20, 2022 at 11:48
4

As others pointed out already \t in a regex does not stand for TAB. So the obvious solution would be to add a literal TAB character, which BASH may make a bit tricky. However you can enter a literal TAB using ^V (Control+vTAB).

Maybe it's more convenient to set TAB='Control+v TAB'. The other thing is that + is treated literally in non-extended regular expressions (BRE) (see "Basic vs Extended Regular Expressions" in man grep), so use:

grep -v "^[^$TAB]\+$TAB[^$TAB]\+$TAB[^$TAB]\+$TAB[^$TAB]\+$"

(Here you could also shorten the variable to just T, and it's not necessary to use ${TAB} (or ${T}), but be prepared for surprises)

Also when you prefer to use egrep you can use a repeated group like this:

egrep -v "^([^$TAB]+$TAB){3}[^$TAB]+$"
4
  • Or TAB=$(printf '\t'). Note that \+ is a GNUism. The standard equivalent is \{1,\}. Or you can switch to ERE with -E and then use +. But here, maybe * would make more sense to allow those TAB-separated fields to be empty like the the awk-based solution does. Aug 17, 2022 at 8:06
  • 1
    Or you can wrap the regex in $'' to have the shell expand \t into literal TAB characters for you. You will however have to escape any backslashes you want to retain.
    – TooTea
    Aug 17, 2022 at 15:25
  • @TooTea: That's also a bashism, not necessarily valid with non-bash shells that are often preferred for running scripts. Aug 17, 2022 at 22:29
  • 1
    @R..GitHubSTOPHELPINGICE True, but OP's question is explicitly about Bash. Also, if you're talking about dash, note that it does support $'', just like many other common shells.
    – TooTea
    Aug 18, 2022 at 11:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .