6

I think it's somewhat a simple problem but I don't understand what's happening. I have a file such as

Scaffold_19:10402053-10402986   Scaffold_33:25064012-25064819   92.806  278 5   5   8   279 269 1   2.84e-107   388
Scaffold_6:54625155-54625833    Scaffold_33:6634011-6634353 96.774  248 6   2   1   248 97  342 1.21e-114   412
Scaffold_9:94800251-94801152    Scaffold_26:56326990-56327448   100.000 297 0   0   1   297 162 458 1.19e-155   549
Scaffold_20:79074547-79075122   Scaffold_20:79075175-79076351   93.269  208 11  3   1   208 972 1176    6.43e-82    303
Scaffold_18:6685836-6686898 Scaffold_19:81178714-81179662   99.789  949 1   1   114 1062    1   948 0.0 1740
Scaffold_18:46725018-46726185   Scaffold_18:46723864-46724938   84.309  752 73  29  1   710 323 1071    0.0 693
Scaffold_9:97974338-97975523    Scaffold_9:97976642-97978267    90.239  1045    81  18  155 1185    1   1038    0.0 1345
Scaffold_16:35667544-35671350   Scaffold_34:16160847-16161438   99.459  185 0   1   1   184 185 1   1.56e-90    335
Scaffold_33:47202231-47202984   Scaffold_33:47205382-47206652   94.763  611 31  1   3   613 642 1251    0.0 950
Scaffold_8:18583199-18586799    Scaffold_25:14231844-14235013   98.460  3116    43  3   1   3112    55  3169    0.0 5483

Now, all I want to do is to print the lines where the thirds column is <= to a number. That should be easy right? For example, print the lines where $3 is <= 100

head q.filtered.noself.blastn|awk '$3 <= 100'

But it returns nothing! Can you help? It's like awk doesn't recognize the third field is a number.

I am using mawk 1.3.4 20200120 on Ubuntu 20.04.2

here is a sample of the file sample

EDIT: here is the output of locale

locale
LANG=en_GB.UTF-8
LANGUAGE=en_GB:en
LC_CTYPE="en_GB.UTF-8"
LC_NUMERIC=fr_BE.UTF-8
LC_TIME=fr_BE.UTF-8
LC_COLLATE="en_GB.UTF-8"
LC_MONETARY=fr_BE.UTF-8
LC_MESSAGES="en_GB.UTF-8"
LC_PAPER=fr_BE.UTF-8
LC_NAME=fr_BE.UTF-8
LC_ADDRESS=fr_BE.UTF-8
LC_TELEPHONE=fr_BE.UTF-8
LC_MEASUREMENT=fr_BE.UTF-8
LC_IDENTIFICATION=fr_BE.UTF-8
LC_ALL=
5
  • I can't reproduce this. Please edit your question and add i) your operating system (I see you tagged as Linux, but please add it to the question body and tell us which Linux), ii) your version/flavor of awk (see awk -V) and iii) upload a sample of the actual file somewhere and link us to it. Maybe there are non-printing characters in there confusing things.
    – terdon
    Apr 19, 2021 at 11:58
  • 5
    Which implementation of awk are you using? POSIX-compliant implementations are required to use the decimal separator defined in your locale when parsing input data, which may not be the dot (.). GNU awk uses the dot by default for that, unless told not to. (And what is the output of locale in the environment you are running awk in?)
    – fra-san
    Apr 19, 2021 at 12:01
  • Yes, good point. Please also add the output of locale.
    – terdon
    Apr 19, 2021 at 12:02
  • OK, try head q.filtered.noself.blastn | LC_ALL=C awk '$3 <= 100' Apr 19, 2021 at 13:18
  • You may replace . with , in the third field to honor your locale. Use the gsub function. For example : { gsub("\.",",",$3); }
    – Stephan
    Apr 20, 2021 at 20:03

1 Answer 1

12

The problem is that you have LC_NUMERIC=fr_BE.UTF-8. Under this locale, the decimal separator is , and not . so 98.9 is not considered a number. To illustrate:

$ echo "98.5" | LC_NUMERIC=C mawk '$1<100' 
98.5
$ echo "98.5" | LC_NUMERIC=fr_BE.UTF-8 mawk '$1<100' 
$ 

So either change your locale to one that has . as the decimal separator, or set the locale explicitly for this command:

head q.filtered.noself.blastn | LC_NUMERIC=C awk '$3 <= 100'
4
  • Hello, yes that was the problem. Thanks a lot!
    – Alessandro
    Apr 19, 2021 at 13:45
  • 1
    that also seems to depend on the version of awk. I can't reproduce that with mawk 1.3.3, while 1.3.4 does indeed exhibit that. And GNU awk ignores what the locale says, unless you use --use-lc-numeric to say you want that.
    – ilkkachu
    Apr 19, 2021 at 13:49
  • @ilkkachu huh. I tested it with 1.3.4, that's the only one I have accessible.
    – terdon
    Apr 19, 2021 at 13:57
  • either that or there's some difference in the locales between the two systems I tried on
    – ilkkachu
    Apr 19, 2021 at 16:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.