1

I distribute a slightly modified version of a public CSV file. To generate the modified CSV file I use a short one-line grep command:

grep -Ev '^([A-Za-zA-Za-z]|[0-90-9]*),' input.csv

Actual project is here.

A user reported that BSD grep on a Mac gets different results from GNU grep even with correct locale settings. It isn't clear to me why this would happen, and I don't have a Mac or BSD system to test on. (Is there some way to install BSD grep on Linux? I didn't find anything searching quickly...)

Given the same locale settings, why would the above command return different results? Sorry for not providing the original CSV file, but it's 200MB+; you can download it here if you like, it's lex.csv in v2.3.0.


To give some more detail about the data, the CSV file is a list of Japanese words. The first field is the literal word and is never quoted. The goal of the regex here is to remove entries that are either:

  1. a single letter in the latin alphabet [A-Za-zA-Za-z]
  2. purely numeric, of any length [0-90-9]*

The Unicode characters are full width latin characters that are used in Japan for legacy reasons.

Here is some example data, picked at random:

Fishstat,16201,17684,438,名詞,普通名詞,一般,*,*,*,エフアイエスエイチエスティーエーティー,FISHSTAT,Fishstat,エフアイエスエイチエスティーエーティー,Fishstat,エフアイエスエイチエスティーエーティー,記号,*,*,*,*,*,*,体,エフアイエスエイチエスティーエーテ
ィー,エフアイエスエイチエスティーエーティー,エフアイエスエイチエスティーエーティー,エフアイエスエイチエスティーエーティー,16,C1,*,25110105422963200,91350
FINA,14662,15435,406,名詞,固有名詞,一般,*,*,*,エフアイエヌエー,FINA,FINA,エフアイエヌエー,FINA,エフアイエヌエー,固,*,*,*,*,*,*,固有名,エフアイエヌエー,エフアイエヌエー,エフアイエヌエー,エフアイエヌエー,7,*,*,57503092366778880,209195
FINA,14662,15435,406,名詞,固有名詞,一般,*,*,*,エフアイエヌエー,FINA,FINA,エフアイエヌエー,FINA,エフアイエヌエー,固,*,*,*,*,*,*,固有名,エフアイエヌエー,エフアイエヌエー,エフアイエヌエー,エフアイエヌエー,7,*,*,57503092366778880,209195
FIA,16210,17693,438,名詞,普通名詞,一般,*,*,*,エフアイエー,FIA,FIA,エフアイエー,FIA,エフアイエー,記号,*,*,*,*,*,*,体,エフアイエー,エフアイエー,エフアイエー,エフアイエー,5,C2,*,1099520251404800,4000
FIA,16210,17693,438,名詞,普通名詞,一般,*,*,*,エフアイエー,FIA,FIA,エフアイエー,FIA,エフアイエー,記号,*,*,*,*,*,*,体,エフアイエー,エフアイエー,エフアイエー,エフアイエー,5,C2,*,1099520251404800,4000
エフ・アイ・シー,14659,15432,-1018,名詞,固有名詞,一般,*,*,*,エフアイシー,FIC-FIC,エフ・アイ・シー,エフアイシー,エフ・アイ・シー,エフアイシー,固,*,*,*,*,*,*,固有名,エフアイシー,エフアイシー,エフアイシー,エフアイシー,5,*,*,52112736611607040,189585
FIC,14659,15432,406,名詞,固有名詞,一般,*,*,*,エフアイシー,FIC-FIC,FIC,エフアイシー,FIC,エフアイシー,固,*,*,*,*,*,*,固有名,エフアイシー,エフアイシー,エフアイシー,エフアイシー,5,*,*,52112736645161472,189585
FIC,14659,15432,406,名詞,固有名詞,一般,*,*,*,エフアイシー,FIC-FIC,FIC,エフアイシー,FIC,エフアイシー,固,*,*,*,*,*,*,固有名,エフアイシー,エフアイシー,エフアイシー,エフアイシー,5,*,*,52112736645161472,189585
FIZ,16209,17692,438,名詞,普通名詞,一般,*,*,*,エフアイゼット,FIZ,FIZ,エフアイゼット,FIZ,エフアイゼット,記号,*,*,*,*,*,*,体,エフアイゼット,エフアイゼット,エフアイゼット,エフアイゼット,5,C1,*,25108181277614592,91343
FIZ,16209,17692,438,名詞,普通名詞,一般,*,*,*,エフアイゼット,FIZ,FIZ,エフアイゼット,FIZ,エフアイゼット,記号,*,*,*,*,*,*,体,エフアイゼット,エフアイゼット,エフアイゼット,エフアイゼット,5,C1,*,25108181277614592,91343
FIT,16209,17692,438,名詞,普通名詞,一般,*,*,*,エフアイティー,FIT,FIT,エフアイティー,FIT,エフアイティー,記号,*,*,*,*,*,*,体,エフアイティー,エフアイティー,エフアイティー,エフアイティー,5,C1,*,25107906399707648,91342
FIT,16209,17692,438,名詞,普通名詞,一般,*,*,*,エフアイティー,FIT,FIT,エフアイティー,FIT,エフアイティー,記号,*,*,*,*,*,*,体,エフアイティー,エフアイティー,エフアイティー,エフアイティー,5,C1,*,25107906399707648,91342

Based on the user report, the number of entries excluded by the regex is larger with BSD grep than with GNU grep (roughly 8000 vs 232). Unfortunately they didn't report the exact difference so I'm not sure what kind of entries are being treated differently.


I contacted the user who reported this and they provided a gist of the entries matched by BSD grep. They indicate the entries were found using this command (I guess the -v was not used):

grep -Ev '^([A-Za-zA-Za-z]|[0-90-9]*),' lex.csv

On a quick look it looks like it has punctuation and kaomoji (like (´・ω・`)). It's not clear why those would be included to me. It also includes multi-character alphabetic entries, which just seems wrong...

5
  • Could you describe what we're seeing? Those ranges are containing characters that obviously are non-ASCII. Can you show any of the data so that we've got something to test on? – Kusalananda Feb 24 at 12:49
  • I'll update the question... – polm23 Feb 24 at 12:56
  • you can download it here if you like - the page you're linking to is entirely in Japanese. Can you provide a full link to the file that could be used with wget? – Arkadiusz Drabczyk Feb 24 at 13:25
  • You can click on the big download button with the down arrow icon, here's a direct link to the zip file. The file is 2.2GB though... unidic.ninjal.ac.jp/unidic_archive/cwj/2.3.0/… – polm23 Feb 24 at 14:13
  • That range [0-90-9] is probably the problem, since it is not numeric. – Thomas Dickey Feb 24 at 20:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.