I distribute a slightly modified version of a public CSV file. To generate the modified CSV file I use a short one-line grep command:
grep -Ev '^([A-Za-zA-Za-z]|[0-90-9]*),' input.csv
Actual project is here.
A user reported that BSD grep on a Mac gets different results from GNU grep even with correct locale settings. It isn't clear to me why this would happen, and I don't have a Mac or BSD system to test on. (Is there some way to install BSD grep on Linux? I didn't find anything searching quickly...)
Given the same locale settings, why would the above command return different results? Sorry for not providing the original CSV file, but it's 200MB+; you can download it here if you like, it's lex.csv
in v2.3.0.
To give some more detail about the data, the CSV file is a list of Japanese words. The first field is the literal word and is never quoted. The goal of the regex here is to remove entries that are either:
- a single letter in the latin alphabet
[A-Za-zA-Za-z]
- purely numeric, of any length
[0-90-9]*
The Unicode characters are full width latin characters that are used in Japan for legacy reasons.
Here is some example data, picked at random:
Fishstat,16201,17684,438,名詞,普通名詞,一般,*,*,*,エフアイエスエイチエスティーエーティー,FISHSTAT,Fishstat,エフアイエスエイチエスティーエーティー,Fishstat,エフアイエスエイチエスティーエーティー,記号,*,*,*,*,*,*,体,エフアイエスエイチエスティーエーテ
ィー,エフアイエスエイチエスティーエーティー,エフアイエスエイチエスティーエーティー,エフアイエスエイチエスティーエーティー,16,C1,*,25110105422963200,91350
FINA,14662,15435,406,名詞,固有名詞,一般,*,*,*,エフアイエヌエー,FINA,FINA,エフアイエヌエー,FINA,エフアイエヌエー,固,*,*,*,*,*,*,固有名,エフアイエヌエー,エフアイエヌエー,エフアイエヌエー,エフアイエヌエー,7,*,*,57503092366778880,209195
FINA,14662,15435,406,名詞,固有名詞,一般,*,*,*,エフアイエヌエー,FINA,FINA,エフアイエヌエー,FINA,エフアイエヌエー,固,*,*,*,*,*,*,固有名,エフアイエヌエー,エフアイエヌエー,エフアイエヌエー,エフアイエヌエー,7,*,*,57503092366778880,209195
FIA,16210,17693,438,名詞,普通名詞,一般,*,*,*,エフアイエー,FIA,FIA,エフアイエー,FIA,エフアイエー,記号,*,*,*,*,*,*,体,エフアイエー,エフアイエー,エフアイエー,エフアイエー,5,C2,*,1099520251404800,4000
FIA,16210,17693,438,名詞,普通名詞,一般,*,*,*,エフアイエー,FIA,FIA,エフアイエー,FIA,エフアイエー,記号,*,*,*,*,*,*,体,エフアイエー,エフアイエー,エフアイエー,エフアイエー,5,C2,*,1099520251404800,4000
エフ・アイ・シー,14659,15432,-1018,名詞,固有名詞,一般,*,*,*,エフアイシー,FIC-FIC,エフ・アイ・シー,エフアイシー,エフ・アイ・シー,エフアイシー,固,*,*,*,*,*,*,固有名,エフアイシー,エフアイシー,エフアイシー,エフアイシー,5,*,*,52112736611607040,189585
FIC,14659,15432,406,名詞,固有名詞,一般,*,*,*,エフアイシー,FIC-FIC,FIC,エフアイシー,FIC,エフアイシー,固,*,*,*,*,*,*,固有名,エフアイシー,エフアイシー,エフアイシー,エフアイシー,5,*,*,52112736645161472,189585
FIC,14659,15432,406,名詞,固有名詞,一般,*,*,*,エフアイシー,FIC-FIC,FIC,エフアイシー,FIC,エフアイシー,固,*,*,*,*,*,*,固有名,エフアイシー,エフアイシー,エフアイシー,エフアイシー,5,*,*,52112736645161472,189585
FIZ,16209,17692,438,名詞,普通名詞,一般,*,*,*,エフアイゼット,FIZ,FIZ,エフアイゼット,FIZ,エフアイゼット,記号,*,*,*,*,*,*,体,エフアイゼット,エフアイゼット,エフアイゼット,エフアイゼット,5,C1,*,25108181277614592,91343
FIZ,16209,17692,438,名詞,普通名詞,一般,*,*,*,エフアイゼット,FIZ,FIZ,エフアイゼット,FIZ,エフアイゼット,記号,*,*,*,*,*,*,体,エフアイゼット,エフアイゼット,エフアイゼット,エフアイゼット,5,C1,*,25108181277614592,91343
FIT,16209,17692,438,名詞,普通名詞,一般,*,*,*,エフアイティー,FIT,FIT,エフアイティー,FIT,エフアイティー,記号,*,*,*,*,*,*,体,エフアイティー,エフアイティー,エフアイティー,エフアイティー,5,C1,*,25107906399707648,91342
FIT,16209,17692,438,名詞,普通名詞,一般,*,*,*,エフアイティー,FIT,FIT,エフアイティー,FIT,エフアイティー,記号,*,*,*,*,*,*,体,エフアイティー,エフアイティー,エフアイティー,エフアイティー,5,C1,*,25107906399707648,91342
Based on the user report, the number of entries excluded by the regex is larger with BSD grep than with GNU grep (roughly 8000 vs 232). Unfortunately they didn't report the exact difference so I'm not sure what kind of entries are being treated differently.
I contacted the user who reported this and they provided a gist of the entries matched by BSD grep. They indicate the entries were found using this command (I guess the -v
was not used):
grep -Ev '^([A-Za-zA-Za-z]|[0-90-9]*),' lex.csv
On a quick look it looks like it has punctuation and kaomoji (like (´・ω・`)
). It's not clear why those would be included to me. It also includes multi-character alphabetic entries, which just seems wrong...
you can download it here if you like
- the page you're linking to is entirely in Japanese. Can you provide a full link to the file that could be used withwget
? – Arkadiusz Drabczyk Feb 24 at 13:25[0-90-9]
is probably the problem, since it is not numeric. – Thomas Dickey Feb 24 at 20:54