1

How do I remove duplicated entries within each separate fields with below sample as data.

0x,9.4,,,#0,#UNIX#unix,#cli#L#فا#0#فا#0#L#SE#Cli#SE,#فارسی#فارسی#۱#1#١#1,bsh,#V & v

expected output(either delete all duplicated ones, case-insensitive, difference in Unicode "Persian /Arabic ", order of entries and which entry (ignore case) should keep doesn't matter here):

0x,9.4,,,#0,#unix,#cli#L#فا#0#SE,#فارسی#١#۱#1,bsh,#V & v

The pattern is in this format #x, x means anything in one or more length of characters.

Unicode table for Persian/Arabic languages alphabet/numbers differences

1

Using a command line in a shell (just a few lines) with a proper parser :

perl -CS -Mopen=":std,IN,OUT,IO,:encoding(utf8)" -MText::CSV -lne '
    BEGIN{
        our $csv = Text::CSV->new({ sep_char => "," });
        sub uniq { my %seen;  grep !$seen{lc $_}++, @_; }
    };
    $csv->parse($_) or die "parse error";
    print join ",", map { join "#", uniq split /#/ } $csv->fields();
' file.csv

Output :

0x,9.4,,,#0,#UNIX,#cli#L#فا#0#SE,#فارسی#۱#1#١,bsh,#V & v

Note :

  • require to install Text::CSV perl module : sudo apt-get install libtext-csv-perl for debian and derivative
1

Complete Python solution:

uniq_arabic_csv.py script:

#!/bin/python

import sys, re

csv_data = sys.stdin.read().strip()
result = []

for item in csv_data.split(','):
    matches = list(re.finditer(r'#[^#]+', item, re.U | re.I))
    if not matches or len(matches) == 1:
        result.append(item)
    else:
        s, res = set(), []
        for m in matches:
            m = m.group()
            if m.lower() not in s: res.append(m)
            s.add(m.lower())
        result.append(''.join(res))

result = ','.join(result)                    
print(result)

Usage:

s="0x,9.4,,,#0,#UNIX#unix,#cli#L#فا#0#فا#0#L#SE#Cli#SE,#فارسی#فارسی#۱#1#۱#1,bsh,#V & v"
python uniq_arabic_csv.py <<<"$s"

The output:

0x,9.4,,,#0,#UNIX,#cli#L#فا#0#SE,#فارسی#۱#1,bsh,#V & v
0

I code below simple awk solution as well:

awk -F, '{ i=0; comma=""; while (++i<=NF) {
    if ( split(tolower($i), arr, /#/)>1 ){ for(v in arr) !unq[arr[v]]++;printf comma;
        for (x in unq) {printf (x!="")?hash""x:""; hash="#"; delete unq[x]}}
    else {printf comma""$i; comma=","}}; print ""
}' infile.csv

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.