0

I have two compressed files, with values organized in columns.

I need to check the first column in each one, and delete the duplicated values from the first file, and keep it in the second one.

5
  • Can you give sample input files and expected output please?
    – chaos
    Jul 14 '15 at 10:42
  • Each file contains huge number of data, the first column of each file contains IDs, such as 55441 55442 55486
    – Eng7
    Jul 14 '15 at 10:47
  • How are they compressed? What algorithm?
    – Minix
    Jul 14 '15 at 10:58
  • they are .gz files
    – Eng7
    Jul 14 '15 at 11:02
  • @Minix, They are .CSV files, Ordinary ASCII files with columns of data separated by delimiter , on each row.
    – Eng7
    Jul 14 '15 at 12:20
1

You need to put the paths to your files into the script and then run it.

Tested the script with these example files:

file1.csv:

11,0
12,b
14,z

file2.csv:

12,a
13,b
14,c

out.csv:

11,0

del_dup.sh:

#!/bin/bash

FILE1=./file1.csv
FILE2=./file2.csv
FILE3=./out.csv

IFS=$'\n' 
for line in $( cat $FILE1 ); do
        ID1="$( echo $line | cut -d ',' -f1 )"
        if [ -z "$( cat $FILE2 | grep ^"$ID1" )" ]; then
                echo $line >> $FILE3
        fi
done
1
  • Forgot the decompression, but with two files, that can be done manually, I guess.
    – Minix
    Jul 14 '15 at 16:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.