How to find out two files and remove the common entries from second file

Question

I have one main directory and it has multiple sub-directory with many files with similar name and extension. I want to find out two files and then remove the common entries from second file.

I tried following command

awk 'NR==FNR{a[$1]++;next;}!($2 in a)' file1_comma_splited_newsort.txt file2_comma_splited_snp.txt >> output.txt 

above awk command compare the 1st column of file1 with 2nd column of file2 remove duplicate entries from 2nd column of file2. after run this command I get desire output. But I need to run this command for many files which are inside the sub-directory. For this I want to find out the both files using find command and place in above awk command, Is it possible to do?

consider my directory contains following files

main directory file1_comma_splited_newsort.txt 
sub-directory1 file2_comma_splited_snp.txt
sub-directory2 file3_comma_splited_snp.txt 
sub-directory3 file4_comma_splited_snp.txt
sub-directory4 file5_comma_splited_snp.txt
sub-directory5 file6_comma_splited_snp.txt
sub-directory6 file7_comma_splited_snp.txt
sub-directory7 file8_comma_splited_snp.txt
sub-directory8 file9_comma_splited_snp.txt

I need to compare all the subdirectory files with 1st column of main directory file i.e file1_comma_splited_newsort.txt one by one so that I can remove duplicate entries present in 2nd column of all sub-directory files.

Answer 1

Assuming a tree like the following

main
|--maindata.txt
|--sub
   |--subdata1.txt
   |--subdata2.txt

and you being in directory main as cdw, this can be done with a bash script (even as a one-liner if wish be).

#!/bin/bash
for file in sub/* ; do
    awk '{actions}' maindata.txt "$file" >> "$file"_differences
done

This will simply iterate over all files in your subdirectory and create a file containing the differences for each file in sub.