0

I have one main directory and it has multiple sub-directory with many files with similar name and extension. I want to find out two files and then remove the common entries from second file.

I tried following command

awk 'NR==FNR{a[$1]++;next;}!($2 in a)' file1_comma_splited_newsort.txt file2_comma_splited_snp.txt >> output.txt

above awk command compare the 1st column of file1 with 2nd column of file2 remove duplicate entries from 2nd column of file2. after run this command I get desire output. But I need to run this command for many files which are inside the sub-directory. For this I want to find out the both files using find command and place in above awk command, Is it possible to do?

consider my directory contains following files

main directory file1_comma_splited_newsort.txt 
sub-directory1 file2_comma_splited_snp.txt
sub-directory2 file3_comma_splited_snp.txt 
sub-directory3 file4_comma_splited_snp.txt
sub-directory4 file5_comma_splited_snp.txt
sub-directory5 file6_comma_splited_snp.txt
sub-directory6 file7_comma_splited_snp.txt
sub-directory7 file8_comma_splited_snp.txt
sub-directory8 file9_comma_splited_snp.txt

I need to compare all the subdirectory files with 1st column of main directory file i.e file1_comma_splited_newsort.txt one by one so that I can remove duplicate entries present in 2nd column of all sub-directory files.

| improve this question | |
  • 1
    So your problem is not with awk but with how to pass the files to your awk command? Then how do you know which two files to compare? This rather be done with a shell script. – Fiximan Nov 21 '16 at 12:23
  • 1
    While showing your attempted solution is a good thing, you should explain the problem you are trying to solve a bit more. There may be better solutions that you haven't considered. It would also help us better analyze your attempted solution to find why it's not working. In this case, provide an example directory structure, and which files you'd want removed. – Patrick Nov 21 '16 at 13:27
  • @Fiximan, yes I have no problem with awk but i cant pass the files in awk command and I need to compare my all sub-directory files with one main directory file. – user6681622 Nov 21 '16 at 16:04
0

Assuming a tree like the following

main
|--maindata.txt
|--sub
   |--subdata1.txt
   |--subdata2.txt

and you being in directory main as cdw, this can be done with a bash script (even as a one-liner if wish be).

#!/bin/bash
for file in sub/* ; do
    awk '{actions}' maindata.txt "$file" >> "$file"_differences
done

This will simply iterate over all files in your subdirectory and create a file containing the differences for each file in sub.

| improve this answer | |
  • I don't want all difference in ONE file single file , as I above show I want to compare all my sub-directory files with one single main directory file and i need each single file with difference in sub-directory i.e it will be total 8 files (one file in each sub -directory). – user6681622 Nov 21 '16 at 17:51
  • I need code or shell script something like this awk 'NR==FNR{a[$1]++;next;}!($2 in a)' "\Desktop\main_directory\file1_comma_splited_newsort.txt" find ./ pwd -name *comma_splited_snp.txt >> output.txt` This code is what I tried to solve this problem but it not run successfully because it can't find out and place second file i.e sub-directory file in awk command. can you please solve this problem – user6681622 Nov 21 '16 at 18:10
  • @user6681622 Try the (now updated) bash loop then (can be done in one line, too). – Fiximan Nov 21 '16 at 20:53
  • Yes now this updated bash script work properly. Thanks for your efforts to solve my problem. – user6681622 Nov 22 '16 at 16:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.