50

I have the following code that I run on my Terminal.

LC_ALL=C && grep -F -f  genename2.txt hg38.hgnc.bed > hg38.hgnc.goi.bed

This doesn't give me the common lines between the two files. What am I missing there?

0

2 Answers 2

103

Use comm -12 file1 file2 to get common lines in both files.

You may also needs your file to be sorted to comm to work as expected.

comm -12 <(sort file1) <(sort file2)

From man comm:

-1     suppress column 1 (lines unique to FILE1)
-2     suppress column 2 (lines unique to FILE2)

Or using grep command you need to add -x option to match the whole line as a matching pattern. The F option is telling grep that match pattern as a string not a regex match.

grep -Fxf file1 file2

Or using awk.

awk 'NR==FNR{seen[$0]=1; next} seen[$0]' file1 file2

This is reading the whole line of file1 into an array called seen where the key is a whole line (in awk the $0 represents the whole current line).

We used NR==FNR as a condition to run the following block only for the first input file1 and not file2 (NR is referring to the number of records across all inputs, and FNR is the file number of records for each individual input. So, FNR is unique for each input file whereas NR is unique for all inputs files.)

The next statement telling awk to not continue the rest of the code and rather start over again until NR is not equal to FNR, which means all lines of file1 are read by awk.

Then next condition seen[$0] will apply only for the second input file2. For each line in file2 it will print every line that was marked as present =1 in file1 in the array.

Another simple option is using sort and uniq:

sort file1 file2|uniq -d

This will print both files sorted then uniq -d will print only duplicated lines. BUT this is granted when there is NO duplicated lines in both files themselves, else below is always granted even if there is a lines duplicated within both files.

uniq -d <(sort <(sort -u file1) <(sort -u file2))
3
  • comm works fine, but in my case grep -Fxf returns: bash: /bin/grep: Argument list too long
    – Marco
    Commented Oct 1, 2019 at 18:11
  • comm -12 file1 file2 didn't correctly get all the common lines but sort file1 file2|uniq -d did Commented Aug 22, 2021 at 21:16
  • grep is faster than awk I found
    – ericcurtin
    Commented Oct 25, 2023 at 18:51
4

Since you're running on Linux, I suppose it's GNU/Linux and you are using the GNU diff command.

If you're running the GNU diff command, this is how to see all changed lines as well as common lines:

diff \
--old-line-format='-%l
' \
--new-line-format='+%l
' \
--unchanged-line-format=' %l
' \
"$@"

This is similar to classic diff output, but no file names or separator lines appear in output, and old lines are marked with -, new lines are prefixed with +, and common lines are prefixed with a space .

Here's an example shell script and the resulting output on test files:

$ cat diffcomm.sh
#!/bin/sh
diff \
--old-line-format='-%l
' \
--new-line-format='+%l
' \
--unchanged-line-format=' %l
' \
"$@"
$ cat > filea
a
b
c
d
$ cat > fileb
a
z
d
$ ./diffcomm.sh  filea fileb
 a
-b
-c
+z
 d
$

You can modify the output format for each class of line.

See man diff or info diff or the GNU diffutils documentation for more information.

Not the answer you're looking for? Browse other questions tagged .