3

I have a file where each line starts with a unique identifier, for example:

$ cat source.txt
aaa text
bbb text
ccc text
ddd text
eee text

I am trying to use fgrep with the option -f to save lines that contain identifiers listed in another file (lets call it "list.txt") into a new file:

fgrep -f list.txt source.txt > new.txt

The problem is that some of the identifiers in the "list.txt" are repeated, for example:

$ cat list.txt
aaa
ccc
ccc
ccc
eee
eee

Here, grep is treating each of these repeated identifiers as if it was listed only once, giving me a result like this:

$ cat new.txt
aaa text
ccc text
eee text

Instead, I would like to save the lines containing identifiers repeated in the list.file the exact amount of times those identifiers are repeated. In this case the preferable end result should be:

$ cat new.txt
aaa text
ccc text
ccc text
ccc text
eee text
eee text

Is there some easy trick to force grep to stop treating the repeated identifiers as if they were listed only once? Alternatively, is there a different way to obtain the result I want without using grep, for example with awk?


For better visualization of the problem, here is an actual line from source.txt:

head -n 1 source.txt | cat -T
GCF_000005825.2_WP_003320558.1 MULTISPECIES: IS21-like element helper ATPase IstB [Bacillaceae]^IMNEQIQAYAKRLKLSWIRENFNQIEAETNEEYLLKLFEKEVQNREERKVNLLLSQAQLPKTGSTPFQWEHIQIPQGIERTAVINGDFIKERENLILYGGVGTGKTYLATLLSLNAIHRFGSQVKFYTVAGLVNKLIEANQKNTLPKLMKQIEKLDLLILDELGYIPLNKEGAELLFQVISMCYENRSIVITTNLQFGQWNHVFGDPILTEAVIDRLIHHSHLLVFKGDSFRYKESLLHQ

And this would be a matching identifier for it:

GCF_000005825.2_WP_003320558.1
2
  • 1
    I would be surprised if you could do this simply with grep. When grep considers a line, the line either matches or not, thus respectively it gets printed or not. Jul 11, 2022 at 14:26
  • 1
    For your lexicographically ordered example above, you could simply use join list.txt source.txt Jul 11, 2022 at 15:50

2 Answers 2

2

Here's one way to do with awk (assuming source.txt isn't too big to run into memory issues).

$ awk 'NR==FNR{a[$1]=$0; next} $0 in a{print a[$0]}' source.txt list.txt
aaa text
ccc text
ccc text
ccc text
eee text
eee text
  • NR==FNR{a[$1]=$0; next} this builds an array with first field as key and entire input line as the value (for the first file, which is source.txt in this case)
  • $0 in a{print a[$0]} when list.txt file is being processed, each line is checked if it is present as a key in array a and the corresponding line is printed

Here's a modified solution that should perform better memory wise (assuming lines in source.txt have exactly two fields and separated by single space).

awk 'NR==FNR{a[$1]=$2; next} $0 in a{print $0, a[$0]}'

Instead of saving entire line, only second field is saved in the array. While printing, the key is prefixed.

3
  • 1
    Thank you! Works like a charm even though the files are indeed a bit large.
    – 6april
    Jul 12, 2022 at 11:15
  • If your input files are indeed sorted as shown in the samples. If so, I'd be interested to know how join list.txt source.txt performs (as mentioned in comments under your question), should be better imo. I've also added an alternate solution that might suit your large inputs better.
    – Sundeep
    Jul 12, 2022 at 13:03
  • 1
    I tested join but it took noticeably longer than your first proposed solution. Interestingly, it would do the work just fine if not for the fact that it exchanged the tabs that were present in each line with normal spaces, which I need in the next step of my workflow. I now edited my post to show how my real data looks like. The second solution unfortunately doesn't work due to the formatting of the data.
    – 6april
    Jul 13, 2022 at 8:59
1

Another approach, using xargs to do the heavy lifting.

$ grep -v "^$" list.txt | xargs -I{} grep "^{} " source.txt
aaa text
ccc text
ccc text
ccc text
eee text
eee text
$
3
  • 1
    Thank you, this works in principle, but with my full file sizes - source.txt has 49 million lines and the different list.txt files have minimum 6 million lines each - takes unfortunately way too long (I tested this also with fgrep). I should have mentioned the sizes to make it clear that this could pose an issue...
    – 6april
    Jul 12, 2022 at 11:13
  • That's doing a partial-line regexp match when a full-field string match is required so it'd output false matches. See stackoverflow.com/questions/65621325/… for more info on the various types of "pattern" matching.
    – Ed Morton
    Jul 12, 2022 at 12:11
  • Good points. Slow if millions of lines. And I've done an edit, to try address the partial-line regexp aspect, thanks for highlighting it.
    – steve
    Jul 12, 2022 at 17:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .