1

There's a large file that I need to download, but it's too big to fit on the server (~1Tb), so instead I'd like to only keep lines that I need, which are found in a pattern text file.

wget www.foo.com/file.txt | grep -f file.txt 

The pattern text file contains patterns like bar and the large text file would be like

test blah blah foo
blah test bar blah 

I've tried various combinations of this but none of them retained the right lines in the file. Is this possible?

1
  • 1
    What is actually extracted with your grep if not the lines that contains the matches of the patterns in file.txt? It is difficult to debug your issue since we don't know what's happening and have no way of testing your code to try to reproduce the issue for ourselves. Is it only the fact that wget saves the file to disk by default?
    – Kusalananda
    Feb 2, 2022 at 15:19

1 Answer 1

4

The simple way with STDOUT (wget -q URL -O-) and the search pattern in a file runs without problems:

echo HERR > /tmp/search.txt
echo Frau >> /tmp/search.txt
wget -q https://archive.org/stream/DieBibel-AltesUndNeuesTestament-NachMartinLuther/DieBibel-Deutsch-Pdf_djvu.txt -O- | grep -f /tmp/search.txt

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .