Extract information from multiple log files

Question

I am trying to make a .txt file that contains only one line gathered from other 30 .log files. To extract only one line I used this:

$ sed -n '/Num mapped reads/p' /home/travc/seq_v2/AgamP4_v2/samples/ERS224561/qualimap/qualimap.log > /data/home/odkirling/Mali/Yeah1.txt

It works great but now I need to do it for other 29 files, how can I do that?

Answer 1

You can try something like

for filepath in $(ls -1 /home/travc/seq_v2/AgamP4_v2/samples/*/qualimap/qualimap.log);
do
  sed -n '/Num mapped reads/p' $filepath >> /data/home/odkirling/Mali/Yeah1.txt
done

This will need the path before and after * to be exactly the same.

Update

If your files are in a sequence, say ERS224561 to ERS224591, you can use seq 61 91. In this case, the script will be

for filenum in $(seq -w 30 150);
do
  sed -n '/Num mapped reads/p' "/home/travc/seq_v2/AgamP4_v2/samples/ERS2245$filenum/qualimap/qualimap.log" >> /data/home/odkirling/Mali/Yeah1.txt
done

Or let's say you have a list of directories in a text file, in the following format

ERS224591
ERS224592
ERS224523
ERS224525
.
.
.

then you can do something like

for dirname in $(cat dir_names.txt);
do
  sed -n '/Num mapped reads/p' "/home/travc/seq_v2/AgamP4_v2/samples/$dirname/qualimap/qualimap.log" >> /data/home/odkirling/Mali/Yeah1.txt
done

dir_names.txt is the file containing your directory names. Similarly, you should be able to replace any part of the file path based on your requirements.

Answer 2

sed operates on an input stream that doesn't need to be limited to a single file; use multiple files instead, e.g. by supplying wild card characters in the files' paths like .../ERS*/.../*log