Remove complete line in text files based on Tags

Question

For example : 01.txt, 02.txt, 03.txt and 04.txt has following values.

HDR|A|B|C|D|F|||||||
DTL|1|2|3|4|5|||||||||||
DTL|5|6|7|8|9|||||||||||
TLR|A|B|C|D|F|||||||

I have to remove complete lines based on tags like HDR, TLR.

Once HDR and TLR lines are removed from four files, it should merge into single file .

we should merge in such way , should maintain files orders after removing HDR , TLR lines.(01_latest.txt, 02_latest_txt, 03_latest.txt and 04_latest.txt)

How can we achieve in UNIX Script?

Answer 1

You can remove those lines using grep too:

$ grep -vE "HDR|TLR" *.txt > merged.txt

This will skip any lines that contain a "HDR" or "TLR". The | is an or operator so you can expand it as needed too, "STR1|STR2|STR3", for example. Looping through the files is guaranteed so long as the names are consistently 01.txt, 02.txt, etc.

You can convince yourself of this with the following example.

$ ls -1
01.txt
02.txt
03.txt
04.txt
05.txt
06.txt
07.txt
08.txt
09.txt
11.txt
12.txt
13.txt
14.txt
15.txt
16.txt
17.txt
18.txt
19.txt

$ echo *.txt
01.txt 02.txt 03.txt 04.txt 05.txt 06.txt 07.txt 08.txt 09.txt 11.txt 12.txt 13.txt 14.txt 15.txt 16.txt 17.txt 18.txt 19.txt

Here you can see that the shell expanded *.txt in the sequential order that you're looking for.

Answer 2

You can remove those lines and merge everything using

sed '/^\(HDR\|TLR\)|/d' [0-9]*.txt > merged.txt

Here it is assumed, that the first entry of the line that should be removed is exactly HDR or TLR.