0

Is there a cleaner way? A single pass of the large input.txt file would be preferred. Solution so far:

grep -e "\[.*fred" input.txt |sed 's/.*\[\([^]]*\)\].*/\1/g' |grep -vf /dev/stdin input.txt

Explanation:

grep -e "\[.*fred" input.txt |

Find the lines tagged with "fred" that have a left square bracket before "fred" in the line. send to

sed 's/.*\[\([^]]*\)\].*/\1/g' |

Find the IDs between the square brackets identifying all "fred" entries in the file. The number of the same ID entries per "fred" tag is not fixed, the example shows 2 lines per "fred" but it could be any number. send to

grep -vf /dev/stdin input.txt

remove all lines with these IDs from the file. (replace /dev/stdin with '-' i.e. piped input, as you require)

input.txt:

Jan  2 10:45:01 TecMint CRON[3383]: pam_unix(cron:session): session opened for user root by (fred)
Jan  2 10:45:01 TecMint CRON[3383]: pam_unix(cron:session): session closed for user root
Jan  2 10:51:34 TecMint sudo:  tecmint : TTY=unknown ; PWD=/home/tecmint ; USER=root ; COMMAND=/usr/lib/linuxmint/mintUpdate/checkAPT.py
Jan  2 10:51:34 TecMint sudo: pam_unix(sudo:session): session opened for user root by (fred)
Jan  2 10:51:39 TecMint sudo: pam_unix(sudo:session): session closed for user root
Jan  2 10:55:01 TecMint CRON[4099]: pam_unix(cron:session): session opened for user root by (fred)
Jan  2 10:55:01 TecMint CRON[4099]: pam_unix(cron:session): session closed for user root
Jan  2 11:05:01 TecMint CRON[4138]: pam_unix(cron:session): session opened for user root by (fred)
Jan  2 11:05:01 TecMint CRON[4138]: pam_unix(cron:session): session closed for user root
Jan  2 11:09:01 TecMint CRON[4146]: pam_unix(cron:session): session opened for user root by (fred)
Jan  2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session opened for user root by (bill)
Jan  2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session closed for user root

Desired output:

Jan  2 10:51:34 TecMint sudo:  tecmint : TTY=unknown ; PWD=/home/tecmint ; USER=root ; COMMAND=/usr/lib/linuxmint/mintUpdate/checkAPT.py
Jan  2 10:51:34 TecMint sudo: pam_unix(sudo:session): session opened for user root by (fred)
Jan  2 10:51:39 TecMint sudo: pam_unix(sudo:session): session closed for user root
Jan  2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session opened for user root by (bill)
Jan  2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session closed for user root
0

I think that if you want to delete anywhere in the file lines that match IDs extracted from anywhere in the file, then there is no way to do it in a single pass.

However if you only need to delete lines going forward from the most recent match, then you could do something like:

$ perl -lne '$pid = $1 if /(\[.*?\]).*fred/; print unless index($_,$pid) > -1' input.txt 
Jan  2 10:51:34 TecMint sudo:  tecmint : TTY=unknown ; PWD=/home/tecmint ; USER=root ; COMMAND=/usr/lib/linuxmint/mintUpdate/checkAPT.py
Jan  2 10:51:34 TecMint sudo: pam_unix(sudo:session): session opened for user root by (fred)
Jan  2 10:51:39 TecMint sudo: pam_unix(sudo:session): session closed for user root
Jan  2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session opened for user root by (bill)
Jan  2 11:05:01 TecMint CRON[4199]: pam_unix(cron:session): session closed for user root
  • thanks, I see your point. it is the latter, similar to most log file entry generation, 'everything' increments going forward/down the file. i'm even less knowledgeable of perl than grep/sed/awk. an elegant solution to be sure and one to fall back on. i've read suggestions that performance might be an issue if processing large files with perl compared with grep/sed/awk. – BillDoor Oct 7 '18 at 7:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.