0

I'm trying to find all lines of a file not being after a specific pattern.

For some time I had an issue with my history using GNU bash (version 4 and 5) where commands appeared in duplicates. I assumed this was due to the fact that in my .bashrc I had the following line:

 PROMPT_COMMAND="history -a; history -n; $PROMPT_COMMAND"

and since I'm using terminal multiplexers (screen and/or tmux) the above mentioned command gets executed several times (therefore echo $PROMPT_COMMAND results in history -a; history -n; history -a; history -n;

In some situations (especially when doing stuff concomitantly on different panes/windows/frames/buffers) the last command I entered was stored twice or even more often in my ~/.bash_history. This led to entries like the following:

#1596110297
yadm list -a | xargs -t ls -l
yadm list -a | xargs -t ls -l

Needless to say, this is pretty annoying. I just (hopefully) found a fix for the history-issue (by changing the command to PROMPT_COMMAND="history -a; history -n) but correction: this did NOT solve the issue with duplicated entries in the history.

Now I'd like to get rid of the duplicated entries.

Therefore I'm currently trying to find a regular expression to mark everything except lines starting with # and one line after that. My first idea was to combine grep -v (to invert the selection) and grep -A 1 (to get additionally one line after the matching pattern). But

grep -v "^#" -A 1 ~/.bash_history

did not yield the result I hoped for.

Therefore my question: does anyone have a good idea on how to do that using grep? If not: how could I accomplish this with other tools (sed, awk, ...)?

0

As far as I understand grep -v "^#" -A 1 means to print the lines that don't start with a hash sign, and one line after each. But don't you want the opposite, print the lines that do start with a hash sign, and one line after?

Given a test file:

#123
echo this
echo this
#456
echo that
echo that
echo that
#789
echo third

grep -A1 ^# history.txt |grep -vxFe -- prints:

#123
echo this
#456
echo that
#789
echo third

The second grep is to get rid of the group separators grep -A prints.

Alternatively uniq history.txt should work to print just one of each set of consecutive identical lines.

3
  • My goal is to select only the lines in the files which don't start with a # and which aren't directly after a line with starts with a # (so that in a next step I could delete those lines). I'm aware that I could use your solution to generate a new file and use that to replace my ~/.bash_history, but I'm interested if a better solution exist.
    – n0542344
    Aug 16 '20 at 21:54
  • @n0542344, any solution that would select the lines to be deleted would still end up having to copy the files to be kept into another temporary file (or into memory) before putting that in place of the original. That's what e.g. sed -i and others do. The OS doesn't really support line-based operations on files. And anyway, even if you had a grep command that printed the duplicate lines, going from there to removing them (based on content) is more trouble than just printing out the ones you want to keep.
    – ilkkachu
    Aug 16 '20 at 22:09
  • you are right of course - thanks for the clarification!
    – n0542344
    Aug 24 '20 at 11:42
0

using Raku (née Perl6)

This seems like a job for the "flip-flop" operator, available in a number of scripting languages. Below is an answer using the Raku programming language (previously known as Perl6). First start by creating a more extensive test file:

$ cat repeated_log.txt
#1596110297_1
A_yadm list -a | xargs -t ls -l
B_yadm list -a | xargs -t ls -l
#1596110297_2
C_yadm list -a | xargs -t ls -l
D_yadm list -a | xargs -t ls -l
E_yadm list -a | xargs -t ls -l
#1596110297_3
F_yadm list -a | xargs -t ls -l
G_yadm list -a | xargs -t ls -l
H_yadm list -a | xargs -t ls -l
I_yadm list -a | xargs -t ls -l
#1596110297_4
#1596110297_5

Now for a one-liner using Raku's fff flip-flop operator, which implements "sed-like" behavior. The capture turns ON for lines where the first regex sees (at the start-of-line ^^) a literal "#" character. Once ON, the capture ignores the first regex and evaluates against the second regex, turning OFF when it finds a match against lines that are missing (at the start-of-line ^^) a "#" character. The 'negative' regex is implemented in the code below using <-[#]>, which is a negative "Enumerated Character Class" and a real feature of the Raku language:

$ raku -ne '.put if /^^ "#" / fff /^^ <-[#]> /;' repeated_log.txt
#1596110297_1
A_yadm list -a | xargs -t ls -l
#1596110297_2
C_yadm list -a | xargs -t ls -l
#1596110297_3
F_yadm list -a | xargs -t ls -l
#1596110297_4
#1596110297_5

Actually, the first regex (to the left of the fff infix operator) could be written using <+[#]> which is a positive "Enumerated Character Class", for a more parallel construction:

$ raku -ne '.put if /^^ <+[#]> / fff /^^ <-[#]> /;' repeated_log.txt
#1596110297_1
A_yadm list -a | xargs -t ls -l
#1596110297_2
C_yadm list -a | xargs -t ls -l
#1596110297_3
F_yadm list -a | xargs -t ls -l
#1596110297_4
#1596110297_5

Also, it seems to me you can improve your regex by demanding a match for-or-against a start-of-line "#" followed by one or more digits, i.e. <digit>+, see below:

$ raku -ne '.put if /^^ <+[#]> <digit>+ / fff /^^ <-[#]> <-digit>+ /;' repeated_log.txt
#1596110297_1
A_yadm list -a | xargs -t ls -l
#1596110297_2
C_yadm list -a | xargs -t ls -l
#1596110297_3
F_yadm list -a | xargs -t ls -l
#1596110297_4
#1596110297_5

[All code above removes the duplicated lines starting with B, D, E, G, H, and I. The only quirk I've noticed is two consecutive target lines like "#1596110297" will appear in your output, but it's not clear to me if your input file will ever contain such consecutive lines].

https://raku.org/

3
  • thanks for the hint, that's a very interesting approach - but I'd need to install raku for that (which I don't have currently). Not that this is a major issue but to be honest I'd prefer a solution using tools from the coreutils/moreutils which are available everywhere...
    – n0542344
    Aug 24 '20 at 11:54
  • @n0542344 well hopefully Raku should be an easy install, see: rakudo.org/downloads . But I get what you're saying. Aug 24 '20 at 20:45
  • true, the even easier solution would be sudo apt install rakudo ;)
    – n0542344
    Aug 25 '20 at 19:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.