1

I'm trying to edit a text file containing several duplicates. The goal is to keep only the first match of a string and remove the rest duplicate lines of the same string.

In the example file

* Title 1
** Subtitle 01
#+begin_src
  Line 001
  Line 002
#+end_src

* Title 1
** Subtitle 02
#+begin_src
  Line 001
  Line 002
#+end_src

* Title 2
** Subtitle 01
#+begin_src
  Line 001
  Line 002
#+end_src

* Title 2
** Subtitle 02
#+begin_src
  Line 001
  Line 002
#+end_src

I'd like to keep one of each * Title N and keep all other unrelated/unspecified duplicate lines on the file. So the result would be:

* Title 1
** Subtitle 01
#+begin_src
  Line 001
  Line 002
#+end_src

** Subtitle 02
#+begin_src
  Line 001
  Line 002
#+end_src

* Title 2
** Subtitle 01
#+begin_src
  Line 001
  Line 002
#+end_src

** Subtitle 02
#+begin_src
  Line 001
  Line 002
#+end_src

The traditional solutions for removing duplicates like

uniq file.txt

Useful AWK One-Liners to Keep Handy:

awk '!a[$0]++' contents.txt

shell - How to delete duplicate lines in a file without sorting it in Unix - Stack Overflow

perl -ne 'print if ! $x{$_}++' file

delete every duplicate indiscriminately.

I tried using variations of these solutions and also GNU sed in a loop format like

duplicateLines=$(grep -E "^\* .*" file.org | uniq)
  printf '%s\n' "$duplicateLines" | while read -r line; do
  sed "s/$line//g2" file.org
done

with no success. I don't mind absolute performance so doing multiple iterations like calling sed inside a loop to remove one specified string at a time is no problem.

Any insight would be very much appreciated.

It would be nice to be able to do this inside a shell script but I'm open to alternative solutions like Python, C, Java, etc., just tell me what the function/library name is and I'm searching for it there.

Thanks.

2 Answers 2

1

You could trivially modify the awk !a[$0]++ paradigm to make the incrementing pattern-dependent:

awk '!a[$0]; /^\* Title/{a[$0]++}' file
1
  • OMG this is exactly what I was searching for. Thank you very much.
    – yeyin33455
    Dec 30, 2021 at 2:14
0

In awk we idiomatically use an array named seen[] to distinguish between the first vs subsequent appearance of a string, e.g.:

awk '!seen[$0]++'

would only output the first occurrence of every line.

In your case you only want to use that when the current line starts with * Title so that'd be:

$ awk '!( /^\* Title/ && seen[$0]++ )' file
* Title 1
** Subtitle 01
#+begin_src
  Line 001
  Line 002
#+end_src

** Subtitle 02
#+begin_src
  Line 001
  Line 002
#+end_src

* Title 2
** Subtitle 01
#+begin_src
  Line 001
  Line 002
#+end_src

** Subtitle 02
#+begin_src
  Line 001
  Line 002
#+end_src

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .