1
https://www.domain.com/files/G5SPNDOF/AAA-1080p.mp4.html
https://www.domain2.com/dl/G5SPNDOF/JHCGTS/AAA-1080p.mp4.html
https://www.domain.com/files/ZQWL80BG/AAA-1080p.mp4.html
https://www.domain.com/files/SVSRS0AD/BBB-1080p.mp4.html
https://www.domain.com/files/UCIONEMA/BBB-1080p.mp4.html

Given a file with above lines, how do I delete the ones that have duplicate files, but links overall are different, to end up with:

https://www.domain.com/files/G5SPNDOF/AAA-1080p.mp4.html
https://www.domain.com/files/SVSRS0AD/BBB-1080p.mp4.html
3
  • Could you say something about why you decided to keep those two lines in particular and e.g. https://www.domain2.com/dl/G5SPNDOF/JHCGTS/AAA-1080p.mp4.html and https://www.domain.com/files/UCIONEMA/BBB-1080p.mp4.html?
    – Kusalananda
    Commented Oct 11, 2022 at 9:17
  • @Kusalananda What about the most obvious reason: Those two are the first appearances of the file names?
    – Philippos
    Commented Oct 12, 2022 at 5:51
  • @Philippos Sure, I can see that, but it's not clear that this is an absolute requirement or not.
    – Kusalananda
    Commented Oct 12, 2022 at 5:58

5 Answers 5

5

Using awk :

awk -F'/' '!seen[$NF]++' file

You can also do it using sort. As your file contain a number of fields that change you need to reverse it first :

rev file | sort -u -t/ -k1,1 | rev
2

Using any sed, you'd choose the lazy "throw all known file names on the hold space" approach:

sed 'G;\_/\(.*\)\n.*\1_d;P;s_.*/__;h;d' file
  • G appends the hold space (with all known file names) to the pattern space
  • The pattern /\(.*\)\n.*\1 fits a filename between / and the newline, which gets repeated (\1) in the collected file names; in this case, delete it, no output, no changes to the hold space
  • At this point, we know we found a new word, so Print the line without the appended hold space
  • s_.*/__ removes the path, so the new path list can be copied to the hold space. d suppresses further output
2

If your list is huge then to avoid storing all of the file names in memory in awk:

$ sort file | awk -F'/' '$NF != prev{print} {prev=$NF}'
https://www.domain.com/files/G5SPNDOF/AAA-1080p.mp4.html
https://www.domain.com/files/SVSRS0AD/BBB-1080p.mp4.html
https://www.domain.com/files/ZQWL80BG/AAA-1080p.mp4.html

Otherwise see @MariusCouet's awk answer.

2
  • Are you sure keeping a list of file names in memory can be a bigger problem than sorting such a big file?
    – Philippos
    Commented Oct 12, 2022 at 6:03
  • @Philippos yes because sort is designed to efficiently handle massive amounts of input by using demand paging, etc. while awk when used as awk -F'/' '!seen[$NF]++' file will slow down greatly as the hash array size increases (it's allocated in increasingly larger blocks) before eventually failing when it runs out of memory.
    – Ed Morton
    Commented Oct 12, 2022 at 13:08
1

Using Raku (formerly known as Perl_6)

~$ raku -e '.put for lines.unique(as => *.IO.basename);'  file

Sample Input:

https://www.domain.com/files/G5SPNDOF/AAA-1080p.mp4.html
https://www.domain2.com/dl/G5SPNDOF/JHCGTS/AAA-1080p.mp4.html
https://www.domain.com/files/ZQWL80BG/AAA-1080p.mp4.html
https://www.domain.com/files/SVSRS0AD/BBB-1080p.mp4.html
https://www.domain.com/files/UCIONEMA/BBB-1080p.mp4.html

Sample Output:

https://www.domain.com/files/G5SPNDOF/AAA-1080p.mp4.html
https://www.domain.com/files/SVSRS0AD/BBB-1080p.mp4.html

Raku understands directory conventions, so you can use .IO to convert your string to an IO object, and get the filename with basename. Raku's unique function is clever enought to take an as parameter, so the full file path is returned.

https://docs.raku.org/routine/unique
https://docs.raku.org/routine/IO
https://docs.raku.org/routine/basename
https://raku.org

0

Using GNU datamash to prefix the original data with the basename (filename at the end) from each URL, and then to group the data by the added column and extract the first URL from each group. The added field is then removed with cut at the end of the pipeline. This is reminiscent of how one might do a Schwartzian transform.

$ datamash basename 1 cut 1 <file | datamash groupby 1 first 2 | cut -f 2
https://www.domain.com/files/G5SPNDOF/AAA-1080p.mp4.html
https://www.domain.com/files/SVSRS0AD/BBB-1080p.mp4.html

You can instead get the last URL from each group:

$ datamash basename 1 cut 1 <file | datamash -s groupby 1 last 2 | cut -f 2
https://www.domain.com/files/ZQWL80BG/AAA-1080p.mp4.html
https://www.domain.com/files/UCIONEMA/BBB-1080p.mp4.html

... or a random one:

$ datamash basename 1 cut 1 <file | datamash -s groupby 1 rand 2 | cut -f 2
https://www.domain2.com/dl/G5SPNDOF/JHCGTS/AAA-1080p.mp4.html
https://www.domain.com/files/SVSRS0AD/BBB-1080p.mp4.html
$ datamash basename 1 cut 1 <file | datamash -s groupby 1 rand 2 | cut -f 2
https://www.domain.com/files/G5SPNDOF/AAA-1080p.mp4.html
https://www.domain.com/files/SVSRS0AD/BBB-1080p.mp4.html
$ datamash basename 1 cut 1 <file | datamash -s groupby 1 rand 2 | cut -f 2
https://www.domain.com/files/ZQWL80BG/AAA-1080p.mp4.html
https://www.domain.com/files/UCIONEMA/BBB-1080p.mp4.html

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .