-2

It took me a while to word this best.

Problem 1:

Is there a way to find lines in a text file that have the same string at the beginning and the same strings at the end?

Beginning and end strings not necessarily equal to each other.

Text between the two strings is not consistant in number of characters or number of hyphens.

A hyphen is in between notable strings (which I added to my large text file).

For example:

File:

  AAAAA-tjhut-ghyut-FILE1.txt
  BBB-fhuj-dnbhg-frt76-FILE2.txt
  AAAAA-ghtujukr-hgute7gr-xhyutt-FILE1.txt
  AAAAA-VGYT-4d7tt-xhyutt-FILE1.txt
  CCCC-fjuk-dryy-fhuj-FILE3.txt

Lines to detect:

  AAAAA-tjhut-ghyut-FILE1.txt
  AAAAA-ghtujukr-hgute7gr-xhyutt-FILE1.txt
  AAAAA-VGYT-4d7tt-xhyutt-FILE1.txt

Problem 2:

Is there a way to remove all but one of the lines containing the duplicate strings at the beginning and end from above?

Before:

  AAAAA-tjhut-ghyut-FILE1.txt
  AAAAA-ghtujukr-hgute7gr-xhyutt-FILE1.txt
  AAAAA-VGYT-4d7tt-xhyutt-FILE1.txt

After:

  AAAAA-tjhut-ghyut-FILE1.txt

Big thanks for any advice.

Note: I think using the occurances of the first hyphen and last hyphen can be used to locate the duplicate strings.

  • 2
    yes, you can use awk to process based on first and last field with - as separator... give it a shot – Sundeep Jun 23 '17 at 4:25
  • could be there multiple different duplicates? – RomanPerekhrest Jun 23 '17 at 7:23
1

Problem 1: The idea:

  • loop twice
  • on first loop extract index (column 1 and last column)
  • count the occurances of the index
  • second loop iterate through all lines again (stored inside first loop)
  • extract index again from current row
  • if index count > 1 do print

Here is my example, store as example.awk:

#!/usr/bin/awk -f
BEGIN {
  # field separator
  FS="-";
}
{
  # remember row for second loop
  allRows[rowCount++]=$0;

  # relevant columns
  colFirst=$1;
  colLast=$NF;

  # create combined index: first column, space, last column
  # count occurences of combined index
  indexes[colFirst" "colLast]++;
}
END {

  # loop through all input rows
  for(currentRow in allRows) {

    # extract combined index from current row
    arrLen = split(allRows[currentRow],splitted,"-");
    colFirst=splitted[1];
    colLast=splitted[arrLen];

    # output row  if index count of current row > 1
    if (indexes[colFirst" "colLast] > 1) {
    # index count > 1, so print
      print allRows[currentRow];
    }
  }
}

Execute like this, if input file is named input.txt:

awk -f example.awk < input.txt

My output:

AAAAA-tjhut-ghyut-FILE1.txt
AAAAA-ghtujukr-hgute7gr-xhyutt-FILE1.txt
AAAAA-VGYT-4d7tt-xhyutt-FILE1.txt

Problem 2: As far as I understand you only want to keep the first line:

awk -f example.awk < input.txt | head -n 1

head: output the first part of files

-n 1: print the first NUM lines, so just 1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.