Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a folder with some files (snippet of the contents of the folder)


My focus is on the 3rd (50,50,51,50) and the 5th (WRS20140,WRS20140,WRS29232,WRS20140) blocks. How can I write a script that displays the duplicate filenames with the same 3rd block AND 5th block (The duplicates of the combination of the 3rd and the 5th block strings)?

So the output should list the following in the above example

share|improve this question
up vote 3 down vote accepted
ls *.DAT | awk -F. '{ if (c[$3$5]) print $0 ; c[$3$5]=$0}'

In the above, awk looks at each file name using . as a field separator. If it has seen the combination of the third and fifth fields before, it prints the file name. With your file names as input, the above produces:


MORE: Let's examine the awk commands in more detail:

if (c[$3$5]) print $0 ; c[$3$5]=$0

The above consists of two statements: one "if" statement and one assignment. The "if" statement is:

if (c[$3$5]) print $0

In this statement, c is an "associative array". This means that that you give it a key and it gives you back a value. We are using $3$5 as the key where $3 is the third "block" (what awk would call the third "field") and $5 is the fifth block. If that key was previously unassigned, then c[$3$5] returns an empty (false) value. So, if this combination of third and fifth blocks was seen before, then print $0 is executed, meaning that the whole of the file name is printed. If not, the print statement is skipped.

The second statement is:


This assigns the name of the file ($0) to the associative array under the key of the third and fifth fields: $3$5. Thus, the next time that those fields are seen in the "if" statement, the print statement will execute.

share|improve this answer
Thanks. Can you please explain the awk part of the script? Between the {} – user6123723 Feb 3 '14 at 6:37
What should I do when two nice folks give the right answer at the same time? – user6123723 Feb 3 '14 at 6:47
@user1324816 Be nice to both and recognize that I was first by two minutes (grin). – John1024 Feb 3 '14 at 6:49
@user1324816 Does the new explanation answer your questions? – John1024 Feb 3 '14 at 6:50
Yes! Thank you! – user6123723 Feb 3 '14 at 6:56

Here's how you could do it with awk: use a variable to count the number of times you've seen the same pair of 3rd and 5th field, and print out the filename if you've already seen that particular pair.

With those filenames in a file called input, this would look like:

$ awk -F. '{if (dups[$3$5]++) print $0}' input

If your filenames could contain whitespace or other funky characters, use find rather than ls to list them, with something like:

$ find . -name 'PAT1.*.DAT' -print0 | \
    awk -F. 'BEGIN{RS="\0"} {if (dups[$3$5]++) print $0}'

As a side benefit, you could inspect the dups variable in an END block to print out how many of each pair you saw in the input.

share|improve this answer
What should I do when two nice folks give the right answer at the same time? – user6123723 Feb 3 '14 at 6:48
Wait for a third person to provide something better? :-) John actually answered first, and provided more details about how this works, so his answer is arguably better. – Mat Feb 3 '14 at 6:53
Lol. Thank you so much! – user6123723 Feb 3 '14 at 6:55
On the other hand, your answer handles hostile file names, and I like the ++ trick. – John1024 Feb 3 '14 at 6:57
Can be shortened to: printf '%s\0' PAT1.*.DAT | awk -F. -vRS='\0' 'dups[$3$5]++'. Note that not all awk implementations support NUL characters in their input – Stéphane Chazelas Feb 3 '14 at 11:57

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.