Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top
bash-3.00$ cat f.txt
-rw-r--r--   1 mukesh   other        102 Nov  5 18:32 f1.txt
-rw-r--r--   1 mukesh   other         19 Nov  5 18:32 f2.txt
-rw-r--r--   1 mukesh   other        204 Nov  5 18:32 f3.txt //DUPLICATE
-rw-r--r--   1 mukesh   other        204 Nov  6 18:32 f3.txt  //DUPLICATE
-rw-r--r--   1 mukesh   other        204 Nov  6 18:32 f100.txt

Lets say i have the above file-names listed in a file called f.txt. I want to list out all the file-names which have a duplicate(only file-name), display the duplicate file name and number of duplicates.

Eg : In the above list f3.txt has a duplicate and there are a total of 2 f3.txt filenames found.

share|improve this question
What do you want to use? – Dadam Nov 6 '11 at 14:21
I could use any standard utility like awk,loops etc or aything you suggest.I think I need to take out the last column using awk and keep comparing using a loop inside a loop(may be a while do loop) I am trying that....Hope it works....I would appreciate any valuable inputs – munish Nov 6 '11 at 14:30
Sort and uniq are standard utilities, they're as likely to be there as awk. – Kevin Nov 6 '11 at 14:47
Thanks, @kevin I hope that it would be..I will edit my question anyway. – munish Nov 6 '11 at 14:51
BTW, as far as I know, no Unix will ever allow your listing to happen. The filenames in a directory must be unique, though the differences might be unprintable characters. – Hack Saw Nov 6 '11 at 20:07
up vote 2 down vote accepted

As you mentioned awk, here is a solution in it:

awk '{d[$9]++}END{for(f in d)if(d[f]>1)print d[f],f}' f.txt

If you have also files with space characters in their names, then some workaround is needed:

awk '{f=$9;for(i=10;i<=NF;i++)f=f" "$i;d[f]++}END{for(f in d)if(d[f]>1)print d[f],f}' f.txt

Note that the above workaround may fail, for example on handling symlinks. After all, parsing ls output is not a good idea. Would be better to generate your f.txt in some other way, if possible. (For example find or stat, where you have better control over the list format.)

share|improve this answer
Your workaround won't work on a file name containing two consecutive space characters. For this input, it would be better to use substr($0,55). – Gilles Nov 6 '11 at 15:55
Oops, I missed the essential in the concatenation: not appended the $i. Thank you. – manatwork Nov 6 '11 at 16:29
Oh, I'd missed that too. My bug report still stands though. – Gilles Nov 6 '11 at 16:32
Ah, got you now. You mean it will consider “a_b” and “a__b” the same. Well, I supposed spaces will appear as “a\_b” and “a\_\_b”, as in the default ls output. (Used underscores as whitespaces are not preserved in comments.) – manatwork Nov 6 '11 at 16:40

Here's a solution in pure bash, since your question clearly indicates that you have bash 3 available:


if [[ -t 0 ]]; then
    printf '%s\n' 'No input!'
    exit 1

while read -ra _file; do

    for _file_name_already_known in "${_files[@]}"; do
        if [[ ${_file_name_already_known} == "${_file}" ]]; then
            for _file_name_already_printed in "${_already_printed[@]}"; do
                [[ ${_file_name_already_printed} == "${_file}" ]] && continue 3
            printf '%s\n' "${_file}"
            _already_printed+=( "${_file}" )

    _files+=( "${_file}" )


$ ./lsdups < f.txt

Run as ./lsdups < file.

Bear in mind that ls output is not the best thing to use, however. http://mywiki.wooledge.org/ParsingLs goes over some reasons why you should not do this.

share|improve this answer

The uniq utility removes or lists duplicates. The input needs to be sorted. Since only the file name part is relevant, tell these utilities to ignore what's to the left of the file names.

sort -k 1.54 f.txt | uniq -s 54 -cd
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.