I have this command to remove duplicate lines from a file and save the unique lines in a new file

uniq telphone.txt output.txt

However Im trying to find a command that scans all .txt files in the current folder and removes duplicates and saves it as the same filename

That will find telphone.txt remove duplicates and save it as telphone.txt.

Then if a new file homes.txt exists, to do the same and save it as homes.txt.

Is it possible?

2 Answers 2



for file in ./*.txt; do
    cp "$file" "$tmpfile" &&
    uniq "$tmpfile" >$file"

rm "$tmpfile"

This loops over all the names that end in .txt in the current directory and removes duplicated lines from each of them in turn.

The file is first copied to a temporary file, and this temporary file is then processed with uniq. The output is written to the original filename.

Doing it this way ensures that the permissions on the original files are not altered (which they may be if one did uniq "$file" >"$tmpfile" followed by cp "$tmpfile" "$file").

The call to uniq is also depending on the cp to be successful. If the cp failed, the uniq command's redirection would otherwise truncate the original file and you would loose that data.


You need to use a temp file, as uniq or > will truncate the file when opening.

Try this,

for f in *.txt; do
    uniq "$f" "$tmpf" && mv "$tmpf" "$f"

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .