Batch sorting multiple files and removing duplicate lines from multiple files - in place if possible

Question

I have a folder containing multiple text files.

apple.txt, book.txt, teeth.txt etc.

I created a bash script with multiple sed commands to process every text file in place as follows:

    ...
    find /directory/. -type f -exec sed -i 's/one/two/g' {} \;
    ...

These lines work fine.

Is there a way to process all the individual text files in place in a similar manner (single command) to perform these operations:

1. sort lines from longest to shortest string (each line contains one string - no spaces).

2. remove any duplicate lines.

Answer 1

I'm not sure what the requirement for a single command is, but this comes pretty close:

(cd /directory/; find -type f -printf '%P\0' | while IFS= read -r -d '' fn; do
    awk '{ print length, $0 }' "$fn" | sort -nur | sed -r 's/^[0-9]+ //' > "$fn.~"
    mv "$fn.~" "$fn"
done)

The first line simply reads the filenames in as robust a fashion as possible.

The second line is where all the work happens:

• awk adds a character count (+ space) to the beginning of each line
• sort -n (numeric), -u (unique), and -r (reverse) process the file
• sed removes the leading character count
• then it is dumped to a temporary file

The mv line then writes the temporary file over the original (as it can't be done during the pipeline).