0

I have many files of the same name distributed in many sub-directories with different names (though all at the same level). I'd like to concatenate all the files of the same into a new file with that name. I'd like this new file to be in the parent directory.

I've already tried a few answers posted here on SE: How to move files with same name and concatenate

Here's what I've tried and the issues:

find */*/*/seq/in/ -type f -name '*.fasta' -exec bash -c 'cat "{}" >> new_file' \;

This concatenated all files with suffix .fasta, into one file call new_file, not just those with matching names.

for file in */*/*/seq/in/*.fasta; 
do 
cat "$file" >> "$file.cat" done

This just made a copy of each file in the same original sub-directory and appended it with .cat

What am I doing wrong? Thank you so much!

|improve this question
  • Your scripts are doing exactly what you specified. The first one creates a file called new file in the same directory and appends the data into it. The second creates a file in the same directory with the suffix of .cat and appends the data into it. If by parent directory, you mean the top level directory then you need to prepend that to the path of the file. You can set the file to be /top/pathto/directory/newfile in your script. – Nasir Riley Apr 5 '18 at 16:31
  • If "user284583", "user284586" and "user284592" are all the same person, I suggest that you log in as the user that you used to write the question, update the question with the relevant information about what issues you have with applying my solutions, and register the account properly. – Kusalananda♦ Apr 5 '18 at 18:16
  • Rejected edit posted by anonymous user: UPDATE: Sorry about the posting/comment issues. I have now gotten your answer working, thank you very much! However just as a note it is very slow - it has been running for a couple hours and has only produced about 70 of the concatenated files out of about 600 total that will be/need to be created. – user259412 Apr 5 '18 at 22:04
3

You will have to do this in two steps:

  1. Find all unique Fasta filenames.
  2. For each name, find all the files with that name and concatenate them.

With code:

Find all Fasta files (this assumes sane filename with no embedded newlines):

find . -type f -path '*/seq/in/*.fasta' -exec basename {} ';' | sort -u -o file.list

Then for each name, concatenate all files with the same name. The resulting files will be placed in the new directory in the current directory. The name of the new file will be the same as that of the concatenated files.

mkdir new
while read -r name; do
    find . -type f -path "*/seq/in/$name" -exec cat {} + >"new/$name"
done <file.list

The thing that you might want to add to each of the find invocations is -mindepth 6 -maxdepth 6 (or the appropriate depth, 6 may be off by one), so that the files are all taken from the same depth in the file hierarchy. Insert these options just before -type f.

Alternative suggested by steeldriver in comments, which bypasses the creation of a separate list of Fasta filenames:

mkdir new
find . -type f -path '*/seq/in/*.fasta' \
    -exec sh -c 'for pathname do cat "$pathname" >>"new/${pathname##*/}"; done' find-sh {} +
|improve this answer
  • Hmm... shouldn't something like find dir -name '*.fasta' -exec sh -c 'for f; do cat "$f" >> "$(basename "$f" ".fasta").fasta"; done' sh {} + work also? – steeldriver Apr 5 '18 at 16:27
  • Thank you all for your replies, but I'm still having issues. The file.list is empty when I try that option. Also, just the be clear, the result will be hundreds of concatenated files - made from numerous files of the same prefix. – user284586 Apr 5 '18 at 16:42
  • @steeldriver Yes, that would also work, except you're doing some unnecessary filename suffix juggling there. I'll add that as an alternative. – Kusalananda♦ Apr 5 '18 at 17:05
  • Thanks yes I realized that after it was too late to edit the comment - I guess I had in mind that the OP might want to name the aggregate files something like basename_all.fasta – steeldriver Apr 5 '18 at 17:07
  • I've now also tried what is posted from steeldriver (amended by Kusalananda), which sounds like what I want, but I don't know enough about the syntax to understand what went it wrong. I produced new files but only concatenated two files and there are many more matching in other sub-directories. – user284592 Apr 5 '18 at 17:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.