Copying files from specific subsubdirectories based on subdirectory name into single directory then renaming

Question

So here's an excerpt of my directory tree:

|-- 20070214_014700.a  
|   |-- info  
|   |-- processed  
|       |-- HH.EL..BHZ  
|       |-- AZ.AS..HHZ  
|       |-- (hundreds more)  
|   |-- raw  
|   |-- resp  
|-- 20100737_055560.a  
|   |-- info  
|   |-- processed  
|   |-- raw  
|   |-- resp  
|-- 20190537_028750.a  
|   |-- info  
|   |-- processed  
|   |-- raw  
|   |-- resp  

I have ~13,000 directories (ending in .a) and each directory has a 'processed' subdirectory which has files I'd like to copy from every processed/ directory into a single directory. Some of these files may have the same filename so I'd also like to rename them based on their parent directory. I'm not too picky but something similar to:

20070214_014700_HH.EL..BHZ

The whole dataset is 3 TB so I've been testing on just a few directories using 'find':

find . -name processed -exec cp -r '{}' 'test/{}' \;

For some reason this dumps some files into test/ but also creates another processed/ directory inside of that. I'm not sure how to include a copy command and renaming function into find at the same time so any advice would be great. Thanks for the help.

Answer 1

find . -type f -path "./*.a/processed/*" -exec sh -c '
for path; do
  prefix=${path%%.a/processed*}
  cp "$path" "test/${prefix##*/}_${path##*processed/}"
done
' sh {} +

Option -type f searches for regular files in the given path and the -exec option starts a shell script with find's result as arguments ({} +). In the for-loop each argument is assigned to the path variable.

Example: If variable path is ./20070214_014700.a/processed/AZ.AS..HHZ, then

• prefix=${path%%.a/processed*} removes the suffix -> ./20070214_014700
• ${prefix##*/} removes the prefix to the first / -> 20070214_014700
• ${path##*processed/} also removes the prefix and leaves the filename -> AZ.AS..HHZ

The resulting target filename of the cp command is test/20070214_014700_AZ.AS..HHZ.