loop through multiple folders and perform commands then write to file

Question

I have 10 folders that I want to loop through while executing the following command:

awk 'FNR > 1' *.csv | sort -sk 1,2 | sort -sk 3,3 > ${f}_appended.dat

With a for loop wrapper, like this:

for f in */ ; do awk 'FNR > 1' *.csv | sort -sk 1,2 | sort -sk 3,3 > ${f}_appended.dat; done

It only writes _appended.dat in each folder and does not perform the command on files in the folder it is looping through, but rather the root folder (if there are any).

DESIRED OUTCOME:

Have an _appended.dat file in each folder that was generated with the awk command. So 10 folders with a unique file generated based on the contents of .csv files in each folder.

Answer 1

This will run the awk command on the *.csv files in each subdirectory, leaving behind a {dirname}_appended.dat file in that directory containing the output:

for f in */
do
    awk 'FNR > 1' "$f"/*.csv | sort -sk 1,2 | sort -sk 3,3 >"${f}${f%/}_appended.dat"
done

Discussion

Consider:

for f in */ ; do awk 'FNR > 1' *.csv | sort -sk 1,2 | sort -sk 3,3 > ${f}_appended.dat; done

On every run through the loop, this processes the *.csv files in the current directory. To get the files in the $f directory, use "$f"/*.csv in place of *.csv.

Also, the above puts the results of each run in the file ${f}_appended.dat

Alternative

Another approach is to cd into each directory and run the awk command:

for f in */
do
    ( cd "$f"; awk 'FNR > 1' *.csv | sort -sk 1,2 | sort -sk 3,3 >"${f%/}_appended.dat" )
done

The cd command is run inside a subshell (delimited by parentheses). Consequently, the current directory is restored to its original value as soon as the subshell exits.