1

I have 10 folders that I want to loop through while executing the following command:

awk 'FNR > 1' *.csv | sort -sk 1,2 | sort -sk 3,3 > ${f}_appended.dat

With a for loop wrapper, like this:

for f in */ ; do awk 'FNR > 1' *.csv | sort -sk 1,2 | sort -sk 3,3 > ${f}_appended.dat; done

It only writes _appended.dat in each folder and does not perform the command on files in the folder it is looping through, but rather the root folder (if there are any).

DESIRED OUTCOME:

Have an _appended.dat file in each folder that was generated with the awk command. So 10 folders with a unique file generated based not eh contents of .csv files in each folder.

2

This will run the awk command on the *.csv files in each subdirectory, leaving behind a {dirname}_appended.dat file in that directory containing the output:

for f in */
do
    awk 'FNR > 1' "$f"/*.csv | sort -sk 1,2 | sort -sk 3,3 >"${f}${f%/}_appended.dat"
done

Discussion

Consider:

for f in */ ; do awk 'FNR > 1' *.csv | sort -sk 1,2 | sort -sk 3,3 > ${f}_appended.dat; done

On every run through the loop, this processes the *.csv files in the current directory. To get the files in the $f directory, use "$f"/*.csv in place of *.csv.

Also, the above puts the results of each run in the file ${f}_appended.dat

Alternative

Another approach is to cd into each directory and run the awk command:

for f in */
do
    ( cd "$f"; awk 'FNR > 1' *.csv | sort -sk 1,2 | sort -sk 3,3 >"${f%/}_appended.dat" )
done

The cd command is run inside a subshell (delimited by parentheses). Consequently, the current directory is restored to its original value as soon as the subshell exits.

  • The files were already going into the respective directories, but I could not get them named with that directory as a prefix. Your solutions do not provide the desired outcome either, though they do offer me insight into the use of quotes and parentheses in calling variables. In fact, the only difference between your suggestion and my original code is putting quotes around the $f variable ahead of the *.csv files. That definitely pointed the control to where the files to append are, but I still cannot get the directory to be the prefix in the filename. – geokrowding Mar 3 '15 at 9:41
  • If you select the files with the globbing pattern */ then each name in the variable will contain the slash as part of the name. To remove that terminating slash for your filename use "${f%/}_appended.dat". – Janis Mar 3 '15 at 13:16
  • @janis Good point. I will fix that as soon as I get a chance to sit in front of a proper keyboard. – John1024 Mar 3 '15 at 19:44
  • John1024, this was actually meant as a comment for geokrowding; I think the fix I posted should solve the problem he still has. – Janis Mar 3 '15 at 21:38
  • @geokrowding Just to remove all doubt, if you have a directory named, say, dir, do you want the output file to be ./dir/_appended.dat or ./dir/dir_appended.dat` or ./dir_appended.dat? – John1024 Mar 3 '15 at 22:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.