I have 30 directories i have more than 300 text files in each, all text files in each directory have same naming format


I would like to sort the directories and text files in sequential order and export the data in each file into csv file

Following is the script i have written

for dir in * ; do

   paste -s -d ',' <(tail -q -n 1 "$dir"/t1/regional_vol*.txt ) >> data.csv


The output csv file that i obtain is unsorted, how can sort all the files in my directory in ascending order and export the data to csv

  • glob expansions are sorted lexically on file names by default. If you want a different sorting order, please be more explicit. – Stéphane Chazelas Nov 24 '16 at 16:30
  • If your problem is that for instance GM10 comes before GM2 in lexical order, you could switch to using zsh with its numerical glob sorting options: for dir in *(/n) and ..regional_col*.txt(n). – Stéphane Chazelas Nov 24 '16 at 16:34
  • @StéphaneChazelas Exactly , my problem is directory GM10 comes before GM2 but i dont have zsh , is there any way to solve this in bash ? – DevanDev Nov 24 '16 at 16:42

If your problem is that the order is lexical instead of being numerical, you could use zsh and do:

for dir in *(n/); do
  tail -q -n 1 "$dir"/t1/regional_vol*.txt(n) | paste -s -d ',' -
done > data.csv

The n glob qualifier causes the sorting to be numerical.

If using zsh is not an option but your ls is the GNU one, an alternative is to use GNU ls's -v option for version sort:

eval "dirs=($(ls -v --quoting-style=shell-always))"
for dir in "${dirs[@]}"; do
  eval "files=($(
    ls -vd --quoting-style=shell-always -- "$dir"/t1/regional_vol*.txt))"
  tail -q -n 1 -- "${files[@]}" | paste -sd , -
done > data.csv

Yes, that's parsing the output of ls and using eval in the same command!

But here, it's safe as ls with --quoting-style=shell-always outputs in the exact format expected by eval.

To add your row and column headers:

  eval "dirs=($(ls -v --quoting-style=shell-always))"
  for dir in "${dirs[@]}"; do
      cd -- "$dir/t1" || exit
      eval "files=($(
        ls -vd --quoting-style=shell-always regional_vol*.txt))"
      if ! "$headers_done"; then
        printf DIR
        printf ',%s' "${files[@]}"
        printf '\n'
      printf %s, "$dir"
      tail -q -n 1 -- "${files[@]}" | paste -sd , -
} > data.csv
  • Thanks it works greatly,Is there any way i can export file name of each file in directory as the name of row like this expected result , since all 30 directories have .txt files of same file names, i just need a one single row which consists of all files names in directory , i have added this line to code: paste -d, $(ls -1v "$dir"/t1/regional_vol*.txt ) > data.csv, but i dont see file names of each file in the first row – DevanDev Nov 24 '16 at 17:01
  • @DevanDev, replace $( with <( in your command. – Stéphane Chazelas Nov 24 '16 at 17:21
  • I did that ,The output i am getting is incorrect output , This is the format of output that i would like to obtain Desired result, what i need is the first row of csv should be the file names arranged in sequential order like regional_vol_GM1.txt ,regional_vol_GM2.txt ..etc . and first column of csv should be directory name like Sub1,Sub2..Sub30. I have 30 directories with naming convention Sub1,Sub2,Sub30 – DevanDev Nov 24 '16 at 17:32
  • @DevanDev, see edit – Stéphane Chazelas Nov 24 '16 at 17:40
  • Thanks a lot works great, except minute issue, in the output file on the top row i just need regional_vol_GM.txt instead of subject1/t1/regional_vol_GM.txt – DevanDev Nov 24 '16 at 17:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.