1

I have 30 directories i have more than 300 text files in each, all text files in each directory have same naming format

regional_vol_GM1.txt
regional_vol_GM2.txt
regional_vol_GM*.txt 

I would like to sort the directories and text files in sequential order and export the data in each file into csv file

Following is the script i have written

for dir in * ; do


   paste -s -d ',' <(tail -q -n 1 "$dir"/t1/regional_vol*.txt ) >> data.csv

 done

The output csv file that i obtain is unsorted, how can sort all the files in my directory in ascending order and export the data to csv

3
  • glob expansions are sorted lexically on file names by default. If you want a different sorting order, please be more explicit. Nov 24, 2016 at 16:30
  • If your problem is that for instance GM10 comes before GM2 in lexical order, you could switch to using zsh with its numerical glob sorting options: for dir in *(/n) and ..regional_col*.txt(n). Nov 24, 2016 at 16:34
  • @StéphaneChazelas Exactly , my problem is directory GM10 comes before GM2 but i dont have zsh , is there any way to solve this in bash ?
    – DevanDev
    Nov 24, 2016 at 16:42

1 Answer 1

1

If your problem is that the order is lexical instead of being numerical, you could use zsh and do:

for dir in *(n/); do
  tail -q -n 1 "$dir"/t1/regional_vol*.txt(n) | paste -s -d ',' -
done > data.csv

The n glob qualifier causes the sorting to be numerical.

If using zsh is not an option but your ls is the GNU one, an alternative is to use GNU ls's -v option for version sort:

eval "dirs=($(ls -v --quoting-style=shell-always))"
for dir in "${dirs[@]}"; do
  eval "files=($(
    ls -vd --quoting-style=shell-always -- "$dir"/t1/regional_vol*.txt))"
  tail -q -n 1 -- "${files[@]}" | paste -sd , -
done > data.csv

Yes, that's parsing the output of ls and using eval in the same command!

But here, it's safe as ls with --quoting-style=shell-always outputs in the exact format expected by eval.

To add your row and column headers:

{
  eval "dirs=($(ls -v --quoting-style=shell-always))"
  headers_done=false
  for dir in "${dirs[@]}"; do
    (
      cd -- "$dir/t1" || exit
      eval "files=($(
        ls -vd --quoting-style=shell-always regional_vol*.txt))"
      if ! "$headers_done"; then
        printf DIR
        printf ',%s' "${files[@]}"
        printf '\n'
        headers_done=true
      fi
      printf %s, "$dir"
      tail -q -n 1 -- "${files[@]}" | paste -sd , -
    )
  done
} > data.csv
6
  • Thanks it works greatly,Is there any way i can export file name of each file in directory as the name of row like this expected result , since all 30 directories have .txt files of same file names, i just need a one single row which consists of all files names in directory , i have added this line to code: paste -d, $(ls -1v "$dir"/t1/regional_vol*.txt ) > data.csv, but i dont see file names of each file in the first row
    – DevanDev
    Nov 24, 2016 at 17:01
  • @DevanDev, replace $( with <( in your command. Nov 24, 2016 at 17:21
  • I did that ,The output i am getting is incorrect output , This is the format of output that i would like to obtain Desired result, what i need is the first row of csv should be the file names arranged in sequential order like regional_vol_GM1.txt ,regional_vol_GM2.txt ..etc . and first column of csv should be directory name like Sub1,Sub2..Sub30. I have 30 directories with naming convention Sub1,Sub2,Sub30
    – DevanDev
    Nov 24, 2016 at 17:32
  • @DevanDev, see edit Nov 24, 2016 at 17:40
  • Thanks a lot works great, except minute issue, in the output file on the top row i just need regional_vol_GM.txt instead of subject1/t1/regional_vol_GM.txt
    – DevanDev
    Nov 24, 2016 at 17:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .