5

I have a huge amount of files having the following naming style:

WBM_MIROC_rcp8p5_mississippi.txt
WBM_GFDL_rcp8p5_nosoc_mississippi.txt
DBH_HADGEM_rcp4p5_co2_mississippi.txt
HMH_IPSL_rcp4p5_mississippi.txt

Those files represent tables with (some of them have a tab delimiter and other one space delimiter) as follow:

YEAR MONTH DAY RES
1971 1 1 1988
1971 1 2 3829
...

I would like to group all the files having rcp8p5 in their name in one big table; and do the same for the files having rcp4p5 in their name. But, I just want to paste the 4 column of each files in order to avoid the redundancy of the first three column that are always the same. I am currently using the following script:

ls |
awk -F_ '{ i=$1; m=$2; s=$3; u=$4;
          if(f[s]=="")add = $0;
          else add = sprintf("<(cut -f4 %s)",$0);
          f[s] = f[s] " " add }
          END{ for(insc in f)
                 printf "paste%s > out_%s.txt\n",f[insc],insc
          }' |bash

It´s unclear why but the output is not as expected. I have the following output:

YEAR MONTH DAY RES YEAR MONTH DAY RES YEAR MONTH DAY RES
1971 1     1   187 1971 1     1   143 1971 1     1   234
1971 1     2   321 1971 1     2   398 1971 1     1   754
...

Instead I would like to have the following output:

YEAR MONTH DAY RES RES RES
1971 1     1   187 143 234
1971 1     2   321 398 754

It could be great if anyone is able to give me an hint!

2

The most likely answer is that your data file columns are not separated by tabs, but by space, for example. You can verify this by running one of them through cat -vet which shows real tabs as ^I.

To change your cut command to use space as a delimiter you need to add the arg -d' ', but since you are already inside single quotes and an awk script you need to change your sprintf(...) to

sprintf("<(cut -d\" \"  -f4 %s)",$0)
  • @ meuh - Indeed you are definitely right. Some of them are separated by tabs (actually few of them) and most of them are separated by space $ when running cat -vet. Shall I first convert all files with a space delimiter? – steve Oct 12 '15 at 18:19
  • perhaps the simplest is to replace the <(cut...) by <(awk '\''{print $4}'\'' %s) which works for spaces or tabs. – meuh Oct 12 '15 at 18:27
2

For not too big files:

while read -r f_part
do
    awk '
        BEGIN{
            SUBSEP=" "
        }
        NR==1{
            for(i=2;i<ARGC;i++)
                $(NF+1)=$NF
            print
        }
        FNR==1{
             next
        }
        {
             RES[$1,$2,$3]=RES[$1,$2,$3] $4 " "
        }
        END{
             for(i in RES)
                 print i, RES[i]
        }' *_${f_part}_* > big_table_${f_part}
done < <(printf '%s\n' *_*_*_*txt | cut -d_ -f3 | sort -u)

Or if you sure in proper order in files:

while read -r f_part
do 
    set -- *_${f_part}_*
    sed -i 's/\s+/:/3;s/\s\+/\t/g;s/\s*$//' "$@"
    while [ $# -gt 1 ]
    do
        join -t: $1 $2 > tmp
        mv tmp big_table_${f_part}
        shift 2
        set -- big_table_${f_part} "$@"
    done
    sed 's/:/\t/g' big_table_${f_part}
done < <(printf '%s\n' *_*_*_*txt | cut -d_ -f3 | sort -u)
2
for f in rcp8p5 rcp4p5
do  : >"$f.txt"
    find . ! -name . -prune ! -type d -name "*_${f}_*txt" -exec \
        sh -c '
            printf "%s\t" YEAR MONTH DAY
            printf "%.0sRES\t" "$@"; echo
            sed -n "
                /^[0-9]/!d;p;:n
                n
                /^[0-9]/s/.*[[:blank:]]//p
                bn
            "  "$@" | paste
    ' --    {} + >>"$f.txt"
done

... I think I might have got the wrong idea about what you were doing before, and so maybe this can make up for it. I'm not so certain this will work, but, if it does, it should be way faster than what you are doing.

Basically find grabs a list of files with names matching either of ...8... or ...4... and hands them off {} + to a shell.

The shell prints a header row beginning with YEAR MONTH DAY each followed by a \tab and after prints as many RES columns as it has arguments.

Afterward sed concatenates all of the file arguments into a single stream, and prints the first line beginning with a digit in full, and all of those lines beginning with a digit that come after only get the last field printed.

All of sed's output is passed to paste, which replaces all \newlines in its input with \tabs on output.

Hopefully, this version will work because invoking a new subshell and opening a new pipe for each file in your list is pretty much a terrible idea.

If it does, this will, however, do a new line in your table for each group of ARGMAX files - which might not be a bad thing, but is easily handled afterward.

  • @ mikeserv - This look promising but the output is not the expected one. It produces just one column of data – steve Oct 12 '15 at 19:11
  • @steve - one column? or one row? ill confess i didnt dissect your stuff as duly as maybe i should have. it should be close at least, though... i think i have an idea though... – mikeserv Oct 12 '15 at 19:14
  • @ mikeserv - ;) no problem, I already have three answer to understand! It actually produce one column. – steve Oct 12 '15 at 19:16
  • @steve - im sorry. i think i got the wrong idea - that wouldnt have ever worked. the edit might, but is way more complicated. if it does, though, you should use it. – mikeserv Oct 12 '15 at 19:34
1

You could also do something like

arr=( *_rcp8p5_*.txt )
paste "${arr[@]}" | cut -f-4,$(seq -s, 8 4 $((4*${#arr[@]}))) >out_rcp8p5.txt

This will paste all *_rcp8p5_*.txt files then extract fields 1-4 and every fourth field after.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.