I asked the same question on stackoverflow, but did not get a satisfactory answer.
I have a directory tree like this:
bacteria > species(num subdirs = 1300) > kmer(num subsudirs = 9)
The directory and subdirectories are agruped this way:
bacteria/specie1/kmer2/csv
kmer3/csv
.
.
kmer9/csv
specie2/kmer2/csv
kmer3/csv
.
.
kmer9/csv
specie1300/kmer2/csv
kmer3/csv
.
.
kmer9/csv
In the kmer subdirectory there are the csv files and depend on the species it has 1 or many files.
The csv files are simple, two coulmns with the kmer and it counts. I really dont know if all bigger kmers (like 6 or upper) will have the same kmer keys, because this I am counting them.
For example:
bacteria/species1/kmer2/example_csv_k2_count.csv
The csv example_csv_k2_count.csv has this structure:
kmer,count
AA, 10
AC, 20
.
.
.
For each species and kmer counts I need to concatenate the csv files in one master csv for each kmer count (2-9). For example for subdiretories of all species with kmer2 counts I need
specie1.csv1
kmer, count
aa, 22
at, 21...
specie1.csv2
kmer, count
aa, 31
at, 18...
For bacteria/species2/kmer2:
specie2.csv1
kmer, count
aa, 22
at, 21...
specie2.csv2
kmer, count
aa, 31
at, 18...
I need a make a master kmer2_count.csv for each species and they kmer counts:
kmer, count
aa, 22, 31,...
at, 21, 18,...
I tried many codes and commands that I search in stack, google, but I always get they stacked:
kmer, count
aa, 31
at, 18...
kmer, count
aa, 22
at, 21...
Example of shell commands I tried:
$ cat */kmer2/*.csv | datamash -t , transpose
datamash: transpose input error: line 275 has 1 fields (previous lines had 2);
Ex2:
awk '
FNR==1 && NR!=1 { while (/^<header>/) getline; }
1 {print}
' */kmer9/*.csv > ex.csv
Now I am using python/pandas and got the job working but for kmer counts bigger than 6, ex 7 my notebook is working for 3 days in a roll. I am trying to merge this to make a mean of counts because many species had they genome sequenced many times or because they are subspecies or strains. Than I will make a master file with the mean of all sequenced genomes.
Because this I would love a shell solution.
Thanks, Paulo
PS = Sorry about the bad text explaining my needs. Hope now it get more clear. My apologies
kmer,count
, you should have exactly 2 columns, not one per input file.