I have multiple large csv files in a folder and I am trying to rbind(concatenate) them into one csv. But, while doing this I want to make sure all column values go in an appropriate column after concatenating. I can't do this in R because of memory limitation. I am pretty much new to shell scripting and I know there might be some way to do that without taking all csv files into memory.
Eg.
> csv1
A B C D E
1 2 4 5 6
4 5 7 8 9
3 5 6 7 8
2 3 4 5 8
> csv1
C B E D A
10 22 43 35 66
14 15 37 48 99
33 25 56 67 88
> Desired Output
A B C D E
1 2 4 5 6
4 5 7 8 9
3 5 6 7 8
2 3 4 5 8
66 22 10 35 43
99 15 14 48 37
88 25 33 67 56
My attempts:
I try to set column order in R for each file while saving and then use below code to concatenate. I want to know a way where I can do everything in linux shell.
My Attempt:
nawk 'FNR==1 && NR!=1{next;}{print}' *.csv > result.csv
Any help is highly appreciated.
Solution by RavindraSingh13-
awk '
BEGIN{
PROCINFO["sorted_in"] = "@ind_num_asc"
}
FNR==1{
for(i=1;i<=NF;i++){
a[$i]=i};
if(FNR==1 && FNR==NR){
print};
next
}
{
for(j in a){
printf("%s ",$a[j])}
print ""
}
' csv1 csv2
But, in above solution it is skipping some rows while concatenating.