I have a directory with 1000s of files from a data recording system, and each file can have as many as 40,000 or more rows. The challenge is that sometimes data is not recorded from one or more sensors and thus will be missing eg
The desired result has all the files merged/joined with a single header. Where an input file is missing a column (because the sensor was broken) that portion is replaced with a null
A,B,C,D,E,F 10,20,10,20,,5 ,20,10,20,5,10 ,,,10,30,20
The last column, F, is always present because that's the date/time stamp.
I found this answer, however it assumes all headers/columns are the same across all the files
I also found this question Merging multiple CSV files for matching and non matching columns but the answer is not sufficiently complete for me to use it.