I have to generate the file dynamically from the source file based on the below control file.
The control file contains the position of columns which I require from the source file.
Example: 1 column, 3 column, 5 column, blank column (-1 indicates blank column) and so on..
I have written a shell to read the position from the control file, one by one, to generate multiple files. Then finally use the paste command to generate new files, incase of -1 values I created touch files. Based on sequence of file order, it is pasted by
So my existing shell looks like below:
if [ position != -1 ] then cut -d, -f$position > file_$var.csv else touch file_$var.csv fi paste -d, $(ls -v file_*.csv) > newe_file.csv
I hope there is a way to minimize the I/O of files, I am looking for something below:
cut -d, -f1,3,5 > file1.csv touch file2.csv cut -d, -f8 > file3.csv touch file4.csv cut -d, -f4 > file5.csv
Or better solution will be great.
The number of columns in the source files will be in the hundreds.
input-file is sample.csv col1,col2,col3,col4,col5,col6,col7,col8 1,2,3,4,5,6,7,8 9,10,11,12,13,14,15,16
col1,col3,col5,-1,col8,-1,col4 1,3,5,,8,,4 9,11,13,,16,,12
output.csv is based on