0

I have 200+ large files that all have exactly 1 column and 76M rows. I want to start a newfile.txt and slap the columns next to each other (match Line 1 of File 1 with Line 1 of File 2... and continue adding until Line 1 of 200). Then repeat this for all rows. I am struggling with this. Any suggestions?

I have tried Gilles and Glens answers here and here but I can't figure out how to loop and repeatedly add tab-delimited columns to the output newfile.txt. I can only use methods that do not store the files in memory (the final file should be 120GB+).

Thanks

  • 3
    try paste file_prefix* > newfile.txt and let me know ... – RomanPerekhrest Feb 6 '18 at 20:14
  • Hey thanks for the response, but paste file_prefix* > newfile.txt doesn't work. I tested with 3 files and the output is 2 columns (should be 3) and the data structure is screwed up. It gives File1_Row1, Blank \n Blank, File2_Row1 \n Blank, File3_Row1... – Ehi Akhirome Feb 6 '18 at 23:52
  • Are all your files proper Unix text files, or may some or all of them be either DOS or old Mac-formatted text files? Roman's suggestion should work on Unix text files. – Kusalananda Feb 7 '18 at 6:56
  • Roman's suggestion should work on Unix text files, provided that the generated command line is not too long. – Kusalananda Feb 7 '18 at 8:15
0

With some patience, and assuming exactly 200 files named "file{1..200}" and 76,000,000 lines in each:

for ((line=1; line <= 76000000; line++))
do 
  printf "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n" "$(sed -n ${line}p "file1")" "$(sed -n ${line}p "file2")" "$(sed -n ${line}p "file3")" "$(sed -n ${line}p "file4")" "$(sed -n ${line}p "file5")" "$(sed -n ${line}p "file6")" "$(sed -n ${line}p "file7")" "$(sed -n ${line}p "file8")" "$(sed -n ${line}p "file9")" "$(sed -n ${line}p "file10")" "$(sed -n ${line}p "file11")" "$(sed -n ${line}p "file12")" "$(sed -n ${line}p "file13")" "$(sed -n ${line}p "file14")" "$(sed -n ${line}p "file15")" "$(sed -n ${line}p "file16")" "$(sed -n ${line}p "file17")" "$(sed -n ${line}p "file18")" "$(sed -n ${line}p "file19")" "$(sed -n ${line}p "file20")" "$(sed -n ${line}p "file21")" "$(sed -n ${line}p "file22")" "$(sed -n ${line}p "file23")" "$(sed -n ${line}p "file24")" "$(sed -n ${line}p "file25")" "$(sed -n ${line}p "file26")" "$(sed -n ${line}p "file27")" "$(sed -n ${line}p "file28")" "$(sed -n ${line}p "file29")" "$(sed -n ${line}p "file30")" "$(sed -n ${line}p "file31")" "$(sed -n ${line}p "file32")" "$(sed -n ${line}p "file33")" "$(sed -n ${line}p "file34")" "$(sed -n ${line}p "file35")" "$(sed -n ${line}p "file36")" "$(sed -n ${line}p "file37")" "$(sed -n ${line}p "file38")" "$(sed -n ${line}p "file39")" "$(sed -n ${line}p "file40")" "$(sed -n ${line}p "file41")" "$(sed -n ${line}p "file42")" "$(sed -n ${line}p "file43")" "$(sed -n ${line}p "file44")" "$(sed -n ${line}p "file45")" "$(sed -n ${line}p "file46")" "$(sed -n ${line}p "file47")" "$(sed -n ${line}p "file48")" "$(sed -n ${line}p "file49")" "$(sed -n ${line}p "file50")" "$(sed -n ${line}p "file51")" "$(sed -n ${line}p "file52")" "$(sed -n ${line}p "file53")" "$(sed -n ${line}p "file54")" "$(sed -n ${line}p "file55")" "$(sed -n ${line}p "file56")" "$(sed -n ${line}p "file57")" "$(sed -n ${line}p "file58")" "$(sed -n ${line}p "file59")" "$(sed -n ${line}p "file60")" "$(sed -n ${line}p "file61")" "$(sed -n ${line}p "file62")" "$(sed -n ${line}p "file63")" "$(sed -n ${line}p "file64")" "$(sed -n ${line}p "file65")" "$(sed -n ${line}p "file66")" "$(sed -n ${line}p "file67")" "$(sed -n ${line}p "file68")" "$(sed -n ${line}p "file69")" "$(sed -n ${line}p "file70")" "$(sed -n ${line}p "file71")" "$(sed -n ${line}p "file72")" "$(sed -n ${line}p "file73")" "$(sed -n ${line}p "file74")" "$(sed -n ${line}p "file75")" "$(sed -n ${line}p "file76")" "$(sed -n ${line}p "file77")" "$(sed -n ${line}p "file78")" "$(sed -n ${line}p "file79")" "$(sed -n ${line}p "file80")" "$(sed -n ${line}p "file81")" "$(sed -n ${line}p "file82")" "$(sed -n ${line}p "file83")" "$(sed -n ${line}p "file84")" "$(sed -n ${line}p "file85")" "$(sed -n ${line}p "file86")" "$(sed -n ${line}p "file87")" "$(sed -n ${line}p "file88")" "$(sed -n ${line}p "file89")" "$(sed -n ${line}p "file90")" "$(sed -n ${line}p "file91")" "$(sed -n ${line}p "file92")" "$(sed -n ${line}p "file93")" "$(sed -n ${line}p "file94")" "$(sed -n ${line}p "file95")" "$(sed -n ${line}p "file96")" "$(sed -n ${line}p "file97")" "$(sed -n ${line}p "file98")" "$(sed -n ${line}p "file99")" "$(sed -n ${line}p "file100")" "$(sed -n ${line}p "file101")" "$(sed -n ${line}p "file102")" "$(sed -n ${line}p "file103")" "$(sed -n ${line}p "file104")" "$(sed -n ${line}p "file105")" "$(sed -n ${line}p "file106")" "$(sed -n ${line}p "file107")" "$(sed -n ${line}p "file108")" "$(sed -n ${line}p "file109")" "$(sed -n ${line}p "file110")" "$(sed -n ${line}p "file111")" "$(sed -n ${line}p "file112")" "$(sed -n ${line}p "file113")" "$(sed -n ${line}p "file114")" "$(sed -n ${line}p "file115")" "$(sed -n ${line}p "file116")" "$(sed -n ${line}p "file117")" "$(sed -n ${line}p "file118")" "$(sed -n ${line}p "file119")" "$(sed -n ${line}p "file120")" "$(sed -n ${line}p "file121")" "$(sed -n ${line}p "file122")" "$(sed -n ${line}p "file123")" "$(sed -n ${line}p "file124")" "$(sed -n ${line}p "file125")" "$(sed -n ${line}p "file126")" "$(sed -n ${line}p "file127")" "$(sed -n ${line}p "file128")" "$(sed -n ${line}p "file129")" "$(sed -n ${line}p "file130")" "$(sed -n ${line}p "file131")" "$(sed -n ${line}p "file132")" "$(sed -n ${line}p "file133")" "$(sed -n ${line}p "file134")" "$(sed -n ${line}p "file135")" "$(sed -n ${line}p "file136")" "$(sed -n ${line}p "file137")" "$(sed -n ${line}p "file138")" "$(sed -n ${line}p "file139")" "$(sed -n ${line}p "file140")" "$(sed -n ${line}p "file141")" "$(sed -n ${line}p "file142")" "$(sed -n ${line}p "file143")" "$(sed -n ${line}p "file144")" "$(sed -n ${line}p "file145")" "$(sed -n ${line}p "file146")" "$(sed -n ${line}p "file147")" "$(sed -n ${line}p "file148")" "$(sed -n ${line}p "file149")" "$(sed -n ${line}p "file150")" "$(sed -n ${line}p "file151")" "$(sed -n ${line}p "file152")" "$(sed -n ${line}p "file153")" "$(sed -n ${line}p "file154")" "$(sed -n ${line}p "file155")" "$(sed -n ${line}p "file156")" "$(sed -n ${line}p "file157")" "$(sed -n ${line}p "file158")" "$(sed -n ${line}p "file159")" "$(sed -n ${line}p "file160")" "$(sed -n ${line}p "file161")" "$(sed -n ${line}p "file162")" "$(sed -n ${line}p "file163")" "$(sed -n ${line}p "file164")" "$(sed -n ${line}p "file165")" "$(sed -n ${line}p "file166")" "$(sed -n ${line}p "file167")" "$(sed -n ${line}p "file168")" "$(sed -n ${line}p "file169")" "$(sed -n ${line}p "file170")" "$(sed -n ${line}p "file171")" "$(sed -n ${line}p "file172")" "$(sed -n ${line}p "file173")" "$(sed -n ${line}p "file174")" "$(sed -n ${line}p "file175")" "$(sed -n ${line}p "file176")" "$(sed -n ${line}p "file177")" "$(sed -n ${line}p "file178")" "$(sed -n ${line}p "file179")" "$(sed -n ${line}p "file180")" "$(sed -n ${line}p "file181")" "$(sed -n ${line}p "file182")" "$(sed -n ${line}p "file183")" "$(sed -n ${line}p "file184")" "$(sed -n ${line}p "file185")" "$(sed -n ${line}p "file186")" "$(sed -n ${line}p "file187")" "$(sed -n ${line}p "file188")" "$(sed -n ${line}p "file189")" "$(sed -n ${line}p "file190")" "$(sed -n ${line}p "file191")" "$(sed -n ${line}p "file192")" "$(sed -n ${line}p "file193")" "$(sed -n ${line}p "file194")" "$(sed -n ${line}p "file195")" "$(sed -n ${line}p "file196")" "$(sed -n ${line}p "file197")" "$(sed -n ${line}p "file198")" "$(sed -n ${line}p "file199")" "$(sed -n ${line}p "file200")"
done > newfile.txt

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.