2

I have two huge files and number of lines in both of the file is not same.

file1

61346877
41724134
85406965
59647779
25199749
86213
45417131
41905714
19415458
1828594
56543876
70603415

File2

212
1231
324234
213

Desired Output

61346877,212
41724134,1231
85406965,324234
59647779,213
25199749,212
86213,1231
45417131,324234
41905714,213
19415458,212
1828594,1231
56543876,324234
70603415,213
4

bash:

size1=$( wc -l < file1 )
size2=$( wc -l < file2 )
i=0
while (( i < size1 )); do
    cat file2
    (( i += size2 ))
done | paste -d, file1 -  | head -n $size1

I pipe into head in case the size of file1 is not an even multiple of file2

output

61346877,212
41724134,1231
85406965,324234
59647779,213
25199749,212
86213,1231
45417131,324234
41905714,213
19415458,212
1828594,1231
56543876,324234
70603415,213

A 3-line awk program

awk -v OFS=, '
    # read the smaller file into memory
    NR == FNR {size2++; file2[FNR] = $0; next} 

    # store the last line of the array as the zero-th element
    FNR == 1 && NR > 1 {file2[0] = file2[size2]} 

    # print the current line of file1 and the corresponding file2 line
    {print $0, file2[FNR % size2]}
' file2 file1
  • You can join two last awk actions (lines) into one: awk -v OFS=, 'NR==FNR{size2++; file2[FNR]=$0; next}{file2[0]=file2[size2]; print $0, file2[FNR%size2]}' file2 file1 – jimmij Aug 11 '15 at 19:08
  • The files are "huge", so it might be better to use awk's getline f2 <file2 to read both files in parallel. If you can get awk to start from the beginning of a file again, when it hits the end... – Peter Cordes Aug 11 '15 at 19:17
  • @jimmij, sure, but you don't really need to do that assignment for every line in file1 – glenn jackman Aug 12 '15 at 0:33
  • @PeterCordes: define "huge": is file2 measured in GB? – glenn jackman Aug 12 '15 at 0:34
  • No clue, have to ask @anurag, since he's the one that wrote that his files are "huge". But yes, it's basically fine unless it start paging out your GUI desktop. And potentially faster to sequential-read one file and then the other, rather than generating seeks back and forth. – Peter Cordes Aug 12 '15 at 0:44
2

What you have there is file2 repeating once all the lines in the file2 have been read. Here's an awk/sed solution that takes number of lines in file2 as variable, and prints lines in file2 according to counter, which is being reset each time we go over the total number of lines in the file

$ awk -v file2lines=$(wc -l file2 | cut -f1 -d' ') 'BEGIN{count=1} {if(count > file2lines) count=1; printf $1",";system("sed -n -e "count"p file2");count++}' file1

61346877,212
41724134,1231
85406965,324234
59647779,213
25199749,212
86213,1231
45417131,324234
41905714,213
19415458,212
1828594,1231
56543876,324234
70603415,213
  • You're invoking sed once for every line in file1 -- this will be really slow. – glenn jackman Aug 12 '15 at 0:35
  • @glennjackman admitedly, if the files are large, that will get slow. With what OP's given, time command reports 0.00user 0.00system 0:00.11elapsed 3%CPU (0avgtext+0avgdata 3496maxresident)k 0inputs+0outputs (0major+2480minor)pagefaults 0swaps – Sergiy Kolodyazhnyy Aug 12 '15 at 0:38
  • A little bit complicated, but it works ;) – A.B. Aug 12 '15 at 6:40
2

Simple using awk ;)

awk 'FNR==NR{a[i++]=$0; max=i; next} {if ((NR % max) == 0) {i=max-1} else {i=(NR%max) - 1}; printf "%s,%s\n",$0,a[i]}' file2 file1

Example output

61346877,212
41724134,1231
85406965,324234
59647779,213
25199749,212
86213,1231
45417131,324234
41905714,213
19415458,212
1828594,1231
56543876,324234
70603415,213

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.