0

I have a data like this:

input.txt

1 0000100101000000
1 0000010100000000
2 1110000001000000
2 1111000000001000
3 0000000111111111
3 1111111100000000
4 8888345500000000
4 0000000000000000

and I want to sum up the values within eachtwo rows with the same row number: output:

output.txt

1 0000110201000000
2 2221000001001000
3 1111111211111111
4 8888345500000000

any suggestion please? my real dat had 8000 rows with 45000 digit in each line

  • Are you against using python for your solution? – Gravy Oct 5 '15 at 21:56
  • @Gravy no I am not against that. but I never used that. if you have a solution in python, I may need the solution in details... :) – zara Oct 5 '15 at 21:57
  • Let me see if I can whip something up here... Have a rough idea, but gotta work out the logic. – Gravy Oct 5 '15 at 21:58
2
sed '
    N                                                       #append next line
    s/$/))/                                                 #add `))` to end
    s/\(\S*\s*\)\(.*\)\n\1/printf "%016d\n" \$((10#\2+10#/  #check Nos, form line
    t                                                       #to end if Nos equal
    s/))$//                                                 #remove `))`
    D                                                       #delete 1st line
    ' file |
bash

Regarding 45000 digits number please note that maximum number which bash can handle is

/* Minimum and maximum values a `signed long int' can hold.  */
#  if __WORDSIZE == 64
#   define LONG_MAX 9223372036854775807L
#  else
#   define LONG_MAX 2147483647L
#  endif

[ 1 ] /usr/include/limits.h

2

How about an all awk solution:

awk 'BEGIN { tag = -1; sum = 0}
    {
        if (tag != $1) {
            if (tag > -1) {printf "%d %016d\n",  tag, sum;}
            tag = $1; sum = $2
        } else { sum += $2 }
    }
    END {print tag, sum}'  input.txt

It isn't clear if your input is sorted by the first column or not. You might have to do something like: sort -k1.1n input.txt and then pipe that into the awk script above.

  • My data file is sorted based on row numbers. how ever this command does not work on my real data. I give and empty output in which row names do not repeated any more but in front of each row is written : inf – zara Oct 5 '15 at 22:37
  • do you have any idea why it does not work on my real data? my real data has 8000 rows and 45000 digits in each row – zara Oct 5 '15 at 22:39
  • @zara - my guess is that 45,000 digits overflows awk's arithmetic, but I don't know for certain. I even assume 16 digits in the "printf" output so it preserves leading zeros. It would be worth noting in your question what the exact input is, as I assumed your example was representative. – Bruce Ediger Oct 5 '15 at 22:44
  • I mentioned at the end of my post that my real data is big. do not you have any other suggestion? – zara Oct 5 '15 at 22:51
  • @zara - if it were me, I'd write a small C program that could take a series of 45,000 digit strings and add them numerically. Then, I'd put that code in the middle of a while-loop that implemented what my awk code above did, which is to keep track of the "tag" number in the first column, and print out an answer when the tag changes, and at the end of file. – Bruce Ediger Oct 5 '15 at 22:57
0

Ruby has bignum support, so you could do

ruby -e '
    sum = Hash.new {|h,k| h[k] = 0} 
    f = File.new(ARGV.shift)
    key, val = f.readline.chomp.split
    width = val.length
    sum[key] = val.to_i
    f.each_line {|line| key,val = line.chomp.split; sum[key] += val.to_i}
    sum.keys.sort.each {|key| printf "%d %0*d\n", key, width, sum[key]}
' file
0

If we can assume that there are always 2 lines to be added (never 3 or 1 or whatever) and that the numbers are always in the second column, separated by a space, then there's an easy solution:

cut -f2 -d' ' input.txt | perl -Mbigint -nle 'print $_ + <>' > output.txt

The cut command just picks the second column of data, and throws the first away. The perl command loops over the incoming lines (using the -n switch) and prints the sum of the current line and the next line (so it works in groups of two). Note the use of the bigint module to treat the long strings as very large numbers. Finally, the output is redirected to output.txt.

If you need the lines numbered in the output, you might consider adding cat -n as a last step in the pipeline, or add it right in the Perl code:

cut -f2 -d' ' input.txt | perl -Mbigint -nle 'print ++$x . " " . ($_ + <>)' > output.txt

Or if you can't assume the input is formatted with spaces as in your example, you can move that processing to the Perl too:

perl -Mbigint -nle 's/.* //; $x=<>; $x =~ s/.* //; print $_ + $x' input.txt > output.txt

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.