0

I can't seem to understand why a script I've been running fine up until now fails on a slightly larger input file.

If I run the script on a file that is 820000x75 (lines by columns), the script works just fine, but if I run it with a file that is 1.4million x 75, the script fails to print anything, with the only error message being "Killed". Am I running into some size limit of awk with my input.

Here is the script, which should just read through a file of floating point numbers and print them out arranged differently in another file:

BEGIN{  i=1; j=1

    getline
    getline
 }{ 
           if($3 ~ /MO/ ){
                  i=1
                  if(0=="1"){print i,j}
                  j++
                }
 else           {
                  # substr(string, start [, length ])

                  if(length($0)>"61"){

                     Mat[i,j]=sprintf("%.8g",substr($0,1,15)) ; i++
                     Mat[i,j]=sprintf("%.8g",substr($0,16,15)); i++
                     Mat[i,j]=sprintf("%.8g",substr($0,31,15)); i++
                     Mat[i,j]=sprintf("%.8g",substr($0,46,15)); i++
                     Mat[i,j]=sprintf("%.8g",substr($0,61,15)); i++

                  }
                  else if(length($0)>"46"){

                     Mat[i,j]=sprintf("%.8g",substr($0,1,15)) ; i++
                     Mat[i,j]=sprintf("%.8g",substr($0,16,15)); i++
                     Mat[i,j]=sprintf("%.8g",substr($0,31,15)); i++
                     Mat[i,j]=sprintf("%.8g",substr($0,46,15)); i++

                  }
                  else if(length($0)>"31"){

                     Mat[i,j]=sprintf("%.8g",substr($0,1,15)) ; i++
                     Mat[i,j]=sprintf("%.8g",substr($0,16,15)); i++
                     Mat[i,j]=sprintf("%.8g",substr($0,31,15)); i++

                  }
                  else if(length($0)>"16"){

                     Mat[i,j]=sprintf("%.8g",substr($0,1,15)) ; i++
                     Mat[i,j]=sprintf("%.8g",substr($0,16,15)); i++

                  }
                  else if(length($0)>"1"){

                     if($0!="0"){  Mat[i,j]=sprintf("%.8g",substr($0,1,15)) ; i++  }
                     else{print "scratch.scr.readMO: a zero element was found!"}

                  }
                  else if(length($0)=="0"){ print "scratch.scr.readMO: error-0"}
                  else{ print "scratch.scr.readMO: error-2"}
                }
}
END{
                if(0=="1"){print (i-1),j}
                for (x=1; x<=(i-1); x++)
                {   for (y=1; y<=j; y++)
                    {         printf "%.8g %s ", Mat[x,y], " "    > "MOs_"label".txt"
                    }
                    if (x<(i-1)) {printf "\n "                    > "MOs_"label".txt" }
                }
}

And here is the command I'm using to run this:

gawk -v label=P -f script file_to_process
  • 2
    With a file that large, you might indeed be running into limits. Try splitting the file into smaller pieces? – DopeGhoti Aug 30 '18 at 22:18
  • @DopeGhoti I should be able to do that as a workaround, though I'm curious to learn what specific size limit I'm running into. Even counting the characters in the file doesn't seem to get me anywhere close to the upper limits of int storage. – Tyberius Aug 30 '18 at 22:36
1

The message Killed usually indicates that the kernel has killed your process, due most likely to either resource exhaustion (running out of memory) or quota exhaustion (running out of permitted memory use). You can see messages from the kernel by perusing your system logs (typically /var/log/messages (or, if you're there when you see the process die, run dmesg) to usually get some insight as to why the kernel decided your process had to die.

  • This seems to have been my issue. I assumed it was something gawk specific, but it seems to have just been a memory issue. – Tyberius Sep 9 '18 at 21:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.