Apologies if this is off topic - it concerns the relative efficiencies of running I/O-heavy Perl/Java scripts in parallel on a Ubuntu system.

I have written two simple versions of a file copy script (Perl and Java) - see below. When I run the scripts on a 15GB file, each takes a similar amount of time on a 48-core machine running Ubuntu Server 12.04 (perl 2m10s, java 2m27s).

However, when I run six instances in parallel, each operating on a different 15GB input file, I observe very different processing times:

  • Perl: one instance completes in 2m6s, all others take 27m26s - 28m10s.
  • Java: all instances take 3m27s - 4m37s.

Looking at the processor cores in top during the long-running Perl processes, I see that the occupied cores have I/O wait percentages (%wa) of 70%+, implying some kind of disk contention (all files are on one HD). Presumably, then, Java's BufferedReader is somehow less sensitive to this disk contention.

Question - Does this seem like a reasonable conclusion? And if so, can anyone suggest any actions I can take at the OS-level or in Perl to make the Perl script as efficient as Java for this kind of task?

Note - my goal is not simply to copy files - my real scripts contain additional logic, but exhibit the same performance behaviour as the simplified scripts below.


#!/usr/bin/perl -w
open(IN, $ARGV[0]) || die();
open(OUT, ">$ARGV[1]") || die();
while (<IN>) {
    print OUT $_


import java.io.*;
public class CopyFileLineByLine {
    public static void main(String[] args) throws IOException {
        BufferedReader br = null;
        PrintWriter pw = null;
        try {
            br = new BufferedReader(new FileReader(new File(args[0])));
            pw = new PrintWriter(new File(args[1]));
            String line;
            while ((line = br.readLine()) != null) {
        finally {
            if (pw != null) pw.close();
            if (br != null) br.close();

The performance difference is most likely in how buffering works between Perl and Java. In this case, you used A bufferedReader in java which gives it an advantage. Perl does buffer around 4k from disk.

You could try a few things here. One is to use the read function in perl to get larger blocks at a time. That may improve performance.

Another option might be to investigate the various mmap related perl modules.


Not really an answer, but code does not format well in comments.

For GNU Parallel I use a version of this to copy. It can deliver in the order for 1 GB/s/core and works well in parallel:

perl -e '$left=-s STDIN;
  }' < in > out

Hello This might not be the case but from first observation your perl script is running in a sequential intrepreted fashion. While your java program is running as a compiled program and is doing so in a paralleled fashion. This may account for the speed of completion difference.

  • What tells me this is the interpreted while loop vs the complied while loop which I believe will execute in the background. – Henry McKelvey Jul 2 '12 at 19:55
  • Perl compiles to bytecode then runs natively on the processor, it's not actually interpreted. – bahamat Jul 2 '12 at 20:34
  • 2
    Bytecode doesn't run natively (it's not machine code, it's meant to be portable). – Alexios Jul 2 '12 at 20:52
  • Sequential vs parallel has nothing to do with interpreted vs compiled. – Gilles 'SO- stop being evil' Jul 2 '12 at 22:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.