Running jobs in parallel on Ubuntu - I/O contention differences between Perl and Java

Question

Apologies if this is off topic - it concerns the relative efficiencies of running I/O-heavy Perl/Java scripts in parallel on a Ubuntu system.

I have written two simple versions of a file copy script (Perl and Java) - see below. When I run the scripts on a 15GB file, each takes a similar amount of time on a 48-core machine running Ubuntu Server 12.04 (perl 2m10s, java 2m27s).

However, when I run six instances in parallel, each operating on a different 15GB input file, I observe very different processing times:

• Perl: one instance completes in 2m6s, all others take 27m26s - 28m10s.
• Java: all instances take 3m27s - 4m37s.

Looking at the processor cores in top during the long-running Perl processes, I see that the occupied cores have I/O wait percentages (%wa) of 70%+, implying some kind of disk contention (all files are on one HD). Presumably, then, Java's BufferedReader is somehow less sensitive to this disk contention.

Question - Does this seem like a reasonable conclusion? And if so, can anyone suggest any actions I can take at the OS-level or in Perl to make the Perl script as efficient as Java for this kind of task?

Note - my goal is not simply to copy files - my real scripts contain additional logic, but exhibit the same performance behaviour as the simplified scripts below.

Perl

#!/usr/bin/perl -w
open(IN, $ARGV[0]) || die();
open(OUT, ">$ARGV[1]") || die();
while (<IN>) {
    print OUT $_
}
close(OUT);
close(IN);

Java

import java.io.*;
public class CopyFileLineByLine {
    public static void main(String[] args) throws IOException {
        BufferedReader br = null;
        PrintWriter pw = null;
        try {
            br = new BufferedReader(new FileReader(new File(args[0])));
            pw = new PrintWriter(new File(args[1]));
            String line;
            while ((line = br.readLine()) != null) {
                pw.println(line);
            }
        }
        finally {
            if (pw != null) pw.close();
            if (br != null) br.close();
        }
    }
}

Answer 1

The performance difference is most likely in how buffering works between Perl and Java. In this case, you used A bufferedReader in java which gives it an advantage. Perl does buffer around 4k from disk.

You could try a few things here. One is to use the read function in perl to get larger blocks at a time. That may improve performance.

Another option might be to investigate the various mmap related perl modules.

Answer 2

Hello This might not be the case but from first observation your perl script is running in a sequential intrepreted fashion. While your java program is running as a compiled program and is doing so in a paralleled fashion. This may account for the speed of completion difference.