I need to merge chunks from a few but large files. Each line contains the file name and byte offsets.

# file  begin   end 
foo/a   11970   12010
foo/a   22995   23035
foo/b   45090   45130
foo/b   46035   46075
foo/b   48150   48190
foo/c   16200   16240
foo/c   17550   17590
foo/c   18540   18580
foo/c   26730   26770
foo/c   34245   34285

The extraction could use tail -c and head -c, but this would reopen the same file many times, slowing down the process. The unique solution I thought of was coding a program that seeked to the beginning of each chunk and printed until the end, opening each file only once.

Do you have any suggestion?

  • Please feel free to fix grammar and semantic errors, so I can improve my English. – Pedro Lacerda Mar 31 '13 at 20:14
  • when you say chunks, do you mean chunk of lines, say lines 300-to-400 of 10,000 lines long file ? Or do you refer something else as chunk ? – MelBurslan Mar 31 '13 at 20:19
  • I mean "file foo/a from byte 11970 to 12010". Thanks, I'll update the question. – Pedro Lacerda Mar 31 '13 at 20:31
  • I don't think you're going to get better than your own idea of opening all the files ahead of time then seeking about in them. That seems pretty efficient especially if there are only three unique ones as in your example. Very quick job in any reasonable scripting language. – grifferz Mar 31 '13 at 21:05

Something like this perl should work. Replace filenames where appropriate.

#!/usr/bin/env perl

use strict;
use warnings;
use IO::Handle;

open(my $list_fh, '<', 'somefile') or die "Failed to open list file: $!";
open(my $out_fh, '>', 'outfile') or die "Failed to open out file: $!";
my $merge_fh = IO::Handle->new();
my $cur_fname = q{};
my $buff;

while ( my $line = <$list_fh> ) {
    next if $line =~ /^\s?#/;
    my ($fname, $begin, $end) = split(/\s+/, $line);
    if ( $cur_fname ne $fname ) {
        $merge_fh->close() if $merge_fh->opened();
        open($merge_fh, '<', $fname) or die "Failed to open file: $!";
        $cur_fname = $fname;

    seek($merge_fh, $begin, 0);
    read($merge_fh, $buff, $end - $begin);
    print {$out_fh} $buff or die "Failed to write to $cur_fname: $!";

  • Great! Just could fail if the chunks get too large, perhaps chop up the read/print into a loop. – vonbrand Mar 31 '13 at 22:25
  • @vonbrand I don't seen anything perl's documentation of read() or my OS's manpage for fread() that indicates there is some limitation to this. Do you mean from a lack of RAM to hold the data? – jordanm Mar 31 '13 at 22:34
  • Exactly. Might push it into swapping. – vonbrand Mar 31 '13 at 22:36
  • I'm looking for a small and elegant solution instead of writing a new program. Anyway this is a nice answer! – Pedro Lacerda Mar 31 '13 at 23:37
  • @Kanvuanza It's not a one-liner, but I would consider it a "small" script. – jordanm Mar 31 '13 at 23:38

With zsh:

zmodload zsh/mapfile
while read -r f b e; do
  [ -f $f ] && printf %s ${${mapfile[$f]}[b+1,e+1]}
done < list.txt > merged

Don't get too enthusiastic though. $mapfile does use mmap but reads the whole file in memory. (see info zsh 'The zsh/mapfile Module' for details).

With ksh93:

PATH=/opt/ast/bin:$PATH export PATH
while read -r f b e; do
  [[ -f $f ]] && head -c "$((e-b+1))" < "$f" <#(($b))
done < list.txt > merged

Setting PATH that way is so that head be the ksh93 builtin (even if there's no /opt/ast/bin directory). <#((n)) is ksh93's frontend to lseek.

PATH=/opt/ast/bin:$PATH export PATH
while read -r f b e; do
  [[ -f $f ]] && head -c "$((e-b+1))" -s "$b" < "$f"
done < list.txt > merged

ksh93's head has the -s option to skip data (uses lseek internally for regular files). It would work as long as ksh93 has been built with the head builtin included.

  • The relevant info page talks about mapping file names to file contents through an associative array, isn't there a more direct way to get a range out of the file? I can't find that Perly syntax on file ranges either (no, no zsh user). – vonbrand Mar 31 '13 at 22:50
  • Very useful module, I'll test with real data tomorrow, but seems pretty fast. When the mapped file is freed up from memory? I'm worried about filling up the RAM because 4 or 6 files can easily reach 100Mb. – Pedro Lacerda Mar 31 '13 at 23:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.