concatenating multiple files with multiple headers

Question

Im trying to concatenate multiple files with multiple headers to have one file with all the information in it for example.

File 1:

Numbers
1
2
3

Letters
A
B
C

File 2:

Numbers
4
5
6

Letters
D
E
F

Will be merged to create File 3:

Numbers
1
2
3
4
5
6

Letters
A
B
C
D
E
F

So far I've tried cat and had file2 added to the end then attempted to grep all data under each header but long story short it failed.

I've tried paste and if the amount of data in each file is the same then its not too bad as it has each files info tabbed next to the last but will create chaos if any file has more entries than another under a header.

Any one got any ideas how to solve this sort of issue?

Answer 1

Assuming '====' under the section keywords the following Python would solve the issue without having to be explicit on the section names:

import sys
from collections import OrderedDict

combined = OrderedDict()
seperator = '===='

for file_name in 'f1.txt f2.txt'.split(): #sys.argv[1:]:
    with open(file_name) as fp:
        lines = fp.readlines()
        data = []
        while len(lines):
            # reverse over the lines
            line = lines.pop(-1)
            if not line.strip(): continue # skip empty
            if line.startswith(seperator):
                name = lines.pop(-1)
                section = combined.setdefault(name, [])
                section.extend(reversed(data))
                data = []
            else:
                data.append(line)

for idx, k in enumerate(reversed(combined)):
    if idx != 0:
        print # insert empty line before all but first
    sys.stdout.write(k)
    print('=' * len(k))
    for line in combined[k]:
        sys.stdout.write(line)

You have to give the file names on the invocation of the commandline.

This generates the output:

Numbers
========
1
2
3
4
5
6

Letters
========
A
B
C
D
E
F

Answer 2

Here's one way to do it with a Perl script:

#!/usr/bin/perl

my @fileInfo;
open(my $fh, "<file1");
push (@fileInfo,<$fh>);
close($fh);
open(my $fh, "<file2");
push (@fileInfo,<$fh>);
close($fh);

my @letLines;
my @numLines;
my $numMode  = 0;
my $letMode  = 0;

foreach my $line (@fileInfo) {

    # skip empty and '==' lines
    next if ($line =~ /^$/);
    next if ($line =~ /==/);

    if ($line =~ /Letters/) {
        $letMode = 1;
        $numMode = 0;
        next;
    }

    if ($line =~ /Numbers/) {
        $numMode = 1;
        $letMode = 0;
        next;
    }

    if ($letMode) {
        push (@letLines, $line);
        next;
    }

    if ($numMode) {
        push (@numLines, $line);
        next;
    }
}

print "Numbers\n";
print "=======\n";
print @numLines;

print "\n";

print "Letters\n";
print "=======\n";
print @letLines;

# vim: set ts=2 nolist :

output

Numbers
=======
1
2
3
4
5
6

Letters
=======
A
B
C
D
E
F

Answer 3

A numbersletters.sh for the input you gave:

#!/bin/bash

echo Numbers

for f in "$@"
do
   grep -E '^[0-9]+$' "$f" 
done

echo
echo Letters

for f in "$@"
do
    grep -E '^[A-Z]+$' "$f"
done

echo

numbersletters.sh file1 file2 returns

Numbers
1
2
3
4
5
6

Letters
A
B
C
D
E
F

You could add | sort -u for the for loops to make it sorted and unique.

Answer 4

Here's a simple Perl solution that will work for arbitrary header names:

#!/usr/bin/env perl 
my ($last,$head); 
my %k;  ## this will hold our data
my $separator="=====";
my @lines=<>; ## Read all lines
for(my $i=0; $i<=$#lines; $i++){ ## for each line...
    next if $lines[$i]=~/$separator/; ##skip the separators
    next if $lines[$i]=~/^\s*$/;  ##skip empty lines
    if ($lines[$i+1]=~/====/) { ## if the NEXT line is a separator, 
    $head=$lines[$i];       ## the this line is a heading
    next;
    }
    else { ## save the contents of this group into the %k hash.
    push @{$k{$head}},$lines[$i];
    }
}

## Now print everything
foreach (keys(%k)) {
    print "$_========\n", @{$k{$_}};
}