Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Im trying to concatenate multiple files with multiple headers to have one file with all the information in it for example.

File 1:

Numbers
1
2
3

Letters
A
B
C

File 2:

Numbers
4
5
6

Letters
D
E
F

Will be merged to create File 3:

Numbers
1
2
3
4
5
6

Letters
A
B
C
D
E
F

So far I've tried cat and had file2 added to the end then attempted to grep all data under each header but long story short it failed.

I've tried paste and if the amount of data in each file is the same then its not too bad as it has each files info tabbed next to the last but will create chaos if any file has more entries than another under a header.

Any one got any ideas how to solve this sort of issue?

share|improve this question
1  
How do you distinguish in general between what is a header (Letters, Numbers) and what is information 'under' a header? –  Anthon Apr 19 '13 at 10:55
    
What do you want to happen in case both files contain the same information, e.g. if one line under "Numbers" is "3" in both files? Do you want it listed twice or only once? –  Jenny D Apr 19 '13 at 11:04
    
in the file a header has a line beneath it with several = symbols so i thought i could use them as a delimiter. If a number/letter is duplicated ideally it will not be replicated however it isnt too much of an issue if it is. –  Dan Apr 19 '13 at 11:06
    
If you accept the edit that someone made on your post, we can probably better see what your data looks like. –  Anthon Apr 19 '13 at 11:20
    
how do i accept an edit? –  Dan Apr 19 '13 at 11:38
add comment

4 Answers

Assuming '====' under the section keywords the following Python would solve the issue without having to be explicit on the section names:

import sys
from collections import OrderedDict

combined = OrderedDict()
seperator = '===='

for file_name in 'f1.txt f2.txt'.split(): #sys.argv[1:]:
    with open(file_name) as fp:
        lines = fp.readlines()
        data = []
        while len(lines):
            # reverse over the lines
            line = lines.pop(-1)
            if not line.strip(): continue # skip empty
            if line.startswith(seperator):
                name = lines.pop(-1)
                section = combined.setdefault(name, [])
                section.extend(reversed(data))
                data = []
            else:
                data.append(line)

for idx, k in enumerate(reversed(combined)):
    if idx != 0:
        print # insert empty line before all but first
    sys.stdout.write(k)
    print('=' * len(k))
    for line in combined[k]:
        sys.stdout.write(line)

You have to give the file names on the invocation of the commandline.

This generates the output:

Numbers
========
1
2
3
4
5
6

Letters
========
A
B
C
D
E
F
share|improve this answer
add comment

Here's one way to do it with a Perl script:

#!/usr/bin/perl

my @fileInfo;
open(my $fh, "<file1");
push (@fileInfo,<$fh>);
close($fh);
open(my $fh, "<file2");
push (@fileInfo,<$fh>);
close($fh);

my @letLines;
my @numLines;
my $numMode  = 0;
my $letMode  = 0;

foreach my $line (@fileInfo) {

    # skip empty and '==' lines
    next if ($line =~ /^$/);
    next if ($line =~ /==/);

    if ($line =~ /Letters/) {
        $letMode = 1;
        $numMode = 0;
        next;
    }

    if ($line =~ /Numbers/) {
        $numMode = 1;
        $letMode = 0;
        next;
    }

    if ($letMode) {
        push (@letLines, $line);
        next;
    }

    if ($numMode) {
        push (@numLines, $line);
        next;
    }
}

print "Numbers\n";
print "=======\n";
print @numLines;

print "\n";

print "Letters\n";
print "=======\n";
print @letLines;

# vim: set ts=2 nolist :

output

Numbers
=======
1
2
3
4
5
6

Letters
=======
A
B
C
D
E
F
share|improve this answer
add comment

A numbersletters.sh for the input you gave:

#!/bin/bash

echo Numbers

for f in "$@"
do
   grep -E '^[0-9]+$' "$f" 
done

echo
echo Letters

for f in "$@"
do
    grep -E '^[A-Z]+$' "$f"
done

echo

numbersletters.sh file1 file2 returns

Numbers
1
2
3
4
5
6

Letters
A
B
C
D
E
F

You could add | sort -u for the for loops to make it sorted and unique.

share|improve this answer
add comment

Here's a simple Perl solution that will work for arbitrary header names:

#!/usr/bin/env perl 
my ($last,$head); 
my %k;  ## this will hold our data
my $separator="=====";
my @lines=<>; ## Read all lines
for(my $i=0; $i<=$#lines; $i++){ ## for each line...
    next if $lines[$i]=~/$separator/; ##skip the separators
    next if $lines[$i]=~/^\s*$/;  ##skip empty lines
    if ($lines[$i+1]=~/====/) { ## if the NEXT line is a separator, 
    $head=$lines[$i];       ## the this line is a heading
    next;
    }
    else { ## save the contents of this group into the %k hash.
    push @{$k{$head}},$lines[$i];
    }
}

## Now print everything
foreach (keys(%k)) {
    print "$_========\n", @{$k{$_}};
}
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.