1

Im trying to concatenate multiple files with multiple headers to have one file with all the information in it for example.

File 1:

Numbers
1
2
3

Letters
A
B
C

File 2:

Numbers
4
5
6

Letters
D
E
F

Will be merged to create File 3:

Numbers
1
2
3
4
5
6

Letters
A
B
C
D
E
F

So far I've tried cat and had file2 added to the end then attempted to grep all data under each header but long story short it failed.

I've tried paste and if the amount of data in each file is the same then its not too bad as it has each files info tabbed next to the last but will create chaos if any file has more entries than another under a header.

Any one got any ideas how to solve this sort of issue?

5
  • 2
    How do you distinguish in general between what is a header (Letters, Numbers) and what is information 'under' a header?
    – Anthon
    Apr 19, 2013 at 10:55
  • What do you want to happen in case both files contain the same information, e.g. if one line under "Numbers" is "3" in both files? Do you want it listed twice or only once?
    – Jenny D
    Apr 19, 2013 at 11:04
  • in the file a header has a line beneath it with several = symbols so i thought i could use them as a delimiter. If a number/letter is duplicated ideally it will not be replicated however it isnt too much of an issue if it is.
    – Dan
    Apr 19, 2013 at 11:06
  • If you accept the edit that someone made on your post, we can probably better see what your data looks like.
    – Anthon
    Apr 19, 2013 at 11:20
  • how do i accept an edit?
    – Dan
    Apr 19, 2013 at 11:38

4 Answers 4

1

Assuming '====' under the section keywords the following Python would solve the issue without having to be explicit on the section names:

import sys
from collections import OrderedDict

combined = OrderedDict()
seperator = '===='

for file_name in 'f1.txt f2.txt'.split(): #sys.argv[1:]:
    with open(file_name) as fp:
        lines = fp.readlines()
        data = []
        while len(lines):
            # reverse over the lines
            line = lines.pop(-1)
            if not line.strip(): continue # skip empty
            if line.startswith(seperator):
                name = lines.pop(-1)
                section = combined.setdefault(name, [])
                section.extend(reversed(data))
                data = []
            else:
                data.append(line)

for idx, k in enumerate(reversed(combined)):
    if idx != 0:
        print # insert empty line before all but first
    sys.stdout.write(k)
    print('=' * len(k))
    for line in combined[k]:
        sys.stdout.write(line)

You have to give the file names on the invocation of the commandline.

This generates the output:

Numbers
========
1
2
3
4
5
6

Letters
========
A
B
C
D
E
F
1

Here's one way to do it with a Perl script:

#!/usr/bin/perl

my @fileInfo;
open(my $fh, "<file1");
push (@fileInfo,<$fh>);
close($fh);
open(my $fh, "<file2");
push (@fileInfo,<$fh>);
close($fh);

my @letLines;
my @numLines;
my $numMode  = 0;
my $letMode  = 0;

foreach my $line (@fileInfo) {

    # skip empty and '==' lines
    next if ($line =~ /^$/);
    next if ($line =~ /==/);

    if ($line =~ /Letters/) {
        $letMode = 1;
        $numMode = 0;
        next;
    }

    if ($line =~ /Numbers/) {
        $numMode = 1;
        $letMode = 0;
        next;
    }

    if ($letMode) {
        push (@letLines, $line);
        next;
    }

    if ($numMode) {
        push (@numLines, $line);
        next;
    }
}

print "Numbers\n";
print "=======\n";
print @numLines;

print "\n";

print "Letters\n";
print "=======\n";
print @letLines;

# vim: set ts=2 nolist :

output

Numbers
=======
1
2
3
4
5
6

Letters
=======
A
B
C
D
E
F
1

A numbersletters.sh for the input you gave:

#!/bin/bash

echo Numbers

for f in "$@"
do
   grep -E '^[0-9]+$' "$f" 
done

echo
echo Letters

for f in "$@"
do
    grep -E '^[A-Z]+$' "$f"
done

echo

numbersletters.sh file1 file2 returns

Numbers
1
2
3
4
5
6

Letters
A
B
C
D
E
F

You could add | sort -u for the for loops to make it sorted and unique.

1

Here's a simple Perl solution that will work for arbitrary header names:

#!/usr/bin/env perl 
my ($last,$head); 
my %k;  ## this will hold our data
my $separator="=====";
my @lines=<>; ## Read all lines
for(my $i=0; $i<=$#lines; $i++){ ## for each line...
    next if $lines[$i]=~/$separator/; ##skip the separators
    next if $lines[$i]=~/^\s*$/;  ##skip empty lines
    if ($lines[$i+1]=~/====/) { ## if the NEXT line is a separator, 
    $head=$lines[$i];       ## the this line is a heading
    next;
    }
    else { ## save the contents of this group into the %k hash.
    push @{$k{$head}},$lines[$i];
    }
}

## Now print everything
foreach (keys(%k)) {
    print "$_========\n", @{$k{$_}};
}

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .