Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I need to extract strings of text from a single file containing one very long line of text with no delimiters. Using the sample line below, these are the following known facts:

???????A1XXXXXXXXXX???????B1XXXX???????A1XXXXXXXXXX???????C1XXXXXXX

1.  It contains 38 fixed width record types 
2.  The record marker is a 7 alphanumeric character followed by, for example, ‘A1’.
3.  Each record type has varying widths, for example, A1 record type will have 10 characters following it, if B1 then 4, and if C1 then 7.
4.  The record types aren’t clumped together and can be in any order. As in the example, its A1,B1,A1,C1
5.  The example above has 4 records and each record type needs to go to separate files. In this case 38 of them.

???????A1XXXXXXXXXX

???????B1XXXX

???????A1XXXXXXXXXX

???????C1XXXXXXX

6.  The record identifier, e.g. ????????A1, can appear in the body of the record so cannot use grep. 
7.  With the last point in mind, I was proposing 3 solutions but not sure on how to script this and of course would greatly appreciate some help. 
a. Traverse through the file from the beginning and sequentially strip out the record to the appropriate output file. For example, strip out first record type A1 to A1file which I know is 10 characters long then re-interrogate the file which will then have B1 which I know is 4 chars long, strip this out to B1file etc.. <<< this seems painful >>
b. Traverse through the file and append some obscure character to each record marker within the same file. Much like above but not strip out. I understand it still will use the same logic but seems more elegant
c. I did think of simply using the proposed grep -oE solution but then re-interrogate the output files to see if any of the 38 record markers exist anywhere other than at the beginning. But this might not always work.
share|improve this question
    
Perl code refactored to take your updates into account. Please see if it helps. –  Joseph R. Dec 14 '13 at 21:50
    
Thank you Joseph. I don't know Perl but wanted to make it clear that the file contains only 1 line of text, i.e. no carriage returns or line breaks. Just wanted to make that clear because i see in your comments you imply the file has more than 1 lines unless like i said i've misread this. Many thanks. –  jags Dec 14 '13 at 22:43
    
This should not make a difference. The Perl code will work the same if it's all on one line or if there are several, as long as each line contains an integer number of well-formed records. –  Joseph R. Dec 14 '13 at 22:53
    
Thank you so much Joseph. It's worked. Tested with if a record marker is in body of record and this back referencing overcomes that. Can anyone offer a Unix equivalent please? –  jags Dec 15 '13 at 14:35
    
Please look at my updated answer. –  Joseph R. Dec 15 '13 at 17:34

3 Answers 3

How about

grep -oE 'A1.{10}|B1.{4}|C1.{7}' input.txt

This prints each record of each record type on a separate line. To redirect grep output to 3 files named A1, B1, C1 respectively,

grep -oE 'A1.{10}|B1.{4}|C1.{7}' input.txt| 
awk -v OFS= -v FS= '{f=$1$2; $1=$2=""; print>f}'
share|improve this answer
    
Thank you so much for this. Do you mind explaining these various script components and switches used so that I can test and extend please. Also how do I add the pattern of 9s before it (which in reality will be alphanumeric characters of 7 characters long). Many thanks. –  jags Dec 14 '13 at 9:14
    
Spoke too soon...I should have also added 1 vital information which was that the pattern.recordmarker might appear in the rest of the record so its been advised we strip out a record at a time to a file and reinterrogate the file which probably means I cannot use grep. –  jags Dec 14 '13 at 9:51
    
Furthermore, I have 2 possible solutions. - traverse through the file, label with an obscure character to denote start of valid record. Move X chars depending on record type and use same obscure character to denote next record. However wary of any buffer issues. Therefore expecting new output to interrogate looking like this "?\\9999999A1XXXXXXXXXX?\\9999999B1XXXX?\\9999999A1XXXXXXXXXX?\\9999999C1XXXXXXX" - use current sol but then search within each output file if the other patterns appear other than at the beginning –  jags Dec 14 '13 at 15:35
    
@jags, you may want to update your original question with truly representative sample data, it's all getting a little bit confusing –  1_CR Dec 14 '13 at 18:00
    
Thank you 1_CR, I've resubmitted the question. Thank you all for your help. Most appreciated. –  jags Dec 14 '13 at 21:34

In Perl:

#!/usr/bin/env perl

use strict;
use warnings;
use re qw(eval);

my %field_widths = (
    A1 => 10,
    B1 =>  4,
    C1 =>  7,
    #...(fill this up with the widths of your 38 record types)
);

# Make a regex of record types; sort with longest first as appropriate for
# ... regex alternation:
my $record_type_regex = join '|', sort { length($b) <=> length($a) } keys %field_widths; 

my %records;
my $marker_length=7; #Assuming the marker is 7 characters long
while(<>){
    chomp;
    while( # Parse each line of input
      m!
        (.{$marker_length})          # Match the record marker (save in $1)
        ($record_type_regex)         # Match any record type (save in $2)
        (
         (??{'.'x$field_widths{$2})} # Match a field of correct width
        )                            # Save in $3
       !xg){
        $records{$2}.="$1$2$3\n";
      }
}
for my $file (sort keys %records){
    open my $OUT,'>',$file or die "Failed to open $file for writing: $!\n";
    print $OUT $records{$file};
    close $OUT
}

Invoke it as:

[user@host]$ ./myscript.pl file_of_data

Code tested and works with your given input.

Update

In your comments, you requested a "Unix equivalent" of the above. I highly doubt there exists such a thing, since the Perl expression used to parse your line is a highly irregular expression and I doubt that vanilla regular expressions can parse your given data format: it is too similar to a famous type of expression that regex can't parse (match any number of a's followed by the same number of b's).

In any case, the closest "Unix" approach I can find is the generalization of 1_CR's answer. You should note that this approach is specific to the GNU implementation of grep and therefore will not work on most Unices. The Perl approach, on the contrary, should work the same on any platform that Perl works on. Here's my suggested GNU grep approach:

cat <<EOF \
| while read -r record width;do
    grep -oE ".{7}$record.{$width}" input_file\ #replace 7 with marker length
     >> "$record"
done
A1 10
B1 4
# enter your 38 record types
EOF

Update

Based on the OP's requests in the comments, instead of passing the filename as a command line argument, it can be opened within the script like so:

open my $IN,'<',$input_file_name or die "Failed to open $input_file: $!\n";
while(<$IN>){ #instead of while(<>)
...

This assumes you have declared the variable $input_file_name to contain, well, the input file name.

As for appending a timestamp to the output file name, you can use the qx{} syntax: between the braces you can put any Unix command you want and it will be run and its standard output read back in place of the qx{} operator:

open my $OUT,'>',"$file_".qx{date +%Y-%m-%d--%I:%M:%S%P}

The qx operator is not restricted to braces, use your favorite character as delimiter, just make sure it's not in the command you need to run:

qx<...>
qx(...)    
qx!...!    
qx@...@

and so on...

In some Perl code you may see backticks (` `) used to serve this function instead, similar to what the shell does. Just think of the qx operator as the generalization of backticks to any delimiter.

By the way, this will give a slightly different timestamp to each file (if the difference of their creation times happens to be a finite number of seconds). If you don't want this, you can do it on two steps:

my $tstamp = qx{...};
open my $OUT,'>',"$file_$tstamp" or die...;
share|improve this answer
    
Hi again....beginning to really love perl. Just have a couple of niggly bits. 1. How to read in the file as opposed to pass in the command line argument. Trying but failing to use Eclipse run configuration. 2. How to append some text to the output filename $file. Most appreciated. –  jags Dec 24 '13 at 10:01
    
@jags Welcome to the club :). Answer updated. See if it helps. –  Joseph R. Dec 24 '13 at 10:24
    
Thanks Joseph. However for the last request I meant to actually append, for example, date/timestamp to the output filename. The current code outputs files A1,B1 & C1. Many thanks again. –  jags Dec 24 '13 at 16:37
    
@jags I see. Please see if the update helps. –  Joseph R. Dec 25 '13 at 11:36
    
Thanks as always Joseph. However I meant append to the actual output filename which in this case is currently A1,B1,C1, i.e. I want to add a date/timestamp, A1_<todays_date>, B1_<todays_date>, C1_<todays_date>. Many thanks. –  jags Dec 27 '13 at 17:24

Here's a posible solution using gawk's FPAT

BEGIN { 
    FPAT="A1.{10}|B1.{4}|C1.{7}" #define field contents
} 
{
    for(i=1;i<=NF;i++) 
        print $i >> substr($i,0,2) #print the field to file A1,B1,etc
}

As a one-liner:

gawk 'BEGIN{FPAT="A1.{10}|B1.{4}|C1.{7}"} {for(i=1;i<=NF;i++)print $i >> substr($i,0,2)}' < datafile
share|improve this answer
    
Note that FPAT requires gawk version 4. See: linuxjournaldigital.com/linuxjournal/201109#pg98 –  Håkon Hægland Dec 24 '13 at 19:44

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.