0

I am receiving a messaging file in pipe delimited format. One message line is very lengthy almost 6000. And the total file size is more than 6gb. Below is the sample format of the file. Need to parse the file and bring everything into one line.

I need to remove the new line character from the middle of data:

File: abc.txt
File_Name|abc.txt|date|04212019|this is one full line|Client_name|Whole
File_Name|abc.txt|date|04212019|half data is good
File_Name|abc.txt|date|04212019|Sample data
is split|Client_Name|Marshals
File_Name|abc.txt|date|04212019|this is good again|Processing_date|03282019
File_Name|abc.txt|date|04212019|line is not good 
again|Processing_date|04232019

I want the data to be like this:

File_Name|abc.txt|date|04212019|this is one full line|Client_name|Whole
File_Name|abc.txt|date|04212019|half data is good
File_Name|abc.txt|date|04212019|Sample data is split|Client_Name|Marshals
File_Name|abc.txt|date|04212019|this is good again|Processing_date|03282019
File_Name|abc.txt|date|04212019|line is not good again|Processing_date|04232019

I am using Linux.

I tried using perl -ef which throwing out of memory error.

  • @Philippos I checked the edit history, and I did not remove that newline. The text after the supposed newline is a long, single "word", so your browser is probably wrapping the whole thing to the next line. Have a look at the original submission to confirm the lack of a newline in the original question. (FWIW I selected the code and pressed Ctrl+k, to minimise the chance of user error.) – Sparhawk Apr 27 at 23:49
  • @Sparhawk You are right, my mistake. Sorry for that. Anyhow there is reason to believe that a newline is intended. – Philippos Apr 29 at 7:05
  • @Philippos I absolutely agree that it's almost certainly a typo, but hopefully the author can confirm. – Sparhawk Apr 29 at 9:24
  • The sed solution by @Philippos seems correct as per your described requirement and sample file. If that solution throws an error as you say, tell us what error exactly. GNU's sed (the one usually present on Linux systems) can deal with files of any size and lines of any length, as long as you have enough RAM memory. I guess it might be that some part of that 6GB file is actually one huge line filling your RAM – LL3 Apr 30 at 0:01
1

Thus, you want to join lines not starting with File_Name to the previous line?

In sed, you use an N;P;D pattern for this kind of task:

sed 'N;/\nFile_Name/!s/\n/ /;P;D' abc.txt
  • N appends the next line to the pattern space
  • /\nFile_Name/ addresses all lines with File_Name after the new line; ! inverts the selection, so the next command is executed only if the second line of the two lines in the pattern space doesn't start with File_Name
  • s/\n/ / replaces the newline between the lines with a whitespace
  • P prints the first line in the pattern space
  • D deletes everything up to the newline and starts a new cycle with the second line still in the pattern space (to the next line gets appended to have a new pair of lines)

Please note that this will only work to join two lines. If lines can be broken into more lines, we need to add a loop or do it differently.

  • sed 'N;/\nFile_name/!s/\n/ /;P;D' abc.txt is throwing error as sed command grabled. – sam Apr 25 at 13:13
  • Which kind of error, please? I tested it and it should be completely portable – Philippos Apr 25 at 14:33
0

Here's another version using perl that will strip multiple new lines from the text:

perl -pe 's/\n//' abc.txt | perl -pe 's/(.)File_Name/\1\nFile_Name/g'

It first strips all newlines from the text, and then inserts new ones before every occurrence of "File_Name", when it is preceded by at least one character.

You can pipe it through more if you need to clean up e.g. multiple spaces:

perl -pe 's/\n/ /' abc.txt \
| perl -pe 's/(.)File_Name/\1\nFile_Name/g' \
| perl -pe 's/ +/ /g'
  • Thanks Guys. Issue with the file is a big message coming in as a Pipe '|" delimited. Line Size itself is 6000 and the file min size is 6gb. So when I use perl -pe it is running out of memory. – sam Apr 25 at 12:47
  • Any Help please – sam Apr 26 at 19:03
0

If, for some reason, the -pe version is blowing up, then here is the standalone perl program: stripper.pm This is the standard way to do things based on what the previous line has in it. You'd run it via

perl stripper.pm <abc.txt >new_abc.txt

#!/usr/bin/perl
my $previous = <STDIN>;

if( defined $previous ){
    chomp $previous;
};

while( $line = <STDIN> ){
    chomp $line;

    unless( $line =~ m/^File_Name/ ){
        $previous .= $line;

    } else { 
        print STDOUT "$previous\n";
        $previous = $line;
    }
}

print STDOUT "$previous\n";

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.