Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have an issue where I am getting a flatfile source with a huge number of records with PIPE delimiters and one of the fields is getting carriage returns (in multiple lines) and starts with a newline (\n). So how can I remove the \n character in the file?

Example:

-000123456|1654321|6/12/2002 8:49:20 AM|
tt Cynthia L Eggleston E456585 remove move the funds adv account in fcle flagged on 710091 pmt due 12-16- 15|

Can anyone suggest how to proceed?

share|improve this question
    
Is it suitable to remove all \n characters, or is there a special case when a \n is not desired? –  Shawn J. Goff Jun 4 at 13:56

3 Answers 3

If you know, that this field begins with \n in all of the records, you can use sed as follows:

sed "N;s/|\n/|/"

to get rid of the new-line. Note, that since sed uses \n as line delimiters, you first have to join the next line to the one processed (the N command), during this process sed inserts the newline character between the joined lines.

If this problem arises randomly (not in every records), you'll have to resort to a stronger tool to parse the file content - basically you need to count the fields. Either awk or perl can do the trick elegantly (I think it could be accomplished in sed as well as the, but you probably don't want go that direction).

share|improve this answer
    
This only works for every other line - and sed is plenty strong enough to handle this - sed is only as strong as its script. Look into the P;D;H;h;x;l commands specifically. You can easily count fields in sed as well like : s/field_delimiter/replace/3 –  mikeserv Jun 4 at 5:21
    
It is supposed to work like that - and it is exactly what the OP needs. –  peterph Jun 4 at 10:36
awk 1 ORS=

Result

-000123456|1654321|6/12/2002 8:49:20 AM|tt Cynthia L Eggleston E456585 remove move the funds adv account in fcle flagged on 710091 pmt due 12-16- 15|
share|improve this answer
    
Won't that remove all the newlines from the file? –  Mat Jun 4 at 5:46
    
@Mat hm I thought that is what he wanted... –  Steven Penny Jun 4 at 5:49

You can use many tools, but sed was built for exactly these kinds of situations:

 sed -i 's,\\n,,g' Flatfile

To show you how it works, here's the same expression with the input you provided just echoed:

$ echo '-000123456|1654321|6/12/2002 8:49:20 AM|\n tt Cynthia L Eggleston E456585 remove move the funds adv account in fcle flagged on 710091 pmt due 12-16- 15|' | sed 's,\\n,,g'
-000123456|1654321|6/12/2002 8:49:20 AM| tt Cynthia L Eggleston E456585 remove move the funds adv account in fcle flagged on 710091 pmt due 12-16- 15|

It just looks for the string \n and replaces it with an empty, zero-length string (effectively removing it). There are two \, so \n loses its special (newline character) meaning and can be matched as two characters.

share|improve this answer
1  
This is not right. First of all echo doesnt recognize \n unless you use echo -e. Secondly sed doesnt recognize newlines either unless you append them to the pattern space as in the other answer. –  Steven Penny Jun 4 at 4:37
    
The question was different when first asked, that's why the answer is now out of sync and appears wrong. I guess I should update it. –  lynxlynxlynx Jun 18 at 21:50

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.