Getting Source as Flatfile and to remove the Carriage returns and \n …special charaters?

Question

I have an issue where I am getting a flatfile source with a huge number of records with PIPE delimiters and one of the fields is getting carriage returns (in multiple lines) and starts with a newline (\n). So how can I remove the \n character in the file?

Example:

-000123456|1654321|6/12/2002 8:49:20 AM|
tt Cynthia L Eggleston E456585 remove move the funds adv account in fcle flagged on 710091 pmt due 12-16- 15|

Can anyone suggest how to proceed?

Answer 1

If you know, that this field begins with \n in all of the records, you can use sed as follows:

sed "N;s/|\n/|/"

to get rid of the new-line. Note, that since sed uses \n as line delimiters, you first have to join the next line to the one processed (the N command), during this process sed inserts the newline character between the joined lines.

If this problem arises randomly (not in every records), you'll have to resort to a stronger tool to parse the file content - basically you need to count the fields. Either awk or perl can do the trick elegantly (I think it could be accomplished in sed as well as the, but you probably don't want go that direction).

Answer 2

You can use many tools, but sed was built for exactly these kinds of situations:

 sed -i 's,\\n,,g' Flatfile

To show you how it works, here's the same expression with the input you provided just echoed:

$ echo '-000123456|1654321|6/12/2002 8:49:20 AM|\n tt Cynthia L Eggleston E456585 remove move the funds adv account in fcle flagged on 710091 pmt due 12-16- 15|' | sed 's,\\n,,g'
-000123456|1654321|6/12/2002 8:49:20 AM| tt Cynthia L Eggleston E456585 remove move the funds adv account in fcle flagged on 710091 pmt due 12-16- 15|

It just looks for the string \n and replaces it with an empty, zero-length string (effectively removing it). There are two \, so \n loses its special (newline character) meaning and can be matched as two characters.