I have a number of files up to a million records coming from a vendor.
These files are originally from a Windows environment and when we get them they have erroneous splits in the records caused by Line Feeds in the middle of text fields.
I think this is caused by characters that get interpreted as Line Feeds during the transfer to Linux, but am not sure as we never see the original Windows files.
What I need is a routine that will count the number of delimiters on a line and if it is below a specified threshold to then delete the Line Feed at the end of the record. So for example we know that a record is 29 columns and should have 28 "pipe" delimiters ("|") when we get the data because of the erroneous Line Feed we get two records, one with, for example 10 fields and 9 delimiters and a second with 19 fields and 18 delimiters. Here is an example with the data changed for security:
9999999999|Duck Donald|87|||999999999|9999999999|XX999999|||Z99999|999 Planet Ln|||Trumpet ville|ZZ|99999||||||ZZ|P|9999999999|F|||
Notice that the line splits on the word Trumpetville. This is because of the Line Feed character that was inserted or mistranslated. Again looking for a woutine to count the delimiters and remove the Line Feed if the number of delimiters falls below specified threshold.