Some files we get from a customer could not be processed properly because they were declared as US-ASCII but contained invalid characters. In order to validate a software fix, I am trying to copy several lines from the original file to a new file. The original files are quite large and I only need a couple of lines.
The original line looks like this in gedit
\D4 part in red and I get a warning about invalid characters. on the console, the original line looks like this:
I have tried to pipe the files through head & tail, but the resulting file contains a 'fixed' character instead of the invalid one.
head -n 449025 invalid.txt | tail > invalid_short.txt
the line looks like this if I cat the resulting file:
but if i open the resulting file with gedit it looks like this:
003002002002ÔM and there are no invalid characters
I have tried to select the lines in gedit and save them as a new file, but I only get a representation of the invalid char, not the original one. It looks like this with cat and gedit (no invalid char warning, no red):
Looks like I am missing something simple, but I have no idea what.
Ubuntu 16.04, zsh