Some files we get from a customer could not be processed properly because they were declared as US-ASCII but contained invalid characters. In order to validate a software fix, I am trying to copy several lines from the original file to a new file. The original files are quite large and I only need a couple of lines.
The original line looks like this in gedit 003002002002\D4M
, the \D4
part in red and I get a warning about invalid characters. on the console, the original line looks like this: 003002002002�M
I have tried to pipe the files through head & tail, but the resulting file contains a 'fixed' character instead of the invalid one.
head -n 449025 invalid.txt | tail > invalid_short.txt
the line looks like this if I cat the resulting file: 003002002002�M
but if i open the resulting file with gedit it looks like this: 003002002002ÔM
and there are no invalid characters
I have tried to select the lines in gedit and save them as a new file, but I only get a representation of the invalid char, not the original one. It looks like this with cat and gedit (no invalid char warning, no red): 003002002002\D4M
Looks like I am missing something simple, but I have no idea what.
Thank you
Ubuntu 16.04, zsh
head
andtail
don't change the data, which is why I don't post this as an answer.java.nio.charset.MalformedInputException: Input length = 1
. It does not break if I advise the codec to replace malformed input with a null byte.file -bi
returnsus-ascii
.