Some files we get from a customer could not be processed properly because they were declared as US-ASCII but contained invalid characters. In order to validate a software fix, I am trying to copy several lines from the original file to a new file. The original files are quite large and I only need a couple of lines.

The original line looks like this in gedit 003002002002\D4M, the \D4 part in red and I get a warning about invalid characters. on the console, the original line looks like this: 003002002002�M

I have tried to pipe the files through head & tail, but the resulting file contains a 'fixed' character instead of the invalid one.

head -n 449025 invalid.txt | tail > invalid_short.txt

the line looks like this if I cat the resulting file: 003002002002�M but if i open the resulting file with gedit it looks like this: 003002002002ÔM and there are no invalid characters

I have tried to select the lines in gedit and save them as a new file, but I only get a representation of the invalid char, not the original one. It looks like this with cat and gedit (no invalid char warning, no red): 003002002002\D4M

Looks like I am missing something simple, but I have no idea what.

Thank you

Ubuntu 16.04, zsh

  • What you show appears to be UTF or similar but head and tail don't change the data, which is why I don't post this as an answer. Jul 19, 2016 at 15:29
  • Yes, I have assumed UTF at first, but when reading the file as UTF, the reader broke with java.nio.charset.MalformedInputException: Input length = 1. It does not break if I advise the codec to replace malformed input with a null byte. file -bi returns us-ascii.
    – kostja
    Jul 19, 2016 at 15:33

1 Answer 1


head or tail will not fix/change the character.

What probably happens is that gedit tries to guess the encoding of the file based on the first few bytes. When that 0xD4 is far within the file, gedit guesses the file is in ASCII or UTF-8 and complains when it sees that 0xD4 byte that is invalid in either ASCII or UTF-8.

While for the second shorter file, since 0xD4 is near the beginning of the file, gedit guesses the character set is some 8 bit character set (probably iso8859-1 where 0xD4 is Ô).

So your:

head -n 449025 invalid.txt | tail > invalid_short.txt

method is correct. If you run sed -n l on both files, you'll see in both cases something like 003002002002\324M. And with hd: 30 30 33 30 30 32 30 30 32 30 30 32 d4 4d |003002002002.M|.

  • even simpler than I hoped for :) Thanks Stephane
    – kostja
    Jul 19, 2016 at 15:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .