6

I scrapped a site with wget.

That site is in German and some of that pages had Ü,ü,Ö,ö,Ä,ä,ß in the URL.

Now some files have a very weird name.
For example one file is called mirror.de/�%9Cbersicht.html

Is there a way to run a command that changes that weird encoding to a proper one?

In the example case I would expect the following as a valid result mirror.de/Uebersicht.html

EDIT: Output of LC_CTYPE=C ls | grep bersicht.html | od -t x1:

0000000 42 69 6e 61 72 79 20 66 69 6c 65 20 28 73 74 61
0000020 6e 64 61 72 64 20 69 6e 70 75 74 29 20 6d 61 74
0000040 63 68 65 73 0a
0000045
6
  • 3
    have you tried --restrict-file-names in wget ?
    – jai_s
    Feb 11, 2016 at 12:36
  • @jai_s no, but I'm going to try that. Feb 11, 2016 at 12:56
  • 1
    Ü in UTF-8 in URL encoding is %c3%9c. What exact byte sequence does the URL consist of (your post has the substitution character)? You have a local file now, right? What's the output of LC_CTYPE=C ls | grep bersicht.html | od -t x1? Feb 11, 2016 at 22:37
  • @Gilles Added the output of the command you asked for. Feb 17, 2016 at 9:49
  • 1
    Sorry, that isn't useful because grep printed Binary file (standard input) matches. I hadn't thought of that. LC_CTYPE=C ls *bersicht* | od -t x1 or echo *bersicht* | od -t x1 would give useful output. But Stéphane's analysis is probably correct, did you try his answer? Feb 17, 2016 at 10:02

2 Answers 2

3

Chances are that if you just decode the %XX URI encodings, you'll get UTF-8 encoded characters. In zsh:

autoload zmv
LC_ALL=C zmv '(**/)(*%*)' '$1${2//(#b)%(??)/${(#):-0x$match}}'

If as noted by Gilles, that � substitution character is for a 0xc3 byte (which in iso8859-1 is a letter which would explain why it hasn't been URI-encoded contrary to 0x9c which is invalid in iso8859-1), then the above command would change your file name to Übersicht.html

0
3

After decoding the %XX URI encodings to valid UTF-8, you could also have used convmv -f utf-8 -t latin-1 to convert the character set of the filename.

That might be an useful alternative solution if zsh is not available.

1
  • Note that my zsh approach only does the URI decoding, not conversion of charset. I don't think the OP wants the file names in latin1. But in any case, convmv is indeed a useful tip if you need to convert the charset in the file name. May 1, 2019 at 8:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.