6

is it possibile to decode a file name by command line?

Suppose I have the following two files:

  • foo.mp3
  • bar.mp3

Is there any command line tool that decodes the files names into their UTF-8 values:

  • 0x66 0x6F 0x6F 0x2E 0x6D 0x70 0x33
  • 0x62 0x61 0x72 0x2E 0x6D 0x70 0x33
  • 2
    Just a simple point: You appear to want to show the file contents, not decode the file name. – Julie Pelletier Mar 16 '17 at 14:48
  • 4
    @JuliePelletier looks like the question is perfectly clear: how to decode a filename into UTF-8 octets – Gianluca Ghettini Mar 16 '17 at 14:59
  • 4
    For for instance, would you like to see 0x20ac (the Unicode code point of that EURO character) or 0xe2 0x82 0xac (the 3 bytes of the UTF-8 encoding of that EURO character)? – Stéphane Chazelas Mar 16 '17 at 16:20
  • 1
    ...and what to do if the input (file name) isn't a valid UTF-8 byte sequence? – ilkkachu Mar 16 '17 at 16:24
  • The target of the question is to get, starting from a file name, the UTF-8 encoded values. So in the case of the € symbol, I expect: 0xe2 0x82 0xac. The solution proposed by Stéphane, based on iconv, tells you if a file name is well encoded or not – Grynium Mar 17 '17 at 10:09
14

The standard (POSIX/Unix) command to get the byte values as hex numbers is od.

file=foo.mp3
printf %s "$file" | od -An -vtx1

Which gives an output similar to:

 66 6f 6f 2e 6d 70 33

$file above contains an arbitrary array of (non-NUL for shells other than zsh) bytes. The character encoding doesn't enter in consideration.

If you want $file to contain an array of characters (so in the locale's encoding) and you want to get the Unicode code points for each of them as hexadecimal numbers, on a Little-Endian system, you could do:

printf %s "$file" | iconv -t UTF-32LE | od -An -vtx4

See also:

printf %s "$file" | recode ..dump

Or:

printf %s "$file" | uconv -x hex/unicode
printf %s "$file" | uconv -x '([:Any:])>&hex/unicode($1)\n'

If you wanted the byte values as hex numbers of the UTF-8 encoding of those characters:

printf %s "$file" | iconv -t UTF-8 | od -An -vtx1

For something like foo.mp3 that contains only ASCII characters, they're all going to be equivalent.

  • You mean UTF-32LE instead of UTF32LE? – cuonglm Mar 16 '17 at 15:45
  • @cuonglm, on my system, both works. But I've changed it to UTF-32LE if that's more portable. – Stéphane Chazelas Mar 16 '17 at 16:04
  • Yep, at least in my OSX El Capital, UTF32LE does not work. – cuonglm Mar 16 '17 at 16:07
  • I tried to use UTF-32LE but it seems not supported:"iconv: conversion to `UTF-32LE' is not supported". It's strange because "iconv --list | grep UTF" returns both UTF-32LE and UTF-32BE. Anyway, specifying UTF-8 works fine – Grynium Mar 16 '17 at 16:18
  • 1
    I'm a bit "out of context", but about the choices of unicode out there, I want to point people reading this answer to a good read : utf8everywhere.org ... spread the word! ^^ – Olivier Dulac Mar 16 '17 at 19:09
4

With perl:

$ perl -CA -le 'print join " ", map { sprintf "0x%X", $_ } unpack "U*" for @ARGV' \
  foo.mp3 bar.mp3 cường
0x66 0x6F 0x6F 0x2E 0x6D 0x70 0x33
0x62 0x61 0x72 0x2E 0x6D 0x70 0x33
0x63 0x1B0 0x1EDD 0x6E 0x67

If you store those list of filenames in a file, then:

perl -CI -lne 'print join " ", map { sprintf "0x%X", $_ } unpack "U*"' <file
4

With plain Bash:

a=abcdefghij    
for ((i=0;i<${#a};i++));do printf %02X \'${a:$i:1};done
6162636465666768696A

Customize the printf format to suit your needs.

  • This will fail for unicode characters. – cuonglm Mar 16 '17 at 15:41
3

I think this could be done with a simple Perl script:

perl -we 'foreach my $file (glob("*")) { 
   printf "0x%02X ", ord($_) foreach split//, $file; print "\n" }; '

resulting in output similar to yours when the directory contains files foo.mp3 and bar.mp3:

0x62 0x61 0x72 0x2E 0x6D 0x70 0x33 
0x66 0x6F 0x6F 0x2E 0x6D 0x70 0x33 

(I think glob() sorts the names.)

Explanation:

foreach my $file (glob("*")) {   # loop over all filenames in current directory 
    printf "0x%02X ", ord($_)    # take the character as a number, and print in hex
      foreach split//, $file;    # .. after splitting the filename to characters
    print "\n";                  # add a newline
}

(* except those that start with a dot)

This will just print the actual bytes in the filenames, as they are stored on the file system. If you have files whose names are not in UTF-8, the script will not convert them.

You could also do something like piping the output of ls to od or xxd, which would also work with other data than lists of file names, but would bring all the problems that come with reading ls, and it would be hard to get the file names separated to different lines.

  • 1
    Note that this doesn't print the unicode codepoint of the unicode characters. – cuonglm Mar 16 '17 at 15:36
  • 2
    @cuonglm Like it says on the lid, it will print the actual bytes, as they are stored on file system. It's a bit hard to say what they want, given the examples in the question only contain plain ASCII characters. – ilkkachu Mar 16 '17 at 15:48
  • Hi ilkkachu, first of all thanks for your answer. The purpouse of the question is to be able to decode every kind of UTF-8 characters, not only ASCII set – Grynium Mar 16 '17 at 16:21
  • And pass the a filename pattern (glob string) to the above perl to only show files that match the pattern desired. – ChuckCottrill Mar 16 '17 at 18:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.