8

I occasionally get tarballs where the filenames are encoded in ISO-8859-1 or some other pre-Unicode scheme. My system uses UTF-8, so when I untar these archives with the usual options (tar xvf foo.tar) I end up with a directory full of mojibake filenames.

Until now I've been using convmv to convert the filenames to UTF-8 after they've been extracted. This is a bit incovenient, because I either need to invoke convmv on each affected file, or else untar the file into a new directory, run convmv on the entire directory, and then move the files to where I wanted them originally. Short of coding this functionality into a shell script, is there some way of converting the archived filenames to UTF-8 on the fly, as they are being untarred?

2 Answers 2

12

Here is a little tar file extractor that modifies the names in memory before extracting:

#!/usr/bin/python27

import tarfile

def transform(data):
    u = data.decode('latin1')
    return u.encode('utf8')

tar = tarfile.open('archive.tar')
for m in tar.getmembers():
    m.name = transform(m.name)

tar.extractall()

Warning: unlike GNU tar, this extractor is not stripping leading /. Either add checking logic to this extractor or check each tar file before extracting with tar -t.

2
  • This is OK as long as you don't need to pass any additional options to tar. I don't suppose there's any way of generalizing your solution so that it supports the same options as GNU tar…?
    – Psychonaut
    Sep 9, 2015 at 7:48
  • @Psychonaut Compression is handled by tarfile transparently. Options that handle on the fly manipulation (change UID, change paths) can be done in the same way as in the example. Filtering (extract only a subset) is also easy: create a list of all member, filter them, iterate over the list with tar.extract(). So it's all doable, just work. However, as GNU tar has tons of options, some of them really obscure, a full compatible version is not worth the effortd. Sep 9, 2015 at 10:07
0

Thomas Erker's answer almost worked for me, except that the getmembers call seems to read the entire archive into memory, and the tar file I have is too big for that. So here is a modified version that reads one file at a time. The name transformation is also different, as in my tar file the file names show up with surrogate characters; I'm not sure if that will be generally applicable.

#!/usr/bin/env python3

import tarfile
import sys
from pathlib import Path

def transform(data):
    try:
        data.encode('utf-8')
        return data
    except UnicodeEncodeError as err:
        u = data.encode('utf-8', 'surrogateescape')
        v = u.decode('latin-1')

        print(f'Replaced filename: {v}')
        v.encode('utf-8') # check that it can be encoded
        return v

tar = tarfile.open(sys.argv[1])
while (m := tar.next()) is not None:
    if m.name == '.':
        continue
    m.name = transform(m.name)
    path = Path(m.name)
    try:
        if path.stat().st_size == m.size:
            print(f'Already exists with the same size: skipping {m.name}')
            continue
        else:
            print(f'Size mismatch, extracting: {m.name}', flush=True)
    except FileNotFoundError:
        print(f'About to extract {m.name}', flush=True)
    tar.extract(m)
    print(f'Extracted {m.name}')

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.