Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a big playlist in .pls format (created with deadbeef playing software) which also contains some badly-encoded strings. Whether this is a bug in deadbeef or one of its plugins does not really interest me, though, since I just want to cut off from a certain position to the end of the line using sed.

However, this is easier than it looks at first sight, since sed will instantly interrupt parsing once it hits a character it deems "invalid":

$ cat archives.pls | grep --perl-regexp Lika.*rar.*02
rar:///home/andy/audio_compressed/world/ru/Lika Star - Ya (RUS 2001).rar:2001- �/02 ���.mp3

This is the original line, unmodified. Now let's try sed on it, to extract the path (note: this is for a script I'm almost finished with to check for "dead" audio files in the directory tree, i. e. those that do no longer exist)

 $ cat archives.pls | grep --perl-regexp Lika.*rar.*02 | sed -e 's/^\(zip\|rar\|7z\):\/\///' -e 's/\(\.\(zip\|rar\|7z\)\):.*/\1/'
 /home/andy/audio_compressed/world/ru/Lika Star - Ya (RUS 2001).rar�/02 ���.mp3

Ugh. This obviously did not work, though the regular expression is definitely correct (since it can be proved to work fine with so-called "valid" characters). The rar� actually comes from a contraction of originally rar:2001- �, which indicates that sed parsed from the colon via the "2001-" to the blank, then stopped abruptly at the invalid � character. Still, there is the question left why doesn't sed just ignore what encoding it is and parses anyway? I mean, we want everything to be cut off after the colon following the "rar", so it ought to be safe to assume that .* does definitely stand for any character?

NOTE: Please refrain from "seemingly elegant" solutions using cut in your answers. Been there, done that. Instead, you had better consider that we're on a UNIX-type OS and we may even legitimately use strange file names that involve artists that originally spell ":wumpscut:" or the like. In other words, there might be a colon something in-between as well, which is why it would not be a good idea to statically cut off at the first colon we come across after the (zip|rar|7z)://.

share|improve this question
1  
What are those characters? Please pipe the grep output through od -c or od -t x1 –  glenn jackman Nov 26 '13 at 20:40
3  
will instantly interrupt parsing once it hits a character it deems "invalid" -> There may not be a choice. E.g. UTF-8 includes multibyte characters, and if you try to process something that's encoded wrongly by that standard, you can hit a byte (!= a character) that doesn't make sense, meaning it is impossible to interpret the next byte because there's no valid context to interpret it in. Note that you haven't reported the actual error here, just a lot of noise about it. –  TAFKA 'goldilocks' Nov 26 '13 at 20:48
    
Your question wording is misleading and unclear. As your example demonstrates, sed parsed the input just fine since you got the .mp3 which came after the questionable characters. Your real question is "Why doesn't . match � in a sed regex, and how do I match �?". Then, it becomes obvious that the real question is what on earth "�" is. I second glennjackman's suggestion - run od on the input. Also what is your shell's locale (echo $LANG)? In all likelihood running LANG=C sed will probably work. –  jw013 Nov 26 '13 at 21:48
add comment

2 Answers

As goldilocks already commented above, UTF-8 is a multibyte variable-width encoding. Each character might consist of up to four bytes. After an invalid byte you can at best hope that the next byte might start a new character.

sed tries to match characters not bytes. As the invalid byte is no character and the . only matches characters it will not match and your result is absolutely correct.

You have to find out what encoding is used for that file. (The file command might help you with this.) Setting the environment variable LANG to the correct value should be enough for sed to work the way you intended it to. In your case probably LANG=C should be enough.

share|improve this answer
add comment

Set your LANG environment variable to some UTF-8 encoding string like LANG='en_US.UTF-8'. Here, on Mac OS X 10.6.8, it is LANG='C' that causes GNU sed 4.2.1 to stop parsing as described in the question. GNU sed 4.2.2, on the other hand, just works fine.

Another modification that works for me (on Mac OS X 10.6.8) is to replace the regex part :.* with: :\([^[:print:]]\|[[:print:]]\)* (POSIX character class).

echo $'rar:///home/andy/audio_compressed/world/ru/Lika Star - Ya (RUS 2001).rar:2001- \357\277\275/02 \357\277\275\357\277\275\357\277\275.mp3' > archives.pls

(
export LANG='C'
#locale charmap
cat archives.pls | grep --perl-regexp 'Lika.*rar.*02'
cat archives.pls | grep --perl-regexp 'Lika.*rar.*02' | gsed -e 's/^\(zip\|rar\|7z\):\/\///' -e 's/\(\.\(zip\|rar\|7z\)\):.*/\1/'    # GNU sed 4.2.1
cat archives.pls | grep --perl-regexp 'Lika.*rar.*02' | gnused -e 's/^\(zip\|rar\|7z\):\/\///' -e 's/\(\.\(zip\|rar\|7z\)\):.*/\1/'  # GNU sed 4.2.2
cat archives.pls | grep --perl-regexp 'Lika.*rar.*02' | gsed -e 's/^\(zip\|rar\|7z\):\/\///' -e 's/\(\.\(zip\|rar\|7z\)\):\([^[:print:]]\|[[:print:]]\)*/\1/'
)


# output:
# rar:///home/andy/audio_compressed/world/ru/Lika Star - Ya (RUS 2001).rar:2001- �/02 ���.mp3
# /home/andy/audio_compressed/world/ru/Lika Star - Ya (RUS 2001).rar�/02 ���.mp3
# /home/andy/audio_compressed/world/ru/Lika Star - Ya (RUS 2001).rar
# /home/andy/audio_compressed/world/ru/Lika Star - Ya (RUS 2001).rar

#--------------

(
export LANG='en_US.UTF-8'
#locale charmap
cat archives.pls | grep --perl-regexp 'Lika.*rar.*02'
cat archives.pls | grep --perl-regexp 'Lika.*rar.*02' | gsed -e 's/^\(zip\|rar\|7z\):\/\///' -e 's/\(\.\(zip\|rar\|7z\)\):.*/\1/'    # GNU sed 4.2.1
cat archives.pls | grep --perl-regexp 'Lika.*rar.*02' | gnused -e 's/^\(zip\|rar\|7z\):\/\///' -e 's/\(\.\(zip\|rar\|7z\)\):.*/\1/'  # GNU sed 4.2.2
cat archives.pls | grep --perl-regexp 'Lika.*rar.*02' | gsed -e 's/^\(zip\|rar\|7z\):\/\///' -e 's/\(\.\(zip\|rar\|7z\)\):\([^[:print:]]\|[[:print:]]\)*/\1/'
)

# output:
# rar:///home/andy/audio_compressed/world/ru/Lika Star - Ya (RUS 2001).rar:2001- �/02 ���.mp3
# /home/andy/audio_compressed/world/ru/Lika Star - Ya (RUS 2001).rar
# /home/andy/audio_compressed/world/ru/Lika Star - Ya (RUS 2001).rar
# /home/andy/audio_compressed/world/ru/Lika Star - Ya (RUS 2001).rar
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.