1

I use the following command to insert new lines between characters in a text file in order to generate a single column of characters:

sed 's/./&\n/g' file.txt

This works for characters within ASCII; for example, from:

123

to:

1

2

3

I cannot get anything to work with multibyte characters such as Õöþùüê¡£¢¥Ÿ£€Ÿ£¡œãòá碜áòá, which appear on web pages that I download as HTML.

Any suggestions?

This is part of a solution I'm working on for another question I posted - I'm ultimately trying to generate a script to convert these characters to English- but I need to get each character on its own line.

closed as unclear what you're asking by Michael Homer, Tomasz, Kusalananda, dhag, GAD3R Feb 14 '17 at 19:20

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • By "funny", I assume you mean "anything that has a multi-byte UTF-8 representation"? It looks like your sed is assuming each byte represents one character - is it operating with a non-UTF-8 locale? (To check, use locale charmap in the same shell process as the sed command). – Toby Speight Feb 14 '17 at 17:45
  • Alternatively, the file might be in a unibyte encoding such as ISO-8859-1, in which case a mismatch between that and a UTF-8 sed could also cause problems. You might want to use od (perhaps with -t x1c) to examine the problematic characters closely. – Toby Speight Feb 14 '17 at 17:47
  • 3
    It is difficult to offer solutions when the problem statement is simply, "it doesn't work". Please edit your question to give a more complete description of what you expected to happen and how that differs from the actual results. See How to Ask for hints on what makes a good explanation. – Toby Speight Feb 14 '17 at 17:53
1

Any suggestions?

First of all, this works fine for me:

echo "Õöþùüê¡£¢" | sed 's/./&\n/g'
Õ
ö
þ
ù
ü
ê
¡
£
¢

Given this locale:

locale
LANG=sv_SE.UTF-8
LANGUAGE=
LC_CTYPE="sv_SE.UTF-8"
LC_NUMERIC="sv_SE.UTF-8"
LC_TIME="sv_SE.UTF-8"
LC_COLLATE="sv_SE.UTF-8"
LC_MONETARY="sv_SE.UTF-8"
LC_MESSAGES="sv_SE.UTF-8"
LC_PAPER="sv_SE.UTF-8"
LC_NAME="sv_SE.UTF-8"
LC_ADDRESS="sv_SE.UTF-8"
LC_TELEPHONE="sv_SE.UTF-8"
LC_MEASUREMENT="sv_SE.UTF-8"
LC_IDENTIFICATION="sv_SE.UTF-8"
LC_ALL=

My guess is that your locale is not UTF-8. Try to set your locale to UTF-8 and try again.

  • Well, echo "Õöþùüê¡£¢" | sed 's/./&\n/g' WORKS... – speld_rwong Feb 14 '17 at 17:19
  • My locale is UTF-8 – speld_rwong Feb 14 '17 at 17:29
  • I solved the problem while attempting to edit my post and give you some more information. I opened the text file, selected all the lines of text and then pasted the text into another text file and VOILA! I can now dump the original text file to a single column of data - go figure. – speld_rwong Feb 14 '17 at 20:01

Not the answer you're looking for? Browse other questions tagged or ask your own question.