7

Suppose I have a (potentially very large) text file that contains a word list with whitespace interjected.  For example, it might look like this:

Cat                           Dog
Soup                          Rat
Cass                          Audrey

I want each word on a separate line (with no whitespace), like this:

Cat
Dog
Soup
Rat
Cass
Audrey

I can do a simple tr -d " " to make that into:

CatDog
SoupRat
CassAudrey

(but that is not what I want).

I do not know what type of blank space separates those words, so assume that it's some combination of ordinary ASCII spaces and tabs.  (We can assume that there are no invisible Unicode characters like em spaces and zero-width thingies.)  Naturally, the words do not contain whitespace, so "à la", "alma mater", "apple pie", "at large" and "ice cream" are not valid words.

Assume that words may contain (non-blank) non-alphabetic characters, such as "AC/DC", "add-on", "AT&T", "audio-visual", "can't", "carbon-14", "jack-o'-lantern", "mother-in-law", "o'clock", "O'Reilly", "RS-232" and "3-D".  Ideally the solution should tolerate non-ASCII characters, as in "Ångström", "Gödel", "naïve", "résumé" and "smörgåsbord".

How do I get rid of all those spaces while preserving (and isolating) the indented words using common Unix/Linux tools like tr, sed or awk?

It would be great if the solution would also work for more general cases of the stated problem; i.e., not just two-column text, but also random arrangements like:

          Once    upon
    a   midnight
                    dreary
while                     I pondered
       weak    and weary
           Over                many
a   quaint  and     curious     volume
 of forgotten lore
5
  • set -f; printf ‘%s\n’ $(<file); set +f. This is halfway a joke, because there are other types of expansion in the shell besides globs, but in some hackish cases it might be a very simple solution.
    – kojiro
    Nov 14 '17 at 12:25
  • 1
    This is not a question "describing a problem that can't be reproduced and seemingly went away on its own (or went away when a typo was fixed)". This question describes a reproducible problem, whose solution(s) are likely to help future readers.  The fact that the OP didn't actually have the problem they described does not invalidate the question, per se. Apr 13 at 22:27
  • @G-Man it looks to me like the OP said in version 3 that "So it was looking like some words were appearing right-justified. I went through the same file more slowly with vim and there were no right-justified words." which sounds to me like they realized that there wasn't actually a problem to solve. If we want to reopen this question for the existing answers, I'd suggest editing the Q down to focus on the problem that they solve.
    – Jeff Schaller
    Apr 14 at 1:04
  • @Jeff: Well, I acknowledged that "the OP didn't actually have the problem they described".  So, what, exactly, are you suggesting?  That I delete the OP's edit (i.e., the last paragraph)?  Or should I purge all references to the “back story” of how a person might land in the situation of having a file like the one described in the question? Apr 14 at 5:19
  • @G-ManSays'ReinstateMonica' I personally think we should keep this question closed, since the OP is in no position to accept an answer. I'll abstain from voting in the reopen queue, though. If we think that this is the best question we have on removing spaces, then I would say to edit the Q to focus on that, removing the backstory and "I didn't actually have this problem" parts.
    – Jeff Schaller
    Apr 14 at 12:43
13

etopylight was almost right:

tr -s ' \t' '\n'

because the question asks to replace tabs, too.

4
  • 1
    The POSIX equivalent would be tr -s ' \t' '[\n*]'. See also tr -s '[:space:]' '[\n*]' or tr -s '[:blank:]' '[\n*]' Nov 14 '17 at 18:45
  • This fails on "AC/DC", "add-on", "AT&T". You need something like tr -s ' \t,."' '\n' <file
    – ImHere
    Apr 20 at 22:56
  • @Isaac Well, that’s debatable. The question says, “Assume that words may contain (non-blank) non-alphabetic characters”. (Disclosure: I edited the question to say that, with the intent of keeping my answer correct.) If words may contain non-alphabetic characters, then "AC/DC", "add-on" and  "AT&T" are all words. And the OP didn’t give us any clue how they want “Mr.”, “Mrs.”, “Ph.D” or “Q.E.D.” to be handled. While comma (and semicolon) maybe should always be separators, people with 20th century technology sometimes used " to denote umlaut / dieresis; e.g., na"ive for naïve. Apr 20 at 23:48
  • Fair enough. @G-ManSays'ReinstateMonica'
    – ImHere
    Apr 20 at 23:56
10

Basically, you could do it in GNU sed:

sed 's/\s\+/\n/g'

There you go...

0
6

You should be able to use

sed -e 's/[[:space:]]\{1,\}/\n/'

to replace any sequence of one or more whitespace characters (including oddities like formfeed and vertical tabs) with a single newline.

1
  • 6
    Almost portable, but most sed versions will insert a backslash and an n, because \n in the replacement is undefined by the standard. Use a literal newline instead (typically by typing backslash, Ctrl-V, Ctrl-J).
    – Philippos
    Nov 14 '17 at 7:29
2

You can use the option -s from tr to squeeze repeated characters into one and replace it into a new line

tr -s " " "\n"
2

If gnu-grep available,

grep -Po '\S+'
0

It could be done in a long list of ways:

tr -s ' \t' '\n' <file for tabs and spaces only.
tr -s ' \t,."' '\n' <file for your strings.
tr -s '[:blank:]' '\n' <file for tabs and spaces only.
tr -s '[:space:]' '\n' <file for \t\n\v\f\r
sed -e 's/ \t/\n/g' -e 's/\n\n*/\n/g' file GNU sed for \n.
sed 's/[ \t".,]\+/\n/g' file | tr -s '\n' GNU sed for \n.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.