Replace any number of tabs and spaces with single new line in Linux?

Question

Suppose I have a (potentially very large) text file that contains a word list with whitespace interjected.  For example, it might look like this:

Cat                           Dog
Soup                          Rat
Cass                          Audrey

I want each word on a separate line (with no whitespace), like this:

Cat
Dog
Soup
Rat
Cass
Audrey

I can do a simple tr -d " " to make that into:

CatDog
SoupRat
CassAudrey

(but that is not what I want).

I do not know what type of blank space separates those words, so assume that it's some combination of ordinary ASCII spaces and tabs.  (We can assume that there are no invisible Unicode characters like em spaces and zero-width thingies.)  Naturally, the words do not contain whitespace, so "à la", "alma mater", "apple pie", "at large" and "ice cream" are not valid words.

Assume that words may contain (non-blank) non-alphabetic characters, such as "AC/DC", "add-on", "AT&T", "audio-visual", "can't", "carbon-14", "jack-o'-lantern", "mother-in-law", "o'clock", "O'Reilly", "RS-232" and "3-D".  Ideally the solution should tolerate non-ASCII characters, as in "Ångström", "Gödel", "naïve", "résumé" and "smörgåsbord".

How do I get rid of all those spaces while preserving (and isolating) the indented words using common Unix/Linux tools like tr, sed or awk?

It would be great if the solution would also work for more general cases of the stated problem; i.e., not just two-column text, but also random arrangements like:

          Once    upon
    a   midnight
                    dreary
while                     I pondered
       weak    and weary
           Over                many
a   quaint  and     curious     volume
 of forgotten lore

Answer 1

etopylight was almost right:

tr -s ' \t' '\n'

because the question asks to replace tabs, too.

Answer 2

Basically, you could do it in GNU sed:

sed 's/\s\+/\n/g'

There you go...

Answer 3

You should be able to use

sed -e 's/[[:space:]]\{1,\}/\n/'

to replace any sequence of one or more whitespace characters (including oddities like formfeed and vertical tabs) with a single newline.

Answer 4

If gnu-grep available,

grep -Po '\S+'

Answer 5

As the default behavior for awk already is to split on any number of blanks (spaces, tabs), one could as well use that feature, just setting the output field separator to "\n" and rebuilding $0.  An open question for the task, however, is: How do you want empty lines to be handled?

To just print them as they are:

awk -v OFS='\n' '{$1 = $1; print}' file

To additionally filter out empty lines:

awk -v OFS='\n' 'NF {$1 = $1; print}' file

(Beware of Windows line endings (containing \r) in a Linux setting, however: awk does not necessarily regard lines with \r as empty and in that case would output them too.  So filter text files with CRLF endings through dos2unix first.

Answer 6

It could be done in a long list of ways:

tr tr -s ' \t' '\n' <file for tabs and spaces only.
tr tr -s ' \t,."' '\n' <file for your strings.
tr tr -s '[:blank:]' '\n' <file for tabs and spaces only.
tr tr -s '[:space:]' '\n' <file for \t\n\v\f\r
sed sed -e 's/[ \t]/\n/g' -e 's/\n\n*/\n/g' file GNU sed for \n.
sed sed 's/[ \t".,]\+/\n/g' file | tr -s '\n' GNU sed for \n.

---

Description

tr

The most basic tool is tr, yes, but it could not understand format. So, a basic

tr -s ' ' '\n' <file

would convert all (repeated) spaces to one newline. That, of course could generate empty lines if lines with only spaces exist or the file start with spaces. There is no way to correct that in tr. It could be done adding a filter to remove empty lines. Like sed '/^$/d'

tr -s ' ' '\n' <file | sed '/^$/d'

Additional characters (like tabs) could be added:

tr -s ' \t' '\n' <file | sed '/^$/d'

Others, that might result from a text paragraph (like this one), like commas and periods could also be added. That is changing the definition of what a word is.

tr -s ' \t,.()' '\n' <file | sed '/^$/d'

sed

Sed is more capable than tr (and could be slower). It could change runs of some characters to one newline. The basic idea would be (in GNU sed for the replacement \n):

sed 's/[ \t]\{1,\}/\n/g' 

In other seds that are not able to use \n in the right side of a replacement we need to use an actual newline (and, sometimes, an actual tab):

sed 's/[   ]\{1,\}/\
/g'

So, sed helps with some issues but makes unnecessarily complex others.
Read 4.1. How do I insert a newline into the RHS of a substitution?

grep

It could be done in grep as well. Just match sequences of non-word characters:

grep -o '[^         ]\{1,\}'     ## explicit space-tab.

In GNU grep, the equivalent \S+ could be used:

grep -Eo '\S+'     ## or grep -o '\S\+'

awk

In awk we get still more power from the tool. It gets quite simple:

awk '{for(i=1;i<=NF;i++) {print $i}}' file

Which is just: print all fields for each line. Where there are fields. If the number of fields is 0 nothing will be printed.

That is using the default FS which delimits fields on repeated spaces, tabs or newlines.

A similar solution could be done with RS (for awks that allow regex separators):

awk -v RS="[ \t\n]+" 'NF'

Which tells awk to split records in runs of space, tab or newlines and print only if there is any no-empty field (the NF).

Answer 7

You can use the option -s from tr to squeeze repeated characters into one and replace it into a new line

tr -s " " "\n"

Answer 8

With GNU awk or any other awk that supports multi-char RS (e.g. newer versions of mawk):

$ awk -v RS='[[:space:]]+' '$0!=""' file
Cat
Dog
Soup
Rat
Cass
Audrey

$ awk -v RS='[[:space:]]+' '$0!=""' file2
Once
upon
a
midnight
dreary
while
I
pondered
weak
and
weary
Over
many
a
quaint
and
curious
volume
of
forgotten
lore

Answer 9

Simple optimal text formatter displays in one column with further deletion of leading spaces:

fmt -1 file | column -t

In awk NF prevents from possible output of the first empty line:

awk 'NF' RS='[[:space:]]+' file