command that will take a file and separate each word so its on its own line

Question

I have a file which has around 17523 words in it and I need to separate each word so it is on its own line (a word is defined to be a contiguous sequence of letters, so 1 letter words don't count) and to remove all blank lines. i don't know why this isn't working:

cat file.txt | tr [[:upper:]] [[:lower:]] | tr -d '\n' | tr " " '\n'

Answer 1

Grep's -o option is perfect for this: it prints each match on its own line.

grep -E -o '[[:alpha:]]{2,}' file.txt

And if you want that lowercased

grep -E -o '[[:alpha:]]{2,}' file.txt | tr '[:upper:]' '[:lower:]'

Note that the grep regex uses double brackets and tr does not: that's because a character class in a regular expression needs to be contained in brackets, while tr does not use regular expressions, it uses sets of characters.

Answer 2

Try:

cat file.txt | tr [[:upper:]] [[:lower:]] | tr '\n' ' ' | sed -E 's/[ \t]+/\n/g'

Your tr -d '\n' is joining all the words by removing spacing. The tr '\n' ' ' above preserve spacing.

Then, the spacing needs to be converted to one space, sed above does that. Converts several spaces (or tabs) [ \t]+ to one newline \n.

Answer 3

To answer the question on the title:

command that will take a file and separate each word so its on its own line

You could do:

<file tr '\n\t\r' ' '' '' ' | tr -s ' ' '\n'   # needs three spaces !

It will convert newlines, tabs and carriage returns to spaces and then …
convert any run (-s) of spaces back to one newline.

You could take advantage of the tr command and use it to also convert uppercase to lowercase in the same call:

<file tr '[:upper:]\n\t\r' '[:lower:]   ' | tr -s ' ' '\n'

Or you can do exactly the same in GNU sed (mind that this will slurp the whole file into memory and assumes that no NUL bytes exist inside the file):

<file sed -zE -e 'y/A-Z\n\t\r/a-z   /;s/ +/\n/g'

Then, to answer the question in the body:

(a word is defined to be a contiguous sequence of letters, so 1 letter words don't count) and to remove all blank lines.

you can remove words with characters other than a-z, one character words, and empty lines:

sed -E '/[^a-z]/d;/^.$/d;/^$/d'

It could be reduced to the slightly more cryptic:

sed -E '/[^a-z]/d;/^(.|)$/d'

All in one line, either:

<file tr '[:upper:]\n\t\r' '[:lower:]   ' | tr -s ' ' '\n' | sed -E '/[^a-z]/d;/^(.|)$/d'

Or:

<file sed -zE -e 'y/A-Z\n\t\r/a-z   /;s/ +/\n/g' | sed -E '/[^a-z]/d;/^(.|)$/d'

Commented version (works in GNU sed):

# Source `file` and use sed with the `zero` option (-z) and Extended Regex (ERE `-E`)
<file sed -zE -e '
    # Transliterate (-y) UPPER to lower and convert control to space.
y/A-Z\n\t\r/a-z   /
    # Restore consecutive spaces to **one** newline.
s/ +/\n/g
    # Second call to sed.
' | sed -E '
    # Delete (d) lines that have nay character not in the range a-z.
/[^a-z]/d
    # delete any line with one character or empty.
/^(.|)$/d
'

Answer 4

cat file.txt | awk 'NF''{gsub(/ /,"\n"); print tolower($0)}'

NF removes lines containing only blanks or tabs, then gsub will change spaces between words / / by new lines"\n". finally, tolower($0) will capital letters.

Answer 5

$ echo '  HTE ONTE NOTEH ONTEH E E O  AOE  ' | perl -pe '$_ =~ s/\b\w\b//g; $_ =~ s/\W*(\w+)\W*/\L$1\n/g'
hte
onte
noteh
onteh
aoe

This uses Perl to first remove any single-character words from the input, and then to extract and lower-case each remaining word with non-word characters removed and with each word on its own line.

Answer 6

perl -nale 'foreach $word (@F) {print $word;}' file.txt

or even more briefly:

perl -nale 'for (@F) {print}' file.txt

The perl command line options are:

-n - process each input line, no printing
-a - auto split each input line into words and store the result in special variable @F (list of fields)
-l - remove line breaks (\n) from the end of each input line and restore line breaks at the end of each output line
-e - execute code given on the command line

The code then iterates over each word in @F and prints it.

Another option is to just print the words in @F separated by newlines:

perl -nale 'print join("\n", @F)' file.txt