1

I have a file which has around 17523 words in it and I need to separate each word so it is on its own line (a word is defined to be a contiguous sequence of letters, so 1 letter words don't count) and to remove all blank lines. i don't know why this isn't working:

cat file.txt | tr [[:upper:]] [[:lower:]] | tr -d '\n' | tr " " '\n'
  • Next time please be explicit instead of saying "i don't know why this isn't working". You haven't said what happened. (You have said what you're trying to achieve though - thank you.) You haven't provided an example file that illustrates the problem. – roaima Sep 29 '18 at 22:37
2

Grep's -o option is perfect for this: it prints each match on its own line.

grep -E -o '[[:alpha:]]{2,}' file.txt

And if you want that lowercased

grep -E -o '[[:alpha:]]{2,}' file.txt | tr '[:upper:]' '[:lower:]'

Note that the grep regex uses double brackets and tr does not: that's because a character class in a regular expression needs to be contained in brackets, while tr does not use regular expressions, it uses sets of characters.

1

Try:

cat file.txt | tr [[:upper:]] [[:lower:]] | tr '\n' ' ' | sed -E 's/[ \t]+/\n/g'

Your tr -d '\n' is joining all the words by removing spacing. The tr '\n' ' ' above preserve spacing.

Then, the spacing needs to be converted to one space, sed above does that. Converts several spaces (or tabs) [ \t]+ to one newline \n.

1

To answer the question on the title:

command that will take a file and separate each word so its on its own line

You could do:

<file tr '\n\t\r' ' '' '' ' | tr -s ' ' '\n'   # needs three spaces !

It will convert newlines, tabs and carriage returns to spaces and then …
convert any run (-s) of spaces back to one newline.

You could take advantage of the tr command and use it to also convert uppercase to lowercase in the same call:

<file tr '[:upper:]\n\t\r' '[:lower:]   ' | tr -s ' ' '\n'

Or you can do exactly the same in GNU sed (mind that this will slurp the whole file into memory and assumes that no NUL bytes exist inside the file):

<file sed -zE -e 'y/A-Z\n\t\r/a-z   /;s/ +/\n/g'

Then, to answer the question in the body:

(a word is defined to be a contiguous sequence of letters, so 1 letter words don't count) and to remove all blank lines.

you can remove words with characters other than a-z, one character words, and empty lines:

sed -E '/[^a-z]/d;/^.$/d;/^$/d'

It could be reduced to the slightly more cryptic:

sed -E '/[^a-z]/d;/^(.|)$/d'

All in one line, either:

<file tr '[:upper:]\n\t\r' '[:lower:]   ' | tr -s ' ' '\n' | sed -E '/[^a-z]/d;/^(.|)$/d'

Or:

<file sed -zE -e 'y/A-Z\n\t\r/a-z   /;s/ +/\n/g' | sed -E '/[^a-z]/d;/^(.|)$/d'

Commented version (works in GNU sed):

# Source `file` and use sed with the `zero` option (-z) and Extended Regex (ERE `-E`)
<file sed -zE -e '
    # Transliterate (-y) UPPER to lower and convert control to space.
y/A-Z\n\t\r/a-z   /
    # Restore consecutive spaces to **one** newline.
s/ +/\n/g
    # Second call to sed.
' | sed -E '
    # Delete (d) lines that have nay character not in the range a-z.
/[^a-z]/d
    # delete any line with one character or empty.
/^(.|)$/d
'
0
cat file.txt | awk 'NF''{gsub(/ /,"\n"); print tolower($0)}'

NF removes lines containing only blanks or tabs, then gsub will change spaces between words / / by new lines"\n". finally, tolower($0) will capital letters.

0
$ echo '  HTE ONTE NOTEH ONTEH E E O  AOE  ' | perl -pe '$_ =~ s/\b\w\b//g; $_ =~ s/\W*(\w+)\W*/\L$1\n/g'
hte
onte
noteh
onteh
aoe

This uses Perl to first remove any single-character words from the input, and then to extract and lower-case each remaining word with non-word characters removed and with each word on its own line.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.