Explanation of sed in shell script

Question

I found a snippet of code online that would actually help with creating the frequency of each word in a text file however I would like someone to explain exactly how it is doing so

Especially the sed command since i am super new to bash and I need to know what are all the seperators are doing ( s/\(.*\)/\L\1/ ).

Here is the code:

cat EnglishText.txt
sed 's/\.//g;s/\(.*\)/\L\1/;s/\ /\n/g' EnglishText.txt | sort | uniq -c

I want to know what exactly comes after the sed, i do understand the uniq -c and the sort but i would like to know what is going on in the matching and so on .. i know this is somehow weird but again i am very new to this

In the same context of the question

sed  's/\([0-9]*\).*/\1/'

What would this mean ?

Answer 1

The sed script consists of three substitute commands. Substitute commands are of the form s/old/new/ which looks for something in the text that matches regular expression old and replaces it with new. If a g is put after the command, then this substitution is done repetitively ("globally"). The first one removes periods. The second one makes the text lower case. The third puts each word on its own line. In more detail:

• s/\.//g

  This matches periods in the input and replaces them with nothing.

• s/\(.*\)/\L\1/

  This matches anything in the input and replaces it with a lower-case version of the same.

• s/\ /\n/g

  This replaces spaces with newlines. This has the effect of putting each word on a separate line.

Example

Note that the period is removed and all words are lowercased and put on separate lines:

$ echo 'This test is this test.' | sed 's/\.//g;s/\(.*\)/\L\1/;s/\ /\n/g'
this
test
is
this
test

This form is suitable for sorting and counting:

$ echo 'This test is this test.' | sed 's/\.//g;s/\(.*\)/\L\1/;s/\ /\n/g' | sort | uniq -c
      1 is
      2 test
      2 this

Improvement

The sed script, as written, does not do anything with other punctuation, like ?"!, or with tabs. With minor modifications to the above code, all can be handled:

$ echo 'This "test(?)" is this test!' | sed 's/[[:punct:]]//g; s/.*/\L&/; s/[[:space:]]/\n/g' | sort | uniq -c
      1 is
      2 test
      2 this

This uses the same type of substitute commands as the original with just small changes:

1. s/[[:punct:]]//g removes all punctuation characters.

2. s/.*/\L&/ converts all upper case characters to lower case.

3. s/[[:space:]]/\n/g replaces all whitespace with newline characters.

Addendum

If a line starts with a number, sed 's/\([0-9]*\).*/\1/' keeps that number and removes everything after it. All other lines are just removed. For example:

$ echo '123 tests' | sed  's/\([0-9]*\).*/\1/'
123
$ echo 'There are 123 tests' | sed  's/\([0-9]*\).*/\1/'