3

I need to identify patterns in a text file for further analysis. So the input files may contain semi-structured text as follows;

file1

905:john: abc123: john@doe.com: US  
920:eric: ericaA: eric007@gmail.com: US  
1000: rio: ri0ri0: rio@yahoo.com: IN  

file2

nathen <tab> nathen@yman.com <tab> 764323545 <tab> UK
thomas <tab> thom@gmail.com <tab> 563363421 <tab> UK
ian <tab> rt@gmail.com <tab> 3453245472 <tab> SP

number of words in a line may vary for each document. delimiter also vary but unique for each document. what I want is to count number of words per each line in each document.

output would be:

for file1

5  
5 
5 
5   

for file2

4  
4 
4 
4

I want generalize this for any file with any delimiter. It could be - | : \space+ \tab+.

Some files are as follows:

| hetro@gmail.com | er34532 |  
| rt@gmail.com | 764474 |
6

For file2 with its simple format of white-space separated words, try:

$ awk '{print NF}' file2
4
4
4

In awk, NF is the number of fields. In awk by default, words characters separated by white-space.

For file1, for which words may be separated by colons or spaces, we need to add : to the field separator:

$ awk -F'[: \t]+' '{print NF}' file1
5
5
5

Improvement

awk will think there are extra fields if a line has leading or trailing whitespace. We can fix that by removing any such whitespace before we count the fields, like this for file1:

awk -F'[: \t]+' '{gsub(/^[: \t]+|[: \t]+$/, ""); print NF}' file1

or this for file2:

awk '{gsub(/^[[:space:]]+|[[:space:]]+$/, ""); print NF}' file2

In the regex, ^[[:space:]]+ matches leading whitespace and [[:space:]]+$ matches trailing whitespace. Their combination in the gsub command removes both.

Generalization

If the field separator could be any of - | : or , then use:

awk -F'[-|: \t]+' '{gsub(/^[[-\|: \t]+|[[-\|: \t]+$/, ""); print NF}'

Example:

$ cat file3
| hetro@gmail.com | er34532 |  
| rt@gmail.com | 764474 |
$ awk -F'[-|: \t]+' '{gsub(/^[[-\|: \t]+|[[-\|: \t]+$/, ""); print NF}' file3
2
2
  • Very informative. Thanks. Please see my reply also – user3148655 May 15 '16 at 4:44
  • @user3148655 Your welcome. Also, I added code that handles all those possible delimiters. – John1024 May 15 '16 at 5:30
2

You could use this trick in perl to count substrings matching a word regex of your choosing e.g. to count sequences of "words" consisting of perl's \w character set augmented with @ and .

perl -lne 'print my $count = () = $_ =~ /[\w@.]+/g' somefile

which gives

$ perl -lne 'print my $count = () = $_ =~ /[\w@.]+/g' file1
5
5
5

and

$ perl -lne 'print my $count = () = $_ =~ /[\w@.]+/g' file2
4
4
4

for your sample files.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.