I would like to count the same words across multiple files then show in which file they are.

File1:

This is so beautiful

File2:

There are so beautiful

File3:

so beautiful

The desired output 1:

so:3
beautiful:3

The desired output 2:

so:
file1:1
file2:1
file3:1

beautiful:
file1:1
file2:1
file3:1
  • 2
    This is a programming question and off-topic here. – RalfFriedl Sep 14 at 5:50
  • 1
    but this is Linux instruction – Yu Quan Sep 14 at 6:14
  • 1
    Hi and Welcome to U&L! Please see the help center to see what this site is about. – mhutter Sep 14 at 6:49
  • I think I should go stackflow to ask this question? sorry I think I go wrong way – Yu Quan Sep 14 at 6:58
  • 2
    This question is not off-topic. It is better to assemble existing and specific utilities than to implement a new program. Moreover, it is easy to solve his problem using Unix utilities. – Fólkvangr Sep 14 at 7:14

Try this,

# Declare the files you want to include
files=( file* )

# Function to find common words in any number of files
wcomm() {
    # If no files provided, exit the function.
    [ $# -lt 1 ] && return 1
    # Extract words from first file
    local common_words=$(grep -o "\w*" "$1" | sort -u)
    while [ $# -gt 1 ]; do
        # shift $1 to next file
        shift
        # Extract words from next file
        local next_words=$(grep -o "\w*" "$1" | sort -u)
        # Get only words in common from $common_words and $next_words
        common_words=$(comm -12 <(echo "${common_words,,}") <(echo "${next_words,,}"))
    done
    # Output the words common to all input files
    echo "$common_words"
}

# Output number of matches for each of the common words in total and per file
for w in $(wcomm "${files[@]}"); do
    echo $w:$(grep -oiw "$w" "${files[@]}" | wc -l);
    for f in "${files[@]}"; do
        echo $f:$(grep -oiw "$w" "$f" | wc -l);
    done;
    echo;
done

Output:

beautiful:3
file1:1
file2:1
file3:1

so:3
file1:1
file2:1
file3:1

Explanation:

Included as comments inside the script.

Features:

  • As many files as your ARG_MAX allows
  • Finds all words separated by anything grep understands as word separator.
  • Ignores case, so "beautiful" and "Beautiful" is the same word.
  • The comparison is only between the first file and the others. In other words, common words are only those included in the first file, therefore, some common words (multiple occurrences) in other files are not selected. – Fólkvangr Sep 16 at 9:05
  • Thats intended. Common words are words that are shared in all files. That's what I understood from the question. – RoVo Sep 16 at 11:08

Try this code. if needed make adjustments

bash-4.1$ cat test.sh
#!/bin/bash

OUTPUT_FILE=/tmp/output.txt

awk '{
for(i=1;i<=NF;i++)
{
        Arr[$i]++
}
}
END{
for (i in Arr){
        if(Arr[i]>1)
        {
                print i":"Arr[i]
        }
}
}' file* > ${OUTPUT_FILE}

cat ${OUTPUT_FILE}
echo ""

IFS=":"
while read WORD TOTAL_COUNT
do
        echo "${WORD}:"
        for FILE_NAME in file*
        do
                COUNT=$(tr ' ' '\n' < ${FILE_NAME} | grep -c "${WORD}")
                if [ "${COUNT}" -gt "0" ]
                then
                        echo "${FILE_NAME}:${COUNT}"
                fi
        done
done < ${OUTPUT_FILE}


bash-4.1$ bash test.sh
beautiful:3
so:3

beautiful:
file1:1
file2:1
file3:1
so:
file1:1
file2:1
file3:1
  • 2
    you should explain your code. Letting code without comment like that could made people unsafe about using it. – Kiwy Sep 14 at 8:16

Using grep to feed the words and files' name and then awk to reformat the output to get desired result:

grep -Ho '\w\+' file* |
awk -F':' '{ words[$1 FS $2]++; seen[$2]++ }
END{ for (x in seen) {
         print x":" seen[x];
         for (y in words) {
            if (y ~ "\\<" x "\\>")print substr(y, 1, length(y)-length(x)), words[y]
         }
     }
}'

This will give you a nice output as following (both desired outputs in one go):

so:3
file1: 1
file2: 1
file3: 1
This:1
file1: 1
beautiful:3
file3: 1
file1: 1
file2: 1
There:1
file2: 1
are:1
file2: 1
is:1
file1: 1

Try this all awk approach:

awk '
        {for (i=1; i<=NF; i++)  {WC[$i]++
                                 FC[$i,FILENAME]++
                                }
        }
END     {for (w in WC) if (WC[w] > 1) print w, WC[w]

         print ""

         for (f in FC)  {split (f, T, SUBSEP)
                         w = T[1]
                         if (WC[w] > 1) {if (!D[w]) print w, ""
                                         print T[2], FC[f]
                                         D[w] = 1
                                        }
                        }
        }

' OFS=":" file[1-3]
so:3
beautiful:3

beautiful:
file3:1
file2:1
file1:1
so:
file1:1
file2:1
file3:1

It collects the respective data (word count & word count per file) for each file, and then, in the END section, produces the desired outputs 1 and 2, based on word count > 1.

perl, counts all the words in the specified files, and for words seen more than once, print the total count and the count per file

$ cat file1
This is so beautiful
foo
foo
foo
$ cat file2
There are so beautiful
foo
bar
bar
$ cat file3
so beautiful
bar
baz

then

perl -lane '
    for (@F) {$count{$_}++; $filecount{$_}{$ARGV}++}
    END {
        for $word (sort keys %count) {
            if ($count{$word} > 1) {
                print "$word:$count{$word}";
                print "$_:$filecount{$word}{$_}" for sort keys %{ $filecount{$word} };
                print "";
            }
        }
    }
' file{1,2,3}
bar:3
file2:2
file3:1

beautiful:3
file1:1
file2:1
file3:1

foo:4
file1:3
file2:1

so:3
file1:1
file2:1
file3:1

To sort the results by descending order of "count", you would use this line in the END block:

for $word (reverse sort {$count{$a} <=> $count{$b}} keys %count) {
  • This work for me!!! Thank you so much! Can the output be from large amount to little? – Yu Quan Sep 15 at 4:25
  • Could you tell me what kind of language is this ? Thank you – Yu Quan Sep 15 at 5:13
  • What kind of language? I don't understand what you mean. You can learn more about perl on the perl tag page. – glenn jackman Sep 15 at 11:09

This process displays the numbers of occurrences of words included in text files located in a directory or subdirectories, for each file and in all files.

  1. Concatenate all text files.
  2. Suppress duplicated words.

The result is a list of words and each word in the list is then counted in each file successively as well as in concatenated files.

  1. Search the number of occurrences for each word in the list.

1. Concatenate all text files

find . -type f -exec cat {} \;

find searches all text files in the current directory or subdirectories and invoke cat to concatenate all matching files.

2. Suppress duplicated words

Words may be placed separately on each line to create a list of words: each character other than a letter is replaced by a newline, then, a sequence of newlines is replaced by one newline.

tr -cs '[:alpha:]' '[\n*]'

Finally, the duplicated words must be suppressed to get a suitable list of words. uniq may filter out repeated lines of text but the lines must be sorted, therefore, sort may be used to sort lines of text.

sort | uniq

or

sort -u

Note: a case-insensitive search may be performed adding the following command in the pipeline.

tr '[:upper:]' '[:lower:]'

Potential issues

An existing script file should not be concatenated with other files even if it is located in the current directory or in subdirectories (c.f. man find).

find . -type f \( \! -name "*${0##*/}" \) -exec cat {} \;

Note: $0 expands to the name of the shell. ${PARAMETER##WORD} expands to the parameter with the longest matching pattern deleted (c.f. shell parameter expansions).

For example, /usr/local/bin/myscript becomes myscript.

Result

list=$(find . -type f \( \! -name "*${0##*/}" \) -exec cat {} \; | 
           tr -cs '[:alpha:]' '[\n*]' | tr '[:upper:]' '[:lower:]' | sort -u)

The command substitution allow to save the output of the pipeline in a variable.

3. Search the number of occurrences...

a) Searches all lines containing the given word, each occurrence on a different line...

grep --exclude="*${0##*/}" -Rowi $word .

... and counts number of lines

grep --exclude="*${0##*/}" -Rowi $word . | wc -l

b) Counts the matching lines for each input file.

grep --exclude="*${0##*/}" -Rowi $word . | tr '[:upper:]' '[:lower:]' |
    uniq -c | sed -E "$sed_script"

Searches all lines containing the given word (each occcurrence is on a different line), prints the number of occurrences as well as the source file and reformats data.

Note: $0 expands to the name of the shell. ${PARAMETER##WORD} expands to the parameter with the longest matching pattern deleted (c.f. shell parameter expansions).

For example, /usr/local/bin/myscript becomes myscript.

Note: --exclude is a GNU extension, not specified for POSIX grep. If this is an issue, then, remove the option and add the following statement at the top of the shell script.

# assign first positional parameter to "dirname"
# and move to this directory

dirname=${1:?first positional parameter missing\!}
cd "$dirname"

Shell script

#!/bin/sh

list=$(find . -type f \( \! -name "*${0##*/}" \) -exec cat {} \; | 
           tr -cs '[:alpha:]' '[\n*]' | tr '[:upper:]' '[:lower:]' | sort -u)

# extract data and print formatted data
sed_script='s/([[:digit:]]+) (.*):[[:alpha:]]+/\2:\1/;s/[[:blank:]]+//'

for word in $list; do
    echo $word:$(grep --exclude="*${0##*/}" -Rowi $word . | wc -l)
    grep --exclude="*${0##*/}" -Rowi $word . | tr '[:upper:]' '[:lower:]' | 
        uniq -c | sed -E "$sed_script"
    echo
done | sed -e "/^[[:alpha:]]\+:1/{N;N;d;}"

Input

prompt% cat file1
Lorem ipsum dolor sit amet. cat cat
prompt% cat file2
Lorem ipsum dolor sit amet, consectetur adipiscing elit. cat ut
prompt% cat file3
Lorem ipsum dolor sit amet, consectetur adipiscing elit.
Sed tristique egestas massa sed facilisis. Duis hendrerit ut. cat
tristique tristique tristique tristique

Output

adipiscing:2
./file2:1
./file3:1

amet:3
./file1:1
./file2:1
./file3:1

cat:4
./file1:2
./file2:1
./file3:1

consectetur:2
./file2:1
./file3:1

dolor:3
./file1:1
./file2:1
./file3:1

elit:2
./file2:1
./file3:1

ipsum:3
./file1:1
./file2:1
./file3:1

lorem:3
./file1:1
./file2:1
./file3:1

sed:2
./file3:2

sit:3
./file1:1
./file2:1
./file3:1

tristique:5
./file3:5

ut:2
./file2:1
./file3:1
  • Thank you for your reply!!! it helps, but actually I'm looking forward to compare two different article and find the same word then print – Yu Quan Sep 14 at 7:57
  • @YuQuan: this answer provides the expected output. What do you mean? – Fólkvangr Sep 14 at 10:46
  • he wants to search common words. so and beautiful are just examples. – RoVo Sep 14 at 10:51
  • Thank you !! But I try this code I didn't got the same output like you – Yu Quan Sep 15 at 4:17
  • @YuQuan: I explained the process in detail, hoping it will help. – Fólkvangr Sep 15 at 7:41

If you don't want to write a code, just using fast way to know the results, you can use this command:

cat list_of_words | while read line; do echo $line; grep -riE '$line'-c where_to_look_or_folder; done

-r :read into files
-i: no casesensitive
-E: regexp is useable if you want something more complicated to search
-c: counter

Output:

word1
path:filename:count

Example:

cat text | while read line; do echo $line; grep -riE '$line'-c somwhwere/nowhere; done

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.