0

I have two text files in which I have to use the comm command to extract all unique words from file 1. So just those that are not in file 2. I was asked to use the comm command (not diff nor join).

I have tried a lot of things such as comm -32 file1 file2, but this returns all the words in file 1.

  • maybe your file1 contains only unique words? what's the input you are feeding? – umläute Oct 29 '13 at 9:12
  • No there are indeed words that are similar, I checked using grep. 'input you are feeding'? I'm comparing sorted lists of words. File1 contains 2574 words and file2 around 40 000 words, using the comm command I get the same list of words of file1, with 2574 words. – hdb004 Oct 29 '13 at 9:14
  • 1
    please provide excerpts of the two files that exhibit the problem. also check whether the files have different EOL-markers (e.g. CRLF vs LF) – umläute Oct 29 '13 at 9:20
  • I don't know those things sorry... – hdb004 Oct 29 '13 at 9:25
  • see wikipedia. it basically boils down to your two files containing invisible characters that make two words differ even if they seem identical. you still haven't provided sample input for your problem. – umläute Oct 29 '13 at 9:52
3

comm expects sorted files where each word has to be on one line (it's the full lines that are compared).

So it's

(export LC_ALL=C
comm -23 <(grep -Po '\S+' file1 | sort) <(grep -Po '\S+' file2 | sort))

(assuming GNU grep with PCRE support and a shell supporting process substitution like ksh93, zsh or bash).

Fixing the locale to C to perform byte to byte comparison for sort and comm.

  • (export LC_ALL=C comm -23 <(grep -Po '\S+' tekst.txt | sort) <(grep -Po '\S+' woordenlijst.txt | sort)) -bash: export: -23': not a valid identifier -bash: export: /dev/fd/63': not a valid identifier -bash: export: `/dev/fd/62': not a valid identifier – hdb004 Oct 29 '13 at 10:37
  • @hdb004, that's 2 lines, the first one sets LC_ALL, the second calls comm. That's in a subshell for limiting the scope of the LC_ALL variable – Stéphane Chazelas Oct 29 '13 at 10:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.