I have two files:

  1. one generated using find command in a folder to list files, sorting them numerically and writing to a file,
  2. and the other generated by a python script, which is not sorted, so I explicitly sort it numerically.

The problem is that my sort output only has two columns and is as follows:


As you can see, even the common entries are shown separately in two columns and the third column is missing altogether, implying that there is nothing common between the two files, whereas these first three entries are indeed same. sort otherwise works as expected with some test files that I make.

I know that comm needs the two files to be lexically sorted, and here is a list of options I tried and failed:

comm <(sort file1.txt) <(sort file2.txt)

from https://unix.stackexchange.com/a/377689/187419 failed. I also tried giving the -d option to sort explicitly, and also tried explicitly rewriting the files with dictionary sort -- both didn't work

comm --check-order <(sort file1.txt) <(sort file2.txt)

from https://unix.stackexchange.com/a/186101/187419 did not return any order error; it ran as usual giving two output columns.

This solution for a problem very close to mine is also not working.

Thinking that it might be because of some additional characters in the file, I also the solution mentioned here to do :set list in vim.

Just to test if sort is causing issues, I deliberately sorted the test files I made (with which comm worked earlier) numerically and comm still worked.

I tried the solutions I could find, to no avail. Any other suggestions?

  • Line endings? Trailing whitespace? (Check with hex, hexdump, od -c, etc.) On a UNIX-type system the line ending should be (just) \n. There should be no \r. – roaima May 7 '18 at 19:42

You are almost certainly right that additional characters on each line are causing corresponding lines to fail to match exactly. Those additional characters might have the form of carriage-return characters from Windows-style line terminators, space or tab characters, or possibly other non-printing characters. For example, maybe the Python script is right-justifying the numbers so that some or all of them have leading spaces.

The surest thing to do would be to filter out all such unwanted characters, and since the data are strictly numeric, that's pretty easy to do with, for example, sed:

sed 's/[^0-9]//g' < input > output

You could interpose that at various points in your process. Here's just one:

comm <(sed 's/[^0-9]//g' file1.txt | sort) <(sed 's/[^0-9]//g' file2.txt | sort)
  • This worked! I want to find out which extra characters are lingering in those files. Can you suggest a way? This is just for my learning. Also, if I know I will be better able to integrate the solution in my larger program. – user128785 May 7 '18 at 20:36
  • @user128785, the comment on your original question suggests severall alternatives aimed exactly at looking for and identifying the extra characters. Which tools are available to you will depend on your particular system. For instance, I don't have hex, but I do have hexdump and od. Alternatively, various editors have options that would reveal the extra characters, but details vary, of course. – John Bollinger May 7 '18 at 21:05
  • Following the suggestion of @roaima, I checked od -c file1.txt and similarly for file2.txt, and found that the file being written by the python script had \r\n line-endings. I installed and used dos2unix to convert the line-endings to \n, and simple comm without any options worked normally. I also modified my python3 script to have the newline='\n' attribute explicitly specified but I have not tested this yet. After this modification I expect the script to work, but if I have issues with it I will post another question. – user128785 May 7 '18 at 21:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.