Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have 2 files.

File1 has 400k numerical records. e.g:

1
2
3
4
5
6
..
and so on

File 2 also has 420k numerical records. e.g:

1
2
3
4
6 
..
and so on

Both these file are in unsorted manner. I want to match the 2 file's and print the difference.

When I try using diff, comm, or grep it take a long time (more than an hour). This is not feasible for me.

How can I do this faster (matching and printing the difference).

I use HP -UX.

share|improve this question
3  
What kind of output are you expecting? 1 hour to compare files a couple of megabytes large. Is your machine from the 80s? What exactly did you try? –  Stéphane Chazelas Feb 9 '13 at 8:16
    
@StephaneChazelas: if he is using HP-UX then probably his hardware is from the 1980s or tops 1990s. –  tripleee Feb 9 '13 at 9:19
    
... In which case, unless you are connecting from a real DEC VT-100 terminal or a ZX-81, copying the files to your local workstation and doing the comparison there might be a good workaround. –  tripleee Feb 9 '13 at 9:21
    
What differences do you need? Line by line, know if there are new values? –  vonbrand Feb 9 '13 at 14:53

1 Answer 1

On a 10 million line file, generated with:

seq 10000000 |
  tee a |
  awk 'rand() < 0.05 {print int(1000000 * rand())}; 1' > b

all of:

diff a b | wc -l

comm -3 <(sort a) <(sort b) | wc -l

(ksh/bash/zsh syntax)

cmp -l a b | wc -l

Took under 30 seconds on a 3 year old low end PC (running Linux).

There could be big variations with diff depending on the content as diff algorithm that needs to detect insertions, deletions or changes will be affected by how the data is laid out, but there wouldn't be so much variation with the other ones.

What exactly did you try?

share|improve this answer
1  
"How much memory" is also a reasonable question. If you only have 128M of RAM, swapping (especially to old, slow disks) will dominate completely over the "real work". Switching to explicit temporary files could be a huge win in that scenario. –  tripleee Feb 9 '13 at 11:36

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.