2

I have two files:

aaaa 11 0.4 12 0.2
aaab 40 0.1 99 0.2 69 0.3
aaac 222 0.5 21 0.3
aaad 2 0.1
aaae 33 0.3
....

and

aaaa
aaac
aaae
....

I need to compare the first column of first file with second file and if a element is present in second file, write each line of the first file to a separate file. I have a script that does that in python but its extremely inefficient. Is it possible to do it from a terminal?

EDIT:

python script: LABEL_FILE would be the first example and other 'file' - list is present_images-list of files in a folder.

 f = open(LABEL_FILE, 'r')
 present_images = iter(os.listdir(os.path.join(IMAGES_PATH, dataset)))

 templab = f.readlines()
 num_info = len(templab)
 image_ids = []
 labels = [] 
 labels_ind = [] 
 for line in templab:
     if len(line[:-1].split(' ')) != 1:
         if (line[:-1].split(' ')[0] in present_images):
             image_ids.append(os.path.join(IMAGES_PATH, dataset, line[:-1].split(' ')[0]))
             line = line[:-1].split(' ')[1:]
             labels_ind.append([int(i) for i in line[::2]])
             labels.append([float(j) for j in line[1::2]])
Improve this question
2
  • So you want to see if the entry in column 1 of the first file is anywhere in the second file, and if so print all of the first file? Or print only those lines in the first file whose first column exists in the second file? Perhaps if you edit your question to show us your python script it will clarify what you want. – Eric Renouf Dec 14 '16 at 16:16
  • @EricRenouf print only those lines in the first file whose first column exists in the second file exactly – jojo Dec 14 '16 at 16:23
2

With those files, you could use grep like:

grep -wf file2 file1

though you'll need to dos2unix file2 first since it has \r characters at the end.

This will match whole words with -w and read the patterns from the file with -f. This would actually match the patterns anywhere in the line, but with the sample input you gave us, it should get the job done.

As for your python code, you might want to consider spliting the line once and using that list many times instead of re-splitting it each time you want part of it

Improve this answer
4
  • You are right with python, that's silly. As for the terminal command it works with small files, but for my task it didn't produce output after computing for hours, I'll try to work around it. – jojo Dec 15 '16 at 9:09
  • @jojo you can also try adding -F to tell grep to treat everything as a fixed string, not a pattern and see if that helps. If it was running for hours without any output, are there only a few matches to be found? It might have been buffering the output and never hit enough results to fill the buffer, or that's a symptom I often find I have with grep when I leave off the last file argument so it's just sitting and waiting for input to process until I kill it and try again – Eric Renouf Dec 15 '16 at 11:42
  • Most of entries in file2 match ones in file1, but I think its really about the size of those files. In the end I want with python, splitting the list by patterns and matching in parts. Solved within 15 min. Thanks for alternative solution suggestion! – jojo Dec 15 '16 at 12:51
  • @jojo yeah, for many things programs will be faster than shell scripts so it would have been a little odd to have gone the other way – Eric Renouf Dec 15 '16 at 23:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.