1

how can I remove duplicate lines from the data set in a .txt file because the problem is that half content of my lines are repeated and I want only first RESULT line. I tried to use the command but this didn't work well

sort myfile.txt uniq -u | newfile.txt

here is the content of my file and my lines of my file are 299873

ligand_06278/out.pdbqt:REMARK  Name = 22626427
ligand_06278/out.pdbqt:REMARK VINA RESULT:      -8.3      2.094      2.612
ligand_06278/out.pdbqt:REMARK VINA RESULT:      -8.3      2.821      8.000
ligand_06278/out.pdbqt:REMARK VINA RESULT:      -8.4      3.333      6.628
ligand_06278/out.pdbqt:REMARK VINA RESULT:      -8.4      4.526      7.557
ligand_06278/out.pdbqt:REMARK VINA RESULT:      -8.5      2.500      4.835
ligand_06278/out.pdbqt:REMARK VINA RESULT:      -8.5      2.516      7.135
ligand_06278/out.pdbqt:REMARK VINA RESULT:      -8.6      2.660      7.148
ligand_06278/out.pdbqt:REMARK VINA RESULT:      -8.8      3.141      6.023
ligand_06278/out.pdbqt:REMARK VINA RESULT:      -8.9      0.000      0.000
ligand_06279/out.pdbqt:REMARK  Name = 22629712
ligand_06279/out.pdbqt:REMARK VINA RESULT:      -6.1      9.841     13.115
ligand_06279/out.pdbqt:REMARK VINA RESULT:      -6.3     15.483     18.543
ligand_06279/out.pdbqt:REMARK VINA RESULT:      -6.3      1.944      5.962
ligand_06279/out.pdbqt:REMARK VINA RESULT:      -6.3      8.946     12.260
ligand_06279/out.pdbqt:REMARK VINA RESULT:      -6.5     14.453     17.240
ligand_06279/out.pdbqt:REMARK VINA RESULT:      -6.8     10.330     14.145
ligand_06279/out.pdbqt:REMARK VINA RESULT:      -6.8      1.727      5.848
ligand_06279/out.pdbqt:REMARK VINA RESULT:      -7.1      7.429     11.509
ligand_06279/out.pdbqt:REMARK VINA RESULT:      -7.3      0.000      0.000
ligand_06280/out.pdbqt:REMARK  Name = 22631372
ligand_06280/out.pdbqt:REMARK VINA RESULT:     -10.0      3.811      7.264
ligand_06280/out.pdbqt:REMARK VINA RESULT:     -10.1      0.000      0.000
ligand_06280/out.pdbqt:REMARK VINA RESULT:      -9.3      5.006      9.020
ligand_06280/out.pdbqt:REMARK VINA RESULT:      -9.4      2.195      8.687
ligand_06280/out.pdbqt:REMARK VINA RESULT:      -9.4      2.712      9.301
ligand_06280/out.pdbqt:REMARK VINA RESULT:      -9.6      2.186      8.354
ligand_06280/out.pdbqt:REMARK VINA RESULT:      -9.7      5.168      7.981
ligand_06280/out.pdbqt:REMARK VINA RESULT:      -9.8      1.961      2.580
ligand_06280/out.pdbqt:REMARK VINA RESULT:      -9.8      2.311      8.341
  • 1
    What is the expected output? Is it one single line or one line per "section"? Or something else? – Kusalananda Mar 28 '18 at 9:50
  • And why put a so long sample input ? You can makes the same 70% shorter. Please, read MCVE – Gilles Quenot Mar 28 '18 at 9:54
1
$ awk -F: '$1 != p && /RESULT/ { print; p = $1 }' file
ligand_06278/out.pdbqt:REMARK VINA RESULT:      -8.3      2.094      2.612
ligand_06279/out.pdbqt:REMARK VINA RESULT:      -6.1      9.841     13.115
ligand_06280/out.pdbqt:REMARK VINA RESULT:     -10.0      3.811      7.264

This outputs the first RESULT line from each file mentioned in the input file. It does this by comparing the first column (the filename) to the first column of the previous line and testing whether the current line contains the word RESULT. When a RESULT line is found which has a different filename from the previous, the line is printed as is, and the value of p is updated.


I'm noticing that the input file looks very much like the result of grep having been run over a number of files, possibly grepping for REMARK.

To find all files and get the first line matching REMARK VINA RESULT from each:

find . -type f -path './ligand_*' -name 'out.pdbqt' -exec sed -n '/REMARK VINA RESULT/{p;q;}' {} ';'

or, as a simple loop:

for name in ligand_*/out.pdbqt; do
    grep -F 'REMARK VINA RESULT' "$name" | head -n 1
done

I've used different approaches here, you pick the one that feels most natural.

  • You can use -m1 instead of head in the for loop. – terdon Mar 28 '18 at 10:18
  • @terdon Yes, with some implementations of grep. – Kusalananda Mar 28 '18 at 10:19
  • Oh, I thought -m was POSIX. Sorry. – terdon Mar 28 '18 at 10:27
  • Thank you so much it is really helpful but I also want each "REMARK Name =" line before 'REMARK VINA RESULT' because REMARK Name is not showing in the output . how will it be processed ?? – Bushra Mar 29 '18 at 4:51
  • @Bushra This contradicts the question where you say "I want only first RESULT line". – Kusalananda Mar 29 '18 at 5:16
0

You could simply grep for Name= including the following line (-A1) and pass the output though grep RESULT to remove the Name= lines:

$ grep -A1 'Name =' file | grep RESULT
ligand_06278/out.pdbqt:REMARK VINA RESULT:      -8.3      2.094      2.612
ligand_06279/out.pdbqt:REMARK VINA RESULT:      -6.1      9.841     13.115
ligand_06280/out.pdbqt:REMARK VINA RESULT:     -10.0      3.811      7.264
  • thank you for help but this output is not showing REMARK Name line. How will we get this one ?? – Bushra Mar 29 '18 at 4:53
  • @Bushra when asking a question, it is important that you explain what you actually need. Kusalananda left you a comment yesterday asking you what your desired output would be, but you never answered so both he and I gave you answers that don't work for you. Anyway, just run grep -A1 'Name =' file and it should be what you want. If not, edit your question and clarify. – terdon Mar 29 '18 at 9:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.