File1 has 8 columns. The column 3 and 4 make a pairing like rs|860 rs|756 and the values associated with the pair is at column 8.
bb yy rs|860 rs|756 xx aa 0.7 2234 bb yy rs|310 rs|260 xx aa 0.3 9838 bb yy rs|110 rs|77 xx aa 0.5 2291 bb yy rs|756 rs|860 xx aa 0.4 2269 bb yy rs|110 rs|77 xx aa 0.9 1112 bb yy rs|756 rs|860 xx aa 0.8 3269 bb yy rs|233 rs|79 xx aa 0.4 1397 bb yy rs|79 rs|233 xx aa 0.7 1397
The pair of column3 and 4 or 4 and 3 should be treated same for instance, rs|860 rs|756 == rs|756 rs|860. Next, there might be repetition in the pairs in row one is also present in row4 and row6 (inversely). Simply here AB = BA = BA. First I want to sort the col3 and col4 row-wise which can make data like AB = AB = AB. Next, in all the occurrence of the pairs only one should retain with maximum value at column8. FOr example, rs|860 rs|756 has maximum value at row6 (i.e., 3269) so the row1 and row4 should be deleted. Likewise in case of rs|110 rs|77 row5 should be deleted. Next, in case of pairs with similar value at column8, one with higher value at column7 should be retained. For example, in case of rs|233 rs|79 row7 with lower value at column7 (0.4) should be deleted. For any pair, if both the column8 and column7 have no higher/lower difference any one can be deleted.
The desired output File2
bb yy rs|260 rs|310 xx aa 0.3 9838 bb yy rs|77 rs|110 xx aa 0.5 2291 bb yy rs|756 rs|860 xx aa 0.8 3269 bb yy rs|79 rs|233 xx aa 0.7 1397