3

I have a file named fields.txt and containing L=300k+ lines which looks like:

field1 field2 field3
field1 field2 field3
field1 field2 field3
... 
field1 field2 field3

In the same folder, I have N files containing only one string (let's identify it s(n)) and named res-0-n-0 , with n anywhere between 0 and L. However N < L.

I generated the file res_numbers_sorted.tmp containing the numerically sorted list of the above-mentioned n with the command (not sure if the most efficient, but it is reasonably fast and I need this sorted for other purposes)

find -maxdepth 1 -type f -name "res-0-*" | sort -t'-' -k3 -n | awk -F'-' '{print $3}'>| res_numbers_sorted.tmp

The file res_numbers_sorted.tmp looks like:

0
1
8
... 
299963

Finally, what I want is a file named results.txt which would like:

field1 field2 field3 s(0)
field1 field2 field3 s(1)
field1 field2 field3
...
field1 field2 field3 s(299963) 
...
field1 field2 field3

where again s(n)is the string contained in the n-th res-0-n-0.

I have achieved what I want first by cp fields.txt resutls.txt and then with the following while loop:

while IFS='' read -r line; do 
     #storing the content of the file in a variable
     res=$(<res-0-"$line"-0)     
     # this is needed in order to take into account that sed addresses the first line of a file with the number 1 whereas the file list starts with 0
     real_line=$(( line + 1 ))     
     sed -i "${real_line}s/.$/ ${res}/" field.txt
done < res_numbers_sorted.tmp

However this is very slow and I need to perform this several times. I suspect that maybe sed is not the right tool for this job.

2
  • sed is not a bad tool for the job, the issue is that you iterate on the same file over and over. What you could do is generate a sed script by iterating on res_numbers_sorted.tmp, and then applying it once on field.txt. Maybe this would improve perfomance
    – Ikaros
    Apr 19, 2018 at 9:31
  • 1
    I compared the two solutions proposed (the sh script and the sed based one) with time. Both the solutions are much faster than mine with the former displaying this result: real 875.849 user 48.629 sys 92.359 pcpu 16.09and the latter this result: real 244.680 user 136.436 sys 9.536 pcpu 59.65 . Therefore I am choosing the first one
    – woland
    Apr 19, 2018 at 11:59

2 Answers 2

2

If I understand correctly, you have afields.txtfile with many line. You have several res-0-n-0 files. And, for each line in fields.txt you what to copy them in results.txt with, if it exist, the content of the res-0-<line_number>file.

I think you can simply read the fields.txt file line by line and echo the line in the results.txtwith the content of res-0-<line_number>file if needed.

I would go with something like this:

#! /bin/sh

LINE_NUMBER=0
while read line;
do
  if [ -f "res-0-$LINE_NUMBER-0" ]
  then
    echo "$line $(cat res-0-$LINE_NUMBER-0)" >> result.txt
  else
    echo "$line" >> result.txt
  fi
  ((LINE_NUMBER++))
done < fields.txt
1

Try generating a sed script and then applying it only once on you file field.txt :

while IFS='' read -r line; do
    res=$(<res-0-"$line"-0)
    real_line=$(( line + 1 ))
    prinft "%s" "${real_line}s/.$/ ${res}/" >> myscript.sed
done < res_numbers_sorted.tmp

and then do :

sed -i -f myscript.sed field.txt

This way, you only iterate once over your large file. Let me know if this helps.

1
  • Thanks, this is a very good solution, but I am choosing the other one as it is faster
    – woland
    Apr 19, 2018 at 12:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .