1

I do not know how to write variables with sed

So, I want to do the following task using for loop:

sed -n '1,200p' big_file.txt  > 1to200.txt
sed -n '201,400p' big_file.txt  > 201to400.txt
sed -n '401,600p' big_file.txt  > 401to600.txt
sed -n '601,800p' big_file.txt  > 601to800.txt
sed -n '801,1000p' big_file.txt  > 801to1000.txt
sed -n '1001,1200p' big_file.txt  > 1001to1200.txt
sed -n '1201,1400p' big_file.txt  > 1201to1400.txt
sed -n '1401,1600p' big_file.txt  > 1401to1600.txt
sed -n '1601,1800p' big_file.txt  > 1601to1800.txt
sed -n '1801,2000p' big_file.txt  > 1801to2000.txt
sed -n '2001,2200p' big_file.txt  > 2001to2200.txt
sed -n '2201,2400p' big_file.txt  > 2201to2400.txt
sed -n '2401,2600p' big_file.txt  > 2401to2600.txt
sed -n '2601,2800p' big_file.txt  > 2601to2800.txt
sed -n '2801,3000p' big_file.txt  > 2801to3000.txt
sed -n '3001,3200p' big_file.txt  > 3001to3200.txt
sed -n '3201,3400p' big_file.txt  > 3201to3400.txt
sed -n '3401,3600p' big_file.txt  > 3401to3600.txt
sed -n '3601,3800p' big_file.txt  > 3601to3800.txt
sed -n '3801,4000p' big_file.txt  > 3801to4000.txt

What I have tried:

j=0
for ((i=1;i<=3801;i=$i+200))
do
    #echo $m,$n
    j=$j + 200
    sed -n '$i,$j p' big_file.txt  > $ito$j.txt 
done

Please help me in doing this. Please provide explanation too.

  • 1
    This looks like a job for split (or even awk) rather than sed – steeldriver Jun 1 '18 at 15:40
  • 1
    Shouldn't that be j=$(( $i + 200 ))? and > "${i}to${j}.txt" – Dani_l Jun 1 '18 at 15:41
  • 3
    split -l 200 big_file.txt and possibly format the suffixes to something usable. – Dani_l Jun 1 '18 at 15:47
  • 1
    @Dani_l, you should make that an answer. – glenn jackman Jun 1 '18 at 15:50
  • 1
    I removed the bash and regular-expression tags, as neither are involved here. – Jeff Schaller Jun 1 '18 at 16:14
1

Close.

In j=$j + 200, you need to explicitly invoke arithmetic expansion, i.e. j=$(( j + 200 )).

And in sed -n '$i,$j p' big_file.txt > $ito$j.txt, 1) you need to use double-quotes instead of single quotes in the argument to sed, since otherwise the $ is taken literally and the variables will not be expanded; 2) you need braces around the i, since $ito would be a valid variable expansion.

Also, in arithmetic contexts you don't need to (and probably shouldn't) use the $ in front of the variable name.

So, I'd rewrite that as:

j=0
for (( i = 1 ; i <= 3801 ; i = i + 200)); do
    j=$((j + 200))
    sed -n "$i,$j p" big_file.txt > "${i}to$j.txt"
done

Like others have commented, there's probably a better way to do that, since now you walk the file in full once for each piece you split to.

  • I accept this solution because my code resembles this. – Dipankar Nalui Jun 4 '18 at 11:00
2

I would do this with a single awk program, instead of calling sed N times

awk '
    BEGIN {incr = 200; i=1-incr; j=0}
    NR % incr == 1 {
        i += incr
        j += incr
        close(out)
        out = i "to" j ".txt"
    }
    {print > out}
' big_file.txt

This should be quicker, as you only have to process the big file once, instead of 4000÷200=20 times.

  • Could you please explain this? – Dipankar Nalui Jun 4 '18 at 11:08
  • NR is the awk variable that holds the current record number (here, a record = a line). For lines 1, 201, 401, ..., we set "out" to hold the desired output file name, closing the previous output file. (Awk string concatenation is a bit odd looking, just put the component strings side by side, there's no special operator: out = i "to" j ".txt") Then, for every line, print the line to the output file. The BEGIN block initializes the variables. – glenn jackman Jun 4 '18 at 11:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.