0

I'm trying to split a csv file with too many rows into chunks of user defined no. of rows, using the following script using awk but its not generating any output, the file is created but its blank. Pls Can anyone point out what's missing...

# !/bin/bash
File=filename.csv
count=`awk 'END {print NR}' $File`
i=2
j=5000
k=$j
while [ $j -le $count ]
  do
    awk 'NR==1 {print $0}' $File > output"$i".csv
    awk 'NR==$i, NR==$j {print $0}' $File > output"$i".csv
    i=`expr $j + 1`
    j=`expr $i + $k`
    if [ $j -gt $count ]
      then
        j=$count
        k=`expr 0 - 1`
    elif [ $j -eq $count]
      then
        break
    fi
done
3
  • PS. I'm new to Linux, and I just wanna figure out what's wrong in the script, I tried finding similar questions but I wasn't able to find any, Please help, Ik we can use sed command but I dont know how to implement it, thanks in advance Jan 27, 2020 at 16:34
  • 2
    Are you fixated on using bash & awk to do this? There's a dedicated tool called "split" that does this job. (As long as your CSV rows are line-delimited)
    – Jeff Schaller
    Jan 27, 2020 at 16:42
  • 1
    Agree using split would be preferred. Also awk can do all the arithmetic better, and switch between files internally, so you only need read the input file once. However, to answer your actual question. You can set awk variables on the command line like awk -v i="$i" -v j="$j" and just use i and j inside awk. Jan 27, 2020 at 17:39

1 Answer 1

1

If you really want to use awk then how about

awk -v c=5000 '
    BEGIN{getline; head=$0; f = -1}
    {if (f != int((NR-2)/c)) {f+=1; fl=sprintf("%03d.csv", f); print head > fl}; print $0 > fl}
    ' filename.csv

As for your script.....

#!/bin/bash
File=filename.csv
count=$(awk 'END {print NR}' $File) # syntax
i=2
j=5000
k=$j
while [ $j -le $count ]
  do
    awk 'NR==1 {print $0}' $File > output"$i".csv
    # you cannot pass shell variables directly into awk, use -v
    # and append >> don't overwrite >
    awk -v st=$i -v fi=$j 'NR==st, NR==fi {print $0}' $File >> output"$i".csv 
    i=`expr $j + 1`
    j=`expr $i + $k`
    if [ $j -gt $count ]
      then
        j=$count
        k=-1
    elif [ $j -eq $count ] # needed a space before the ]
      then
        break
    fi
done
1
  • Thanks A Lot @bu5hman it worked Jan 28, 2020 at 10:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .