I have a CSV that is 40 000 lines long. What is the simplest tool to read from line x to line y?

I am sure there are more professional ways to do it but I would just like a bash script that works in the form of

 readFile --from 10 --to 20

so that I can process ten lines, then call it again and bring in ten more lines. I was considering just using an AWK script but can you do it even more simply with something like cat, grep, tail or head?


while getopts "f:t:" option; do
  case "$option" in 
    f) from=$OPTARG ;;
    t) to=$OPTARG ;;
if [ -z "$from" ] || [ -z "$to" ]; then
  echo "must give both -f and -t" >&2
  exit 1

awk -v "from=$from" -v "to=$to" 'from <= NR && NR <= to' filename

# or:
sed -n "$from,$to p; $to q" filename

# or:    
while read line; do
  i=$(( i + 1 ))
  if [ $i -ge $from ] && [ $i -le $to ]; then
    printf "%s\n" "$line"
  [ $i -eq $to ] && break
done < filename
  • Thank you that was awesome I hope I can at least get to the point where I am that good at the simple things in life. That last one is completely beyond me at the moment. – Arcabard Sep 2 '11 at 12:42
  • I should note that this takes as arguments -f and -t instead of the requested --from and --to – glenn jackman Sep 2 '11 at 14:59
  • It seems this script is doing the same thing three times :) – Bernhard Jan 1 '14 at 10:38
  • and you get to pick one of them. That's what "or" means – glenn jackman Jan 1 '14 at 13:17

Extracting lines 10 to 20 of a file is a simple combination of head and tail:

tail -n +11 | head -n 10 | mycommand

This command skips 10 lines and processes the next 10. If you want to process all the lines in the file in sequence, but by groups of 10, there are better ways. You can read 10 lines with head repeatedly. Note the use of chunk=$(…; echo a) followed by stripping away the a, to work around the suppression of final newlines in command substitutions. This approach has the advantage of working even if the input is a pipe (which you can't rewind), and it's faster for a regular file too. Warning, untested code, typed directly in the browser.

while chunk=$(head -n 10; echo a); chunk=${chunk#a}; [ -n "$chunk" ]; do
  printf %s "$chunk" | mycommand
done <filename

Alternatively, you can make awk do the job. Again, untested.

awk '
    {chunk = chunk $0 RS}
    NR % 10 {print chunk | "mycommand"; close("mycommand"); chunk="" }
    END {if (chunk != "") {print chunk | "mycommand"; close("mycommand"); chunk="" }}
' <filename
  • I love the magic of combining different utilities ty. – Arcabard Sep 3 '11 at 0:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.