Parsing giant files into Ruby

Question

I have a CSV that is 40 000 lines long what is the simplest tool to read from line x to line y. I am sure there are more professional ways to do it but I would just like a bash script that works in the form of readFile --from 10 --to 20 so that I can process ten lines call it again and bring in ten more lines. I was considering just using an AWK script but can you do it even more simply with something like cat grep tail or head?

Answer 1

#!/bin/sh

while getopts "f:t:" option; do
  case "$option" in 
    f) from=$OPTARG ;;
    t) to=$OPTARG ;;
  esac
done
if [ -z "$from" ] || [ -z "$to" ]; then
  echo "must give both -f and -t" >&2
  exit 1
fi

awk -v "from=$from" -v "to=$to" 'from <= NR && NR <= to' filename

# or:
sed -n "$from,$to p; $to q" filename

# or:    
i=0
while read line; do
  i=$(( i + 1 ))
  if [ $i -ge $from ] && [ $i -le $to ]; then
    printf "%s\n" "$line"
  fi
  [ $i -eq $to ] && break
done < filename

Answer 2

Extracting lines 10 to 20 of a file is a simple combination of head and tail:

tail -n +11 | head -n 10 | mycommand

This command skips 10 lines and processes the next 10. If you want to process all the lines in the file in sequence, but by groups of 10, there are better ways. You can read 10 lines with head repeatedly. Note the use of chunk=$(…; echo a) followed by stripping away the a, to work around the suppression of final newlines in command substitutions. This approach has the advantage of working even if the input is a pipe (which you can't rewind), and it's faster for a regular file too. Warning, untested code, typed directly in the browser.

while chunk=$(head -n 10; echo a); chunk=${chunk#a}; [ -n "$chunk" ]; do
  printf %s "$chunk" | mycommand
done <filename

Alternatively, you can make awk do the job. Again, untested.

awk '
    {chunk = chunk $0 RS}
    NR % 10 {print chunk | "mycommand"; close("mycommand"); chunk="" }
    END {if (chunk != "") {print chunk | "mycommand"; close("mycommand"); chunk="" }}
' <filename