Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a CSV that is 40 000 lines long. What is the simplest tool to read from line x to line y?

I am sure there are more professional ways to do it but I would just like a bash script that works in the form of

 readFile --from 10 --to 20

so that I can process ten lines, then call it again and bring in ten more lines. I was considering just using an AWK script but can you do it even more simply with something like cat, grep, tail or head?

share|improve this question

2 Answers 2

up vote 1 down vote accepted
#!/bin/sh

while getopts "f:t:" option; do
  case "$option" in 
    f) from=$OPTARG ;;
    t) to=$OPTARG ;;
  esac
done
if [ -z "$from" ] || [ -z "$to" ]; then
  echo "must give both -f and -t" >&2
  exit 1
fi

awk -v "from=$from" -v "to=$to" 'from <= NR && NR <= to' filename

# or:
sed -n "$from,$to p; $to q" filename

# or:    
i=0
while read line; do
  i=$(( i + 1 ))
  if [ $i -ge $from ] && [ $i -le $to ]; then
    printf "%s\n" "$line"
  fi
  [ $i -eq $to ] && break
done < filename
share|improve this answer
    
Thank you that was awesome I hope I can at least get to the point where I am that good at the simple things in life. That last one is completely beyond me at the moment. –  James Andino Sep 2 '11 at 12:42
    
I should note that this takes as arguments -f and -t instead of the requested --from and --to –  glenn jackman Sep 2 '11 at 14:59
    
It seems this script is doing the same thing three times :) –  Bernhard Jan 1 at 10:38
    
and you get to pick one of them. That's what "or" means –  glenn jackman Jan 1 at 13:17

Extracting lines 10 to 20 of a file is a simple combination of head and tail:

tail -n +11 | head -n 10 | mycommand

This command skips 10 lines and processes the next 10. If you want to process all the lines in the file in sequence, but by groups of 10, there are better ways. You can read 10 lines with head repeatedly. Note the use of chunk=$(…; echo a) followed by stripping away the a, to work around the suppression of final newlines in command substitutions. This approach has the advantage of working even if the input is a pipe (which you can't rewind), and it's faster for a regular file too. Warning, untested code, typed directly in the browser.

while chunk=$(head -n 10; echo a); chunk=${chunk#a}; [ -n "$chunk" ]; do
  printf %s "$chunk" | mycommand
done <filename

Alternatively, you can make awk do the job. Again, untested.

awk '
    {chunk = chunk $0 RS}
    NR % 10 {print chunk | "mycommand"; close("mycommand"); chunk="" }
    END {if (chunk != "") {print chunk | "mycommand"; close("mycommand"); chunk="" }}
' <filename
share|improve this answer
    
I love the magic of combining different utilities ty. –  James Andino Sep 3 '11 at 0:26

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.