Use `less` to view comma-aligned data

Question

I have a big CSV file.

I would like to view my file using less or some command like it which doesn't have to read the whole file at once to show me part of it.

Is there a command out there which can show me my file in comma-aligned columns?

Answer 1

I'm not sure if that is enough for you, but you could make use of column program and read the selected parts of the file using head and/or tail like this:

 head -n 300 myfile.csv | tail -n 100 | column -ts ',' | less
 head -n-300 myfile.csv | head -n 100 | column -ts ',' | less

You could wrap it up in some script to view different parts of the file at a time (probably without less then). Apart from that, I'm afraid it would be a problem to use only the GNU tools for the job.

Answer 2

Use less but when you want to see the CSV data column-aligned, pipe the current page through the column -ts , command:

       | <m> shell-command
              <m>  represents  any  mark letter.  Pipes a section of the input
              file to the given shell command.  The section of the file to  be
              piped  is  between  the first line on the current screen and the
              position marked by the letter.   may also be ^ or $ to  indi-
              cate beginning or end of file respectively.  If  is . or new-
              line, the current screen is piped.

       m      Followed by any lowercase letter,  marks  the  current  position
              with that letter.

so use |.column -ts , in less

Answer 3

This isn't really a unixy question, but in any case, I'd recommend using something different from less for viewing csv files. It isn't really the right tool. Try something like R, which has good support for looking at and if necessary, working with CSV files. E.g. to read 5 rows of a csv file, do

$ R
> read.csv("pheno.csv",nrows=5)
  faid   expid pid mid sex pheno
1 1420 NA12003   0   0   1     0
2 1420 NA12004   0   0   2     0
3 1420 NA10838   9  10   1     0
4 1420 NA12005   0   0   1     0
5 1420 NA12006   0   0   2     0

See

> help(read.csv)

for help. See also

> help(write.csv)

for writing to a file etc.

EDIT: I happened to have a csv file which is 1.1G and is 934991 lines long.

$ time Rscript -e 'read.csv("GenomeWideSNP_6.na29.annot.csv", skip=500000, nrows=5, header=FALSE)'

This skips 500,000 lines and reads 5 lines. R takes 1 minute to return this, and tops out at 620M usage (!) It looks like it may be reading the skipped lines into memory, though that makes no sense.

The upshot: I'm not really an R fan, but for handling small to medium data sets, especially if you want to do statistical analyses on it, you could do worse than R. An alternative is Python and some csv processing library.