I have a big CSV file.
I would like to view my file using
less or some command like it which doesn't have to read the whole file at once to show me part of it.
Is there a command out there which can show me my file in comma-aligned columns?
I'm not sure if that is enough for you, but you could make use of
column program and read the selected parts of the file using
tail like this:
head -n 300 myfile.csv | tail -n 100 | column -ts ',' | less head -n-300 myfile.csv | head -n 100 | column -ts ',' | less
You could wrap it up in some script to view different parts of the file at a time (probably without
less then). Apart from that, I'm afraid it would be a problem to use only the GNU tools for the job.
less but when you want to see the CSV data column-aligned, pipe the current page through the
column -ts , command:
| <m> shell-command <m> represents any mark letter. Pipes a section of the input file to the given shell command. The section of the file to be piped is between the first line on the current screen and the position marked by the letter. may also be ^ or $ to indi- cate beginning or end of file respectively. If is . or new- line, the current screen is piped. m Followed by any lowercase letter, marks the current position with that letter.
|.column -ts , in
I've had good experience with
tabview, though it does not always behave nicely with large files. However, in conjunction with
head it's pretty decent.
This isn't really a unixy question, but in any case, I'd recommend using something different from
less for viewing csv files. It isn't really the right tool. Try something like
R, which has good support for looking at and if necessary, working with CSV files. E.g. to read 5 rows of a csv file, do
$ R > read.csv("pheno.csv",nrows=5) faid expid pid mid sex pheno 1 1420 NA12003 0 0 1 0 2 1420 NA12004 0 0 2 0 3 1420 NA10838 9 10 1 0 4 1420 NA12005 0 0 1 0 5 1420 NA12006 0 0 2 0
for help. See also
for writing to a file etc.
EDIT: I happened to have a csv file which is 1.1G and is 934991 lines long.
$ time Rscript -e 'read.csv("GenomeWideSNP_6.na29.annot.csv", skip=500000, nrows=5, header=FALSE)'
This skips 500,000 lines and reads 5 lines. R takes 1 minute to return this, and tops out at 620M usage (!) It looks like it may be reading the skipped lines into memory, though that makes no sense.
The upshot: I'm not really an R fan, but for handling small to medium data sets, especially if you want to do statistical analyses on it, you could do worse than R. An alternative is Python and some csv processing library.