Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a big CSV file.

I would like to view my file using less or some command like it which doesn't have to read the whole file at once to show me part of it.

Is there a command out there which can show me my file in comma-aligned columns?

share|improve this question
1  
I notice a problem here: you would like to have comma-aligned view, but at the same time wish that the program does not read the whole file... But to show the data correctly aligned, the program would have to read through the file to determine the maximum width of a column! –  rozcietrzewiacz Aug 16 '11 at 20:21
    
I thought of that, but I don't see that as being an impediment. Only a fixed part of a file is viewable at a time and I don't so much care if the columns adjust their size dynamically. Either that, or perhaps whatever this magic program is, you press a key and it realigns. –  Richard Aug 16 '11 at 20:25
    
That is, @rozcietrzewiacz, that, for viewing purposes, only the local maxima are important. Especially if you remember the largest local maximum you've seen yet, which will tend towards the global maximum. –  Richard Aug 16 '11 at 20:26
1  
Cross posting? askubuntu.com/questions/57369/… –  enzotib Aug 17 '11 at 4:57
    
Indeed, @enzotib. I'd thought that this wasn't appropriate for stackoverflow originally, but other people pointing back there seemed to implicitly imply that it was. –  Richard Aug 17 '11 at 14:18

3 Answers 3

I'm not sure if that is enough for you, but you could make use of column program and read the selected parts of the file using head and/or tail like this:

 head -n 300 myfile.csv | tail -n 100 | column -ts ',' | less
 head -n-300 myfile.csv | head -n 100 | column -ts ',' | less

You could wrap it up in some script to view different parts of the file at a time (probably without less then). Apart from that, I'm afraid it would be a problem to use only the GNU tools for the job.

share|improve this answer
    
In my experience, less doesn't try to read the whole file by default, so simply colum -ts ',' myfile.csv | less should work. –  Klox Aug 17 '11 at 1:52
1  
less obviously doesn't, but column does - so it's not a good solution after all... –  rozcietrzewiacz Aug 17 '11 at 6:07
1  
See also 'man cut' to view only specific columns, e.g. cut -d',' -f1,4-6 would return only columns 1,4,5 and 6 –  symcbean Aug 17 '11 at 10:03

Use less but when you want to see the CSV data column-aligned, pipe the current page through the column -ts , command:

       | <m> shell-command
              <m>  represents  any  mark letter.  Pipes a section of the input
              file to the given shell command.  The section of the file to  be
              piped  is  between  the first line on the current screen and the
              position marked by the letter.   may also be ^ or $ to  indi-
              cate beginning or end of file respectively.  If  is . or new-
              line, the current screen is piped.

       m      Followed by any lowercase letter,  marks  the  current  position
              with that letter.

so use |.column -ts , in less

share|improve this answer

This isn't really a unixy question, but in any case, I'd recommend using something different from less for viewing csv files. It isn't really the right tool. Try something like R, which has good support for looking at and if necessary, working with CSV files. E.g. to read 5 rows of a csv file, do

$ R
> read.csv("pheno.csv",nrows=5)
  faid   expid pid mid sex pheno
1 1420 NA12003   0   0   1     0
2 1420 NA12004   0   0   2     0
3 1420 NA10838   9  10   1     0
4 1420 NA12005   0   0   1     0
5 1420 NA12006   0   0   2     0

See

> help(read.csv)

for help. See also

> help(write.csv)

for writing to a file etc.

EDIT: I happened to have a csv file which is 1.1G and is 934991 lines long.

$ time Rscript -e 'read.csv("GenomeWideSNP_6.na29.annot.csv", skip=500000, nrows=5, header=FALSE)'

This skips 500,000 lines and reads 5 lines. R takes 1 minute to return this, and tops out at 620M usage (!) It looks like it may be reading the skipped lines into memory, though that makes no sense.

The upshot: I'm not really an R fan, but for handling small to medium data sets, especially if you want to do statistical analyses on it, you could do worse than R. An alternative is Python and some csv processing library.

share|improve this answer
    
R may be wonderful but is it the right tool for simply viewing the middle 20 rows of a 100,000 row CSV file? The question sought a program that didn't first load the whole file into memory. –  RedGrittyBrick Aug 18 '11 at 11:00
    
@RedGrittyBrick: Reasonable point. See added example. R's memory usage in this case is nuts. For this reason, if memory usage was a concern, Python or Perl might be better. They would also probably be faster. –  Faheem Mitha Aug 18 '11 at 15:09

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.