Comma-separated values: files containing data arrange in a table, with commas to separate columns

learn more… | top users | synonyms

18
votes
9answers
6k views

Is there a robust command line tool for processing csv files?

I work with CSV files and sometimes need to quickly check the contents of a row or column from the command line. In many cases cut, head, tail, and friends will do the job; however, cut cannot easily ...
17
votes
4answers
32k views

Convert a .xlsx (MS Excel) file to .csv on command line with semicolon separated fields

I realize that this is not an entirely unix/linux related question. But since this is something I'll do on linux, I hope someone has an answer. I have an online excel file (.xlsx) which gets updated ...
17
votes
5answers
4k views

Command line friendly spreadsheets

Does such a thing exist? Text-based spreadsheets that display well in a CLI environment. I'm aware that I could cat foobar.csvand do as I please, but it isn't particularly practical or attractive. I ...
14
votes
4answers
19k views

How to manipulate a CSV file with sed or awk?

How can I do the following to a CSV file using sed or awk? Delete a column Duplicate a column Move a column I have a big table with over 200 rows, and I'm not that familiar with sed.
11
votes
8answers
725 views

How do I get the character count of words in a particular column?

I have a CSV file like this: abd,123,egypt,78 cde,456,england,45 How can I get the character count of only the 3rd column words? I can't figure out how to get wc to do this.
10
votes
2answers
2k views

sort inplace - like sed --in-place - exists?

Am I blind or is there no option like --in-place for sort? In order to save results to the input file, sed uses -i (--in-place). Redirecting the output of sort to the input file sort < f > f ...
8
votes
5answers
8k views

Remove comma between the quotes only in a comma delimited file

I have a input file delimited with commas (,). There are some fields enclosed in double quotes that are having a comma in them. Here is the sample row 123,"ABC, DEV 23",345,534.202,NAME I need to ...
8
votes
3answers
2k views

command to layout tab separated list nicely

Sometimes, I'm getting as an input tab separated list, which is not quite aligned, for instance var1 var2 var3 var_with_long_name_which_ruins_alignment var2 var3 Is there an easy way to render ...
6
votes
4answers
669 views

uniq a csv file ignoring a column, awk maybe?

Given this file (annotations are not part of file, but form part of explanation)... x,a,001,b,c,d,y x,a,002,b,c,e,yy x,bb,003,b,d,e,y x,c,004,b,d,e,y x,c,005,b,d,e,y # nb - dupe of row 4 ...
6
votes
3answers
372 views

Remove only the commas present within the double quotes

In a text file, I want to remove , (commas) and also the " (quotes) (only if the double quotes contains numbers seperated by commas). 56,72,"12,34,54",x,y,"foo,a,b,bar" Expected ouput ...
6
votes
3answers
273 views

adding an empty first line inside csv file

I input data into a csv by 'echo'-ing some values into another csv. I tried echo "" and in the next line echo "values" but i get intermittent empty lines.. I need empty line only at the start....
6
votes
3answers
809 views

Use `less` to view comma-aligned data

I have a big CSV file. I would like to view my file using less or some command like it which doesn't have to read the whole file at once to show me part of it. Is there a command out there which can ...
5
votes
3answers
137 views

Robustly segmenting a csv file containing noisy data

Say I have a csv file with these columns: timestamp,cpu,memory An example of the shape of the actual data would be this chart: Having the data visualized in this way, it's easy for the human eye ...
5
votes
5answers
3k views

Parsing a delimited text file in bash as command arguments

I have a text file split up like so: field1,field2,field3 xield1,xield2,xield3 dield1,dield2,dield3 gield1,gield2,gield3 Each of these columns will be a parameter to a program, and I would like ...
5
votes
3answers
5k views

Search for values in 2 columns from CSV

I have a csv files with say 20 odd columns. I need to get the 14th and 15th column values where value in 14th column is say "VALUE1" and value in 15th column is "VALUE2". My condition gets satisfied ...
4
votes
4answers
1k views

A unix command to truncate each line of a file

I have a CSV file and I want to truncate it from the third semicolon. For example, if I have this file: 1;foo;bar;baz;x;y;z 2;foo;bar;baz;x;y;z 3;foo;bar;baz;x;y;z I want to get the following ...
4
votes
5answers
4k views

extract the 4th column from a csv file using unix command

I have a csv file formatted as below. "col1","col2","col3","col4" "1","text1","<p>big html text</p> ","4th column" "2","text2","<p>big2 html2 text2</p> ...
4
votes
5answers
4k views

How can I convert tab delimited data to comma delimited data?

I'm requesting a list of ec2 snapshots via amazon's ec2 command line tool: ec2-describe-snapshots -H --hide-tags > snapshots.csv The data looks something like this: SnapshotId VolumeId ...
4
votes
2answers
64 views

awk pairwise differences

I run into this problem a lot with data, if I have some integers, I frequently want to look at differences between adjacent integers, I usually solve this in Ruby or Python, but I think such a thing ...
4
votes
2answers
2k views

Is there a command line utility to transpose a csv-file?

Given a file like so First, Last, Age Cory, Klein, 27 John Jacob, Smith, 30 Is there a command line utility to transpose the contents so the output appears like so First, Cory, John Jacob Last, ...
4
votes
3answers
3k views

Filter a .CSV file based on the 5th column values of a file and print those records into a new file

I have a .CSV file with the below format: "column 1","column 2","column 3","column 4","column 5","column 6","column 7","column 8","column 9","column 10 "12310","42324564756","a simple string with a , ...
4
votes
1answer
693 views

How to display TSV (csv) in console, when empty cells are missed by: `column -t -s $'\t' `

I have file with columns spearated with tab. I have file when some rows have empty cells (on begining, in middle). In such cases column -t -s $'\t' simply fails: Input: $ echo -e ...
3
votes
4answers
323 views

How to Prefix a column values with an apostrophe ( ' )?

I have a CSV file with multiple columns and 1000's of records, I need to prefix all the values of one of the columns (lets say 2nd column) with an apostrophe ' except in the first line or header line, ...
3
votes
3answers
584 views

Using CSV line as command parameters

I have a CSV file like: Name,Age,Address Daniel Dvorkin,28,Some Address St. 1234 (... N ...) Foo Bar,90,Other Address Av. 3210 And I have a command that take this parameters: ./mycommand ...
3
votes
2answers
364 views

need shell script to grock a csv

Need some pointing in right direction on script to fetch and regex or sed. Site24x7 provides a URL with a CSV list of their source IP's used for monitoring. (they also provide other formats, CSV ...
3
votes
1answer
34 views

update line based on previous line

I have a issue since I'm not familiar with awk. I have csv file generated from sar -d output converted to csv style: 12:33:41 unix,restarts 12:35:00,lofi4096,0,0.0,0,0,0.0,0.0 ...
3
votes
1answer
120 views

Wrapping long cells in a tsv to keep them in same column

Formatting a TSV with long cells with column makes its readability very low. It simply breaks lines like this: $ python3 -c 'for i in (1,2,3): print(((str(i)*50)+"\t")*3)' | column -s $'\t' -t ...
3
votes
1answer
360 views

An efficient filter for .sql files with just “INSERT INTO”

Several times I've found myself working on data that has been exported to .sql files that I want to process rather than import into a database. I'm only talking about "pure data" sql files. Ones that ...
3
votes
0answers
271 views

Catdoc xls2csv converting only one sheet

I have Catdoc's xls2csv installed on Debian/Squeeze. It used to work beautifully when I copied files to a folder like /var/www/xyz, converting all sheets into .csv format, separated by ^L. But now, ...
2
votes
3answers
3k views

Extracting column from comma separated text

I have a long comma-separated delimited file with 20K lines. Here's a sample: "","id","number1","number2","number3","number4","number5","number6","number7" ...
2
votes
4answers
457 views

Format a Date field of a .CSV file with multiple commas in a string field

I have a .CSV file (file.csv) whose data are all enclosed in double quotes. Sample format of the file is as below: column1,column2,column3,column4,column5,column6, column7, Column8, Column9, Column10 ...
2
votes
4answers
938 views

gawk join two TSV by columns (a'la sql join)

How to join, to tsv files, examples: a.tsv c 7 r z d 6 s w f 1 f f b 8 p y a 9 q x b.tsv a q a c r ccc b p bb 0 0 0 d s dddd Here I'd like to ...
2
votes
3answers
761 views

Replacing the values in one file with the values in another file in bash

I am having a csv file naming List.csv in following format: Location,IP Address,Host Name,Domain,Domain Name, User Name,Manufacturer,Model,System Type, Serial Number, Operating System,RAM ...
2
votes
4answers
6k views

How can I find and replace with a new line?

I have a CSV delimited by commas and I want to delimit it by newlines instead. Input: a, b, c Output: a b c I've written Java parsers that do this stuff, but couldn't this be done with vim or ...
2
votes
3answers
517 views

Select lines from text file which have ids listed in another file

I use a lot of grep awk sort in my unix shell to work with medium-sized (around 10M-100M lines) tab-separated column text files. In this respect unix shell is my spreadsheet. But I have one huge ...
2
votes
6answers
4k views

How to swap columns in such a file?

I have a text file, each line is stored like this : "Video or movie" "parent" "Media or entertainment" "1" "1" "1" "0" "0" I want to swap the columns 3 with 2, i.e. "Video or movie" ...
2
votes
2answers
159 views

Create an ASCII art table from tabular data

Given perhaps comma- or tab-delimited input, I'd like to present a series of appropriately padded columns to stdout, so I can easily scan columnar information which would otherwise present rather ...
2
votes
3answers
126 views

Extract data from a file and place in different files based on1 column value

We will generate a csv file with below values yp1234,577,1,3 yp5678,577,3,5 yp9012,132,8,9 I need to extract data and create files based on second column. If it's 577 then the whole line has to be ...
2
votes
3answers
166 views

I need data from a csv file which is not present in the second csv file

For example: I have a CSV file (first file ) with below data (vertically): 1 2 3 4 5 6 Second CSV with below data (vertically): 5 6 7 8 I want my resultant file (action needs to be performed ...
2
votes
2answers
75 views

How can I show the number of positions lines have moved in diff output?

top1 1,facebook.com 2,google.com 3,youtube.com 4,yahoo.com 5,wikipedia.org 6,baidu.com 7,live.com 8,amazon.com 9,qq.com 10,twitter.com top2 1,facebook.com 2,google.com 3,youtube.com 4,yahoo.com ...
2
votes
2answers
74 views

CSV file + set CSV parameters and values in bash script

I have the following CSV file: Alabama,Alaska,Arizona,Arkansas,California,Colorado,Connecticut,Delaware,Florida,Georgia,Hawaii,Idaho,Illinois,Indiana,Iowa 1000,"1 0 0 1",1002,1002,1003,1004,1005,"1 0 ...
2
votes
2answers
2k views

Convert csv to HTML table

I have a Medical.csv file with rows of following format, field: 'participation.type', displayName: 'program_type', type: 'String',path:'participation' field: 'participation.program', ...
2
votes
3answers
570 views

How can I turn ugly output into pretty, useful information?

How can I turn this ugly output into pretty, useful data? The output: /* ---------- TA#box#AbC_p ---------- */ insert_job: TA#box#AbC_p job_type: a #owner: bob permission: gx date_conditions: 1 ...
2
votes
2answers
184 views

Print some fields of /etc/passwd

I have a question concerning the usage of wild cards in Linux. Here I have some lines from a long list of lines in the /etc/passwd -file. root:x:0:0:root:/root:/bin/bash ...
2
votes
1answer
192 views

solaris awk problems with date and if statements

I have daily files that come in via FTP with incorrect dates in the first column of the file. I have figured out how to deduct one day to derive the correct date and print this to a new file. The main ...
2
votes
1answer
131 views

How to extract single value from single line tuples?

I have a dump file of a database in the following form ('value1','value2',value3','value4'),('value1','value2',value3','value4'),... I want first to have each tuple in a single line and next i ...
2
votes
2answers
697 views

How to remove an apostrophe ( ' ) from couple of columns of a .CSV file?

I have a .CSV file with 7 fields, and the 3rd and 4th columns of the file has a number starting with an apostrophe ( ' ). Please see the example below. col0,col1,col2,col3,col4,col5,col6, ...
2
votes
1answer
122 views

Read only new parts of a file?

I have a large CSV file, that grows by some entries every day (created by an external software which writes always to this one file, no splitting possible). For performance reasons and further ...
2
votes
1answer
2k views

Swap two columns in a CSV using SED

I have a CSV file that contains 10 different fields (, is the deliminator). Example data: student-id,last,first,hwk1,hwk2,hwk3,exam1,hwk4,hwk5,exam2 pts-avail,,,100,150,100,200,150,100,300 ...
2
votes
1answer
63 views

merge csv files by first column

I have 3 csv files like this. csv 1: 1,aaaa,bbb,2014-04-01 2,qwe,rty,2014-04-03 3,zxc,cvb,2014-04-05 csv 2: 2,j,k,2014-04-01 3,a,s,2014-04-04 5,g,h,2014-04-08 csv 3: 2,a,s,d,f,g,2014-04-01 ...