Files containing data arrange in a table, often with commas (hence Comma Separated Values), to separate columns. Rows are separated by newlines (but not all newlines are row separators as fields can be quoted to contain the separator newlines. Use this tag for full-fledged CSV data not the simpler ...

learn more… | top users | synonyms

1
vote
1answer
32 views

Calculate hits within a timespan from log file [closed]

I need to use UNIX utilities that implement out-of-core divide-and-conquer algorithms so to decrease the amount of virtual memory used (cap of 1 mb). I have a large data set that is an apache log file....
0
votes
1answer
2k views

How to create CSV file without unnecessary spaces

I use the xls2csv binary in order to translate XLS doc to CSV in my linux red-hat machine, example: ( from man page ) xls2csv -x "1252spreadsheet.xls" -b WINDOWS-1252 -c "ut8csvfile.csv" -a UTF-8 ...
4
votes
2answers
40 views

Is it possible to print an iterated range of arguments via cli from a csv? (see example)

I have hundreds of logs spread across a few thousand VM's and I'm trying to create a few scripts to quickly scan logs and I have figured out the majority of the presentation and wanted to know if ...
1
vote
4answers
853 views

How to extract column name (header) from a CSV file which contains the max value in a row?

I am trying to extract the column name with the maximum value in each row using bash script i.e., the column header value or the value from the same column in the first row. I am using the following ...
16
votes
7answers
25k views

Remove comma between the quotes only in a comma delimited file

I have a input file delimited with commas (,). There are some fields enclosed in double quotes that are having a comma in them. Here is the sample row 123,"ABC, DEV 23",345,534.202,NAME I need to ...
24
votes
2answers
1k views

How to merge two text files into one file without using cat or sed [duplicate]

I have two files File1: a b c File2: 1 2 3 now I need to combine them to one csv file a;1 b;2 c;3 As the files are really huge, I would rather not use cat and sed to process the second file. ...
0
votes
3answers
37 views

Combining several text files into a single csv [closed]

Input files: File: Article1.txt: paragraph1 It is a long established fact that a reader will...... paragraph2 It is a long established fact that a reader will...... paragraph3 It is a long ...
0
votes
3answers
61 views

How can I retrieve and compare two values from a file? [closed]

How can I retrieve and compare two values from the first two lines of a file? I have the following text: 05-24-2016, 2:59:32,0,0 05-24-2016, 2:59:37,0,0 05-24-2016, 2:59:42,0,0 05-24-2016, 2:59:47,0,...
3
votes
2answers
105 views

Question about concatenation instructions

I have a semicolon-delimited csv file and a Korn shell script that do some elaboration. I'm searching to understand what these two instructions do in Korn shell: num_rec=`cat test.csv |wc -l|tr -d " "...
1
vote
2answers
72 views

Removing spaces from the fields in a semicolon-delimited file using shell script

I have a csv file like this: 03139; 5; IT1234978208; 20150930 ; CTZ 13/31.12.15; 03137; 6; IT1234978206; 20151015 ; CTZ 13/31.11.18; .... .... 03134; 8; IT1234974406; ...
0
votes
1answer
62 views

Matching 3 or more words from fields in separate csv files

I have two csv files: This is csv1: 11, The Sun Still Shines in Reading, 64312, 464566 13, You and Me Together Again London, 564564, 131355 12, What's the Story Now Your Gone, 4545646, 1124545 17, ...
0
votes
1answer
60 views

Match words in array line by line to a seperate file

Here is a sample of my input data: $9.99,Titan the Power,www.example.com,565654 $15.99,Conan The Barbarian,www.sure.com,565438 $1.99,Julia Cesar,www.bfg.com,69722 I have written this code so ...
1
vote
2answers
28 views

Can python extract org-mode tables from org documents?

I'm trying to write some python code that reads specified tables in org mode documents. For example, I have a file ~/foo.org $ cat ~/foo.org #+Title: Example Org Document * Section One Proin quam ...
2
votes
3answers
46 views

Bash read with delimiter past newlines

I have a csv for which I have to extract a specific field. The csv is pipe-delimited (|), with double quotes (") to protect text fields and (this is the catch) newlines in some of the text. eg. "aaa"...
3
votes
1answer
37 views

If there are more than x number of pipes in a csv line, then delete the 2nd instance

I have a csv file which is supposed to contain 4 columns of data which include a product number, a title, a url and a price. Each column is separated by a | delimiter (this has to be maintained, there ...
2
votes
3answers
47 views

linux + ssh + perform two tasks in one ssh and print to CSV

the following command print the two values in one ssh shut ssh test01 "hostname && cat /etc/sysconfig/rhn/rhnsd " test01 INTERVAL=240 but how to change the syntax in order to print the ...
1
vote
3answers
2k views

How to do a fast convert of a .xls (MS Excel) file to .csv

How can I do a fast convert of excel to csv? I have tried with php, perl, JavaScript. but it takes too much time. For example, for a 5MB file it will take 3-5 minutes.
3
votes
1answer
41 views

Copy csv data while simultaneously filling serial column

I've created my first postgres table and i'm trying to work out how to copy a .csv file into the table, whilst simultaneously generating a serial number number in column 1 (ID) which starts at number ...
41
votes
4answers
13k views

sort inplace - like sed --in-place - exists?

Am I blind or is there no option like --in-place for sort? In order to save results to the input file, sed uses -i (--in-place). Redirecting the output of sort to the input file sort < f > f ...
2
votes
3answers
69 views

Splitting file based on last digit of first column in UNIX

I have a file sales_$date.csv. I want to split it into 10 files based on the last digit of first column (ITEM). so in reality file would be split in 10 files as sales_$date-01.csv, sales_$date-02.csv,....
0
votes
4answers
86 views

How do I remove all \r\n from a file, but preserve \n

I have a CSV with unix line endings, but some of the string values have windows line endings in them: date,notes\n 2014-01-01,"Blah Blah Blah"\n 2014-01-02,"Two things:\r\n - first thing\r\n - ...
0
votes
1answer
66 views

Extract data from csv

friends! I have a HUGE csv file and I need extract rows with headers where column called "data.status" equals "complete", then delete empty columns from it and write to another csv file. Can I achieve ...
1
vote
0answers
53 views

How to make excel interpret Chinese characters in CSV file

I am sending a report from a unix shell script that contains chinese characters. When I open the CSV report in plaintext editor like textmate I can see the chinese characters but when I open CSV ...
0
votes
2answers
72 views

Compare a file with a column of another file

I have a file File1 with data as aaa bbb And another file File2 with data as: 2,aaa,234 w,bbb,589 4,ccc,675 I need to compare the File1 data with column2 data of File2 and print the matching ...
0
votes
1answer
29 views

Parsing a delimited in ksh as command argument

My system generates a flat file with 7 arguments : Field1,Field2,Field3,Field4,Field5,Field6,Field7 Field1,Field2,Field3,Field4,Field5,Field6,Field7 Field1,Field2,Field3,Field4,Field5,Field6,Field7 ...
0
votes
1answer
102 views

text to xls file creation

I have 4 text files in my unix system want to put data into a xls file and send it via mail. text1.txt: 100 150 130 120 110 text2.txt: 200 230 240 250 260 .. .. .. so on ... I want to have ...
1
vote
0answers
43 views

CSV copyscript not copying 5 files out of 4000+

I was provided a great bit of code for searching and copying files listed in a csv from a directory and its subfolders in an answer to my previous question. However, some files are not copied. The ...
4
votes
3answers
126 views

Delete consecutive lines in CSV with duplicate values in one field, but keep the last line

I have a long CSV file with two columns, which includes runs of consecutive duplicates like this: ... 1500,1533 1554,1678 1554,1703 1554,1728 1593,1766 ... I need to delete all of these duplicates ...
-3
votes
2answers
30 views

How can I reformat MAC address and IP pairs, ensuring each byte of the MAC has a leading zero?

I've got a table like this: 10.6.6.12 0:50:56:9b:a6:2 10.6.6.13 fe:9:31:b6:71:1d I must have something like this: 10.6.6.12,00:50:56:9b:a6:02 10.6.6.13,fe:09:31:b6:71:1d A 0 is always added ...
1
vote
2answers
42 views

I would like join two csv files

I would like join two csv files: file1: 19/02/2016 22:00;16.70 19/02/2016 22:30;16.80 19/02/2016 23:00;16.80 19/02/2016 23:30;16.70 file2: 19/02/2016 22:00;39.50 19/02/2016 22:30;41.00 19/02/2016 ...
1
vote
4answers
169 views

Split comma separated list ignoring commas within matching {}

I want to split a csv, but ignore any commas within matching curly brace groups, and loop through each of the list members. The code below works great but does not consider commas within curly brace ...
0
votes
3answers
75 views

How to find specific csv files and extract their second row into a single csv file

I'm trying to find out how to filter specific csv files by name and then extract their second row into a single csv file. I'm not sure whether it is a good idea to find files and then pipe it with ...
1
vote
0answers
192 views

How to redirect linux command outputs to specific row and column in csv?

I'm working on Unix Build post documentation process, I have a CSV file to be filled in and uploaded for every new build, to ensure the servers are build as per the original request. The CSV file ...
4
votes
5answers
215 views

How do I print all lines of a file with duplicate values in a certain column

Given a file like this: 1,768,12,46576457,7898 1,123,435,134,146 2,345,6756856,12312,1311 5,234,567465,12341,1341 1,3245,4356345,2442,13 9,423,2342,121,463 9,989,342,121,1212 I would like to list ...
6
votes
7answers
699 views

Keep only the lines containing exact number of delimiters

I have a huge csv file with 10 fields separated by commas. Unfortunately, some lines are malformed and do not contain exactly 10 commas (what causes some problems when I want to read the file into R). ...
-4
votes
1answer
54 views

How to transform row to collumn in csv file?

2015-12-2119:30:00;ManagedElement=1,Equipment=1,Subrack=1 ;pmEs ;0 I need this result 2015-12-2119:30:00;ManagedElement=1,Equipment=1,Subrack=1;pmEs;0 Extra data from comment: 2015-12-21 19:30:...
5
votes
7answers
781 views

Remove lines from tab-delimited file with missing values

I have a large (~900MB) tab-delimited text file that I will be processing in a downstream program. I need to delete any line with a missing value. The correct number of columns are on each line (so ...
2
votes
0answers
42 views

Command like `column -t` but encoding-type agnostic? [closed]

Essentially, I am looking for an encoding-type agnostic version of the column -t command in linux. I need to convert delimited files to flat and then write to an outfile. The specific delimiter ...
1
vote
1answer
1k views

How to select rows from a CSV file based on different column values?

I have a CSV file with values like the following: col1,col2,col3,col4,col5,col6,col7 1,0,0,0,0,BTS,Active 4,5,3,1,1,LocalMode,Offlne 1,2,5,6,3,PermFault,Offline 1,2,6,6,2,BSC,Active 7,8,2,3,2,NE,...
1
vote
4answers
292 views

Modify an SVG file based on information in a CSV file

I've got a SVG file of the map of the US and I want to add colors to it. Because it would take me to much time, I thought I could use a bash script. I have also another file containing the states and ...
3
votes
1answer
256 views

Extract specific columns from text file

There is a text file, with aboot 10000+ lines, in the format: field1 field2 field3 field4 field5 ... fieldX It needs to be converted to look like: field1,field2,field4,field3,field5 to fieldX The ...
0
votes
0answers
36 views

Assign Tracking # to a CSV file from a Master List TXT File

I have a text file that contains a list of tracking numbers (MasterList.txt). I have another file, Orders.csv, file with 3 columns. Row 1 is the headers. The last column is the 'Tracking Number' ...
1
vote
4answers
179 views

Execute Linux command only if column in CSV file has a value greater than 1

I am trying to figure out a way to execute a Linux command only if a specific column in my CSV file has a value greater than 1. Sample CSV file: Item,Quantity Pen,1 Pencil,1 Stapler,3 ...
1
vote
1answer
142 views

get mysql count of id in shell script

I am trying to get count of column user_Id using count(user_Id) from mysql as follows: count=$(mysql -uroot -proot csv_imports -e "select count(user_Id) from test_data where user_Id=\"12345\";") I ...
10
votes
6answers
18k views

Using jq to extract values and format in CSV

I have the below JSON file: { "data": [ { "displayName": "First Name", "rank": 1, "value": "VALUE" }, { "displayName": "Last Name", "rank": 2, ...
13
votes
5answers
1k views

How can I count the number of lines in a file after a grep match?

I am trying to count the number of lines after a problematic row in a csv file. I am aware I can use the grep -a # syntax to output # number of lines after a match has been found. I'm only interested ...
-1
votes
1answer
30 views

Reformat text contents into a CSV

I have a text file that I wish to turn into a CSV. The pattern of its contents are as follows: 1 mm/dd/yyyy LastName, FirstName MiddleName ID-NUMBER-HERE GENDER 2 mm/dd/yyyy LastName, FirstName ...
2
votes
3answers
89 views

Addition of values in second columns if the first column entry is same in UNIX

I am trying to aggregate a file containing the following data in UNIX. I need to add the amounts if the key is same. Key,amount,date,Time abc-xyz-12234,45,15-08-91,23:00 pqr-vgh-5241,15,15-08-91,21:...
0
votes
1answer
50 views

AWK Help ending standalone script

I started to do this.. to swap fields in a csv file but I don't get to finish the script. Any help? What I want to do is to concatenate fields from 2 to 7 and group them by the first field. Fields 1, ...
1
vote
4answers
235 views

Changing column of multiple csv files

I have a csv file called scenario1.csv in which the second column has column names like "0-4 years high risk", "65+ years first responder" etc. There are 20 such values. The 21st row, 2nd column has ...