Files containing data arrange in a table, often with commas (hence Comma Separated Values), to separate columns. Rows are separated by newlines (but not all newlines are row separators as fields can be quoted to contain the separator newlines. Use this tag for full-fledged CSV data not the simpler ...

learn more… | top users | synonyms

-1
votes
0answers
9 views

While using a ruby script to export db info into a csv, all data for an asset is copied into the same cell even when using “--tabdelim” [on hold]

I have the following ruby script which exports all the asset info from database to a csv file. #!/usr/bin/env ruby FCSVRCLIENTCMD = "/Library/Application\\ Support/Final\\ Cut\\ Server/Final\\ ...
2
votes
3answers
31 views

linux + ssh + perform two tasks in one ssh and print to CSV

the following command print the two values in one ssh shut ssh test01 "hostname && cat /etc/sysconfig/rhn/rhnsd " test01 INTERVAL=240 but how to change the syntax in order to print the ...
1
vote
3answers
2k views

How to do a fast convert of a .xls (MS Excel) file to .csv

How can I do a fast convert of excel to csv? I have tried with php, perl, JavaScript. but it takes too much time. For example, for a 5MB file it will take 3-5 minutes.
3
votes
1answer
30 views

Copy csv data while simultaneously filling serial column

I've created my first postgres table and i'm trying to work out how to copy a .csv file into the table, whilst simultaneously generating a serial number number in column 1 (ID) which starts at number ...
39
votes
4answers
12k views

sort inplace - like sed --in-place - exists?

Am I blind or is there no option like --in-place for sort? In order to save results to the input file, sed uses -i (--in-place). Redirecting the output of sort to the input file sort < f > f ...
0
votes
1answer
1k views

How to create CSV file without unnecessary spaces

I use the xls2csv binary in order to translate XLS doc to CSV in my linux red-hat machine, example: ( from man page ) xls2csv -x "1252spreadsheet.xls" -b WINDOWS-1252 -c "ut8csvfile.csv" -a UTF-8 ...
2
votes
3answers
64 views

Splitting file based on last digit of first column in UNIX

I have a file sales_$date.csv. I want to split it into 10 files based on the last digit of first column (ITEM). so in reality file would be split in 10 files as sales_$date-01.csv, ...
0
votes
4answers
64 views

How do I remove all \r\n from a file, but preserve \n

I have a CSV with unix line endings, but some of the string values have windows line endings in them: date,notes\n 2014-01-01,"Blah Blah Blah"\n 2014-01-02,"Two things:\r\n - first thing\r\n - ...
0
votes
1answer
52 views

Extract data from csv

friends! I have a HUGE csv file and I need extract rows with headers where column called "data.status" equals "complete", then delete empty columns from it and write to another csv file. Can I achieve ...
1
vote
0answers
31 views

How to make excel interpret Chinese characters in CSV file

I am sending a report from a unix shell script that contains chinese characters. When I open the CSV report in plaintext editor like textmate I can see the chinese characters but when I open CSV ...
0
votes
2answers
50 views

Compare a file with a column of another file

I have a file File1 with data as aaa bbb And another file File2 with data as: 2,aaa,234 w,bbb,589 4,ccc,675 I need to compare the File1 data with column2 data of File2 and print the matching ...
1
vote
1answer
29 views

Calculate hits within a timespan from log file [closed]

I need to use UNIX utilities that implement out-of-core divide-and-conquer algorithms so to decrease the amount of virtual memory used (cap of 1 mb). I have a large data set that is an apache log ...
0
votes
1answer
26 views

Parsing a delimited in ksh as command argument

My system generates a flat file with 7 arguments : Field1,Field2,Field3,Field4,Field5,Field6,Field7 Field1,Field2,Field3,Field4,Field5,Field6,Field7 Field1,Field2,Field3,Field4,Field5,Field6,Field7 ...
0
votes
1answer
88 views

text to xls file creation

I have 4 text files in my unix system want to put data into a xls file and send it via mail. text1.txt: 100 150 130 120 110 text2.txt: 200 230 240 250 260 .. .. .. so on ... I want to have ...
1
vote
0answers
43 views

CSV copyscript not copying 5 files out of 4000+

I was provided a great bit of code for searching and copying files listed in a csv from a directory and its subfolders in an answer to my previous question. However, some files are not copied. The ...
4
votes
3answers
117 views

Delete consecutive lines in CSV with duplicate values in one field, but keep the last line

I have a long CSV file with two columns, which includes runs of consecutive duplicates like this: ... 1500,1533 1554,1678 1554,1703 1554,1728 1593,1766 ... I need to delete all of these duplicates ...
-3
votes
2answers
26 views

How can I reformat MAC address and IP pairs, ensuring each byte of the MAC has a leading zero?

I've got a table like this: 10.6.6.12 0:50:56:9b:a6:2 10.6.6.13 fe:9:31:b6:71:1d I must have something like this: 10.6.6.12,00:50:56:9b:a6:02 10.6.6.13,fe:09:31:b6:71:1d A 0 is always added ...
1
vote
2answers
38 views

I would like join two csv files

I would like join two csv files: file1: 19/02/2016 22:00;16.70 19/02/2016 22:30;16.80 19/02/2016 23:00;16.80 19/02/2016 23:30;16.70 file2: 19/02/2016 22:00;39.50 19/02/2016 22:30;41.00 19/02/2016 ...
1
vote
4answers
149 views

Split comma separated list ignoring commas within matching {}

I want to split a csv, but ignore any commas within matching curly brace groups, and loop through each of the list members. The code below works great but does not consider commas within curly brace ...
0
votes
3answers
63 views

How to find specific csv files and extract their second row into a single csv file

I'm trying to find out how to filter specific csv files by name and then extract their second row into a single csv file. I'm not sure whether it is a good idea to find files and then pipe it with ...
0
votes
0answers
138 views

How to redirect linux command outputs to specific row and column in csv?

I'm working on Unix Build post documentation process, I have a CSV file to be filled in and uploaded for every new build, to ensure the servers are build as per the original request. The CSV file ...
4
votes
5answers
175 views

How do I print all lines of a file with duplicate values in a certain column

Given a file like this: 1,768,12,46576457,7898 1,123,435,134,146 2,345,6756856,12312,1311 5,234,567465,12341,1341 1,3245,4356345,2442,13 9,423,2342,121,463 9,989,342,121,1212 I would like to list ...
6
votes
7answers
595 views

Keep only the lines containing exact number of delimiters

I have a huge csv file with 10 fields separated by commas. Unfortunately, some lines are malformed and do not contain exactly 10 commas (what causes some problems when I want to read the file into R). ...
-4
votes
1answer
53 views

How to transform row to collumn in csv file?

2015-12-2119:30:00;ManagedElement=1,Equipment=1,Subrack=1 ;pmEs ;0 I need this result 2015-12-2119:30:00;ManagedElement=1,Equipment=1,Subrack=1;pmEs;0 Extra data from comment: 2015-12-21 ...
5
votes
7answers
601 views

Remove lines from tab-delimited file with missing values

I have a large (~900MB) tab-delimited text file that I will be processing in a downstream program. I need to delete any line with a missing value. The correct number of columns are on each line (so ...
2
votes
0answers
41 views

Command like `column -t` but encoding-type agnostic? [closed]

Essentially, I am looking for an encoding-type agnostic version of the column -t command in linux. I need to convert delimited files to flat and then write to an outfile. The specific delimiter ...
1
vote
1answer
1k views

How to select rows from a CSV file based on different column values?

I have a CSV file with values like the following: col1,col2,col3,col4,col5,col6,col7 1,0,0,0,0,BTS,Active 4,5,3,1,1,LocalMode,Offlne 1,2,5,6,3,PermFault,Offline 1,2,6,6,2,BSC,Active ...
1
vote
4answers
264 views

Modify an SVG file based on information in a CSV file

I've got a SVG file of the map of the US and I want to add colors to it. Because it would take me to much time, I thought I could use a bash script. I have also another file containing the states and ...
3
votes
1answer
202 views

Extract specific columns from text file

There is a text file, with aboot 10000+ lines, in the format: field1 field2 field3 field4 field5 ... fieldX It needs to be converted to look like: field1,field2,field4,field3,field5 to fieldX The ...
14
votes
7answers
24k views

Remove comma between the quotes only in a comma delimited file

I have a input file delimited with commas (,). There are some fields enclosed in double quotes that are having a comma in them. Here is the sample row 123,"ABC, DEV 23",345,534.202,NAME I need to ...
0
votes
0answers
34 views

Assign Tracking # to a CSV file from a Master List TXT File

I have a text file that contains a list of tracking numbers (MasterList.txt). I have another file, Orders.csv, file with 3 columns. Row 1 is the headers. The last column is the 'Tracking Number' ...
1
vote
4answers
146 views

Execute Linux command only if column in CSV file has a value greater than 1

I am trying to figure out a way to execute a Linux command only if a specific column in my CSV file has a value greater than 1. Sample CSV file: Item,Quantity Pen,1 Pencil,1 Stapler,3 ...
1
vote
1answer
78 views

get mysql count of id in shell script

I am trying to get count of column user_Id using count(user_Id) from mysql as follows: count=$(mysql -uroot -proot csv_imports -e "select count(user_Id) from test_data where user_Id=\"12345\";") I ...
9
votes
6answers
15k views

Using jq to extract values and format in CSV

I have the below JSON file: { "data": [ { "displayName": "First Name", "rank": 1, "value": "VALUE" }, { "displayName": "Last Name", "rank": 2, ...
13
votes
5answers
1k views

How can I count the number of lines in a file after a grep match?

I am trying to count the number of lines after a problematic row in a csv file. I am aware I can use the grep -a # syntax to output # number of lines after a match has been found. I'm only interested ...
-1
votes
1answer
29 views

Reformat text contents into a CSV

I have a text file that I wish to turn into a CSV. The pattern of its contents are as follows: 1 mm/dd/yyyy LastName, FirstName MiddleName ID-NUMBER-HERE GENDER 2 mm/dd/yyyy LastName, FirstName ...
2
votes
3answers
73 views

Addition of values in second columns if the first column entry is same in UNIX

I am trying to aggregate a file containing the following data in UNIX. I need to add the amounts if the key is same. Key,amount,date,Time abc-xyz-12234,45,15-08-91,23:00 ...
0
votes
1answer
49 views

AWK Help ending standalone script

I started to do this.. to swap fields in a csv file but I don't get to finish the script. Any help? What I want to do is to concatenate fields from 2 to 7 and group them by the first field. Fields 1, ...
1
vote
4answers
219 views

Changing column of multiple csv files

I have a csv file called scenario1.csv in which the second column has column names like "0-4 years high risk", "65+ years first responder" etc. There are 20 such values. The 21st row, 2nd column has ...
1
vote
0answers
24 views

exchange a column of many many csv files using awk [duplicate]

There is a similar post, but this one is slightly different. I have a csv file in which the second column contains some alphanumeric data. I need to delete this entire column and replace it with a new ...
1
vote
2answers
151 views

How to convert my text files (each containing a username+password pair) into CSV in order to import them to KeePass via bash(cygwin)

I'd like to use Keepass (win10x64). The problem is that, I have my passwords gathered all these years like this: each username/password pair in a separate text file, having as filename the domain ...
0
votes
3answers
76 views

how to export all numbers that are unique in a few text files into another file?

I have a.txt,b.txt,c.txt. Each has different numbers as below: a.txt: 12 14 111 1 15 2 b.txt 12 18 22 23 1 2 c.txt 12 14 15 16 17 1200 The output should contain all the numbers from ...
1
vote
2answers
53 views

Stack data from multiple delimited files into one, with variable columns

I'm new to scripting. Got struck with a file merge issue in Unix. Was looking for some direction and stumbled upon this site. I saw many great posts and replies but couldn't find a solution to my ...
3
votes
3answers
150 views

awk when both delimiter and quotes are used for a field

I have a file in the following format: field1|field2|field3 field1|"field2|field2"|field3 Notice the second row contains double quotes. The string within the double quotes belongs to field 2. How ...
0
votes
1answer
77 views

Single read filter csv based on 1st column values

I have a gigantic csv file that is around a GB in size. The csv's 1st column contains around 300 unique integer values. I want to process the csv once and save the output to different files based on ...
1
vote
1answer
2k views

How can I use multiple if statement inside another if statement of a awk program?

I have a CSV File and I want to have multiple checks to retrieve the data from the CSV file. I have the following conditions which needs to be taken care of and I am trying to use awk in a shell ...
2
votes
1answer
67 views

Processing table with comma separated values in different columns

I'm trying to do the following for a while but without success. The data I received has comma separated values in each separate columns. The first value in column 6 before the comma is always related ...
2
votes
1answer
106 views

Adding two columns together in CSV and outputting to new CSV file

I have a problem I have been trying to figure out: We have a stock file CSV, which contains the stock in multiple locations. The csv looks like this: ...
0
votes
3answers
36 views

List users who have the maximum number of processes running with maximum cpu usage

starting off with the command ps -eo user,pcpu,pid,args | sort -rnk 3 | head -30 How do i parse the output of this command into a csv file with users having most processess and consuming cpu ...
0
votes
0answers
12 views

csvkit - File-object must be opened in binary mode

I'm trying to use csvkit to convert excel (*xlsx) files to csv. This application has worked brilliantly but I seem to have some odd files this time around. I get the error: File-object must be ...