Files containing data arrange in a table, often with commas (hence Comma Separated Values), to separate columns. Rows are separated by newlines (but not all newlines are row separators as fields can be quoted to contain the separator newlines. Use this tag for full-fledged CSV data not the simpler ...

learn more… | top users | synonyms

2
votes
1answer
31 views

If there are more than x number of pipes in a csv line, then delete the 2nd instance

I have a csv file which is supposed to contain 4 columns of data which include a product number, a title, a url and a price. Each column is separated by a | delimiter (this has to be maintained, there ...
-1
votes
0answers
11 views

While using a ruby script to export db info into a csv, all data for an asset is copied into the same cell even when using “--tabdelim” [on hold]

I have the following ruby script which exports all the asset info from database to a csv file. #!/usr/bin/env ruby FCSVRCLIENTCMD = "/Library/Application\\ Support/Final\\ Cut\\ Server/Final\\ ...
2
votes
3answers
34 views

linux + ssh + perform two tasks in one ssh and print to CSV

the following command print the two values in one ssh shut ssh test01 "hostname && cat /etc/sysconfig/rhn/rhnsd " test01 INTERVAL=240 but how to change the syntax in order to print the ...
3
votes
1answer
31 views

Copy csv data while simultaneously filling serial column

I've created my first postgres table and i'm trying to work out how to copy a .csv file into the table, whilst simultaneously generating a serial number number in column 1 (ID) which starts at number ...
2
votes
3answers
64 views

Splitting file based on last digit of first column in UNIX

I have a file sales_$date.csv. I want to split it into 10 files based on the last digit of first column (ITEM). so in reality file would be split in 10 files as sales_$date-01.csv, ...
0
votes
4answers
64 views

How do I remove all \r\n from a file, but preserve \n

I have a CSV with unix line endings, but some of the string values have windows line endings in them: date,notes\n 2014-01-01,"Blah Blah Blah"\n 2014-01-02,"Two things:\r\n - first thing\r\n - ...
0
votes
1answer
52 views

Extract data from csv

friends! I have a HUGE csv file and I need extract rows with headers where column called "data.status" equals "complete", then delete empty columns from it and write to another csv file. Can I achieve ...
1
vote
0answers
31 views

How to make excel interpret Chinese characters in CSV file

I am sending a report from a unix shell script that contains chinese characters. When I open the CSV report in plaintext editor like textmate I can see the chinese characters but when I open CSV ...
0
votes
2answers
52 views

Compare a file with a column of another file

I have a file File1 with data as aaa bbb And another file File2 with data as: 2,aaa,234 w,bbb,589 4,ccc,675 I need to compare the File1 data with column2 data of File2 and print the matching ...
1
vote
1answer
29 views

Calculate hits within a timespan from log file [closed]

I need to use UNIX utilities that implement out-of-core divide-and-conquer algorithms so to decrease the amount of virtual memory used (cap of 1 mb). I have a large data set that is an apache log ...
1
vote
0answers
43 views

CSV copyscript not copying 5 files out of 4000+

I was provided a great bit of code for searching and copying files listed in a csv from a directory and its subfolders in an answer to my previous question. However, some files are not copied. The ...
0
votes
1answer
89 views

text to xls file creation

I have 4 text files in my unix system want to put data into a xls file and send it via mail. text1.txt: 100 150 130 120 110 text2.txt: 200 230 240 250 260 .. .. .. so on ... I want to have ...
-3
votes
2answers
26 views

How can I reformat MAC address and IP pairs, ensuring each byte of the MAC has a leading zero?

I've got a table like this: 10.6.6.12 0:50:56:9b:a6:2 10.6.6.13 fe:9:31:b6:71:1d I must have something like this: 10.6.6.12,00:50:56:9b:a6:02 10.6.6.13,fe:09:31:b6:71:1d A 0 is always added ...
4
votes
3answers
118 views

Delete consecutive lines in CSV with duplicate values in one field, but keep the last line

I have a long CSV file with two columns, which includes runs of consecutive duplicates like this: ... 1500,1533 1554,1678 1554,1703 1554,1728 1593,1766 ... I need to delete all of these duplicates ...
1
vote
2answers
38 views

I would like join two csv files

I would like join two csv files: file1: 19/02/2016 22:00;16.70 19/02/2016 22:30;16.80 19/02/2016 23:00;16.80 19/02/2016 23:30;16.70 file2: 19/02/2016 22:00;39.50 19/02/2016 22:30;41.00 19/02/2016 ...
0
votes
1answer
26 views

Parsing a delimited in ksh as command argument

My system generates a flat file with 7 arguments : Field1,Field2,Field3,Field4,Field5,Field6,Field7 Field1,Field2,Field3,Field4,Field5,Field6,Field7 Field1,Field2,Field3,Field4,Field5,Field6,Field7 ...
1
vote
4answers
151 views

Split comma separated list ignoring commas within matching {}

I want to split a csv, but ignore any commas within matching curly brace groups, and loop through each of the list members. The code below works great but does not consider commas within curly brace ...
0
votes
3answers
63 views

How to find specific csv files and extract their second row into a single csv file

I'm trying to find out how to filter specific csv files by name and then extract their second row into a single csv file. I'm not sure whether it is a good idea to find files and then pipe it with ...
0
votes
0answers
140 views

How to redirect linux command outputs to specific row and column in csv?

I'm working on Unix Build post documentation process, I have a CSV file to be filled in and uploaded for every new build, to ensure the servers are build as per the original request. The CSV file ...
4
votes
5answers
176 views

How do I print all lines of a file with duplicate values in a certain column

Given a file like this: 1,768,12,46576457,7898 1,123,435,134,146 2,345,6756856,12312,1311 5,234,567465,12341,1341 1,3245,4356345,2442,13 9,423,2342,121,463 9,989,342,121,1212 I would like to list ...
6
votes
7answers
599 views

Keep only the lines containing exact number of delimiters

I have a huge csv file with 10 fields separated by commas. Unfortunately, some lines are malformed and do not contain exactly 10 commas (what causes some problems when I want to read the file into R). ...
-4
votes
1answer
53 views

How to transform row to collumn in csv file?

2015-12-2119:30:00;ManagedElement=1,Equipment=1,Subrack=1 ;pmEs ;0 I need this result 2015-12-2119:30:00;ManagedElement=1,Equipment=1,Subrack=1;pmEs;0 Extra data from comment: 2015-12-21 ...
5
votes
7answers
610 views

Remove lines from tab-delimited file with missing values

I have a large (~900MB) tab-delimited text file that I will be processing in a downstream program. I need to delete any line with a missing value. The correct number of columns are on each line (so ...
2
votes
0answers
42 views

Command like `column -t` but encoding-type agnostic? [closed]

Essentially, I am looking for an encoding-type agnostic version of the column -t command in linux. I need to convert delimited files to flat and then write to an outfile. The specific delimiter ...
1
vote
4answers
264 views

Modify an SVG file based on information in a CSV file

I've got a SVG file of the map of the US and I want to add colors to it. Because it would take me to much time, I thought I could use a bash script. I have also another file containing the states and ...
3
votes
1answer
204 views

Extract specific columns from text file

There is a text file, with aboot 10000+ lines, in the format: field1 field2 field3 field4 field5 ... fieldX It needs to be converted to look like: field1,field2,field4,field3,field5 to fieldX The ...
0
votes
0answers
34 views

Assign Tracking # to a CSV file from a Master List TXT File

I have a text file that contains a list of tracking numbers (MasterList.txt). I have another file, Orders.csv, file with 3 columns. Row 1 is the headers. The last column is the 'Tracking Number' ...
1
vote
4answers
146 views

Execute Linux command only if column in CSV file has a value greater than 1

I am trying to figure out a way to execute a Linux command only if a specific column in my CSV file has a value greater than 1. Sample CSV file: Item,Quantity Pen,1 Pencil,1 Stapler,3 ...
1
vote
1answer
78 views

get mysql count of id in shell script

I am trying to get count of column user_Id using count(user_Id) from mysql as follows: count=$(mysql -uroot -proot csv_imports -e "select count(user_Id) from test_data where user_Id=\"12345\";") I ...
-1
votes
1answer
29 views

Reformat text contents into a CSV

I have a text file that I wish to turn into a CSV. The pattern of its contents are as follows: 1 mm/dd/yyyy LastName, FirstName MiddleName ID-NUMBER-HERE GENDER 2 mm/dd/yyyy LastName, FirstName ...
2
votes
3answers
73 views

Addition of values in second columns if the first column entry is same in UNIX

I am trying to aggregate a file containing the following data in UNIX. I need to add the amounts if the key is same. Key,amount,date,Time abc-xyz-12234,45,15-08-91,23:00 ...
0
votes
1answer
50 views

AWK Help ending standalone script

I started to do this.. to swap fields in a csv file but I don't get to finish the script. Any help? What I want to do is to concatenate fields from 2 to 7 and group them by the first field. Fields 1, ...
1
vote
0answers
24 views

exchange a column of many many csv files using awk [duplicate]

There is a similar post, but this one is slightly different. I have a csv file in which the second column contains some alphanumeric data. I need to delete this entire column and replace it with a new ...
1
vote
4answers
219 views

Changing column of multiple csv files

I have a csv file called scenario1.csv in which the second column has column names like "0-4 years high risk", "65+ years first responder" etc. There are 20 such values. The 21st row, 2nd column has ...
1
vote
2answers
152 views

How to convert my text files (each containing a username+password pair) into CSV in order to import them to KeePass via bash(cygwin)

I'd like to use Keepass (win10x64). The problem is that, I have my passwords gathered all these years like this: each username/password pair in a separate text file, having as filename the domain ...
3
votes
3answers
155 views

awk when both delimiter and quotes are used for a field

I have a file in the following format: field1|field2|field3 field1|"field2|field2"|field3 Notice the second row contains double quotes. The string within the double quotes belongs to field 2. How ...
0
votes
1answer
78 views

Single read filter csv based on 1st column values

I have a gigantic csv file that is around a GB in size. The csv's 1st column contains around 300 unique integer values. I want to process the csv once and save the output to different files based on ...
1
vote
2answers
53 views

Stack data from multiple delimited files into one, with variable columns

I'm new to scripting. Got struck with a file merge issue in Unix. Was looking for some direction and stumbled upon this site. I saw many great posts and replies but couldn't find a solution to my ...
2
votes
1answer
68 views

Processing table with comma separated values in different columns

I'm trying to do the following for a while but without success. The data I received has comma separated values in each separate columns. The first value in column 6 before the comma is always related ...
2
votes
1answer
106 views

Adding two columns together in CSV and outputting to new CSV file

I have a problem I have been trying to figure out: We have a stock file CSV, which contains the stock in multiple locations. The csv looks like this: ...
0
votes
3answers
36 views

List users who have the maximum number of processes running with maximum cpu usage

starting off with the command ps -eo user,pcpu,pid,args | sort -rnk 3 | head -30 How do i parse the output of this command into a csv file with users having most processess and consuming cpu ...
0
votes
0answers
12 views

csvkit - File-object must be opened in binary mode

I'm trying to use csvkit to convert excel (*xlsx) files to csv. This application has worked brilliantly but I seem to have some odd files this time around. I get the error: File-object must be ...
0
votes
2answers
120 views

Simple efficient concatenation of CSV files [duplicate]

Consider a large number of CSV files (*.csv) living in some folder. They all have the same exact header. How can I efficiently concatenate them all into a single CSV file with the same single ...
1
vote
3answers
213 views

Help with awk / sed shell script

I have to make a script using the info in the following table (fake info) AnimalNumber,DOB,Gender,Breed,Date-moved-in IE161289240602,04/02/2010,M,AAX,20/07/2011, ...
0
votes
2answers
209 views

Quick way to delete a line from an extremely large file [duplicate]

I have a 67GB .csv file (I know... I know...) I need to remove the 4,125,878th line from the file as it is corrupt. My CSV parsing tools won't ignore or skip it. I could use sed to do something ...
1
vote
2answers
62 views

Cut every 100th column from csv

I have a data file of numbers seperated by tabs, like this 1 2 3 4 2 4 6 8 My real file is 50000 columns wide and I only need every 100th column (column 100, 200, 300, 400, ...). Now I would like ...
5
votes
2answers
1k views

Join multiple sed commands in one script for processing CSV file

Having a CSV file like this: HEADER "first, column"|"second "some random quotes" column"|"third ol' column" FOOTER and looking for result like: HEADER first, column|second "some random quotes" ...
0
votes
0answers
24 views

Properly sorting a csv file [duplicate]

The sort function looked promising, but as far as I know, it can't handle escaped line feeds, and the field deliminator option -t doesn't have an option that will take into account escaped commas. ...
1
vote
2answers
47 views

Is there a command line spell to save specific columns by name in a CSV-file?

Suppose I have a crazy data set in CSV: ...
4
votes
3answers
247 views

How can I change the first line of a big gzip file without decompressing all of it?

I currently have a compressed file, A.gz which contains lots of tabulated data, including a header in the first line. I want to create another file, B.gz, which has the same data as the previous file, ...