Files containing data arrange in a table, often with commas (hence Comma Separated Values), to separate columns. Rows are separated by newlines (but not all newlines are row separators as fields can be quoted to contain the separator newlines. Use this tag for full-fledged CSV data not the simpler ...

learn more… | top users | synonyms

3
votes
3answers
291 views

awk when both delimiter and quotes are used for a field

I have a file in the following format: field1|field2|field3 field1|"field2|field2"|field3 Notice the second row contains double quotes. The string within the double quotes belongs to field 2. How ...
0
votes
1answer
109 views

Single read filter csv based on 1st column values

I have a gigantic csv file that is around a GB in size. The csv's 1st column contains around 300 unique integer values. I want to process the csv once and save the output to different files based on ...
1
vote
2answers
59 views

Stack data from multiple delimited files into one, with variable columns

I'm new to scripting. Got struck with a file merge issue in Unix. Was looking for some direction and stumbled upon this site. I saw many great posts and replies but couldn't find a solution to my ...
2
votes
1answer
78 views

Processing table with comma separated values in different columns

I'm trying to do the following for a while but without success. The data I received has comma separated values in each separate columns. The first value in column 6 before the comma is always related ...
2
votes
1answer
175 views

Adding two columns together in CSV and outputting to new CSV file

I have a problem I have been trying to figure out: We have a stock file CSV, which contains the stock in multiple locations. The csv looks like this: stock_no,primary,secondary,tertiary,cstock,...
0
votes
3answers
37 views

List users who have the maximum number of processes running with maximum cpu usage

starting off with the command ps -eo user,pcpu,pid,args | sort -rnk 3 | head -30 How do i parse the output of this command into a csv file with users having most processess and consuming cpu usage?...
0
votes
0answers
14 views

csvkit - File-object must be opened in binary mode

I'm trying to use csvkit to convert excel (*xlsx) files to csv. This application has worked brilliantly but I seem to have some odd files this time around. I get the error: File-object must be ...
0
votes
2answers
180 views

Simple efficient concatenation of CSV files [duplicate]

Consider a large number of CSV files (*.csv) living in some folder. They all have the same exact header. How can I efficiently concatenate them all into a single CSV file with the same single ...
1
vote
3answers
239 views

Help with awk / sed shell script

I have to make a script using the info in the following table (fake info) AnimalNumber,DOB,Gender,Breed,Date-moved-in IE161289240602,04/02/2010,M,AAX,20/07/2011, IE141424490333,13/01/2009,M,LMX,21/09/...
0
votes
2answers
412 views

Quick way to delete a line from an extremely large file [duplicate]

I have a 67GB .csv file (I know... I know...) I need to remove the 4,125,878th line from the file as it is corrupt. My CSV parsing tools won't ignore or skip it. I could use sed to do something ...
2
votes
2answers
76 views

Cut every 100th column from csv

I have a data file of numbers seperated by tabs, like this 1 2 3 4 2 4 6 8 My real file is 50000 columns wide and I only need every 100th column (column 100, 200, 300, 400, ...). Now I would like ...
6
votes
2answers
3k views

Join multiple sed commands in one script for processing CSV file

Having a CSV file like this: HEADER "first, column"|"second "some random quotes" column"|"third ol' column" FOOTER and looking for result like: HEADER first, column|second "some random quotes" ...
0
votes
0answers
25 views

Properly sorting a csv file [duplicate]

The sort function looked promising, but as far as I know, it can't handle escaped line feeds, and the field deliminator option -t doesn't have an option that will take into account escaped commas. ...
1
vote
2answers
49 views

Is there a command line spell to save specific columns by name in a CSV-file?

Suppose I have a crazy data set in CSV: SOS_VOTERID,COUNTY_NUMBER,COUNTY_ID,LAST_NAME,FIRST_NAME,MIDDLE_NAME,SUFFIX,DATE_OF_BIRTH,REGISTRATION_DATE,PARTY_AFFILIATION,RESIDENTIAL_ADDRESS1,...
4
votes
3answers
348 views

How can I change the first line of a big gzip file without decompressing all of it?

I currently have a compressed file, A.gz which contains lots of tabulated data, including a header in the first line. I want to create another file, B.gz, which has the same data as the previous file, ...
0
votes
0answers
128 views

Convert vnstat log file to csv

My goal is to log daily network activity (rx/tx) from each ethernet interface. I installed vnstat, and while it reports daily and monthly stats, it doesn't offer a nice way to convert to csv to ...
1
vote
3answers
81 views

Process text file - Extract data starting with number

I have a list of contact information I need to process in .txt format seperated by , . We only want to keep SIC codes that start with 65 which is real estate. The command should only check for the ...
10
votes
6answers
778 views

Remove lines from a file depending on lines found in another file

File file1.txt contains lines like: /api/purchase/<hash>/index.html For example: /api/purchase/12ab09f46/index.html File file2.csv contains lines like: <hash>,timestamp,ip_address ...
1
vote
3answers
3k views

How to do a fast convert of a .xls (MS Excel) file to .csv

How can I do a fast convert of excel to csv? I have tried with php, perl, JavaScript. but it takes too much time. For example, for a 5MB file it will take 3-5 minutes.
1
vote
2answers
3k views

Open CSV File And Go Straight To Spreadsheet

Part of my software issues various commands to open and view different file types. For instance I use atril for PDFs and eom for PNGs. However I have a slight problem with CSV files. I can open them ...
2
votes
3answers
1k views

How to sum column values for each row in two csv files using bash script?

I have two CSV files and i have the requirement to calculate the sum of the 8th column from File1 and File2 for every row. The two CSV files have exactly 24 rows each and to simplify the things i have ...
0
votes
1answer
305 views

How to install and run xlsx2csv on Fedora 19?

I was not able to run xlsx2csv at the command line on Fedora 19 [ec2-user@localhost Downloads]$ xlsx2csv bash: xlsx2csv: command not found... So I looked to install the corresponding RPM package ...
3
votes
3answers
226 views

If statement string comparison

I'm reading a csv file with 14 values per line. I want to convert the HEX value from 10th [f10] value to a DEC value. In order to do this if the 7th [f7] value must hold EnHr or EnSt or SpJb or Chem ...
0
votes
2answers
145 views

How can I combine different csv files from differenet folders with the name of each file as a column name in the combined file?

I have a bunch of directories in which there are 3 .csv files with different names. For example, in my directories aa bb cc dd there are 3 files in each: aa: EA_sing_aa.csv EA_ska_aa.csv ...
1
vote
4answers
939 views

How to extract column name (header) from a CSV file which contains the max value in a row?

I am trying to extract the column name with the maximum value in each row using bash script i.e., the column header value or the value from the same column in the first row. I am using the following ...
3
votes
5answers
727 views

Efficient data extraction from multiple files to a single CSV file

I have a large collection of XML files with the same exact structure: $ cat file_<ID>.xml ... ... ... <double>1.2342</double> <double>2.3456</double> ...
1
vote
2answers
343 views

Remove lines from CSV file if the second column is in a word list

I have found a long list of free email providers that I want to remove from my email lists - https://gist.github.com/tbrianjones/5992856 Below are two commands I currently use that do the same job ...
2
votes
4answers
77 views

CSV - Only keep certain entries

I have a bunch of data inside "" and separated by , - Example below of two lines "stampthisandthat.com","GANDI SAS","3fdb1bcc220148658e7ea1a96aef1585-1184687@contact.gandi.net","whois.gandi.net","A....
2
votes
5answers
329 views

Duplicate lines and make them unique?

I have the following structure of text lines: 3923 001 L05 LV,L05 RM 3923 002 L12 RA,L12 LA 3923 003 I06 ALL 3923 004 G04 RV,Z09 ALL but i would need this: 3923 001 L05 LV 3923 001 ...
1
vote
1answer
2k views

How can I use multiple if statement inside another if statement of a awk program?

I have a CSV File and I want to have multiple checks to retrieve the data from the CSV file. I have the following conditions which needs to be taken care of and I am trying to use awk in a shell ...
0
votes
2answers
92 views

Merging large data set in fastest possible way

I have a large data set ~100 GBs having smaller csv files ~ 100,000. I wish to merge all .csv files together in the fastest possible way. Files are located in two different folders. Someone ...
1
vote
1answer
2k views

How to select rows from a CSV file based on different column values?

I have a CSV file with values like the following: col1,col2,col3,col4,col5,col6,col7 1,0,0,0,0,BTS,Active 4,5,3,1,1,LocalMode,Offlne 1,2,5,6,3,PermFault,Offline 1,2,6,6,2,BSC,Active 7,8,2,3,2,NE,...
1
vote
0answers
258 views

extract text to csv file from output of reading multiple files into specific rows

I am trying to write a bash script to generate a csv file from text within multiple pdf documents. I have a script for converting pdf to text, but not for generating the csv file. Each text document ...
1
vote
1answer
245 views

How to parse a CSV file based with groups made of specific number of rows?

I have a CSV file which has the following format: Date,Towns,Count1,Count2 6/26/2015,HYDERABAD,234552,0 ,KARIM NAGAR,7520,0 ,RAJAMUNDRY,7248,256 ,KURNOOL,5848,32,0 ...
1
vote
1answer
346 views

Pick columns from a variable length csv file

I have two CSV files, which share a column that is unique to each row in each file, like an ID. The files don't have a header. The file_2 has variable length columns e.g. 1,2,3 4,5,6,7,8 I joined ...
13
votes
5answers
1k views

How can I count the number of lines in a file after a grep match?

I am trying to count the number of lines after a problematic row in a csv file. I am aware I can use the grep -a # syntax to output # number of lines after a match has been found. I'm only interested ...
1
vote
2answers
72 views

Alignment when printing a series of records

How can I resolve the problem of the tables going to the right. I just want it to be shown under 1. Here's my script with START added. The alignment has gone wonky now: while IFS="," read f1 f2 f3 ...
4
votes
3answers
323 views

Compare two files with four columns

I have two csv files with some log information. I need to compare line to line the field1 (file1) with field2 (file2) with a range of 1 minute (-00:00:01 < time < 00:00:01) and the field5 (file1)...
1
vote
0answers
108 views

How to create a Kate highlight syntax to CSV files

I'm working with CSV files and I'm using vim because of the rainbow csv plugin that makes the editing easier: http://www.vim.org/scripts/script.php?script_id=4598 But I prefer using Kate and as I ...
0
votes
3answers
4k views

convert xml to csv

i need to convert a xml to csv using script. I found a solution with xmlstarlet but it is not available so i am returning to point 0. Here is my xml. <root> <record> <...
0
votes
1answer
214 views

Appending to a CSV file after obtaining datum from a database

I can [finally] extract data from the MySQL DB my_database_query="SELECT * FROM my_database.database" mysql -uUSER -p'PASSWORD' << EndOfFile $my_database_query EndOfFile How can I append this ...
2
votes
1answer
165 views

Using fixed strings in regular expressions?

I am writing a bash script which accepts a list of CSV files as arguments and outputs e-mail addresses only found in the first file. To accomplish this, for each record in the first CSV file I look ...
1
vote
1answer
185 views

Switch Columns in .csv files so that they are all the same

I have 20 excel files having 6x6 rows and columns, where the first row and column are string headers. Each of these files has 4 same columns and row headers except 1 different. I want to know how I ...
2
votes
1answer
467 views

Convert Tab Separated csv using unoconv

I am attempting to convert a tab separated csv into an xls file using unoconv. However, the tabs are not being recognized as the field separator. Can someone please tell me how this can be done. I've ...
2
votes
1answer
37 views

Match and delete lines with ~100 strings

I have a list of roughly 100 entries to be deleted from a csv-delimited file. They are already in another text file called 'tbd.txt; My first thought is to write a bash for loop around 'sed -i' but ...
0
votes
2answers
737 views

I want to print the value from csv file dynamically

I want to print the values from properties file dynamically I am using below code but getting wrong output. testProperty.properties edmcws,edmbws,edmwt Code file=/ThreadDump/testProperty....
2
votes
6answers
536 views

Scripting to split a single CSV row into multiple

I need a script that can take a CSV file with that has a column of semicolon-delimited attributes, and break it into multiple rows to normalize that multi-valued attribute? Here is the scenario: ...
0
votes
0answers
13 views

how to keep the spaces when inserting line to parameter [duplicate]

When I want to see for example the line number 2 , from the CSV file I get the following line example1 LINE_NUM=2 # sed -n "${LINE_NUM}p" file.csv cwdfwefwe,"aaaaa ","\dsw\dawd\aw ...
-3
votes
1answer
37 views

Empty columns which were seen before

I have a large CSV file and want to empty certain columns if they were seen before. So I have (to illustrate my problem): Category | Subcategory ---------+------------ foo | bar foo | bar ...
0
votes
1answer
463 views

csv file with multiple columns to zenity

How could I output the contents of a .csv file to a Zenity list. The file has multiple rows and columns and the rows are no fixed numbers. I would also like it so that when the user double clicks, on ...