Questions tagged [csv-simple]

Text files containing simple comma separated values. Use this for data that is a subset of real CSV without newlines in (quoted) field content, that can be handled with line oriented tools without the need for a real parser.

Filter by
Sorted by
Tagged with
25
votes
12answers
14k views

Converting CSV to TSV

I have a number of large CSV files and would like them in TSV (tab separated format). The complication is that there are commas in the fields of the CSV file, eg: A,,C,"D,E,F","G",I,"K,L,M",Z ...
5
votes
7answers
890 views

How to remove duplicate value in a tab-delimited text file

I have a tab delimited column text like below A B1 B1 C1 B B2 D2 C C12 C13 C13 D D3 D5 D9 G F2 F2 how could I convert the above table ...
5
votes
1answer
4k views

Print each field of CSV on newline without knowing number of fields

I was playing with IFS today and created a quick text file with a list of numbers separated by commas on 1 line. 1,2,3,4,5 I then tried to write a script to print each number on a newline. I was ...
4
votes
4answers
203 views

Double quote value assignments stored in a CSV?

I have a file that contains text as follows: dt=2016-06-30,path=path1,site=US,mobile=1 dt=2016-06-21,path=path2,site=UK,mobile=0 I want to convert it to text with double-quoted values in the key-...
4
votes
5answers
772 views

Count unique associated values in awk (or perl)

I've already found "How to print incremental count of occurrences of unique values in column 1", which is similar to my question, but the answer isn't sufficient for my purposes. First let me just ...
3
votes
3answers
7k views

Determine maximum column length for every column in a simplified csv-file (one line per row)

To determine the maximum length of each column in a comma-separated csv-file I hacked together a bash-script. When I ran it on a linux system it produced the correct output, but I need it to run on OS ...
3
votes
3answers
3k views

Count the number of occurrences of a column value in a TSV file with AWK

I have a TSV tab-separated file with 3 cols: ID\tTEXT\tTYPE To print the TYPE column I do cat /dataset.csv | awk -F $'\t' '{print $3}' Those values are an enumeration of values like {CLASS_A,...
3
votes
4answers
19k views

Compare two .csv files

I have two files: 'file1' has employee ID numbers. 'file2' has the complete database of the employees. Here is what they look like: file1 123123 222333 file2 111222 Jones Sally 111333 Johnson ...
3
votes
1answer
83 views

Why is column adding a newline in the middle of my row where one is not present in the original data?

I'm analyzing some packetfilter logs and wanted to make a nice table of some output, which normally works fine when I use column -t. I can't use a tab as my output field separator (OFS) in this case ...
3
votes
1answer
375 views

Create table with frequency of unique names retrieved from multiple .csv files

I have 32 CSV files containing fetched information from a database. I need to make a frequency table in TSV/CSV format, where the names of the rows are the name of each file, and the names of the ...
2
votes
5answers
1k views

Use AWK to split substring by last n characters in to a new column

I have a large .csv file where I need to split a specific column by string length. I'm trying to take the last 6 characters of column 2 and move them into a new column. Current: 3102017,90131112,0,...
2
votes
3answers
1k views

Search a directory recursively for files listed in a csv, and copy them to another location

I have a directory (let's call it "Movies") which contains many files and folders. I have a long list of file names in a .csv file (around 4000 entries) which refer to files which are located ...
2
votes
2answers
2k views

CSV > Excel > Graph?

Q: I currently have MQTT client that dumps data in a csv file, is it possible to convert this data into a graph with BASH or Python [automatically]? Can such a thing be done? Or would this be an Excel ...
2
votes
1answer
2k views

How to insert CSV data into an SQLite table via a shell pipe?

I have written a program that outputs the result to the standard output in strict pure CSV form (every line represents a single record and contain the same set of comma-separated fields, fields only ...
2
votes
1answer
23 views

Update table header regarding file name

I have several files named has follow (I just give 2 names as examples)(note: they have a semi colon ; field separator): component1_component2_component3.csv e.g. almourol_gfdl_85.csv ...
2
votes
1answer
346 views

Transform .csv into 3 columns and a row

Sample Input: id,Product1,Product2,Product3,Product4 1,0.1,0.3,0.8,0.7 2,0.6,0.7,0.5,0.9 I need output as : id,productname,product_val 1,Product1,0.1 1,Product2,0.3 1,Product3,0.8 1,Product4,0.7 2,...
2
votes
5answers
2k views

awk: split column, print left side of delimiter and paste in the middle of csv

Input file: AAA, BBB:XXX, CCC, DDD, EEE, FFF, GGG, HHH Output should be: AAA, BBB, BBB:XXX, CCC, DDD, EEE, FFF, GGG, HHH I thought about something like: awk -F, '{n=split($2,a,":"); a[n]} {$2=$NF"...
2
votes
4answers
6k views

How to check for null fields in CSV file?

My CSV file will look like this (pipe separated): apple|banana|pear||grapefruit lemon|lime|damson|jackfruit |tangerine|nectarine|plum apricot|orange|pineapple|coconut| (Assume that any sort of ...
2
votes
1answer
781 views

AWK, Sum of category

I have tons of CSV files with similar content. The values are usually comma separated and they look like this. product_a, domestic, 500 product_a, abroad, 15 product_b, domestic, 313 ...
2
votes
2answers
246 views

Delete rows from a simplified CSV where some columns matching specific pattern

I have the following simplified CSV file (no separators or newlines embedded in fields): ID,PDBID,FirstResidue,FirstChain,SecondResidue,SecondChain,ThirdResidue,ThirdChain,FourthResidue,FourthChain,...
2
votes
2answers
3k views

Delete rows in a simplified csv file (one newline per row) based on a specific column value

I have a csv file with the following headers - Interaction_ID, PDB_ID, First_Residue, First_Chain, Second_Residue, Third_Residue, Third_Chain & Pattern: AM_AUTO_1,1cx0,A166,B,C118,B,G129,B,A-...
2
votes
1answer
474 views

Command to “fill down” columns in text file, a la the Excel fill down function

I have a text file with rows, and columns within those rows. I want to, essentially, replicate the Excel "fill down" function. In other words, if there is a blank "cell" on a line, it will look to the ...
2
votes
1answer
1k views

Using fixed strings in regular expressions?

I am writing a bash script which accepts a list of CSV files as arguments and outputs e-mail addresses only found in the first file. To accomplish this, for each record in the first CSV file I look ...
1
vote
4answers
884 views

In bash I have to ask for each row from first csv file, second csv file and the answers put to third csv file [closed]

In bash I have to ask for each row from first csv file, second csv file and the answers put to third csv file ... first file. e.g. aaa.csv: aaa bbb aaa aaa ccc fff aaa second file, e.g. bbb.csv: ...
1
vote
3answers
4k views

Group by and sum in shell script without awk

I have a file like: $ cat input.csv 201,100 201,300 300,100 300,500 100,400 I want to add the values in column 2 which has same value in column 1. Expected output is as follows: $ cat output.csv ...
1
vote
2answers
33 views

Data extraction problem based on upper case words

I have a data file in the following format: source.dat (Example for illustration purposes.) ALBANIA Duck 1000 BELGIUM CHARLEROI Donald Duck 10234 CYPRUS J. Mickey 567 I'm looking for a Linux script ...
1
vote
4answers
82 views

Remove any occurrence of '-' from a field in shell script

below is my csv file,I want to remove all occurrences of - from retailer_id field and create new csv. >IPAY_USER_ID,RETAILER_ID,CUST_FIRST_NAME,CUST_LAST_NAME,CUST_MIDDLE_NAME,ACTIVATION_ACTOR_ID,...
1
vote
2answers
2k views

Compare two csv files and append value using awk [closed]

I have two files as follows: file1.csv +------------+----------+--------+---------+ | Account_ID | Asset_ID | LOT_ID | FLAG_F1 | +------------+----------+--------+---------+ | 10000 | 20000 | ...
1
vote
3answers
182 views

Filter csv file based on extended column values

I have the following csv file: ID,PDBID,FirstResidue,SecondResidue,ThirdResidue,FourthResidue,Pattern RZ_AUTO_1,4tov,1404,1405,1518,1519,CG/AA Canonical ribose-zipper RZ_AUTO_2,4tov,1405,1406,1517,...
1
vote
1answer
1k views

Merging CSV (single line per row) files without losing the header line

I need to merge two CSV files: datasorted1.csv and datasorted2.csv, but my headers keep getting sorted. head -n 1 datasorted1.csv datasorted2.csv > resultado.csv && tail -n +2 datasorted1....
1
vote
3answers
369 views

remove duplicated pattern/entries within each field in CSV file

How do I remove duplicated entries within each separate fields with below sample as data. 0x,9.4,,,#0,#UNIX#unix,#cli#L#فا#0#فا#0#L#SE#Cli#SE,#فارسی#فارسی#۱#1#١#1,bsh,#V & v expected output(...
1
vote
3answers
227 views

merge multiple lines of data having one common field

I have a large file with 9+ rows of data seperates with semi-colon(;) and I want to merge data in column 3 (seperating with ,) the lines where data in column 5 matches. The data is held on a Linux ...
1
vote
3answers
5k views

Delete lines that contain X in specific column .csv [duplicate]

I currently use the following to delete all emails that contain EXAMPLE sed -i '/EXAMPLE/d' newname.csv However that deletes lines that have EXAMPLE in any of the columns. I want it to only delete ...
1
vote
2answers
111 views

Using join with space in key

I'm having issues merging two files using the join command: The first file is a csv file: NAIN GENIEU 01/01/1900,A,B NAIN GENIEUR 01/01/1917,C,D NAINGENIEUR 21/01/1917,E,F The second file contains ...
1
vote
3answers
794 views

CSV Units Conversion

I've got a CSV file that looks like the following: miami,20,in lansing,2,cm austin,3,mm chicago,5,miles phoenix,2,feet The first field is the name, the next field is the numeric value and the third ...
1
vote
3answers
875 views

How can I output sample rows from a .csv?

I'm doing a lot of data merge printing and would like to be able to output, say, 10 samples from a large database. My goal would be to extract the 10 rows spread evenly from the supplied .csv (...
1
vote
2answers
293 views

Remove a line if regex exists only in 1st column of each line in a csv file

I have the following comma-delimited data (spaces used for visual clarity): 1 2014 23 2352 19.24 7.18 1 2014 23 2353 19.78 4.65 24 2014 23 2354 20.26 25.34 1 2014 23 2355 26.65 42.76 1 2014 23 ...
1
vote
2answers
1k views

find field pattern in a csv file then count unique field values

I am using Cygwin (bash) to create a script to find, group and count fields in multiple CSV files. Each row will have comma-separated fields with each field following a similar convention. There is a ...
1
vote
2answers
3k views

Count the maximum character length for all the data fields in a simplified csv file and output to txt

Given a simplified CSV (max one line per row) with many data fields (>50), how can I count the maximum character length for each data field and then export all the counts to a txt file? BTW, I want to ...
1
vote
2answers
80 views

How to convert uA/mA to uA in each lines of csv file using c shell?

I have a *.csv file as bellow. What I want to do is converting all units to one unit (uA). Anyone can help to do this in c shell??
1
vote
2answers
1k views

Convert command output to csv, with time stamp

I want to test a speed connection with terminal command: ./speedtest-cli And it returns This: Retrieving speedtest.net configuration... Retrieving speedtest.net server list... Testing from Moscow, ...
1
vote
1answer
171 views

How to filter a basic csv by columns

How do I filter by third column of the following file list.txt [[ list of people ]] billy baxter - @baxter - b.baxter@woods.com james woods - @woods - j.woods@someother.com I currently have cat ...
1
vote
2answers
707 views

Grep matching cell of the csv file and index of that row

I have csv file and I want to see numbers that start with 0.99 and their row's index which is the first cell of that row. This is what I have so far: cat fil.csv | grep '0\\.99'| tee > (cut -d, ...
1
vote
2answers
135 views

text processing : extract parts of a file and write them into a single line

I have several files formatted like this: ... <title> Field1 : DATA_FIELD_1</title> ... <i class="blablabla"></i> <b>Field2 : </b> <span>DATA_FIELD_2</...
1
vote
1answer
901 views

I need to merge multiple .csv files together side by side with unequal number of rows

I have anywhere from 3-5 .csv files and need to be able to merge them together while keeping everything in its respective columns below is an easy example with files with varying number of rows. file1 ...
0
votes
1answer
251 views

appending three CSV files [duplicate]

I have three CSV files which have the same same row count. 1.csv 1,2,3 4,5,6 7,8,9 2.csv 4 5 6 3.csv 8 9 10 After combining I want the result as result.csv 1,2,3,4,8 4,5,6,5,9 7,8,9,6,10 So I ...
0
votes
2answers
44 views

Checking if a text file has all entries and is formatted correctly

I have a file that is like this 0001;4;5;30;33;41;52 0002;9;37;39;41;43;49 0003;10;11;29;30;36;47 0004;1;5;6;27;42;59 0005;1;2;6;16;19;46 0006;7;13;19;22;40;47 0007;3;5;20;21;38;56 ... etc The first ...
0
votes
2answers
135 views

Transform row of integer into multi-column output

I am trying to get the input of a file to generate me a list with rows and columns of that numbers in sorted order. This will be use to generate a printable document later, so i am able to replace the ...
0
votes
3answers
1k views

How to concatenate a variable number of csv, removing their header rows?

I have a directory with several hundred csv files whose filenames start with two digits {01..84}. Several hundred >> 84, so obviously some filenames start with the same prefix. I wish to concatenate ...
0
votes
2answers
202 views

Append to second column

There is a csv file ip,time,name 1.1.1.1,2018-08-31 10:01:03,postgres 2.1.1.1,2018-08-31 12:51:13,oracle 3.1.1.1,2018-08-31 14:32:53,sql I need output as below ip,time,name 1.1.1.1,2018-08-31 10:...