Questions tagged [csv]
Files containing data arrange in a table, often with commas (hence Comma Separated Values), to separate columns. Rows are separated by newlines (but not all newlines are row separators as fields can be quoted to contain the separator newlines. Use this tag for full-fledged CSV data not the simpler case of one record per line or completely unquoted (use csv-simple for that kind of data).
630
questions
2
votes
0answers
46 views
shuffle two parallel text files not giving same lines even if same random source
This is similar to Shuffle two parallel text files
I have:
two large csv files with parallel lines. (they represent 'before' and 'after' states for particular items). The fields are sometimes ...
0
votes
2answers
60 views
csv text-processing question
Sample Input
0bef-82-46-8a-9a0b.xml "Fruits/Mango Apple /Plum cherry date">1446815.ABC
0bef-82-46-8a-9a0b 5da-0-ba-c1-1a9 "Fruits/Pear Banana/Plum orange mango"
0bef-82-46-8a-9a0b ac-94-4ab-91-23 "...
0
votes
1answer
24 views
Merging two .csv tables one with header other - not
I need to merge two .csv tables,
one that contains the following info with NO HEADER:
gene0 ABCD
gene1 xyz
gene2 1234
..etc
Another table has a header and information like that:
gene_id ...
0
votes
4answers
58 views
Shell : How to replace character only inside double quoted text in a file?
I have following line in text file
abc|45|"Do not replace | in this"|0.23
I want a way to replace | character only in text in double quotes, resulting
abc|45|"Do not replace in this"|0.23
I have ...
0
votes
1answer
72 views
How can I edit txt to xls in Unix?
I want to convert my txt file to xls/csv.
The first row should print datetime and second row should print the remaining all data (from tid to table in below example)
When m using the below command
...
0
votes
4answers
56 views
How to delete and replace field separator from column using awk?
I have CSV file like this:
name_var;type_var;
id;string;
dt_depot;string;
num_artc;string
I want to delete the first's ; and replace the last ones by , so the output would look like:
name_var ...
0
votes
1answer
46 views
Read csv from shell script and print the column header based on closest match to user input
I Have a csv file. It has multiple columns. I want to develop a shell script which prompts the use for a numerical value, find the closest match in the csv and print the header of the column. For ...
0
votes
1answer
34 views
Output specific field values only if the specific values of an entire random line are equal to some variable
So I have a .csv file containing about 1000+ lines. The file consists of variety of movies; in the first column is the name, in the second is the rating (PG-13, R, etc), in the fifth is the year it ...
0
votes
1answer
43 views
Compare two .csv file and write diff into 3rd file
I have two .csv files test_case_list.csv and test_case_summary.csv and I want to generate a third file that will have the difference of two files.
test_case_list.csv looks like:
cli-verify-cvlan-...
0
votes
2answers
57 views
separating an array into comma separated values
I have a small snippet which gives me some ips of my current network:
#!/bin/bash
read -p "network:" network
data=$(nmap -sP $network | awk '/is up/ {print up}; {gsub (/\(|\)/,""); up = $NF}')
it ...
0
votes
3answers
42 views
filter csv file based on list of id in another file
I have a large csv file with these format
file 1:
id,ICD,CT
16214497,008.8,1
16227244,401.1,45
27154870,780.79,1
I have another txt file with one column of list of id (header id).
file 2:
id
...
3
votes
5answers
36 views
Delete the character and numbers from partiular position in comma delimeted file in unix
I want to delete the characters and numbers only from particular position from comma delimited file in unix.
My data like this
jagan1,*,*,*,*,*,W,24,25,26
jagan2,*,*,*,*,*,1,24,25,26
jagan3,*,*,*,*,*...
-1
votes
1answer
39 views
Check if a two columns are completely numeric in a csv using awk
I have a csv with columns A & B as below which has numeric or alphanumeric data.I need to write column C such that if,
A or B is blank -> write "Error" in C
A or B not numeric -> write "Error"...
1
vote
1answer
46 views
Export command output specific lines and put it into CSV file
Could you assist me how to retrieve specific lines and put it into CSV file?
How to process the output via Bash and Python3?
Remark:
"objectID" represents each found server
"An instance of Path" ...
0
votes
1answer
28 views
How to execute multiple commands from an input file and string it to a new output file
I want to execute multiple commands (one after the other) from an input.txt file and output it to a new .csv file.
All the commands run successfully when executed individually, but nothing happens ...
0
votes
2answers
59 views
Merge CSV Files based on ID even when Unmatched IDS and Indefinite Number of Columns
I'm writing a bash script and part of the process requires to piece csv files together whilst preserving the integrity of the columns; for example I have two files in the following format:
F1
ID,MD,...
1
vote
2answers
24 views
How to combine a first line with several next lines?
I have data that I have filtered and made a CSV file like this:
%%TRACERT: IPTYPE=IPv4, LOCALIP4="10.10.10.10", PEERIP4="20.20.20.20", MAXHOP=10;%%
1,3,1,1,1,10.10.10.10
2,3,1,1,1,10.37.112.5
3,0,...
2
votes
2answers
34 views
How to delete a record, a special value in Unix with the comma separated file n
How to delete a record, a special value in Unix with the comma separated file n
My data
100,india,*,*,P,45000,location
200,india,*,*,*,45000,location
300,india,*,*,P,45000,location
400,india,*,*,*,...
0
votes
3answers
83 views
Edit first column of csv with sed or awk in bash
I have a csv file and want to edit just the first column containing title of scientific papers by deleting everything after the ";" sign.
So a typical title would have the following form:
long ...
0
votes
2answers
29 views
converting variables in a shell script from strings to numbers
I have a file laid out like this
gender,yearsExperience,yearsSchool,wage
female,9,13,6.3152956461
female,12,12,5.4797699786
female,11,11,3.6421699174
female,9,14,4.5933365997
female,8,14,2.4181574607
...
0
votes
3answers
69 views
Compare 2 comma separated files and output the difference
I would like to compare 2 similar files on a common column and output the difference. The files will have identical headers. And i am looking to see the difference of particular column.
Second file ...
0
votes
2answers
36 views
Match values between two files awk
I want to compare the first columns from two files. If there is a match, export the corresponding values of the second file to the first file.
File 1
username, fields
File 2
username, other_fields
...
-2
votes
2answers
58 views
How can I manipulate contents of a text file into a new output file
I have an input text file that looks like this:
table columns are:
number of vertices
total surface area (mm^2)
total gray matter volume (mm^3)
average cortical thickness +- standard deviation (mm)
...
1
vote
1answer
38 views
How do I print matching field data from a doc with a qualifier?
I have a CSV Doc that has three columns. Column 1 has an MD5 checksum. Column 2 has a path to the file. Column 3 is either blank or has a unique identifier.
Example
0000801f8b7a5c3b483809ef069d4d82,/...
-1
votes
1answer
37 views
Parsing a CSV file with comma in one of its field [duplicate]
Is there a way that I can parse a CSV file which has comma as a valid character in one of its fields (which can a free form text like DESCRIPTION)?
The only alternative (easy way) I have tried to ...
0
votes
3answers
81 views
CSV dataset to text dataset
I have a csv dataset student.csv with 1000 rows and 10 columns.
The sample data looks like this:
s_id,s_name,s_dob,s_class,s_marks,s_parentname,
1,abc,1/1/18,5,49,def,
2,xyz,1/1/17,4,85,abc,
...
1
vote
2answers
52 views
match a string in two columns then print count AWK
I have a csv file that contains 21 columns. I want to search a string in two columns then count the total number of occurrences.
sample of my dataset
columnA columnB columnC columnD
abc ...
2
votes
5answers
133 views
Filtering CSV dataset to find rows and store them in text file
I'm new to shell commands. I'm kinda struggling with this question based on a CSV dataset fbnews.csv.
The CSV dataset looks like this:
D,E,F, message, score, A,B,C, ID
d,e,f, ...
0
votes
1answer
154 views
Counting unique values in all columns of a csv file
I have a CSV file, a.csv, which have 15000 rows and 16 columns.
It have numbers, sentences, dates in different columns.
It looks like this:
A, B, C, D
a b c d
e f g h
e i j k
a m n k
...
2
votes
2answers
70 views
Finding a substring in a column of CSV file
I have a csv file with around 15000 lines and multiple columns. The data set looks like this:
A,B,C, message D, E, F
a,b,c,Hi how are you ? d,e,f
a,b,c,Hi Hello.Are you ok? ...
1
vote
4answers
75 views
Given Pattern Replacement With Comma
Below follows test.txt contents, with generally 22 columns:
BusinessDate,SourceSystemId,IceIndexId,IceIndexName,ComponentId,ComponentReferenceType,ComponentType,ComponentName,ComponentIssuerCIS,...
0
votes
2answers
68 views
extract columns from csv to a text file
It is a simple problem. I have a csv file with multiple columns I would like to extract 3 columns and save the output to a text file.
sample of my dataset:
page_id post_name link ...
0
votes
3answers
165 views
splitting a column in Bash based on '_'
Hello I have a similar problem to the one in this post
But since I am new to Bash and my file is a bit different I could not modify and apply the answers to my code.
I have a csv file with multiple ...
0
votes
2answers
50 views
CSV - Add quotes around missing fields
My CSV file looks like this -
1,2,3,4,5,6,7,8
"AK","87","86","81","83","78",,
How do I transform it like this -
1,2,3,4,5,6,7,8
"AK","87","86","81","83","78","",""
1
vote
1answer
76 views
How to out the output of the command 'cal' to csv.?
How can I format the output of the command cal -y 2020 as a csv? I want to prepare the calendar in a table processor. Can this be done?
I am using Slackware 14.2.
1
vote
1answer
250 views
create a folder from dynamic path location
I would like to create a folder which location is available in abc.csv file. abc.scv content the following fields id,path and file name we should read the file name and move the file in the path ...
1
vote
1answer
75 views
UNIX/SED - Wildcard in file name
File name is: file_201909091143.csv
How to add a wildcard to pull all files that start with 'file' ?
#!/bin/bash
#cleans blank rows
sed -i '/^$/d' file?.csv #wildcard missing
UPDATE
directory's ls:
...
0
votes
2answers
60 views
how separate each string in csv line to lines
we want to separate each string in csv line to lines
example of line ( could be other line , with the same concept )
machine23,machine094,machine73,machine83
so we try this
echo machine23,...
3
votes
2answers
104 views
Parsing csv file to filter rows basing on a matching set of characters from a column value
Consider the following csv file:
A,3300
B,8440
B,8443
B,8444
C,304
C,404
M,5502
M,5511
The real csv file is really big (around 60,000 rows) but I only included a small version for ...
3
votes
3answers
42 views
Text Manipulation - Transpose column into rows depending on the value
I want to write a code to be able to manipulate text on my CSV file which contains the following:
71w - Rus,51200
71w - Phi,307200
71w - Ukr,307200
71w - Ukr,51200
71w - Mic,102400
71w - Mic,51200
...
-1
votes
1answer
31 views
Merge files in different directories
I have 3 dirs with csv files with the same name
/dir1/ex1.csv
/dir2/ex1.csv
/dir3/ex1.csv
And i would like to merge the content in just one destination file (Just concat file content, since is a csv ...
-2
votes
2answers
66 views
Help with indexing a list of values
I need to track a huge log and more exactly a column. In this column I have some integer values in a rage between 103 and 17431. Example of original file
402
402
402
667
942
342
990
402
For every ...
0
votes
1answer
54 views
How to split CSV file by first column being a multiple of n
I have large csv file that looks like this:
1 , aaa, bbb, ...
1 , ccc, ddd, ...
2 , aaa, bbb, ...
..
10 , aaa, bbb, ...
..
1000 , aaa, bbb, ...
I would like to split it into separate csv files ...
0
votes
3answers
38 views
I want to delete a column from a row which begins with a particular string in a csv file
I have a csv file:
test_1,2,data,hi,cat
test_2,3,4,5,6
test_1,3,7,8,9
I want to delete column 3 of the rows which begin with test_1.
I used the cut command to delete column 3 but I do not know how ...
0
votes
1answer
66 views
How to write a csvcut script to cut column by header with multiple files?
Since csvcut (from csvkit) does not take more than a single file at a time, I need to write a script to process multiple files using it.
The first parameter should be the delimiter, the second ...
2
votes
2answers
385 views
How can I categorize files in a directory based on their content?
I have a directory which contains a lot of CSV files. The CSV files have many columns the first of which is a timestamp (as number of seconds since the UNIX Epoch). I want to categorize files in the ...
0
votes
0answers
34 views
Large file processing performance
I did a simple performance test of csv processing a while ago, and want to share results with community, maybe you can point out, what test could be more precise and fair.
Firstly i took out 42 MB ...
1
vote
3answers
524 views
Joining / merging CSV files that do not share all their headers / column
I have a directory with 1000s of files from a data recording system, and each file can have as many as 40,000 or more rows. The challenge is that sometimes data is not recorded from one or more ...
0
votes
1answer
35 views
using sed how to append N number of lines
I have the following csv file:
hostname1,this is a test,001^M
hostname2,this is
a test,002^M
hostname3,this
is
a
test,003^M
The first line is the proper format of the line. I want the other lines to ...
0
votes
1answer
32 views
AWK - multiply two columns and add result with trailing zeros
I have a csv file like this:
Row 1,1,4.50,
Row 2,8,0.30,
Row 3,14,0.30,
Row 4,2,2.50,
Row 5,20,0.30,
I want to multiply the second coulmn with the third and add the result as a fourth.
I got this ...
