Questions tagged [csv]

Files containing data arrange in a table, often with commas (hence Comma Separated Values), to separate columns. Rows are separated by newlines (but not all newlines are row separators as fields can be quoted to contain the separator newlines. Use this tag for full-fledged CSV data not the simpler case of one record per line or completely unquoted (use csv-simple for that kind of data).

Filter by
Sorted by
Tagged with
2
votes
0answers
46 views

shuffle two parallel text files not giving same lines even if same random source

This is similar to Shuffle two parallel text files I have: two large csv files with parallel lines. (they represent 'before' and 'after' states for particular items). The fields are sometimes ...
0
votes
2answers
60 views

csv text-processing question

Sample Input 0bef-82-46-8a-9a0b.xml "Fruits/Mango Apple /Plum cherry date">1446815.ABC 0bef-82-46-8a-9a0b 5da-0-ba-c1-1a9 "Fruits/Pear Banana/Plum orange mango" 0bef-82-46-8a-9a0b ac-94-4ab-91-23 "...
0
votes
1answer
24 views

Merging two .csv tables one with header other - not

I need to merge two .csv tables, one that contains the following info with NO HEADER: gene0 ABCD gene1 xyz gene2 1234 ..etc Another table has a header and information like that: gene_id ...
0
votes
4answers
58 views

Shell : How to replace character only inside double quoted text in a file?

I have following line in text file abc|45|"Do not replace | in this"|0.23 I want a way to replace | character only in text in double quotes, resulting abc|45|"Do not replace in this"|0.23 I have ...
0
votes
1answer
72 views

How can I edit txt to xls in Unix?

I want to convert my txt file to xls/csv. The first row should print datetime and second row should print the remaining all data (from tid to table in below example) When m using the below command ...
0
votes
4answers
56 views

How to delete and replace field separator from column using awk?

I have CSV file like this: name_var;type_var; id;string; dt_depot;string; num_artc;string I want to delete the first's ; and replace the last ones by , so the output would look like: name_var ...
0
votes
1answer
46 views

Read csv from shell script and print the column header based on closest match to user input

I Have a csv file. It has multiple columns. I want to develop a shell script which prompts the use for a numerical value, find the closest match in the csv and print the header of the column. For ...
0
votes
1answer
34 views

Output specific field values only if the specific values of an entire random line are equal to some variable

So I have a .csv file containing about 1000+ lines. The file consists of variety of movies; in the first column is the name, in the second is the rating (PG-13, R, etc), in the fifth is the year it ...
0
votes
1answer
43 views

Compare two .csv file and write diff into 3rd file

I have two .csv files test_case_list.csv and test_case_summary.csv and I want to generate a third file that will have the difference of two files. test_case_list.csv looks like: cli-verify-cvlan-...
0
votes
2answers
57 views

separating an array into comma separated values

I have a small snippet which gives me some ips of my current network: #!/bin/bash read -p "network:" network data=$(nmap -sP $network | awk '/is up/ {print up}; {gsub (/\(|\)/,""); up = $NF}') it ...
0
votes
3answers
42 views

filter csv file based on list of id in another file

I have a large csv file with these format file 1: id,ICD,CT 16214497,008.8,1 16227244,401.1,45 27154870,780.79,1 I have another txt file with one column of list of id (header id). file 2: id ...
3
votes
5answers
36 views

Delete the character and numbers from partiular position in comma delimeted file in unix

I want to delete the characters and numbers only from particular position from comma delimited file in unix. My data like this jagan1,*,*,*,*,*,W,24,25,26 jagan2,*,*,*,*,*,1,24,25,26 jagan3,*,*,*,*,*...
-1
votes
1answer
39 views

Check if a two columns are completely numeric in a csv using awk

I have a csv with columns A & B as below which has numeric or alphanumeric data.I need to write column C such that if, A or B is blank -> write "Error" in C A or B not numeric -> write "Error"...
1
vote
1answer
46 views

Export command output specific lines and put it into CSV file

Could you assist me how to retrieve specific lines and put it into CSV file? How to process the output via Bash and Python3? Remark: "objectID" represents each found server "An instance of Path" ...
0
votes
1answer
28 views

How to execute multiple commands from an input file and string it to a new output file

I want to execute multiple commands (one after the other) from an input.txt file and output it to a new .csv file. All the commands run successfully when executed individually, but nothing happens ...
0
votes
2answers
59 views

Merge CSV Files based on ID even when Unmatched IDS and Indefinite Number of Columns

I'm writing a bash script and part of the process requires to piece csv files together whilst preserving the integrity of the columns; for example I have two files in the following format: F1 ID,MD,...
1
vote
2answers
24 views

How to combine a first line with several next lines?

I have data that I have filtered and made a CSV file like this: %%TRACERT: IPTYPE=IPv4, LOCALIP4="10.10.10.10", PEERIP4="20.20.20.20", MAXHOP=10;%% 1,3,1,1,1,10.10.10.10 2,3,1,1,1,10.37.112.5 3,0,...
2
votes
2answers
34 views

How to delete a record, a special value in Unix with the comma separated file n

How to delete a record, a special value in Unix with the comma separated file n My data 100,india,*,*,P,45000,location 200,india,*,*,*,45000,location 300,india,*,*,P,45000,location 400,india,*,*,*,...
0
votes
3answers
83 views

Edit first column of csv with sed or awk in bash

I have a csv file and want to edit just the first column containing title of scientific papers by deleting everything after the ";" sign. So a typical title would have the following form: long ...
0
votes
2answers
29 views

converting variables in a shell script from strings to numbers

I have a file laid out like this gender,yearsExperience,yearsSchool,wage female,9,13,6.3152956461 female,12,12,5.4797699786 female,11,11,3.6421699174 female,9,14,4.5933365997 female,8,14,2.4181574607 ...
0
votes
3answers
69 views

Compare 2 comma separated files and output the difference

I would like to compare 2 similar files on a common column and output the difference. The files will have identical headers. And i am looking to see the difference of particular column. Second file ...
0
votes
2answers
36 views

Match values between two files awk

I want to compare the first columns from two files. If there is a match, export the corresponding values of the second file to the first file. File 1 username, fields File 2 username, other_fields ...
-2
votes
2answers
58 views

How can I manipulate contents of a text file into a new output file

I have an input text file that looks like this: table columns are: number of vertices total surface area (mm^2) total gray matter volume (mm^3) average cortical thickness +- standard deviation (mm) ...
1
vote
1answer
38 views

How do I print matching field data from a doc with a qualifier?

I have a CSV Doc that has three columns. Column 1 has an MD5 checksum. Column 2 has a path to the file. Column 3 is either blank or has a unique identifier. Example 0000801f8b7a5c3b483809ef069d4d82,/...
-1
votes
1answer
37 views

Parsing a CSV file with comma in one of its field [duplicate]

Is there a way that I can parse a CSV file which has comma as a valid character in one of its fields (which can a free form text like DESCRIPTION)? The only alternative (easy way) I have tried to ...
0
votes
3answers
81 views

CSV dataset to text dataset

I have a csv dataset student.csv with 1000 rows and 10 columns. The sample data looks like this: s_id,s_name,s_dob,s_class,s_marks,s_parentname, 1,abc,1/1/18,5,49,def, 2,xyz,1/1/17,4,85,abc, ...
1
vote
2answers
52 views

match a string in two columns then print count AWK

I have a csv file that contains 21 columns. I want to search a string in two columns then count the total number of occurrences. sample of my dataset columnA columnB columnC columnD abc ...
2
votes
5answers
133 views

Filtering CSV dataset to find rows and store them in text file

I'm new to shell commands. I'm kinda struggling with this question based on a CSV dataset fbnews.csv. The CSV dataset looks like this: D,E,F, message, score, A,B,C, ID d,e,f, ...
0
votes
1answer
154 views

Counting unique values in all columns of a csv file

I have a CSV file, a.csv, which have 15000 rows and 16 columns. It have numbers, sentences, dates in different columns. It looks like this: A, B, C, D a b c d e f g h e i j k a m n k ...
2
votes
2answers
70 views

Finding a substring in a column of CSV file

I have a csv file with around 15000 lines and multiple columns. The data set looks like this: A,B,C, message D, E, F a,b,c,Hi how are you ? d,e,f a,b,c,Hi Hello.Are you ok? ...
1
vote
4answers
75 views

Given Pattern Replacement With Comma

Below follows test.txt contents, with generally 22 columns: BusinessDate,SourceSystemId,IceIndexId,IceIndexName,ComponentId,ComponentReferenceType,ComponentType,ComponentName,ComponentIssuerCIS,...
0
votes
2answers
68 views

extract columns from csv to a text file

It is a simple problem. I have a csv file with multiple columns I would like to extract 3 columns and save the output to a text file. sample of my dataset: page_id post_name link ...
0
votes
3answers
165 views

splitting a column in Bash based on '_'

Hello I have a similar problem to the one in this post But since I am new to Bash and my file is a bit different I could not modify and apply the answers to my code. I have a csv file with multiple ...
0
votes
2answers
50 views

CSV - Add quotes around missing fields

My CSV file looks like this - 1,2,3,4,5,6,7,8 "AK","87","86","81","83","78",, How do I transform it like this - 1,2,3,4,5,6,7,8 "AK","87","86","81","83","78","",""
1
vote
1answer
76 views

How to out the output of the command 'cal' to csv.?

How can I format the output of the command cal -y 2020 as a csv? I want to prepare the calendar in a table processor. Can this be done? I am using Slackware 14.2.
1
vote
1answer
250 views

create a folder from dynamic path location

I would like to create a folder which location is available in abc.csv file. abc.scv content the following fields id,path and file name we should read the file name and move the file in the path ...
1
vote
1answer
75 views

UNIX/SED - Wildcard in file name

File name is: file_201909091143.csv How to add a wildcard to pull all files that start with 'file' ? #!/bin/bash #cleans blank rows sed -i '/^$/d' file?.csv #wildcard missing UPDATE directory's ls: ...
0
votes
2answers
60 views

how separate each string in csv line to lines

we want to separate each string in csv line to lines example of line ( could be other line , with the same concept ) machine23,machine094,machine73,machine83 so we try this echo machine23,...
3
votes
2answers
104 views

Parsing csv file to filter rows basing on a matching set of characters from a column value

Consider the following csv file: A,3300 B,8440 B,8443 B,8444 C,304 C,404 M,5502 M,5511 The real csv file is really big (around 60,000 rows) but I only included a small version for ...
3
votes
3answers
42 views

Text Manipulation - Transpose column into rows depending on the value

I want to write a code to be able to manipulate text on my CSV file which contains the following: 71w - Rus,51200 71w - Phi,307200 71w - Ukr,307200 71w - Ukr,51200 71w - Mic,102400 71w - Mic,51200 ...
-1
votes
1answer
31 views

Merge files in different directories

I have 3 dirs with csv files with the same name /dir1/ex1.csv /dir2/ex1.csv /dir3/ex1.csv And i would like to merge the content in just one destination file (Just concat file content, since is a csv ...
-2
votes
2answers
66 views

Help with indexing a list of values

I need to track a huge log and more exactly a column. In this column I have some integer values in a rage between 103 and 17431. Example of original file 402 402 402 667 942 342 990 402 For every ...
0
votes
1answer
54 views

How to split CSV file by first column being a multiple of n

I have large csv file that looks like this: 1 , aaa, bbb, ... 1 , ccc, ddd, ... 2 , aaa, bbb, ... .. 10 , aaa, bbb, ... .. 1000 , aaa, bbb, ... I would like to split it into separate csv files ...
0
votes
3answers
38 views

I want to delete a column from a row which begins with a particular string in a csv file

I have a csv file: test_1,2,data,hi,cat test_2,3,4,5,6 test_1,3,7,8,9 I want to delete column 3 of the rows which begin with test_1. I used the cut command to delete column 3 but I do not know how ...
0
votes
1answer
66 views

How to write a csvcut script to cut column by header with multiple files?

Since csvcut (from csvkit) does not take more than a single file at a time, I need to write a script to process multiple files using it. The first parameter should be the delimiter, the second ...
2
votes
2answers
385 views

How can I categorize files in a directory based on their content?

I have a directory which contains a lot of CSV files. The CSV files have many columns the first of which is a timestamp (as number of seconds since the UNIX Epoch). I want to categorize files in the ...
0
votes
0answers
34 views

Large file processing performance

I did a simple performance test of csv processing a while ago, and want to share results with community, maybe you can point out, what test could be more precise and fair. Firstly i took out 42 MB ...
1
vote
3answers
524 views

Joining / merging CSV files that do not share all their headers / column

I have a directory with 1000s of files from a data recording system, and each file can have as many as 40,000 or more rows. The challenge is that sometimes data is not recorded from one or more ...
0
votes
1answer
35 views

using sed how to append N number of lines

I have the following csv file: hostname1,this is a test,001^M hostname2,this is a test,002^M hostname3,this is a test,003^M The first line is the proper format of the line. I want the other lines to ...
0
votes
1answer
32 views

AWK - multiply two columns and add result with trailing zeros

I have a csv file like this: Row 1,1,4.50, Row 2,8,0.30, Row 3,14,0.30, Row 4,2,2.50, Row 5,20,0.30, I want to multiply the second coulmn with the third and add the result as a fourth. I got this ...