Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
The results are in! See what nearly 90,000 developers picked as their most loved, dreaded, and desired coding languages and more in the 2019 Developer Survey.

Questions tagged [csv]

Files containing data arrange in a table, often with commas (hence Comma Separated Values), to separate columns. Rows are separated by newlines (but not all newlines are row separators as fields can be quoted to contain the separator newlines. Use this tag for full-fledged CSV data not the simpler case of one record per line or completely unquoted (use csv-simple for that kind of data).

-4
votes
3answers
54 views

Find entry with maximum value in one column [closed]

In the below example I want to only display UID and Process Name in the Access.csv file which consume the most memory. Write a shell scripts for this goal. Access.csv looks like UID Process ...
1
vote
2answers
52 views

Remove duplicates from particular columns

I have a file in below format, where columns are separated with comma. [1], Value1, UAC, AB [2.2], Check1, BOH D2A D2A BOH, SD [63], name2, MFB MFB, k ... I want ...
1
vote
3answers
66 views

How to compare two CSV files and display unique records?

I have two csv file as below, i want to compare file1 with file2 and if records present in file2 that complete row should remove from file1. field to compare here is ID and in original file its at ...
0
votes
5answers
56 views

Getting values from comma seperated file

I have a file structured like this: [IP ADRESS], [ID-Number], [Name] [IP ADRESS], [ID-Number], [Name] [IP ADRESS], [ID-Number], [Name] ... (The actual values are not in brackets) How do I get (...
2
votes
3answers
78 views

Unix file with newlines within quotes

I've got a CSV weird file with quotes within quotes and newlines and what not in one single column. Now I need to identify that column with "newlines" as one column and replace newlines with some ...
-1
votes
0answers
59 views

Compare CSV file with while read loop in bash

I have two csv file like this : File 1 : Monday,XXXXX,XXXXX,XXXXX XXXXX,XXXXX,XXXXX,Friday XXXXX,XXXXX,XXXXX,Thursday XXXXX,Wednesday,XXXXX,XXXXX File 2 : Monday,XXXXX,XXXXX,XXXXX XXXXX,XXXXX,...
4
votes
3answers
84 views

Shell script to extract data from a list of files and saving it as a csv

I'm on CentOS. I have a list of files that I want to read, extract data from them and organize it as a csv file. The log files text format is: ... {"name":"test-api","hostname":"ci47","pid":3202,"...
-5
votes
2answers
57 views

How to parse through csv file with bash

I have CSV file with IP addresses and open ports: IP,1,3,4,6,7,9,13,17,19,20,21,22,23,24,25,26 1.1.1.2,,,,,,,,,,,open,,,,, 1.1.1.3,,,,,,,,,,,open,,,,, 1.1.1.4,,open ,open,,,,,,,,open,,,,, 1.1.1.5,,,,,...
0
votes
4answers
67 views

Sorting the CSV based on Unix timestamp

I have a large number of CSV files, each file having multiple numeric columns. The first column is Unix timestamp. How can I sort such file? Thank you. An example row of the file as below: ...
-1
votes
2answers
50 views

filtering output

for i in $(cat HPE_FRAMES_IP); do ssh Reports@$i showsys | awk '{print $2,$5}' | column -t ; ssh Reports@$i showpd -degraded -failed | awk '{print $1,$2,$3,$5}' ; done | grep -v "MiB" | grep -v "...
2
votes
0answers
29 views

JSON -> csv creating header line and padding header if found empty field

I have a programme in bash that get JSONline files with several million of these object per line (See source) { "company_number": "09626947", "data": { "address": { "address_line_1": "...
0
votes
1answer
41 views

csv filter on linux bash

Problem: I have a CSV file with 5 columns (1st column is a string and the other 4 are int). I would like to filter based on the third column, Revenues, the largest on the top and smallest on the ...
0
votes
2answers
85 views

Splitting text file into CSV with multiple delimiters in bash?

Trying to parse a text file into CSV. Catch is that I have multiple delimiters currently, that I'd ideally like to use as column headers but can strip out of the csv results. Ideally would rather ...
1
vote
2answers
41 views

How could I (painlessly) split or reverse “Last, First” within a record in Miller?

I have a tab-delimited file where one of the columns is in the format "LastName, FirstName". What I want to do is split that record out into two separate columns, last, and first, use cut or some ...
-3
votes
2answers
33 views

pipeline delimiter and some fields having comma those should be placed in double quotes and it supposed to be taken as single filed [closed]

i Have one csv file. input. India|Andhra|asd,ged,ijn|telangana|41657|hgjd,dfef,edf|dfd output India,Andhra,"asd,ged,ijn",telangana,41657,"hgjd,dfef,edf",dfd
9
votes
1answer
544 views

awk unexpectedly removes dot from string

I want to add a column (2nd place) in a .csv file and I want the values of that column to be strings and to be quoted; The following command does add the column but without quotes: awk -F"," 'BEGIN {...
1
vote
1answer
51 views

Using NF with FPAT – what am I doing wrong?

When NF is used with FPAT regex, the comma is considered a field. I prefer using NF and FPAT: 1) NF – to limit the output to the actual number of fields for the record 2) FPAT – to handle an ...
1
vote
4answers
75 views

Sorting elements of a CSV file

I have a csv file with seven numbers per line like this: 1083,20,28,42,23,10,43 1084,20,5,29,59,40,33 1085,39,50,21,12,40,55 1086,45,4,6,23,10,2 1087,36,46,28,32,3,20 I want to keep the first number ...
0
votes
0answers
36 views

How to store output(data) in csv format? [duplicate]

I am making process monitoring application. Now my need is To get process related data(pid, process name, cpu usage, memory usage and virtual memory usage). After completing first step I want to ...
1
vote
1answer
37 views

I have a pipe separated file,read file row wise and search for unclosed quotes(") in each row and close that

Below is a row and i need to close "20 with "20" 2019-02-15T10:25:19+0100|ttt|werewrw|erewr|fddsfsdf|dfsdf|07|2ee7|1b65d04|"20|19|-02-|15t10:03|45435435|765767||SIP;rr=200;text="hyt"|
0
votes
5answers
49 views

Output to last column depending on first column if is greater, less than or equal to 0

I have a large .csv (comma separated, with over 90,000 rows), the format looks like this: xcoord,m1,m2,m3 9,TRUE,FALSE,FALSE -1,TRUE,FALSE,FALSE 0,TRUE,FALSE,FALSE I was able to add an extra column ...
-1
votes
2answers
76 views

Extract data from CSV file, where some fields have comma in field value

How to I parse the CSV file file name : abc.csv (csv file) The above file containing data like abv,sfs,,hju,',',jkk wff,fst,,rgr,',',rgr ere,edf,erg,',',rgr,rgr I have a requirement ...
0
votes
1answer
35 views

How to concatenate several CSV files avoiding some lines at the beginning and in some at the end

I would like to join 5 CSV files. In all of them, the data starts at the 8th row. In two of them, I want to take out some lines at the end of the file. From my precedent question "How to concatenate ...
0
votes
1answer
37 views

CSV processing: moving column/row value to different row where column value matches

I have a csv file with around 50 columns, can be anywhere between 20 and 100 rows. The individual records have IDs, and some records can be in a group of 2. Essentially what I need to do is add an ...
0
votes
0answers
54 views

How can I parse through a log file, extract certain values and construct a tab or comma delimited text file?

I am trying to convert a pre-formatted section within a log file into a comma or tab delimited tabular view. Here's a sample section contained in the log file: Appliance-01> home Appliance-01>...
0
votes
3answers
40 views

How to tell the difference between changed and added lines?

I have two csv files where I want to figure out the deleted, modified and added lines. (The first field is an unique id.) The first file looks like: 0, WILL_BE_REMOVED 1, OLD 2, SAME 3, SAME and a ...
2
votes
1answer
44 views

How can I call an external command from within Miller (mlr)’s DSL?

Suppose I have the following CSV: $ cat test.csv id,domain 1,foo.com 2,bar.com Using mlr put, I can easily map any function over a field in the CSV, as long as I can define it in the Miller DSL. So, ...
0
votes
2answers
51 views

Adding headers to a CSV

I have an SQL command which fetches you an output like 70 138 200 This is stored in a csv and I have added headers to it but appears like A,B,C 70 138 200 I wanted it to look like A 70 B 138 C ...
0
votes
1answer
38 views

Find count of unique values in columns based on date in column 1

I have 3 comma separated quoted fields. last crawled,linking page,domain "Nov 17, 2018","https://allestoringen.be/problemen/bwin/antwerpen","allestoringen.be" "Aug 11, 2017","http://casino.linkplek....
1
vote
2answers
44 views

Remove lines from a file based on patterns in another file which may partially match a particular column in first file

I have searched for similar answers but none solve the partial match problem. Patterns file is file2 and lines to be removed are in file1.csv which is a pretty big file with far more columns than ...
0
votes
2answers
366 views

shell script read from csv column and search files

I am creating a shell script to take input from CSV files, having two rows ( One Column mentioning time and another file string ).. My script works when there is one row... But it isnot working in ...
0
votes
2answers
42 views

writing file names from subdirectories to a csv file in linux

Previously I have asked a very similar question but it was only for one subdirectory writing files in a subdirectory to a csv file and save it to parent directory in linux command line. I have a ...
1
vote
2answers
66 views

writing files in a subdirectory to a csv file and save it to parent directory in linux command line

I have a directory called 'dir', and there is a subdirectory in it called 'subdir'. Subdir has many files in it. I would like write to list of these files on a CSV file and save it to in 'dir' ...
2
votes
1answer
68 views

How to write a file list with foreign names to a csv file in the terminal?

I have some files with names in foreign languages in my terminal folder. On the terminal when I do 'ls' i see the list on the right in the picture below. When I do ls -t | tail -n +2, I see the real ...
0
votes
2answers
59 views

Getting file as rows and columns data matrix

I have gene.csv file which is about 1.3 GB in size and has got 300 columns and more than a million rows. it looks like following id1 id2 id3 id4 count1 count2 S1001 450 ...
2
votes
2answers
86 views

Having trouble parsing a csv file from the commandline

I have a CSV file that I've been working on for most of the day and I'm not having any luck getting it parsed properly using a Regex with awk. awk is not processing the Regex as expected. Here are ...
2
votes
4answers
75 views

SED delete csv column if exists

I need to remove a column from my tabulated CSV file if this column exists. My CSV file: GENE REF ALT AKT A G AKT G G Desired output: if column REF exists delete this column GENE ALT AKT ...
1
vote
1answer
74 views

Data Cleaning Using Shellscript

I want a combined query suitable for selecting rows where lang = en i.e. language is equal to English and removing the other rows while I convert this file from JSON to CSV. json2csv -i Downloads/30....
0
votes
0answers
42 views

AWK FS with CSV file [duplicate]

I have below file.CSV file, "ID","CODE","RATE","AMOUNT" "1","A","0",","1,2000.00" "2","B","12,0.5","1300.00" I am using below code with other logic but here, value for 3rd and column has comma in ...
0
votes
3answers
82 views

Changing CSV Column Layout

Does anyone have a good solution for taking a CSV file and adjusting the column layout? So for example, I have a file with headers + data: foo,bar,baz 1,2,3 But I want to change the column layout ...
1
vote
2answers
82 views

convert tab delimited data to , delimited records with double quoted fields “ ” [closed]

I want to convert tab delimited csv into " ", add empty space if value is empty FirstName LastName Address1 Address2 City State ZIP John1 ...
1
vote
1answer
29 views

Awk csv treatment output malformed line to stderr

I have a script that takes several CSV files from a folder and puts them together in a gzip file with one field extracted from the file name : awk -F";" 'BEGIN { OFS = FS } FNR == 1 { match(FILENAME,...
1
vote
1answer
62 views

Parsing CSV that has multiple line values with carriage returns

I have a CSV that I'm trying to remove the double quote values from using the following sed one-liner. Details about this can be found in my original post to figure that out sed -i.bak 's/^"//g;s/","/...
0
votes
1answer
38 views

awk or bash to read IDs from CSV, match rows in second CSV, change value in 2nd column

I have two .csv items-to-change.txt is a single column list of unique ids: 218294 222084 197931 198161 197432 201021 214009 items-by-location.txt has unique ids and location number (tab delimited) ...
0
votes
2answers
93 views

SED not removing last double quote from every line

I have a CSV that was exported with double quotes around every header and value and I need them gone but without removing double quotes that might actually exist as values. For example: "HEADER1","...
5
votes
3answers
172 views

Add lines to files to make them equal length

I have a bunch of .csv files with N columns and different number of rows (lines). I would like to add as many empty lines ;...; (N semicolons) to make them the same length. I can get the length of the ...
1
vote
1answer
17 views

How to extract data from csv file with modification to data

For eg i have a file called test.csv containing emp1,dep1 emp2,dep2 emp3,dep1 emp4,dep3 emp5,dep2 expected output is emp1 works in dep1 emp2 works in dep2 . . emp5 works in dep2
-1
votes
1answer
29 views

Truncate long line and append indicating symbol [closed]

I have sample: "this is a long title must in a doule quotes" this_is_a_solid_long_line_without_space_or_tabs "this is a long title must in a doule quotes" is a long line in double quotes that ...
2
votes
1answer
25 views

Layout tab/spaces [closed]

I've read this thread https://unix.stackexchange.com/a/7718/256195, that only if var doesn't contain any tab/spaces but in my case it does contain spaces, like below example: "this is a test" ...
0
votes
3answers
153 views

awk - Print all values with a suffix

I want to use awk to print all values in a delimited string with a static suffix appended to each one. Input: stack,over,flow Delimiter: Comma Output: stack suffix, over suffix, flow suffix