Questions tagged [csv]
Files containing data arrange in a table, often with commas (hence Comma Separated Values), to separate columns. Rows are separated by newlines (but not all newlines are row separators as fields can be quoted to contain the separator newlines. Use this tag for full-fledged CSV data not the simpler case of one record per line or completely unquoted (use csv-simple for that kind of data).
-4
votes
3answers
54 views
Find entry with maximum value in one column [closed]
In the below example I want to only display UID and Process Name in the Access.csv file which consume the most memory. Write a shell scripts for this goal.
Access.csv looks like
UID Process ...
1
vote
2answers
52 views
Remove duplicates from particular columns
I have a file in below format, where columns are separated with comma.
[1], Value1, UAC, AB
[2.2], Check1, BOH D2A D2A BOH, SD
[63], name2, MFB MFB, k
...
I want ...
1
vote
3answers
66 views
How to compare two CSV files and display unique records?
I have two csv file as below, i want to compare file1 with file2 and if records present in file2 that complete row should remove from file1. field to compare here is ID and in original file its at ...
0
votes
5answers
56 views
Getting values from comma seperated file
I have a file structured like this:
[IP ADRESS], [ID-Number], [Name]
[IP ADRESS], [ID-Number], [Name]
[IP ADRESS], [ID-Number], [Name]
...
(The actual values are not in brackets)
How do I get (...
2
votes
3answers
78 views
Unix file with newlines within quotes
I've got a CSV weird file with quotes within quotes and newlines and what not in one single column. Now I need to identify that column with "newlines" as one column and replace newlines with some ...
-1
votes
0answers
59 views
Compare CSV file with while read loop in bash
I have two csv file like this :
File 1 :
Monday,XXXXX,XXXXX,XXXXX
XXXXX,XXXXX,XXXXX,Friday
XXXXX,XXXXX,XXXXX,Thursday
XXXXX,Wednesday,XXXXX,XXXXX
File 2 :
Monday,XXXXX,XXXXX,XXXXX
XXXXX,XXXXX,...
4
votes
3answers
84 views
Shell script to extract data from a list of files and saving it as a csv
I'm on CentOS. I have a list of files that I want to read, extract data from them and organize it as a csv file.
The log files text format is:
...
{"name":"test-api","hostname":"ci47","pid":3202,"...
-5
votes
2answers
57 views
How to parse through csv file with bash
I have CSV file with IP addresses and open ports:
IP,1,3,4,6,7,9,13,17,19,20,21,22,23,24,25,26
1.1.1.2,,,,,,,,,,,open,,,,,
1.1.1.3,,,,,,,,,,,open,,,,,
1.1.1.4,,open ,open,,,,,,,,open,,,,,
1.1.1.5,,,,,...
0
votes
4answers
67 views
Sorting the CSV based on Unix timestamp
I have a large number of CSV files, each file having multiple numeric columns. The first column is Unix timestamp. How can I sort such file? Thank you.
An example row of the file as below:
...
-1
votes
2answers
50 views
filtering output
for i in $(cat HPE_FRAMES_IP); do ssh Reports@$i showsys | awk '{print $2,$5}' | column -t ; ssh Reports@$i showpd -degraded -failed | awk '{print $1,$2,$3,$5}' ; done | grep -v "MiB" | grep -v "...
2
votes
0answers
29 views
JSON -> csv creating header line and padding header if found empty field
I have a programme in bash that get JSONline files with several million of these object per line (See source)
{
"company_number": "09626947",
"data": {
"address": {
"address_line_1": "...
0
votes
1answer
41 views
csv filter on linux bash
Problem:
I have a CSV file with 5 columns (1st column is a string and the other 4 are int). I would like to filter based on the third column, Revenues, the largest on the top and smallest on the ...
0
votes
2answers
85 views
Splitting text file into CSV with multiple delimiters in bash?
Trying to parse a text file into CSV. Catch is that I have multiple delimiters currently, that I'd ideally like to use as column headers but can strip out of the csv results. Ideally would rather ...
1
vote
2answers
41 views
How could I (painlessly) split or reverse “Last, First” within a record in Miller?
I have a tab-delimited file where one of the columns is in the format "LastName, FirstName". What I want to do is split that record out into two separate columns, last, and first, use cut or some ...
-3
votes
2answers
33 views
pipeline delimiter and some fields having comma those should be placed in double quotes and it supposed to be taken as single filed [closed]
i Have one csv file.
input.
India|Andhra|asd,ged,ijn|telangana|41657|hgjd,dfef,edf|dfd
output
India,Andhra,"asd,ged,ijn",telangana,41657,"hgjd,dfef,edf",dfd
9
votes
1answer
544 views
awk unexpectedly removes dot from string
I want to add a column (2nd place) in a .csv file and I want the values of that column to be strings and to be quoted;
The following command does add the column but without quotes:
awk -F"," 'BEGIN {...
1
vote
1answer
51 views
Using NF with FPAT – what am I doing wrong?
When NF is used with FPAT regex, the comma is considered a field. I prefer using NF and FPAT:
1) NF – to limit the output to the actual number of fields for the record
2) FPAT – to handle an ...
1
vote
4answers
75 views
Sorting elements of a CSV file
I have a csv file with seven numbers per line like this:
1083,20,28,42,23,10,43
1084,20,5,29,59,40,33
1085,39,50,21,12,40,55
1086,45,4,6,23,10,2
1087,36,46,28,32,3,20
I want to keep the first number ...
0
votes
0answers
36 views
How to store output(data) in csv format? [duplicate]
I am making process monitoring application. Now my need is
To get process related data(pid, process name, cpu usage, memory usage and virtual memory usage).
After completing first step I want to ...
1
vote
1answer
37 views
I have a pipe separated file,read file row wise and search for unclosed quotes(") in each row and close that
Below is a row and i need to close "20 with "20"
2019-02-15T10:25:19+0100|ttt|werewrw|erewr|fddsfsdf|dfsdf|07|2ee7|1b65d04|"20|19|-02-|15t10:03|45435435|765767||SIP;rr=200;text="hyt"|
0
votes
5answers
49 views
Output to last column depending on first column if is greater, less than or equal to 0
I have a large .csv (comma separated, with over 90,000 rows), the format looks like this:
xcoord,m1,m2,m3
9,TRUE,FALSE,FALSE
-1,TRUE,FALSE,FALSE
0,TRUE,FALSE,FALSE
I was able to add an extra column ...
-1
votes
2answers
76 views
Extract data from CSV file, where some fields have comma in field value
How to I parse the CSV file
file name : abc.csv (csv file)
The above file containing data like
abv,sfs,,hju,',',jkk
wff,fst,,rgr,',',rgr
ere,edf,erg,',',rgr,rgr
I have a requirement ...
0
votes
1answer
35 views
How to concatenate several CSV files avoiding some lines at the beginning and in some at the end
I would like to join 5 CSV files.
In all of them, the data starts at the 8th row.
In two of them, I want to take out some lines at the end of the file.
From my precedent question "How to concatenate ...
0
votes
1answer
37 views
CSV processing: moving column/row value to different row where column value matches
I have a csv file with around 50 columns, can be anywhere between 20 and 100 rows.
The individual records have IDs, and some records can be in a group of 2. Essentially what I need to do is add an ...
0
votes
0answers
54 views
How can I parse through a log file, extract certain values and construct a tab or comma delimited text file?
I am trying to convert a pre-formatted section within a log file into a comma or tab delimited tabular view.
Here's a sample section contained in the log file:
Appliance-01> home
Appliance-01>...
0
votes
3answers
40 views
How to tell the difference between changed and added lines?
I have two csv files where I want to figure out the deleted, modified and added lines. (The first field is an unique id.)
The first file looks like:
0, WILL_BE_REMOVED
1, OLD
2, SAME
3, SAME
and a ...
2
votes
1answer
44 views
How can I call an external command from within Miller (mlr)’s DSL?
Suppose I have the following CSV:
$ cat test.csv
id,domain
1,foo.com
2,bar.com
Using mlr put, I can easily map any function over a field in the CSV, as long as I can define it in the Miller DSL. So, ...
0
votes
2answers
51 views
Adding headers to a CSV
I have an SQL command which fetches you an output like
70
138
200
This is stored in a csv and I have added headers to it but appears like
A,B,C
70
138
200
I wanted it to look like
A 70
B 138
C ...
0
votes
1answer
38 views
Find count of unique values in columns based on date in column 1
I have 3 comma separated quoted fields.
last crawled,linking page,domain
"Nov 17, 2018","https://allestoringen.be/problemen/bwin/antwerpen","allestoringen.be"
"Aug 11, 2017","http://casino.linkplek....
1
vote
2answers
44 views
Remove lines from a file based on patterns in another file which may partially match a particular column in first file
I have searched for similar answers but none solve the partial match problem.
Patterns file is file2 and lines to be removed are in file1.csv which is a pretty big file with far more columns than ...
0
votes
2answers
366 views
shell script read from csv column and search files
I am creating a shell script to take input from CSV files, having two rows ( One Column mentioning time and another file string ).. My script works when there is one row... But it isnot working in ...
0
votes
2answers
42 views
writing file names from subdirectories to a csv file in linux
Previously I have asked a very similar question but it was only for one subdirectory writing files in a subdirectory to a csv file and save it to parent directory in linux command line.
I have a ...
1
vote
2answers
66 views
writing files in a subdirectory to a csv file and save it to parent directory in linux command line
I have a directory called 'dir', and there is a subdirectory in it called 'subdir'. Subdir has many files in it. I would like write to list of these files on a CSV file and save it to in 'dir' ...
2
votes
1answer
68 views
How to write a file list with foreign names to a csv file in the terminal?
I have some files with names in foreign languages in my terminal folder. On the terminal when I do 'ls' i see the list on the right in the picture below.
When I do ls -t | tail -n +2, I see the real ...
0
votes
2answers
59 views
Getting file as rows and columns data matrix
I have gene.csv file which is about 1.3 GB in size and has got 300 columns and more than a million rows. it looks like following
id1 id2 id3 id4 count1 count2
S1001 450 ...
2
votes
2answers
86 views
Having trouble parsing a csv file from the commandline
I have a CSV file that I've been working on for most of the day and I'm not having any luck getting it parsed properly using a Regex with awk.
awk is not processing the Regex as expected.
Here are ...
2
votes
4answers
75 views
SED delete csv column if exists
I need to remove a column from my tabulated CSV file if this column exists.
My CSV file:
GENE REF ALT
AKT A G
AKT G G
Desired output:
if column REF exists delete this column
GENE ALT
AKT ...
1
vote
1answer
74 views
Data Cleaning Using Shellscript
I want a combined query suitable for selecting rows where lang = en i.e. language is equal to English and removing the other rows while I convert this file from JSON to CSV.
json2csv -i Downloads/30....
0
votes
0answers
42 views
AWK FS with CSV file [duplicate]
I have below file.CSV file,
"ID","CODE","RATE","AMOUNT"
"1","A","0",","1,2000.00"
"2","B","12,0.5","1300.00"
I am using below code with other logic but here, value for 3rd and column has comma in ...
0
votes
3answers
82 views
Changing CSV Column Layout
Does anyone have a good solution for taking a CSV file and adjusting the column layout?
So for example, I have a file with headers + data:
foo,bar,baz
1,2,3
But I want to change the column layout ...
1
vote
2answers
82 views
convert tab delimited data to , delimited records with double quoted fields “ ” [closed]
I want to convert tab delimited csv into " ", add empty space if value is empty
FirstName LastName Address1 Address2 City State ZIP
John1 ...
1
vote
1answer
29 views
Awk csv treatment output malformed line to stderr
I have a script that takes several CSV files from a folder and puts them together in a gzip file with one field extracted from the file name :
awk -F";" 'BEGIN { OFS = FS }
FNR == 1 { match(FILENAME,...
1
vote
1answer
62 views
Parsing CSV that has multiple line values with carriage returns
I have a CSV that I'm trying to remove the double quote values from using the following sed one-liner. Details about this can be found in my original post to figure that out
sed -i.bak 's/^"//g;s/","/...
0
votes
1answer
38 views
awk or bash to read IDs from CSV, match rows in second CSV, change value in 2nd column
I have two .csv
items-to-change.txt is a single column list of unique ids:
218294
222084
197931
198161
197432
201021
214009
items-by-location.txt has unique ids and location number (tab delimited)
...
0
votes
2answers
93 views
SED not removing last double quote from every line
I have a CSV that was exported with double quotes around every header and value and I need them gone but without removing double quotes that might actually exist as values. For example:
"HEADER1","...
5
votes
3answers
172 views
Add lines to files to make them equal length
I have a bunch of .csv files with N columns and different number of rows (lines). I would like to add as many empty lines ;...; (N semicolons) to make them the same length. I can get the length of the ...
1
vote
1answer
17 views
How to extract data from csv file with modification to data
For eg i have a file called test.csv containing
emp1,dep1
emp2,dep2
emp3,dep1
emp4,dep3
emp5,dep2
expected output is
emp1 works in dep1
emp2 works in dep2
.
.
emp5 works in dep2
-1
votes
1answer
29 views
Truncate long line and append indicating symbol [closed]
I have sample:
"this is a long title must in a doule quotes" this_is_a_solid_long_line_without_space_or_tabs
"this is a long title must in a doule quotes" is a long line in double quotes that ...
2
votes
1answer
25 views
Layout tab/spaces [closed]
I've read this thread https://unix.stackexchange.com/a/7718/256195, that only if var doesn't contain any tab/spaces but in my case it does contain spaces, like below example:
"this is a test" ...
0
votes
3answers
153 views
awk - Print all values with a suffix
I want to use awk to print all values in a delimited string with a static suffix appended to each one.
Input:
stack,over,flow
Delimiter:
Comma
Output:
stack suffix, over suffix, flow suffix
