Questions tagged [csv]
Files containing data arrange in a table, often with commas (hence Comma Separated Values), to separate columns. Rows are separated by newlines (but not all newlines are row separators as fields can be quoted to contain the separator newlines. Use this tag for full-fledged CSV data not the simpler case of one record per line or completely unquoted (use csv-simple for that kind of data).
-1
votes
2answers
64 views
Extract data from CSV file, where some fields have comma in field value
How to I parse the CSV file
file name : abc.csv (csv file)
The above file containing data like
abv,sfs,,hju,',',jkk
wff,fst,,rgr,',',rgr
ere,edf,erg,',',rgr,rgr
I have a requirement ...
-1
votes
0answers
26 views
Perl - working with hashes and arrays for CSV processing
I have a perl script I am working on that I am using to process a csv file. There are 55 columns, and for this example, let's say 10 rows. Some records need to be grouped by a particular column. Some ...
0
votes
1answer
30 views
How to concatenate several CSV files avoiding some lines at the beginning and in some at the end
I would like to join 5 CSV files.
In all of them, the data starts at the 8th row.
In two of them, I want to take out some lines at the end of the file.
From my precedent question "How to concatenate ...
0
votes
1answer
30 views
CSV processing: moving column/row value to different row where column value matches
I have a csv file with around 50 columns, can be anywhere between 20 and 100 rows.
The individual records have IDs, and some records can be in a group of 2. Essentially what I need to do is add an ...
0
votes
0answers
46 views
How can I parse through a log file, extract certain values and construct a tab or comma delimited text file?
I am trying to convert a pre-formatted section within a log file into a comma or tab delimited tabular view.
Here's a sample section contained in the log file:
Appliance-01> home
Appliance-01>...
0
votes
3answers
35 views
How to tell the difference between changed and added lines?
I have two csv files where I want to figure out the deleted, modified and added lines. (The first field is an unique id.)
The first file looks like:
0, WILL_BE_REMOVED
1, OLD
2, SAME
3, SAME
and a ...
1
vote
1answer
29 views
How can I call an external command from within Miller (mlr)’s DSL?
Suppose I have the following CSV:
$ cat test.csv
id,domain
1,foo.com
2,bar.com
Using mlr put, I can easily map any function over a field in the CSV, as long as I can define it in the Miller DSL. So, ...
0
votes
2answers
48 views
Adding headers to a CSV
I have an SQL command which fetches you an output like
70
138
200
This is stored in a csv and I have added headers to it but appears like
A,B,C
70
138
200
I wanted it to look like
A 70
B 138
C ...
0
votes
1answer
36 views
Find count of unique values in columns based on date in column 1
I have 3 comma separated quoted fields.
last crawled,linking page,domain
"Nov 17, 2018","https://allestoringen.be/problemen/bwin/antwerpen","allestoringen.be"
"Aug 11, 2017","http://casino.linkplek....
1
vote
2answers
34 views
Remove lines from a file based on patterns in another file which may partially match a particular column in first file
I have searched for similar answers but none solve the partial match problem.
Patterns file is file2 and lines to be removed are in file1.csv which is a pretty big file with far more columns than ...
0
votes
2answers
39 views
shell script read from csv column and search files
I am creating a shell script to take input from CSV files, having two rows ( One Column mentioning time and another file string ).. My script works when there is one row... But it isnot working in ...
0
votes
2answers
38 views
writing file names from subdirectories to a csv file in linux
Previously I have asked a very similar question but it was only for one subdirectory writing files in a subdirectory to a csv file and save it to parent directory in linux command line.
I have a ...
1
vote
2answers
52 views
writing files in a subdirectory to a csv file and save it to parent directory in linux command line
I have a directory called 'dir', and there is a subdirectory in it called 'subdir'. Subdir has many files in it. I would like write to list of these files on a CSV file and save it to in 'dir' ...
2
votes
1answer
63 views
How to write a file list with foreign names to a csv file in the terminal?
I have some files with names in foreign languages in my terminal folder. On the terminal when I do 'ls' i see the list on the right in the picture below.
When I do ls -t | tail -n +2, I see the real ...
0
votes
2answers
55 views
Getting file as rows and columns data matrix
I have gene.csv file which is about 1.3 GB in size and has got 300 columns and more than a million rows. it looks like following
id1 id2 id3 id4 count1 count2
S1001 450 ...
2
votes
2answers
70 views
Having trouble parsing a csv file from the commandline
I have a CSV file that I've been working on for most of the day and I'm not having any luck getting it parsed properly using a Regex with awk.
awk is not processing the Regex as expected.
Here are ...
2
votes
4answers
67 views
SED delete csv column if exists
I need to remove a column from my tabulated CSV file if this column exists.
My CSV file:
GENE REF ALT
AKT A G
AKT G G
Desired output:
if column REF exists delete this column
GENE ALT
AKT ...
1
vote
1answer
70 views
Data Cleaning Using Shellscript
I want a combined query suitable for selecting rows where lang = en i.e. language is equal to English and removing the other rows while I convert this file from JSON to CSV.
json2csv -i Downloads/30....
0
votes
0answers
37 views
AWK FS with CSV file [duplicate]
I have below file.CSV file,
"ID","CODE","RATE","AMOUNT"
"1","A","0",","1,2000.00"
"2","B","12,0.5","1300.00"
I am using below code with other logic but here, value for 3rd and column has comma in ...
0
votes
3answers
68 views
Changing CSV Column Layout
Does anyone have a good solution for taking a CSV file and adjusting the column layout?
So for example, I have a file with headers + data:
foo,bar,baz
1,2,3
But I want to change the column layout ...
1
vote
2answers
78 views
convert tab delimited data to , delimited records with double quoted fields “ ” [closed]
I want to convert tab delimited csv into " ", add empty space if value is empty
FirstName LastName Address1 Address2 City State ZIP
John1 ...
1
vote
1answer
24 views
Awk csv treatment output malformed line to stderr
I have a script that takes several CSV files from a folder and puts them together in a gzip file with one field extracted from the file name :
awk -F";" 'BEGIN { OFS = FS }
FNR == 1 { match(FILENAME,...
1
vote
1answer
56 views
Parsing CSV that has multiple line values with carriage returns
I have a CSV that I'm trying to remove the double quote values from using the following sed one-liner. Details about this can be found in my original post to figure that out
sed -i.bak 's/^"//g;s/","/...
0
votes
1answer
32 views
awk or bash to read IDs from CSV, match rows in second CSV, change value in 2nd column
I have two .csv
items-to-change.txt is a single column list of unique ids:
218294
222084
197931
198161
197432
201021
214009
items-by-location.txt has unique ids and location number (tab delimited)
...
0
votes
2answers
77 views
SED not removing last double quote from every line
I have a CSV that was exported with double quotes around every header and value and I need them gone but without removing double quotes that might actually exist as values. For example:
"HEADER1","...
4
votes
3answers
165 views
Add lines to files to make them equal length
I have a bunch of .csv files with N columns and different number of rows (lines). I would like to add as many empty lines ;...; (N semicolons) to make them the same length. I can get the length of the ...
1
vote
1answer
15 views
How to extract data from csv file with modification to data
For eg i have a file called test.csv containing
emp1,dep1
emp2,dep2
emp3,dep1
emp4,dep3
emp5,dep2
expected output is
emp1 works in dep1
emp2 works in dep2
.
.
emp5 works in dep2
-1
votes
1answer
28 views
Truncate long line and append indicating symbol [closed]
I have sample:
"this is a long title must in a doule quotes" this_is_a_solid_long_line_without_space_or_tabs
"this is a long title must in a doule quotes" is a long line in double quotes that ...
2
votes
1answer
23 views
Layout tab/spaces [closed]
I've read this thread https://unix.stackexchange.com/a/7718/256195, that only if var doesn't contain any tab/spaces but in my case it does contain spaces, like below example:
"this is a test" ...
0
votes
3answers
61 views
awk - Print all values with a suffix
I want to use awk to print all values in a delimited string with a static suffix appended to each one.
Input:
stack,over,flow
Delimiter:
Comma
Output:
stack suffix, over suffix, flow suffix
1
vote
3answers
40 views
Add a pattern before the delimiter
I'm looking for a command to add a pattern before or the delimiter.
My sting:
1 |Chris|ubuntu
Here my delimiter is| and I want to generate a string with this input.
Output
1 = ID and Chris = ...
2
votes
1answer
59 views
Extract the number of pages of PDFs recursively and dump them to a CSV
Apologies for the fairly trivial question but I am a Linux beginner.
I need to get a list of PDFs in various sub-folders of a directory structure and the number of pages of each PDF file. All saved ...
0
votes
1answer
29 views
Is the an elegant way to select rows in a file where a given column has a given value?
I have a file with a lot of lines like this
0 file:/home/knappen/somefilename.txt 7 0.2838356973995272 19 0.21823286052009455 18 0.10121158392434988 15 0....
0
votes
3answers
59 views
Organizing data by a header
I have two csv files.
File 1 contains header.
File 2 contains data
file 1 format : file1.csv
id,abc,xyz,aaa,bbb,ccc
file 2 format : file2.csv
id,source,value
1,abc,100
1,xyz,...
3
votes
6answers
696 views
Replace the third comma on each line with newline
I have these long IDs which consist of four sections:
AKJHGFGUIKL,OIUYT,KJHBTYUI,98765434567
RTYUIKHGFGH,TYUJI,TGHYJKJKLJKL,6789876
ETRYTUUI,YTYUIL,UIOKJHGFGH,34567898766
I want to put the numbers ...
0
votes
2answers
73 views
How to sort the data from CSV file by comparing data from text file using awk output it as a TXT file?
There are two files I am going to extract the data:
file1.txt:
Type Serial ID Element Hit_Possibility
Yasuo 19-2 19623 Hasaki
file2.csv:
Date,Name,Order,Hit Possibility
12-Aug,Ken,1,256
12-Aug,...
-1
votes
3answers
155 views
Field separator part of a column - incorrect parsing unix
I want to do a check the number of columns in a CSV file before processing it. The problem is that the delimiter (comma) also occurs in the text of some fields and because of that I cannot parse it ...
0
votes
2answers
101 views
How to remove duplicate items within the same line from a csv file?
I have a csv file with ~4000 lines, each one containing between 2 and 30 names separated by commas. The names are including titles (for example mr. X Adams or ms. Y Sanders). Some names exist multiple ...
1
vote
4answers
118 views
CSV file: Group and sum values based on first pattern of each value
I am using bash to create a script to find, group and sum fields in a CSV file. Each row will have comma-separated fields with each field following a similar convention. For each comma-separated field,...
3
votes
4answers
711 views
Bash: join data from two csv files
I have two csv files that contain various user data; they share one common field (username).
file A:
username ; Fullname ; mail
Bob ; Bob Hope ; bob@hope.org
file B:
username ; LastLogonTime ; ...
0
votes
1answer
98 views
Append lines from csv file to another based on column 1 matches
I have 2 .csv files, file1.csv and file2.csv. Say a line from file2.csv has the same 1st field as a line from file1.csv. I would like the full line from file1.csv to replace the full line from file2....
0
votes
3answers
101 views
AWK csv manipulation
I have a csv file containing something like this:
abc;!!!!;22
abc;!!!!;23
23;!!!!!;22
Now I want to delte every line that contains a 23 in the third column.
Since there is a possibility that there ...
0
votes
2answers
89 views
Rename a csv tab from UNIX bash script
I have CSV file that is generated from a bash script: UNIX_REPORT.sh. When you open CSV the file on the windGrr side with excel, the tab (or sheet) in the lower left corner is named UNIX_REPORT.sh.
...
0
votes
1answer
54 views
Interpolate Columnar Data
I am trying to find a way to interpolate between two lines of data in a CSV file, likely using awk. Right now, each line represents a data point at Hour 0 and Hour 6. I am looking to fill in missing ...
1
vote
3answers
87 views
Using AWK to add a new column with values to a csv file without creating empty new lines in between rows
I would like to add a new column to the end of a csv file and populating the column with values. I have used the following codes and it ends up showing new blank lines in between each record. Please ...
0
votes
2answers
80 views
Split datetime column in csv file into MM/YYYY and time column separately and write to the same csv
I am trying to use awk, split and then print but running into error
Input:
id day userId itemId
1 12/1/17 8:32 2232 tv
2 1/12/18 10:18 3232 fdfs
3 2/9/18 10:50 232 fdsf
...
-1
votes
3answers
131 views
remove the lines in CSV that contain specific pattern on the second field
How to remove the lines in .csv file that second field contain the word content?
example :
ams-hbase-log4j,content:\n#LicensedtotheApacheSoftwareFoundation(ASF)underone\n#...
1
vote
2answers
38 views
How to merge different columns in the same file?
I have three files, as shown below:
FILE 1:
DATE PGTO_CRED
20180801 50.00
20180802 150.00
20180803 130.00
20180804 110.00
20180805 200.00
FILE 2:
DATE PGTO_TOTAL
20180801 ...
1
vote
1answer
35 views
Save each line of a .csv file as a separate .csv [duplicate]
I have a .csv file with no headers and want to have each line saved as a separate .csv file with the new file name to be the value in column 2.
0
votes
0answers
48 views
file command gives incorrect encoding type
I have a csv file in hand. When I ran 'file -i filename', it shows that it's encoded as us-ascii. But when I ran cat filename | csvcut -t -e us-ascii, I got an error:
"Your file is not "us-ascii" ...
