Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [csv]

Files containing data arrange in a table, often with commas (hence Comma Separated Values), to separate columns. Rows are separated by newlines (but not all newlines are row separators as fields can be quoted to contain the separator newlines. Use this tag for full-fledged CSV data not the simpler case of one record per line or completely unquoted (use csv-simple for that kind of data).

-1
votes
2answers
64 views

Extract data from CSV file, where some fields have comma in field value

How to I parse the CSV file file name : abc.csv (csv file) The above file containing data like abv,sfs,,hju,',',jkk wff,fst,,rgr,',',rgr ere,edf,erg,',',rgr,rgr I have a requirement ...
-1
votes
0answers
26 views

Perl - working with hashes and arrays for CSV processing

I have a perl script I am working on that I am using to process a csv file. There are 55 columns, and for this example, let's say 10 rows. Some records need to be grouped by a particular column. Some ...
0
votes
1answer
30 views

How to concatenate several CSV files avoiding some lines at the beginning and in some at the end

I would like to join 5 CSV files. In all of them, the data starts at the 8th row. In two of them, I want to take out some lines at the end of the file. From my precedent question "How to concatenate ...
0
votes
1answer
30 views

CSV processing: moving column/row value to different row where column value matches

I have a csv file with around 50 columns, can be anywhere between 20 and 100 rows. The individual records have IDs, and some records can be in a group of 2. Essentially what I need to do is add an ...
0
votes
0answers
46 views

How can I parse through a log file, extract certain values and construct a tab or comma delimited text file?

I am trying to convert a pre-formatted section within a log file into a comma or tab delimited tabular view. Here's a sample section contained in the log file: Appliance-01> home Appliance-01>...
0
votes
3answers
35 views

How to tell the difference between changed and added lines?

I have two csv files where I want to figure out the deleted, modified and added lines. (The first field is an unique id.) The first file looks like: 0, WILL_BE_REMOVED 1, OLD 2, SAME 3, SAME and a ...
1
vote
1answer
29 views

How can I call an external command from within Miller (mlr)’s DSL?

Suppose I have the following CSV: $ cat test.csv id,domain 1,foo.com 2,bar.com Using mlr put, I can easily map any function over a field in the CSV, as long as I can define it in the Miller DSL. So, ...
0
votes
2answers
48 views

Adding headers to a CSV

I have an SQL command which fetches you an output like 70 138 200 This is stored in a csv and I have added headers to it but appears like A,B,C 70 138 200 I wanted it to look like A 70 B 138 C ...
0
votes
1answer
36 views

Find count of unique values in columns based on date in column 1

I have 3 comma separated quoted fields. last crawled,linking page,domain "Nov 17, 2018","https://allestoringen.be/problemen/bwin/antwerpen","allestoringen.be" "Aug 11, 2017","http://casino.linkplek....
1
vote
2answers
34 views

Remove lines from a file based on patterns in another file which may partially match a particular column in first file

I have searched for similar answers but none solve the partial match problem. Patterns file is file2 and lines to be removed are in file1.csv which is a pretty big file with far more columns than ...
0
votes
2answers
39 views

shell script read from csv column and search files

I am creating a shell script to take input from CSV files, having two rows ( One Column mentioning time and another file string ).. My script works when there is one row... But it isnot working in ...
0
votes
2answers
38 views

writing file names from subdirectories to a csv file in linux

Previously I have asked a very similar question but it was only for one subdirectory writing files in a subdirectory to a csv file and save it to parent directory in linux command line. I have a ...
1
vote
2answers
52 views

writing files in a subdirectory to a csv file and save it to parent directory in linux command line

I have a directory called 'dir', and there is a subdirectory in it called 'subdir'. Subdir has many files in it. I would like write to list of these files on a CSV file and save it to in 'dir' ...
2
votes
1answer
63 views

How to write a file list with foreign names to a csv file in the terminal?

I have some files with names in foreign languages in my terminal folder. On the terminal when I do 'ls' i see the list on the right in the picture below. When I do ls -t | tail -n +2, I see the real ...
0
votes
2answers
55 views

Getting file as rows and columns data matrix

I have gene.csv file which is about 1.3 GB in size and has got 300 columns and more than a million rows. it looks like following id1 id2 id3 id4 count1 count2 S1001 450 ...
2
votes
2answers
70 views

Having trouble parsing a csv file from the commandline

I have a CSV file that I've been working on for most of the day and I'm not having any luck getting it parsed properly using a Regex with awk. awk is not processing the Regex as expected. Here are ...
2
votes
4answers
67 views

SED delete csv column if exists

I need to remove a column from my tabulated CSV file if this column exists. My CSV file: GENE REF ALT AKT A G AKT G G Desired output: if column REF exists delete this column GENE ALT AKT ...
1
vote
1answer
70 views

Data Cleaning Using Shellscript

I want a combined query suitable for selecting rows where lang = en i.e. language is equal to English and removing the other rows while I convert this file from JSON to CSV. json2csv -i Downloads/30....
0
votes
0answers
37 views

AWK FS with CSV file [duplicate]

I have below file.CSV file, "ID","CODE","RATE","AMOUNT" "1","A","0",","1,2000.00" "2","B","12,0.5","1300.00" I am using below code with other logic but here, value for 3rd and column has comma in ...
0
votes
3answers
68 views

Changing CSV Column Layout

Does anyone have a good solution for taking a CSV file and adjusting the column layout? So for example, I have a file with headers + data: foo,bar,baz 1,2,3 But I want to change the column layout ...
1
vote
2answers
78 views

convert tab delimited data to , delimited records with double quoted fields “ ” [closed]

I want to convert tab delimited csv into " ", add empty space if value is empty FirstName LastName Address1 Address2 City State ZIP John1 ...
1
vote
1answer
24 views

Awk csv treatment output malformed line to stderr

I have a script that takes several CSV files from a folder and puts them together in a gzip file with one field extracted from the file name : awk -F";" 'BEGIN { OFS = FS } FNR == 1 { match(FILENAME,...
1
vote
1answer
56 views

Parsing CSV that has multiple line values with carriage returns

I have a CSV that I'm trying to remove the double quote values from using the following sed one-liner. Details about this can be found in my original post to figure that out sed -i.bak 's/^"//g;s/","/...
0
votes
1answer
32 views

awk or bash to read IDs from CSV, match rows in second CSV, change value in 2nd column

I have two .csv items-to-change.txt is a single column list of unique ids: 218294 222084 197931 198161 197432 201021 214009 items-by-location.txt has unique ids and location number (tab delimited) ...
0
votes
2answers
77 views

SED not removing last double quote from every line

I have a CSV that was exported with double quotes around every header and value and I need them gone but without removing double quotes that might actually exist as values. For example: "HEADER1","...
4
votes
3answers
165 views

Add lines to files to make them equal length

I have a bunch of .csv files with N columns and different number of rows (lines). I would like to add as many empty lines ;...; (N semicolons) to make them the same length. I can get the length of the ...
1
vote
1answer
15 views

How to extract data from csv file with modification to data

For eg i have a file called test.csv containing emp1,dep1 emp2,dep2 emp3,dep1 emp4,dep3 emp5,dep2 expected output is emp1 works in dep1 emp2 works in dep2 . . emp5 works in dep2
-1
votes
1answer
28 views

Truncate long line and append indicating symbol [closed]

I have sample: "this is a long title must in a doule quotes" this_is_a_solid_long_line_without_space_or_tabs "this is a long title must in a doule quotes" is a long line in double quotes that ...
2
votes
1answer
23 views

Layout tab/spaces [closed]

I've read this thread https://unix.stackexchange.com/a/7718/256195, that only if var doesn't contain any tab/spaces but in my case it does contain spaces, like below example: "this is a test" ...
0
votes
3answers
61 views

awk - Print all values with a suffix

I want to use awk to print all values in a delimited string with a static suffix appended to each one. Input: stack,over,flow Delimiter: Comma Output: stack suffix, over suffix, flow suffix
1
vote
3answers
40 views

Add a pattern before the delimiter

I'm looking for a command to add a pattern before or the delimiter. My sting: 1 |Chris|ubuntu Here my delimiter is| and I want to generate a string with this input. Output 1 = ID and Chris = ...
2
votes
1answer
59 views

Extract the number of pages of PDFs recursively and dump them to a CSV

Apologies for the fairly trivial question but I am a Linux beginner. I need to get a list of PDFs in various sub-folders of a directory structure and the number of pages of each PDF file. All saved ...
0
votes
1answer
29 views

Is the an elegant way to select rows in a file where a given column has a given value?

I have a file with a lot of lines like this 0 file:/home/knappen/somefilename.txt 7 0.2838356973995272 19 0.21823286052009455 18 0.10121158392434988 15 0....
0
votes
3answers
59 views

Organizing data by a header

I have two csv files. File 1 contains header. File 2 contains data file 1 format : file1.csv id,abc,xyz,aaa,bbb,ccc file 2 format : file2.csv id,source,value 1,abc,100 1,xyz,...
3
votes
6answers
696 views

Replace the third comma on each line with newline

I have these long IDs which consist of four sections: AKJHGFGUIKL,OIUYT,KJHBTYUI,98765434567 RTYUIKHGFGH,TYUJI,TGHYJKJKLJKL,6789876 ETRYTUUI,YTYUIL,UIOKJHGFGH,34567898766 I want to put the numbers ...
0
votes
2answers
73 views

How to sort the data from CSV file by comparing data from text file using awk output it as a TXT file?

There are two files I am going to extract the data: file1.txt: Type Serial ID Element Hit_Possibility Yasuo 19-2 19623 Hasaki file2.csv: Date,Name,Order,Hit Possibility 12-Aug,Ken,1,256 12-Aug,...
-1
votes
3answers
155 views

Field separator part of a column - incorrect parsing unix

I want to do a check the number of columns in a CSV file before processing it. The problem is that the delimiter (comma) also occurs in the text of some fields and because of that I cannot parse it ...
0
votes
2answers
101 views

How to remove duplicate items within the same line from a csv file?

I have a csv file with ~4000 lines, each one containing between 2 and 30 names separated by commas. The names are including titles (for example mr. X Adams or ms. Y Sanders). Some names exist multiple ...
1
vote
4answers
118 views

CSV file: Group and sum values based on first pattern of each value

I am using bash to create a script to find, group and sum fields in a CSV file. Each row will have comma-separated fields with each field following a similar convention. For each comma-separated field,...
3
votes
4answers
711 views

Bash: join data from two csv files

I have two csv files that contain various user data; they share one common field (username). file A: username ; Fullname ; mail Bob ; Bob Hope ; bob@hope.org file B: username ; LastLogonTime ; ...
0
votes
1answer
98 views

Append lines from csv file to another based on column 1 matches

I have 2 .csv files, file1.csv and file2.csv. Say a line from file2.csv has the same 1st field as a line from file1.csv. I would like the full line from file1.csv to replace the full line from file2....
0
votes
3answers
101 views

AWK csv manipulation

I have a csv file containing something like this: abc;!!!!;22 abc;!!!!;23 23;!!!!!;22 Now I want to delte every line that contains a 23 in the third column. Since there is a possibility that there ...
0
votes
2answers
89 views

Rename a csv tab from UNIX bash script

I have CSV file that is generated from a bash script: UNIX_REPORT.sh. When you open CSV the file on the windGrr side with excel, the tab (or sheet) in the lower left corner is named UNIX_REPORT.sh. ...
0
votes
1answer
54 views

Interpolate Columnar Data

I am trying to find a way to interpolate between two lines of data in a CSV file, likely using awk. Right now, each line represents a data point at Hour 0 and Hour 6. I am looking to fill in missing ...
1
vote
3answers
87 views

Using AWK to add a new column with values to a csv file without creating empty new lines in between rows

I would like to add a new column to the end of a csv file and populating the column with values. I have used the following codes and it ends up showing new blank lines in between each record. Please ...
0
votes
2answers
80 views

Split datetime column in csv file into MM/YYYY and time column separately and write to the same csv

I am trying to use awk, split and then print but running into error Input: id day userId itemId 1 12/1/17 8:32 2232 tv 2 1/12/18 10:18 3232 fdfs 3 2/9/18 10:50 232 fdsf ...
-1
votes
3answers
131 views

remove the lines in CSV that contain specific pattern on the second field

How to remove the lines in .csv file that second field contain the word content? example : ams-hbase-log4j,content:\n#LicensedtotheApacheSoftwareFoundation(ASF)underone\n#...
1
vote
2answers
38 views

How to merge different columns in the same file?

I have three files, as shown below: FILE 1: DATE PGTO_CRED 20180801 50.00 20180802 150.00 20180803 130.00 20180804 110.00 20180805 200.00 FILE 2: DATE PGTO_TOTAL 20180801 ...
1
vote
1answer
35 views

Save each line of a .csv file as a separate .csv [duplicate]

I have a .csv file with no headers and want to have each line saved as a separate .csv file with the new file name to be the value in column 2.
0
votes
0answers
48 views

file command gives incorrect encoding type

I have a csv file in hand. When I ran 'file -i filename', it shows that it's encoded as us-ascii. But when I ran cat filename | csvcut -t -e us-ascii, I got an error: "Your file is not "us-ascii" ...