Skip to main content

Questions tagged [csv]

Files containing data arrange in a table, often with commas (hence Comma Separated Values), to separate columns. Rows are separated by newlines (but not all newlines are row separators as fields can be quoted to contain the separator newlines. Use this tag for full-fledged CSV data not the simpler case of one record per line or completely unquoted (use csv-simple for that kind of data).

Filter by
Sorted by
Tagged with
0 votes
3 answers
65 views

command-line tool to sum the values in a column of a CSV file

I am looking for a command-line tool to calculate the sum of the values in a specified column of a CSV file. (Update: The CSV file might have quoted fields, so a simple solution just to break on a ...
halloleo's user avatar
  • 589
1 vote
1 answer
84 views

Print csv columns according to user input

I have a .csv file with multiple lines with the following format: ¬Country¬,¬Year¬,¬Singer¬,¬Song Title¬ ¬Japan¬,¬1999¬,¬Utada Hikaru¬,¬First Love¬ ¬South Korea¬,¬1999¬,¬Lee Jung Hyun¬,¬Wa¬ ... I can ...
baiserekku's user avatar
3 votes
5 answers
165 views

How to remove double quotes within the double-quoted field values in .dat file

I have a text file that has around 15 columns.  The fields are separated by comma.  One column that is description is double-quoted and also has some words which are double-quoted.  I need to retain ...
Mythri's user avatar
  • 43
2 votes
7 answers
734 views

How to remove embedded newlines from CSV fields

I am struggling with processing a file. Normally I should have a line for each host definition. But sometimes, someone split some fields on different lines. Here is an example: "host1","...
Nicolas Greder's user avatar
1 vote
1 answer
37 views

Add list of urls to the Qutebrowser’s sqlite3 history

General overview I have a `history.csv CSV file of urls formated like this: <url>;<title>;2024-03-30T12:00:00;2024-03-30T12:00:00 (The 2024-03-30T12:00:00 is the same) The goal is to get ...
fauve's user avatar
  • 1,278
0 votes
1 answer
40 views

Certain file types not showing in Ranger preview

I have a bunch of csv files that are all just plain text csv files. $ file ./* file1.csv: ASCII text, with very long lines file2.csv: CSV Text I even generated file2.csv from file1.csv using cut. ...
blackthorne18's user avatar
5 votes
5 answers
685 views

Remove lines from a csv file based on column value

I have a csv file with 12 million lines in the following format: mcu_i,INIT,200,iFlash, 11593925, 88347,,0x00092684,r,0x4606b570, ok,, 32,single,op-c,0,, 0, 0, 0,...
raphui's user avatar
  • 84
-1 votes
4 answers
174 views

awk command to extract columns from a csv file whose header contains a certain substring

I have a huge csv file that is of the format aaa1, "aaa2, aa214", aa21, "aa, a14", aa211, aa44, aaa445 data, data, data, data, data, data, data, ......................................
User's user avatar
  • 254
-2 votes
1 answer
59 views

CSV file with many columns, print each using loop [closed]

I have a CSV file saved on my desktop with many columns. The CSV file has only one row but 195 columns and I have to print each column in Unix using loop command. The path of the file is /home/...
Kinjal Sinha's user avatar
3 votes
5 answers
421 views

How to create a new column and add a random identifier to it with miller

I want to add a column with a randomly created "case number" to my csv file. The first 2 letters of the casenumber must be any letter from A-Z in capitals. followed by 5 random numbers. ...
pwrsheller's user avatar
0 votes
1 answer
117 views

Bash script with input ipaddress file & csv file

I have a bash script with an ipaddress.txt input file, that I want to use against a csv file. I want find the IP and then display the values of Column A (for now). I may want to display two or three ...
burownidl's user avatar
-1 votes
1 answer
181 views

Create new CSV files based on latest timestamp data

I have a script that accesses multiple dat files and produce csv files based on previous day’s data. these DAT files are updated on per minute bases with data from various instruments. Script snippet: ...
dante_t's user avatar
  • 151
-1 votes
1 answer
49 views

Returning lines from a JSON log file which contain IP and timestamps read from a CSV file?

so I'm new to Shell Scripting and could use some help. So in my CSV file, I want to see if IP and timestamp values exists in some way in the line entries of the JSON log file, and if so, return that ...
Linuxnoob's user avatar
-2 votes
1 answer
56 views

How to convert a CSV with paragraphs to JSON on macOS's BSD Terminal?

I have a huge text file that contains this structure: title;description;date The problem is that description main contain paragraphs, like: title;description;data MyText;Lorem ipsum dolor sit amet, ...
Duck's user avatar
  • 4,714
0 votes
3 answers
299 views

Convert lower-case to uppercase with the output to a new column using miller

I want to copy column NAME to column NAME-LOWERCASE. NAME-LOWERCASE should only contain lowercase letters. The uppercase should be untouched in all column expect NAME-LOWERCASE input NAME,test PTC,N ...
pwrsheller's user avatar
1 vote
5 answers
789 views

awk on date conditions

I am trying delete rows from a person.csv (below) for the condition that person who are not born in last 1 year: Dataset1: "Index","User Id","First Name","Last Name&...
zoomer's user avatar
  • 21
0 votes
1 answer
109 views

Need Help with Date expression - AWK in a csv file

Situation: csv file has date column with different date formats, I want to convert them into a unambiguous date format (like +"%m-%d-%Y"). Example File data : File name = Date_Test_new.csv 3/...
ArielUnix2023's user avatar
3 votes
5 answers
827 views

Replace a block of spaces with a comma

I have a file in txt format that I want to convert to csv. Between each field there are blocks of spaces. The amount of spaces is not the same between every field. Is there a command with sed or awk ...
pwrsheller's user avatar
2 votes
2 answers
3k views

Convert json to csv with headers in jq

Is it possible to convert this json: [ { "bytes": 276697, "checked": false }, { "bytes": 276697, "checked": false ...
Daniel Krajnik's user avatar
0 votes
1 answer
323 views

Process CSV file through command line: Remove only middle rows between consecutive row entries if consecutive entries have the same 2nd column value

I have a CSV file with only 2 columns (but lots of rows) and the occasional irregular row which always starts with an asterisk (*) character and may span more than two columns. Using just the Linux ...
Carla H.'s user avatar
  • 161
3 votes
3 answers
282 views

Creating text files with comma separated list of values from one column based on shared value in second column from the commandline

I am hoping to create a series of text files each containing a comma separated list of values from column 1 of an input file based on their shared value in column 2 of the input file. Input file ...
Zoe Diaz-Martin's user avatar
1 vote
4 answers
232 views

Sed or Awk (or something else) removing several columns from csv file

I have several files from tool suppliers which I need to merge together. However only a few of the columns in the csv file gets used. So I want to remove the irrelevant columns. I have used sed to do ...
Danny's user avatar
  • 175
5 votes
6 answers
3k views

sort the whole .csv based on the value in a certain column

I have a csv file with different revenues. I want to sort the csv file on revenue from high to low value. I can't find how to do this in terminal without the use of python. I don't want to use python. ...
pwrsheller's user avatar
6 votes
2 answers
404 views

Deduplicate CSV rows based on a specific column, with a CSV parser

I searched for this task, and found the following older questions: Removing Duplicates from a CSV based on specified columns Identify unique records on CSV based on specific columns But I can't use ...
Mévatlavé Kraspek's user avatar
0 votes
5 answers
1k views

How to remove commas from a certain field in CSV?

There is a field/column in a CSV file Im working with called name, this is always the second field. The values in this column are like: "Smith, John" "Brady, Tom" "Manning, ...
ababa's user avatar
  • 1
8 votes
8 answers
1k views

Convert a fixed-width file to CSV and remove trailing space

My input file is: $ cat -e myfile.txt 999a bcd efgh555$ 8 z 7 $ 1 xx xx xx 48 $ And I need a CSV without trailing spaces in the columns: 999,a bcd efgh,555 8,z,7 1,xx xx xx,48 So far, ...
Thomas's user avatar
  • 133
0 votes
2 answers
459 views

Append string to the 2nd column if a condition is satisfied, using sed

CSV file: A,12345,hello B,67890,hi A,123456,yo Expected output: Should add two zeros at the end of the 2nd column if the 1st column is A: A,1234500,hello B,67890,hi A,12345600,yo
user569568's user avatar
-1 votes
1 answer
439 views

Read column1 of csv file2 and match with column2 of file1, if text matches then add column1 value from file1 to file2

file1.csv 9,hello-world.com.,A,192.168.0.9 1,hello1-world.com.,A,192.168.0.1 2,hello2-world.com.,A,192.168.0.2 3,hello3-world.com.,A,192.168.0.3 4,hello4-world.com.,A,192.168.0.4 5,hello5-world.com.,A,...
Sha's user avatar
  • 11
1 vote
1 answer
897 views

how to download postgres data in csv format through bash script?

I am able to connect to db also getting output but it's not coming in proper format. So how to get the output in csv like format and also want to save that in csv format. psql -d <db> -U <...
zircon's user avatar
  • 113
2 votes
1 answer
134 views

gawk is treating NF and custom variables differently even though they have the same value

I have the below awk script where I'm using the length of the array passed, as a variable. BEGIN { FPAT = "([^,]*)|(\"[^\"]*\")" len=split(_arr,arr," ") } { ...
Parikshith Raju's user avatar
-1 votes
6 answers
399 views

Minimum value from ";"-delimited list of values in each field in CSV file

I have a big CSV file in which, in some cells, there is more than one value. How can I change these cells so that they instead only contain the minimum value? For example, given this input: id,...
Solomon123's user avatar
0 votes
1 answer
79 views

Trying to sort output of mailbox sized

Trying to sort an output file of mailbox sizes. [email protected], 1.35 MB [email protected], 2.19 GB [email protected], 0 B [email protected], 0 B [email protected], 1.66 MB [email protected], 0 ...
dj423's user avatar
  • 1
4 votes
3 answers
1k views

Visually aligning columns of a CSV file

Using sed or awk, is it possible to visually align columns in a CSV file? For example: e.g. from: a,b,c,some stuff,"some, other, stuff",d,2023-03-10 18:37:00 y,x,z,t,cool,thing,2022-04-12 21:...
s.k's user avatar
  • 471
-3 votes
2 answers
227 views

Bash script to calculate within a CSV [closed]

This is my CSV File: Number;Reference;EANNumber;Manufacturer;Price_B2B;Price_B2B_Dis;Price_B2B_DisPer;Price_B2B_DisAmount;Price_B2C_exVAT;Price_B2C_inVAT 2330113;BP3141;1,31304E+11;APC;13;13;0;0;0;0;;;...
dima1002's user avatar
5 votes
3 answers
968 views

Rounding many values in a csv to 3 decimals (printf ?)

I have a paste command like this paste -d , file1.csv file2.csv file3.csv And file2.csv contains numbers like this 0.2 0.3339 0.111111 I want the values in file2.csv having 3 decimals like this: 0....
R 9000's user avatar
  • 167
0 votes
3 answers
105 views

Combine .csv-files with different amount of lines

Here I asked already for a similar problem. Now I have 3 .csv-files but one of them has only 1 line. file1.csv dog cats mouse file2.csv 001a 002a 003c file3.csv WORD The output should be dog,001a,...
R 9000's user avatar
  • 167
1 vote
1 answer
72 views

Combine .csv-files with text between each line

I have two (or maybe more) files: file1.csv dog cats mouse file2.csv 001a 002a 003c If I use paste file1.csv file2.csv the output is dog 001a cats 002a mouse 003c Of course I can use paste -d ,...
R 9000's user avatar
  • 167
0 votes
1 answer
2k views

Converting data in text file to csv tables

My data in a text file have the below format today HKL1234/2, R1B HKL1274/2, R4E HREF457-74, R1A I need to present this data in excel file in two different columns like below: Product ...
Dalea Kaldeby's user avatar
0 votes
1 answer
175 views

Converting key+n-values text file to a CSV file

Newbie on this site, so apologies if I ask this question badly: I have a text file in the following format: file: abc value: 123 value: 234 value: 567 file: def value: 999 file: ghi value: 123 value: ...
Jeff Day's user avatar
1 vote
1 answer
247 views

Create csv with filename, mtime and filesize in GB

I use the below command to extract the filename, filesize, and date for a big directory with many subdirectories. find MY_PATH -type f -exec ls -la --block-size=GB --time-style=+%Y-%m-%d_%H:%M:%S {} \;...
Niyaz's user avatar
  • 127
0 votes
2 answers
498 views

Choose columns with sort and cut in a csv with a comma delimiter ',' ignoring data on quotes with comma "text,text"

I am parsing a huge csv file with rows and columns with different parameters. However, some fields contain large descriptions within quotes that contain commas. How can I choose columns with sort and ...
infinity's user avatar
3 votes
5 answers
392 views

Replace empty lines in one file with lines from another file

I have file1.csv "word 1" "" "word 3" "" "word 5" "word 6" and file2.csv "replacement text 1" "replacement text 2" &...
R 9000's user avatar
  • 167
0 votes
2 answers
370 views

How to cut a string CSV variable with unknown values count in bash, and loop over its values?

It might seem very easy, but I'm stuck at this. I have a string that is CSV and the length of items is unknown. "item1,item2,item3,..." I want to cut it, and loop over its items. I tried: ...
Saeed Neamati's user avatar
-1 votes
3 answers
679 views

Searching in a CSV for multiple values and get the associated values from it

I have a file.csv "ItemNo","Name","Weight" "a001","Item a","1.1" "a002","Item x","1.2" "a003","...
R 9000's user avatar
  • 167
1 vote
4 answers
524 views

Count unique values and add result values as new column

I have large csv file and want to build simple ranking: $ cat file.csv 2022-12-01 RED 1 2022-12-01 RED 1 2022-12-01 RED 2 2022-12-01 RED 2 2022-12-...
tryingToDevelopMyself's user avatar
0 votes
3 answers
74 views

Read column value in a CSV file starting from another column value?

I am working on extracting certain data from OECD statistics. These are available in a CSV file and look as follows (excerpt): "COUNTRY","Country","DAGEGR","Age ...
John's user avatar
  • 103
1 vote
1 answer
6k views

jq: error (at input.json:45): Cannot index array with string "value"

I try to convert my .json files to .csv with jq. I'm getting cannot index array with string value on all my values. .json { "organic_data": [ { "description": "...
pwrsheller's user avatar
0 votes
4 answers
292 views

Awk: Concatenate columns if they are quoted, and remove commas

I have this data: COL1, COL2, COL3 a, b, c d, "e,f,g", h I want Awk to treat it as: COL1, COL2, COL3 a, b, c d, efg, h Each comma-separated string could be any number of characters. I just ...
eplictical's user avatar
-1 votes
2 answers
891 views

Commands runs in shell, but not in script: no directory found error

I have a few commands that don't run as a script. It's supposed to create a bulleted list of the file names, then sort the columns of each file alphabetically, grab the second and its matching third ...
pierogie's user avatar
0 votes
1 answer
296 views

Quoting a string in output from awk

I am fetching a string from a file using awk as shown below. Now I would like to double quote it. Any support would be highly appreciated. awk -F',' '{print $(NF)}' sample.csv| tail +2 output: ...
user3777713's user avatar

1
2 3 4 5
19