3

I want to capture the csv lines from the third field until and without the double quote (")

more test

"linux02","PLD26","net2-thrift-netconf","net.driver.memory","2"
"linux02","PLD26","net2-thrift-netconf","net.executor.cores","2"
"linux02","PLD26","net2-thrift-netconf","net.executor.instances","2"
"linux02","PLD26","net2-thrift-netconf","net.executor.memory","2"
"linux02","PLD26","net2-thrift-netconf","net.sql.shuffle.partitions","141"
"linux02","PLD26","net2-thrift-netconf","net.dynamicAllocation.enabled","true"
"linux02","PLD26","net2-thrift-netconf","net.dynamicAllocation.initialExecutors","2"
"linux02","PLD26","net2-thrift-netconf","net.dynamicAllocation.minExecutors","2"
"linux02","PLD26","net2-thrift-netconf","net.dynamicAllocation.maxExecutors","20"

I tried this

sed s'/,/ /g' test | awk '{print $3","$4","$5}' | sed s'/"//g'
,,
net2-thrift-netconf,net.driver.memory
net2-thrift-netconf,net.executor.cores
net2-thrift-netconf,net.executor.instances
net2-thrift-netconf,net.executor.memory
net2-thrift-netconf,net.sql.shuffle.partitions
net2-thrift-netconf,net.dynamicAllocation.enabled
net2-thrift-netconf,net.dynamicAllocation.initialExecutors
net2-thrift-netconf,net.dynamicAllocation.minExecutors
net2-thrift-netconf,net.dynamicAllocation.maxExecutors
,,

but I have a problem with my syntax because this syntax also prints ",," and the second syntax isn't elegant.

expected output:

net2-thrift-netconf,net.driver.memory,2
net2-thrift-netconf,net.executor.cores,2
net2-thrift-netconf,net.executor.instances,2
net2-thrift-netconf,net.executor.memory,2
net2-thrift-netconf,net.sql.shuffle.partitions,141
net2-thrift-netconf,net.dynamicAllocation.enabled,true
net2-thrift-netconf,net.dynamicAllocation.initialExecutors,2
net2-thrift-netconf,net.dynamicAllocation.minExecutors,2
net2-thrift-netconf,net.dynamicAllocation.maxExecutors,20
  • see please my update – yael Mar 21 '18 at 10:02
  • Consider also, you can use python (included in just about every distro) and import the csv library for an easy time working with CSVs. docs.python.org/2/library/csv.html – Centimane Mar 21 '18 at 10:51
  • Can the values in your fields possibly contain any commas? – reinierpost Mar 21 '18 at 13:07
3

With sed only:

sed -E 's/"//g; s/^([^,]*,){2}//' infile
  • s/"//g, strips all double quotes.
  • ^([^,]*,){2}, start from begging of the line, strips everything followed by a comma and repeat this to maximum two times.

Or with awk:

awk -F\" '{$1=$2=$3=$4=$5=""}1' OFS="" infile
7

It looks like it is just a matter or removing the quotes and then printing from the 3rd field until the end of the line:

$ tr -d \" < file | cut -d, -f3-
net2-thrift-netconf,net.driver.memory,2
net2-thrift-netconf,net.executor.cores,2
net2-thrift-netconf,net.executor.instances,2
net2-thrift-netconf,net.executor.memory,2
net2-thrift-netconf,net.sql.shuffle.partitions,141
net2-thrift-netconf,net.dynamicAllocation.enabled,true
net2-thrift-netconf,net.dynamicAllocation.initialExecutors,2
net2-thrift-netconf,net.dynamicAllocation.minExecutors,2
net2-thrift-netconf,net.dynamicAllocation.maxExecutors,20

So tr -d \" removes the quotes and cut -d, -f3- prints from the 3rd to the last ,-separated fields.

  • can you change it in case we want to set your syntax after pipe line as cat file | tr ................ – yael Mar 21 '18 at 10:14
  • @yael feel free to use the code as you wish. As is now, I think the answer is more complete and easy to reproduce to others. – fedorqui Mar 21 '18 at 10:17
2

You should really use a proper CSV parser for CSV data. Here's a way using ruby

ruby -rcsv -e '
  CSV.foreach(ARGV.shift) do |row|
    wanted = row.drop(2)   # ignore first 2 fields
    puts CSV.generate_line(wanted, :force_quotes=>false)
  end
' test
net2-thrift-netconf,net.driver.memory,2
net2-thrift-netconf,net.executor.cores,2
net2-thrift-netconf,net.executor.instances,2
net2-thrift-netconf,net.executor.memory,2
net2-thrift-netconf,net.sql.shuffle.partitions,141
net2-thrift-netconf,net.dynamicAllocation.enabled,true
net2-thrift-netconf,net.dynamicAllocation.initialExecutors,2
net2-thrift-netconf,net.dynamicAllocation.minExecutors,2
net2-thrift-netconf,net.dynamicAllocation.maxExecutors,20

or as a one-liner

ruby -rcsv -e 'CSV.foreach(ARGV.shift) {|r| puts CSV.generate_line(r.drop(2), :force_quotes=>false)}' test

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.