👋
You need to look into a CSV-aware tool. As you've already seen, you can write some very complicated awk and sed and still get it wrong. @cas mentioned miller and csvkit, I haven't used those so I cannot say if they can do what you need.
I do use GoCSV which is a very simple tool with a number of commands that can help you achieve your goal.
Just taking that one line by itself, the first thing GoCSV needs is a header, so I'll feed that line into its cap command to create a dummy header with a default name of 'Col':
echo '"Basic","""21,21""","[""21"",""21""]","","","","",""' | \
gocsv cap --default-name Col
and I get:
Col 1,Col 2,Col 3,Col 4,Col 5,Col 6,Col 7,Col 8
Basic,"""21,21""","[""21"",""21""]",,,,,
('Col' was appended with unique index numbers for each column)
You can see the command stripped away the unnecessary empty quotes at the end. Now, on to the quoted quotes.
I'll take that output and pipe it into the replace command where I can specify a regex patten to find, "
, and a replacement of empty string:
echo '"Basic","""21,21""","[""21"",""21""]","","","","",""' | \
gocsv cap --default-name Col | \
gocsv replace --regex '"' --repl ''
and I get:
Col 1,Col 2,Col 3,Col 4,Col 5,Col 6,Col 7,Col 8
Basic,"21,21","[21,21]",,,,,
If you need the header gone, pipe that into behead. Here's the complete pipeline:
echo '"Basic","""21,21""","[""21"",""21""]","","","","",""' | \
gocsv cap --default-name Col | \
gocsv replace --regex '"' --repl '' | \
gocsv behead
Basic,"21,21","[21,21]",,,,,