Here is the example data I already sorted (3 columns):

CustomerId, fname, lname, email, gender, amount spent, creditcard, transaction-id, transactiondate, street, city, state, zip, phone


by using the command:

awk -F ',' '{ state = $12 ? $12: "exceptions.csv"; print}' < "03_conv_gender.tmp" > "04_filter_bad_data.tmp"

I check each row if they are missing a {state} value and if so it gets put in a new file called "exceptions.csv".

How would I delete the original bad data from the "04_filter_bad_data.tmp" or is there a better way to do this?

  • The data can be one of two states, "good" or "bad", correct? Pseudo-code would be "if (GOOD) then (OUPUT to GOOD.TXT) else (OUTPUT to BAD.TXT) endif". This is basic construct/algorithm in any computer science course, and has countless examples on the web. Please do your research.
    – C. M.
    Apr 27, 2021 at 9:57
  • Does this answer your question? Compare two columns of a csv file and show unmatched records only Apr 27, 2021 at 13:46
  • 1
    We have redacted your example data as it looked like it contained personal information. Please update your question with fake data instead.
    – Kusalananda
    Apr 28, 2021 at 17:45

1 Answer 1

awk -F, '{ print >($12==""?"badData.csv":"goodData.csv") }' infile

this splits input file to two separate output files, those line that having empty column#12 (STATE field) will be written into the file "badData.csv" otherwise it will be written into the "goodData.csv" output file.

  • This worked, but I was wondering how it works exactly because I want to learn. lets see if I got this right " I'm printing out everyting comparing at {STATE} -> if its blank? its baddata.csv or if its gooddata.csv read in from file?
    – puuyo
    Apr 26, 2021 at 23:15
  • @puuyo just added a short explain Apr 27, 2021 at 4:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .