I need to count the number of columns in CSV files. If less than 11, rename the file appending the .bad suffix. I tried this but it's short of what I need:

head -1 myfile.csv | sed 's/[^,]//g' | wc -c

I need to do it for all files in a directory.

  • 1
    You should give "Data Science at the Command Line" a look. The post provides some tools for handling csv-files from the command line (they're Python based command line programs). – AllanLRH Dec 4 '17 at 18:59
for file in *.csv; do
    cols=$( awk -F, 'NR == 1 {print NF; exit}' "$file" )
    if [[ "$cols" -lt 11 ]]; then
        mv -v "$file" "$file.bad"

or, with bash without awk

for file in *.csv; do
    IFS=, read -ra fields <"$file"
    if [[ "${#fields[@]}" -lt 11 ]]; then
        mv -v "$file" "$file.bad"
  • 2
    good one, NR==1 not needed as exit is there.. – Sundeep Dec 4 '17 at 14:46

Another short awk approach:

for f in *.csv; do 
    awk -F, '{ exit (NF < 11? 0:1) }' "$f" && mv "$f" "${f}.bad"

  • NF < 11? - if number of fields NF is less than 11
  • the statement mv "$f" "${f}.bad" will be executed only if preceded awk statement returned exit 0 (as a positive evaluation)

perl has rename built-in function (with caveats, so check the documentation)

perl -F, -ane '$f=$ARGV; $c=$#F+1; close ARGV; rename $f,"$f.bad" if $c<11' *.csv
  • -F, use , as field separator, see https://perldoc.perl.org/perlrun.html#Command-Switches for details on other options
  • $f=$ARGV save filename
  • $c=$#F+1 get number of columns
  • close ARGV close the file
  • rename $f,"$f.bad" if $c<11 rename if number of columns is less than 11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.