2

I want to remove the files which have only header and no data. I tried the below command and it is working fine, Only issue is it is giving exit status as 1 which should be zero.

wc -l /Path/File_*  | while read CNT FN; do [ $CNT -lt 2 ] && [  "$FN" != total ] && rm "$FN"; done

I am running this command at Datastage ETL tool and my job is failing.

is there any way to modify the command to get the exit status as 0

  • I think this will help: unix.stackexchange.com/q/14270/388654 – Quasímodo Apr 28 at 19:15
  • When read can not read more because there is no more data to read, it will fail. Is that what you mean is an issue? – Kusalananda Apr 28 at 19:30
  • there are multiple files in the folder, but i want to remove the files which have only header and no data. For that, I have written to do wc and check if the count is less than 2 remove the file – Aditya Kutcharlapati Apr 28 at 20:07
2

To avoid having to read the whole files (like wc does), when you know after the first line that a file has at least 2 lines, I'd do (on a GNU system):

LC_ALL=C gawk -v ORS='\0' '
  FNR == 2 {nextfile}
  ENDFILE  {if (FNR < 2) print FILENAME}' /path/File_* |
  xargs -r0 rm -f

That's also more efficient in that it minimises the number of commands being run.

More reliable as it works with arbitrary file names.

As a functional difference with wc-based solutions: it would not delete files that contain one delimited line followed by one non-delimited one.

That one only returns a non-zero exit status if a file could not be removed (and was there in the first place).


Your problem is that the exit status of that pipe line is the exit status of the right-most command in it (as long as you don't use the pipefail option).

The right-most command here is the while loop. The exit status of a loop is that of the last command run in the body of the loop. In your case, it will be the [ "$FN" != total ] command run on the last line of the input, which will be non-zero unless there is only one /path/File_* file (in which case wc doesn't print the total).

If you changed it to:

[ "$CNT" -gt 1 ] || [ "$FN" = total ] || rm -f -- "$FN"

You'd only get a non-zero exit status if the last header file could not be removed.

| improve this answer | |
0

Your conditions return false at the end of the loop:

# false           # false              = false
[ $CNT -lt 2 ] && [  "$FN" != total ]

For example, if wc -l /path returns

2 total

and you run the command

                                 # true           # true              = true
wc -l *  | while read CNT FN; do [ $CNT = 2 ] && [  "$FN" = total ] && echo $CNT "$FN"; done
echo $?
0

but if you run

                                 # false           # true              = false
wc -l *  | while read CNT FN; do [ $CNT = 3 ] && [  "$FN" = total ] && echo $CNT "$FN"; done
echo $?  
1

In other words:

#false         # false      = false
[ 2 -eq 3 ] && [ bar = foo ]
echo $?
1

#true         # false      = false
[ 3 -eq 3 ] && [ bar = foo ]
echo $?
1

# false         # true      = false
[ 2 -eq 3 ] && [ foo = foo ]
echo $?
1

# true         # true      = true
[ 3 -eq 3 ] && [ foo = foo ]
echo $?
0

I would go with a for loop instead of a while loop and a function to return a value if no matches found:

foo() {
  ret=1 # no matches found
  for file in *.txt; do # your extension
    [[ $(wc -l "$file" | cut -d' ' -f1) -eq 1 ]] && echo "rm $file" && ret=0
  done
  return $ret
}
| improve this answer | |
  • I want to remove the files which have only header and no data in the files – Aditya Kutcharlapati Apr 28 at 20:14
  • Thank you, It worked ``` for file in Path/Filename*; do [[ $(wc -l "$file" | cut -d' ' -f1) -eq 1 ]] && echo "rm $file" ; done ``` – Aditya Kutcharlapati Apr 28 at 20:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.