3

I have a large number of files in a particular folder. I would like to move these files to a subfolder ONLY IF they have at least 1 value above 0.5 in any row of column 4. In a separate command I'd like to do the same but with files having at least 2 rows with values above 0.5 in column 4.

This is the general format of the files (with header):

col1  col2  col3  col4  col5  col6
ABC   DEF   5.10  0.94  GHI   JKL
MNO   PQR   8.31  0.37  STU   VWX
ABC   DEF   6.49  0.84  GHI   JKL
MNO   PQR   3.32  0.21  STU   VWX

Some of the numbers in column 4 are in scientific notation: 8.934553871039306e-05

The code below is what I have tried so far to move files with at least 1 value above 0.5 in column 4. It ends up moving every file into the subfolder, even ones that don't match the condition.

#!/bin/bash

find . -type f -exec awk '$4 >= 0.5' {} \; -exec mv -n {} ./NewFolder/ \;

  • Please specify in the question if the columns are separated by spaces or a tab. – Bodo Apr 7 at 9:01
  • The columns are tab separated – Sarah Apr 7 at 21:30
  • This information should be in the question. – Bodo Apr 8 at 7:17
3

To make your command work you have to make awk exit with code 0 if a match is found or with a non-zero exit code if no match is found.

In addition to this you should skip the first line because a non-numeric value will be compared as a string which may lead to an unexpected match.

find . -type f -exec awk 'FNR==1 {next} $4 >= 0.5 {found=1; exit} END {exit !found}' {} \; -exec mv -n {} ./NewFolder/ \;

Note: If the awk script is called with more than one file the exit code means that a match was found in any of the files. The find command will make sure only one file at a time is passed to awk, so this is not a problem here.

2nd Edit:

To select files that have at least 2 matching rows you can count the matches.

find . -type f -exec awk 'FNR==1 {next} $4 >= 0.5 {found++; if(found >= 2) exit} END {exit found >= 2}' {} \; -exec mv -n {} ./NewFolder/ \;

Edit:

To debug the problem that the script moves files that don't have a matching value in column 4, you can add code to the awk script to print information about the matching line. The following code will print the file name, line number and the matching line if a match was found.

find . -type f -exec awk 'FNR==1 {next} $4 >= 0.5 {found=1; printf "%s:%d:%s\n", FILENAME, FNR, $0; exit} END {exit !found}' {} \; -exec mv -n {} ./NewFolder/ \;

You will get something like

threshold.txt:2:ABC   DEF   5.10  0.94  GHI   JKL

I suggest to do this first to find the cause of the problem.

If there are lines that have a non-numeric text in column 4, the values will be compared as text. This would result in e.g. "abc" being greater than "0.5".

Another possible cause could be a line that has spaces in column 1 or 2 which will lead to a wrong assignment of the text to the columns.

In case there are non-numeric values in column 4 and you want to ignore these lines, you can force a numeric interpretation by adding the value to 0 as in 0 + $4.

find . -type f -exec awk 'FNR==1 {next} 0 + $4 >= 0.5 {found=1; printf "%s:%d:%s\n", FILENAME, FNR, $0; exit} END {exit !found}' {} \; -exec mv -n {} ./NewFolder/ \;

If the reason for the problem is that your fields are separated by a tab and that values can contain spaces you can specify the field separator (-F "\t"). The following script combines this with the other modifications.

find . -type f -exec awk -F "\t" 'FNR==1 {next} 0 + $4 >= 0.5 {found=1; printf "%s:%d:%s\n", FILENAME, FNR, $0; exit} END {exit !found}' {} \; -exec mv -n {} ./NewFolder/ \;
| improve this answer | |
  • This transferred some of the files and not others, but when I manually checked, there are a few files that get transferred into the folder but should not because they don't have any values above 0.5 – Sarah Apr 6 at 21:35
  • @Sarah Please show the files that get transferred and should not. Show the contents of the file as a code block in your question. – Bodo Apr 6 at 21:59
  • Thank you for the help with debugging! I ran your code and I believe there is a problem with column assignment in the files that got transferred incorrectly so I'm going to fix this. – Sarah Apr 7 at 21:44
3

Your awk doesn't actually work, it will find all files because the string col4 satisfies the >=0.5:

$ echo col4 | awk '$1>=0.5'
col4

So you need to skip the header. You also need to tell awk to exit with success if the file matches your criteria and with failure if it doesn't. Something like this:

find . -type f \
    -exec awk -va=1 '{ if($4 >= 0.5 && NR>1){a=0}} END{exit a}' {} \; \
    -exec mv -n {} ./NewFolder/ \;
| improve this answer | |
1

With a for loop you can try this:

for i in *; do # *.extension
  [[ -f "$i" && $(awk 'NR>1 && $4 >= 0.5' "$i") ]] && mv "$i" NewFolder/
done

And for two values:

for i in *; do  # *.extension
  [[ -f "$i" ]] && [[ $(awk 'NR>1 && $4 >= 0.5' "$i" | wc -l) -ge 2 ]] 
  mv "$i" NewFolder
done
| improve this answer | |
  • 1
    The find as used in the question will search for files in all subdirectories. The for i in * loop here will only process files in the current directory. This may or may not be what the OP needs. – Bodo Apr 7 at 9:10
  • @Bodo Thanks, you are right, I thought of that. But I just wanted to suggest an option in case all the files where in the same directory, in which case I think a for loop is a better approach than find. – schrodigerscatcuriosity Apr 7 at 15:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.