0

The problem

In the current working directory I have several (very many) folders and some of them have lots (like 100,000+) of files inside.

My goal is getting just the folder with 10,000+ files and delete folderName10001.ext and above (where folderName is the actual name of the containing folder and ext is a whatever extension).

Approach taken

I am trying to print nameFolder numberOfFiles with bash.

find . -maxdepth 1 -type d -name 'acer' -exec sh -c "echo {}; ls {} | wc -l" \;

This works just fine (for the sample folder acer), but the output is

./acer
6058

I would like to have 1-line output per folder

find . -maxdepth 1 -type d -name 'acer' -exec sh -c "echo {} `ls {} | wc -l`" \;

says ls: cannot access {}: No such file or directory.


Questions

  1. How can I get to display in one line nameFolder numberOfFiles?
  2. Is there an easier way to delete every file folderName/folderName#.ext, with # > 10000?

Question 2

Some more details may help understand what's going on.

I have a folder with the following tree

.
├── a
│   ├── a1.ext
│   ├── a2.ext
│   ├── a3.ext
│   └── a4.ext
├── b
│   ├── b1.ext
│   ├── b2.ext
│   ├── b3.ext
│   └── b4.ext
└── c
    ├── c1.ext
    ├── c2.ext
    ├── c3.ext
    └── c4.ext

I'd like to remove every file with a number greater than 2. (In my specific case the threshold is set to 10000.)


Answer 1

It's all about who interprets what (thanks to @Bratchley for the advise in the comments).

Since I'm typing into bash, bash gets first to interpret what I type. If I want bash to send to find's bash a backtick, then I need to escape it.

find . -maxdepth 1 -type d -name 'acer' -exec sh -c "echo {} \`ls {} | wc -l;\`" \;

which gives me

./acer 6058

@Barmar points out that single- and double-quotes behaves differently, and therefore

find . -maxdepth 1 -type d -name 'acer' -exec sh -c 'echo {} `ls {} | wc -l`' \;

would work just fine (note " replaced with ').

Answer 2

This has been provided by Walter A, and it is the accepted answer for this question.

More errors (a couple of examples follows)...

black: 390120
./clean.sh: line 6: /bin/ls: Argument list too long
rm: missing operand
Try 'rm --help' for more information.

leather: 118240
./clean.sh: line 6: /bin/ls: Argument list too long
rm: missing operand
Try 'rm --help' for more information.

OK, I've fixed it with

ls -d */ | cut -d/ -f1 | while read dir; do
COUNT=$(ls $dir | wc -l);
   if [ ${COUNT} -gt 10000 ]; then
      echo "$dir: ${COUNT}" ;
      for i in `seq 10001 ${COUNT}`; do
         rm ${dir}/${dir}${i}.*
      done
   fi
done
  • working on the second question but the first one would involve giving echo the -n option. – Bratchley Apr 3 '15 at 18:28
  • 1
    Does something like find . -maxdepth 1 -type d -exec sh -c "echo {} \$(ls {} | wc -l)" \; work for you? – Bratchley Apr 3 '15 at 18:35
  • Wow! Yes, it works! I tried $(...) but it was giving be the same problem of the backtick. What does exactly the ` \ ` do? I mean, the \$(...) works like a charm. – Atcold Apr 3 '15 at 18:41
  • 1
    "Sounds like Arabic" - Italian expression for addressing incomprehensible speech or text. About the interpretation of subshell characters, I think I got it. So, basically, what I type into bash is executed first by bash itself, and then it is sent to the find's bash. OK, a bit confusing, but I got it. – Atcold Apr 3 '15 at 18:59
  • 1
    Use single quotes instead of double quotes, then you won't have to escape the backticks. – Barmar Apr 3 '15 at 20:06
1

You could write

ls -d */ | while read dir; do
    echo "$dir: $(ls $dir | wc -l)"
done

When the filenames are numbered without leading zeroes, you can try

ls -d */ | cut -d/ -f1 | while read dir; do
     COUNT=$(ls $dir | wc -l);
     echo "$dir: ${COUNT}" ;
     if [ ${COUNT} -gt 10000 ]; then
        ls ${dir}/${dir}?????*.ext | grep -v ${dir}/${dir}10000.ext | xargs rm
     fi
done

Another approach would be possible when you have leading zeroes in the filenames:

ls -d */ | cut -d/ -f1 | while read dir; do
   ls ${dir}/${dir}*.ext | tail -n +10001 | xargs rm
done

Edit: Included dir+filename in ls commands, and I added cut -d/ -f1 | to cut off the trailing /.

  • Wow! I believe I want to replace folderName with $dir and .ext with .*, right? – Atcold Apr 3 '15 at 22:06
  • I added the dir in the ls commands. Your question suggests that you only want to delete files named with extension .ext. When you want all extensions you can replace .ext by .*. You can use ${dir}/${dir}* in the last example, but that's not possible with ${dir}?????*.* (you want to have a minimum number of digits). – Walter A Apr 4 '15 at 12:45
  • One more question: why did you use ${} instead of simply $? Like ${COUNT} instead of $COUNT. – Atcold Apr 6 '15 at 15:33
  • Note that this breaks for directory names containing newlines, and directory names containing backslashes, and directory names containing spaces. – godlygeek Apr 6 '15 at 15:37
  • Here $ would be enough. I was used to omit {} when that wouldn't matter and add {} when the variable-name is followed by other characters that could also be part of the var, keeping myself sharp. A colleague said he always used {}, just to be safe and consequent. I decided to change my habbits when I adopted google-styleguide.googlecode.com/svn/trunk/shell.xml. – Walter A Apr 6 '15 at 15:38
2

Use single quotes instead of double quotes, so that backticks and $ don't get interpreted by the original shell:

find . -maxdepth 1 -type d -name 'acer' -exec sh -c 'echo {} $(ls {} | wc -l)' \;

For the second question, I would put what you want to do into a separate script, that takes the directory name as an argument. Then do:

find . -maxdepth 1 -type d -name 'acer' -exec ./scriptname {} \;

This way you don't have to deal with all the quoting issues from using -exec sh -c.

  • I thought single and double quotes were interchangeable (like in lua). I was wrong. What about the second question? Is there an easier way to delete all the files above the 10,000-th in each and every folder with 10,000+ files named folderName#.ext, where # > 10000? – Atcold Apr 3 '15 at 21:01
  • No, they're like Perl and PHP: single quotes are for strings that should get no interpretation, double quotes are for strings that allow substitution. I'm not sure I understand your second question. Do you want to remove the whole folder, or just all the files after 10,000 within it? I would write a script that does what you want, and then run that script with -exec, rather than try to cram everything into a one-liner. – Barmar Apr 3 '15 at 21:06
  • OK, perhaps I figured out an alternative way. I can find every file (within .) and then check if the # is > 10000. I guess I have to figure out now how to extract the number from the file name. I'm adding more info to the question above. – Atcold Apr 3 '15 at 21:09
  • Question updated. Let me know if it is more clear now. OK, I guess I will make a script as you suggest, but I will give up bash for lua, so at least I know how to move around with strings. – Atcold Apr 3 '15 at 21:18
  • That should be a new question. Don't mix two different questions in the same post. – Barmar Apr 3 '15 at 21:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.