0

I have a problem with a very large number of files in one directory.

The file system is ext4.

I reached the 2**32 file limit and couldn't even write any file on this partition.

The problem is very big.

I don't know how I can move some of the files to other resources?

The classic "ls" and "mv" are not working. The files is too much...

Is there any way to quickly output any file in bash?

One arbitrary file per directory, which is almost 2**32 files.

If I can download one file quickly, I can write a script.

Any ideas?

Improve this question
4
  • 1
    It's not clear whether you know file names or not. Also, it's not clear what you really want. You want a file listing? You want to inspect individual files? – Artem S. Tashkinov Jul 21 '20 at 17:30
  • It's difficult to identify files names patterns. It's fair to assume that there is no known file name pattern. Is it possible to get the names of, for example, one or ten or a hundred files in this directory quickly? – Luk Jul 21 '20 at 17:36
  • 1
    Does ls --sort=none --no-group work for you? It will take some time to run but it will be finite. – Artem S. Tashkinov Jul 21 '20 at 17:43
  • @ArtemS.Tashkinov Good reply, but you can also head the output and get a list of maybe 100 files. Because of the non-sort, these will also be physically at the start of the directory, so quick to mv them too. – Paul_Pedant Jul 21 '20 at 17:59
2

Please try running:

ls --sort=none --no-group

or limit to some number of files, e.g.

ls --sort=none --no-group | head -500
Improve this answer
7
  • The original poster is unresponsive, so maybe we'd better wait for his reply before you deem my answer unsatisfactory. He's had troubles running ls and mv without arguments. – Artem S. Tashkinov Jul 21 '20 at 21:24
  • @Isaac Of course, this is an answer to the question. I might have given the same. – Hauke Laging Jul 22 '20 at 0:16
  • It's about the only possible answer for a huge directory. ls with any kind of sorting (the default) has to read it all before the first output comes. The only possible alternative is find (but I never tried that). Or C using readdir() if you need the exercise. – Paul_Pedant Jul 22 '20 at 9:00
  • Artem, or @HaukeLaging or anyone else: any idea if this is some sort of GNU thing or will all ls implementations work this way? This --sort=none trick is really useful! (update: I guess it must be GNU, I don't see this option in unix.com/man-page/POSIX/1posix/ls) – terdon♦ Jul 22 '20 at 12:50
  • 2
    @roaima I'm not sure. I have created a directory with a ridiculous number of files (18544022). Running command ls does eventually work, although it takes a few minutes to start printing. Conversely, command ls --sort=none --no-group happily lists out the files after a few seconds. So the options certainly help. I haven't managed to reproduce a case where command ls alone fails completely though. (using command to ensure no aliases with any other options are involved). – terdon♦ Jul 23 '20 at 9:16
1

Using ls to operate on a large directory is very inefficient, since GNU ls will read all of the entries in the directory before returning any of them, even with --sort=none, because it wants the output to be "pretty". This is both slow and uses a lot of RAM, since an ext4 directory can have many millions of files in it.

Instead, you should use find to list files in the directory, which will print out the filenames as soon as they are read from the directory. If you want to find particular files (e.g. all the "*.jpg" files smaller than 1MB), you could run e.g.

find /my/directory -type f -name "*.jpg" -size -1M

See the find(1) man page for full details on how to use it.

Once you find a bunch of of files you want to do something with, then you can use xargs to run a command for each file. For example, to delete temporary files use e.g.:

find /my/directory -name "*.tmp" -type f -print0 | xargs -0 rm

or to move them into a different directory like:

find /my/directory -name "*.jpg" -print0 | xargs -0 -I '{}' mv '{}' /my/otherdirectory

or any number of things. The xargs program runs the specific command for each file that it reads from the standard input, see xargs(1) man page for details. The mv command is a bit more complex than rm because mv needs to put the target directory at the end of the command, while xargs normally adds all the files after the specified command.

You could instead save the list of files to an output file like find ... > /tmp/file_list and then edit file_list to contain only the files you want to delete/move, and pipe it into xargs separately:

xargs -a /tmp/file_list -I '{}' mv '{}' /my/otherdir
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.