For example, I have directory with multiple files created by this way:
touch files/{1..10231}_file.txt
I want to move them into new directory new_files_dir
.
The simplest way to do this is:
for filename in files/*; do
mv "${filename}" -t "new_files_dir"
done
This script works for 10 seconds on my computer. It is slow. The slowness happens due execution of mv
command for every file.
###Edit start###
I have understood, that in my example the simplest way will be just
mv files/* -t new_files_dir
or, if the "Argument list too long":
printf '%s\0' files/* | xargs -0 mv -t new_files_dir
but aforementioned case is a part of task. The whole task is in this question: Moving large number of files into directories based on file names in linux.
So, the files must be moved into corresponding subdirectories, the correspondence of which is based on a number in the filename. This is the cause of for
loop usage and other oddities in my code snippets.
###Edit end###
There is possibility to speedup this process by passing bunch of files to mv
command instead of a single file, like this:
batch_num=1000
# Counting of files in the directory
shopt -s nullglob
file_list=(files/*)
file_num=${#file_list[@]}
# Every file's common part
suffix='_file.txt'
for((from = 1, to = batch_num; from <= file_num; from += batch_num, to += batch_num)); do
if ((to > file_num)); then
to="$file_num"
fi
# Generating filenames by `seq` command and passing them to `xargs`
seq -f "files/%.f${suffix}" "$from" "$to" |
xargs -n "${batch_num}" mv -t "new_files_dir"
done
In this case the script works for 0.2 seconds. So, the performance has increased by 50 times.
But there is a problem: at any moment the program can refuse to work due "Argument list too long", because I can't guarantee that the bunch of filenames length is less than max allowable length.
My idea is to calculate the batch_num
:
batch_num = "max allowable length" / "longest filename length"
and then use this batch_num
in xargs
.
Thus, the question: How can max allowable length be calculated?
I have done something:
Overall length can be found by this way:
$ getconf ARG_MAX 2097152
The environment variables contributes into the argument size too, so probably they should be subtracted from
ARG_MAX
:$ env | wc -c 3403
Made a method to determine the max number of files of equal sizes by trying different amount of files before the right value is found (binary search is used).
function find_max_file_number { right=2000000 left=1 name=$1 while ((left < right)); do mid=$(((left + right) / 2)) if /bin/true $(yes "$name" | head -n "$mid") 2>/dev/null; then left=$((mid + 1)) else right=$((mid - 1)) fi done echo "Number of ${#name} byte(s) filenames:" $((mid - 1)) } find_max_file_number A find_max_file_number AA find_max_file_number AAA
Output:
Number of 1 byte(s) filenames: 209232 Number of 2 byte(s) filenames: 190006 Number of 3 byte(s) filenames: 174248
But I can't understand the logic/relation behind these results yet.
Have tried values from this answer for calculation, but they didn't fit.
Wrote a C program to calculate the total size of passed arguments. The result of this program is close, but some non-counted bytes are left:
$ ./program {1..91442}_file.txt arg strings size: 1360534 number of pointers to strings 91443 argv size: 1360534 + 91443 * 8 = 2092078 envp size: 3935 Overall (argv_size + env_size + sizeof(argc)): 2092078 + 3935 + 4 = 2096017 ARG_MAX: 2097152 ARG_MAX - overall = 1135 # <--- Enough bytes are # left, but no additional # filenames are permitted. $ ./program {1..91443}_file.txt bash: ./program: Argument list too long
program.c
#include <stdio.h> #include <string.h> #include <unistd.h> int main(int argc, char *argv[], char *envp[]) { size_t chr_ptr_size = sizeof(argv[0]); // The arguments array total size calculation size_t arg_strings_size = 0; size_t str_len = 0; for(int i = 0; i < argc; i++) { str_len = strlen(argv[i]) + 1; arg_strings_size += str_len; // printf("%zu:\t%s\n\n", str_len, argv[i]); } size_t argv_size = arg_strings_size + argc * chr_ptr_size; printf( "arg strings size: %zu\n" "number of pointers to strings %i\n\n" "argv size:\t%zu + %i * %zu = %zu\n", arg_strings_size, argc, arg_strings_size, argc, chr_ptr_size, argv_size ); // The enviroment variables array total size calculation size_t env_size = 0; for (char **env = envp; *env != 0; env++) { char *thisEnv = *env; env_size += strlen(thisEnv) + 1 + sizeof(thisEnv); } printf("envp size:\t%zu\n", env_size); size_t overall = argv_size + env_size + sizeof(argc); printf( "\nOverall (argv_size + env_size + sizeof(argc)):\t" "%zu + %zu + %zu = %zu\n", argv_size, env_size, sizeof(argc), overall); // Find ARG_MAX by system call long arg_max = sysconf(_SC_ARG_MAX); printf("ARG_MAX: %li\n\n", arg_max); printf("ARG_MAX - overall = %li\n", arg_max - (long) overall); return 0; }
I have asked a question about the correctness of this program on StackOverflow: The maximum summarized size of argv, envp, argc (command line arguments) is always far from the ARG_MAX limit.