0

I have a folder is populated each day with a series of files, for example content1_20231110.csv, content2_20231110.csv, etc. For reasons out of my control, this folder holds an entire year's worth of records, thousands of files.

I am currently manually copying each day's files from this folder to another share drive. This is quite easy because File Explorer allows you to sort by date-modified, placing all the files I want together. The files are generated overnight, so I still have to identify the ones I want by the date in their file name, not necessarily just by date-modified.

I've created a bash shell script to automate this task. The core method is to use the following command to identify files by the date in their name and copy them:

cp $originalFolderMapping/*$targetDate* $destinationFolderMapping

If the files I want are isolated in their own folder, which I did for testing, this method works fine. But the task takes prohibitively long to search through the unsorted original folder. It's possible that the files being in a shared drive is also slowing this down.

It's quick and easy to sort by date-modified in file explorer, making all the files I want adjacent to each other and convenient to copy manually. Is there a way to do something similar in a shell script to execute this task quickly? Thank you in advance.

3
  • 1
    The source directory is on the local system? Or is it a network drive/share? Nov 14, 2023 at 21:39
  • Please add the output of time /bin/ls $originalFolderMapping/*$targetDate* >/dev/null (should just ouput 3 lines of timing info) to the body of your question (Please, NOT as a comment (;-!) . Are you sure it isn't the cp that is taking the time? Else someday you're not going to have enough room in $originalFolderMapping/, so why not get started managing that. As a seperrate /follow-on task, compare files between source and target and if OK the delete $originalFolderMapping/*$targetDate* . Good luck.
    – shellter
    Nov 14, 2023 at 23:36
  • how long do time echo "$originalFolderMapping"/*"$targetDate"* | wc -l and time ( cd "originalFolderMapping" && find . ! -name . -prune -name "*$targetDate*" | wc -l ) take? (the quoting on second is not the same as the first)
    – jhnc
    Nov 14, 2023 at 23:38

1 Answer 1

1

Mention of File Explorer makes me wonder if you're using Windows, and perhaps then that bash is running within cygwin. If this were the case, the cygwin intermediate layer may possibly be making the already relatively slow NTFS file system appear rather a lot slower.

As Chris Davies noted, if the filesystem is (also) on a network share that will make the glob (file path expansion) in your "cp" that much slower.

One option to try (may not work) is to consider asking for a complete directory listing -- e.g. ls -1 $dir >/tmp/file -- and then using tools like awk, python or perl to filter that list and do the needed changes. Given that the files are created overnight the listing file should only need updating infrequently, so it might be worth keeping track of how up to date it is and whether the current operation needs it to be updated.

I note you're using "cp", so the problem is getting worse over time as no files are being removed but new ones added. While modern file systems are in general a lot better than old ones, there is always a lookup cost to using a very large directory. NTFS and others use a hashed directory structure, which means lookup of a specific filename is much faster than any kind of search.

So as a secondary task, work on the organisers of this folder to enable it to be split it up in some way - e.g. by year, or week number, whatever works. What you have is unsustainable and prone to multiple forms of error.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .