2

So, I made a little stupid mistake. During the collection process of data from a source, I dumped everything to the root of my external disk, which now has like 600k+ files (totaling 23 gigs). Here are the following few options I would like to do:

  1. Move a certain number of files from the disk to my hard disk.
  2. Create sub folders (with 1000 files each subfolder) and distribute files between them.
  3. Tar the entire thing or some part of it (say 50,000 files) and move them to my disk.

But anything that I try to do, it seems like it takes forever. Can somebody please help me about the optimal/most efficient way of doing any of these? Thanks.

  • I'm not clear on choice 2. Does that involve copying files from the external disk to the internal disk, or keeping everything on the external disk and just reorganizing the directory structure? – Mark Plotnick Nov 2 '15 at 16:43
  • If re-organizing the data structure on the external disk itself works (like eases the process of viewing and operating through files), then I will go for it as well. I was willing to run this: i=0; for f in *; do d=dir_$(printf %03d $((i/1000))); mkdir -p $d; mv "$f" $d; let i++; done .. but i am afraid it will become unresponsive and ultimately data will be lost/screwed. I am using a mac machine by the way (operating through terminal).. – khan Nov 2 '15 at 16:47
0

The following will create a subdirectory called new, and then 50 numbered subdirectories of new called new/01 .. new/50. It will then move a maximum of 1000 files from the current directory into each of the new/* directories.

NOTE1: It will not work if any of the filenames have newlines (\n) in them.

NOTE2: this requires GNU xargs and GNU mv. You are running linux so these are standard.

You can change the numbers (50 and 1000) and the base directory to suit your needs.

#! /bin/sh

BASEDIR='new'
SUBDIRS=50
FILECOUNT=1000

mkdir -p "$BASEDIR"

for i in $(seq -w 1 "$SUBDIRS") ; do
   mkdir -p "$BASEDIR/$i"

   # move maximum of $FILECOUNT files to new/$i
   find . -maxdepth 1 -type f | head -n "$FILECOUNT" | xargs -d'\n' mv -t "$BASEDIR/$i"
done

If the files you want moved all match a particular pattern then you can use find's -name or -regex options to select them. add the option(s) immediately after the -type f. e.g. if you wanted to move only .txt files:

find . -maxdepth 1 -type f -name '*.txt' | head -n "$FILECOUNT" | xargs -d'\n' mv -t "$BASEDIR/$i"

A non-GNU version can't use mv -t. it also can't use xargs -d option but you can use tr '\n' '\0' to convert newlines into NUL characters for feeding into xargs -0r - again, this copes with any file that doesn't have a \n in the filename. -R -1 is non-posix, a freebsd extension to xargs, you can safely remove that option but the mv operation will run a lot slower.

#! /bin/sh

BASEDIR='new'
SUBDIRS=50
FILECOUNT=1000

mkdir -p "$BASEDIR"

for i in $(seq -w 1 "$SUBDIRS") ; do
   mkdir -p "$BASEDIR/$i"

   # move maximum of $FILECOUNT files to new/$i
   find . -maxdepth 1 -type f | head -n "$FILECOUNT" | tr '\n' '\0' | xargs -0r -R -1 -I % mv % "$BASEDIR/$i"
done

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.