7

I have a folder containing 2Gb of images, with sub-folders several levels deep.

I'd like to archive only N files of each (sub) folder in a tar file. I tried to use find then tail then tar but couldn't manage to get it to work. Here is what I tried (assuming N = 10):

find . | tail -n 10 | tar -czvf backup.tar.gz

… which outputs this error:

Cannot stat: File name too long

What's wrong here? thinking of it - even if it works I think it will tar only the first 10 files of all folders, not the 10 files of each folder.

How can I get N files of each folder? (No file order needed )

  • +0. Why do you want to do this? – unforgettableid Aug 17 '15 at 20:39
  • Do you wish to have the first 10 files, sorted numerically, alphabetically, by date or does the order not matter? – Alexander Aug 18 '15 at 9:54
  • @unforgettableid : I have to work on a web site locally, but there are 9Gb of images on server and I don't need all of them ! – Sam Aug 19 '15 at 7:22
  • @Sam: Ah OK. Upvoted. – unforgettableid Jul 6 '16 at 1:13
4

If your pax supports the -0 option, with zsh:

print -rN dir/**/*(D/e:'reply=($REPLY/*(ND^/[1,10]))':) |
  pax -w0 | xz > file.tar.xz

It includes the first 10 non-directory files of each directory in the list sorted by file name. You can choose a different sorting order by adding the om glob qualifier (order by modification time, Om to reverse the order), oL (order by length), non (sort by name but numerically)...

If you don't have the standard pax command, or it doesn't support -0 but you have the GNU tar command, you can do:

print -rN -- dir/**/*(D/e:'reply=($REPLY/*(ND^/[1,10]))':) |
  tar --null -T - -cjf file.tar.xz

If you can't use zsh, but have access to bash (the shell of the GNU project), you could do:

find dir -type d -exec bash -O nullglob -O dotglob -c '
  for dir do
    set -- "$dir/*"; n=0
    for file do
      if [ ! -d "$file" ] || [ -L "$file" ]; then
        printf "%s\0" "$file"
        (( n++ < 10 )) || break
      fi
    done
  done' bash {} + | pax -0w | xz > file.tar.xz

That would be significantly less efficient though.

  • print -rN -- dir/**/*(D/e:'reply=($REPLY/*(ND^/[1,10]))':) -> Get error message : -bash: syntax error near unexpected token `(' – Sam Aug 19 '15 at 15:08
  • 1
    @user1916171, It seems you missed the with zsh and If you can't use zsh parts of that answer. bash is a different shell, the shell of the GNU project which is much more limited compared to zsh. See the end of the answer for a solution that would work with bash. – Stéphane Chazelas Aug 19 '15 at 15:22
2

Suppose your master directory is /tmp/dir from which you want to archive only N (e.g N=10) files of each (sub) folder under it to a backup.tar.gz file.

Example tree for /tmp/dir:

dir/                                                                                                                                                                                                           
├── one
│   ├── one10.txt
│   ├── one11.txt
│   ├── one1.txt
│   ├── one2.txt
│   ├── one3.txt
│   ├── one4.txt
│   ├── one5.txt
│   ├── one6.txt
│   ├── one7.txt
│   ├── one8.txt
│   ├── one9.txt
│   └── one_deep
│       ├── one_deep1
│       ├── one_deep10
│       ├── one_deep11
│       ├── one_deep2
│       ├── one_deep3
│       ├── one_deep4
│       ├── one_deep5
│       ├── one_deep6
│       ├── one_deep7
│       ├── one_deep8
│       └── one_deep9
├── three
│   ├── three10.txt
│   ├── three11.txt
│   ├── three1.txt
│   ├── three2.txt
│   ├── three3.txt
│   ├── three4.txt
│   ├── three5.txt
│   ├── three6.txt
│   ├── three7.txt
│   ├── three8.txt
│   ├── three9.txt
│   └── three_deep
│       ├── three_deep1
│       ├── three_deep10
│       ├── three_deep11
│       ├── three_deep2
│       ├── three_deep3
│       ├── three_deep4
│       ├── three_deep5
│       ├── three_deep6
│       ├── three_deep7
│       ├── three_deep8
│       └── three_deep9

Code:

cd /tmp; for i in `find dir/* -type d`; do find $i -maxdepth 1 -type f | tail -n 10 | xargs -I file tar -rf backup.tar file; done; gzip backup.tar

This will create a backup.tar.gz with 10 files of each sub folder from under /tmp/dir.

  • If for some reason cd /tmp fails, you'll run that command in the wrong directory. You should always check the exit status of cd: cd /tmp && for... – Stéphane Chazelas Aug 18 '15 at 9:37
  • find dir/* -type d means you won't be processing the hidden directories in the current directory, but will be processing them in subdirectories. Either use find dir -type d or if you don't want the files in the current directory: find dir/. ! -name . -type d or find dir ! -path dir -type d. – Stéphane Chazelas Aug 18 '15 at 9:39
  • Using `find...` means invoking the split+glob operator (split only in zsh). Here you don't want the glob part, and you want to split on newline only (though newline is a valid character in a file name, so that approach is flawed anyway, and you should use find's -exec instead). – Stéphane Chazelas Aug 18 '15 at 9:41
  • Leaving that $i unquoted also means invoking the split+glob operator which doesn't make sense here. Use find "$i" – Stéphane Chazelas Aug 18 '15 at 9:42
  • Note that even with -I, xargs still treats quote and backslash characters specially in its input. It also means running one tar command per line which is not very efficient. – Stéphane Chazelas Aug 18 '15 at 9:47
2

Since the output of find is flat, you don't really know which files belong to the same directories without looking at the paths. The alternative is to use multiple finds (one per folder), without having to look at the paths. This is what I did. In order to tar up to a maximum of 10 files of each subfolder, use something like this:

for dir in $(find . -type d); do
  find "$dir" -maxdepth 1 -type f -printf "\"%p\"\n" | tail -10
done | xargs tar cvfz backup.tar.gz

This recursively finds all directories in the current folder. For each directory, it finds up to 10 files in exactly that folder (-maxdepth 1). Once the entire loop is finished, the tar command is executed on all files that were output by the loop. I also accounted for directory and folder names with spaces by quoting $dir and having find print each filename within quotes using the -printf option.

1
for d in ./*/
do
    cd "$d"
    tar -rvf ../backup.tar $(ls | tail -10)
    cd ..
done
gzip backup.tar

other variant

find * -prune -type d -exec bash -c 'printf "%s\n" $0/* | tail -10' {} \; |
tar czvf backup.tar.gz -T -
0

Use a hash on the directory name, and only emit the filename if the hash value count is below the threshold. E.g.

find . -depth -type f \
| perl -MFile::Spec -nle '(undef,$d,$f)=File::Spec->splitpath($_); print if $seen{$d}++ < 3' \
| tar ...
0

The easiest (or most easy to understand) way is to use xargs with the -N max-args option.

Keep in mind, that your input always needs to be something, which does not require a commandline, so echo *.* will work as input, where ls *.* does not (too long ls commandline)

find should be fine, as its argument is only the path, not a list of files.

0

OP asked this over in Stackoverflow too. Here's the answer I offered there.

The selection and order of files in this answer is determined by the order out of find, so "first" isn't well-defined here. This may also depend on GNU Awk 4.1.0.

find . -type f |
awk -v N=10 -F / 'match($0, /.*\//, m) && a[m[0]]++ < N' |
xargs -r -d '\n' tar -rvf /tmp/backup.tar

gzip /tmp/backup.tar

Comments:

  1. use find . -type f to ensure that files have a leading directory-name prefix, so the next step can work
  2. the awk command tracks such leading directory names, and emits full path names until N (10, here) files with the same leading directory have been emitted (perhaps simpler awk use - splitting info pattern and program - might be more portable)
  3. use xargs to invoke tar - we're gathering regular file names, and they need to be arguments to that archiving command
  4. xargs may invoke tar more than once, so we'll append (-r option) to a plain archive, then compress it after it's all written

Also, you may not want to write a backup file into the current directory, since you're scanning that - that's why this suggestion writes into /tmp.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.