-1

I have thousands of photos with year of birth in the filename. I need to find and copy at least 100 files for each birthyear, let's say 100 files for birthyear 2000, 100 files for 2001,..., and so on.

Here's the format of the filenames:

35077502_1995-02-01_2012.jpg

2012 is the year the photo is taken, i guess.

Can it be done using bash script?

Thanks

  • Can you edit your question to clarify how the files are named? Is the birthyear part of the filename? And just to make sure, you need exactly n files for each year, right? – PawkyPenguin Aug 4 '17 at 6:09
  • can you give us directory listing ? – Rahul Aug 4 '17 at 6:33
  • The solution provided by Philippos below already does the job but I'm still finding a tweak for it so that the search will skip corrupted files (files with 333 or 0 bytes size) – tr3s Aug 4 '17 at 8:01
  • Oh, so “In the case of the sample file, 1995 is the year of birth and 2012 is the year the photo was taken.” That’s what edits are for. – G-Man Aug 8 '17 at 5:46
1

If there are no nasty things in the filenames you can do

for year in 2000 2001; do
  cp `ls *${year}*.jpg|head -n 100` destination
done
  • actually, they do include month and day. can i do *[2000-2001]* ? – tr3s Aug 4 '17 at 6:19
  • You need the loop to take 100 pictures from each year. – Philippos Aug 4 '17 at 6:28
  • Thank you for assistance Philippos. By the way, the filenames are in these format 32714_1945-01-15_2001.jpg, 2001 being the date taken. So I need the filter the 1945. Sorry for overlooking this on the original question – tr3s Aug 4 '17 at 6:36
  • In this case change the pattern to *_${year}-*jpg to make sure you don't catch the wrong number. – Philippos Aug 4 '17 at 6:38
  • Now it's working. Lastly, how can I filter files with at least KB in size? I just noticed many of the files copied are corrupted and have 333 bytes only in size. Hopefully this is not too much. Thanks again – tr3s Aug 4 '17 at 6:46
3
#!/bin/bash

IFS=$'\n' years=( $(find . -maxdepth 1 -name '*.jpg' -print0 | 
                    sed -zEn 's/^.*_([0-9][0-9][0-9][0-9])-.*\.jpg/\1/p' | 
                    tr '\0' '\n' | 
                    sort -u)
                )

for year in "${years[@]}" ; do
  mkdir -p "$year"
  find . -iname "*_${year}-*.jpg" -size +1k -print0 |
    head -z -n 100 |
    xargs -0r cp -t "$year"
done

This constructs an array ($years) containing the unique set of 4-digit years extracted from filenames in the current directory, where the year is preceded by an underscore (_) and followed by a dash (-). This requires the GNU version of sed for the -z aka --null-data option.

For each year, it first creates a directory for that year if one doesn't already exist, then uses find to list all filenames matching those required pattern that are larger than 1 KB in size. That list is then pipe through head to get only the first 100 lines, and then into xargs to copy the files to the appropriate directory.

The filename list is NUL-terminated throughout the entire pipeline so that it works with all valid filenames (i.e. it will not break if spaces, tabs, newlines, or other unusual but perfectly valid characters are in the filenames)

This also requires the GNU version of head (which is standard on Linux), because it uses the -z option (aka --zero-terminated) for NUL-terminated input. Specifically, it requires a version more recent than 13 Jan 2016. It also requires GNU cp for the -t (aka --target-directory) option, which allows the target directory to be the first argument rather than the last.

If the files need to be sorted, then sort -z can be inserted between the find and head commands - e.g. find ... -print0 | sort -z ... | head -z .... This also requires the GNU version of sort.

This assumes that, as indicated in the revision to your question, the filenames have an underscore followed by the year as the last thing before the .jpg extension.

If the year can appear anywhere in the filename, you may need to use -iname "*${year}*.jpg" (without the underscore and with a second * between the ${year} and the .jpg) but watch out for files where the eight-digit number at the beginning is something like 60420017, which contains 2001 as a substring.

This also assumes that all your files have (case-insensitive) .jpg extensions (and not .jpeg, .jpe, .jfif, .gif, .png, etc.). If multiple filename extensions are required, the -iregex option could be used instead of -iname.

  • 1
    (1) Thanks for providing an answer that doesn’t parse the output of ls and that handles weird filenames. (2) I edited it; you are of course free to tweak further or rollback. – G-Man Aug 4 '17 at 20:09
  • You have misinterpreted the info on the filenames. In the case of the sample file, 1995 is the year of birth and 2012 is the year the photo was taken. By the way, may I redirect you to this new question: link. I have created a new thread because the concept of the problem is different. Thanks – tr3s Aug 7 '17 at 5:51
  • @tr3s see updated version. this one now uses the birth year, not year the photo was taken. it also first builds a list of birth years to search for. – cas Aug 7 '17 at 6:18
  • BTW, unless you really need copies, creating symlinks or hard links would be a lot faster and use a lot less disk space. – cas Aug 7 '17 at 6:40
  • Agree. It's working correctly now. Thank you for your help – tr3s Aug 7 '17 at 8:02
0

With zsh:

for y ({1995..2017}) (cp -- **/*_$y.jpg(.LK+1[1,100]) destination)
  • **/: in any level of sub-directories, sorted alphabetically
  • .: regular files only
  • LK+1: more than 1KiB in length
  • [1,100]: first hundred ones.

(as the sorting order will determine which files are copied, you may want to add the n glob qualifier for the sorting to be numerical).

Or to avoid hard coding the list of years and crawl the directory several times:

typeset -A files n
for f (**/*_<->.jpg(.LK+1)) {
  y=${${f##*_}%.*}
  ((++n[$y] > 100)) || files[$y]+=$f$'\0'
}
for y (${(k)files}) {
  mkdir -p $y && cp -- ${(0)files[$y]} $y
}

(untested)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.