Sorting files into folders based on date in filename?

Question

Solved: See updates in comments.

I'm trying to sort files with a date in the filename into folders based on that date. The files used to have a space in the filename, like this. There are a few older files, so I need to sort those too.

Dispatch Fax_20180521121901.pdf

We're now getting the files with no space in the filename, so it's just Dispatch_20180521124202.pdf

The date is the 20180521 and the rest of it i think are message IDs, but we don't need that. Just the date.

This attempt at a bash script (ubuntu 18.04 box) ended up creating a single folder called 2018-05-21 one directory level up from where the files lived, and just put all of the pdf files in that folder.

    for x in /home/tb/temp/*.pdf
do
    d=$(date -r "$x" +%Y-%m-%d)
    mkdir -p "$d"
    mv -- "$x" "$d/"
done

This ended up moving into a folder called "e" somehow.

for x in /home/tb/temp/*.pdf
do
    d="${x:4:4}-${x:8:2}-${x:10:2}"
    mkdir -p "$d"
    mv -- "$x" "$d/"
done

This attempt? I ended up with a folder for each PDF.

/home/tb/20180521124202.pdf/Dispatch Fax_20180521124202.pdf /home/tb/20180521121901.pdf/Dispatch Fax_20180521121901.pdf

for x in /home/tb/temp/*.pdf
do
    d=$(echo "$x" | awk -F _ '{print $2}')
    mkdir -p "$d"
    mv -- "$x" "$d/"
done

The scripts are from previous posts here and here and those worked for them, but I'm not having the same luck even though the goal is the same.

My goal is to eventually populate a temp directory from rsync (pulling these files from another server) and then via cron, run this script to move things into folders based on date, and serve that up via Apache with "Options +Indexes" enabled. It's an intranet only page, and this particular machine has no outside access. Directory structure is simple. Eventually something like /var/www/html/2018/5/21 etc, following the YYYY/MM/DD format.

I'm just at a loss on how to get this working and my knowledge of bash scripting is very rusty. I have looked at a couple other examples like this one but then I see that it won't work with files that have spaces in the filenames.

I'm open to ideas!

Answer 1

Your question isn't completely clear to me, but I think I might understand what you're trying to do.

Here this the loop that you're using:

for x in /home/tb/temp/*.pdf
do
    d="${x:4:4}-${x:8:2}-${x:10:2}"
    mkdir -p "$d"
    mv -- "$x" "$d/"
done

The example filenames you gave were:

1. Dispatch Fax_20180521121901.pdf
2. Dispatch_20180521124202.pdf

One problem that jumps out at me is that your parameter-substitution expression (d="${x:4:4}-${x:8:2}-${x:10:2}") doesn't seem to produce the correct results, e.g.:

> x='Dispatch Fax_20180521121901.pdf'
> d="${x:4:4}-${x:8:2}-${x:10:2}"
> echo ${d}
atch- F-ax

> x='Dispatch_20180521124202.pdf'
> d="${x:4:4}-${x:8:2}-${x:10:2}"
> echo ${d}
atch-_2-01

You could try using grep instead, e.g.:

> x='Dispatch Fax_20180521121901.pdf'
> d=$(echo "${x}" | grep -Po '\d{8}')
> echo ${d}
20180521

> x='Dispatch Fax_20180521121901.pdf'
> d=$(echo "${x}" | grep -Po '\d{8}')
> echo ${d}
20180521

Or, if you want to add hyphens, you could use the following sed-based command-substitution:

> x='Dispatch Fax_20180521121901.pdf'
> d=$(echo "${x}" | sed -E 's/^[^0-9]*([0-9]{4})([0-9]{2})([0-9]{2}).*$/\1-\2-\3/')
> echo ${d}
2018-05-21

> x='Dispatch Fax_20180521121901.pdf'
> d=$(echo "${x}" | sed -E 's/^[^0-9]*([0-9]{4})([0-9]{2})([0-9]{2}).*$/\1-\2-\3/')
> echo ${d}
2018-05-21

Assuming that this is the desired result (i.e. that you want subdirectories of the form %Y-%m-%d), we could substitute your parameter-substitution expression with the grep-based command-substitution. This would give us the following modified loop instead:

for filename in /home/tb/temp/*.pdf; do
    datestring=$(echo "${filename}" | sed -E 's/^[^0-9]*([0-9]{4})([0-9]{2})([0-9]{2}).*$/\1-\2-\3/')
    mkdir -p "${datestring}"
    mv -i -- "${filename}" "${datestring}/"
done

This should produce subdirectories of the form %Y-%m-%d and populate them with the appropriate files.

---

UPDATE: Based on your comment, it sounds like what you want are nested subdirectories of the form %Y/%m/%d. For that you would need to extract the year, month, and day substrings separately, e.g.:

> x='Dispatch Fax_20180521121901.pdf'
> d=$(echo "${x}" | grep -Po '\d{8}')
> year=${d:0:4}
> month=${d:4:2}
> day=${d:6:2}
> echo "${year}/${month}/${day}/"
2018/05/21/

This leads us to the following loop:

for filename in /home/tb/temp/*.pdf; do
    datestring=$(echo "${filename}" | grep -Po '\d{8}')
    year=${datestring:0:4}
    month=${datestring:4:2}
    day=${datestring:6:2}
    directory="${year}/${month}/${day}/"
    mkdir -p "${directory}"
    mv -i -- "${filename}" "${directory}"
done

Answer 2

If you don't have to deal with actively hostile filenames,

sed -E 's/([0-9]{0,7}[^0-9]+)*([0-9]{8}).*/mkdir -p \2; mv "&" \2/e'
#         111111111111111111 2222222222 

The pattern is zero-to-seven-digits-followed-by-some-nondigits, any number of times, followed by 8 digits, once. The second pattern, the 8-digit one, matches the date.

If you don't have GNU sed you can instead of the e flag pipe the result into a shell.