1

Solved: See updates in comments.

I'm trying to sort files with a date in the filename into folders based on that date. The files used to have a space in the filename, like this. There are a few older files, so I need to sort those too.

Dispatch Fax_20180521121901.pdf

We're now getting the files with no space in the filename, so it's just Dispatch_20180521124202.pdf

The date is the 20180521 and the rest of it i think are message IDs, but we don't need that. Just the date.

This attempt at a bash script (ubuntu 18.04 box) ended up creating a single folder called 2018-05-21 one directory level up from where the files lived, and just put all of the pdf files in that folder.

    for x in /home/tb/temp/*.pdf
do
    d=$(date -r "$x" +%Y-%m-%d)
    mkdir -p "$d"
    mv -- "$x" "$d/"
done

This ended up moving into a folder called "e" somehow.

for x in /home/tb/temp/*.pdf
do
    d="${x:4:4}-${x:8:2}-${x:10:2}"
    mkdir -p "$d"
    mv -- "$x" "$d/"
done

This attempt? I ended up with a folder for each PDF.

/home/tb/20180521124202.pdf/Dispatch Fax_20180521124202.pdf /home/tb/20180521121901.pdf/Dispatch Fax_20180521121901.pdf

for x in /home/tb/temp/*.pdf
do
    d=$(echo "$x" | awk -F _ '{print $2}')
    mkdir -p "$d"
    mv -- "$x" "$d/"
done

The scripts are from previous posts here and here and those worked for them, but I'm not having the same luck even though the goal is the same.

My goal is to eventually populate a temp directory from rsync (pulling these files from another server) and then via cron, run this script to move things into folders based on date, and serve that up via Apache with "Options +Indexes" enabled. It's an intranet only page, and this particular machine has no outside access. Directory structure is simple. Eventually something like /var/www/html/2018/5/21 etc, following the YYYY/MM/DD format.

I'm just at a loss on how to get this working and my knowledge of bash scripting is very rusty. I have looked at a couple other examples like this one but then I see that it won't work with files that have spaces in the filenames.

I'm open to ideas!

  • You might want to explicitly state what you want the directory names to be. – igal May 22 '18 at 0:37
  • @igal - I thought I did. They just need to be the date. eventually something like /var/www/html/2018/5/21 etc, following the YYYY/MM/DD format. – Taco Bob May 22 '18 at 4:07
  • "The date" is not sufficiently explicit. What format do you want? – igal May 22 '18 at 4:09
  • @igal I'm not sure how to be more explicit, I'm sorry. I'm trying the best I can but am having trouble putting words to thoughts. The date is in the filename for each fax. Today's were 20180521 for example. I need to be able to sort them into directories, for year, month, day. If there's no folder for /var/www/html/2018/05/21 then it needs to be created. If it already exists, continue and put the file there. We get maybe 20 faxes a day, and they all have the Dispatch_YYYYMMDD*.pdf filename format. – Taco Bob May 22 '18 at 4:19
2

Your question isn't completely clear to me, but I think I might understand what you're trying to do.

Here this the loop that you're using:

for x in /home/tb/temp/*.pdf
do
    d="${x:4:4}-${x:8:2}-${x:10:2}"
    mkdir -p "$d"
    mv -- "$x" "$d/"
done

The example filenames you gave were:

  1. Dispatch Fax_20180521121901.pdf
  2. Dispatch_20180521124202.pdf

One problem that jumps out at me is that your parameter-substitution expression (d="${x:4:4}-${x:8:2}-${x:10:2}") doesn't seem to produce the correct results, e.g.:

> x='Dispatch Fax_20180521121901.pdf'
> d="${x:4:4}-${x:8:2}-${x:10:2}"
> echo ${d}
atch- F-ax

> x='Dispatch_20180521124202.pdf'
> d="${x:4:4}-${x:8:2}-${x:10:2}"
> echo ${d}
atch-_2-01

You could try using grep instead, e.g.:

> x='Dispatch Fax_20180521121901.pdf'
> d=$(echo "${x}" | grep -Po '\d{8}')
> echo ${d}
20180521

> x='Dispatch Fax_20180521121901.pdf'
> d=$(echo "${x}" | grep -Po '\d{8}')
> echo ${d}
20180521

Or, if you want to add hyphens, you could use the following sed-based command-substitution:

> x='Dispatch Fax_20180521121901.pdf'
> d=$(echo "${x}" | sed -E 's/^[^0-9]*([0-9]{4})([0-9]{2})([0-9]{2}).*$/\1-\2-\3/')
> echo ${d}
2018-05-21

> x='Dispatch Fax_20180521121901.pdf'
> d=$(echo "${x}" | sed -E 's/^[^0-9]*([0-9]{4})([0-9]{2})([0-9]{2}).*$/\1-\2-\3/')
> echo ${d}
2018-05-21

Assuming that this is the desired result (i.e. that you want subdirectories of the form %Y-%m-%d), we could substitute your parameter-substitution expression with the grep-based command-substitution. This would give us the following modified loop instead:

for filename in /home/tb/temp/*.pdf; do
    datestring=$(echo "${filename}" | sed -E 's/^[^0-9]*([0-9]{4})([0-9]{2})([0-9]{2}).*$/\1-\2-\3/')
    mkdir -p "${datestring}"
    mv -i -- "${filename}" "${datestring}/"
done

This should produce subdirectories of the form %Y-%m-%d and populate them with the appropriate files.


UPDATE: Based on your comment, it sounds like what you want are nested subdirectories of the form %Y/%m/%d. For that you would need to extract the year, month, and day substrings separately, e.g.:

> x='Dispatch Fax_20180521121901.pdf'
> d=$(echo "${x}" | grep -Po '\d{8}')
> year=${d:0:4}
> month=${d:4:2}
> day=${d:6:2}
> echo "${year}/${month}/${day}/"
2018/05/21/

This leads us to the following loop:

for filename in /home/tb/temp/*.pdf; do
    datestring=$(echo "${filename}" | grep -Po '\d{8}')
    year=${datestring:0:4}
    month=${datestring:4:2}
    day=${datestring:6:2}
    directory="${year}/${month}/${day}/"
    mkdir -p "${directory}"
    mv -i -- "${filename}" "${directory}"
done
  • I edited my original post. The goal is to have faxes from each day in their respective folders, for example today would be /var/www/html/2018/05/21 - and this script would run every 10 minutes (we don't get that many per day) and would sort stuff based on the date which is in the filename handed to us by the Canon fax server. And thanks for the pointer with grep. that'll help me with something else I was working on. – Taco Bob May 22 '18 at 4:15
  • I updated my post. Take a look at the new solution (at the bottom of the post). – igal May 22 '18 at 12:11
  • this solution worked perfectly. Thank you so much! I was having a difficult time with getting the date and my sed attempts were making a mess. But this works and I have something else to learn from too. Cheers! – Taco Bob May 23 '18 at 15:59
1

If you don't have to deal with actively hostile filenames,

sed -E 's/([0-9]{0,7}[^0-9]+)*([0-9]{8}).*/mkdir -p \2; mv "&" \2/e'
#         111111111111111111 2222222222 

The pattern is zero-to-seven-digits-followed-by-some-nondigits, any number of times, followed by 8 digits, once. The second pattern, the 8-digit one, matches the date.

If you don't have GNU sed you can instead of the e flag pipe the result into a shell.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.