I have some huge amount of image files (around 100 000) spared into 50 folders and sub-folders and i need to write a script in order to treat this data automatically

I am trying to write a Shell script for a bit complicated task, and i am now struggling to make it run the correct way. So, in order to give you the best description possible, i am going to write down the main things that i want to get done with this script as follows:

First: The script must go over Folders and sub-folders and extract filenames and full Path

Second: The filenames contain time and date informations .ie: 20180612074405680. I need the script to convert this to the UTC format .ie: 2018 - 06 - 12 T 07:44:05 TZ +01:00

Finally: I need all this to be exported to a .csv file,

The final .csv file should then contain the following informations:

File path, filename, time
C:/folder/sub-folder/file, 20180612074405680_ZTRDEFO_Blackgen.jpg, Time in UTC

The timestamp in the filename needs to be formatted the way described above !

I have been trying to use the find command in order to walk folders and sub-folders and get filenames but i still couldn't get the full path of the files

Can anyone please help or give some hints !

  • "UTC format" does not exist. UTC is a time zone designator not a time format. Perhaps you are confusing UTC with ISO 8601.
    – andcoz
    Jun 20 '18 at 10:04
  • What should the CSV file contain. Should the resulting filename really contain spaces as you have shown?
    – Kusalananda
    Jun 20 '18 at 10:06
  • Can you give us some full filename example? Is "20180612074405680" a complete filename?
    – andcoz
    Jun 20 '18 at 10:09
  • Is the timestamp in the filename in UTC?
    – andcoz
    Jun 20 '18 at 10:43
  • 1
    It appears you have not registered your WYA account, and so are attempting to update it with a separate new user. Please register your WYA account, or merge the two accounts, so that you can edit your own posts (and more!). unix.stackexchange.com/help/merging-accounts
    – Jeff Schaller
    Jun 20 '18 at 11:14

Quick and dirty:

find "/full/path/of/the/base/dir" -type f -printf "%f;%h;%f\n" \
    | sed -r 's/;([0-9][0-9][0-9][0-9])([0-9][0-9])([0-9][0-9])([0-9][0-9])([0-9][0-9])([0-9][0-9])([0-9][0-9][0-9])([^;]*)$/;\1-\2-\3T\4:\5:\6/ TZ +01:00'

This works if:

  1. Timestamps in filename are in +01:00 timezone,
  2. Filenames and pathnames do not contain any ';' character.

Assumption: You do not want to convert between timezones, just simply parse the timestamp in the filename as a string. Also, the three last digits are insignificant and may be discarded.

Using find, calling a bash script:

find . -type f -exec bash -c '
    fmt="\"%s\",\"%s\",%4d-%02d-%02dT%02d:%02d:%02d TZ +01:00\n"
    for pathname do
        ! [[ "${pathname##*/}" =~ $pattern ]] && continue
        printf "$fmt" "${pathname%/*}" "${pathname##*/}" "${BASH_REMATCH[@]:1}"
    done' bash {} + >report.csv

The find command feeds every file found in the current directory (.) to a short bash script. If you want to look for a particular filename suffix, add e.g. -name '*.jpg' before -exec.

The bash script, with annotations:

# The regular expression that we'd like to match against each pathname.
#        (year    )(month   )(day     )(hour    )(minute  )(second  ) last 3 digits ignored

# The format we'd like our output in (quoting the first two fields)
fmt="\"%s\",\"%s\",%4d-%02d-%02dT%02d:%02d:%02d TZ +01:00\n"

for pathname do
    # If we can't match the pattern against the filename, ignore this file
    ! [[ "${pathname##*/}" =~ $pattern ]] && continue

    # Output according to the format.
    printf "$fmt" "${pathname%/*}" "${pathname##*/}" "${BASH_REMATCH[@]:1}"

When outputting, ${pathname%/*} is the directory of the found file (could be written $( dirname "$pathname" )), ${pathname##*/} is the filename of the found file (could be written $( basename "$pathname" )) and the ${BASH_REMATCH[@]:1} will be each individual parts captured by the regular expression.

The output of the whole command is written to report.csv using a redirection at the end of the find command.


`-- dir
    |-- 20180612074405680_ZTRDEFO_Blackgen.jpg
    |-- file20180612074405680-1.txt
    |-- file20180612074405680-10.txt
    |-- file20180612074405680-2.txt
    |-- file20180612074405680-3.txt
    |-- file20180612074405680-4.txt
    |-- file20180612074405680-5.txt
    |-- file20180612074405680-6.txt
    |-- file20180612074405680-7.txt
    |-- file20180612074405680-8.txt
    |-- file20180612074405680-9.txt
    `-- some-other-file

Running the command generates report.csv which looks like

"./dir","file20180612074405680-1.txt",2018-06-12T07:44:05 TZ +01:00
"./dir","file20180612074405680-2.txt",2018-06-12T07:44:05 TZ +01:00
"./dir","file20180612074405680-3.txt",2018-06-12T07:44:05 TZ +01:00
"./dir","file20180612074405680-4.txt",2018-06-12T07:44:05 TZ +01:00
"./dir","file20180612074405680-5.txt",2018-06-12T07:44:05 TZ +01:00
"./dir","file20180612074405680-6.txt",2018-06-12T07:44:05 TZ +01:00
"./dir","file20180612074405680-7.txt",2018-06-12T07:44:05 TZ +01:00
"./dir","file20180612074405680-8.txt",2018-06-12T07:44:05 TZ +01:00
"./dir","file20180612074405680-9.txt",2018-06-12T07:44:05 TZ +01:00
"./dir","file20180612074405680-10.txt",2018-06-12T07:44:05 TZ +01:00
"./dir","20180612074405680_ZTRDEFO_Blackgen.jpg",2018-06-12T07:44:05 TZ +01:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.