0

I have some huge amount of image files (around 100 000) spared into 50 folders and sub-folders and i need to write a script in order to treat this data automatically

I am trying to write a Shell script for a bit complicated task, and i am now struggling to make it run the correct way. So, in order to give you the best description possible, i am going to write down the main things that i want to get done with this script as follows:

First: The script must go over Folders and sub-folders and extract filenames and full Path

Second: The filenames contain time and date informations .ie: 20180612074405680. I need the script to convert this to the UTC format .ie: 2018 - 06 - 12 T 07:44:05 TZ +01:00

Finally: I need all this to be exported to a .csv file,

The final .csv file should then contain the following informations:

File path, filename, time
C:/folder/sub-folder/file, 20180612074405680_ZTRDEFO_Blackgen.jpg, Time in UTC

The timestamp in the filename needs to be formatted the way described above !

I have been trying to use the find command in order to walk folders and sub-folders and get filenames but i still couldn't get the full path of the files

Can anyone please help or give some hints !

5
  • "UTC format" does not exist. UTC is a time zone designator not a time format. Perhaps you are confusing UTC with ISO 8601.
    – andcoz
    Jun 20, 2018 at 10:04
  • What should the CSV file contain. Should the resulting filename really contain spaces as you have shown?
    – Kusalananda
    Jun 20, 2018 at 10:06
  • Can you give us some full filename example? Is "20180612074405680" a complete filename?
    – andcoz
    Jun 20, 2018 at 10:09
  • Is the timestamp in the filename in UTC?
    – andcoz
    Jun 20, 2018 at 10:43
  • 1
    It appears you have not registered your WYA account, and so are attempting to update it with a separate new user. Please register your WYA account, or merge the two accounts, so that you can edit your own posts (and more!). unix.stackexchange.com/help/merging-accounts
    – Jeff Schaller
    Jun 20, 2018 at 11:14

2 Answers 2

0

Quick and dirty:

find "/full/path/of/the/base/dir" -type f -printf "%f;%h;%f\n" \
    | sed -r 's/;([0-9][0-9][0-9][0-9])([0-9][0-9])([0-9][0-9])([0-9][0-9])([0-9][0-9])([0-9][0-9])([0-9][0-9][0-9])([^;]*)$/;\1-\2-\3T\4:\5:\6/ TZ +01:00'

This works if:

  1. Timestamps in filename are in +01:00 timezone,
  2. Filenames and pathnames do not contain any ';' character.
0

Assumption: You do not want to convert between timezones, just simply parse the timestamp in the filename as a string. Also, the three last digits are insignificant and may be discarded.

Using find, calling a bash script:

find . -type f -exec bash -c '
    pattern="([0-9]{4})([0-9]{2})([0-9]{2})([0-9]{2})([0-9]{2})([0-9]{2})[0-9]{3}"
    fmt="\"%s\",\"%s\",%4d-%02d-%02dT%02d:%02d:%02d TZ +01:00\n"
    for pathname do
        ! [[ "${pathname##*/}" =~ $pattern ]] && continue
        printf "$fmt" "${pathname%/*}" "${pathname##*/}" "${BASH_REMATCH[@]:1}"
    done' bash {} + >report.csv

The find command feeds every file found in the current directory (.) to a short bash script. If you want to look for a particular filename suffix, add e.g. -name '*.jpg' before -exec.

The bash script, with annotations:

# The regular expression that we'd like to match against each pathname.
pattern="([0-9]{4})([0-9]{2})([0-9]{2})([0-9]{2})([0-9]{2})([0-9]{2})[0-9]{3}"
#        (year    )(month   )(day     )(hour    )(minute  )(second  ) last 3 digits ignored

# The format we'd like our output in (quoting the first two fields)
fmt="\"%s\",\"%s\",%4d-%02d-%02dT%02d:%02d:%02d TZ +01:00\n"

for pathname do
    # If we can't match the pattern against the filename, ignore this file
    ! [[ "${pathname##*/}" =~ $pattern ]] && continue

    # Output according to the format.
    printf "$fmt" "${pathname%/*}" "${pathname##*/}" "${BASH_REMATCH[@]:1}"
done

When outputting, ${pathname%/*} is the directory of the found file (could be written $( dirname "$pathname" )), ${pathname##*/} is the filename of the found file (could be written $( basename "$pathname" )) and the ${BASH_REMATCH[@]:1} will be each individual parts captured by the regular expression.

The output of the whole command is written to report.csv using a redirection at the end of the find command.

Example:

.
`-- dir
    |-- 20180612074405680_ZTRDEFO_Blackgen.jpg
    |-- file20180612074405680-1.txt
    |-- file20180612074405680-10.txt
    |-- file20180612074405680-2.txt
    |-- file20180612074405680-3.txt
    |-- file20180612074405680-4.txt
    |-- file20180612074405680-5.txt
    |-- file20180612074405680-6.txt
    |-- file20180612074405680-7.txt
    |-- file20180612074405680-8.txt
    |-- file20180612074405680-9.txt
    `-- some-other-file

Running the command generates report.csv which looks like

"./dir","file20180612074405680-1.txt",2018-06-12T07:44:05 TZ +01:00
"./dir","file20180612074405680-2.txt",2018-06-12T07:44:05 TZ +01:00
"./dir","file20180612074405680-3.txt",2018-06-12T07:44:05 TZ +01:00
"./dir","file20180612074405680-4.txt",2018-06-12T07:44:05 TZ +01:00
"./dir","file20180612074405680-5.txt",2018-06-12T07:44:05 TZ +01:00
"./dir","file20180612074405680-6.txt",2018-06-12T07:44:05 TZ +01:00
"./dir","file20180612074405680-7.txt",2018-06-12T07:44:05 TZ +01:00
"./dir","file20180612074405680-8.txt",2018-06-12T07:44:05 TZ +01:00
"./dir","file20180612074405680-9.txt",2018-06-12T07:44:05 TZ +01:00
"./dir","file20180612074405680-10.txt",2018-06-12T07:44:05 TZ +01:00
"./dir","20180612074405680_ZTRDEFO_Blackgen.jpg",2018-06-12T07:44:05 TZ +01:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .