1

I have written a script that checks for two particular extensions in a zip archive. A couple checks are made that determine if the archive has only 2 files then it process them. If the archive has less than 2 it will move the archive to a "BAD" folder. If the archive has more than 2 it will move the archive to a "FIX" directory.

The process step is to extract the files, and rename them to the same name that the zip archive has.

This all works fine, when conditions are perfect. But when they are not perfect... it gets ugly.

I am running into a case where even though the files may have the correct 2 files and extension, if the files have special characters (such as Ø, backticks, comma, apostrophe, etc.)... they get processed (I assume) like regex syntax or expressions.

Here is the code:

#! /bin/bash

prefix="0000_"

mkdir -p ${prefix}{DONE,FIX,BAD}

shopt -s nocaseglob
for i in *.ZIP;
    do
        zip_name="$i"
        pair_exists=$(unzip -Z1 "$i" | grep -E -- '.cdg|.CDG|.mp3|.MP3' | wc -l)
        log="${prefix}LOG.txt"

        if [ $pair_exists -eq 2 ]
            then
                cdg_name=$(unzip -Z1 "$i" | grep -E -- '.cdg|.CDG' | awk '{print substr($0,index($0,$1))}')
                mp3_name=$(unzip -Z1 "$i" | grep -E -- '.mp3|.MP3' | awk '{print substr($0,index($0,$1))}')
                new_cdg_name="$(echo "${zip_name%.*}.cdg")"
                new_mp3_name="$(echo "${zip_name%.*}.mp3")"
                7za x "$i" -aoa -y -ba >> ./$log
                mv ./"$cdg_name" ./"$new_cdg_name"
                mv ./"$mp3_name" ./"$new_mp3_name"
                mv ./"$zip_name" ./${prefix}DONE/

        elif [ $pair_exists -gt 2 ]
            then
                echo ""
                echo "NEEDS FIXED: $zip_name"
                mv ./"$zip_name" ./${prefix}FIX/

        elif [ $pair_exists -lt 2 ]
            then
                echo ""
                echo "ARCHIVE IS BAD: $zip_name"
                mv ./"$zip_name" ./${prefix}BAD/

        else
                echo ""
                echo "MUST BE BROKE!"
                echo ""
        fi
    done
exit

Everything is cool, until I get to the mv statements.

                mv ./"$cdg_name" ./"$new_cdg_name"
                mv ./"$mp3_name" ./"$new_mp3_name"

I feel that perhaps mv may not be the correct method to be using here, but I was having serious issues with rename as well. I think more so that I need some code that tells the commands to no treat any characters within the variables as actions to be performed.

Here is what I see happening...

----- EXAMPLE 1 -----

ZIP FILE NAME: 

     CB30035-05 - SIMPLE PLAN  - I'D DO ANYTHING.zip

FILES WITHIN ZIP:

     CB30035-05 - Simple Plan  - I'd Do Anything.cdg
     CB30035-05 - Simple Plan  - I'd Do Anything.mp3

ERROR:

     mv: cannot stat './CB30035-05 - Simple PlanI\'d Do Anything.cdg': No such file or directory
     mv: cannot stat './CB30035-05 - Simple PlanI\'d Do Anything.mp3': No such file or directory

----- EXAMPLE 2 -----

ZIP FILE NAME:

     CBSE5-0068 - CARPENTERS, THE - FOR ALL WE KNOW.zip

FILES WITHIN ZIP:

     cbscdge450-5-0068 - Carpenters - For All We Know.cdg
     cbscdge450-5-0068 - Carpenters - For All We Know.mp3

ERROR:
mv: cannot stat './cbscdge450-5-0068 - Carpenters - For All We Know.cdg\ncbscdge450-5-0068 - Carpenters - For All We Know.mp3': No such file or directory

I have searched around for a similar issue, but the topics I have found did not really fit my issue or some of the code was a bit over my head to try and figure out how to incorporate into my script.

I would appreciate any help. Thank you!

(NOTE: I am aware my "awk" in the script above is not doing anything. I moved to "unzip -Z1" and that appears to have solved my previous efforts to just get the filename out of the zip files. I have left it in and adjusted it just to keep it around in case I needed it.)



EDIT 2020120601:


In response to @Wieland, I removed the double space from the zip file name. But left the double space on the files inside. I won't be able to fix the inside of every file as there are to many, so I need to figure out how to fix them as is. The removal of the double space on the zip file did not change my results.

In response to @steeldriver, Here is a bit more information. Below is the return for each of the commands I have attempted to use (notice the 7za... it does not have a way of just producing file names, so I was using awk previously).

I also changed the code to read \.cdg$|\.CDG$|\.mp3$|\.MP3$ and this did not change my results, but I agree would cover that base.

zipinfo -1 "CB30035-05 - SIMPLE PLAN - I'D DO ANYTHING.zip" 

  CB30035-05 - Simple PlanI'd Do Anything.cdg
  CB30035-05 - Simple PlanI'd Do Anything.mp3

unzip -Z1 "CB30035-05 - SIMPLE PLAN - I'D DO ANYTHING.zip" 

  CB30035-05 - Simple PlanI'd Do Anything.cdg
  CB30035-05 - Simple PlanI'd Do Anything.mp3

7za -ba l "CB30035-05 - SIMPLE PLAN - I'D DO ANYTHING.zip"           

   2003-06-27 14:41:56 ....A      1516512       379652  CB30035-05 - Simple PlanI'd Do Anything.cdg
   2003-06-27 14:42:22 ....A      3369876      3112004  CB30035-05 - Simple PlanI'd Do Anything.mp3



EDIT 2020120701:


@G-Man Says 'Reinstate Monica'

Thank you for wrapping so much detail around your explanations. I appreciate that. I will be incorporating your suggested mods to the scripting. In regards to

If you “changed the code to read .cdg$|.CDG$|.mp3$|.MP3$ and this did not change my results”, then you did it wrong...

I am pretty sure I followed your example closely. I had commented on saying I did that, but the backslash was stripped out of my remark. I had also already implemented the grep "c". But apparently seriously missed the "i" option. That would definitely clean that up.

Here is the code as it stands now...

#! /bin/bash

prefix="00001_"

mkdir -p ${prefix}{DONE,FIX,BAD}

shopt -s nocaseglob
for i in *.ZIP;
    do
        zip_name="$i"
        pair_exists=$(unzip -Z1 "$i" | grep -Eci -- '\.cdg$|\.mp3$')

        if [ $pair_exists -eq 2 ]
            then
                cdg_name=$(unzip -Z1 "$i" | grep -E -- '\.cdg$|\.CDG$')
                mp3_name=$(unzip -Z1 "$i" | grep -E -- '\.mp3$|\.MP3$')

                base_name="${zip_name%.*}"
                new_cdg_name="$base_name.cdg"
                new_mp3_name="$base_name.mp3"

                        printf 'cdg_name = [%s]\n' "$cdg_name"
                        printf 'mp3_name = [%s]\n' "$mp3_name"

                unzip -qq "$i"
                mv -- "${cdg_name}" "${new_cdg_name}"
                mv -- "${mp3_name}" "${new_mp3_name}"
                mv ./"$zip_name" ./${prefix}DONE/

        elif [ $pair_exists -gt 2 ]
            then
                echo ""
                echo "NEEDS FIXED: $zip_name"
                mv ./"$zip_name" ./${prefix}FIX/

        elif [ $pair_exists -lt 2 ]
            then
                echo ""
                echo "ARCHIVE IS BAD: $zip_name"
                mv ./"$zip_name" ./${prefix}BAD/

        else
                echo ""
                echo "HMM"
                echo ""
        fi
    done
exit

I incorporated your changes as well.

In regards to a debug, I once again had used "echo". This is what is in my test script...

echo ""
echo "-----"
echo   $pair_exists
echo   $zip_name
echo   $cdg_name
echo   $mp3_name
echo   $new_cdg_name
echo   $new_mp3_name
echo   $prefix
echo   $log
echo "-----"
echo ""

It was producing the same thing as your "printf". However I am quite fond of your fancy ways, and will be adopting your style. :)

To answer point 5. I got that idea from using 7z.exe. Actually looking at it through a GUI. Now I am going to post a screen shot of what my Windows box see's and again I will post what my linux box produces.

WINDOWS:

enter image description here

LINUX:

enter image description here

I have NO CLUE why that is happening. After looking at that for a while, a part of me thinks just maybe... I screwed up my environment somehow. I have not ran a crossed a issue like this before, and it leaves me pulling my hair out!

NOW! Having given you all this information, and maybe I have not given you enough. But I need to tell you. After having made your changes to the code, it is producing the correct outcome! Even though my system still strips the " - " out of the file names, it is producing the desired end result. I wanted the files to take on the name of the zip file and that is what it's doing now.

enter image description here

The changes I made...

  1. cleaned up the pair_exists with
    unzip -Z1 "$i" | grep -Eci -- '\.cdg$|\.mp3$'

  2. clean up the cdg and mp3_name extraction and removed the awk
    cdg_name=$(unzip -Z1 "$i" | grep -E -- '\.cdg$|\.CDG$')
    mp3_name=$(unzip -Z1 "$i" | grep -E -- '\.mp3$|\.MP3$')

  3. incorporated the new base_name suggestion and removal of echo
    base_name="${zip_name%.*}"
    new_cdg_name="$base_name.cdg"
    new_mp3_name="$base_name.mp3"

  4. added the "printf" debug lines (will comment them out when doing 150k+ file)
    printf 'cdg_name = [%s]\n' "$cdg_name"
    printf 'mp3_name = [%s]\n' "$mp3_name"

  5. I change the decompressor to unzip to stay consistent with the toolset
    unzip -qq "$i"

I don't know where the fix happened, but I really appreciate your help @G-Man Says 'Reinstate Monica' and helping me through this and giving some very solid advice.

Thank you!


5
  • The FILES WITHIN ZIP from your first example have a different number of spaces than in the error messages from mv. Which one is correct?
    – Wieland
    Dec 6, 2020 at 21:07
  • I see what your seeing now. I am not certain how that space was corrected. I copied those lines directly from the terminal screen.
    – K.P.
    Dec 6, 2020 at 21:33
  • 1
    I wonder if this issue is related to using zipinfo / unzip -Z to extract the filenames, but are actually using 7za to extract the files? perhaps there is a difference in how they handle special and/or unicode characters? (I don't think it's significant here, but your grep regexes should probably be \.cdg$|\.CDG$ etc. since unescaped . matches any character) Dec 6, 2020 at 22:48
  • I will test that out and also post 7za, zipinfo, and unzip results for comparison. Thank you.
    – K.P.
    Dec 6, 2020 at 23:05
  • I posted a edit to the original information, along with some details of the results for 7za, zipinfo and unzip.
    – K.P.
    Dec 6, 2020 at 23:30

1 Answer 1

1
  1. steeldriver’s comment almost certainly identifies part of the problem.  If you “changed the code to read \.cdg$|\.CDG$|\.mp3$|\.MP3$ and this did not change my results”, then you did it wrong — specifically, you did it incompletely.  Part of the problem lies in the command

    cdg_name=$(unzip -Z1 "$i" | grep -E -- '.cdg|.CDG' | awk '{print substr($0,index($0,$1))}')
    

    which must be changed to

    cdg_name=$(unzip -Z1 "$i" | grep -E -- '\.cdg$|\.CDG$' | awk '{print substr($0,index($0,$1))}')
    

    because cbscdge450-5-0068 - Carpenters … matches .cdg, and so cdg_name is getting set to the concatenation of both names, separated by a newline.  This is quite clear from the mv error message.

  2. Just for simplification purposes, you could change

    unzip -Z1 "$i" | grep -E -- '\.cdg$|\.CDG$|\.mp3$|\.MP3$' | wc -l
    

    to

    unzip -Z1 "$i" | grep -Ec -- '\.cdg$|\.CDG$|\.mp3$|\.MP3$'
    

    (using the count option of grep) or even

    unzip -Z1 "$i" | grep -Eci -- '\.cdg$|\.mp3$'
    

    (using the ignore case option of grep).

  3. There’s hardly ever any reason to say $(echo "something").  In fact, I’m inclined to say that there’s never a reason to do that, but somebody will probably identify a weird corner case where it’s beneficial.  Specifically,

    new_cdg_name="$(echo "${zip_name%.*}.cdg")"
    new_mp3_name="$(echo "${zip_name%.*}.mp3")"
    

    can be changed to

    new_cdg_name="${zip_name%.*}.cdg"
    new_mp3_name="${zip_name%.*}.mp3"
    

    I might even go so far as to change them to

    base_name="${zip_name%.*}"
    new_cdg_name="$base_name.cdg"
    new_mp3_name="$base_name.mp3"
    

    P.S. Strictly speaking, the quotes in the above aren’t necessary, but it’s good practice to use them all the time unless you have a good reason not to.
    P.P.S. In the wrong context, $(echo "something") could do things like change Plan  - I to Plan - I (i.e., compress multiple spaces into one).

  4. At the risk of being politically incorrect, imagine that you’re the police trying to solve a crime, and that all you can do is sit outside the criminals’ headquarters with binoculars.  Wouldn’t it be so much better if you could get a wiretap or an informant, so you could know what was happening inside the building?  Debugging is like trying to solve a crime — while the external information (i.e., the output of zipinfo and 7za, run separately) is important to understand the problem, it really helps to get the inside information.  So, as a routine debugging step, I recommend adding statements like

    printf 'cdg_name = [%s]\n' "$cdg_name"
    printf 'mp3_name = [%s]\n' "$mp3_name"
    

    to the script. This would have made it obvious that cdg_name was getting set to the concatenation of the two names, and it might help you track down where I'd Do Anything changes to I\'d Do Anything.

  5. Strictly speaking, this should be a comment, but, as long as I’m here: where did you get the idea that the names of the member files within CB30035-05 - SIMPLE PLAN - I'D DO ANYTHING.zip were

    • CB30035-05 - Simple Plan - I'd Do Anything.cdg and
    • CB30035-05 - Simple Plan - I'd Do Anything.mp3

    when you haven’t shown us any command that shows anything between the Plan and the I?

  6. As I mentioned above, the I\'d Do Anything is a puzzle.  But do you actually have any examples of problems related to commas, backticks, or non-ASCII characters (like ‘é’, ‘Φ’, ‘Ø’, ‘θ’, ‘½’ or ‘∞’)?

3
  • Thank you! I gave a full response to your questions and comments in my post above.
    – K.P.
    Dec 7, 2020 at 23:13
  • (1) The way Windows lists the “Simple Plan” filenames differently certainly is peculiar. I’m glad it isn’t causing any further problems. (2) The I\'d Do Anything puzzle remains a mystery. I’m glad it went away by itself. (3) No big deal, but I’m not steeldriver. Dec 8, 2020 at 3:33
  • Oh geez! I am sorry! Those credits were definitely yours. My apologize. Thank you G-Man. I do seriously appreciate your help!
    – K.P.
    Dec 8, 2020 at 7:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.