1

I have array of files , example file=[a.yml, a.json,b.yml,b.json]. I'm iterating using for loop. I need to exclude the .json files from being executed when it has both .ymlor.yaml and .json in the array. but if I have only .json in the array (example [a.json,b.json], it needs to pass through the loop. Is that possible with shell script ?

Basically i'm trying to compare the strings in an array and exclude the duplicate dynamically.

Is this possible with shell ?


filename=$(git show --pretty="format:" --name-only $CODEBUILD_RESOLVED_SOURCE_VERSION)
echo "$filename"
mkdir report || echo "dir report exists"
for file in ${filename}; do
    echo ${file}
    ext=${file##*.}
    if [ $ext == "yaml" ] || [ $ext == "yml" ] || [ $ext == "json" ]; then
        if [ ${file} != "buildspec.yml" ] && [ ${file} != "stackupdatebuildspec.yml" ] && [ ${file} != "specs.json" ]; then
            stack=$(echo ${file} | cut -d "." -f 1)
            stackName="${stack//[\/]/-}"
            echo ${stackName}
            howmany() { echo $#; }
            numOfFilesValidated=$(howmany $listOfFilesToScan)
            echo "=========================================== Syntax validation started =============================================================="
            cfSyntaxLogFile="cf-syntax-validation-output"
            numOfFailures=0
            numOfValidatedFiles=0
            for file_to_scan in $listOfFilesToScan; do
                if [[ $(cfn-lint -t "$file_to_scan" --parameter-values-path "${stack}.json" --append-rules ./append_rules --override-spec ./over_ride_spec/spec.json |& tee -a $cfSyntaxLogFile) == "" ]]; then
                    echo "INFO: Syntax validation of template $file: SUCCESS"
                    ((numOfValidatedFiles++))
                else
                    echo "ERROR: Syntax validation of template $file: FAILURE"
                    ((numOfFailures++))
                fi
            done'''
10
  • 1
    1. quote your variables. curly-braces are not a substitute for quoting, they serve a different purpose (disambiguating variable names from adjacent text). 2. you say "array of files", but you are not using an array. You are using, at best, a single newline-delimited string. Use an array. 3. i don't even know where to begin with the rest of your script.
    – cas
    Jun 5, 2021 at 6:39
  • @cas Thanks for your inputs. Im a newbie to bash and will make the code looks understandable .
    – Sriram Ram
    Jun 6, 2021 at 7:04
  • @cas Thanks for your inputs. Im a newbie to bash and will make the code looks understandable . The variable file name will be assigned with the list of files that was changed as part of a commit. So, the values for the filename is dynamic. My concern is, if i commit file1.yml and file2.json the loop will be iterated for 2 times and which does the same operation twice. So is there any way that we can compare and exclude .json if .yml for the same filename is present. TIA
    – Sriram Ram
    Jun 6, 2021 at 7:16
  • Please show us (in your question, and formatted as code) the output from the first echo command Jun 6, 2021 at 7:23
  • You mention only yml suffix files in your question text but your code also considers the yaml suffix. If this is required please say so in your text. Jun 6, 2021 at 7:24

2 Answers 2

1

You can see if you have the other value before moving on: Check if a bash array contains a value.

You could also keep just the .json files in the array and check later in the loop for the existence of the .yml file.

bash has a nice parameter substitution for that:

${parameter%word}

Remove matching suffix pattern.

In your case, it would be something like this (stripping the .json and adding the .yml:

if [ ! -f "${filename%.json}.yml" ]
then
    # process
fi
1

Try something like this:

declare -A fhash

# Load $files array with 'git show -z' - NUL-separated filenames in case
# of spaces, newlines, etc.
mapfile -d '' -t files < <(git show -z --pretty="format:" --name-only "$CODEBUILD_RESOLVED_SOURCE_VERSION")

# build an associative array (hash) from the files array, so we can easily
# check if a matching .yml filename exists for .json and .yaml files.
for f in "${files[@]}"; do
  fhash["$f"]=1
done

# now process each of the filenames in the $files array.
for f in "${files[@]}"; do
  # ignore these filenames
  [[ $f =~ ^(buildspec.yml|stackupdatebuildspec.yml|specs.json)$ ]] && continue

  base=${f%.*}       # get base filename
  ext=".${f##*.}"    # get file's "extension"

  # ignore '.json' and .'yaml' files if there is a matching .yml filename.
  # also ignore .json files if there is a matching .yaml filename
  [[ $ext =~ \.(json|yaml)$ ]] && [ "${fhash[$base.yml]}"  -eq 1 ] && continue
  [[ $ext =~ \.json$        ]] && [ "${fhash[$base.yaml]}" -eq 1 ] && continue

  # The code so far has skipped/ignored all of the files we don't want to
  # process, so you can do whatever you need with "$f".

  # ... your code here ...
done
4
  • but when i tried the above snippet, it still iterates through the .json file even if the .yml | .yaml extension is present in the array.
    – Sriram Ram
    Jun 6, 2021 at 12:27
  • try it now. the ext=${f##*.} was stripping the . before the extension, but my test was expecting it. fixed now.
    – cas
    Jun 6, 2021 at 14:29
  • Great Thanks @cas it works!!. i'm using aws cli for creating/updating stack. can i use ${filename}.(yml|yaml)$ so it maps filename.yml or filename.yaml and picks the file from a directory. is above syntax valid one ?
    – Sriram Ram
    Jun 6, 2021 at 16:31
  • I don't know what your question means. There's nothing invalid about that as a regex but: a) as I've already mentioned, curly braces are not a substitute for quoting; b) where and how do you want to use that? You can use it for a regex match (i.e. on the RHS of =~ inside a [[ ... ]] test) but not, e.g., as an index for an associative array (that requires a fixed string, not a regex)....or anywhere else in bash (=~ only works inside [[ ... ]]). You can also use it with an external program that understands extended regexps, e.g. awk or sed -E or grep -E.
    – cas
    Jun 7, 2021 at 2:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .