Exclude similar file names from an array using shell script

Question

I have array of files , example file=[a.yml, a.json,b.yml,b.json]. I'm iterating using for loop. I need to exclude the .json files from being executed when it has both .ymlor.yaml and .json in the array. but if I have only .json in the array (example [a.json,b.json], it needs to pass through the loop. Is that possible with shell script ?

Basically i'm trying to compare the strings in an array and exclude the duplicate dynamically.

Is this possible with shell ?

filename=$(git show --pretty="format:" --name-only $CODEBUILD_RESOLVED_SOURCE_VERSION)
echo "$filename"
mkdir report || echo "dir report exists"
for file in ${filename}; do
    echo ${file}
    ext=${file##*.}
    if [ $ext == "yaml" ] || [ $ext == "yml" ] || [ $ext == "json" ]; then
        if [ ${file} != "buildspec.yml" ] && [ ${file} != "stackupdatebuildspec.yml" ] && [ ${file} != "specs.json" ]; then
            stack=$(echo ${file} | cut -d "." -f 1)
            stackName="${stack//[\/]/-}"
            echo ${stackName}
            howmany() { echo $#; }
            numOfFilesValidated=$(howmany $listOfFilesToScan)
            echo "=========================================== Syntax validation started =============================================================="
            cfSyntaxLogFile="cf-syntax-validation-output"
            numOfFailures=0
            numOfValidatedFiles=0
            for file_to_scan in $listOfFilesToScan; do
                if [[ $(cfn-lint -t "$file_to_scan" --parameter-values-path "${stack}.json" --append-rules ./append_rules --override-spec ./over_ride_spec/spec.json |& tee -a $cfSyntaxLogFile) == "" ]]; then
                    echo "INFO: Syntax validation of template $file: SUCCESS"
                    ((numOfValidatedFiles++))
                else
                    echo "ERROR: Syntax validation of template $file: FAILURE"
                    ((numOfFailures++))
                fi
            done'''

Answer 1

You can see if you have the other value before moving on: Check if a bash array contains a value.

You could also keep just the .json files in the array and check later in the loop for the existence of the .yml file.

bash has a nice parameter substitution for that:

${parameter%word}

Remove matching suffix pattern.

In your case, it would be something like this (stripping the .json and adding the .yml:

if [ ! -f "${filename%.json}.yml" ]
then
    # process
fi

Answer 2

Try something like this:

declare -A fhash

# Load $files array with 'git show -z' - NUL-separated filenames in case
# of spaces, newlines, etc.
mapfile -d '' -t files < <(git show -z --pretty="format:" --name-only "$CODEBUILD_RESOLVED_SOURCE_VERSION")

# build an associative array (hash) from the files array, so we can easily
# check if a matching .yml filename exists for .json and .yaml files.
for f in "${files[@]}"; do
  fhash["$f"]=1
done

# now process each of the filenames in the $files array.
for f in "${files[@]}"; do
  # ignore these filenames
  [[ $f =~ ^(buildspec.yml|stackupdatebuildspec.yml|specs.json)$ ]] && continue

  base=${f%.*}       # get base filename
  ext=".${f##*.}"    # get file's "extension"

  # ignore '.json' and .'yaml' files if there is a matching .yml filename.
  # also ignore .json files if there is a matching .yaml filename
  [[ $ext =~ \.(json|yaml)$ ]] && [ "${fhash[$base.yml]}"  -eq 1 ] && continue
  [[ $ext =~ \.json$        ]] && [ "${fhash[$base.yaml]}" -eq 1 ] && continue

  # The code so far has skipped/ignored all of the files we don't want to
  # process, so you can do whatever you need with "$f".

  # ... your code here ...
done