0

I have a bash script below that is intended to curl against a list of domains, from a file, and output the results.

#!/bin/bash
    
baseurl=https://csp.infoblox.com
domains=/home/user/Documents/domainlist
B1Dossier=/tide/api/data/threats/state/host?host=$domains
APIKey=<REDACTED>
AUTH="Authorization: Token $APIKey"
        
        
for domain in $domains; do curl -H "$AUTH" -X GET ${baseurl}${B1Dossier} > /tmp/outputfile; done

Unfortunately, the script is not going through each domain in the file whatsoever.

To help understand, I have listed the expectation/explanation of the script:

  • Within the file, /home/user/Documents/domainlist, I have a handful of domains.
  • I'm attempting to use the API to check each domain in the file, by appending the variable $domains at the end of B1Dossier
  • The expectation is that it would run the specified curl command against each domain, within the file, and output the results.

For added visibility, I included the working curl command used for a single domain below:

curl -H 'Authorization: Token <REDACTED>' -X GET https://csp.infoblox.com/tide/api/data/threats/state/host?host=<place domain here>

Can someone assist in what I'm doing wrong and how I can fix this?

3 Answers 3

2

You can read the domains from file to an array and loop for them.

baseurl="https://csp.infoblox.com"
B1Dossier="/tide/api/data/threats/state/host?host="
url="${baseurl}${B1Dossier}"

# read domains to an array
mapfile -t domains < /home/user/Documents/domainlist

# loop for domains
for d in "${domains[@]}"; do
    curl -H "$AUTH" -X GET "${url}${d}" >> temp
done

Notes:

In your command, using B1Dossier into the loop, has no effect, it seems you were waiting some kind of a recursive evaluation, because the domain is contained in B1Dossier and you loop for domain. But your url doesn't change inside the loop this way.

Also, you have to append the responses to your destination file using >> or else every next response would overwrite the previous one.

2
  • Just learnt about mapfile. +1
    – annahri
    Commented Apr 30, 2022 at 7:26
  • Thanks. Your explanation made perfect sense and explained what was going wrong. I suspected so but you were able to clarify. The solution works!
    – Nina G
    Commented May 1, 2022 at 22:25
1

You seem to be iterating over the name of the file that holds the actual domains, which is why your code is not doing what you want it to do. You also seem to be outputting everything to the same file, which you overwrite with each call to curl.

Instead of looping over a list with a shell loop, you may use curl with a configuration file that has several url settings.

The configuration file doesn't actually need to be stored on disk though, so we can create it on the fly and pass it in a pipeline to curl:

domain_list=/home/user/Documents/domainlist

base_url=https://csp.infoblox.com
endpoint=tide/api/data/threats/state/host

APIKey=<REDACTED>
auth_header="Authorization: Token $APIKey"

awk -v h="$auth_header" -v b="$base_url" -v e="$endpoint" '
        BEGIN {
                printf "header = \"%s\"\n", h
        }
        {
                printf "url = %s/%s?host=%s\n", b, e, $0
        }' "$domain_list" |
curl --config -

This first uses awk to prefix each line of the domain list with

url = https://csp.infoblox.com/tide/api/data/threats/state/host?host=

This is then used by curl as the list of URLs to contact.

The awk code also sets the authorization header once at the start of the configuration data. There is no need to explicitly specify the method.

The result is written to standard output, but you could also insert output = somefilename options before each url line to output the result to a specific file.


The same script, but possibly simplified somewhat:

domain_list=/home/user/Documents/domainlist

base_url=https://csp.infoblox.com
endpoint=tide/api/data/threats/state/host

APIKey=<REDACTED>
auth_header="Authorization: Token $APIKey"

{
    printf 'header = "%s"\n' "$auth_header"
    prefix="url = $base_url/$endpoint?host=" \
    awk '{ print ENVIRON["prefix"] $0 }' "$domain_list" 
} | curl --config -
1

Use while loop:

#!/bin/bash
    
baseurl=https://csp.infoblox.com
domains=/home/user/Documents/domainlist
APIKey=<REDACTED>
AUTH="Authorization: Token $APIKey"

while read -r domain; do
    url="${baseurl}/tide/api/data/threats/state/host?host=${domain}"
    curl -H "$AUTH" -X GET "$url" >> /tmp/outputfile
done < "$domains"

The while loop will read line by line the input file and assign it to domain variable (defined by read domain), in this case $domains.

1
  • 1
    Ow yes, I didn't double check it. Edited. @Kusalananda
    – annahri
    Commented May 1, 2022 at 2:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .