Curl bad URL (3)

Question

I iterate through a file and I'd like to curl an api with the line (word) in the URL.

The content of the list2csv.sh is:

#!/bin/bash

for word in $( cat $1 )
do
    echo "https://api.dictionaryapi.dev/api/v2/entries/en/$word" 
    curl "http://api.dictionaryapi.dev/api/v2/entries/en/$word" 
done

Content of list file:

timber
clatter

When I run ./list2csv.sh list the output is:

https://api.dictionaryapi.dev/api/v2/entries/en/timber
curl: (3) URL using bad/illegal format or missing URL
https://api.dictionaryapi.dev/api/v2/entries/en/clatter
curl: (3) URL using bad/illegal format or missing URL

and if I try to curl the echoed URLs, I get:

$ curl https://api.dictionaryapi.dev/api/v2/entries/en/timber
[{"word":"timber","phonetic":"ˈtɪmbə","phonetics":[{"text":"ˈtɪmbə","audio":"//ssl.gstatic.com/dictionary/static/sounds/20200429/timber--_gb_1.mp3"}],"origin":"Old English in the sense ‘a building’, also ‘building material’, of Germanic origin; related to German Zimmer ‘room’, from an Indo-European root meaning ‘build’.","meanings":[{"partOfSpeech":"noun","definitions":[{"definition":"wood prepared for use in building and carpentry.","example":"the exploitation of forests for timber","synonyms":["wood","logs","firewood","planks","wood products","forest","woodland","woods","lumber"],"antonyms":[]},{"definition":"personal qualities or character.","example":"she is frequently hailed as presidential timber","synonyms":[],"antonyms":[]}]}]}]%

$ curl https://api.dictionaryapi.dev/api/v2/entries/en/clatter
[{"word":"clatter","phonetic":"ˈklatə","phonetics":[{"text":"ˈklatə","audio":"//ssl.gstatic.com/dictionary/static/sounds/20200429/clatter--_gb_1.mp3"}],"origin":"Old English (as a verb), of imitative origin.","meanings":[{"partOfSpeech":"noun","definitions":[{"definition":"a continuous rattling sound as of hard objects falling or striking each other.","example":"the horse spun round with a clatter of hooves","synonyms":[],"antonyms":[]}]},{"partOfSpeech":"verb","definitions":[{"definition":"make or cause to make a continuous rattling sound.","example":"her coffee cup clattered in the saucer","synonyms":["rattle","clank","clink","clunk","clang","bang","blatter"],"antonyms":[]}]}]}]%

I use macOS, but I've tried in other os's as well.

Answer 1

Your input file is a DOS text file. It contains an extra carriage-return character at the end of each line, and these are becoming part of the word variable's value, which provokes that specific error from curl:

$ curl $'http://localhost/somepath\r'
curl: (3) URL using bad/illegal format or missing URL

Without the carriage-return at the end, I get the expected error (there is no web-server running on this machine):

$ curl 'http://localhost/somepath'
curl: (7) Failed to connect to localhost port 80 after 0 ms: Connection refused

Consider converting the input file to a Unix text file using e.g. dos2unix.

You also have issues in your code in that you force the shell to read the whole input file in one go, split the contents of the file on spaces, tabs, and newlines, and the perform filename globbing on the resulting words. Additionally, you split the pathname given on the command line in the same way.

It would be safer to use a while loop to read a word at a time:

#!/bin/sh

cat -- "$@" |
while IFS= read -r word; do
    curl "https://api.dictionaryapi.dev/api/v2/entries/en/$word"
done

Or, using xargs,

#!/bin/sh

cat -- "$@" |
xargs -t -I {} \
    curl 'https://api.dictionaryapi.dev/api/v2/entries/en/{}'

Both the above scripts concatenates all files given as arguments on the command line, and passes the output to curl, one line at a time.

Note that I have also corrected the HTTP to HTTPS in your URL. With HTTP, you'll get redirected to the HTTPS site by the remote service, which require you to use -L with curl to follow automatically.