2

I iterate through a file and I'd like to curl an api with the line (word) in the URL.

The content of the list2csv.sh is:

#!/bin/bash

for word in $( cat $1 )
do
    echo "https://api.dictionaryapi.dev/api/v2/entries/en/$word" 
    curl "http://api.dictionaryapi.dev/api/v2/entries/en/$word" 
done

Content of list file:

timber
clatter

When I run ./list2csv.sh list the output is:

https://api.dictionaryapi.dev/api/v2/entries/en/timber
curl: (3) URL using bad/illegal format or missing URL
https://api.dictionaryapi.dev/api/v2/entries/en/clatter
curl: (3) URL using bad/illegal format or missing URL

and if I try to curl the echoed URLs, I get:

$ curl https://api.dictionaryapi.dev/api/v2/entries/en/timber
[{"word":"timber","phonetic":"ˈtɪmbə","phonetics":[{"text":"ˈtɪmbə","audio":"//ssl.gstatic.com/dictionary/static/sounds/20200429/timber--_gb_1.mp3"}],"origin":"Old English in the sense ‘a building’, also ‘building material’, of Germanic origin; related to German Zimmer ‘room’, from an Indo-European root meaning ‘build’.","meanings":[{"partOfSpeech":"noun","definitions":[{"definition":"wood prepared for use in building and carpentry.","example":"the exploitation of forests for timber","synonyms":["wood","logs","firewood","planks","wood products","forest","woodland","woods","lumber"],"antonyms":[]},{"definition":"personal qualities or character.","example":"she is frequently hailed as presidential timber","synonyms":[],"antonyms":[]}]}]}]%

$ curl https://api.dictionaryapi.dev/api/v2/entries/en/clatter
[{"word":"clatter","phonetic":"ˈklatə","phonetics":[{"text":"ˈklatə","audio":"//ssl.gstatic.com/dictionary/static/sounds/20200429/clatter--_gb_1.mp3"}],"origin":"Old English (as a verb), of imitative origin.","meanings":[{"partOfSpeech":"noun","definitions":[{"definition":"a continuous rattling sound as of hard objects falling or striking each other.","example":"the horse spun round with a clatter of hooves","synonyms":[],"antonyms":[]}]},{"partOfSpeech":"verb","definitions":[{"definition":"make or cause to make a continuous rattling sound.","example":"her coffee cup clattered in the saucer","synonyms":["rattle","clank","clink","clunk","clang","bang","blatter"],"antonyms":[]}]}]}]%

I use macOS, but I've tried in other os's as well.

1
  • 1
    Use https instead of http in the curl call in your script. This works: curl "https://api.dictionaryapi.dev/api/v2/entries/en/$word"
    – Edward
    Commented Jan 24, 2022 at 17:04

1 Answer 1

3

Your input file is a DOS text file. It contains an extra carriage-return character at the end of each line, and these are becoming part of the word variable's value, which provokes that specific error from curl:

$ curl $'http://localhost/somepath\r'
curl: (3) URL using bad/illegal format or missing URL

Without the carriage-return at the end, I get the expected error (there is no web-server running on this machine):

$ curl 'http://localhost/somepath'
curl: (7) Failed to connect to localhost port 80 after 0 ms: Connection refused

Consider converting the input file to a Unix text file using e.g. dos2unix.

You also have issues in your code in that you force the shell to read the whole input file in one go, split the contents of the file on spaces, tabs, and newlines, and the perform filename globbing on the resulting words. Additionally, you split the pathname given on the command line in the same way.

It would be safer to use a while loop to read a word at a time:

#!/bin/sh

cat -- "$@" |
while IFS= read -r word; do
    curl "https://api.dictionaryapi.dev/api/v2/entries/en/$word"
done

Or, using xargs,

#!/bin/sh

cat -- "$@" |
xargs -t -I {} \
    curl 'https://api.dictionaryapi.dev/api/v2/entries/en/{}'

Both the above scripts concatenates all files given as arguments on the command line, and passes the output to curl, one line at a time.

Note that I have also corrected the HTTP to HTTPS in your URL. With HTTP, you'll get redirected to the HTTPS site by the remote service, which require you to use -L with curl to follow automatically.

1
  • I forgot that I've generated the list file in windows. dos2unix solved it. I usually use https, after I've tried to debug I left http. And also cheers for the solution with xargs! That was new to me.
    – buduboti
    Commented Jan 24, 2022 at 22:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .