-1
cat ips.txt | xargs -I% curl -s http://ipinfo.io/%/geo > temp.txt

Here ips.txt is a file with a list of IP addresses, one on each line. Sample:

103.240.8.0 
116.236.230.0

This works perfectly fine with the terminal in Mac:

{
"ip": "103.240.8.0",
"city": "Nagpur",
"region": "Maharashtra",
"country": "IN",
"loc": "21.1500,79.1000"
"postal": "440002"
}

{
"ip": "116.236.230.0",
city": "Shanghai",
"region": "Shanghai",
"country": "CN",
"loc": "31.0456,121.3997"
"postal": "440002"
}

but throws the following error in Linux and also in cygwin.

xargs: curl: No such file or directory

Please help!

  • 4
    If you type curl (just that word, no arguments) to the Linux shell, it may output instructions on how to install it. If not, please tell us the Linux distribution you're running. – Mark Plotnick Apr 20 '17 at 18:39
  • See if you have wget instead of curl. – Barmar Apr 20 '17 at 19:09
2

It will work in Linux if you install curl. Or try

xargs -I% wget -q -O - http://ipinfo.io/%/geo <ips.txt >temp.txt
  • Hey is there a way to bulk map 40,000 IP addresses faster? I have tried data scraping with python as well along with the Linux shell command but both are very slow and I have about 600 files each with about 40k unique IP addresses! Although the Linux command is faster, it is still slow for those many addresses. – Rohan Sharma Apr 23 '17 at 9:54
0

For faster processing consider using GNU Parallel:

cat ips.txt | parallel -j 100 curl -s http://ipinfo.io/{}/geo > temp.txt

It also makes sure the output from different jobs are not mixed together, so if you use the output you are guaranteed that you will not get half-a-line from two different jobs.

GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to.

If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:

Simple scheduling

GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:

GNU Parallel scheduling

Installation

For security reasons you should install GNU Parallel with your package manager, but if GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:

(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README

Learn more

See more examples: http://www.gnu.org/software/parallel/man.html

Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html

Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.