5

I am using iptables with ipset on an Ubuntu server firewall. I am wondering if there is a command for importing a file containg a list of ip's to ipset. To populate an ipset, right now, I am adding each ip with this command:

ipset add manual-blacklist x.x.x.x

It would be very helpfull if I can add multiple ip's with a single command, like importing a file or so.

At command

 for ip in `cat /home/paul/ips.txt`; do ipset add manual-blacklist $ip;done

I get this response

resolving to IPv4 address failed to parse 46.225.38.155

for each ip in ips.txt

I do not know how to apply it.

3
  • Well, why are you using two different commands?  ipset add badipipset add manual-blacklist. Nov 2, 2017 at 5:29
  • Sorry, I copied the wrong command, the good one is ipset add manual-blacklist x.x.x.x
    – byte00
    Nov 2, 2017 at 5:30
  • I'll bet that your ips.txt file was created under DOS/Windows and has CR/LF on the line ends, so tyour $ip variable has an IP address ending in the bad character CR. Sep 10, 2018 at 21:26

4 Answers 4

13

You can use ipset save/restore commands.

ipset save manual-blacklist

You can run above command and see how you need to create your save file.

Example output:

create manual-blacklist hash:net family inet hashsize 1024 maxelem 65536 
add manual-blacklist 10.0.0.1 
add manual-blacklist 10.0.0.2

And restore it with below command.

ipset restore -! < ips.txt

Here we use -! to ignore errors mostly because of duplication.

1
  • 2
    It must be noted that restore works blazingly fast compared to add when adding lots of records. On my PC it took 17 sec to add 10000 lines, and only 0.05 sec to restore
    – AlexM
    Dec 5, 2018 at 5:53
1

If your list is geting really realy big for example 200 000 lines or even more and you have enough memory in server to support this, it is nice to run this whole oneliner cycle with nice:

nice -n 5 bash -c "for IP in \$(cat textfile.txt); do ipset add <setname> \$IP -exist timeout <seconds>; done"

Then your other services can step up from bash and you do not get any networking or mysql or apache or other services lag because bash is using all the resources.

1
  • 1
    If you have to add many lines, it's better to use restore command, which works very fast as opposed to adding the lines one by one
    – AlexM
    Dec 5, 2018 at 5:49
1

I know this is a few years old but why not just make a simple shell script with all the IP addreses and then run that shell script. I add hundreds of IP addresses this way.

Example: (done in Centos 8 using vi as the editor)

vi manual-blacklist.sh

Then in the manual-blacklist.sh file add all your IP addresses as listed below: (Press the i key to insert your lines)

ipset add manual-blacklist 123.45.67.111
ipset add manual-blacklist 123.45.67.112
ipset add manual-blacklist 123.45.67.113

And so on until all your IP addresses are added. To speed things up I use Excel to make my lists of IP addresses and then export it as a text file and then do a copy and paste into my shell script file.

Save the file. Press ESC key and then :wq! and then press the Enter key to save the file.

Then to run it use the following command:

./manual-blacklist.sh

All your IP addresses will be added to your manual-blacklist ipset file. And if it notices any duplicates it will ignore them as they were already added prior.

Then save this file: (I save it to my home directory and the etc directory)

ipset save > /etc/ipset.conf
ipset save > /home/username/ipset.conf

And to restore:

ipset restore -f /etc/ipset.conf

The -f switch specifies a filename to print into instead of stdout (list or save commands) or read from instead of stdin (restore command).

Hope this helps.

1
  • Hello! As a conclusion, I am currently using the restore version. I prepare my file in excel and then copy / paste in txt and restore. It works. Thanks for this solution too.
    – byte00
    Apr 1 at 4:37
0

Try this command:

for ip in $(cat </file.txt>); do ipset -A <set-name> $ip;done

If you still get the a error then check your text file for spaces and unreadable\weird characters in a advanced text editor (Notepad++, SublimeText).Delete the spaces and unreadable\weird characters and try again.

1
  • Note that it is much quicker to use restore than the ipset -A command. Feb 11, 2019 at 0:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.