iblocklist.com generic ipset loader for ipset v6 and v4

[redhat27]

That's a lot of lists you are blocking!
Use the MatchIP function to see which list(s) are the culprit:
These are the dropbox ips:

admin@RT-AC66R-D700:/tmp/home/root# hostip dropbox.com
162.125.248.1
admin@RT-AC66R-D700:/tmp/home/root# hostip www.dropbox.com
162.125.7.1

See see which list(s) are blocking those IPs

 

[VZ3]

src is where the traffic originates. dst is the the target for the traffic. Blocking the [src] is mostly all that is needed, specifically if the idea is to block incoming traffic from outside (potentially dangerous/mal-intentioned sites on the internet)
You'd typically use [dst] to prevent your lan machines to "dial out" to unwanted destinations. Example is when you use microsoft windows and want stop your machines from sending identifying or other data to M$ telemetry servers.

Yes, I understand [src|dst] concept. But I thought, reading comments in iblocklist-loader-v2 ( "You can also control the inboud/outbound/both traffic for each of the lists: Just modify the [Traffic] column (the last one) in each list), that I can block inboud traffic from say UK but still can access UK websites myself if I will initiate the connection.

Looking further into script itself I see that this is not the case.
First, you set iptables rules in PREROUTING chain, so all local originated traffic will not hit that table going out, so it does not matter [src]or[dst] there will be no match.

Second, reply packets coming from (UK as in my example) will match blocksets in PREROUTING chain if I have [src] or will not if I have [dst]) and will act apropriatly.

So, It's not possible to block incoming access from country (UK) and still be able to browse UK websites.

I guess that was Amplatfus meant to say.

PS: on other hand WAN and LAN interfaces are all remote from router perspective... so, all of the above is true only for the router itself. Well, got a bit messed up.

 

[redhat27]

You are right, earlier version of the script used the FORWORD chain of the filter table. Please change the iptable rule to use that, instead of raw PREROUTING and see if that works.

I'll revert the script to use that when I am able to. I had changed it to raw PREROUTING without thinking much into it.

 

[VZ3]

Redhat27, thanks again for your scripts.

I was trying to get rid of script-kiddies scanning and brute-forcing my ssh, vpn ports. Not that I was worried they could break it but just annoyed with all that unwanted noise coming from different directions (mostly from China and then some).

So, I was entertained by idea of blocking those countries completely but I found that your other script - ya-malware-block based on FireHOL block-list aggregate is covering all those pesky IP's like 99% without such radical and in some sense less efficient way of blocking whole countries.

I will run it and report back if I find something worth your attention.

 

[amplatfus]

That's a lot of lists you are blocking!
Use the MatchIP function to see which list(s) are the culprit:
These are the dropbox ips:

admin@RT-AC66R-D700:/tmp/home/root# hostip dropbox.com
162.125.248.1
admin@RT-AC66R-D700:/tmp/home/root# hostip www.dropbox.com
162.125.7.1

See see which list(s) are blocking those IPs

Thank you. I have been searching for how to install MatchIP. Could you please guide me?
Because:

root@rooter:/tmp/home/root# MatchIP
-sh: MatchIP: not found

And a question, why site dropbox is blocked when all list are active with only src source, not dst?
Thank you!

 

[VZ3]

Thank you. I have been searching for how to install MatchIP. Could you please guide me?
Because:

root@rooter:/tmp/home/root# MatchIP
-sh: MatchIP: not found

And a question, why site dropbox is blocked when all list are active with only src source, not dst?
Thank you!

Put this code into /jffs/configs/profile.add and you will have MatchIP function.

MatchIP() { # Check IP against ipset lists
  if [ -z "$1" ]; then
    echo "Specify IP to check through ipset lists. Exiting."
  else
    GREEN='\033[0;32m'
    RED='\033[0;31m'
    NC='\033[0m' # No Color
    for TestList in $( (iptables -L -t raw && iptables -L) | grep "match-set" | tr -s ' ' | cut -d' ' -f7 | tr '\n' ' '); do
      ipset -q test $TestList $1 && echo -e "$1 found in ${GREEN}${TestList}${NC}" || echo -e "$1 not found in ${RED}${TestList}${NC}"
    done
  fi
}

There is no way with this script to block sites one way only. Meaning if you have [src] then ALL packets from that source will be blocked. So, if you ping your dropbox, your ping will reach dropbox but reply will be dropped on your firewall giving you impressions that you cannot reach that site.

 

[amplatfus]

Put this code into /jffs/configs/profile.add and you will have MatchIP function.

MatchIP() { # Check IP against ipset lists
  if [ -z "$1" ]; then
    echo "Specify IP to check through ipset lists. Exiting."
  else
    GREEN='\033[0;32m'
    RED='\033[0;31m'
    NC='\033[0m' # No Color
    for TestList in $( (iptables -L -t raw && iptables -L) | grep "match-set" | tr -s ' ' | cut -d' ' -f7 | tr '\n' ' '); do
      ipset -q test $TestList $1 && echo -e "$1 found in ${GREEN}${TestList}${NC}" || echo -e "$1 not found in ${RED}${TestList}${NC}"
    done
  fi
}

There is no way with this script to block sites one way only. Meaning if you have [src] then ALL packets from that source will be blocked. So, if you ping your dropbox, your ping will reach dropbox but reply will be dropped on your firewall giving you impressions that you cannot reach that site.

Appreciate your suport. Could you please share also how it must be called? Because when i did MatchIP 162.125.248.1 it not worked.

root@rooter:/tmp/home/root# MatchIP 162.125.248.1
BusyBox v1.25.1 (2017-07-16 12:57:53 EDT) multi-call binary.

Usage: grep [-HhnlLoqvsriwFE] [-m N] [-A/B/C N] PATTERN/-e PATTERN.../-f FILE [FILE]...

Search for PATTERN in FILEs (or stdin)

    -H    Add 'filename:' prefix
    -h    Do not add 'filename:' prefix
    -n    Add 'line_no:' prefix
    -l    Show only names of files that match
    -L    Show only names of files that don't match
    -c    Show only count of matching lines
    -o    Show only the matching part of line
    -q    Quiet. Return 0 if PATTERN is found, 1 otherwise
    -v    Select non-matching lines
    -s    Suppress open and read errors
    -r    Recurse
    -i    Ignore case
    -w    Match whole words only
    -x    Match whole lines only
    -F    PATTERN is a literal (not regexp)
    -E    PATTERN is an extended regexp
    -m N    Match up to N times per file
    -A N    Print N lines of trailing context
    -B N    Print N lines of leading context
    -C N    Same as '-A N -B N'
    -e PTRN    Pattern to match
    -f FILE    Read pattern from file
-sh: match-set: not found
root@rooter: /tmp/home/root# MatchIP
Specify IP to check through ipset lists. Exiting.

Thank you,
amplatfus

 

[VZ3]

well, you have to re-login first.

And show me output of :
MatchIP

let see if you got it in profile correctly, and then:

cat /jffs/configs/profile.add..

you must have some non-Linux chars in the script when you copy&paste it.

PS: or you know what... do you have ipset script installed?

show me output of :

iptables -L -t raw

 

[amplatfus]

well, you have to re-login first.

And show me output of :
MatchIP

let see if you got it in profile correctly, and then:

cat /jffs/configs/profile.add..

you must have some non-Linux chars in the script when you copy&paste it.

I restarted the router immediately after I changed the profile.add.
Output for MatchIP is:

root@rooter:/tmp/home/root# MatchIP
Specify IP to check through ipset lists. Exiting.

And cat/jffs/configs/profile.add is:

root@rooter:/tmp/home/root# cat /jffs/configs/profile.add
alias blockstats='iptables -L -v | sed "2q;d"; iptables -L -v | grep "match-set"; ip6tables -L -v | grep "match-set"'
alias blockmali='iptables -nvL logdrop --line -t filter'
MatchIP() { # Check IP against ipset lists
  if [ -z "$1" ]; then
    echo "Specify IP to check through ipset lists. Exiting."
  else
    GREEN='\033[0;32m'
    RED='\033[0;31m'
    NC='\033[0m' # No Color
    for TestList in $( (iptables -L -t raw && iptables -L) | grep
"match-set" | tr -s ' ' | cut -d' ' -f7 | tr '\n' ' '); do
      ipset -q test $TestList $1 && echo -e "$1 found in
${GREEN}${TestList}${NC}" || echo -e "$1 not found in
${RED}${TestList}${NC}"
    done
  fi
}

 

[VZ3]

I restarted the router immediately after I changed the profile.add.
Output for MatchIP is:

root@rooter:/tmp/home/root# MatchIP
Specify IP to check through ipset lists. Exiting.

And cat/jffs/configs/profile.add is:

root@rooter:/tmp/home/root# cat /jffs/configs/profile.add
alias blockstats='iptables -L -v | sed "2q;d"; iptables -L -v | grep "match-set"; ip6tables -L -v | grep "match-set"'
alias blockmali='iptables -nvL logdrop --line -t filter'
MatchIP() { # Check IP against ipset lists
  if [ -z "$1" ]; then
    echo "Specify IP to check through ipset lists. Exiting."
  else
    GREEN='\033[0;32m'
    RED='\033[0;31m'
    NC='\033[0m' # No Color
    for TestList in $( (iptables -L -t raw && iptables -L) | grep
"match-set" | tr -s ' ' | cut -d' ' -f7 | tr '\n' ' '); do
      ipset -q test $TestList $1 && echo -e "$1 found in
${GREEN}${TestList}${NC}" || echo -e "$1 not found in
${RED}${TestList}${NC}"
    done
  fi
}

You have extra LF chars in your MatchIP script, remove them with your build-in nano editor... also, your blockstats does not looks into raw CHAIN - this is where this script will install ipset blocks.

Basically you should have it like this:

alias blockstats='iptables -vL | sed "2q;d"; (iptables -vL -t raw; iptables -vL) | grep "match-set"; ip6tables -vL | grep "match-set"'

MatchIP() { # Check IP against ipset lists
  if [ -z "$1" ]; then
    echo "Specify IP to check through ipset lists. Exiting."
  else
    GREEN='\033[0;32m'
    RED='\033[0;31m'
    NC='\033[0m' # No Color
    for TestList in $( (iptables -L -t raw && iptables -L) | grep "match-set" | tr -s ' ' | cut -d' ' -f7 | tr '\n' ' '); do
      ipset -q test $TestList $1 && echo -e "$1 found in ${GREEN}${TestList}${NC}" || echo -e "$1 not found in ${RED}${TestList}${NC}"
    done
  fi
}

See, there is iptables -t raw selector. Also MatchIP function has no extra LF in grep line and line beneath.
Also give me output of
iptables -L -t raw

 

[amplatfus]

Thank you for prompt support. I pasted again form post #250 and now it works. For instance dropbox.com/ 162.125.248.

root@rooter:/tmp/home/root# MatchIP 162.125.248.1
162.125.248.1 not found in Whitelist
162.125.248.1 not found in Blacklist
162.125.248.1 found in WhitelistDomains
162.125.248.1 not found in BlacklistDomains
162.125.248.1 found in TBGHijackedCIDR
162.125.248.1 not found in TBGHijackedSingle
162.125.248.1 not found in CruzITCruzITWebAttacksCIDR
162.125.248.1 not found in CruzITCruzITWebAttacksSingle
162.125.248.1 not found in Malc0deMalc0deCIDR
162.125.248.1 not found in Malc0deMalc0deSingle
162.125.248.1 not found in CI-ArmyMaliciousCIDR
162.125.248.1 not found in CI-ArmyMaliciousSingle
162.125.248.1 not found in AbuseSpyEyeCIDR
162.125.248.1 not found in AbuseSpyEyeSingle
162.125.248.1 not found in BluetackWebexploitCIDR
162.125.248.1 not found in BluetackWebexploitSingle
162.125.248.1 not found in BluetackForumspamCIDR
162.125.248.1 not found in BluetackForumspamSingle
162.125.248.1 not found in BluetackDshieldCIDR
162.125.248.1 not found in BluetackDshieldSingle
162.125.248.1 found in BluetackHijackedCIDR
162.125.248.1 not found in BluetackHijackedSingle
162.125.248.1 not found in BluetackBadpeersCIDR
162.125.248.1 not found in BluetackBadpeersSingle
162.125.248.1 not found in BluetackProxyCIDR
162.125.248.1 not found in BluetackProxySingle
162.125.248.1 not found in BluetackSpywareCIDR
162.125.248.1 not found in BluetackSpywareSingle
162.125.248.1 not found in BluetackRangetestCIDR
162.125.248.1 not found in BluetackRangetestSingle
162.125.248.1 not found in Blacklist
root@rooter:/tmp/home/root#

All the best,
amplatfus

 

[amplatfus]

...Also give me output of iptables -L -t raw

Forgot about it. Please, this is output it is OK? 10q

root@rooter:/tmp/home/root# iptables -L -t raw
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination       

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination       
root@rooter:/tmp/home/root#

 

[VZ3]

Forgot about it. Please, this is output it is OK? 10q

root@rooter:/tmp/home/root# iptables -L -t raw
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination      

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination      
root@rooter:/tmp/home/root#

Yes, that's fine. I see you are using earlier version of ipset blocking script which did not use PREROUTING chain. That's ok.

 

[truglodite]

edit/update:
Turns out my problem with Pandora had nothing to do with the iblocklist-loader script. Last night my n66u router was experiencing issues with 100% cpu load without this script running. After turning off QOS and re-enabling NAT acceleration, everything became snappy again. Acting on a hunch this morning, I re-visited iblocklist-loader. With QOS off it works flawlessly. Lists 2,10,11,13,15,17 load much faster, and last but not least, Pandora whitelisting now works (yay music!).

On my home network with 30Mbup/30Mbdown fiber in home isp, it's common to see our machines with the lowest ping on a game server (w/o QOS). So QOS isn't exactly a necessity for me, but I wouldn't mind it if I could somehow use it to squeeze out lower pings, without destroying my router CPU, but that's way OT.

Thanks Redhat for another awesome script (I also run ya-malware).

Kev

Edit... sigh, 10min later pandora stopped again. I need a break from this lol.

 

[Builder71]

Sorry for a bit off-topic.
I have only the ya-malware-block.sh script running on my RT-N66U with the LTS firmware from John.
So I tried the script from post #250 but that returns only the prompt with no info at all.

Is it possible to make this MatchIP function work with ya-malware-block?

 

[john9527]

Sorry for a bit off-topic.
I have only the ya-malware-block.sh script running on my RT-N66U with the LTS firmware from John.
So I tried the script from post #250 but that returns only the prompt with no info at all.

Is it possible to make this MatchIP function work with ya-malware-block?

The N66 uses a different version of iptables with a different syntax for some of the commands, so the script can't find what it is looking for. I can't try it right now, but you can try changing all the instances of
grep "match-set"
to
grep "set"

If that doesn't work, more extensive changes will probably be needed.

 

[redhat27]

Is it possible to make this MatchIP function work with ya-malware-block?

@Builder71 John is correct. You have ipset 4.x. a tested version of the function for that ipset is in the wiki

 

[redhat27]

Edit... sigh, 10min later pandora stopped again. I need a break from this lol.

Not sure which blocklist is the one blocking it, but you should be able to find that out from the function mentioned in the above post. There are couple of things to try:
[1] Do not use the blocklist that blocks pandora
or
[2] Put the domains affected (eg: "pandora.com") in the whitelist-domains.txt file. A sample file of how that file should look like is here

 

[Builder71]

@Builder71 John is correct. You have ipset 4.x. a tested version of the function for that ipset is in the wiki

Awesome! Totally missed the wiki.
Great for troubleshooting to see if IP is blocked.

admin@RT-N66U:/jffs# MatchIP 8.8.8.8
8.8.8.8 not found in YAMalwareBlockCIDR
8.8.8.8 not found in YAMalwareBlock1IP

 

[truglodite]

Not sure which blocklist is the one blocking it, but you should be able to find that out from the function mentioned in the above post.

Pandora.com and tuner.pandora.com are both blocked by list#2 (but not sure this is actually the problem... explained below). I had already had this info in my post prior to editing; I should avoid editing while I am in the fog of frustration. (...tldr got the best of me)

[1] Do not use the blocklist that blocks pandora
or
[2] Put the domains affected (eg: "pandora.com") in the whitelist-domains.txt file. A sample file of how that file should look like is

[1] didn't work. I removed list 2 and still no pandora. Again, MatchIP did not turn up any other lists for the 4 pandora related IP's I have found. The pandora app did also connect to facebook.com (why? sigh...), but nslookup+matchip came up negative for facebook. I have also tried running the script with both Y and N cache settings, and deleting .gz blocking files to redownload in case they were corrupt. Again, this was foolishly edited out of my original post.

[2] pandora.com and tuner.pandora.com are both on my whitelist. Console seems to confirm they are processed correctly when I run(rerun) the script (4 ip's added from the whitelist). As with [1], I also tried both y/n cache, and deleting .gz's.

I'm truly stumped and wondering if there's something else in my config that is causing problems. To be clear, I'm currently running without QOS, and I have ab-solution and ya-malware (both the latest) running happily on my 380.68 n66u router.

This weekend I've got some extra time I can put in to this. I'm happy to try any other troubleshooting stuffs to get it working. Any further help is welcome.

Kevin