iblocklist.com generic ipset loader for ipset v6 and v4

[Jack Yaz]

One of the teachers is a bing search engine user. But it is being blocked by multiple sources:

MatchIP 204.79.197.200
204.79.197.200 not found in WhitelistDomains
204.79.197.200 found in BlacklistDomains
204.79.197.200 not found in BluetackProxyCIDR
204.79.197.200 not found in BluetackProxySingle
204.79.197.200 not found in BluetackWebexploitCIDR
204.79.197.200 not found in BluetackWebexploitSingle
204.79.197.200 not found in BluetackDshieldCIDR
204.79.197.200 not found in BluetackDshieldSingle
204.79.197.200 not found in BluetackSpiderCIDR
204.79.197.200 not found in BluetackSpiderSingle
204.79.197.200 found in YAMalwareBlock1IP
204.79.197.200 not found in YAMalwareBlock2IP
204.79.197.200 not found in YAMalwareBlockCIDR
204.79.197.200 not found in BlockedCountries
204.79.197.200 not found in TorNodes
204.79.197.200 not found in BruteForceLogins
204.79.197.200 not found in Whitelist
204.79.197.200 not found in Blacklist
204.79.197.200 found in YAMalwareBlock1IP
204.79.197.200 not found in YAMalwareBlock2IP
204.79.197.200 not found in YAMalwareBlockCIDR
204.79.197.200 not found in Blacklist
204.79.197.200 found in MicrosoftSpyServers

As a result, how do you recommend I proceed in getting it to work?

Whitelist the IP/domain since Whitelist takes precedence?

 

[Xentrk]

Whitelist the IP/domain since Whitelist takes precedence?

Good call. I've come up with this so far:
1. Remove 13.107.21.200 from create-ipset-list.sh
2. Add bing.com to /jffs/ipset_lists/whitelists-domains.txt

But I don't see a whitelist for ya-malware-block.sh script

 

[redhat27]

@Xentrk Interesting that the bing call is also identified as a telemetry server. Obviously bing is a very popular search engine. I would make an exception and remove it in both places.

 

[Jack Yaz]

Good call. I've come up with this so far:
1. Remove 13.107.21.200 from create-ipset-list.sh
2. Add bing.com to /jffs/ipset_lists/whitelists-domains.txt

But I don't see a whitelist for ya-malware-block.sh script

You shouldn't need one, asssuming your whitelist accept is at the top, prior to the drop rules, it will be accepted. I think!

iptables -t raw -S

 

[redhat27]

But I don't see a whitelist for ya-malware-block.sh script

If you run the iblocklist-loader script right after the ya-malware-block script, the WHITELIST_DOMAINS_FILE from iblocklist-loader script would ensure that any domain you whitelist is bypassed from blocking.

Edit: @Jack Yaz said the same thing. Thanks for helping out.

 

[Xentrk]

If you run the iblocklist-loader script right after the ya-malware-block script, the WHITELIST_DOMAINS_FILE from iblocklist-loader script would ensure that any domain you whitelist is bypassed from blocking.

Edit: @Jack Yaz said the same thing. Thanks for helping out.

Thank you @Jack Yaz and @redhat27

I reordered the scripts in services-start as follows and bing.com is still blocked

sh /jffs/scripts/create-ipset-lists.sh
sh /jffs/scripts/IPSET_Block.sh init
sh /jffs/scripts/ya-malware-block.sh
sh /jffs/scripts/iblocklist-loader.sh

contents of whitelist-domains.txt

mail.live.com # Hotmail blocked by TBG Primary-Threats
icloud.com # blocked by TBG Primary-Threats
apple.com # blocked by TBG Primary-Threats
bing.com # bing search engine

MatchIP 13.107.21.200

ping bing.com
PING bing.com (13.107.21.200): 56 data bytes
MatchIP 13.107.21.200
13.107.21.200 not found in WhitelistDomains
13.107.21.200 found in BlacklistDomains
13.107.21.200 not found in BluetackProxyCIDR
13.107.21.200 not found in BluetackProxySingle
13.107.21.200 not found in BluetackWebexploitCIDR
13.107.21.200 not found in BluetackWebexploitSingle
13.107.21.200 not found in BluetackDshieldCIDR
13.107.21.200 not found in BluetackDshieldSingle
13.107.21.200 not found in BluetackSpiderCIDR
13.107.21.200 not found in BluetackSpiderSingle
13.107.21.200 found in YAMalwareBlock1IP
13.107.21.200 not found in YAMalwareBlock2IP
13.107.21.200 not found in YAMalwareBlockCIDR
13.107.21.200 not found in BlockedCountries
13.107.21.200 not found in TorNodes
13.107.21.200 not found in BruteForceLogins
13.107.21.200 not found in Whitelist
13.107.21.200 not found in Blacklist
13.107.21.200 found in YAMalwareBlock1IP
13.107.21.200 not found in YAMalwareBlock2IP
13.107.21.200 not found in YAMalwareBlockCIDR
13.107.21.200 not found in Blacklist
13.107.21.200 not found in MicrosoftSpyServers

 

[Jack Yaz]

I'm not sure of the code, but can IP addresses go into Whitelist? If not, i might be able to throw together a whitelist script

 

[Xentrk]

I noticed the bing.com ip address changed from my first post on this topic and the second. So, before running the MatchIP, I would first ping bing.com to see what IP it was using. I would only get one ping reply before it would hang and I had to ctrl-C to get the prompt back.

 

[redhat27]

I noticed the bing.com ip address changed from my first post on this topic and the second. So, before running the MatchIP, I would first ping bing.com to see what IP it was using. I would only get one ping reply before it would hang and I had to ctrl-C to get the prompt back.

Can you re-run the ya-malware-block once more (no need to re-download the ya-malware-block script or delete your ya-malware-block.url_list sources) and check if bing.com can be accessed properly?

I've removed the 2 IPs and the offending entry from the blacklist-domains.

EDIT: You need to make the corresponding changes to you local /jffs/ipset_lists/blacklist-domains.txt or just get the latest one from github with

wget --no-check-certificate -O /jffs/ipset_lists/blacklist-domains.txt https://raw.githubusercontent.com/shounak-de/iblocklist-loader/master/blacklist-domains.txt

If you download the blacklist-domains.txt from github, you'll lose your local changes to that file (if you have any) of course.

 

[Xentrk]

Can you re-run the ya-malware-block once more (no need to re-download the ya-malware-block script or delete your ya-malware-block.url_list sources) and check if bing.com can be accessed properly?

I've removed the 2 IPs and the offending entry from the blacklist-domains.

EDIT: You need to make the corresponding changes to you local /jffs/ipset_lists/blacklist-domains.txt or just get the latest one from github with

wget --no-check-certificate -O /jffs/ipset_lists/blacklist-domains.txt https://raw.githubusercontent.com/shounak-de/iblocklist-loader/master/blacklist-domains.txt

If you download the blacklist-domains.txt from github, you'll lose your local changes to that file (if you have any) of course.

Kawpkoon Krap (Thank you in Thai) @redhat27. bing.com is working now. I had to reboot after downloading the updated blacklist for it to work though.

 

[redhat27]

I had to reboot after downloading the updated blacklist for it to work though.

My bad. I should have mentioned you'd need to run the iblocklist-loader script once after you make the blacklist-domains.txt change

 

[Xentrk]

My bad. I should have mentioned you'd need to run the iblocklist-loader script once after you make the blacklist-domains.txt change

Not a problem. Thanks again for taking the time to test on your end and making the changes so I could get bing search engine working.

 

[redhat27]

All users who use AB-Solution and ya-malware-block please take a look at post #163 on that thread. I'll update the info on the OP of ya-malware-block on how to whitelist some unintended blocked sources using the whitelist-domains file of this iblocklist-loader

Updated OP with a whitelisting section on that thread.

 

[Xentrk]

I updated ya-malware-block.sh. Having trouble with yahoo.com loading slowly or not at all. Yahoo mail can not be accessed from windows 10 client. Okay on ipad app. The block appears to be from iblocklist-loader.sh. But I am suffering from script burnout and perhaps not seeing clearly. So another pair of eyes would be helpful.

I tried whitelisting the yahoo IP addresses in ya-malware-blocks.white

98.138.253.109
98.139.183.24
206.190.36.45
209.73.190.11
209.73.190.12

and the url in whitelist-domains.txt

yahoo.com # bocked by BluetackSpiderCIDR
mg.smtp.yahoo.com # blocked by BluetackSpiderCIDR

This is one of the mg.mail.yahoo.com ip addresses

admin@RT-AC88U:/jffs/scripts# MatchIP 209.73.190.12
209.73.190.12 not found in BlacklistDomains
209.73.190.12 not found in BluetackProxyCIDR
209.73.190.12 not found in BluetackProxySingle
209.73.190.12 not found in BluetackWebexploitCIDR
209.73.190.12 not found in BluetackWebexploitSingle
209.73.190.12 not found in BluetackDshieldCIDR
209.73.190.12 not found in BluetackDshieldSingle
209.73.190.12 found in BluetackSpiderCIDR
209.73.190.12 not found in BluetackSpiderSingle
209.73.190.12 not found in YAMalwareBlockCIDR
209.73.190.12 not found in YAMalwareBlock2IP
209.73.190.12 not found in YAMalwareBlock1IP
209.73.190.12 not found in BlockedCountries
209.73.190.12 not found in TorNodes
209.73.190.12 not found in Whitelist
209.73.190.12 not found in Blacklist
209.73.190.12 not found in MicrosoftSpyServers
209.73.190.12 not found in WhitelistSRCPort
209.73.190.12 not found in Whitelist
209.73.190.12 not found in Blacklist
admin@RT-AC88U:/jffs/scripts# nslookup mg.mail.yahoo.com
Server:    127.0.0.1
Address 1: 127.0.0.1 localhost.localdomain

Name:      mg.mail.yahoo.com
Address 1: 2001:4998:28:800::4001 e2.ycpi.vip.laa.yahoo.com
Address 2: 209.73.190.11 e1.ycpi.vip.laa.yahoo.com
Address 3: 209.73.190.12 e2.ycpi.vip.laa.yahoo.com

 

[redhat27]

I don't see an ipset called WhitelistDomains in the list... Is the path to your WHITELIST_DOMAINS_FILE set properly?

Also, after you add the entry on the whitelist-domains.txt, make sure you run the iblocklist-loader once.

Also note, it is not necessary to add these to ya-malware-block.whites at all. You'd only need to add an IP if it is present in any of the YAMalwareBlock?IP sets.

 

[Xentrk]

I don't see an ipset called WhitelistDomains in the list... Is the path to your WHITELIST_DOMAINS_FILE set properly?

Also, after you add the entry on the whitelist-domains.txt, make sure you run the iblocklist-loader once.

Also note, it is not necessary to add these to ya-malware-block.whites at all. You'd only need to add an IP if it is present in any of the YAMalwareBlock?IP sets.

It may have been a copy/paste issue. Is this the white list entry I should be seeing followed by the Black List entry?

209.73.190.12 not found in WhitelistDomains
209.73.190.12 not found in BlacklistDomains

And the source of the following CIDR is iblocklist-loader or ya-malware-filter?

190.12 found in BluetackSpiderCIDR

Thanks!

 

[bayern1975]

how to whitelist something? i create manualy whitelist.txt, add one domain inside but didn`t whitelisted?

 

[Xentrk]

how to whitelist something? i create manualy whitelist.txt, add one domain inside but didn`t whitelisted?

The whitelist file is located here:
/jffs/ipset_lists/whitelist-domains.txt

You need to edit this line in the script if you want it to reside in a different location:
WHITELIST_DOMAINS_FILE="/jffs/ipset_lists/whitelist-domains.txt"

 

[bayern1975]

The whitelist file is located here:
/jffs/ipset_lists/whitelist-domains.txt

yes, but when i put one domain inside, reset,restart iblocklist-loader i see zero domain in syslog? but should be one?

Firewall: iblocklist-loader.sh: Added WhitelistDomains (0 entries)

 

[Xentrk]

yes, but when i put one domain inside, reset,restart iblocklist-loader i see zero domain in syslog? but should be one?

Firewall: iblocklist-loader.sh: Added WhitelistDomains (0 entries)

Is your /jffs partition on a usb drive?