iblocklist.com generic ipset loader for ipset v6 and v4

[redhat27]

I'm truly stumped and wondering if there's something else in my config that is causing problems.

Try eliminating each script one by one. Have BLOCKLIST_INDEXES="" (empty) or disable running ya-malware-block, see which one is the one stopping pandora. Are you able to ping pandora.com when it is being blocked?

 

[truglodite]

I hope this isn't deja vu, and it is still working a few hours from now (will report back if not), but for now it seems to be working. The 'magical' procedure I used this morning (following a reboot yesterday with iblocklist-loader off, ya-malware on, and ab-solution on):

I removed all lists from the loader script (BLOCKLIST_INDEXES=""), deleted .gz blockfiles, left the whitelist items in there, and ran the iblocklist-loader script.
I tested pandora on my phone (through router obviously) and it still worked.
Then I added lists one by one, starting with 17, then reran the loader script, and tested pandora.
Pandora didn't fail as I worked my way through 2,10,11,13,15,17 (in reverse the order if that matters... hopefully not).

Then, to test reliability, I uncommented the loader script starting line in my services-start, and rebooted the router (with cache set to Y). A few minutes after reboot, the ya-malware and iblocklist scripts did their thing, and so far so good... pandora is still working! (*fingers crossed*) I'm not sure how this worked vs just loading the script with all the lists as usual, but whatever magic was involved seems to work?

FWIW, the bottom of my services-start looks like this:

###end, ab-solution maintained parts

# start, added for Ya-Malware
sleep 600
sh /jffs/scripts/ya-malware-block.sh
# end, added for Ya-Malware

# start, added for iblocklist-loader script
sh /jffs/scripts/iblocklist-loader.sh
# end, added for iblocklist-loader script

Since it took so long after reboot to load both scripts (due to the 'sleep 600'), I'm curious if it would be better to move the blocklist loader above ya-malware to start it earlier. Not sure it matters for security, but I suppose if it's not too much extra exposure, I'd rather leave the unbroken unfixed if you know what I mean.

Thanks,
Kevin

[Here I am, about 30min later, once again without Pandora. It appears whatever bug is causing this takes a long time to manifest. The syslog did show the lists all loaded and processed. No syslogs that coincide with the loss of pandora. Any extra logging I could use to illuminate this?

I noticed in my syslogs, when ya-malware ran it 'downloaded' but then showed '(0) ip's added'. Not sure wth that happened ?!!! (browsing through winscp, everything looked ok). So I deleted yam, rebooted, and reinstalled yam. After running it added IPs as usual. I followed up by running iblocklist-loader (with all the lists & whites) and it appeared to also load correctly. Pandora is working again... for now?

I sure hope I didn't accidentally add/delete stuff from yam to cause this. Is it possible the cru yam run to act differently than a manual cli run? Will report back later...]

 

[redhat27]

There could be multiple factors at play here. Try going a whole day with each of the ad-block, ya-malware-block, iblocklists, etc. Also, I found this and this on the web. Maybe its waiting to serve an ad?

 

[truglodite]

After reading your post, I got the idea to just play with skipping songs (that usually brings the ads faster) and sure enough, that killed Pandora. I went to my router, started playing with the whitelist and mxtoolbox, and noticed a few things. First, the exact urls are a moving target that is unfortunately not whitelistable (given my knowledge). For example, mediaserver-dc6-tX-X-vX.pandora.com where X seems to change between songs, and they're all showing up as blocked using MatchIP (all on list #2). These whitelistings may or may not have helped:

pandora.com
tuner.pandora.com
feeds.pandora.com
beta.savagebeast.com

Now with these in the whitelist, say I add our moving target "mediaserver..." url to the whitelist, the buffering does eventually stop and a song plays after 20-30sec. If I hit skip, it buffers the next song but won't play it, then after the 20-30sec wait, it randomly skips to another song on it's own, then starts to play it. I also noticed the album art no longer shows up no matter what I do. If I close the app altogether and restart it fresh, the buffering won't stop, until I locate the moving target url and whitelist it (weird?).

So, I removed list 2 from my loader script, and now it seems to work no matter what I do (reboot router, reload the app, etc). So I we'd need wildcard whitelisting to use list 2 with pandora?

As mentioned before, I've run abs and yam for a while and never had problems. This seems to be a problem purely with using the loader's list #2. It seems the list contains a lot of IP's, of which I'm sure most are one's I should be blocking. Any alternatives?

Thanks,
Kev

ps: sorry for blowing up this thread on this matter... felt like a space junk tracker alarm... wasn't sure what was relevant or not but didn't want to miss any potentially important details. You should see how long my mxtoolbox page got in the process of discovering the moving target IPs.

 

[Jack Yaz]

Which list is #2? A lot of the iblocklists aren't well maintained anymore

 

[truglodite]

Jack, it's Bluetack level1:

List002="level1 Bluetack http://list.iblocklist.com/?list=ydxerpxkpcfqjaybcssw src"

Searching that file for "pandora" brings up a lot of IP's, many of which I didn't see on my phone while troubleshooting. Not sure if it's well maintained or not, but this pesky line appears to be up to date:

Pandora Media, Inc:208.85.40.0-208.85.47.255

Most of the pandora associated connections my phone hits when playing music are within that range. There were a few out of range, including savagebeast.com and facebook.com. Savagebeast.com was blocked, facebook wasn't... there may be more but I've put enough IT time in today.

Kev

 

[Lynoad]

The script is taking hours to run today, with every gzipped download resulting in

gzip: stdin: unexpected end of file
iblocklist-loader.sh: Loaded I-Blocklist* blocklist with 1 entries

I've made sure nothing's blocking my access to www. iblocklist.com. I'm not using a blacklist, I have just an empty file for that. And for testing, I've just commented all but four lists. What else might I be overlooking as a cause for this?

Partial dir listing for my eventual downloads:

0 Mar 30 13:43 ipset_lists/I-BlocklistTimeWarnerCble.gz
0 Mar 30 13:25 ipset_lists/I-BlocklistCoxComm.gz
0 Mar 30 13:08 ipset_lists/I-BlocklistATT.gz
0 Mar 30 12:50 ipset_lists/I-BlocklistVerizon.gz
0 Mar 30 12:33 ipset_lists/I-BlocklistCablevision.gz

Edit: Also, I updated gzip and sed, just in case.

 

[Lynoad]

My installation of the iblocklist-loader.sh started working again spontaneously so I'm presuming some WAN issue between me and iblocklist.com. Then today, it stopped again. I notice that whether I have cache set to Y or N, the script attempts to wget -q -O /jffs/ipset_lists/I-Blocklist*.gz. In attempt to force a redownload I deleted my gzips and am now stuck with hang, whether caching or not. Do I need to download gzipped files manually from iblocklist.com on a "first" run? I don't recall ever having to do so.

EDIT: The script runs if (and only if?) I first kill dnsmasq. [Although it seems many of the downloaded files are zero-byte. ]

 

[Lynoad]

...am now stuck with hang, whether caching or not.

I've looked at the ungzipped contents of files which partially download before a hang and have concluded there's a parsing issue. They are all ranges with hyphens, and not CIDRs. I realize this is an effectively dead thread but appreciate if anyone might confirm/deny that the blockfiles from IBlocklist.com are no longer working in this script. Thanks.

 

[Lynoad]

I have fairly recent ipset save files available for restores as necessary (eg, after reboot) but if anyone knows whether there's an alternative script for peerguardian-like importation, or an impending rewrite of this one to deal with the loss of freely compatible lists from iblocklist, please make my day. H/T!

 

[Adamm]

I have fairly recent ipset save files available for restores as necessary (eg, after reboot) but if anyone knows whether there's an alternative script for peerguardian-like importation, or an impending rewrite of this one to deal with the loss of freely compatible lists from iblocklist, please make my day. H/T!

iBlocklist isn't a great source imo. Most of the lists are either old revisions of stolen lists from other providers or haven't been updated in 1 - 3 years. Firehol provide a much better database of lists and analytic data to go with it.

 

[Lynoad]

iBlocklist isn't a great source imo. Most of the lists are either old revisions of stolen lists from other providers or haven't been updated in 1 - 3 years. Firehol provide a much better database of lists and analytic data to go with it.

Yep, thought so. More work for me since utilization will appear to require configuring an IDS, which I absolutely would do if responsible for other users. Thanks very much @Adamm in any event!

 

[Lynoad]

I tried redirecting source URLs to working ones but run into problems I've identified as:

1. I-Blocklist changed schema at some point to require https for list retrieval and there is some hardcoded use of http with wget certificate-ignore, probably not really an issue because of how I'm trying to reimplement my saved sets from lists garnered elsewise;
   
2. Script requirement for gz format with unclear (to me) possibility for override - again, perhaps no longer pertinent - and most consequentially,
3. I have had some success using other sources for lists obtained outside of this script, from which I've been able to leverage ipset v6 save/restore sets. When it worked, the script seems to have also established references for each set created, and I don't know how to establish those despite a read-through (to best of my ability) of the ipset manpage.

I'd love to be able to use ipsets on the fly, saving & restoring more or less manually as necessary without having to resort to configuration of a snort-type IDS. Is there a simple way to set refs so that ipset will write my set selection for iptables to use?