Diversion Diversion 5.1.3 - the Router Ad-Blocker, May 09, 2024

[HorseCalledHorse]

Seems like it, if you want to use commands... would sure be nicer if there was an interface. Did you check articles like this?

How to use SCP to transfer files in the macOS Terminal | AppleInsider

Secure Copy is a UNIX standard used to transfer files from one computer to another. He's how to use the function in macOS, all via the Terminal window.

appleinsider.com

I did. And still failed.

 

[Viktor Jaep]

I did. And still failed.

You mind posting some of the output you're typing in and errors it's returning? Perhaps we can help figure out why?

Edit: perhaps create a new topic, so we can get this one back on topic.

 

[HorseCalledHorse]

Had an odd issue over the last couple of days where the macOS and iOS mail clients on my network stopped showing inline images in emails and presented me with this message:

Your network preferences prevent content from loading privately.

(Pressing a "Load images" button then does the trick).

I had never encountered this message before. I do not have Apple Private Relay enabled on any of my devices, but I do have Mail Privacy Protection enabled, which has never caused an issue before with my Merlin/Diversion/Skynet setup.

Searching for solutions, I found this article, which explains the problem: https://robpickering.com/macos-monterey-and-mail-privacy/

The solution offered was to whitelist these two domains: mask.icloud.com and mask-h2.icloud.com.

As far as I could tell, neither site was blocked by my Diversion block list (I'm using the default Large list), but I added them to the allowlist and everything now appears to work again.

But I'm interested in learning why the problem surfaced in the first place. Did a Diversion or Skynet blocklist update mean these two domains were inadvertently added to the list of blocked domains?

I'm also interested to know if allowing these domains means my macOS and iOS devices will now bypass Diversion for all DNS queries (even though I don't use, and never have used, Apple Private Relay). This seems to be a concern for Pi Hole, which blocks these domains by default: https://docs.pi-hole.net/ftldns/configfile/?h=mask#icloud_private_relay

 

[dave14305]

But I'm interested in learning why the problem surfaced in the first place. Did a Diversion or Skynet blocklist update mean these two domains were inadvertently added to the list of blocked domains?

Running an alpha version?

rc: disable the use of Apple's iCloud Private Relay when enabling "Pr… · RMerl/asuswrt-merlin.ng@b155542

…event client auto DoH" Reference: https://developer.apple.com/support/prepare-your-network-for-icloud-private-relay/

github.com

 

[HorseCalledHorse]

Running an alpha version?

rc: disable the use of Apple's iCloud Private Relay when enabling "Pr… · RMerl/asuswrt-merlin.ng@b155542

…event client auto DoH" Reference: https://developer.apple.com/support/prepare-your-network-for-icloud-private-relay/

github.com

You may have hit the nail on the head, dave14305! It's not technically an alpha, but I did just update to the beta of 386.13 for my RT-AC86U. I'm assuming that change is also in that firmware. I'll post in the thread and ask.

So what has that commit changed? And is adding those two domains to the Diversion allowlist the right way to go about fixing my problem? As I said, I don't use Apple Private Relay, but the Mail Privacy Protection appears to use the same servers.

EDIT: Kicking myself now for not realising this was a firmware problem. It's even mentioned in the change notes! Setting Prevent Client Auto DoH to disabled in the WAN DNS setting should do the trick.

 

[Aziron5]

Hi

Slowly converting and adjusting my denylists. Report: the sort and verify function doesn't recognize duplicates if line differs at all, like having a suffix comment.

au.download.windowsupdate.com
au.download.windowsupdate.com ## COMMENT

 

[thelonelycoder]

Hi

Slowly converting and adjusting my denylists. Report: the sort and verify function doesn't recognize duplicates if line differs at all, like having a suffix comment.

au.download.windowsupdate.com
au.download.windowsupdate.com ## COMMENT

It’s not a duplicate line if it differs, that’s how sort works.
You edited the list manually, and Diversion 5.x no longer allows comments in any of the files for simplicity.

 

[Aziron5]

It’s not a duplicate line if it differs, that’s how sort works.
You edited the list manually, and Diversion 5.x no longer allows comments in any of the files for simplicity.

Sure, but ...

This is the result of the first "verify and sort" step in the Edit Denylist menu, without processing.
Processing all lists then goes to remove the comments in the "denylist" as well, but if it's the "denylist.conf" that actually gets parsed by dnsmasq and diversion, where comments probably aren't even supported anyway and the ladder sign means something else, what's the point of removing the comments in the processed "denylist" as well?

Processing of the lists does this and more, what is the point of a separate pre-verification step then if it's not really needed necessairly? But ofcourse I do understand having extra use-case functionality, nothing wrong, just wondering what was the thinking behind it and perhaps how it could be useful in more use cases or improved.

The process I use right now is(and I'm not claiming I'm some linux expert with hotkeys and work environment all tuned up, I'm super busy with other chores atm and hurrying with this):

1. 1. Local PC: denylist_description_comments_date_version-num.txt
2. 2. Upload to Router
3. 3. Delete "denylist"
4. 4. Rename Uploaded "denylist_description_comments_date_version-num.txt" to "denylist"
5. 5. SSH Login to Router, Diversion ...,... "Process All Lists"
6. 6. SSH Diversion follow DNSMASQ to verify processing
7. ... rinse repeat

Here's the idea, it would be a killer feature to have a multiple denylist stash saved on the router/jffs/usb-storage, aka profile banks, preloaded versions/editions of denylists, and then be able to switch via SSH which one to load/activate and do other actions. Whitelist stash as well!
Could possibly even be timed let's say "enable this whitelist stash for X minutes or Y hours, and switch back to stash 03", maybe I'm asking for too much, but I wouldn't be surprised if this idea was floated around in the community before, doesn't it make sense?

This was something I had on my mind before the latest NXDOMAIN transition, didn't spoke about it yet until now.

Sometimes I want to just quickly disable/enable a particular domain (to let a specific update through, to get something that requires it) without touching other rules, and doing this process of dealing with upload/download/edit/rename and keep everything organized can become confusing and time consuming.

By the way, wasn't there a "Process Blacklist" previously in older versions, and the equivalent "Process Denylist" isn't there now anymore, not that's it's that of a big deal, just something I noticed.

Thanks

 

[thelonelycoder]

Sure, but ...

This is the result of the first "verify and sort" step in the Edit Denylist menu, without processing.
Processing all lists then goes to remove the comments in the "denylist" as well, but if it's the "denylist.conf" that actually gets parsed by dnsmasq and diversion, where comments probably aren't even supported anyway and the ladder sign means something else, what's the point of removing the comments in the processed "denylist" as well?

Processing of the lists does this and more, what is the point of a separate pre-verification step then if it's not really needed necessairly? But ofcourse I do understand having extra use-case functionality, nothing wrong, just wondering what was the thinking behind it and perhaps how it could be useful in more use cases or improved.

The process I use right now is(and I'm not claiming I'm some linux expert with hotkeys and work environment all tuned up, I'm super busy with other chores atm and hurrying with this):

1. 1. Local PC: denylist_description_comments_date_version-num.txt
2. 2. Upload to Router
3. 3. Delete "denylist"
4. 4. Rename Uploaded "denylist_description_comments_date_version-num.txt" to "denylist"
5. 5. SSH Login to Router, Diversion ...,... "Process All Lists"
6. 6. SSH Diversion follow DNSMASQ to verify processing
7. ... rinse repeat

Here's the idea, it would be a killer feature to have a multiple denylist stash saved on the router/jffs/usb-storage, aka profile banks, preloaded versions/editions of denylists, and then be able to switch via SSH which one to load/activate and do other actions. Whitelist stash as well!
Could possibly even be timed let's say "enable this whitelist stash for X minutes or Y hours, and switch back to stash 03", maybe I'm asking for too much, but I wouldn't be surprised if this idea was floated around in the community before, doesn't it make sense?

This was something I had on my mind before the latest NXDOMAIN transition, didn't spoke about it yet until now.

Sometimes I want to just quickly disable/enable a particular domain (to let a specific update through, to get something that requires it) without touching other rules, and doing this process of dealing with upload/download/edit/rename and keep everything organized can become confusing and time consuming.

By the way, wasn't there a "Process Blacklist" previously in older versions, and the equivalent "Process Denylist" isn't there now anymore, not that's it's that of a big deal, just something I noticed.

Thanks

That‘s a lot of words and ideas. And I have reasons to do as I did and how I handle these lists and what features I am willing to add.
One wants this, the next wants that feature. I cannot satisfy all.

It takes a lot of time and requires recoding a lot to make a simple change in the deny and allow lists. It is as it is and it works AFAIK.

 

[ugandy]

the deny list. i want to block some sites for my kids during certain hours of the day. i imagined i could use crontab to modify the denylist and trigger a re process...

I cannot do something like this shell command:
> diversion "el 3"

would the following trigger a re process of the deny/allow lists?
> diversion bu

 

[thelonelycoder]

would the following trigger a re process of the deny/allow lists?

Yes diversion bu does that too.

 

[Aziron5]

Sorry, there is indeed a "Process Denylist" option there.

 

[bibikalka]

Installed 386.13_0 on AC86U. Every other reboot appear to be back to the prior troubles. Forced diversion 5.1.1 to force update itself, will see how it goes!

Apr 15 08:30:32 RT-AC86U-9988 dnsmasq[5555]: FAILED to start up
Apr 15 08:30:32 RT-AC86U-9988 dnsmasq[5584]: FAILED to start up
Apr 15 08:30:34 RT-AC86U-9988 dnsmasq[6658]: FAILED to start up
Apr 15 08:30:38 RT-AC86U-9988 dnsmasq[6852]: FAILED to start up
Apr 15 08:30:40 RT-AC86U-9988 dnsmasq[8805]: FAILED to start up
Apr 15 08:30:42 RT-AC86U-9988 dnsmasq[8918]: FAILED to start up
Apr 15 09:03:26 RT-AC86U-9988 dnsmasq[5122]: FAILED to start up
Apr 15 09:03:27 RT-AC86U-9988 dnsmasq[5704]: FAILED to start up
Apr 15 09:03:28 RT-AC86U-9988 dnsmasq[5780]: FAILED to start up
Apr 16 04:06:47 RT-AC86U-9988 dnsmasq[4257]: FAILED to start up
Apr 16 04:06:48 RT-AC86U-9988 dnsmasq[4292]: FAILED to start up
Apr 16 04:06:56 RT-AC86U-9988 dnsmasq[10874]: FAILED to start up
Apr 16 04:06:57 RT-AC86U-9988 dnsmasq[11150]: FAILED to start up
Apr 16 04:11:09 RT-AC86U-9988 dnsmasq[22860]: FAILED to start up
Apr 16 04:11:10 RT-AC86U-9988 dnsmasq[22898]: FAILED to start up
Apr 16 04:11:31 RT-AC86U-9988 dnsmasq[24256]: FAILED to start up
Apr 16 04:11:32 RT-AC86U-9988 dnsmasq[24286]: FAILED to start up

 

[Rhialto]

Just noticed this...

In Options the Blocking list is set to update at 2am every Sunday. Does it update or not? I'm confused.

 

[Ripshod]

@Rhialto Which list are you using? Does a manual update work? Have you tried swapping to a different one, then swapping back?

 

[thelonelycoder]

In Options the Blocking list is set to update at 2am every Sunday. Does it update or not? I'm confused.

I have seen this message posted here multiple times now. Don't know where it comes from but I will investigate.

In any case, re-selecting the list will fix this:
- Using the SSH UI (b, 1): Re-select the Large list
- Using the WebUI (In the Lists tab): Select the Standard blocking list type and click "Save type". Let it do its thing and then come back and select the Large blocking list type. The two steps are necessary as directly selecting the Large type will not offer the Save type option.

 

[Rhialto]

@Rhialto Which list are you using? Does a manual update work? Have you tried swapping to a different one, then swapping back?

Large list like shown in 1st column in original picture.

There is no manual update button.

 

[Rhialto]

- Using the WebUI (In the Lists tab): Select the Standard blocking list type and click "Save type". Let it do its thing and then come back and select the Large blocking list type. The two steps are necessary as directly selecting the Large type will not offer the Save type option.

Great, this has fixed it...

Merci! I see Blocked domains count is also much higher now...

 

[thelonelycoder]

Diversion 5.1.2 is now available

What's new in this patch update
- Fixes disappearing entries in the hostslist. Please run a manual update in the WebUI or SSH UI to check if this is the case for your installation after updating. Thanks @Rhialto and others for reporting.
- Now shows encountered blocking list update errors in the WebUI.
- A couple of logical corrections.

 

[thelonelycoder]

Well, it finally happened - I met a forum member and user of my scripts in real life here in my hometown Lucerne, Switzerland. @visortgw is in town for two days with his charming wife and friends for a trip across part of Europe. Thank you for dinner, I enjoyed the in person exchange after so many years of collaboration and discussions on this board.