Menu
What's new
By:
Filters Better Search

x3mRouting x3mRouting ~ Selective Routing for Asuswrt-Merlin Firmware (1-Nov-2020)

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

abracadabra11 said:
Recently upgraded from 384.18 to 384.19 on my RT-AX3000.

Having an issue with the script update function for x3mRouting. Currently running x3m v2.0 and when I try to run option (5) Check for updates to existing x3mRouting installation, it does not update the install to 2.4 which shows as an available update in the amtm main menu.

Do I just need to run the install command to get to v2.4?
Click to expand...
Some updates require that you first run option [7] - Update x3mRouting Menu. The option will only appear if it detects a version or checksum difference with the master repo on GitHub.

1609194802640.png


Then, after updating the menu, you have to run option 5 to check for updates.
 
mister said:
Dear Xentrk,
thanks a lot for your hints and your suggestions.
Meanwhile I have a workaround for the problem, because my problem is not easy to reproduce:

1. I paused "#" all ASN number entries
2. I add the following command into the services-start file: sleep 60 && sh ./jffs/scripts/nat-start

So the nat-start is running twice. With that approach all lists are populated correctly without any lock and the routing is working.
I will test your approach as well.

Hugo.
Click to expand...
I just flashed beta 2 and think I have a better idea what is occurring. nat-start gets run as indicated by the system log entry:

Code: 
Dec 29 08:30:32 RT-AC88U-8248 custom_script: Running /jffs/scripts/nat-start

I then see the system log entries for x3mRouting. I see messages that the lock file is busy. But eventually, they start once the lock file is freed up. Then, to further complicate things, another instance of nat-start begins before the first one can complete running the x3mRouting scripts. There are more messages that the lock file is busy but they eventually get their turn and complete. I'll keep looking around to see if there is something else I can do. Such as increasing wait time, locking nat-start so only one instance can run at a time rather than x3mRouting script may be a better solution.

Did you every try replacing all of the AWS regions with just the GLOBAL region? That will speed things along.
 
@mister

Edit:

Please test this version of x3mRouting to see if it solves the boot issue. I added a counter so it loops for up to 120 seconds if the lock file is busy.

Code: 
/usr/sbin/curl --retry 3 "https://raw.githubusercontent.com/Xentrk/Asuswrt-Merlin-Linux-Shell-Scripts/master/x3mRouting.sh" -o "/jffs/scripts/x3mRouting/x3mRouting.sh"

System Log example.

Note 8607 is waiting for the lock file to be free. PID 10362 then got the lock followed by 10498 and 10633 before 8607 could run.
Code: 
Dec 29 14:25:07 RT-AC88U-8248 (x3mRouting.sh): 8607 x3mRouting Lock File in use by PID 10113 - wait time 116 secs left
Dec 29 14:25:07 RT-AC88U-8248 (x3mRouting.sh): 10113 Completed Script Execution
Dec 29 14:25:07 RT-AC88U-8248 (x3mRouting.sh): 10362 Starting Script Execution 2 0 LAZADA dnsmasq=lazada.co.th,lazada.com
Dec 29 14:25:08 RT-AC88U-8248 (x3mRouting.sh): 10362 CRON schedule created: #LAZADA# '0 2 * * * ipset save LAZADA'
Dec 29 14:25:08 RT-AC88U-8248 (x3mRouting.sh): 10362 Selective Routing Rule via WAN deleted for LAZADA fwmark 0x8000/0x8000
Dec 29 14:25:08 RT-AC88U-8248 (x3mRouting.sh): 10362 Selective Routing Rule via WAN created for LAZADA fwmark 0x8000/0x8000
Dec 29 14:25:08 RT-AC88U-8248 (x3mRouting.sh): 10362 Completed Script Execution
Dec 29 14:25:08 RT-AC88U-8248 (x3mRouting.sh): 10498 Starting Script Execution server=1 client=2
Dec 29 14:25:08 RT-AC88U-8248 (x3mRouting.sh): 10498 Completed Script Execution
Dec 29 14:25:08 RT-AC88U-8248 (x3mRouting.sh): 10633 Starting Script Execution ALL 3 BBC_WEB4 dnsmasq=2cnt.net,bbc.com,bbcverticals.com,co.uk,llnwi.net,net.uk
Dec 29 14:25:08 RT-AC88U-8248 (x3mRouting.sh): 10633 CRON schedule created: #BBC_WEB4# '0 2 * * * ipset save BBC_WEB4'
Dec 29 14:25:08 RT-AC88U-8248 (x3mRouting.sh): 10633 Selective Routing Rule via VPN Client 3 deleted for BBC_WEB4 fwmark 0x4000/0x4000
Dec 29 14:25:08 RT-AC88U-8248 (x3mRouting.sh): 10633 Selective Routing Rule via VPN Client 3 created for BBC_WEB4 fwmark 0x4000/0x4000
Dec 29 14:25:09 RT-AC88U-8248 (x3mRouting.sh): 10633 Completed Script Execution
Dec 29 14:25:11 RT-AC88U-8248 (x3mRouting.sh): 8607 CRON schedule created: #CBS_Web# '0 2 * * * ipset save CBS_Web'
Dec 29 14:25:11 RT-AC88U-8248 (x3mRouting.sh): 8607 Selective Routing Rule via VPN Client 2 deleted for CBS_Web fwmark 0x2000/0x2000
Dec 29 14:25:11 RT-AC88U-8248 (x3mRouting.sh): 8607 Selective Routing Rule via VPN Client 2 created for CBS_Web fwmark 0x2000/0x2000
Dec 29 14:25:11 RT-AC88U-8248 (x3mRouting.sh): 8607 Completed Script Execution
 
Last edited:
Xentrk said:
@Sean Rhodes
I uploaded the list to a miscellaneous repo on GitHub created from the following:
Code: 
sh /jffs/scripts/x3mRouting/x3mRouting.sh ALL 3 BBC_WEB4 dnsmasq=2cnt.net,bbc.com,bbcverticals.com,co.uk,llnwi.net,net.uk

Download BBC_WEB4 ipset list:
Code: 
/usr/sbin/curl --retry 3 "https://github.com/Xentrk/Asuswrt-Merlin-Linux-Shell-Scripts/blob/master/BBC_WEB4" -o "/opt/tmp/BBC_WEB4"

Please give it a try. You should be able to set Accept DNS Configuration to exclusive. The list will no longer populate from dnsmasq. but the routing rule will still work for the ipv4 addresses in the ipset list.
Click to expand...
I think the command is wrong, its creating an HTML doc in opt/tmp.

If I look at BBC_WEB3 I see the following:
Code: 
cat /opt/tmp/BBC_WEB3
create BBC_WEB3 hash:net family inet hashsize 1024 maxelem 65536
add BBC_WEB3 52.17.32.6
add BBC_WEB3 99.81.225.236
add BBC_WEB3 18.200.89.143
add BBC_WEB3 151.101.126.219
add BBC_WEB3 52.210.48.87
add BBC_WEB3 151.101.126.133
add BBC_WEB3 52.211.5.165
add BBC_WEB3 13.225.65.50
add BBC_WEB3 52.210.183.246
add BBC_WEB3 63.33.52.106
add BBC_WEB3 54.154.197.66
add BBC_WEB3 151.101.126.110
add BBC_WEB3 34.249.112.184
add BBC_WEB3 151.101.126.217
add BBC_WEB3 54.76.62.59
.
.
.
.
etc.
but If I look in BBC_WEB4, then its completely different:

It's actually copying the whole git webpage with head and body tags.

I will use vi and copy the raw data since I assume that was the intent
 
Sean Rhodes said:
I think the command is wrong, its creating an HTML doc in opt/tmp.

If I look at BBC_WEB3 I see the following:
Code: 
cat /opt/tmp/BBC_WEB3
create BBC_WEB3 hash:net family inet hashsize 1024 maxelem 65536
add BBC_WEB3 52.17.32.6
add BBC_WEB3 99.81.225.236
add BBC_WEB3 18.200.89.143
add BBC_WEB3 151.101.126.219
add BBC_WEB3 52.210.48.87
add BBC_WEB3 151.101.126.133
add BBC_WEB3 52.211.5.165
add BBC_WEB3 13.225.65.50
add BBC_WEB3 52.210.183.246
add BBC_WEB3 63.33.52.106
add BBC_WEB3 54.154.197.66
add BBC_WEB3 151.101.126.110
add BBC_WEB3 34.249.112.184
add BBC_WEB3 151.101.126.217
add BBC_WEB3 54.76.62.59
.
.
.
.
etc.
but If I look in BBC_WEB4, then its completely different:

It's actually copying the whole git webpage with head and body tags.

I will use vi and copy the raw data since I assume that was the intent
Click to expand...

I posted the url for the webpage and not the raw code.

Here is the corrected syntax with the updated URL:
Code: 
/usr/sbin/curl --retry 3  https://raw.githubusercontent.com/Xentrk/Asuswrt-Merlin-Linux-Shell-Scripts/master/BBC_WEB4 -o "/opt/tmp/BBC_WEB4"
 
Xentrk said:
I posted the url for the webpage and not the raw code.

Here is the corrected syntax with the updated URL:
Code: 
/usr/sbin/curl --retry 3  https://raw.githubusercontent.com/Xentrk/Asuswrt-Merlin-Linux-Shell-Scripts/master/BBC_WEB4 -o "/opt/tmp/BBC_WEB4"
Click to expand...
Thanks Xentrk
 
Rafael Viana said:
i need to get this out of my vpn 1

Americanas - A maior loja. Os menores preços.

minhaconta.americanas.com.br minhaconta.americanas.com.br


my nordvpn seems to have been blocked by the store! I need to go wan

TKS
Click to expand...
I am having issues accessing the website as well.

You won't need to install x3mRouting as there is only one IPv4 address for the website.

Code: 
 nslookup minhaconta.americanas.com.br
Server:    1.1.1.1
Address 1: 1.1.1.1 one.one.one.one

Name:      minhaconta.americanas.com.br
Address 1: 2403:6200:ffff:fca6::19fe
Address 2: 23.42.145.128 a23-42-145-128.deploy.static.akamaitechnologies.com

You can use the Policy Routing section of the OpenVPN Client Screen to route the 23.42.145.128 IPv4 address to the WAN.
 
is it ok to use latest x3mRouting with 386.1b4?
thx
 
Last edited:
ugandy said:
is it ok to use latest x3mRouting with 386.1b4?
thx
Click to expand...
Yes, with the exception that you need to download the OpenVPN Client Screen separately, if you use that option, until 386.1 goes into production. Please see link below for instructions.

www.snbforums.com

x3mRouting - x3mRouting ~ Modified OpenVPN Client Screen for 386.1 Asuswrt-Merlin release (31 Jan 2020 update)

Applies to those who use the modified OpenVPN Client Screen available in x3mRouting + 386.1 release. RMerlin made some updates to the OpenVPN Client Screen in the 386.1. I need another day to merge the change into the x3mRouting master branch. If you want to proceed immediately, you can...
www.snbforums.com www.snbforums.com
 
on 386.1b4 when i do "x3mRouting ipset_name=MYIPSET del", the vpn up/dw scripts are correctly updated, but then it issues error saying that ipset can't be deleted because kernel is using it. i rebooted and ipset is gone. expected?
 
ugandy said:
on 386.1b4 when i do "x3mRouting ipset_name=MYIPSET del", the vpn up/dw scripts are correctly updated, but then it issues error saying that ipset can't be deleted because kernel is using it. i rebooted and ipset is gone. expected?
Click to expand...
That message will appear if the iptables rule still exists for the IPSET and an attempt is made to remove it using the 'ipset destroy'. In this example, I create the IPSET list "TEMP" followed by the 'ipset destroy' command

Code: 
# x3mRouting ALL 1 TEMP ip=8.8.8.8
(x3mRouting): 6745 Starting Script Execution ALL 1 TEMP ip=8.8.8.8
Successfully added 8.8.8.8
(x3mRouting): 6745 IPSET created: TEMP
(x3mRouting): 6745 Selective Routing Rule via VPN Client 1 created for TEMP fwmark 0x1000/0x1000
(x3mRouting): 6745 iptables -t mangle -D PREROUTING -i br0 -m set --match-set TEMP dst -j MARK --set-mark 0x1000/0x1000 2>/dev/null added to /jffs/scripts/x3mRouting/vpnclient1-route-up
(x3mRouting): 6745 iptables -t mangle -A PREROUTING -i br0 -m set --match-set TEMP dst -j MARK --set-mark 0x1000/0x1000 added to /jffs/scripts/x3mRouting/vpnclient1-route-up
(x3mRouting): 6745 iptables -t mangle -D PREROUTING -i br0 -m set --match-set TEMP dst -j MARK --set-mark 0x1000/0x1000 2>/dev/null added to /jffs/scripts/x3mRouting/vpnclient1-route-pre-down
(x3mRouting): 6745 sh /jffs/scripts/x3mRouting/x3mRouting.sh ALL 1 TEMP added to /jffs/scripts/nat-start
(x3mRouting): 6745 Completed Script Execution

# ipset destroy TEMP
ipset v7.6: Set cannot be destroyed: it is in use by a kernel component
For some reason, the iptables rule may not have gotten removed. The following message should have gotten displayed.

Code: 
Deleted PREROUTING Chain $CHAIN_NUM for IPSET List $IPSET_NAME
x3mRouting first purges the ipset reference from the up/down and other files as appropriate.

x3mRouting then performs a query of the iptables PREROUTING chain for the iptables rule for the specified IPSET list before issuing the command to remove the rule. I wonder if one of the grep conditions are not being met? If you want to test the code on the command line, just copy/paste the code below to SSH prompt and substitute IPSET_NAME for a valid ipset.

Please create a test ipset list like I did above, then run this command to validate the iptables chain got created:
Code: 
iptables -nvL PREROUTING -t mangle --line

Then, run this code to see if it returns the chain number and ipset name. Substitue "IPSET_NAME " to be the name of the ipset you created.
Code: 
iptables -nvL PREROUTING -t mangle --line | grep "br0" | grep "IPSET_NAME " | grep "match-set" | awk '{print $1, $12}' | sort -nr

One thing I just noted is to remove the space after the "IPSET_NAME " and use the "-w" grep option to match on the word which is a better coding technique. So test this code as well, remember to change the name of "IPSET_NAME"

Code: 
iptables -nvL PREROUTING -t mangle --line | grep "br0" | grep -w "IPSET_NAME" | grep "match-set" | awk '{print $1, $12}' | sort -nr
 
Hi Xentrk,

EDIT: RESOLVED - See post #135 where I have left the last 2 questions. I am not deleting these posts as they may help somebody having similar issues - MODS please let me know if you prefer I delete them.

following your suggestion, I have now switched my VPN clients so I no longer have priority issues. Now all my traffic goes via VPN client 4.

However now I am facing an issue with the Netflix IPSET that always used to work...

I had to regenerate the rules for Netflix and Amazon as they used to be built on client 1 and now I need them on VPN client 4.

I have entered the following command:

x3mRouting 4 0 NETFLIX autoscan=netflix,nflx

and I get the following:

Code: 
(x3mRouting): 12365 Starting Script Execution 4 0 NETFLIX autoscan=netflix,nflx

Done.
(x3mRouting): 12365 IPSET created: NETFLIX hash:net family inet hashsize 1024 maxelem 65536
(x3mRouting): 12365 CRON schedule created: #NETFLIX# '0 2 * * * ipset save NETFLIX'
(x3mRouting): 12365 Selective Routing Rule via WAN created for NETFLIX fwmark 0x8000/0x8000
(x3mRouting): 12365 iptables -t mangle -D PREROUTING -i br0 -m set --match-set NETFLIX dst -j MARK --set-mark 0x8000/0x8000 2>/dev/null added to /jffs/scripts/x3mRouting/vpnclient4-route-up
(x3mRouting): 12365 iptables -t mangle -A PREROUTING -i br0 -m set --match-set NETFLIX dst -j MARK --set-mark 0x8000/0x8000 added to /jffs/scripts/x3mRouting/vpnclient4-route-up
(x3mRouting): 12365 iptables -t mangle -D PREROUTING -i br0 -m set --match-set NETFLIX dst -j MARK --set-mark 0x8000/0x8000 2>/dev/null added to /jffs/scripts/x3mRouting/vpnclient4-route-pre-down
(x3mRouting): 12365 sh /jffs/scripts/x3mRouting/x3mRouting.sh 4 0 NETFLIX dnsmasq=netflix.com,netflix.net,nflxext.com,nflximg.com,nflxso.net,nflxvideo.net added to /jffs/scripts/nat-start
(x3mRouting): 12365 Completed Script Execution

but when I do liststats, I see no entries created in the NETFLIX set:

Code: 
xxx@RT-AX88U-8880:/tmp/home/root# liststats
AMAZON - 2446
NETFLIX - 0

I have tried using your autoscan script and the domains are extracted:

Code: 
xxx@RT-AX88U-8880:/jffs/scripts/x3mRouting# sh autoscan.sh autoscan=net
flix,nflx

netflix.com
netflix.net
nflxext.com
nflximg.com
nflxso.net
nflxvideo.net

What am I doing wrong, why no entries are inserted in the NETFLIX IPSET?

Thank you!
 
Last edited:
Mmm I think it is just a matter of waiting as I now see 6 entries... Do they get added retrospectively?

If so, do I have a way to use both the dnsmasq filter and the ASNUM? Or the latter I execute deletes the entries created by the former?

In other words, can I do

x3mRouting 4 0 NETFLIX autoscan=netflix,nflx

and

x3mRouting 4 0 NETFLIX asnum=AS2906

Or will the second command delete whatever the first command did?

Thanks!
 
Mmm not sure what I said above is correct. I have added the ASNUM and I got 148 entries. But when I use the autoscan or the dnsmasq file the number of entries doesn't increase.

I have tried running nat-start and I see it executes a lot of commands, including those I'd expect to be deleted (i.e. with the old VPN client setup).

So I edited nat-start and now I am only left with the following entries:

Code: 
#!/bin/sh
sh /jffs/scripts/x3mRouting/x3mRouting.sh 4 0 AMAZON aws_region=US,CA,AP,CN,EU,SA,GLOBAL
sh /jffs/scripts/x3mRouting/x3mRouting.sh 4 0 NETFLIX dnsmasq=netflix.com,netflix.net,nflxext.com,nflximg.com,nflxso.net,nflxvideo.net

I have restarted the router and, when doing liststats, I see 2 entries in my ipset NETFLIX.

It looks as if it is not automatically adding the entries from dnsmasq. I have tried following the dnsmasq log in Diversion and I can see many netflix related IPs however my Netflix set doesn't get extended.

I am out of ideas... what can I do?

EDIT: I have checked the content of dnsmasq.conf.add and doesn't seem right... but I am not sure how to fix it...

Code: 
ipset=/netflix.com/nflxext.com/nflximg.net/nflxso.net/nflxvideo.net/NETFLIX      dnsmasq=netflix.com,nflxext.com,nflximg.net,nflxso.net,nflxvideo.net
ipset=/netflix.com/nflxext.com/nflxso.net/nflxvideo.net/NETFLIX      autoscan=netflix.com,nflxext.com,nflximg.net,nflxso.net,nflxvideo.net
ipset=/netflix.com/netflix.net/nflxext.com/nflximg.com/nflxso.net/nflxvideo.net/NETFLIX
ipset=/netflix.com/netflix.net/NETFLIX

EDIT2: I think I have solved it. I have first deleted the ipset NETFLIX and made sure all entries were removed both from NAT-START and from dnsmasq.conf.add.

Then I have manually removed all the leftover entries from dnsmasq.conf.add

Once empty, I have reexecuted the following command

Code: 
x3mRouting 4 0  NETFLIX  dnsmasq=netflix.com,netflix.net,nflxext.com,nflximg.net,nflxso.net,nflxvideo.net

and now my dnsmasq.conf.add looks as follows:

Code: 
ipset=/netflix.com/netflix.net/nflxext.com/nflximg.net/nflxso.net/nflxvideo.net/NETFLIX

The number of entries in the NETFLIX IPSET is slowly increasing and I no longer get the proxy error.

Now I am only left with 2 questions:

1) Can I use both the dnsmasq file and the asnum or will the last one executed overwrite the previous one?
2) Are there any other folders where I may need to do some cleansing of leftover files and/or file entries?

Thanks!
 
Last edited:
CannaLucente said:
Hi Xentrk,

following your suggestion, I have now switched my VPN clients so I no longer have priority issues. Now all my traffic goes via VPN client 4.
Click to expand...
You need to enable Policy Rules to the x3mRouting to work properly. The only time I turn it off is when I am doing some analysis and mining dnsmasq.log. For example, HBOMAX recently offered a 6 month special that I purchased. Since they block known VPN servers, I had to analyze the domains used. To do that,
I disable Policy Rules on VPN Client 1 and changed it so that ALL traffic goes thru VPN Client 1. VPN Client 1 is my private IP in US that is not blocked. I then went to HBO website and the app on my streaming device and selected the various options to generate and log the domains being queried in dnsmasq.log. Once this was done, I ran the autoscan.sh script available in Option 4 to list the domains being used for dnsmasq method:

Code: 
sh autoscan.sh scan=hbo,hbomax

IPSET Format
-------------------------------------
hbo.com
hbomax.com
warnermediacdn.com
Please note that I updated the autoscan.sh script to include additional information and plan to publish it later today.

From the above, I created the following rule:
Code: 
x3mRouting ALL 1 HBOMAX dnsmasq=hbo.com,hbomax.com,warnermediacdn.com

I then turned turned Policy Rules back on. As I continue to watch HBOMAX, more IPv4 addresses are added to the IPSET list as domains are queried and HBO traffic is routed to VPN Client 1
[/QUOTE]

CannaLucente said:
However now I am facing an issue with the Netflix IPSET that always used to work...

I had to regenerate the rules for Netflix and Amazon as they used to be built on client 1 and now I need them on VPN client 4.

I have entered the following command:

x3mRouting 4 0 NETFLIX autoscan=netflix,nflx

and I get the following:

Code: 
(x3mRouting): 12365 Starting Script Execution 4 0 NETFLIX autoscan=netflix,nflx

Done.
(x3mRouting): 12365 IPSET created: NETFLIX hash:net family inet hashsize 1024 maxelem 65536
(x3mRouting): 12365 CRON schedule created: #NETFLIX# '0 2 * * * ipset save NETFLIX'
(x3mRouting): 12365 Selective Routing Rule via WAN created for NETFLIX fwmark 0x8000/0x8000
(x3mRouting): 12365 iptables -t mangle -D PREROUTING -i br0 -m set --match-set NETFLIX dst -j MARK --set-mark 0x8000/0x8000 2>/dev/null added to /jffs/scripts/x3mRouting/vpnclient4-route-up
(x3mRouting): 12365 iptables -t mangle -A PREROUTING -i br0 -m set --match-set NETFLIX dst -j MARK --set-mark 0x8000/0x8000 added to /jffs/scripts/x3mRouting/vpnclient4-route-up
(x3mRouting): 12365 iptables -t mangle -D PREROUTING -i br0 -m set --match-set NETFLIX dst -j MARK --set-mark 0x8000/0x8000 2>/dev/null added to /jffs/scripts/x3mRouting/vpnclient4-route-pre-down
(x3mRouting): 12365 sh /jffs/scripts/x3mRouting/x3mRouting.sh 4 0 NETFLIX dnsmasq=netflix.com,netflix.net,nflxext.com,nflximg.com,nflxso.net,nflxvideo.net added to /jffs/scripts/nat-start
(x3mRouting): 12365 Completed Script Execution

but when I do liststats, I see no entries created in the NETFLIX set:

Code: 
xxx@RT-AX88U-8880:/tmp/home/root# liststats
AMAZON - 2446
NETFLIX - 0

I have tried using your autoscan script and the domains are extracted:

Code: 
xxx@RT-AX88U-8880:/jffs/scripts/x3mRouting# sh autoscan.sh autoscan=netflix,nflx

netflix.com
netflix.net
nflxext.com
nflximg.com
nflxso.net
nflxvideo.net

What am I doing wrong, why no entries are inserted in the NETFLIX IPSET?

Thank you!
Click to expand...
I suspect it was a timing thing. Did you watch Netflix before running liststats? That step is required for records to get added to the IPSET list. The IPv4 address get added to the IPSET list dynamically. The top level domains are gathered from the query records. The reply records then gets added to the IPSET list as queries

This will display the domains that are added to the IPSET list NETFLIX
Code: 
grep -w "NETFLIX" /opt/var/log/dnsmasq.log | awk '{print $9}' | sort -u

The above is one of the new items I added to to the autoscan.sh script to help with the analysis.
 
Last edited:
CannaLucente said:
Mmm I think it is just a matter of waiting as I now see 6 entries... Do they get added retrospectively?

If so, do I have a way to use both the dnsmasq filter and the ASNUM? Or the latter I execute deletes the entries created by the former?

In other words, can I do

x3mRouting 4 0 NETFLIX autoscan=netflix,nflx

and

x3mRouting 4 0 NETFLIX asnum=AS2906

Or will the second command delete whatever the first command did?

Thanks!
Click to expand...
You don't need to use both the asnum and dnsmasq method for Netflix.

Here is an example of what you should be seeing in dnsmasq.log file. Note the first record is a query record that gets forwarded. The reply record then gets added to dnsmasq.
Code: 
Jan 13 03:35:20 dnsmasq[4534]: query[A] occ-0-3996-3997.1.nflxso.net from 192.168.22.165
Jan 13 03:35:20 dnsmasq[4534]: forwarded occ-0-3996-3997.1.nflxso.net to 1.0.0.1
Jan 13 03:35:20 dnsmasq[4534]: forwarded occ-0-3996-3997.1.nflxso.net to 1.1.1.1
Jan 13 03:35:20 dnsmasq[4534]: ipset add NETFLIX-DNS 45.57.100.138 occ-0-3996-3997.1.nflxso.net
Jan 13 03:35:20 dnsmasq[4534]: reply occ-0-3996-3997.1.nflxso.net is 45.57.100.138
Jan 13 03:35:20 dnsmasq[4534]: ipset add NETFLIX-DNS 45.57.100.142 occ-0-3996-3997.1.nflxso.net
Jan 13 03:35:20 dnsmasq[4534]: reply occ-0-3996-3997.1.nflxso.net is 45.57.100.142

You can use the follow the log file in diversion function or use the "tail -f /opt/var/log/dnsmasq.log" command to view the log in real time while you watch Netflix to make sure the entries are added to the IPSET list.
 
Last edited:
CannaLucente said:
Mmm not sure what I said above is correct. I have added the ASNUM and I got 148 entries. But when I use the autoscan or the dnsmasq file the number of entries doesn't increase.

I have tried running nat-start and I see it executes a lot of commands, including those I'd expect to be deleted (i.e. with the old VPN client setup).

So I edited nat-start and now I am only left with the following entries:

Code: 
#!/bin/sh
sh /jffs/scripts/x3mRouting/x3mRouting.sh 4 0 AMAZON aws_region=US,CA,AP,CN,EU,SA,GLOBAL
sh /jffs/scripts/x3mRouting/x3mRouting.sh 4 0 NETFLIX dnsmasq=netflix.com,netflix.net,nflxext.com,nflximg.com,nflxso.net,nflxvideo.net

I have restarted the router and, when doing liststats, I see 2 entries in my ipset NETFLIX.

It looks as if it is not automatically adding the entries from dnsmasq. I have tried following the dnsmasq log in Diversion and I can see many netflix related IPs however my Netflix set doesn't get extended.

I am out of ideas... what can I do?

EDIT: I have checked the content of dnsmasq.conf.add and doesn't seem right... but I am not sure how to fix it...

Code: 
ipset=/netflix.com/nflxext.com/nflximg.net/nflxso.net/nflxvideo.net/NETFLIX      dnsmasq=netflix.com,nflxext.com,nflximg.net,nflxso.net,nflxvideo.net
ipset=/netflix.com/nflxext.com/nflxso.net/nflxvideo.net/NETFLIX      autoscan=netflix.com,nflxext.com,nflximg.net,nflxso.net,nflxvideo.net
ipset=/netflix.com/netflix.net/nflxext.com/nflximg.com/nflxso.net/nflxvideo.net/NETFLIX
ipset=/netflix.com/netflix.net/NETFLIX

EDIT2: I think I have solved it. I have first deleted the ipset NETFLIX and made sure all entries were removed both from NAT-START and from dnsmasq.conf.add.

Then I have manually removed all the leftover entries from dnsmasq.conf.add

Once empty, I have reexecuted the following command

Code: 
x3mRouting 4 0  NETFLIX  dnsmasq=netflix.com,netflix.net,nflxext.com,nflximg.net,nflxso.net,nflxvideo.net

and now my dnsmasq.conf.add looks as follows:

Code: 
ipset=/netflix.com/netflix.net/nflxext.com/nflximg.net/nflxso.net/nflxvideo.net/NETFLIX

The number of entries in the NETFLIX IPSET is slowly increasing and I no longer get the proxy error.

Now I am only left with 2 questions:

1) Can I use both the dnsmasq file and the asnum or will the last one executed overwrite the previous one?
2) Are there any other folders where I may need to do some cleansing of leftover files and/or file entries?

Thanks!
Click to expand...
You should use separate IPSET name for each method. Pick either the dnsmasq method or ASN method for Netflix. Shouldn't need both.

For Amazon, I now use only the GLOBAL region and all is good. You shouldn't need to specify local regions if one uses GLOBAL from the testing I've done with it for the past several months. The first one will remove the current code you have. The second line will create the updated list for AMAZON.
Code: 
x3mRouting.sh ipset_name=AMAZON del
x3mRouting.sh 4 0 AMAZON aws_region=GLOBAL
 
Xentrk said:
That message will appear if the iptables rule still exists for the IPSET and an attempt is made to remove it using the 'ipset destroy'. In this example, I create the IPSET list "TEMP" followed by the 'ipset destroy' command

Code: 
# x3mRouting ALL 1 TEMP ip=8.8.8.8
(x3mRouting): 6745 Starting Script Execution ALL 1 TEMP ip=8.8.8.8
Successfully added 8.8.8.8
(x3mRouting): 6745 IPSET created: TEMP
(x3mRouting): 6745 Selective Routing Rule via VPN Client 1 created for TEMP fwmark 0x1000/0x1000
(x3mRouting): 6745 iptables -t mangle -D PREROUTING -i br0 -m set --match-set TEMP dst -j MARK --set-mark 0x1000/0x1000 2>/dev/null added to /jffs/scripts/x3mRouting/vpnclient1-route-up
(x3mRouting): 6745 iptables -t mangle -A PREROUTING -i br0 -m set --match-set TEMP dst -j MARK --set-mark 0x1000/0x1000 added to /jffs/scripts/x3mRouting/vpnclient1-route-up
(x3mRouting): 6745 iptables -t mangle -D PREROUTING -i br0 -m set --match-set TEMP dst -j MARK --set-mark 0x1000/0x1000 2>/dev/null added to /jffs/scripts/x3mRouting/vpnclient1-route-pre-down
(x3mRouting): 6745 sh /jffs/scripts/x3mRouting/x3mRouting.sh ALL 1 TEMP added to /jffs/scripts/nat-start
(x3mRouting): 6745 Completed Script Execution

# ipset destroy TEMP
ipset v7.6: Set cannot be destroyed: it is in use by a kernel component
For some reason, the iptables rule may not have gotten removed. The following message should have gotten displayed.

Code: 
Deleted PREROUTING Chain $CHAIN_NUM for IPSET List $IPSET_NAME
x3mRouting first purges the ipset reference from the up/down and other files as appropriate.

x3mRouting then performs a query of the iptables PREROUTING chain for the iptables rule for the specified IPSET list before issuing the command to remove the rule. I wonder if one of the grep conditions are not being met? If you want to test the code on the command line, just copy/paste the code below to SSH prompt and substitute IPSET_NAME for a valid ipset.

Please create a test ipset list like I did above, then run this command to validate the iptables chain got created:
Code: 
iptables -nvL PREROUTING -t mangle --line

Then, run this code to see if it returns the chain number and ipset name. Substitue "IPSET_NAME " to be the name of the ipset you created.
Code: 
iptables -nvL PREROUTING -t mangle --line | grep "br0" | grep "IPSET_NAME " | grep "match-set" | awk '{print $1, $12}' | sort -nr

One thing I just noted is to remove the space after the "IPSET_NAME " and use the "-w" grep option to match on the word which is a better coding technique. So test this code as well, remember to change the name of "IPSET_NAME"

Code: 
iptables -nvL PREROUTING -t mangle --line | grep "br0" | grep -w "IPSET_NAME" | grep "match-set" | awk '{print $1, $12}' | sort -nr
Click to expand...

"ipset destroy TEMP" issued the "ipset v7.6: Set cannot be destroyed: it is in use by a kernel component" message

all the other commands you mentioned responded with correct results
 
Hi Xentrk,

first let me thank you as usual for your replies.

I have condensed all my answers in a single post.

Policy Rules (strict) has always been active for me. My first question is how do you analyse dnsmasq.log? I know how to follow it from Diversion (or use the tail command you gave) but is there a way to analyse it retrospectively? I see you use a grep command below, is there some sort of editor I can use? Or is the grep the best way to query dnsmasq.log?

I fully understand what you were doing for HBO and this is the same I was trying for Netflix but, for some reason, it wasn't adding the entries to the IPSET (I guess this was due to the funny looking entries in the dnsmasq config file as, once cleansed, it started working again).

Xentrk said:
Did you watch Netflix before running liststats? That step is required for records to get added to the IPSET list. The IPv4 address get added to the IPSET list dynamically. The top level domains are gathered from the query records. The reply records then gets added to the IPSET list as queries
Click to expand...

I watched Netflix before and after. I realised the entries were being added when watching it but, for some reason, the counter was stuck at 6 entries, until I did all the cleansing. I will monitor if this happens again. I am very new to Unix scripts etc so it may well be that I was doing something wrong. And I understand why I don't need both the dnsmasq and the ASNUM.

Xentrk said:
This will display the domains that are added to the IPSET list NETFLIX
Code: 
grep -w "NETFLIX" /opt/var/log/dnsmasq.log | awk '{print $9}' | sort -u
The above is one of the new items I added to to the autoscan.sh script to help with the analysis.
Click to expand...

The above is very useful, thanks! I assume I will get the new autoscan once you release it in GitHub.

All the rest (Amazon and Disney+) are working well so I am not planning to change anything there.

Thanks again!
 
You must log in or register to reply here.

Similar threads

Y
how to create a China IPset and implement traffic diversion based on IPset
Replies
0
Views
415
ysl11
Y
W
Looking for an add-on that does...
Replies
5
Views
2K
JGrana
JGrana
R
Help with RT-ac88u network switch issues - causing reboot
Replies
0
Views
172
rkalinka
R
L
ASUS RT-AX58U - devices kicked-off wifi over 60 times a day
2
Replies
21
Views
682
ColinTaylor
C
W
Skynet SOLVED: Scrollbar Disappears on Skynet Tab on Asuswrt-Merlin 386.14
Replies
10
Views
1K
bennor
B
Similar threads
Thread starter Title Forum Replies Date
C How do I get x3mrouting working with AdGuard Home ? Asuswrt-Merlin AddOns 6
A x3mRouting x3mRouting vs Domain based VPN routing Asuswrt-Merlin AddOns 29
R DomainVPNRouting Domain VPN Routing v3.0.3 ***Release*** Asuswrt-Merlin AddOns 45
C Question about DomainBased Routing & Re-connection Asuswrt-Merlin AddOns 0
K DomainVPNRouting Domain VPN Routing help Asuswrt-Merlin AddOns 4
A DomainVPNRouting Domain VPN Routing Question Asuswrt-Merlin AddOns 32
johndoe85 DomainVPNRouting Domain-based VPN routing script Asuswrt-Merlin AddOns 2
R DomainVPNRouting Domain VPN Routing Asuswrt-Merlin AddOns 96

Similar threads

Latest threads

Support SNBForums w/ Amazon

If you'd like to support SNBForums, just use this link and buy anything on Amazon. Thanks!

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!

Members online

Total: 946 (members: 23, guests: 923)
Top