x3mRouting x3mRouting ~ Selective Routing for Asuswrt-Merlin Firmware

[Xentrk]

I'm not sure why the IP address list downloaded via ASN number is always 2KB and not complete. I'm trying to exclude all Akamai CDN servers and the list downloaded from ipinfo never seem to be a complete list.

What is the AS number so I can test and compare to other sources?

https://bgp.he.net/search?search%5Bsearch%5D=Akamai&commit=Search

 

[OKLY]

What is the AS number so I can test and compare to other sources?

https://bgp.he.net/search?search%5Bsearch%5D=Akamai&commit=Search

AS16625

 

[Kingp1n]

I ended up dropping the US region a month or so ago when I was having an issue and went GLOBAL. It's been working fine since then. Let me know your experience. I'll change the references on the readme to use GLOBAL going forward once confirmed.

Sorry for the late reply. So I was using GLOBAL and some videos would play thru Netflix and some would not (#ALIVE). I would keep getting the VPN error. I switched back to US but I'm still gettting the VPN errors only for some videos i.e. movie #ALIVE. The reason I switched was due to Disney+ not working again. Once I switched to amazon_US, for now everything works (Hulu/Disney/Prime etc) with the exception of some videos inside Netflix....any ideas I can try?

These are the lines I'm using with option 3 of the script:

sh /jffs/scripts/x3mRouting/x3mRouting.sh 1 0 NETFLIX asnum=AS2906
sh /jffs/scripts/x3mRouting/x3mRouting.sh 1 0 NETFLIX_DNS dnsmasq=netflix.com,nflxext.com,nflximg.com,nflximg.net,nflxso.net,nflxvideo.net,amazonaws.com

UPDATE: Not sure what happened but it "magically" started working again! haha sorry for the trouble!!

 

[Xentrk]

AS16625

Thank you. I backed out the last code updates to the Download_ASN_Ipset_List function I made and it's working like it should now. Appears the CIDR notations were not getting loaded.

Use option 5 from the x3mRouting Menu to get the updated code.

Edit:

Out of curiosity, I used the backup source https://api.hackertarget.com/aslookup/?q=AS16625 and get 3135 entries compared to the 100 entries gathered from ipinfo.io. I may have to do with CIDR notation. I will have to investigate further.

 

[OKLY]

Thank you. I backed out the last code updates to the Download_ASN_Ipset_List function I made and it's working like it should now. Appears the CIDR notations were not getting loaded.

Use option 5 from the x3mRouting Menu to get the updated code.

Edit:

Out of curiosity, I used the backup source https://api.hackertarget.com/aslookup/?q=AS16625 and get 3135 entries compared to the 100 entries gathered from ipinfo.io. I may have to do with CIDR notation. I will have to investigate further.

So it still doesn't work after updating the code right? I used option 5 to update the code and it is still only retrieving 100 lines.

 

[Xentrk]

So it still doesn't work after updating the code right? I used option 5 to update the code and it is still only retrieving 100 lines.

I can duplicate. But I do get the CIDR format now after the update. I'll make it a priority. But it looks like I may be temporarily banned from ipinfo.io due to my many downloads today. It will take a few hours to get unblocked. I may change to another source.

Does this return a count of 100?

curl -fsL --retry 3 --connect-timeout 3 "https://ipinfo.io/AS16625" | grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}/[0-9]{1,2}
' | sort -gt '/' -k 1 | sort -ut '.' -k 1,1n -k 2,2n -k 3,3n -k 4,4n | grep -c '.'

This source returns 3103

curl -fsL --retry 3 --connect-timeout 3 "https://api.hackertarget.com/aslookup/?q=AS16625" |  sort -gt '/' -k 1 | sort
 -ut '.' -k 1,1n -k 2,2n -k 3,3n -k 4,4n | grep -c "."
3103

 

[Xentrk]

@OKLY I pushed a hotfix to bypass ipinfo.io as the source. It will use https://hackertarget.com/as-ip-lookup/ as the source instead. I will continue to analyze the source sites for the one that appears to be the best authority. https://geekflare.com/asn-tools/. ipinfo.io has been the source for this project and others. It may have to do that ipinfo.io changed the method of storage to json format and the code needs to be updated to process it. The hackertarget source is in a text file and requires no further processing other than removing the header record and performing a sort.

 

[OKLY]

I can duplicate. But I do get the CIDR format now after the update. I'll make it a priority. But it looks like I may be temporarily banned from ipinfo.io due to my many downloads today. It will take a few hours to get unblocked. I may change to another source.

Does this return a count of 100?

curl -fsL --retry 3 --connect-timeout 3 "https://ipinfo.io/AS16625" | grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}/[0-9]{1,2}
' | sort -gt '/' -k 1 | sort -ut '.' -k 1,1n -k 2,2n -k 3,3n -k 4,4n | grep -c '.'

This source returns 3103

curl -fsL --retry 3 --connect-timeout 3 "https://api.hackertarget.com/aslookup/?q=AS16625" |  sort -gt '/' -k 1 | sort
-ut '.' -k 1,1n -k 2,2n -k 3,3n -k 4,4n | grep -c "."
3103

Yes, if ipinfo it returns result of 100.

 

[Xentrk]

Looks like ipinfo is moving towards a paid model to obtain the IP addresses and I will have to drop them as a source. (note to @Adamm)

ISP lookup for IP addresses is available to IPinfo customers on the Standard and Pro plans. Our ASN API endpoint returns a JSON response with data for both IPv4 and IPv6 prefixes, including netblocks, IDs, names, and countries for networks and internet service providers (ISPs).

ASN Lookup API - IPInfo.io

Our ASN API provides both IP to ASN or just ASN details, including every ASN's related domains, allocation date, registry name, number of IPs, and assigned prefixes.

ipinfo.io

Flexible pricing for the most accurate IP data

Flexible, customized, and affordable IP data powers today's applications. Choose from precise IP insights that scale from startups to enterprises.

ipinfo.io

I got matches when I performed a random sample of ip addresses in the api.hackertarget.com site with Hurricane Electric. I'll still do some comparison with the other sites.

 

[Adamm]

Looks like ipinfo is moving towards a paid model to obtain the IP addresses and I will have to drop them as a source. (note to @Adamm)

Another ip lookup service being crippled in the name of greed, fantastic

fwiw keep in mind hackertarget has a daily lookup limit of 50 ASN's.

 

[Xentrk]

Another ip lookup service being crippled in the name of greed, fantastic

fwiw keep in mind hackertarget has a daily lookup limit of 50 ASN's.

Yep, I got blocked last night when testing changes. I found the whob command can get the prefixes assigned to ASN. I wrote a small script to test and it works.

#!/bin/sh
set -x
VAR=2906
whob -a $VAR | awk '{print $2}' |  grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}/[0-9]{1,2}'

I added the -V option to whob for debugging...

sh test15.sh
+ VAR=2906
+ whob -aVVV 2906
+ awk {print $2}
+ grep -oE ([0-9]{1,3}\.){3}[0-9]{1,3}/[0-9]{1,2}
WhoB version 3.91 firing up... (verbosity level 3)
Data sources: whois.pwhois.org (pWhoIs).
Using user-supplied ASN 2906 for lookup.
23.246.0.0/18
23.246.2.0/24
23.246.3.0/24

<snip>

But when I place the code in x3mRouting.sh, it does not work. I added /opt/bin/ path too. Here is a trace

+ /opt/bin/whob -aVVV 2906
+ grep -oE ([0-9]{1,3}\.){3}[0-9]{1,3}/[0-9]{1,2}
+ awk {print $2}
WhoB version 3.91 firing up... (verbosity level 3)
Data sources: whois.pwhois.org (pWhoIs).
Using Prefix WhoIs for bulk file resolution.
End of file reached after reading 0 lines.
-exit

Ugh. Not sure why it returns the prefixes in one script but not the other using the same code. Nothing gets returned. I see the awk is done after the grep in the x3mRouting.sh code. Yet, it's the same code. Any ideas? @Jack Yaz @thelonelycoder

 

[raven-au]

But when I place the code in x3mRouting.sh, it does not work. I added /opt/bin/ path too. Here is a trace

+ /opt/bin/whob -aVVV 2906
+ grep -oE ([0-9]{1,3}\.){3}[0-9]{1,3}/[0-9]{1,2}
+ awk {print $2}
WhoB version 3.91 firing up... (verbosity level 3)
Data sources: whois.pwhois.org (pWhoIs).
Using Prefix WhoIs for bulk file resolution.
End of file reached after reading 0 lines.
-exit

Ugh. Not sure why it returns the prefixes in one script but not the other using the same code. Nothing gets returned. I see the awk is done after the grep in the x3mRouting.sh code. Yet, it's the same code. Any ideas? @Jack Yaz @thelonelycoder

It's gotta be some sort of change to escaping of the regex between a the test script the the other ...
Maybe you have set a shell option in the target script that changes behaviour or try sticking the whole regex in () or ...

 

[mister]

I also noticed something strange while using it:
As already written before, I am trying to route certain TV media libraries via specific VPNs. Now I have found that on my Windows 10 computer with Firefox I have full access to the media libraries and it is running as expected. But not with my Android smartphone Samsung S10 - I get an error message that the content is not available in my country - so the phone is routed differently.
Now I have run the getdomainnames.sh script for the IPs of the two devices. While the content was collected on the computer and I received a list with different domains, the list remained empty on the Android.

Is there a possibility why that could be?
What could i do? Thank you in advance for the ideas.

 

[Xentrk]

I also noticed something strange while using it:
As already written before, I am trying to route certain TV media libraries via specific VPNs. Now I have found that on my Windows 10 computer with Firefox I have full access to the media libraries and it is running as expected. But not with my Android smartphone Samsung S10 - I get an error message that the content is not available in my country - so the phone is routed differently.
Now I have run the getdomainnames.sh script for the IPs of the two devices. While the content was collected on the computer and I received a list with different domains, the list remained empty on the Android.

Is there a possibility why that could be?
What could i do? Thank you in advance for the ideas.

It may be using the cellular location services on the phone giving away your true location. I have a music app that works on my phone when it is connected to my home network that is connected to a VPN server. When I use cellular data though, it won't work, even if I am connected to the VPN using the app of the provider. I plan on ttrying one of the geo spoofing apps available on the Google Play store.

 

[mister]

Sorry , my mistake. I had at one VPN "force all traffic over VPN" activated instead of using policies. :-( .

Sorry!

Sorry again, the problem occurred again and now all routing clients are on "routing policies".

Instantly after reboot, I am able to establish a VPN connection to my server - a few minutes later it isn´t possibe anymore.

If you take a look into the VPN Server WebUI the username is "UNDEF" , and it doesn´t go forward.....

Any ideas ? How to bypass VPN Servers to WAN ?

 

[mister]

It may be using the cellular location services on the phone giving away your true location. I have a music app that works on my phone when it is connected to my home network that is connected to a VPN server. When I use cellular data though, it won't work, even if I am connected to the VPN using the app of the provider. I plan on ttrying one of the geo spoofing apps available on the Google Play store.

Thanks a lot, but that can´t be the reason, because I tried to access via browser as well. And that doesn´t work also.
And why getdomainnames.sh script doesn´t collect the domain names......?

 

[Xentrk]

It's gotta be some sort of change to escaping of the regex between a the test script the the other ...
Maybe you have set a shell option in the target script that changes behaviour or try sticking the whole regex in () or ...

I moved the code to another location and got it work. I tried this before without success though. I wonder if the $2 in the "awk '{print $2}'" code is inheriting the value passed to the function. This now gives me something to go forward on.

So far, I've determined the code does not return ip addresses when called from within a for loop. I have so see if I can code around the issue.

EDIT: I did some code revisions and got it working now. Doing final testing.

Using whob as the source would be the ideal solution due to it's up-to-date information.

https://pwhois.org/

The Prefix WhoIs Project

The Prefix WhoIs Project provides a whois-compatible client and server framework for disclosing various up-to-date routing information. Instead of using registrar-originated network information (which is often unspecific or inaccurate), Prefix WhoIs uses the Internet's global routing table as gleaned from a number of routing peers around the world. Other sources of information, such as imported data from ARIN are also supported (a separate agreement with ARIN is required).

 

[Xentrk]

Thanks a lot, but that can´t be the reason, because I tried to access via browser as well. And that doesn´t work also.
And why getdomainnames.sh script doesn´t collect the domain names......?

Device isn't using router internal dns e.g. dnsmasq? See if the device has query records in dnsmasq.log file by searching by the device IP addresses assigned via dhcp server:
grep <deviceip> /opt/var/log/dnsmasq.log

 

[mister]

Yes, maybe that could be the reason, because I am using quad9 as the "private dns entry" in android. I will test it this evening.

I thought, that the DNS setting of the device is only responsable for the translation "domainname to IP" . And your script compares the IP collected by the dnsmasq commands in the router with the IP request of the device and is routing then the IPs according to the rules. Now it seems to me , that I didn´t understood, how your script is working.
So every device, which is not using the DNS of the router (and has its own DNS entries) is not able to be routed according to the x3mrouting policies, right ?

Sorry for all the questions and my bad background.

Currently I am trying to get the OpenVPN Server working again .

Thanks a lot again.

 

[Xentrk]

Yes, maybe that could be the reason, because I am using quad9 as the "private dns entry" in android. I will test it this evening.

I thought, that the DNS setting of the device is only responsable for the translation "domainname to IP" . And your script compares the IP collected by the dnsmasq commands in the router with the IP request of the device and is routing then the IPs according to the rules. Now it seems to me , that I didn´t understood, how your script is working.
So every device, which is not using the DNS of the router (and has its own DNS entries) is not able to be routed according to the x3mrouting policies, right ?

Sorry for all the questions and my bad background.

Currently I am trying to get the OpenVPN Server working again .

Thanks a lot again.

dnsmasq has to be enabled on the router and the device logging to dnsmasq.log for the script to work and show the domain names.

Some devices have hard coded DNS. DNS Filter will force the device to use the DNS specified on the rotuer. Do you have DNS Filter turned on under the LAN tab? This should force the phone to use dnsmasq for lookups.