A Guide About Installing ZeroTier on ASUS AC68U Router

[MissingTwins]

Client side
After connecting to zerotier app
install and connect shadowsocks client app with below configuration
Configuration
ServerIP https://my.zerotier.com/manageIP for Asus router
Port 8388
password:"YOURPASSWORD"
encryption:"chacha20-ietf-poly1305"

Please advice!

What client were you using?
On the client-side with shadowsocks client, zerotier is not needed anymore, all zerotier traffic will go through your ss-server.

To me for Quantumult on iOS.
1. SSH to your router, make sure your ss-server is running by
$ /opt/etc/init.d/S22shadowsocks-server status
# or you can run this for debugging, you should see activities while you're connecting from the client.
$ ss-server -s `nvram get lan_ipaddr` -p 8388 -k YOURPASSWORD -m chacha20-ietf-poly1305 -u --reuse-port --no-delay -v

2. Add a server on your iOS device
Protocol Sadowsocks
Address: YourDDNS.domain.org # Or 192.168.1.1 for the test inside your LAN.
Port: 8388
Password: "YOURPASSWORD"
encryption:"chacha20-ietf-poly1305"

3. At the filter configuration,
a. Add an IP-CIDR redirect your zerotier to PROXY
Type IP-CIDR
Action PROXY
Match 10.9.8.0/24 # your zerotier ip address
b. Add another IP-CIDR redirect your local network traffic to PROXY
Match 192.168.1.1/24 # your local network behind the router.

4. Clear the filters.
a. Remove all rules except 127.0.0.1/8, 172.16.0.0/12,
b. Change FINAL to PROXY

5. Turn on the switch on the Home tab. You should see VPN sign on the top of your phone

I tried above script and when I'm on my local network I can connect to the shadowsocks server on my router. When I'm outside my local network I cannot connect.

You may need to try this. If your wan is PPPoE, or change ppp0 to eth0 if your wan is automatic/static IP, and make sure your wan IP is accessible from the Internet.
$ iptables -I INPUT -i ppp0 -p tcp --destination-port 8388 -j ACCEPT
$ iptables -I INPUT -i ppp0 -p udp --destination-port 8388 -j ACCEPT

You may get your self a DDNS, and set it up in Router WebUI => WAN => DDNS
On your client use your DDNS replace your server IP address, you will be able to access from both behind and outside of your router.

 

[Pradrone]

Thanks for inputs, will give a try.
I) Can u please let me know below steps will works if I am behind a Double NAT / CGNAT?

What client were you using?
II) clients
a) macOS -https://github.com/shadowsocks/ShadowsocksX-NG/releases
b) AndroidOS - https://play.google.com/store/apps/details?id=com.github.shadowsocks

On the client-side with shadowsocks client, zerotier is not needed anymore, all zerotier traffic will go through your ss-server.

To me for Quantumult on iOS.
1. SSH to your router, make sure your ss-server is running by
$ /opt/etc/init.d/S22shadowsocks-server status
# or you can run this for debugging, you should see activities while you're connecting from the client.
$ ss-server -s `nvram get lan_ipaddr` -p 8388 -k YOURPASSWORD -m chacha20-ietf-poly1305 -u --reuse-port --no-delay -v

2. Add a server on your iOS device
Protocol Sadowsocks
Address: YourDDNS.domain.org # Or 192.168.1.1 for the test inside your LAN.
Port: 8388
Password: "YOURPASSWORD"
encryption:"chacha20-ietf-poly1305"

III) Sure, will check ddns here, since I am using no-ip (https://www.noip.com/) ddns setting but not able to open router home page using browser but with asus app it is working fine.

3. At the filter configuration,
a. Add an IP-CIDR redirect your zerotier to PROXY
Type IP-CIDR
Action PROXY
Match 10.9.8.0/24 # your zerotier ip address
b. Add another IP-CIDR redirect your local network traffic to PROXY
Match 192.168.1.1/24 # your local network behind the router.

4. Clear the filters.
a. Remove all rules except 127.0.0.1/8, 172.16.0.0/12,
b. Change FINAL to PROXY

IV) Not very much clear about the 3 and 4 steps, since I don't see any filter options in android/mac app.

5. Turn on the switch on the Home tab. You should see VPN sign on the top of your phone



You may need to try this. If your wan is PPPoE, or change ppp0 to eth0 if your wan is automatic/static IP, and make sure your wan IP is accessible from the Internet.
$ iptables -I INPUT -i ppp0 -p tcp --destination-port 8388 -j ACCEPT
$ iptables -I INPUT -i ppp0 -p udp --destination-port 8388 -j ACCEPT

V) My wan is PPPoE, automatic ip behind a Double NAT / CGNAT

You may get your self a DDNS, and set it up in Router WebUI => WAN => DDNS
On your client use your DDNS replace your server IP address, you will be able to access from both behind and outside of your router.[/QUOTE]

 

[MissingTwins]

I) Can u please let me know below steps will works if I am behind a Double NAT / CGNAT?
V) My wan is PPPoE, automatic ip behind a Double NAT / CGNAT

It will work behind double NAT, BUT will not work if your router is behind cell network.

Shadowssocks Plan
------------------------------------------------------------------
1. For example, this is my settings, it works perfectly, my test router is ac68u and it's behind ac86u.
(192.168.7.0/24)ac68u->(192.168.9.2)eth0->(192.168.9.0/24)ac86u->pppoe->Internet->cell-tower->[pocket wi-fi(192.168.179.0/24)->window 10->shadowsocks-windows-client]/[or iphone with Quantumult]
a). ac68u (Firewall is turned off)
$ ss-server -s `nvram get lan_ipaddr` -p 8388 -k YOURPASSWORD -m chacha20-ietf-poly1305 -u --reuse-port --no-delay -v
$ iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 8388 -j DNAT --to-destination `nvram get lan_ipaddr`
$ iptables -t nat -A PREROUTING -i eth0 -p udp --dport 8388 -j DNAT --to-destination `nvram get lan_ipaddr`
In case you're behind a firewall, those two are nessary, change to eth0/ppp0 accordingly
$ iptables -I INPUT -i eth0 -p tcp --destination-port 8388 -j ACCEPT
$ iptables -I INPUT -i eth0 -p udp --destination-port 8388 -j ACCEPT
b). ac86u(Firewall is turned on)
WAN - Virtual Server / Port Forwarding
udp/tcp 8388->192.168.9.2

2. In my Quantumult
a). Setup
sever mydomain.com
port 8388
password YOURPASSWORD
method chacha20-ietf-poly1305

b). Server Status
Check on both Tcp Handshake and Latency.

3. Logs on ac68u
```
root@RT-AC68U:/tmp/home/root# ss-server -s `nvram get lan_ipaddr` -p 8388 -k YOURPASSWORD -m chacha20-ietf-poly1305 -u --reuse-port --no-delay -v
2019-12-29 08:35:58 INFO: enable TCP no-delay
2019-12-29 08:35:58 INFO: UDP relay enabled
2019-12-29 08:35:58 INFO: enable TCP no-delay
2019-12-29 08:35:58 INFO: initializing ciphers... chacha20-ietf-poly1305
2019-12-29 08:35:58 INFO: tcp server listening at 192.168.7.1:8388
2019-12-29 08:35:58 INFO: udp server listening at 192.168.7.1:8388
2019-12-29 08:35:58 INFO: running from root user
2019-12-29 08:41:19 INFO: new connection from client, 1 opened client connections
2019-12-29 08:41:19 INFO: [8388] connect to www.gstatic.com:80
2019-12-29 08:41:19 INFO: found address name v6 address www.gstatic.com
2019-12-29 08:41:19 INFO: found address name v4 address www.gstatic.com
2019-12-29 08:41:19 INFO: successfully resolved www.gstatic.com
2019-12-29 08:41:19 INFO: new connection to remote, 1 opened remote connections
2019-12-29 08:41:19 INFO: close a connection to remote, 0 opened remote connections
2019-12-29 08:41:19 INFO: close a connection from client, 0 opened client connections
2019-12-29 08:54:33 INFO: new connection from client, 1 opened client connections
2019-12-29 08:54:33 INFO: [8388] connect to www.gstatic.com:80
2019-12-29 08:54:33 INFO: found address name v4 address www.gstatic.com
2019-12-29 08:54:33 INFO: found address name v6 address www.gstatic.com
2019-12-29 08:54:33 INFO: successfully resolved www.gstatic.com
2019-12-29 08:54:33 INFO: new connection to remote, 1 opened remote connections
2019-12-29 08:54:33 INFO: close a connection to remote, 0 opened remote connections
2019-12-29 08:54:33 INFO: close a connection from client, 0 opened client connections

```

III) Sure, will check ddns here, since I am using no-ip (https://www.noip.com/) ddns setting but not able to open router home page using browser but with asus app it is working fine.

It's fine, no-ip will work.

II) clients

IV) Not very much clear about the 3 and 4 steps, since I don't see any filter options in android/mac app.

For Mac and Android, they're all based on socks5, so basically all your browser's traffic will go through the proxy.
So you can safely ignore the 3 and 4 steps.

1. Mac ShadowsocksX-NG
Since you're running ShadowsocksX-NG, configure your browser use socks5 proxy, then you are all set.

2. Shadowsocks-Android
a). In your profile config, turn on Apps VPN mode, select your browser goes through the proxy
Or you can use socks5 mode
a). You can turn on socks5 in settings=>Adanced. Assuming your socks5 proxy port is 1080
b). Type "about:config" in your firefox app address bar
c). Search for "network.proxy"
d). Change those settings
network.proxy.socks=127.0.0.1
network.proxy.socks_port=1080
network.proxy.soccks_remote_dns=true


------------------------------------------------------------
Zerotier Plan

If you're behind a cell network, follow this link, let zerotier handle all your traffics , I haven't done this yet, you'll have to figure it out.

https://zerotier.atlassian.net/wiki...693/Overriding+Default+Route+Full+Tunnel+Mode

 

[Pradrone]

It will work behind double NAT, BUT will not work if your router is behind cell network.

Shadowssocks Plan
------------------------------------------------------------------
1. For example, this is my settings, it works perfectly, my test router is ac68u and it's behind ac86u.
(192.168.7.0/24)ac68u->(192.168.9.2)eth0->(192.168.9.0/24)ac86u->pppoe->Internet
a). ac68u (Firewall is turned off)
$ ss-server -s `nvram get lan_ipaddr` -p 8388 -k YOURPASSWORD -m chacha20-ietf-poly1305 -u --reuse-port --no-delay -v
$ iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 8388 -j DNAT --to-destination `nvram get lan_ipaddr`
$ iptables -t nat -A PREROUTING -i eth0 -p udp --dport 8388 -j DNAT --to-destination `nvram get lan_ipaddr`
b). ac86u(Firewall is turned on)
WAN - Virtual Server / Port Forwarding
udp/tcp 8388->192.168.9.2

2. In my Quantumult
a). Setup
sever mydomain.com
port 8388
password YOURPASSWORD
method chacha20-ietf-poly1305

b). Server Status
Check on both Tcp Handshake and Latency.

3. Logs on ac68u
```
root@RT-AC68U:/tmp/home/root# ss-server -s `nvram get lan_ipaddr` -p 8388 -k YOURPASSWORD -m chacha20-ietf-poly1305 -u --reuse-port --no-delay -v
2019-12-29 08:35:58 INFO: enable TCP no-delay
2019-12-29 08:35:58 INFO: UDP relay enabled
2019-12-29 08:35:58 INFO: enable TCP no-delay
2019-12-29 08:35:58 INFO: initializing ciphers... chacha20-ietf-poly1305
2019-12-29 08:35:58 INFO: tcp server listening at 192.168.7.1:8388
2019-12-29 08:35:58 INFO: udp server listening at 192.168.7.1:8388
2019-12-29 08:35:58 INFO: running from root user
2019-12-29 08:41:19 INFO: new connection from client, 1 opened client connections
2019-12-29 08:41:19 INFO: [8388] connect to www.gstatic.com:80
2019-12-29 08:41:19 INFO: found address name v6 address www.gstatic.com
2019-12-29 08:41:19 INFO: found address name v4 address www.gstatic.com
2019-12-29 08:41:19 INFO: successfully resolved www.gstatic.com
2019-12-29 08:41:19 INFO: new connection to remote, 1 opened remote connections
2019-12-29 08:41:19 INFO: close a connection to remote, 0 opened remote connections
2019-12-29 08:41:19 INFO: close a connection from client, 0 opened client connections
2019-12-29 08:54:33 INFO: new connection from client, 1 opened client connections
2019-12-29 08:54:33 INFO: [8388] connect to www.gstatic.com:80
2019-12-29 08:54:33 INFO: found address name v4 address www.gstatic.com
2019-12-29 08:54:33 INFO: found address name v6 address www.gstatic.com
2019-12-29 08:54:33 INFO: successfully resolved www.gstatic.com
2019-12-29 08:54:33 INFO: new connection to remote, 1 opened remote connections
2019-12-29 08:54:33 INFO: close a connection to remote, 0 opened remote connections
2019-12-29 08:54:33 INFO: close a connection from client, 0 opened client connections

```



It's fine, no-ip will work.




For Mac and Android, they're all based on socks5, so basically all your browser's traffic will go through the proxy.
So you can safely ignore the 3 and 4 steps.

1. Mac ShadowsocksX-NG
Since you're running ShadowsocksX-NG, configure your browser use socks5 proxy, then you are all set.

2. Shadowsocks-Android
a). In your profile config, turn on Apps VPN mode, select your browser goes through the proxy
Or you can use socks5 mode
a). You can turn on socks5 in settings=>Adanced. Assuming your socks5 proxy port is 1080
b). Type "about:config" in your firefox app address bar
c). Search for "network.proxy"
d). Change those settings
network.proxy.socks=127.0.0.1
network.proxy.socks_port=1080
network.proxy.soccks_remote_dns=true


------------------------------------------------------------
Zerotier Plan

If you're behind a cell network, follow this link, let zerotier handle all your traffics , I haven't done this yet, you'll have to figure it out.

https://zerotier.atlassian.net/wiki...693/Overriding+Default+Route+Full+Tunnel+Mode

Thanks a-lot!
Let me give a try for Zerotier Plan as well, since above steps not working for me.

I am trying to build below architecture.
VPN build - Able to access lan and internet (public ip w.x.y.z)from home router after connecting cellphone using shadowsocks using mobile data (public ip a.b.c.d).

1) Cellphone [ShadowSocks -AndroidApp] <---->Internet (public ip a.b.c.d)-------Uses 4G/LTE Mobile tower
~~~Internet (public ip w.x.y.z) <--->(pppoe)-home Router- Asus68u<---->ShadowSocks server running.

2) Macbook[ShadowSocks -MacClient] <->(hotspot)-->Cellphone<---->Internet (public ip a.b.c.d)---Uses 4G/LTE Mobile tower
~~~Internet (public ip w.x.y.z)<--->(pppoe)-->home Router- Asus68u<---->ShadowSocks server running

 

[MissingTwins]

1) Cellphone [ShadowSocks -AndroidApp] <---->Internet (public ip a.b.c.d)-------Uses 4G/LTE Mobile tower
~~~Internet (public ip w.x.y.z) <--->(pppoe)-home Router- Asus68u<---->ShadowSocks server running.

2) Macbook[ShadowSocks -MacClient] <->(hotspot)-->Cellphone<---->Internet (public ip a.b.c.d)---Uses 4G/LTE Mobile tower
~~~Internet (public ip w.x.y.z)<--->(pppoe)-->home Router- Asus68u<---->ShadowSocks server running

I didn't see the problem with your home server side. It should work. As long as your server-side is not behind cell-network(if you can ping your router), ss-server should be reachable.

There is a third plan, run both ss-server and zerotier on a VPS, add an ip route on the VPS which points to your home router.

$ ip route add 192.168.1.1 via 10.9.8.2
# Enable IP forwading
sudo sysctl -w net.ipv4.ip_forward=1
sudo sysctl -w net.ipv4.conf.zt0.route_localnet=1
# Enable BBR
sudo sysctl -w net.core.default_qdisc=fq
sudo sysctl -w net.ipv4.tcp_congestion_control=bbr

The third plan also works for me.

 

[Pradrone]

It will work behind double NAT, BUT will not work if your router is behind cell network.

Shadowssocks Plan
------------------------------------------------------------------
1. For example, this is my settings, it works perfectly, my test router is ac68u and it's behind ac86u.
(192.168.7.0/24)ac68u->(192.168.9.2)eth0->(192.168.9.0/24)ac86u->pppoe->Internet->cell-tower->[pocket wi-fi(192.168.179.0/24)->window 10->shadowsocks-windows-client]/[or iphone with Quantumult]
a). ac68u (Firewall is turned off)
$ ss-server -s `nvram get lan_ipaddr` -p 8388 -k YOURPASSWORD -m chacha20-ietf-poly1305 -u --reuse-port --no-delay -v
$ iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 8388 -j DNAT --to-destination `nvram get lan_ipaddr`
$ iptables -t nat -A PREROUTING -i eth0 -p udp --dport 8388 -j DNAT --to-destination `nvram get lan_ipaddr`
In case you're behind a firewall, those two are nessary, change to eth0/ppp0 accordingly
$ iptables -I INPUT -i eth0 -p tcp --destination-port 8388 -j ACCEPT
$ iptables -I INPUT -i eth0 -p udp --destination-port 8388 -j ACCEPT
b). ac86u(Firewall is turned on)
WAN - Virtual Server / Port Forwarding
udp/tcp 8388->192.168.9.2

2. In my Quantumult
a). Setup
sever mydomain.com
port 8388
password YOURPASSWORD
method chacha20-ietf-poly1305

b). Server Status
Check on both Tcp Handshake and Latency.

3. Logs on ac68u
```
root@RT-AC68U:/tmp/home/root# ss-server -s `nvram get lan_ipaddr` -p 8388 -k YOURPASSWORD -m chacha20-ietf-poly1305 -u --reuse-port --no-delay -v
2019-12-29 08:35:58 INFO: enable TCP no-delay
2019-12-29 08:35:58 INFO: UDP relay enabled
2019-12-29 08:35:58 INFO: enable TCP no-delay
2019-12-29 08:35:58 INFO: initializing ciphers... chacha20-ietf-poly1305
2019-12-29 08:35:58 INFO: tcp server listening at 192.168.7.1:8388
2019-12-29 08:35:58 INFO: udp server listening at 192.168.7.1:8388
2019-12-29 08:35:58 INFO: running from root user
2019-12-29 08:41:19 INFO: new connection from client, 1 opened client connections
2019-12-29 08:41:19 INFO: [8388] connect to www.gstatic.com:80
2019-12-29 08:41:19 INFO: found address name v6 address www.gstatic.com
2019-12-29 08:41:19 INFO: found address name v4 address www.gstatic.com
2019-12-29 08:41:19 INFO: successfully resolved www.gstatic.com
2019-12-29 08:41:19 INFO: new connection to remote, 1 opened remote connections
2019-12-29 08:41:19 INFO: close a connection to remote, 0 opened remote connections
2019-12-29 08:41:19 INFO: close a connection from client, 0 opened client connections
2019-12-29 08:54:33 INFO: new connection from client, 1 opened client connections
2019-12-29 08:54:33 INFO: [8388] connect to www.gstatic.com:80
2019-12-29 08:54:33 INFO: found address name v4 address www.gstatic.com
2019-12-29 08:54:33 INFO: found address name v6 address www.gstatic.com
2019-12-29 08:54:33 INFO: successfully resolved www.gstatic.com
2019-12-29 08:54:33 INFO: new connection to remote, 1 opened remote connections
2019-12-29 08:54:33 INFO: close a connection to remote, 0 opened remote connections
2019-12-29 08:54:33 INFO: close a connection from client, 0 opened client connections

```



It's fine, no-ip will work.




For Mac and Android, they're all based on socks5, so basically all your browser's traffic will go through the proxy.
So you can safely ignore the 3 and 4 steps.

1. Mac ShadowsocksX-NG
Since you're running ShadowsocksX-NG, configure your browser use socks5 proxy, then you are all set.

2. Shadowsocks-Android
a). In your profile config, turn on Apps VPN mode, select your browser goes through the proxy
Or you can use socks5 mode
a). You can turn on socks5 in settings=>Adanced. Assuming your socks5 proxy port is 1080
b). Type "about:config" in your firefox app address bar
c). Search for "network.proxy"
d). Change those settings
network.proxy.socks=127.0.0.1
network.proxy.socks_port=1080
network.proxy.soccks_remote_dns=true


------------------------------------------------------------
Zerotier Plan

If you're behind a cell network, follow this link, let zerotier handle all your traffics , I haven't done this yet, you'll have to figure it out.

https://zerotier.atlassian.net/wiki...693/Overriding+Default+Route+Full+Tunnel+Mode

Thanks A lot, I have re try above steps again with logs enabled. It seams like issue with DDNS registration (internal/external)

WAN->DDNS->Method to retrieve WAN IP - Internal - working fine in home network and not working remotely
WAN->DDNS->Method to retrieve WAN IP - External - not working remotely either cases.

I tried to use replace ddns with no-ip with asuscomm.com, same results.
https://test.asuscomm.com:port does not work remotely.
http://test.asuscomm.com:port works on home network.
All the respective setting router is enabled .
Administration ->Remote Access Config

 

[MissingTwins]

WAN->DDNS->Method to retrieve WAN IP - Internal - working fine in home network and not working remotely
WAN->DDNS->Method to retrieve WAN IP - External - not working remotely either cases.

Can you ping your home router from outside?

 

[Pradrone]

Can you ping your home router from outside?

Please find below details

Only case 2, able to ping using zero tier
1) Using DDNS
a) WAN->DDNS->Method to retrieve WAN IP - Internal
Internal | WAN IP: private ip dynamically assigned (10.XX.XXX.1) | NOT able to access home router outside network
ping ddns | not reachable

b) PublicIP WAN (Common IPs for multiple home networks) -> MyHomeRouter (private IP (10.XX.XXX.1)/)
External | WAN IP : Public ip (https://whatismyipaddress.com:port8433/) | NOT able to access home router outside network
ping ddns | reachable

2) Using zeroTier
a) zeroTier app -> ManageIP ZeroTier -> router | Able to access home router outside network
ping IP ZeroTier server (10.xx.xx.xxx)| reachable

Thanks a-lot!

 

[MissingTwins]

Please find below details

Only case 2, able to ping using zero tier
1) Using DDNS
a) WAN->DDNS->Method to retrieve WAN IP - Internal
Internal | WAN IP: private ip dynamically assigned (10.XX.XXX.1) | NOT able to access home router outside network
ping ddns | not reachable

b) PublicIP WAN (Common IPs for multiple home networks) -> MyHomeRouter (private IP (10.XX.XXX.1)/)
External | WAN IP : Public ip (https://whatismyipaddress.com:port8433/) | NOT able to access home router outside network
ping ddns | reachable

I see. Currently, Plan A(Direct access to your public WAN IP) would not work with your ISP configuration.
So let's put Plan A aside.

2) Using zeroTier
a) zeroTier app -> ManageIP ZeroTier -> router | Able to access home router outside network
ping IP ZeroTier server (10.xx.xx.xxx)| reachable

Plan B should be working for you, and zerotier alone should be enough.
Let's forget about the shadowsocks and stick to the zerotier.

Just reroute default traffic to your home route, you should be able to access your LAN while surfing the Internet

 

[Pradrone]

I see. Currently, Plan A(Direct access to your public WAN IP) would not work with your ISP configuration.
So let's put Plan A aside.



Plan B should be working for you, and zerotier alone should be enough.
Let's forget about the shadowsocks and stick to the zerotier.

Just reroute default traffic to your home route, you should be able to access your LAN while surfing the Internet

Appreciate you time and guidance . Do u have any steps to be followed to reroute default traffic to home route ?
Thanks alot.

 

[MissingTwins]

Appreciate you time and guidance . Do u have any steps to be followed to reroute default traffic to home route ?
Thanks alot.

I haven't tested yet, please follow this guide:

https://zerotier.atlassian.net/wiki...693/Overriding+Default+Route+Full+Tunnel+Mode

-----------------------------------
By the way, if you have IPv6, that would be globally reachable, maybe Plan A is still feasible.

Here is a very interesting thread about UPNP behind CGNAT:

https://www.reddit.com/r/HomeNetwor..._on_carriergrade_nat_cgn_and_port_forwarding/

And here is a UPNP MAPPER tool, may be it will help.

https://www.howtogeek.com/122227/ho...ts-on-your-router-from-a-desktop-application/

 

[pjv]

Thanks very much @MissingTwins for posting this guide. I have been using Tinc for many years for this purpose and was interested in trying ZeroTier as a potential replacement because it seems quite a bit quicker and easier to deploy and manage.

In case this helps anyone else, I wanted to post about my experience setting up ZeroTier on my router and devices because it went a little different than in your original steps and I think my iptables rules are a little simpler than in your original.

For me, everything is exactly the same for steps 1 - 5, so I will start with...

6. Join ZeroTier network

zerotier-cli join <your network id>

I am running my own ZeroTier network controller in a docker container, so what I see may be a little different than what it looks like on my.zerotier.com, but it is functionally the same. I want to be able to reach the devices on my LAN from any device connected to my ZeroTier network (and vice versa), so I added a Managed Route on the ZeroTier network for my LAN's subnet and set that managed route's gateway to be the ZeroTier IP address of my router. This sets up a simple L3 route that gets pushed by ZeroTier to any device you connect to your ZeroTier network. On my ZeroTier network controller, it looks like this:

(On my router, my actual LAN is a /24 subnet, but I am specifying a /23 subnet in the managed route that ZeroTier pushes in order to 'trick' devices that are both physically on my LAN and also connected to my ZeroTier network to prefer using the /24 route when communicating with other devices on my physical LAN. I THINK this works... But if you will never have devices that are on your physical LAN also independently connected to ZeroTier, you can make the managed route a /24 or whatever exactly matches your LAN subnet and it will work fine like that as well.)

7. Test the network; assuming 10.9.8.10 is a device on your ZeroTier network, this SHOULD WORK PERFECTLY (it did for me).

ping 10.9.8.10 -c 4

8. Add iptables rules (SLIGHTLY DIFFERENT RULES HERE)

NOTE WELL: These rules create a permissive two-way L3 route between your LAN and your ZeroTier network. You should only connect your router to a PRIVATE ZeroTier network of your own making. Don't connect your router to 8056c2e21c000001 (the open ZeroTier earth test network).

iptables -I INPUT -p udp --dport 9993 -j ACCEPT
iptables -I INPUT -i zt+ -j ACCEPT
iptables -I FORWARD -i zt+ -j ACCEPT

9. Test the network, should still work perfectly, and you should now also be able to ping devices on your LAN from any device connected to your ZeroTier network and also ping any device connected to your ZeroTier network from any device connected to your LAN.

10. I did not need to do this. My rules from step 8 already allow me to access the router via http, ssh, etc, and the router considers such connections to be LAN connections. I have both the SSH and HTTP Administration settings set to LAN only (no WAN) and I can manage my router (via either SSH or HTTP) from an external device connected to my ZeroTier network.

11 - 14 Same for me.

 

[Saullofiro]

What exactly should i do on step 13? I'm getting errors and starting the SKYNET firewall.

Saulo@RT-AC68U-0770:/tmp/home/root# /jffs/scripts/nat-start
-sh: /jffs/scripts/nat-start: not found
Saulo@RT-AC68U-0770:/tmp/home/root# /jffs/scripts/wan-start
-sh: /jffs/scripts/wan-start: not found
Saulo@RT-AC68U-0770:/tmp/home/root# /jffs/scripts/firewall-start
STARTS SKYNET Firewall
Saulo@RT-AC68U-0770:/tmp/home/root# /opt/etc/init.d/S90zerotier-one.sh
Usage: /etc/init.d/zerotier-one {start|stop|status}
Saulo@RT-AC68U-0770:/tmp/home/root# /opt/etc/init.d/S90zerotier-one.sh start
713
ZeroTier-One is already running.


Everything seems to be working fine, but i`m not sure it will if i reboot the router.
I'm running version 384.15

 

[MissingTwins]

What exactly should i do on step 13? I'm getting errors and starting the SKYNET firewall.

Saulo@RT-AC68U-0770:/tmp/home/root# /jffs/scripts/nat-start
-sh: /jffs/scripts/nat-start: not found
Saulo@RT-AC68U-0770:/tmp/home/root# /jffs/scripts/wan-start
-sh: /jffs/scripts/wan-start: not found
Saulo@RT-AC68U-0770:/tmp/home/root# /jffs/scripts/firewall-start
STARTS SKYNET Firewall
Saulo@RT-AC68U-0770:/tmp/home/root# /opt/etc/init.d/S90zerotier-one.sh
Usage: /etc/init.d/zerotier-one {start|stop|status}
Saulo@RT-AC68U-0770:/tmp/home/root# /opt/etc/init.d/S90zerotier-one.sh start
713
ZeroTier-One is already running.


Everything seems to be working fine, but i`m not sure it will if i reboot the router.
I'm running version 384.15

Apparently, you need to create those scripts before using them. Did you see the scripts below step 14, **All scripts ** part.

 

[Saullofiro]

Apparently, you need to create those scripts before using them. Did you see the scripts below step 14, **All scripts ** part.

Wow, I haven't. At first I saw only the GitHub hosted script (the S90zerotier-one.sh).
Sorry for the silly question.

I'll test the connectivity after a reboot and then figure out how to connect two different networks (my home network and in-law's home network).

We both use AC-68U's and unfortunately our ISP uses CGNAT, so I'm not sure the speeds will be good, if it even connects reliable. My main goal is to give them access to my Plex Server (Their "not so" Smart TV uses only IPV4 and therefore cant use Plex outside the same network properly because of the CGNAT) and some other services.

 

[Saullofiro]

I've been using this solution for quite a good time to connect 2 networks and handle IPV4 routing between 2 Asus RT-AC68U (as shown in the image bellow). I could use IP address to connect to my Unraid (Plex Server basically) despite being in Network B (with dumb devices that can't run Zerotier, since everything is handled by the routers), or being elsewhere using my laptop. But now something seems to be broken and i cant find what it's.

So I've been thinking if some update/changes on the router`s firmware, Zerotier, my local ISP (both networks use CGNAT since day one, but that hadn't stopped this setup from working flawlessly in the beginning) or if somehow i messed up the iptables rules.

I'd appreciate some help:

Spoiler: Network layout

Spoiler: Ping Test

Overall i can say that Zerotier in the routers isn't working as before/I expected and that the 2 networks aren't linked anymore (Despite the rules in the Zerotier network (as stated in the first image)
The strangest thing i can notice is that only My PC can ping the Router #1 via the Zerotier IP, and I have no clue how and why.

This is my firewall-start script:

#!/bin/sh

logger -t "custom iptables" "Enter" -p user.notice
iptables -C INPUT -i zt+ -j ACCEPT
if [ $? != 0 ]; then
        iptables -I INPUT -p udp --dport 9993 -j ACCEPT
        iptables -I INPUT -i zt+ -j ACCEPT
        iptables -I FORWARD -i zt+ -j ACCEPT
        iptables -t nat -I PREROUTING -i zt+ -d 10.10.10.0/24 -p tcp -m multiport --dport 21,22,80 -j DNAT --to-destination `nvram get lan_ipaddr`
        logger -t "custom iptables" "rules added" -p user.notice
        logger -t "custom iptables" "Firewall Rules Created" -p user.notice
else
        logger -t "custom iptables" "rules existed skip" -p user.notice
fi

 

[MissingTwins]

There is one thing I would like you to confirm, what version of Zerotier you are using on your ac68u?
As what I have known, the Zerotier v1.6.x on ac68u is still broken, you have to fallback to v1.4.6.

I saw you were missing 9399 tcp port, you might need that as well.

And one more thing, if you want LAN over Zerotier, you may need add an extra iptables clause.
Router table is also needed.

Add this to Network B router
ip route add 192.168.0.0/24 via 10.10.10.1
iptables -I INPUT -s 192.168.2.0/24 -j ACCEPT
Add this to Network A router
ip route add 192.168.2.0/24 via 10.10.10.11
iptables -I INPUT -s 192.168.0.0/24 -j ACCEPT

Nesting Zerotier devices is not recommended, it will definitely slowing down your inner lan Zerotier devices' performances. And Nesting Zerotier devices sometimes causing strange drop-out or unreachable nodes.

 

[Saullofiro]

There is one thing I would like you to confirm, what version of Zerotier you are using on your ac68u?
As what I have known, the Zerotier v1.6.x on ac68u is still broken, you have to fallback to v1.4.6.

I'm on 1.6.5, now i just need to find out how to fallback. Is there any easy way or do i need to compile from source?

I saw you were missing 9399 tcp port, you might need that as well.

In addition to UDP 9993?

And one more thing, if you want LAN over Zerotier, you may need add an extra iptables clause.

Added

Spoiler: New firewall-start script

#!/bin/sh

logger -t "custom iptables" "Enter" -p user.notice
iptables -C INPUT -i zt+ -j ACCEPT
if [ $? != 0 ]; then
        iptables -I INPUT -p udp --dport 9993 -j ACCEPT
        iptables -I INPUT -p tcp --dport 9399 -j ACCEPT
        iptables -I INPUT -i zt+ -j ACCEPT
        iptables -I INPUT -s 192.168.0.0/24 -j ACCEPT
        iptables -I FORWARD -i zt+ -j ACCEPT
        iptables -t nat -I PREROUTING -i zt+ -d 10.10.10.0/24 -p tcp -m multiport --dport 21,22,80 -j DNAT --to-destination `nvram get lan_ipaddr`
        logger -t "custom iptables" "rules added" -p user.notice
        logger -t "custom iptables" "Firewall Rules Created" -p user.notice
else
        logger -t "custom iptables" "rules existed skip" -p user.notice
fi

RouterB got updated aswell

Router table is also needed.

Routing table rules seems to be pushed by zerotier (added them via the Managed Routes on the website) and i used /23 routes to make the local network preferred.

RouterA@RT-AC68U:/tmp/home/root# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.100.1   *               255.255.255.255 UH    0      0        0 eth0
192.168.100.0   *               255.255.255.0   U     0      0        0 eth0
192.168.0.0     *               255.255.255.0   U     0      0        0 br0
10.10.10.0      *               255.255.255.0   U     0      0        0 ztxxxxxxxx
192.168.2.0     10.10.10.11     255.255.254.0   UG    0      0        0 ztxxxxxxxx
127.0.0.0       *               255.0.0.0       U     0      0        0 lo
default         192.168.100.1   0.0.0.0         UG    0      0        0 eth0

RouterB gets the other route.

RouterB@RT-AC68U:/tmp/home/root# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.100.1   *               255.255.255.255 UH    0      0        0 eth0
192.168.100.0   *               255.255.255.0   U     0      0        0 eth0
192.168.2.0     *               255.255.255.0   U     0      0        0 br0
10.10.10.0      *               255.255.255.0   U     0      0        0 ztxxxxxxxx
192.168.0.0     10.10.10.1      255.255.254.0   UG    0      0        0 ztxxxxxxxx
127.0.0.0       *               255.0.0.0       U     0      0        0 lo
default         192.168.100.1   0.0.0.0         UG    0      0        0 eth0

Nesting Zerotier devices is not recommended, it will definitely slowing down your inner lan Zerotier devices' performances. And Nesting Zerotier devices sometimes causing strange drop-out or unreachable nodes.

Planning to use Tailscale for device to device connection, might leave only the routers using Zerotier.

 

[MissingTwins]

Add route table also need to your laptop.

route -p ADD 192.168.0.0 MASK 255.255.255.0 10.10.10.1
route -p ADD 192.168.2.0 MASK 255.255.255.0 10.10.10.11

I'm on 1.6.5, now i just need to find out how to fallback. Is there any easy way or do i need to compile from source?

Yes pretty easy.

opkg remove zerotier
opkg install http://bin.entware.net/armv7sf-k2.6/archive/zerotier_1.4.6-5_armv7-2.6.ipk

In addition to UDP 9993?

Yes

Routing table rules seems to be pushed by zerotier (added them via the Managed Routes on the website) and i used /23 routes to make the local network preferred.

Managed push seems not functioning properly, so manually adding ip route though automatic bash script is recommended. You may add cru or add to /jffs/scripts/wan-event

cru a ZeroTierRouteDaemon "*/2 * * * * /jffs/scripts/lan-route-table.sh"

#!/bin/sh
# wan-event

echo "wan-event Enter"
logger -t "wan-event" -c "WAN"$1", eventType="$2" " -p user.notice

if [ "connected" == "$2" ]; then
    echo "Detect connection"
    logger -t "wan-event" "Detect connection" -p user.notice
    /jffs/scripts/lan-route-table.sh
fi

echo "wan-event leaving"
logger -t "wan-event" "Leaving" -p user.notice

#! /bin/sh
#lan-route-table.sh

TEST_ARGS=$(ip route show 192.168.0.0/24 | wc -l)
if [ $TEST_ARGS -eq 0 ]; then
    ip route add 192.168.0.0/24 via 10.10.10.1
    logger -t "lan-route-table.sh" -c "zerotier LAN route added" -p user.notice
fi

 

[DoradoUS]

I didn't see the problem with your home server side. It should work. As long as your server-side is not behind cell-network(if you can ping your router), ss-server should be reachable.

There is a third plan, run both ss-server and zerotier on a VPS, add an ip route on the VPS which points to your home router.

$ ip route add 192.168.1.1 via 10.9.8.2
# Enable IP forwading
sudo sysctl -w net.ipv4.ip_forward=1
sudo sysctl -w net.ipv4.conf.zt0.route_localnet=1
# Enable BBR
sudo sysctl -w net.core.default_qdisc=fq
sudo sysctl -w net.ipv4.tcp_congestion_control=bbr

The third plan also works for me.

Hi, MissingTwins
Thanks for providing such a detailed zerotier guide, now my iPhone and android phone, iPad, MacBook Pro, etc., can login my ac88u router via WebUI or Qnap NAS app, through wifi in-home/outside, even with cell phone network, via both zerotier generated IPs or home router's local LAN IP when the zerotier clients running in the background. And the speed is awesome.

I'm very interested in your 3rd plan, run both ss-server and zerotier on a VPS. Currently I'm running a docker for SS+V2ray plugin SS-server on the GCP cloud VM and the SS client on my Ac88U, firmware of Asuswrt-merlin 386.3.2. Though I'm not a IT guy but just have followed a great githuber's scripts. docker-shadowsocks-libev and shadowsocks-asuswrt-merlin made it work that let all devices surfing ineternet, bypass restriction in home and use SS clients when outside with cellphone data. I like the concept of your 3rd plan but don't understand the steps clearly, and how to apply it with my current docker ss-v2ray server, please help if convenient. BTW: there's a article in digitalocean talk about VPN plan for zerotier on VPS which logic I can understand but prefer shadowsocks for its ability of proxy traffic on demand.

Questions:

1. on the VPS, do i need other iptables rules for docker ss-server after add ip route 192.168.1.1 to ponint home router's zerotier ip 172.30.55.44 and enable ip forwording?
2. what about the part for the VPS's route setting In the zerotier managed route section?
3. Devices using cellphone data outside, what's the 'server address' in SS clinet's configuration

{
"server": "??????", #vps public ip or vps zerotier ip or home router's zerotier ip?
"server_port": 9443,
"local_address": "0.0.0.0", #or newly added ip route, 192.168.1.1
"local_port": 10XX,
"password": "PASSWORD",
"timeout": 600,
"method": "aes-256-gcm",
"mode": "tcp_only",
"plugin": "v2ray-plugin",
"plugin_opts": "tls;host=mydomain.com;path=/XXXX"
}

INFO OF MY SETTINGS:
1.My VPS info
1.1 ip route (#add ip route, point to home router zerotier ip?)
sudo ip route add 192.168.1.1 via 172.30.55.44
default via 10.138.0.1 dev eth0
10.138.0.1 dev eth0 scope link
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
172.18.0.0/16 dev br-4657f14ca6e5 proto kernel scope link src 172.18.0.1
172.20.0.0/16 dev docker_gwbridge proto kernel scope link src 172.20.0.1 linkdown
172.30.0.0/16 dev ztMYSPECIFIC proto kernel scope link src 172.30.165.217
192.168.1.1 via 172.30.55.44 dev ztMYSPECIFIC proto static

#172.30.0.0/16 zerotier network, 192.168.50.0/24 my ac88u LAN subnet, 172.30.55.44 my ac88u zerotier ip; 172.30.165.217 my vps zerotier ip)

1.2 VPS iptables related with docker-proxy
sudo iptables -v -t nat -L
...
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- br-4657f14ca6e5 any anywhere anywhere
0 0 RETURN all -- docker0 any anywhere anywhere
0 0 RETURN all -- docker_gwbridge any anywhere anywhere
7863 470K DNAT tcp -- !br-4657f14ca6e5 any anywhere anywhere tcp dpt:9443 to:172.18.0.2:9443
214 52176 DNAT udp -- !br-4657f14ca6e5 any anywhere anywhere udp dpt:9443 to:172.18.0.2:9443

POSTROUTING
sudo iptables -v -t nat -L POSTROUTING
Chain POSTROUTING (policy ACCEPT 16359 packets, 1082K bytes)
pkts bytes target prot opt in out source destination
26568 1622K MASQUERADE all -- any !br-4657f14ca6e5 172.18.0.0/16 anywhere
0 0 MASQUERADE all -- any !docker0 172.17.0.0/16 anywhere
0 0 MASQUERADE all -- any !docker_gwbridge 172.20.0.0/16 anywhere
0 0 MASQUERADE tcp -- any any 172.18.0.2 172.18.0.2 tcp dpt:9443
0 0 MASQUERADE udp -- any any 172.18.0.2 172.18.0.2 udp dpt:9443

2.My zerotier info:
managed routes
172.30.0.0/16 (LAN)
192.168.50.0/24 via 172.30.55.44 #asus ac88u lan subnet via ac88u zerotier ip.

3.My ac88u with zerotier:
3.1.ip route
/tmp/mnt/AC88USD/entware# ip route
172.56.0.1 dev ppp0 proto kernel scope link
192.168.50.0/24 dev br0 proto kernel scope link src 192.168.50.1
172.30.0.0/16 dev ztMYSPECIFIC proto kernel scope link src 172.30.55.44
169.254.0.0/16 dev eth0 proto kernel scope link src 169.254.148.159
127.0.0.0/8 dev lo scope link
default via 172.56.0.1 dev ppp0

3.2.ip tables:
iptables -I INPUT 3 -i zt+ -j ACCEPT
iptables -t nat -I PREROUTING -i zt+ -d 172.30.0.0/16 -p tcp -m multiport --dport 21,22.80.443 -j DNAT --to-destination `nvram get lan_ipaddr`
iptables -t nat -A POSTROUTING -o br0 -s 172.30.0.0/16 -j SNAT --to-source `nvram get lan_ipaddr`
iptables -I FORWARD -i zt+ -d `nvram get lan_ipaddr`/24 -j ACCEPT
iptables -I FORWARD -i br0 -d 172.30.0.0/16 -j ACCEPT