What's new

Wireguard Session Manager - Discussion (3rd) thread

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

FYI,

wireguard_manager Beta v4.16bB now makes an exception for IPSets of Type: hash:mac whereby, rather than insisting on creating a duplicate IPv6 IPset (mirroring the contents of the IPv4 MAC IPSet), the IPv6 firewall rule will now also be created if appropriate.
Following @ZebMcKayhan changes recommended in #54 and updating to Beta v4.16bB, my setup appears to be completely broken. The update took quite a while before happening and (just in case I ran it again). Somehow this also messed up unbound (logging was not correct and time server had reset to ntp from Chrony) and the IPv6 WAN interface was completely missing. Once I had done the necessary repairs I noted that my VPN (MAC assigned) device was no longer going through the VPN.

I had been experimenting with where and how to add the IPv6 alias and restore the ipset on boot and before making the above changes I had come to the following conclusions
  • that the ipv6 alias for br0 needs to be up before unbound is started
  • that the ipset needs to be restored before wg_manager is started (which is after unbound)
  • that ipv6 full functionality appear to only happen well after ipv4 (e.g. adding an ipv4 alias works in wan-event or nat-start with no wait state, it does not for an ipv6 alias)
  • selecting the wait time was somewhat hit and miss.
I therefore added the required steps (ipv6 alias and ipset restore) by adding S60wg11-mac to /opt/etc/init.d - my understanding is this would force it to run just before S61unbound - I am sure there are better ways but it appears to work for now
Code:
#!/bin/sh
ip -6 address add dev br0 fd36:7ef1:2add:aa88:100::1/128
IPSET_NAME=wg11-mac
if [ "$(ipset list -n "$IPSET_NAME" 2>/dev/null)" != "$IPSET_NAME" ]; then #if ipset does not already exist
   if [ -s "/opt/tmp/$IPSET_NAME" ]; then #if a backup file exists
      ipset restore -! <"/jffs/addons/wireguard/$IPSET_NAME" #restore ipset
    fi
fi
and wg11-up.sh and wg11-down.sh were
Code:
#!/bin/sh
###############################################################################
WanIp6=$(nvram get ipv6_rtr_addr)     #WanIp6=2001:1111:2222:3333::1
iptables -t nat -I POSTROUTING ! -s <vpn_ipv4> -o wg11 -j MASQUERADE
ip6tables -t nat -I POSTROUTING ! -s <vpn_ipv6> -o wg11 -j MASQUERADE
ip6tables -t nat -A WGDNS1 -i br0 -j DNAT --to-destination ${WanIp6}
ip6tables -t mangle -I PREROUTING -m set --match-set wg11-mac src -j MARK --set-mark 0x1000/0x1000
iptables -t mangle -I PREROUTING -m set --match-set wg11-mac src -j MARK --set-mark 0x1000/0x1000
ip rule add from all fwmark 0x1000 table 121 prio 9900
ip -6 rule add from all fwmark 0x1000 table 121 prio 9900
echo 2 > /proc/sys/net/ipv4/conf/wg11/rp_filter
Code:
#!/bin/sh
###############################################################################
iptables -t nat -D POSTROUTING ! -s <vpn_ipv4> -o wg11 -j MASQUERADE
ip6tables -t nat -D POSTROUTING ! -s <vpn_ipv6> -o wg11 -j MASQUERADE
ip6tables -t mangle -D PREROUTING -m set --match-set wg11-mac src -j MARK --set-mark 0x1000/0x1000
iptables -t mangle -D PREROUTING -m set --match-set wg11-mac src -j MARK --set-mark 0x1000/0x1000
ip rule del from all fwmark 0x1000 table 121 prio 9900
ip -6 rule del from all fwmark 0x1000 table 121 prio 9900

After making the recommended changes and adding
Code:
E:Option ==> peer wg11 add ipset wg11-mac
wg11-up.sh and wg11-down.sh were
Code:
#!/bin/sh
###############################################################################
WanIp6=$(nvram get ipv6_rtr_addr)     #WanIp6=2001:1111:2222:3333::1
iptables -t nat -I POSTROUTING ! -s 10.0.11.124 -o wg11 -j MASQUERADE
ip6tables -t nat -I POSTROUTING ! -s 2a0e:1c80:4:2000::b7d -o wg11 -j MASQUERADE
ip6tables -t nat -A WGDNS1 -i br0 -j DNAT --to-destination ${WanIp6}
Code:
#!/bin/sh
###############################################################################
iptables -t nat -D POSTROUTING ! -s 10.0.11.124 -o wg11 -j MASQUERADE
ip6tables -t nat -D POSTROUTING ! -s 2a0e:1c80:4:2000::b7d -o wg11 -j MASQUERADE
Although I can see that the ipset and aliases exist and unbound is routing its DNS requests via the ipv6 alias (I have ipv6 preferred), these are all on port 53 and
  • All of the devices are showing the ISP assigned IPv4 address as the sole DNS server (before they showed the Azire DNS servers)
  • the MAC device is not going through the VPN at all.
If it is relevant running
Code:
E:Option ==> ?

        Router RT-AX88U Firmware (v386.5_2)

        [✔] Entware Architecture arch=aarch64

        v4.16bB WireGuard Session Manager (Change Log: https://github.com/MartineauUK/wireguard/commits/dev/wg_manager.sh)
        MD5=6bd27234bb2fcbe38ef7d4ac7edb5814 /jffs/addons/wireguard/wg_manager.sh
.
.
.
.
        [✔] IPv6 Service is dhcp6
Where my IPv6 is set to native and running SLAAC rather than dhcp6

As I changed the vpn settings and updated the version at the same time (and it did not appear to run smoothly) I am at a loss as how best to troubleshoot. Ideally I would like to revert to my old settings and then check step by step, but that would require the previous dev version.
 
As I changed the vpn settings and updated the version at the same time (and it did not appear to run smoothly) I am at a loss as how best to troubleshoot. Ideally I would like to revert to my old settings and then check step by step, but that would require the previous dev version.
When you say "update did not go well" what does that mean? Some error message or just that it took time? Running uf dev again should re-download all files again. Github could sometimes be slow and take time to apply updates. Maybee try update again today?

If the ipset is somehow misbehaving, try remove it from wgm and use the rules manually (or did you check so wgm applies all your rules?)

There is a pending Entware update that seem to cause some issues for some people. did you update Entware as well?

if your policy rules dont seems to apply, check so the peer is really up (check handshake i.e. wg show)

check routing rules, so your policy rules points to table 121:
INI:
ip rule
ip -6 rule

and the policy routing table 121 so it default routes to wg11.
Code:
ip route show table 121
ip -6 route show table 121

while it is understandable that Unbound is picky that the interface specified in outgoing-interface actually exist and may refuse to start if it dont exist, it should not make any difference on ipv4 and ipv6. my router sets up my br0 ipv6 so it is created early but it is completely broken until wg11 start so I actually get an ipv6 connection, but this dont appear to be a problem for Unbound.

I use X3mRouting wich restores my ipset in nat-start after a sleep 10 which also does not appear to cause any issue. but I'm only using it for netflix vpn bypass and apparently my vpn is not blocked by netflix (yet, just trying to be prepared) so I might not notice if the ipset/rules does not apply if the ipset did not exist when wgm tried to apply the rules

maybe your ipv6 interface alias could be added even earlier, perhaps even init-start? I'll have to read up on @RMerlin page to find roughly the first instance after the ordinary interfaces are set up.

Edit: Think that init-start would be appropriate to setup your ipv4+ipv6 addresses to br0, what if you put this in init-start (and comment other instances of it):
Code:
#!/bin/sh
ifconfig br0:1 192.168.3.1 netmask 255.255.255.255
ip -6 address add dev br0 fd36:7ef1:2add:aa88:100::1/128

then at least the addresses exist when Unbound and whatnot starts.
 
Last edited:
Following @ZebMcKayhan changes recommended in #54 and updating to Beta v4.16bB, my setup appears to be completely broken. The update took quite a while before happening and (just in case I ran it again). Somehow this also messed up unbound (logging was not correct and time server had reset to ntp from Chrony) and the IPv6 WAN interface was completely missing. Once I had done the necessary repairs I noted that my VPN (MAC assigned) device was no longer going through the VPN.

I had been experimenting with where and how to add the IPv6 alias and restore the ipset on boot and before making the above changes I had come to the following conclusions
  • that the ipv6 alias for br0 needs to be up before unbound is started
  • that the ipset needs to be restored before wg_manager is started (which is after unbound)
  • that ipv6 full functionality appear to only happen well after ipv4 (e.g. adding an ipv4 alias works in wan-event or nat-start with no wait state, it does not for an ipv6 alias)
  • selecting the wait time was somewhat hit and miss.
I therefore added the required steps (ipv6 alias and ipset restore) by adding S60wg11-mac to /opt/etc/init.d - my understanding is this would force it to run just before S61unbound - I am sure there are better ways but it appears to work for now
Code:
#!/bin/sh
ip -6 address add dev br0 fd36:7ef1:2add:aa88:100::1/128
IPSET_NAME=wg11-mac
if [ "$(ipset list -n "$IPSET_NAME" 2>/dev/null)" != "$IPSET_NAME" ]; then #if ipset does not already exist
   if [ -s "/opt/tmp/$IPSET_NAME" ]; then #if a backup file exists
      ipset restore -! <"/jffs/addons/wireguard/$IPSET_NAME" #restore ipset
    fi
fi
and wg11-up.sh and wg11-down.sh were
Code:
#!/bin/sh
###############################################################################
WanIp6=$(nvram get ipv6_rtr_addr)     #WanIp6=2001:1111:2222:3333::1
iptables -t nat -I POSTROUTING ! -s <vpn_ipv4> -o wg11 -j MASQUERADE
ip6tables -t nat -I POSTROUTING ! -s <vpn_ipv6> -o wg11 -j MASQUERADE
ip6tables -t nat -A WGDNS1 -i br0 -j DNAT --to-destination ${WanIp6}
ip6tables -t mangle -I PREROUTING -m set --match-set wg11-mac src -j MARK --set-mark 0x1000/0x1000
iptables -t mangle -I PREROUTING -m set --match-set wg11-mac src -j MARK --set-mark 0x1000/0x1000
ip rule add from all fwmark 0x1000 table 121 prio 9900
ip -6 rule add from all fwmark 0x1000 table 121 prio 9900
echo 2 > /proc/sys/net/ipv4/conf/wg11/rp_filter
Code:
#!/bin/sh
###############################################################################
iptables -t nat -D POSTROUTING ! -s <vpn_ipv4> -o wg11 -j MASQUERADE
ip6tables -t nat -D POSTROUTING ! -s <vpn_ipv6> -o wg11 -j MASQUERADE
ip6tables -t mangle -D PREROUTING -m set --match-set wg11-mac src -j MARK --set-mark 0x1000/0x1000
iptables -t mangle -D PREROUTING -m set --match-set wg11-mac src -j MARK --set-mark 0x1000/0x1000
ip rule del from all fwmark 0x1000 table 121 prio 9900
ip -6 rule del from all fwmark 0x1000 table 121 prio 9900

After making the recommended changes and adding
Code:
E:Option ==> peer wg11 add ipset wg11-mac
wg11-up.sh and wg11-down.sh were
Code:
#!/bin/sh
###############################################################################
WanIp6=$(nvram get ipv6_rtr_addr)     #WanIp6=2001:1111:2222:3333::1
iptables -t nat -I POSTROUTING ! -s 10.0.11.124 -o wg11 -j MASQUERADE
ip6tables -t nat -I POSTROUTING ! -s 2a0e:1c80:4:2000::b7d -o wg11 -j MASQUERADE
ip6tables -t nat -A WGDNS1 -i br0 -j DNAT --to-destination ${WanIp6}
Code:
#!/bin/sh
###############################################################################
iptables -t nat -D POSTROUTING ! -s 10.0.11.124 -o wg11 -j MASQUERADE
ip6tables -t nat -D POSTROUTING ! -s 2a0e:1c80:4:2000::b7d -o wg11 -j MASQUERADE
Although I can see that the ipset and aliases exist and unbound is routing its DNS requests via the ipv6 alias (I have ipv6 preferred), these are all on port 53 and
  • All of the devices are showing the ISP assigned IPv4 address as the sole DNS server (before they showed the Azire DNS servers)
  • the MAC device is not going through the VPN at all.
If it is relevant running
Code:
E:Option ==> ?

        Router RT-AX88U Firmware (v386.5_2)

        [✔] Entware Architecture arch=aarch64

        v4.16bB WireGuard Session Manager (Change Log: https://github.com/MartineauUK/wireguard/commits/dev/wg_manager.sh)
        MD5=6bd27234bb2fcbe38ef7d4ac7edb5814 /jffs/addons/wireguard/wg_manager.sh
.
.
.
.
        [✔] IPv6 Service is dhcp6
Where my IPv6 is set to native and running SLAAC rather than dhcp6

As I changed the vpn settings and updated the version at the same time (and it did not appear to run smoothly) I am at a loss as how best to troubleshoot. Ideally I would like to revert to my old settings and then check step by step, but that would require the previous dev version.
Not sure if the implication that simply upgrading script wireguard_manager to Beta v4.16bB somehow not only borked unbound, switched the NTPD server from Chrony but also removed the IPv6 WAN configuration is justified/possible?

The change in Beta v4.16bB is (hopefully) fairly trivial, but it could be that the requested modification to auto create the matching IPv6 IPSet Selective Routing iptables rule from the IPv4 IPSet broke the intended insertion of both the IPv4 and IPv6 rule.

Anyway, to manually backout Beta v4.16bB to revert wireguard_manager to Beta v4.16bA use:
Code:
curl https://raw.githubusercontent.com/MartineauUK/wireguard/2c13af47c6b7eea910ba3a11d30bdf3d579755d4/wg_client -o /jffs/addons/wireguard/wg_client

Code:
curl https://raw.githubusercontent.com/MartineauUK/wireguard/e83d0e74b0046c613908c9723fa99ce1350d037b/wg_manager.sh -o /jffs/addons/wireguard/wg_manager.sh
 
When you say "update did not go well" what does that mean? Some error message or just that it took time? Running uf dev again should re-download all files again. Github could sometimes be slow and take time to apply updates. Maybee try update again today?

If the ipset is somehow misbehaving, try remove it from wgm and use the rules manually (or did you check so wgm applies all your rules?)

There is a pending Entware update that seem to cause some issues for some people. did you update Entware as well?

if your policy rules dont seems to apply, check so the peer is really up (check handshake i.e. wg show)

check routing rules, so your policy rules points to table 121:
INI:
ip rule
ip -6 rule

and the policy routing table 121 so it default routes to wg11.
Code:
ip route show table 121
ip -6 route show table 121

while it is understandable that Unbound is picky that the interface specified in outgoing-interface actually exist and may refuse to start if it dont exist, it should not make any difference on ipv4 and ipv6. my router sets up my br0 ipv6 so it is created early but it is completely broken until wg11 start so I actually get an ipv6 connection, but this dont appear to be a problem for Unbound.

I use X3mRouting wich restores my ipset in nat-start after a sleep 10 which also does not appear to cause any issue. but I'm only using it for netflix vpn bypass and apparently my vpn is not blocked by netflix (yet, just trying to be prepared) so I might not notice if the ipset/rules does not apply if the ipset did not exist when wgm tried to apply the rules

maybe your ipv6 interface alias could be added even earlier, perhaps even init-start? I'll have to read up on @RMerlin page to find roughly the first instance after the ordinary interfaces are set up.

Edit: Think that init-start would be appropriate to setup your ipv4+ipv6 addresses to br0, what if you put this in init-start (and comment other instances of it):
Code:
#!/bin/sh
ifconfig br0:1 192.168.3.1 netmask 255.255.255.255
ip -6 address add dev br0 fd36:7ef1:2add:aa88:100::1/128

then at least the addresses exist when Unbound and whatnot starts.
The update was just very slow and while I did run it a second time yesterday I can try this again today.

I tested with and without using the ipset (but I may not have done this correctly so I will redo this today). I am not sure what you meant by 'did you check so wgm applies all your rules?'

I have already added the entware updates - there was an issue with Unbound (/opt/etc/init.d/61unbound was over written and some permissions changed) but these were repaired.

Looking at wg show
Code:
interface: wg11
  public key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx=
  private key: (hidden)
  listening port: 57009

peer: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx=
  endpoint: <vpn_ip4>:<vpn_port>
  allowed ips: 0.0.0.0/0, ::/0
  latest handshake: 12 minutes, 49 seconds ago
  transfer: 1.56 KiB received, 14.62 KiB sent
ip rule
Code:
0:      from all lookup local
9911:   from 192.168.3.1 lookup 121
9991:   from all fwmark 0x1000/0x1000 lookup 121
32766:  from all lookup main
32767:  from all lookup default
ip -6 rule
Code:
0:      from all lookup local
9911:   from fd36:7ef1:2add:aa88:100::1 lookup 121
9991:   from all fwmark 0x1000/0x1000 lookup 121
32766:  from all lookup main
ip route show table 121
Code:
10.0.0.0/8 dev br0 proto kernel scope link src 10.50.60.10
10.50.60.0/24 dev br0 proto kernel scope link src 10.50.60.1
ip -6 route show table 121
Code:
<LAN IPv6 Address> dev br0 proto kernel metric 256 pref medium
fd36:7ef1:2add:aa88:100::1 dev br0 proto kernel metric 256 pref medium
fe80::/64 dev br0 proto kernel metric 256 pref medium
So no reference to wg11 and I don't remember seeing the last line (fe80:: ...) before.
Looking at E:Option ==> peer wg11
Code:
Client  Auto  IP                                        Endpoint                   DNS                                   MTU  Annotate
wg11    P     <vpn_ipv4_range>, <vpn_ipv6_range>  nl1.wg.azirevpn.net:pppp  10.50.60.1,fe80::aa5e:45ff:xxxx:xxxx       # N/A

        Selective Routing RPDB rules
ID  Peer  Interface  Source                      Destination  Description
4   wg11  VPN        fd36:7ef1:2add:aa88:100::1  Any          Unbound6VPN
3   wg11  VPN        192.168.3.1                 Any          Unbound4VPN

IPSet     Enable  Peer  FWMark  DST/SRC
wg11-mac  Y       wg11  0x1000  src

In regard to getting the ipv6 alias in place before unbound starts, this is only a problem as as I have set ipv6 as the preferred lookup protocol. I could remove this, but I would prefer to keep it - as a check that 'everything is in in place' if nothing else.

In case it is relevant I have also started looking at the iptables rules (I had noted all these earlier - just in case) and have observed some changes (these may be as a result of the entware updates) which appear to relate to multicasting, though I don't know why they have appeared.
For instance if I look at iptables -nvL, new first chain INPUT and FORWARD rules have been added.
Code:
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     2    --  eth0   *       0.0.0.0/0            0.0.0.0/0
Code:
Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  eth0   *       0.0.0.0/0            224.0.0.0/4
and in ip6tables -nvL the 3rd entry in Chain FORWARD is also new
Code:
Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    2   192 REJECT     udp      br0    *       ::/0                 ::/0                 udp dpt:123 reject-with icmp6-port-unreachable
    0     0 REJECT     tcp      br0    *       ::/0                 ::/0                 tcp dpt:123 reject-with icmp6-port-unreachable
    0     0 ACCEPT     all      eth0   *       ::/0                 ff00::/8
Looking at iptables -nvL -t nat, rule 5 in PREROUTING is new
Code:
Chain PREROUTING (policy ACCEPT 158K packets, 31M bytes)
 pkts bytes target     prot opt in     out     source               destination
14098  946K WGDNS1     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:53 /* WireGuard 'client1 DNS' */
   16   668 WGDNS1     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53 /* WireGuard 'client1 DNS' */
    0     0 DNAT       tcp  --  br0    *       0.0.0.0/0            0.0.0.0/0            tcp dpt:123 to:10.55.63.1
  299 22822 DNAT       udp  --  br0    *       0.0.0.0/0            0.0.0.0/0            udp dpt:123 to:10.55.63.1
    0     0 ACCEPT     all  --  eth0   *       0.0.0.0/0            224.0.0.0/4
and while the POSTROUTING rules are unchanged, the traffic distribution has altered
Code:
Chain POSTROUTING (policy ACCEPT 59118 packets, 4259K bytes)
 pkts bytes target     prot opt in     out     source               destination
76402 5639K MASQUERADE  all  --  *      wg11    10.50.60.0/24        0.0.0.0/0            /* WireGuard 'client' */
35317 2831K MASQUERADE  all  --  *      wg11   !VPN                 0.0.0.0/0
82865 6026K PUPNP      all  --  *      eth0    0.0.0.0/0            0.0.0.0/0
61866 4115K MASQUERADE  all  --  *      eth0   !WAN                 0.0.0.0/0
 4193  438K MASQUERADE  all  --  *      br0     10.50.60.0/24        10.50.60.0/24
Code:
Chain POSTROUTING (policy ACCEPT 38269 packets, 2715K bytes)
 pkts bytes target     prot opt in     out     source               destination
   15   850 MASQUERADE  all  --  *      wg11    10.50.60.0/24        0.0.0.0/0            /* WireGuard 'client' */
    0     0 MASQUERADE  all  --  *      wg11   !VPN                  0.0.0.0/0
 150K   12M PUPNP      all  --  *      eth0    0.0.0.0/0             0.0.0.0/0
 146K   12M MASQUERADE  all  --  *      eth0   !WAN                  0.0.0.0/0
  694 72477 MASQUERADE  all  --  *      br0     10.50.60.0/24        10.50.60.0/24
and for ip6tables -nvL -t nat, no new rules but the same changes to POSTROUTING
Code:
Chain POSTROUTING (policy ACCEPT 23377 packets, 6912K bytes)
 pkts bytes target     prot opt in     out     source               destination
26789 3215K MASQUERADE  all      *      wg11    WAN                 ::/0                 /* WireGuard 'client' */
 104K   10M MASQUERADE  all      *      wg11   !VPN                 ::/0
Code:
Chain POSTROUTING (policy ACCEPT 110K packets, 13M bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MASQUERADE  all      *      wg11    WAN                 ::/0                 /* WireGuard 'client' */
    0     0 MASQUERADE  all      *      wg11   !VPN                 ::/0
 
Continued....
and with iptables -nvL -t mangle
Code:
Chain PREROUTING (policy ACCEPT 12M packets, 8148M bytes)
 pkts bytes target     prot opt in     out     source               destination
3648K 1511M MARK       all  --  wg11   *       0.0.0.0/0            0.0.0.0/0            /* WireGuard 'client' */ MARK xset 0x1/0x7
3811K 4656M MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MAC 00:15:5D:XX:YY:ZZ MARK or 0x1000

Chain INPUT (policy ACCEPT 4352K packets, 1887M bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 7567K packets, 6257M bytes)
 pkts bytes target     prot opt in     out     source               destination
3810K 4655M MARK       all  --  *      wg11    0.0.0.0/0            0.0.0.0/0            /* WireGuard 'client' */ MARK xset 0x1/0x7
 2366  134K TCPMSS     tcp  --  wg11   *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* WireGuard 'client' */ TCPMSS clamp to PMTU
13520  811K TCPMSS     tcp  --  *      wg11    0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* WireGuard 'client' */ TCPMSS clamp to PMTU

Chain OUTPUT (policy ACCEPT 6605K packets, 5239M bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 14M packets, 11G bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain BWDPI_FILTER (0 references)
 pkts bytes target     prot opt in     out     source               destination
Code:
Chain PREROUTING (policy ACCEPT 21M packets, 18G bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MARK       all  --  wg11   *       0.0.0.0/0            0.0.0.0/0            /* WireGuard 'client' */ MARK xset 0x1/0x7
3891K 4114M MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set wg11-mac src /* WireGuard 'client' */ MARK or 0x1000

Chain INPUT (policy ACCEPT 758K packets, 136M bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 20M packets, 17G bytes)
 pkts bytes target     prot opt in     out     source               destination
  138  8056 MARK       all  --  *      wg11    0.0.0.0/0            0.0.0.0/0            /* WireGuard 'client' */ MARK xset 0x1/0x7
    0     0 TCPMSS     tcp  --  wg11   *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* WireGuard 'client' */ TCPMSS clamp to PMTU
  110  6880 TCPMSS     tcp  --  *      wg11    0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* WireGuard 'client' */ TCPMSS clamp to PMTU

Chain OUTPUT (policy ACCEPT 604K packets, 132M bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 21M packets, 18G bytes)
 pkts bytes target     prot opt in     out     source               destination
Similarly with ip6tables -nvL -t mangle (I have ignored the DNSFILTER* items which appear to relate to the traffic for devices outside the default policy based routing )
Code:
Chain PREROUTING (policy ACCEPT 880K packets, 809M bytes)
 pkts bytes target     prot opt in     out     source               destination
61181   76M MARK       all      wg11   *       ::/0                 ::/0                 /* WireGuard 'client' */ MARK xset 0x1/0x7
44537   20M MARK       all      *      *       ::/0                 ::/0                 MAC 00:15:5D:XX:YY:ZZ MARK or 0x1000

Chain FORWARD (policy ACCEPT 853K packets, 799M bytes)
 pkts bytes target     prot opt in     out     source               destination
42582   20M MARK       all      *      wg11    ::/0                 ::/0                 /* WireGuard 'client' */ MARK xset 0x1/0x7
   19  1392 TCPMSS     tcp      wg11   *       ::/0                 ::/0                 tcp flags:0x06/0x02 /* WireGuard 'client' */ TCPMSS clamp to PMTU
   44  3520 TCPMSS     tcp      *      wg11    ::/0                 ::/0                 tcp flags:0x06/0x02 /* WireGuard 'client' */ TCPMSS clamp to PMTU
Code:
Chain PREROUTING (policy ACCEPT 3319K packets, 3149M bytes)
 pkts bytes target     prot opt in     out     source               destination
   11  1276 MARK       all      wg11   *       ::/0                 ::/0                 /* WireGuard 'client' */ MARK xset 0x1/0x7
91262   54M MARK       all      *      *       ::/0                 ::/0                 match-set wg11-mac src /* WireGuard 'client' */ MARK or 0x1000

Chain FORWARD (policy ACCEPT 3166K packets, 3075M bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MARK       all      *      wg11    ::/0                 ::/0                 /* WireGuard 'client' */ MARK xset 0x1/0x7
    0     0 TCPMSS     tcp      wg11   *       ::/0                 ::/0                 tcp flags:0x06/0x02 /* WireGuard 'client' */ TCPMSS clamp to PMTU
    0     0 TCPMSS     tcp      *      wg11    ::/0                 ::/0                 tcp flags:0x06/0x02 /* WireGuard 'client' */ TCPMSS clamp to PMTU
 
Continued....
and with iptables -nvL -t mangle
Code:
Chain PREROUTING (policy ACCEPT 12M packets, 8148M bytes)
 pkts bytes target     prot opt in     out     source               destination
3648K 1511M MARK       all  --  wg11   *       0.0.0.0/0            0.0.0.0/0            /* WireGuard 'client' */ MARK xset 0x1/0x7
3811K 4656M MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MAC 00:15:5D:XX:YY:ZZ MARK or 0x1000

Chain INPUT (policy ACCEPT 4352K packets, 1887M bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 7567K packets, 6257M bytes)
 pkts bytes target     prot opt in     out     source               destination
3810K 4655M MARK       all  --  *      wg11    0.0.0.0/0            0.0.0.0/0            /* WireGuard 'client' */ MARK xset 0x1/0x7
 2366  134K TCPMSS     tcp  --  wg11   *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* WireGuard 'client' */ TCPMSS clamp to PMTU
13520  811K TCPMSS     tcp  --  *      wg11    0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* WireGuard 'client' */ TCPMSS clamp to PMTU

Chain OUTPUT (policy ACCEPT 6605K packets, 5239M bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 14M packets, 11G bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain BWDPI_FILTER (0 references)
 pkts bytes target     prot opt in     out     source               destination
Code:
Chain PREROUTING (policy ACCEPT 21M packets, 18G bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MARK       all  --  wg11   *       0.0.0.0/0            0.0.0.0/0            /* WireGuard 'client' */ MARK xset 0x1/0x7
3891K 4114M MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set wg11-mac src /* WireGuard 'client' */ MARK or 0x1000

Chain INPUT (policy ACCEPT 758K packets, 136M bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 20M packets, 17G bytes)
 pkts bytes target     prot opt in     out     source               destination
  138  8056 MARK       all  --  *      wg11    0.0.0.0/0            0.0.0.0/0            /* WireGuard 'client' */ MARK xset 0x1/0x7
    0     0 TCPMSS     tcp  --  wg11   *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* WireGuard 'client' */ TCPMSS clamp to PMTU
  110  6880 TCPMSS     tcp  --  *      wg11    0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* WireGuard 'client' */ TCPMSS clamp to PMTU

Chain OUTPUT (policy ACCEPT 604K packets, 132M bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 21M packets, 18G bytes)
 pkts bytes target     prot opt in     out     source               destination
Similarly with ip6tables -nvL -t mangle (I have ignored the DNSFILTER* items which appear to relate to the traffic for devices outside the default policy based routing )
Code:
Chain PREROUTING (policy ACCEPT 880K packets, 809M bytes)
 pkts bytes target     prot opt in     out     source               destination
61181   76M MARK       all      wg11   *       ::/0                 ::/0                 /* WireGuard 'client' */ MARK xset 0x1/0x7
44537   20M MARK       all      *      *       ::/0                 ::/0                 MAC 00:15:5D:XX:YY:ZZ MARK or 0x1000

Chain FORWARD (policy ACCEPT 853K packets, 799M bytes)
 pkts bytes target     prot opt in     out     source               destination
42582   20M MARK       all      *      wg11    ::/0                 ::/0                 /* WireGuard 'client' */ MARK xset 0x1/0x7
   19  1392 TCPMSS     tcp      wg11   *       ::/0                 ::/0                 tcp flags:0x06/0x02 /* WireGuard 'client' */ TCPMSS clamp to PMTU
   44  3520 TCPMSS     tcp      *      wg11    ::/0                 ::/0                 tcp flags:0x06/0x02 /* WireGuard 'client' */ TCPMSS clamp to PMTU
Code:
Chain PREROUTING (policy ACCEPT 3319K packets, 3149M bytes)
 pkts bytes target     prot opt in     out     source               destination
   11  1276 MARK       all      wg11   *       ::/0                 ::/0                 /* WireGuard 'client' */ MARK xset 0x1/0x7
91262   54M MARK       all      *      *       ::/0                 ::/0                 match-set wg11-mac src /* WireGuard 'client' */ MARK or 0x1000

Chain FORWARD (policy ACCEPT 3166K packets, 3075M bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MARK       all      *      wg11    ::/0                 ::/0                 /* WireGuard 'client' */ MARK xset 0x1/0x7
    0     0 TCPMSS     tcp      wg11   *       ::/0                 ::/0                 tcp flags:0x06/0x02 /* WireGuard 'client' */ TCPMSS clamp to PMTU
    0     0 TCPMSS     tcp      *      wg11    ::/0                 ::/0                 tcp flags:0x06/0x02 /* WireGuard 'client' */ TCPMSS clamp to PMTU
The problem is that your policy tables 121 contains no routes to wg11, so it is effectively not used and the rules pointing devices to these tables will probably be broken.

The big question is why wgm does not add the default route to the policy tables? Perhaps a start wg11 debug could shed some light of what is happening.

Meanwhile you could add the routes manually:
Code:
ip route add default dev wg11 table 121
ip -6 route add default dev wg11 table 121
and see if your system starts behaving.
 
Debug is
Code:
start wg11 debug

        Requesting WireGuard VPN Peer start (wg11)

        WireGuard-clientwg11: Initialising WireGuard VPN 'client' Peer (wg11) in Policy Mode to nl1.wg.azirevpn.net:<port> (# N/A) DNS=10.50.60.1,fe80::aa5e:45ff:xxxx:yyyy
[#] iptables -t nat -N WGDNS1
[#] ip6tables -t nat -N WGDNS1
[#] ip link add dev wg11 type wireguard
[#] wg setconf wg11 /tmp/wg11.12485 #(/opt/etc/wireguard.d/wg11.conf)
[#] ip address add dev wg11 <vpn_ipv4_range>
[#] ip -6 address add dev wg11 <vpn_ipv6_range>
[#] ip link set up dev wg11
[#] ip -6 link set up dev wg11
[#] ifconfig wg11 mtu 1420
[#] ifconfig wg11 txqueuelen 1000
[#] ip route add <vpn_ip4> via WAN
[#] ip rule add from 0/0 fwmark 0x1000/0x1000 table 121 prio 9991
[#] ip -6 rule add from ::/0 fwmark 0x1000/0x1000 table 121 prio 9991
[#] iptables -t mangle -A PREROUTING -m set --match-set wg11-mac src -j MARK --set-mark 0x1000/0x1000 -m comment --comment WireGuard 'client'
[#] ip6tables -t mangle -A PREROUTING -m set --match-set wg11-mac src -j MARK --set-mark 0x1000/0x1000 -m comment --comment WireGuard 'client'
[#] ip route add table 121 10.0.0.0/8 proto kernel scope link src 10.50.60.10 dev br0
[#] ip route add table 121 10.50.60.0/24 proto kernel scope link src 10.50.60.1 dev br0
[#] ip -6 route add table 121 <LAN IPV6 Range> proto kernel metric 256 pref medium dev br0
[#] ip -6 route add table 121 fd36:7ef1:2add:aa88:100::1 proto kernel metric 256 pref medium dev br0
[#] ip -6 route add table 121 fe80::/64 proto kernel metric 256 pref medium dev br0
[+] wg11-up.sh
[#] iptables -t mangle -I FORWARD -o wg11 -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu -m comment --comment WireGuard 'client'
[#] iptables -t mangle -I FORWARD -i wg11 -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu -m comment --comment WireGuard 'client'
[#] iptables -t mangle -I FORWARD -o wg11 -j MARK --set-xmark 0x01/0x7 -m comment --comment WireGuard 'client'
[#] iptables -t mangle -I PREROUTING -i wg11 -j MARK --set-xmark 0x01/0x7 -m comment --comment WireGuard 'client'
[#] iptables -t nat -I POSTROUTING -s 10.50.60.1/24 -o wg11 -j MASQUERADE -m comment --comment WireGuard 'client'
[#] iptables -t nat -I PREROUTING -p tcp -m tcp --dport 53 -j WGDNS1 -m comment --comment WireGuard 'client1 DNS'
[#] iptables -t nat -I PREROUTING -p udp -m udp --dport 53 -j WGDNS1 -m comment --comment WireGuard 'client1 DNS'
[#] ip6tables -t mangle -I FORWARD -o wg11 -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu -m comment --comment WireGuard 'client'
[#] ip6tables -t mangle -I FORWARD -i wg11 -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu -m comment --comment WireGuard 'client'
[#] ip6tables -t mangle -I FORWARD -o wg11 -j MARK --set-xmark 0x01/0x7 -m comment --comment WireGuard 'client'
[#] ip6tables -t mangle -I PREROUTING -i wg11 -j MARK --set-xmark 0x01/0x7 -m comment --comment WireGuard 'client'
[#] ip6tables -t nat -I POSTROUTING -s <WAN IPv6 Address>/64 -o wg11 -j MASQUERADE -m comment --comment WireGuard 'client'
[#] ip6tables -t nat -I PREROUTING -p tcp -m tcp --dport 53 -j WGDNS1 -m comment --comment WireGuard 'client1 DNS'
[#] ip6tables -t nat -I PREROUTING -p udp -m udp --dport 53 -j WGDNS1 -m comment --comment WireGuard 'client1 DNS'
        WireGuard-clientwg11: Initialisation complete.
 
and adding the rules
Code:
ip route add default dev wg11 table 121
ip -6 route add default dev wg11 table 121
manually puts everything back as it should be, so the question is what did I do (fail to do) so that they are missing.
 
Debug is
Code:
start wg11 debug

        Requesting WireGuard VPN Peer start (wg11)

        WireGuard-clientwg11: Initialising WireGuard VPN 'client' Peer (wg11) in Policy Mode to nl1.wg.azirevpn.net:<port> (# N/A) DNS=10.50.60.1,fe80::aa5e:45ff:xxxx:yyyy
[#] iptables -t nat -N WGDNS1
[#] ip6tables -t nat -N WGDNS1
[#] ip link add dev wg11 type wireguard
[#] wg setconf wg11 /tmp/wg11.12485 #(/opt/etc/wireguard.d/wg11.conf)
[#] ip address add dev wg11 <vpn_ipv4_range>
[#] ip -6 address add dev wg11 <vpn_ipv6_range>
[#] ip link set up dev wg11
[#] ip -6 link set up dev wg11
[#] ifconfig wg11 mtu 1420
[#] ifconfig wg11 txqueuelen 1000
[#] ip route add <vpn_ip4> via WAN
[#] ip rule add from 0/0 fwmark 0x1000/0x1000 table 121 prio 9991
[#] ip -6 rule add from ::/0 fwmark 0x1000/0x1000 table 121 prio 9991
[#] iptables -t mangle -A PREROUTING -m set --match-set wg11-mac src -j MARK --set-mark 0x1000/0x1000 -m comment --comment WireGuard 'client'
[#] ip6tables -t mangle -A PREROUTING -m set --match-set wg11-mac src -j MARK --set-mark 0x1000/0x1000 -m comment --comment WireGuard 'client'
[#] ip route add table 121 10.0.0.0/8 proto kernel scope link src 10.50.60.10 dev br0
[#] ip route add table 121 10.50.60.0/24 proto kernel scope link src 10.50.60.1 dev br0
[#] ip -6 route add table 121 <LAN IPV6 Range> proto kernel metric 256 pref medium dev br0
[#] ip -6 route add table 121 fd36:7ef1:2add:aa88:100::1 proto kernel metric 256 pref medium dev br0
[#] ip -6 route add table 121 fe80::/64 proto kernel metric 256 pref medium dev br0
[+] wg11-up.sh
[#] iptables -t mangle -I FORWARD -o wg11 -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu -m comment --comment WireGuard 'client'
[#] iptables -t mangle -I FORWARD -i wg11 -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu -m comment --comment WireGuard 'client'
[#] iptables -t mangle -I FORWARD -o wg11 -j MARK --set-xmark 0x01/0x7 -m comment --comment WireGuard 'client'
[#] iptables -t mangle -I PREROUTING -i wg11 -j MARK --set-xmark 0x01/0x7 -m comment --comment WireGuard 'client'
[#] iptables -t nat -I POSTROUTING -s 10.50.60.1/24 -o wg11 -j MASQUERADE -m comment --comment WireGuard 'client'
[#] iptables -t nat -I PREROUTING -p tcp -m tcp --dport 53 -j WGDNS1 -m comment --comment WireGuard 'client1 DNS'
[#] iptables -t nat -I PREROUTING -p udp -m udp --dport 53 -j WGDNS1 -m comment --comment WireGuard 'client1 DNS'
[#] ip6tables -t mangle -I FORWARD -o wg11 -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu -m comment --comment WireGuard 'client'
[#] ip6tables -t mangle -I FORWARD -i wg11 -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu -m comment --comment WireGuard 'client'
[#] ip6tables -t mangle -I FORWARD -o wg11 -j MARK --set-xmark 0x01/0x7 -m comment --comment WireGuard 'client'
[#] ip6tables -t mangle -I PREROUTING -i wg11 -j MARK --set-xmark 0x01/0x7 -m comment --comment WireGuard 'client'
[#] ip6tables -t nat -I POSTROUTING -s <WAN IPv6 Address>/64 -o wg11 -j MASQUERADE -m comment --comment WireGuard 'client'
[#] ip6tables -t nat -I PREROUTING -p tcp -m tcp --dport 53 -j WGDNS1 -m comment --comment WireGuard 'client1 DNS'
[#] ip6tables -t nat -I PREROUTING -p udp -m udp --dport 53 -j WGDNS1 -m comment --comment WireGuard 'client1 DNS'
        WireGuard-clientwg11: Initialisation complete.
Great, hopefully @Martineau could figure it out. Wierd, did a search through latest wg_client.sh and could not find were the routes to wg11 are added to table 121, only commented parts... corrupted upload??? Or just me sucking at reading code.

Don't think you did anything wrong but we'll see.
 
Last edited:
Great, hopefully @Martineau could figure it out. Wierd, did a search through latest wg_client.sh and could not find were the routes to wg11 are added to table 121, only commented parts... corrupted upload??? Or just me sucking at reading code.

Don't think you did anything wrong but we'll see.
After more testing I added the ipv6 alias into wan-event, preceded by sleep 40 - in my setup wan-event first occurs in the syslog after 8 seconds and /opt/etc/61unbound after 50 seconds.

The issue in running it too early is that not only is the command lost, but the ipv6 default route
Code:
ip -6 route show
default via fe80::4255:82ff:xxxx:xxxx dev eth0 proto ra metric 1024 expires 3645sec hoplimit 64 pref medium is created
may not be created, meaning that for all clients not on the VPN, IPV6 is down - I then need to disable and re-enable ipv6 from the GUI to get it back.
 
After more testing I added the ipv6 alias into wan-event, preceded by sleep 40 - in my setup wan-event first occurs in the syslog after 8 seconds and /opt/etc/61unbound after 50 seconds.

The issue in running it too early is that not only is the command lost, but the ipv6 default route
Code:
ip -6 route show
default via fe80::4255:82ff:xxxx:xxxx dev eth0 proto ra metric 1024 expires 3645sec hoplimit 64 pref medium is created
may not be created, meaning that for all clients not on the VPN, IPV6 is down - I then need to disable and re-enable ipv6 from the GUI to get it back.
Aha, interesting find... but the timing could change so you might want more margin than 2 sec?

Did you get the ipset to work in wgm?
 
Aha, interesting find... but the timing could change so you might want more margin than 2 sec?

Did you get the ipset to work in wgm?
I am running with ipset in wgm, the only issue seems to be the missing routes which are (for now) in wg11-route-up.sh

With regard to the alias timing the challenge is finding a gap that is big enough for the command to stick and short enough for it to be before unbound starts (all while avoiding being lynched by the family for constantly rebooting). It was for that reason that I initially added it as the first line after #!/bin/sh in 61unbound and then decided, as that may be changed by updates, to create 60wg11-mac and put it there, as this will always run just before 61unbound (if I have understood the ordering in init.d).
 
I am running with ipset in wgm, the only issue seems to be the missing routes which are (for now) in wg11-route-up.sh

With regard to the alias timing the challenge is finding a gap that is big enough for the command to stick and short enough for it to be before unbound starts (all while avoiding being lynched by the family for constantly rebooting). It was for that reason that I initially added it as the first line after #!/bin/sh in 61unbound and then decided, as that may be changed by updates, to create 60wg11-mac and put it there, as this will always run just before 61unbound (if I have understood the ordering in init.d).
Yea, I got the same problem with the family. I'm bannad to tinkering during day time so I have to do it at night, when everyone at sleep and if something goes wrong... well its a long night and not really productive the next day. Now I'm happily acouple of wgm releases behind...

Maybe you could let the event control it:
wan-event

Called after an event related to the WAN interface occurs. The script will receive two parameters:

$1 will contain the WAN unit (0 for Primary, or 1 for Secondary).
$2 will contain the event type, from the following list (the list can vary between firmware versions):

init
connecting
connected
disconnected
stopped
disabled
stopping

The old wan-start script would occur on a "connected" event. Note that after a connected event occurs, the Internet connection might not yet be fully functional. You should probably add a slight pause before you try to initiate anything requiring Internet access, or write your own code to wait until the connection becomes functional.

So something like:
Code:
if [ "$2" = "connected" ]; then
ifconfig br0:1 192.168.3.1 netmask 255.255.255.255 
ip -6 address add dev br0 fd36:7ef1:2add:aa88:100::1/128
fi
 
Last edited:
Yea, I got the same problem with the family. I'm bannad to tinkering during day time so I have to do it at night, when everyone at sleep and if something goes wrong... well its a long night and not really productive the next day. Now I'm happily acouple of wgm releases behind...

Maybe you could let the event control it:


So something like:
Code:
if [ "$1" = "0" ] && [ "$2" = "connected" ]; then
ifconfig br0:1 192.168.3.1 netmask 255.255.255.255
ip -6 address add dev br0 fd36:7ef1:2add:aa88:100::1/128
fi
I will give this a go on Friday (too late for reboots today and out tomorrow) though I hope that these stages are not just based around IPv4 connectivity. As previously noted this looks to happen ahead of full ipv6 functionality. I will report back when I have tested.
 
Great, hopefully @Martineau could figure it out. Wierd, did a search through latest wg_client.sh and could not find were the routes to wg11 are added to table 121, only commented parts... corrupted upload??? Or just me sucking at reading code.

Don't think you did anything wrong but we'll see.
Whoops! :eek:

I decided that the existence of either 0.0.0.0/0 or ::0/0 in the AllowedIPs = directive (even though invariably they are always present) shouldn't dictate if the default route is added to the Selective Routing table, but as @ZebMcKayhan states, the relevant code block in the script was erroneously completely commented out rather than just the checking code.

1650529983153.png


Hopefully I've corrected the regression in the latest wireguard_manager Beta v4.16.bC ( Stable v4.15 is unaffected)

Code:
e  = Exit Script [?]

E:Option ==> uf dev
 
Last edited:
I will give this a go on Friday (too late for reboots today and out tomorrow) though I hope that these stages are not just based around IPv4 connectivity. As previously noted this looks to happen ahead of full ipv6 functionality. I will report back when I have tested.
Had some time this morning, checked out @Martineau latest version (working and I have removed the extra routing from wg11-route-up.sh) and also tried the script in wan-event, unfortunately it did not work. Checking ifconfig br0 & br0:1 I could see the ipv4 alias had loaded successfully but the ipv6 alias had not and (I assume as a result) the IPV6 WAN default route (assume to the link-local address of the upstream ISP device)
Code:
default via fe80::4255:82ff:xxxx:xxxx dev eth0 proto ra metric 1024 expires 3645sec hoplimit 64 pref medium is created
was not created
Also whenever this (ipv6 alias not added) occurs I also see this in the syslog
Code:
lldpd[xxxx]: removal request for address of <ipv6_alias>%nn, but no knowledge of it
it does not happen where the alias is successfully mounted.
 
hecking ifconfig br0 & br0:1 I could see the ipv4 alias had loaded successfully but the ipv6 alias had not and (I assume as a result) the IPV6 WAN default route (assume to the link-local address of the upstream ISP device)
looks like the ipv6 is interfering with the assignment of br0 ipv6 from your ISP... guess the router dont assign ipv6 if its already assigned.... maybee giving the ipv6 to br0 is just a bad idea?

would it be possible to add ipv6 to your alias interface??? according to "internet experts" there is no ipv6 alias, but worth a test, like:
Code:
ifconfig br0:1 192.168.3.1 netmask 255.255.255.255
ip -6 address add dev br0:1 fd36:7ef1:2add:aa88:100::1/128
There is a risk that the ipv6 will be added to br0 anyway if the "internet experts" are correct.

or perhaps add it to one of the interfaces inside br0? like:
Code:
ifconfig br0:1 192.168.3.1 netmask 255.255.255.255
ip -6 address add dev eth3 fd36:7ef1:2add:aa88:100::1/128
(dont know if any firewall rules would be needed for this to work though as source interface might now be eth3 instead of br0 for communication from Unbound, but it will only be used in output and input chain and not from anything connected to eth3... would be interesting to test.

or some other interface maybee?
 
Last edited:
looks like the ipv6 is interfering with the assignment of br0 ipv6 from your ISP... guess the router dont assign ipv6 if its already assigned.... maybee giving the ipv6 to br0 is just a bad idea?

would it be possible to add ipv6 to your alias interface??? according to "internet experts" there is no ipv6 alias, but worth a test, like:
Code:
ifconfig br0:1 192.168.3.1 netmask 255.255.255.255
ip -6 address add dev br0:1 fd36:7ef1:2add:aa88:100::1/128

or perhaps add it to one of the interfaces inside br0? like:
Code:
ifconfig br0:1 192.168.3.1 netmask 255.255.255.255
ip -6 address add dev eth3 fd36:7ef1:2add:aa88:100::1/128
(dont know if any firewall rules would be needed for this to work though as source interface might now be eth3 instead of br0 for communication from Unbound, but it will only be used in output and input chain and not from anything connected to eth3... would be interesting to test.

or some other interface maybee?
I don't think the issue is if you can add an ipv6 alias, rather when. Once the router is up a running (i.e. by the time the GUI is available) then adding an alias (or as many as you like) will work, the issue appears to be how much earlier this can happen.

As I see it (and I am happy to be corrected) the ipv6 functionality in Asuswrt (and thence Asuswrt-Merlin) is an adjunct to and secondary to IPv4 functionality and as such it currently takes considerably longer for it to 'complete' and until it is 'complete' some programs do not work properly with IPv6. In this case while adding an IPv4 alias (whether via ip or ifconfig) work 'out-of-the-box' after around 8 seconds (that is when wan-event runs on my boot according to the syslog) but the extended functionality for IPv6 takes very much longer - when I was testing they did not work at 25 seconds in but did at 40, I have not checked in between as I need to minimise the number of reboots, as each time it fails not only does the alias not take, but it (appears to) prevent the loading of the default 'WAN (eth0) to ISP route', so I then have to restore this before retrying.

Which is why I have settled on putting the entry into a script in /opt/etc/init.d As I want to run unbound and I know that
  1. the aliases must be up before unbound starts
  2. unbound is started from /opt/etc/init.d/61unbound
  3. scripts in /opt/etc/init.d run in numeric order
so adding a new script to /opt/etc/init.d called 60<anything you like> will always run before 61unbound tries to start and the alias will always be ready.

n.b I had already found that adding the alias as ifconfig br0:n would add it to br0 not br0:n
 
I don't think the issue is if you can add an ipv6 alias, rather when. Once the router is up a running (i.e. by the time the GUI is available) then adding an alias (or as many as you like) will work, the issue appears to be how much earlier this can happen.
I'm just concerned about you have to find the small time-slot between WAN ipv6 completed its task and Unbound starts... it sound like the outcome of this could change over time. sure, you guarantee that your alias is added just before Unbound. but what if the ISP ipv6 handshake is delayed somewhat, then you will have problems again.
but its your call.

My view on why this is what it is, is quite different from yours. For ipv4 wan is assigned ip from isp. Br0 is applied by router as a private address. For ipv6 br0 is wierdly assigned assumed not via dhcp6 to isp but maybee just assumed and grabbed from the prefix. also different, dnsmasq hands out ipv4 ips based on hard-coded set values in GUI but for ipv6 it is based on br0 interface address which is set from ISP in some way. the ipv4 and ipv6 system is really different so it may not be a too big suprise that what is working perfectly for ipv4 might not be soo good for ipv6.

I think, assigning an ipv6 to br0 before router assigns it creates some conflict in the router, based on above. the ipv6 grabbing from the prefix might be messed up, the number of things that needs to happen for the default route to be placed correctly might not be fulfilled. it is probably a situation that the router software was not prepared for.

Also dnsmasq gives ipv6 based on br0 to clients and if multiple addresses exist it may give out many prefixes... hence my comment that maybe seek another interface than br0 to put this on, since ipv6 system setup depends too heavily on br0 interface.
 
Last edited:
I'm just concerned about you have to find the small time-slot between WAN ipv6 completed its task and Unbound starts... it sound like the outcome of this could change over time. sure, you guarantee that your alias is added just before Unbound. but what if the ISP ipv6 handshake is delayed somewhat, then you will have problems again.
but its your call.

My view on why this is what it is, is quite different from yours. For ipv4 wan is assigned ip from isp. Br0 is applied by router as a private address. For ipv6 br0 is wierdly assigned assumed not via dhcp6 to isp but maybee just assumed and grabbed from the prefix. also different, dnsmasq hands out ipv4 ips based on hard-coded set values in GUI but for ipv6 it is based on br0 interface address which is set from ISP in some way. the ipv4 and ipv6 system is really different so it may not be a too big suprise that what is working perfectly for ipv4 might not be soo good for ipv6.

I think, assigning an ipv6 to br0 before router assigns it creates some conflict in the router, based on above. the ipv6 grabbing from the prefix might be messed up, the number of things that needs to happen for the default route to be placed correctly might not be fulfilled. it is probably a situation that the router software was not prepared for.

Also dnsmasq gives ipv6 based on br0 to clients and if multiple addresses exist it may give out many prefixes... hence my comment that maybe seek another interface than br0 to put this on, since ipv6 system setup depends too heavily on br0 interface.
Your reasoning may well be correct, I am still at the 'I know that I don't know' stage. As you have noted the WAN IPV4 is provided (typically) as a x.x.x.x/32 address (on eth0) which does not need to be replicated on br0 as it will be NAT'd to the router defined subnet(s), whereas the WAN IPV6 is provided as <ipv6>/56 or /64 and does need to be replicated across the default bridge. However I would need to find the time to do a lot more reading and testing before I have any grasp on why this translates into the time delay problem

For now the good news is that using an eth device can work. I have tested (only lightly) with each of
#ip -6 address add dev eth1 fd36:7ef1:2add:aa88:100::1/128
#ip -6 address add dev eth3 fd36:7ef1:2add:aa88:100::1/128
#ip -6 address add dev eth5 fd36:7ef1:2add:aa88:100::1/128
#ip -6 address add dev eth7 fd36:7ef1:2add:aa88:100::1/128
where eth1 (LAN4) connects to my secondary router (for guests and testing), eth3 (LAN 2) connects to nothing, eth5 also connects to nothing but is the bridge to LAN 5-8 and eth7 is the 5.0 GHz WLAN. The relevant factor appears to be whether the device has state UP (works) or DOWN (does not), but I will need to run this a few more times to verify.
Code:
ip addr show | grep eth
11: eth1: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
    link/ether <router MAC> brd ff:ff:ff:ff:ff:ff
12: eth2: <NO-CARRIER,BROADCAST,MULTICAST,ALLMULTI,UP> mtu 1500 qdisc pfifo_fast master br0 state DOWN group default qlen 1000
    link/ether <router MAC> brd ff:ff:ff:ff:ff:ff
13: eth3: <NO-CARRIER,BROADCAST,MULTICAST,ALLMULTI,UP> mtu 1500 qdisc pfifo_fast master br0 state DOWN group default qlen 1000
    link/ether <router MAC> brd ff:ff:ff:ff:ff:ff
14: eth4: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
    link/ether <router MAC> brd ff:ff:ff:ff:ff:ff
15: eth5: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
    link/ether <router MAC> brd ff:ff:ff:ff:ff:ff
    link/ether <router MAC> brd ff:ff:ff:ff:ff:ff
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
20: eth6: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
    link/ether <router MAC> brd ff:ff:ff:ff:ff:ff
21: eth7: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
    link/ether <5.0 GHz MAC> brd ff:ff:ff:ff:ff:ff
    link/ether <router MAC> brd ff:ff:ff:ff:ff:ff
It may also be relevant that I have no VLANs so everything LANside is bridged through br0.

As the above seems to do what is needed, the next issue is how this affects any 'road-warrior' devices and what I should expect.

If I just run the wg21 server, with wg11 down, then (using ipleak.net and browserleaks.com)
Device IP
IPv4 = WAN IPv4 address
IPv6 = from WanWg21_PrefIp range e.g. #2001:1111:2222:3333:100::2

WebRTC
IPv4 = Local: from WG21 IPv4 subnet. Public: Device IPv4
IPv6 = from WG21 IPv6 subnet + Device IPv6

DNS
WAN IPv4 address

If I have both wg21 and wg11 up then the only change is that the DNS changes to the VPN DNS and if I use passthru (peer wg21 passthru add wg11 sam21a) it makes no difference at all and so I have some questions / observations

Adding wg21 to wg11 or vice-versa (without passthru) generates error messages
Code:
[#] ip link add dev wg21 type wireguard
[#] ip -6 link add dev wg21 type wireguard
[#] wg set wg21 fwmark 11501
[#] wg setconf wg21 /tmp/wg21.11969 #(/opt/etc/wireguard.d/wg21.conf)
[#] ip link set up dev wg21
[#] ip -6 link set up dev wg21
[#] ip address add dev wg21 10.50.1.1/24
ERR: bdmf_attrelem_add_as_num#4276: system: status:No resources. attribute:ipv4_host_address_table  index:0 value:171049217
Code:
[#] iptables -t nat -N WGDNS1
[#] ip6tables -t nat -N WGDNS1
[#] ip link add dev wg11 type wireguard
[#] wg setconf wg11 /tmp/wg11.14312 #(/opt/etc/wireguard.d/wg11.conf)
[#] ip address add dev wg11 <vpn_ipv4_range>
ERR: bdmf_attrelem_add_as_num#4276: system: status:No resources. attribute:ipv4_host_address_table  index:0 value:167775100
if I add passthru
Code:
[#] ip link add dev wg21 type wireguard
[#] ip -6 link add dev wg21 type wireguard
[#] wg set wg21 fwmark 11501
[#] wg setconf wg21 /tmp/wg21.29147 #(/opt/etc/wireguard.d/wg21.conf)
[#] ip link set up dev wg21
[#] ip -6 link set up dev wg21
[#] ip address add dev wg21 10.50.1.1/24
[#] ip -6 address add dev wg21 aa36:7ef1:2add:aa88:100::1/120
[#] ifconfig wg21 mtu 1420
[#] ifconfig wg21 txqueuelen 1000
[#] ip route add default dev wg21 table 210
[#] ip rule add fwmark 0x000d2 table 210 prio 9810
[#] ip -6 route add default dev wg21 table 210
[#] ip -6 rule add fwmark 0x000d2 table 210 prio 9810
Error: any valid prefix is expected rather than "10.50.1.1/24,aa36:7ef1:2add:aa88:100::1/120".
Error: inet6 prefix is expected rather than "10.50.1.1/24,aa36:7ef1:2add:aa88:100::1/120".
Code:
[#] iptables -t nat -N WGDNS1
[#] ip6tables -t nat -N WGDNS1
[#] ip link add dev wg11 type wireguard
[#] wg setconf wg11 /tmp/wg11.32267 #(/opt/etc/wireguard.d/wg11.conf)
[#] ip address add dev wg11 <vpn_ipv4_range>
ERR: bdmf_attrelem_add_as_num#4276: system: status:No resources. attribute:ipv4_host_address_table  index:0 value:167775100
[#] ip -6 address add dev wg11 <vpn_ipv6_range>
[#] ip link set up dev wg11
[#] ip -6 link set up dev wg11
[#] ifconfig wg11 mtu 1420
[#] ifconfig wg11 txqueuelen 1000
[+] wg11-route-up.sh
[#] ip route add <VPN IP4> via <WAN IP4>
[#] ip rule add from 0/0 fwmark 0x1000/0x1000 table 121 prio 9991
[#] ip -6 rule add from ::/0 fwmark 0x1000/0x1000 table 121 prio 9991
[#] iptables -t mangle -A PREROUTING -m set --match-set wg11-mac src -j MARK --set-mark 0x1000/0x1000 -m comment --comment WireGuard 'client'
[#] ip6tables -t mangle -A PREROUTING -m set --match-set wg11-mac src -j MARK --set-mark 0x1000/0x1000 -m comment --comment WireGuard 'client'
[#] ip route add 0/1 dev wg11 table 121
[#] ip route add 128/1 dev wg11 table 121
[#] ip -6 route add 0::/1 dev wg11 table 121
[#] ip -6 route add 8000::/1 dev wg11 table 121
[#] ip route add table 121 10.0.0.0/8 proto kernel scope link src 10.50.60.10 dev br0
[#] ip route add table 121 10.50.60.0/24 proto kernel scope link src 10.50.60.1 dev br0
[#] ip -6 route add table 121 <ISP_IPv6_Range> proto kernel metric 256 pref medium dev br0
[#] ip -6 route add table 121 fe80::/64 proto kernel metric 256 pref medium dev br0
Error: any valid prefix is expected rather than "10.50.1.3/32,aa36:7ef1:2add:aa88:100::3/128".
iptables v1.4.21: invalid mask `128' specified
Try `iptables -h' or 'iptables --help' for more information.
Failed to send flush request: No such process
continued......
 

Support SNBForums w/ Amazon

If you'd like to support SNBForums, just use this link and buy anything on Amazon. Thanks!

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top