WireGuard Client & WireGuard Server/InstantGuard at the same time?

[ZebMcKayhan]

Is there some underlying problem with Android not working in this kind of set up then? :S

I'm just saying that you should not exclude this possibility. But it could be app dependant as Android only implemented mDNS natively quite recently (2021 I think). Apps that needs to do this on older phones would have to implement mDNS themselves. Then what constraints put from the kernel I dont know. But some apps (like "service browser") could just refuse to work when not on wifi, because that's how it was designed. Another app may not have this restriction. Whats to say your app is happy with working over VPN?

My guess is router level because no device other than Windows can ping eachothers .local address no matter what setting I mess with.

If ping .local fails does not need to mean problems. Systems that does not do mDNS natively will likely fail this (like my Samsung S9). That does not mean that apps that implements this themselves will not work, but it will only work given the constraints of the app.

I think you need to get deeper and sniff packets on each side to figure out where it goes wrong. Tcpdump could be installed on router to do this.

 

[privacyguy123]

I'm just saying that you should not exclude this possibility. But it could be app dependant as Android only implemented mDNS natively quite recently (2021 I think). Apps that needs to do this on older phones would have to implement mDNS themselves. Then what constraints put from the kernel I dont know. But some apps (like "service browser") could just refuse to work when not on wifi, because that's how it was designed. Another app may not have this restriction. Whats to say your app is happy with working over VPN?

True yes, good point. I am on a S20 FE which is newish, but not bleeding edge. I plan on grabbing the new Pixel when it's out ... maybe we can see if it's any different.

If ping .local fails does not need to mean problems. Systems that does not do mDNS natively will likely fail this (like my Samsung S9). That does not mean that apps that implements this themselves will not work, but it will only work given the constraints of the app.

I think you need to get deeper and sniff packets on each side to figure out where it goes wrong. Tcpdump could be installed on router to do this.

tcpdump is all well and good if you know what you're looking at/for ... unfortunately I don't. Ping .local outright refuses to work, so I guess the firmware is missing some packages or libs.

 

[ZebMcKayhan]

tcpdump is all well and good if you know what you're looking at/for

Yea, it will be a process of going through packets one after one to figure out what each means a.s.o
But I'm thinking more like

tcpdump -i wgs1 -s0 -vv net 224.0.0.0/4

And just se if any multicast pops out of the interface from your Android app.
And see if your router seems to send anything.
Then redo on Android side to se if whatever the router is sending pops out on this side and/or whether or not the app send something this way... maybe you dont need to understand every content?

 

[privacyguy123]

tcpdump -i wgs1 -s0 -vv net 224.0.0.0/4

Tidal outputs this after a while, I assume it's looking for the Tidal Connect receiver which is not at that address.

19:18:24.787993 IP (tos 0x0, ttl 255, id 26371, offset 0, flags [DF], proto UDP (17), length 207)
10.6.0.2.40566 > 239.255.255.250.upnp: [udp sum ok] UDP, length 179
19:18:31.591738 IP (tos 0x0, ttl 255, id 26626, offset 0, flags [DF], proto UDP (17), length 207)
10.6.0.2.51215 > 239.255.255.250.upnp: [udp sum ok] UDP, length 179

 

[privacyguy123]

As far as I can tell mcpd is being used to route VPN stuff because when I kill it to use 3rd party stuff the WG Server fails to connect at all ...

Which means that stuff like this can't coexist on this router at all as far as I can see

Making DLNA through site-to-site VPN work

There are articles about how does multicast routing across subnets work, but none of them was complete for my setup. Here is my solution.

an0n-r0.medium.com

Can you specify multiple interfaces in this config file perhaps?

##### IGMP configuration #####
igmp-default-version 3
igmp-query-interval 20
igmp-query-response-interval 100
igmp-last-member-query-interval 10
igmp-robustness-value 2
igmp-max-groups 25
igmp-max-sources 25
igmp-max-members 25
igmp-fast-leave 1
igmp-admission-required 0
igmp-admission-bridging-filter 0
igmp-proxy-interfaces eth4
igmp-snooping-interfaces br0
igmp-mcast-interfaces eth4

Someone on reddit said this writeup works but it explained in ipv6 terms so I don't know how to test it with my ipv4 setup.

VXLAN & Linux

VXLAN is an overlay network for L2 traffic over an existing IP network. Let's explore how to configure it on Linux.

vincent.bernat.ch

 

[ZebMcKayhan]

Tidal outputs this after a while, I assume it's looking for the Tidal Connect receiver which is not at that address.

Is this on the router?? It would mean that you app is sending multicast over wg vpn. So thats working!

But this is upnp/SSDP and not mDNS https://en.m.wikipedia.org/wiki/Simple_Service_Discovery_Protocol
So, obviously there is more than mDNS needed to get this going.
But the reply to this packet would be a unicast if the little I read up on it is true, so your Tidal device (or Avahi reflector) may be replying to this, but you dont see it since you are only looking at multicast...

 

[privacyguy123]

Is this on the router?? It would mean that you app is sending multicast over wg vpn. So thats working!

Yes !?

But this is upnp/SSDP and not mDNS https://en.m.wikipedia.org/wiki/Simple_Service_Discovery_Protocol
So, obviously there is more than mDNS needed to get this going.
But the reply to this packet would be a unicast if the little I read up on it is true, so your Tidal device (or Avahi reflector) may be replying to this, but you dont see it since you are only looking at multicast...

Whoosh

I'm reading more and more saying "use VXLAN" - perhaps you could help me decode this?

 

[ZebMcKayhan]

I'm reading more and more saying "use VXLAN" - perhaps you could help me decode this?

It would take some time for me to grasp. But it's seems to involve using an interface type of vxlan, which has a tag, just like vlan.

Good news is that vxlan interface seems supported on our routers:

admin@RT-AX86U_Pro:/tmp/home/root# ip link add vxlan100 type
 vxlan id 100 dstport 4789
admin@RT-AX86U_Pro:/tmp/home/root# ifconfig vxlan100
vxlan100  Link encap:Ethernet  HWaddr CA:yy:xx:aa:F9:7B
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

But adding vlans and bridges are type of stuff that usually breaks things on our routers...

 

[privacyguy123]

It would take some time for me to grasp. But it's seems to involve using an interface type of vxlan, which has a tag, just like vlan.

Good news is that vxlan interface seems supported on our routers:

admin@RT-AX86U_Pro:/tmp/home/root# ip link add vxlan100 type
 vxlan id 100 dstport 4789
admin@RT-AX86U_Pro:/tmp/home/root# ifconfig vxlan100
vxlan100  Link encap:Ethernet  HWaddr CA:4B:DD:01:F9:7B
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

But adding vlans and bridges are type of stuff that usually breaks things on our routers...

I'd appreciate any guidance!

If I break it I just reset and give up at this point ...

Maybe some relevant stuff:

VXLAN & Linux

VXLAN is an overlay network for L2 traffic over an existing IP network. Let's explore how to configure it on Linux.

vincent.bernat.ch

GitHub - m13253/VxWireguard-Generator: Utility to generate VXLAN over Wireguard mesh SD-WAN configuration

Utility to generate VXLAN over Wireguard mesh SD-WAN configuration - GitHub - m13253/VxWireguard-Generator: Utility to generate VXLAN over Wireguard mesh SD-WAN configuration

github.com

VXLAN over Wireguard with VLAN-VNI mapping

I am currently attempting to setup a L2 bridge between two sites using VXLAN to provide the L2 connectivity and Wireguard as transport/L3, eventually with plans to expand it multiple sites. I've

networkengineering.stackexchange.com

 

[privacyguy123]

Got a little further with smcroute ... I had glossed over the fact it can run with multiple multicast tables because our kernel has support for it built in.

Here's the config suggest online (edited to suit my adapters) to get upnp through the wireguard tunnel

phyint wg0 enable
phyint eth0 enablemgroup from eth0 group 239.255.255.250
mroute from eth0 group 239.255.255.250 to wg0mgroup from wg0 group 239.255.255.250
mroute from wg0 group 239.255.255.250 to eth0

The docs suggest:

-t ID Set multicast routing table ID. Remember to also create routing rules directing packets to the table. This example uses routing table ID 123:
ip mrule add iif eth0 lookup 123
ip mrule add oif eth0 lookup 123

I added these rules for eth4 and br0, I don't know if thats needed or not recommended.

ip mrule
0: from all iif br0 lookup 123
32764: from all oif eth4 lookup 123
32765: from all iif eth4 lookup 123
32766: from all oif br0 lookup 123
32767: from all lookup default

Now we run smcroute with table ID 123 and it doesn't complain about multicast API in use or whatever, but it still doesn't work and nothing connects lol ...

smcrouted -t 123 -n -N -l debug
smcroute[13473]: SMCRoute v2.5.6
smcroute[13473]: Found new interface lo, adding ...
smcroute[13473]: Found new interface ifb0, adding ...
smcroute[13473]: Found new interface ifb1, adding ...
smcroute[13473]: Found new interface imq0, adding ...
smcroute[13473]: Found new interface imq1, adding ...
smcroute[13473]: Found new interface imq2, adding ...
smcroute[13473]: Found new interface ip_vti0, adding ...
smcroute[13473]: Found new interface ip6_vti0, adding ...
smcroute[13473]: Found new interface sit0, adding ...
smcroute[13473]: Found new interface ip6tnl0, adding ...
smcroute[13473]: Found new interface ip6gre0, adding ...
smcroute[13473]: Found new interface archer, adding ...
smcroute[13473]: Found new interface bcmsw, adding ...
smcroute[13473]: Found new interface eth0, adding ...
smcroute[13473]: Found new interface eth1, adding ...
smcroute[13473]: Found new interface eth2, adding ...
smcroute[13473]: Found new interface eth3, adding ...
smcroute[13473]: Found new interface eth4, adding ...
smcroute[13473]: Found new interface dpsta, adding ...
smcroute[13473]: Found new interface eth5, adding ...
smcroute[13473]: Found new interface eth6, adding ...
smcroute[13473]: Found new interface br0, adding ...
smcroute[13473]: Found new interface wgs1, adding ...
smcroute[13473]: Found new interface wgc1, adding ...
smcroute[13473]: Found lo, updating ...
smcroute[13473]: Found lo:0, updating ...
smcroute[13473]: Found eth4, updating ...
smcroute[13473]: Found br0, updating ...
smcroute[13473]: Found wgs1, updating ...
smcroute[13473]: Found wgc1, updating ...
smcroute[13473]: Setting IPv4 multicast routing table id 123
smcroute[13473]: Setting IPv6 multicast routing table id 123
smcroute[13473]: Cannot set IPv6 multicast routing table id: Protocol not supported
smcroute[13473]: Make sure your kernel has CONFIG_IPV6_MROUTE_MULTIPLE_TABLES=y
smcroute[13473]: NOFILE: current 1024 max 4096
smcroute[13473]: NOFILE: set new current 4096 max 4096
smcroute[13473]: Binding IPC socket to /opt/var/run/smcroute.sock
smcroute[13473]: /opt/etc/smcroute.conf line 1: phyint br0 enable ttl-threshold 5
smcroute[13473]: Creating/updating multicast VIF for br0 TTL 5
smcroute[13473]: Map iface br0 => VIF 0 ifindex 22 flags 0x0008 TTL threshold 5
smcroute[13473]: No IPv6 multicast socket
smcroute[13473]: /opt/etc/smcroute.conf line 2: phyint wgs1 enable ttl-threshold 5
smcroute[13473]: Creating/updating multicast VIF for wgs1 TTL 5
smcroute[13473]: Map iface wgs1 => VIF 1 ifindex 24 flags 0x0008 TTL threshold 5
smcroute[13473]: No IPv6 multicast socket
smcroute[13473]: /opt/etc/smcroute.conf line 3:
smcroute[13473]: /opt/etc/smcroute.conf line 4: mgroup from br0 group 239.255.255.250
smcroute[13473]: Group socket 9 count 1 of MAX 20
smcroute[13473]: Join group (*,239.255.255.250) on ifindex 22 and socket 9 ...
smcroute[13473]: /opt/etc/smcroute.conf line 5: mroute from br0 group 239.255.255.250 to wgs1
smcroute[13473]: /opt/etc/smcroute.conf line 5: mroute: checking for input iface br0 ...
smcroute[13473]: /opt/etc/smcroute.conf line 5: mroute: input iface br0 has vif 0
smcroute[13473]: /opt/etc/smcroute.conf line 5: mroute: checking for wgs1 ...
smcroute[13473]: mroute: adding route from br0 (0.0.0.0/32,239.255.255.250/32)
smcroute[13473]: /opt/etc/smcroute.conf line 6:
smcroute[13473]: /opt/etc/smcroute.conf line 7: mgroup from wgs1 group 239.255.255.250
smcroute[13473]: Group socket 9 count 2 of MAX 20
smcroute[13473]: Join group (*,239.255.255.250) on ifindex 24 and socket 9 ...
smcroute[13473]: /opt/etc/smcroute.conf line 8: mroute from wgs1 group 239.255.255.250 to br0
smcroute[13473]: /opt/etc/smcroute.conf line 8: mroute: checking for input iface wgs1 ...
smcroute[13473]: /opt/etc/smcroute.conf line 8: mroute: input iface wgs1 has vif 1
smcroute[13473]: /opt/etc/smcroute.conf line 8: mroute: checking for br0 ...
smcroute[13473]: mroute: adding route from wgs1 (0.0.0.0/32,239.255.255.250/32)
smcroute[13473]: Parse error in /opt/etc/smcroute.conf
smcroute[13473]: Creating PID file /opt/var/run/smcroute.pid
smcroute[13473]: Ready, waiting for client request or kernel event.
smcroute[13473]: New multicast data from 192.168.50.71 to group 239.255.255.250 on br0
smcroute[13473]: Add 192.168.50.71 -> 239.255.255.250 from VIF 0
smcroute[13473]: New multicast data from 192.168.50.71 to group 239.255.255.250 on br0
smcroute[13473]: Add 192.168.50.71 -> 239.255.255.250 from VIF 0

The replies are from a SkyQ Box which is an airplay/chromecast receiver so the YouTube app should see them and it does when not connected through the WG tunnel, of course.
I read a little more and understand that WireGuard can't possibly work with this kind of traffic because it's on Layer 3 and it would need to be Layer 2. This is what using VXLAN fixes ... hopefully we can decode it into some sort of usable English though ...

 

[privacyguy123]

The newest nonsense in this saga.

With aforementioned repeaters/routers/whatever they're called I can use mdns-scan from my rooted phone via Termux connected in via WG tunnel and it will find all the devices just fine. As you mentioned before, apps like Service Browser will fail to load anything and just keep a loading circle spinning forever. YouTube times out looking for the Sky boxes. Makes exactly 0 sense to me - some apps can find the multicast/upnp/dlna/whatevercast devices and some can't.

In conclusion, seems Android is unsupported for this kind of set up ...

 

[ZebMcKayhan]

The newest nonsense in this saga.

With aforementioned repeaters/routers/whatever they're called I can use mdns-scan from my root phone via Termux connected in via WG tunnel and it will find all the devices just fine. As you mentioned before, apps like Service Browser will fail to load anything and just keep a loading circle spinning forever. YouTube times out looking for the Sky boxes. Makes exactly 0 sense to me - some apps can find the multicast/upnp/dlna/whatevercast devices and some can't.

In conclusion, seems Android is unsupported for this kind of set up ...

Yup... but it just means that we have not found all bits and pieces just yet. We started out with mDNS and found upnp commands and continuing it probably doesnt stop there.
Multicast is, as I recently learned through this, a vast protocol and its probably not possible to cover them all.

Broadcast is another form that we didnt consider.

Perhaps trying to tweak wg into doing something it was not intended to do is just a bad idea.

I have had vxlan in my mind since last discussion but it will probably mean a complete re-route of the interface data on both ends... surely doable but alot of work.

 

[privacyguy123]

Yup... but it just means that we have not found all bits and pieces just yet. We started out with mDNS and found upnp commands and continuing it probably doesnt stop there.
Multicast is, as I recently learned through this, a vast protocol and its probably not possible to cover them all.

Broadcast is another form that we didnt consider.

Perhaps trying to tweak wg into doing something it was not intended to do is just a bad idea.

I have had vxlan in my mind since last discussion but it will probably mean a complete re-route of the interface data on both ends... surely doable but alot of work.

Quite annoying as it does feel like we're 90% there ... what other things need covered to get these devices talking? Is it possible it's just some firewall/iptables thing? I find iptables impossible to understand as well lol.

 

[ZebMcKayhan]

Quite annoying as it does feel like we're 90% there ... what other things need covered to get these devices talking? Is it possible it's just some firewall/iptables thing? I find iptables impossible to understand as well lol.

Dont know, you will have to sniff around with tcpdump and see what packets are sent by the app, decode them and look for replies. Find where the packets gets lost.

It could be that the app just refuses to work over VPN interface and then I dont think there is anything to do. Poke the app developer maybe, but it will probably lead nowhere.

 

[privacyguy123]

Dont know, you will have to sniff around with tcpdump and see what packets are sent by the app, decode them and look for replies. Find where the packets gets lost.

It could be that the app just refuses to work over VPN interface and then I dont think there is anything to do. Poke the app developer maybe, but it will probably lead nowhere.

If big apps like YouTubes cast function does not work through a WG tunnel out of the box there is little hope, Google themselves designed that one right?

 

[ZebMcKayhan]

If big apps like YouTubes cast function does not work through a WG tunnel out of the box there is little hope, Google themselves designed that one right?

No idea,I don't know even how they could determine to disqualify an interface. Too little Android knowledge.
If they only look at interface tags you have already taken care of it.

 

[privacyguy123]

No idea,I don't know even how they could determine to disqualify an interface. Too little Android knowledge.
If they only look at interface tags you have already taken care of it.

Can't believe they gave me a reason to bash Android ... as far as I can see Mac/IOS can make use of multicast from tunnels. I see them doing it here on a ZeroTier presentation video which I can only assume is basically just a nice fancy GUI for what we are trying to set up here manually in the terminal. Either that or I need to try installing ZeroTier on the router, I see it exists in opkg. Not today though.

 

[ZebMcKayhan]

I need to try installing ZeroTier on the router, I see it exists in opkg. Not today though.

There is a guide about this in Addons section: https://www.snbforums.com/threads/a...rotier-on-asus-ac68u-router.42648/post-363199

 

[privacyguy123]

There is a guide about this in Addons section: https://www.snbforums.com/threads/a...rotier-on-asus-ac68u-router.42648/post-363199

Unfortunately I can't see it working as it seems on some deep level multicast in a VPN tunnel is blocked in Android.

Strangely another music remote app for HQPlayer connects immediately on app open without having to scan for a device name ... and when I scan it shows no devices. Just weirdness after weirdness.

 

[privacyguy123]

Someone on Android Reddit now suggests that it does work for them but the devices won't populate if the signal strength isn't high enough ... Back down the rabbit hole we go