Maybe, but since VLC player is working fine when plugged directly, there is no problem with any type of join or heartbeats, right?
Tried everything - nothing helps.
No, carrier not listed in profiles, also I tried all LAN->IPTV settings, still stops.
Also, I tred UDP Proxy (UDPxy) on that page. It states, that udp proxy can convert multicast UDP streams to unicast HTTP streams.
"Enter the port (such as 4000) that you want to assign for the IPTV application, then use this format to enter the address in the play list to watch IPTV streams: http://[Router IP]:[Udpxy port]/[protocol]/[channel IP address]:[channel port]
(such as
http://192.168.1.1:4000/udp/239.255.1.1:5000 or
http://192.168.1.1:4000/rtp/239.255.1.1:5000)."
This also do not work at all, not for a second!
And it kills the udp://@239.255.*.*:5500 in VLC!
Unfortunately no, multicast directly connected vs. through a router is completely different.
First thing you have to keep in mind, home router manufacturers added multicast support for one particular thing - forwarding IPTV service to a set top box. Over time they added some additional features like IGMP proxy but it really is not a primary function of the router, they had to do it to gain some more market share with people who use those ISPs, that's about it.
Professional routers use PIM to forward the data and build the multicast "tree". Your ISP is likely using that on their upstream router (in a building somewhere near you), so when you plug directly in, your ISP router receives your IGMP join, then PIM takes care of everything beyond that.
Home routers like the Asus don't support PIM, instead they use something called IGMP Proxy (they call it "multicast routing" but that is a gross exaggeration) to make your ISP think that the directly connected device (your router) is the one joining, when in fact it is really sort of NAT'ing IGMP from a device behind it. In theory (and often in practice) it does work, but it can be very particular.
Now to add more confusion, the "IPTV" page is under LAN but in reality it is a mix of LAN and WAN stuff. Some of the settings affect the WAN only, some affect the LAN only, some affect both (and one setting can change what another setting affects).
I haven't used the features on the Asus but from what I can tell - if you set an "STB port" - that is telling the router to put that LAN port bridged directly into the WAN VLAN for IPTV. (I don't want to take down my internet to try it but if you wanted to you could SSH into the router and use "robocfg show" to see what VLANs are assigned where with each setting). This setup theoretically means it bypasses the router, firewall, etc and essentially directly connects the LAN port to the ISP's upstream router. I'm not positive on that but it looks that way.
Whereas if you don't select an STB port and instead select Multicast Routing (IGMP Proxy) then any LAN port should be able to join the stream, but it has to play nicely with IGMP proxy, firewall, etc, and sometimes having more than one device doing multicast will confuse it and interrupt things. Since many LAN/IOT devices use multicast for local communication, sometimes that can also confuse the router. I believe several of their routers will not let you do both an STB port and Multicast Routing at the same time, it is one or the other, because they do totally different things which conflict with each other.
Then the other settings all come into play too. "Fast Leave" depends on your ISP, it can break things, or it might be the only way to get it working. IGMP Snooping/Efficient multicast, this is mostly a LAN setting, but it can break stuff too (but not having it enabled can consume LAN bandwidth and possibly slow down your wifi). DHCP routes is if your set top box requires some special parameters to be passed to it via DHCP. Etc, etc. So everything has to be just right. If you can't find someone online that has figured out the perfect combo for your ISP, you may just have to keep toying with stuff.
First thing I'd try is downgrading to 386 code base (388 is very new) and factory reset. Then you can try setting your PC port to be an "STB" port, but be aware that it may not have full/normal internet access and if it does, it may be bypassing the firewall and not have all the features a normal port would. Then you could try disabling that and instead enabling multicast routing/igmp proxy and toying with the different settings for fast leave, routes, and snooping.
This asus page gives a bit of info on what each setting does, but not a ton
https://www.asus.com/us/support/FAQ/1011708/
When you plugged your PC directly into the ISP device, were you setting any VLANs or trunking on the PC? If not then it sounds like they aren't using VLANs for IPTV but that would be strange since you said they have several VLANs. Maybe they allow joins over both VLAN 1 (default/untagged) and the IPTV VLAN, dunno.
Then if and when you get it all working, you need to toy with the wireless settings to make things work properly there, however that is usually less fussy (though it can certainly consume a good amount of wireless bandwidth and CPU).
UDP proxy converts multicast into unicast. So if you use that, you would need to tell VLC to access the http:// stream, the udp:// stream won't be accessible anymore, it is effectively behind a NAT now. If VLC can't figure out what to do with that stream you may need to tweak some settings to tell it what you're trying to do.