Hello,
At first, read EDIT 2 (At the end of this post). There is another way to configure VLAN.
After read several posts and scripts posted here (and also the one from Chinese site and others related, this Chinese site was the most important to do everything to work) I could make the VLAN working between AC86U and AC68U (both running RMerlin firmware). I tried with a Raspberry Pi 2 running OpenWRT with success.
I have 3 wifi networks:
- Main network (2,4Ghz / 5Ghz)
- Guest network (2,4Ghz / 5Ghz)
- IoT network (2,4Ghz only and AP Isolation enabled)
Each of the networks are configured in different bridges so that I can control the communication using IPTABLES.
The AC68U is in AP mode with the same 3 networks connected to AC86U using only 1 port with tagged VLAN (in this case, the routing and any other feature, like DHCP, DNS are all disabled)
The main purpose here is to explain the configuration on AC86U side, that I'm seeing there is lack of documentation, for AC68U you can find easly how to do and for Raspberry Pi, you can do it like any other Linux system using 5 or 6 commands (ip command can do everything VERY easy).
I'm just sharing what worked for me to be a start point for those who want to try, if you decide to use, it's your responsibility. If you make any mistake, be aware I can't support you to recovery your device. And finally, its important you have some knowledge for troubleshooting.
#!/bin/sh
# With this script I'm going to use eth2 (physical port 3) to be my
# Trunk port to another router Asus RT-AC68U, so that I can split my
# Wi-Fi networks (Main Network, Guest Network and IoT Network).
# Configure file resolv.conf to make it to use the router itself as DNS Server, instead ONT. It's important if you are using DNSCrypt.
# Configura o arquivo resolv.conf para apontar apenas pra loopback, usando assim o DNSCrypt configurado
echo "nameserver 127.0.0.1" > /tmp/resolv.conf
# Remove the interface eth2 from br0 (to use it as trunk with 802.1Q Tags). This interface can't be inside any bridge.
# Remove a interface eth2 da br0 (para utilizar como Trunk) e cria as VLANs com TAG. Esta interface não pode estar em nenhum bridge
brctl delif br0 eth2
# Creating VLAN 100 and the Input / Output rules (main network)
# Criando a VLAN 100 e as regras de entrada e saída (rede principal)
vlanctl --mcast --if-create eth2 100
vlanctl --if eth2 --rx --tags 1 --filter-vid 100 0 --pop-tag --set-rxif eth2.v100 --rule-append
vlanctl --if eth2 --tx --tags 0 --filter-txif eth2.v100 --push-tag --set-vid 100 0 --rule-append
ifconfig eth2.v100 up
# Creating VLAN 200 and the Input / Output rules (guest network)
# Criando a VLAN 200 e as regras de entrada e saída (rede de convidados)
vlanctl --mcast --if-create eth2 200
vlanctl --if eth2 --rx --tags 1 --filter-vid 200 0 --pop-tag --set-rxif eth2.v200 --rule-append
vlanctl --if eth2 --tx --tags 0 --filter-txif eth2.v200 --push-tag --set-vid 200 0 --rule-append
ifconfig eth2.v200 up
# Creating VLAN 300 and the Input / Output rules (IoT network)
# Criando a VLAN 300 e as regras de entrada e saída (rede IoT)
vlanctl --mcast --if-create eth2 300
vlanctl --if eth2 --rx --tags 1 --filter-vid 300 0 --pop-tag --set-rxif eth2.v300 --rule-append
vlanctl --if eth2 --tx --tags 0 --filter-txif eth2.v300 --push-tag --set-vid 300 0 --rule-append
ifconfig eth2.v300 up
# This is a very important step, otherwithse (if keep in ONT mode), the broadcasts received in 1 VLAN will be spread to others and your DHCP will become crazy.
# Este passo é muito importante, para evitar que os broadcasts em uma VLAN sejam divulgados em outras, caso contrário haverá problemas com o DHCP.
vlanctl --if eth2 --set-if-mode-rg
# This command is not in the help page of vlanctl, but I found the tx table is ACCEPT by default, so, I changed to DROP. I think if not rule matches, then the packet will be blocked.
# Este comando não está na página de ajuda do commando vlanctl, mas percebi que a tabela de tx sempre é criada com política ACCEPT por padrão, então alterai para DROP. Acredito que se nenhuma regra bater, o pacote é bloqueado.
vlanctl --if eth2 --tx --tags 0 --default-miss-drop
# Organize the bridges, in my case, I have an additional WiFi 2,4/5Ghz for guests and only one 2,4Ghz for IoT (no need 5Ghz for IoT, at least for me).
# Arruma as bridges (br0 br1 br2), no meu caso tenho uma rede adicionao de 2,4/5Ghz para convidados e apenas uma de 2,4Ghz para IoT (não preciso de 5Ghz para IoT)
brctl stp br0 on
brctl addbr br1
brctl addbr br2
brctl delif br0 wl0.1
brctl delif br0 wl1.1
brctl delif br0 wl0.2
brctl addif br0 eth2.v100
brctl addif br1 wl0.1
brctl addif br1 wl1.1
brctl addif br1 eth2.v200
brctl addif br2 wl0.2
brctl addif br2 eth2.v300
# Define IPs for new bridges
# Define os IPs para as novas bridges
ifconfig br1 192.168.20.1 netmask 255.255.255.0 up
ifconfig br2 192.168.30.1 netmask 255.255.255.0 up
# Adjust NVRAM config
# Ajusta configs do NVRAM
nvram set lan_ifnames="eth1 eth2.v100 eth3 eth4 eth5 eth6"
nvram set br0_ifnames="eth1 eth2.v100 eth3 eth4 eth5 eth6"
nvram set br1_ifname=br1
nvram set br1_ifnames="eth2.v200 wl0.1 wl1.1"
nvram set lan1_ifname=br1
nvram set lan1_ifnames="eth2.v200 wl0.1 wl1.1"
nvram set br2_ifname=br2
nvram set br2_ifnames="eth2.v300 wl0.2"
nvram set lan2_ifname=br2
nvram set lan2_ifnames="eth2.v300 wl0.2"
nvram set wl0.2_ap_isolate="1"
nvram commit
# eapd restart (when change NVRAM, it's necessary)
# Restart eapd (autenticacao wifi, necessário quando altera a NVRAM)
killall eapd
eapd
# So, the last step, if you want to communicate between LAN interfaces (1 to 4), you must disable HW Switching (this will increase the CPU usage, but my tests showed you can reach 1Gbps with no problem).
# Para funcionar tráfego entre as interfaces físicas (1 a 4), desativar o HW Switching, isto aumentará o uso de CPU, mas de acordo com os meus testes, você consegue atingir 1Gbps sem problemas.
ethswctl -c hw-switching -o disable
EDIT: this last command is necessary, it will increase the CPU usage, but if you keep the hw-switching enabled, the packats will bypass the system processing and it will work as a normal switch, not a managed one.
EDIT 2: I performed some additional tests, after disable hw-switching, it's possible to create VLAN using "normal" Linux commands, so, it's not necessary to use VLANCTL.