I am very ignorant with unix/linux, what kind of commands are used to say make two of the LAN ports assigned a separate subnet such as 192.168.10.xxx? If I make changes using vlanctl, I assume it has to be scheduled in a script?
First, if you're on 386 or higher code base and have guest wireless 1 enabled, you should have VLAN 501 (2.4Ghz guest) and VLAN 502 (5ghz Guest) already created, each with their own subnet and DHCP pool. So they've kind of done some of the work for you. So I'd highly recommend going that route to get a lot of the work out of the way automatically. I have moved a wired port into 502 (untagged) and also trunked (tagged) 501 to my outdoor 2.4Ghz AP on my non-HND router and it works well. If you want some ports in their own subnet/vlan that is NOT shared with a guest wireless, you can still use this pre-configured functionality, just hide the SSID for guest wireless 1 and use it strictly for wired clients, then if you want guest you can use GW2 and/or 3 for that (which will share a subnet with your LAN but use firewall rules to block communication as long as you enable the option to block LAN access from guest.
If you search here for RT-AC86U VLANs you'll find some discussions. Bear in mind a lot of the discussions were before Asus created these two new networks for you so they are actually doing a lot more than you need to now. I believe now all you have to do is move the physical interface into the guest bridge, and use ifconfig (and possibly vlanctl or ip link) to enable that new sub-interface. I believe you'll also have to disable hardware switching with
ethswctl -c hw-switching -o disable for it to work. In one of the threads it is stated that once you disable hardware switching you can also use the more standard linux "ip link" commands to do some of the stuff for you.
Yes any changes you make using vlanctl, brctl, ip link, etc will need to be in a script. Services-start script works, you just have to keep in mind any changes you make in the router could potentially overwrite your script changes until you reboot or re-run the script. Service-event scripts may be more appropriate since they'll run after various services are restarted and reapply your configs.
I think you're going to need to do the following
disable hardware switching
brctrl to remove the physical eth port from the main bridge
brctl to add that port into the guest bridge
ifconfig to bring up the new subinterface like eth0.v501
Without having access to an HND router all I can really tell you is to go look at how VLAN 501 and 502 are set up in ifconfig, vlanctl, ip link, brctl, etc and reverse engineer how to move a physical port over.
Your other option is to upgrade to an AX router (either a Pro model which has native VLAN support, or a non-pro that uses non-HND chipset and robocfg which makes the script a lot easier). You could also look at a different AC router like the RT-AC88U which uses non-HND, or even the RT-AC1900P. Probably can get them cheap used. I believe the 86 is considered a dud of a router so you may want to replace it anyway.