I connected to my 5ghz guest network and I was able to ssh to my router.
Then I have no idea. The only guest isolation bug I'm aware of was with the RT-AC87U's 5 GHz band, which was fixed by Asus in 378_4950. You can try again with 378.53 when it's released to see if you get better luck.
Guest isolation is implemented at the ebtables level, so make sure you're not doing any changes to that table that might break it.