NOT asking how to setup the Plex server.
Asking if anyone has with the RMerlin firmware on a 68U and if so did you encounter any issues, fixes, tweaks?
I run/administer quite a few Plex Servers on Networks for myself, family, friends, and clients.
Most of these Plex Servers are on network that use Asus RT-N66, RT-AC66, RT-AC68, and one that uses an RT-AC87 router. Only the network that uses the RT-AC87 router uses the RMerlin firmware. I have all the others set up on either the John9527 Fork or HGomez Fork off of the RMerlin firmware.
I have never had any problems with the Plex Server when running any version of the John9527 firmware that could be attributed to the router firmware itself. Just make sure that you have the Port Forwarding set correctly and that you set up the PMS server with a static LAN IP.
I have one Plex Server on a Network that has a RT-AC68 running the HGGomez Fork firmware. That one is closest to the most current RMerlin firmware. I like the ability to monitor network traffic and utilization with that firmware. Again, I have no problems with the Plex Server that I can attribute to the router, or firmware.
Most of the problems I have found with Plex Servers are from the OS (The Firewall settings are critical), Browser used, and the PMS software itself (PMS settings are critical if trying to communicate with client devices outside the network that the server is on).
As for Tips & Tricks, there are so many different things to say, it is hard to know where to start. The few most important (IMO) are:
1. Don't use an under-powered server box if you want good transcoding and streaming without hangups and buffering. More CPU and RAM really is better.
2. Forget about wireless. Connect your PMS with a CAT6 Ethernet cable to the router. Connect your router by CAT6 Ethernet Cable to your media devices. If you want to guarantee smooth, trouble free streaming without hang-ups or buffering of your 1080p or 4k video streams, Ethernet cable is the best. If you have a very good PLA with new electrical wiring that is properly grounded (and clean power from the electrical company) that can work almost as good as Ethernet cables.
3. If you are streaming to client devices outside your network, make sure that you have internet service capable of upload speeds of 10Mbps, or higher for 1080p video streams. Streaming 4k videos will require at least 15Mbps. - Yes, I know that you can set the maximum upload speed on the PMS to limit the upload speed outside (or inside) your network. That works good when you have a very capable CPU capable of transcoding the original video file to a lower quality. If you don't, you run into hang-ups.
4. Newer, more powerful smart phones are reasonable devices for wireless streaming because they have the capability of transcoding the video stream (less load on the PMS) and can have excellent wireless chips that pick up wireless signals good. Older phones and tablets are not so great and can be the source of problems.
5. Make sure you take into account the number of client devices that are going to use the PMS simultaneously. Combine that with the type of streams to get the maximum network, and PMS load just for that PMS. If the network also has clients who are gaming, torrenting, or doing heavy duty file transfers at the same time as streaming media. The traffic may overload the network and the Plex experience will be degraded.