I had similar problems when I tried to setup deluge over a vpn.
It sounds like there's a need to understand what's happening and work out a way to verify each part of the packet traversal. The first part would be to verify the provider port is being forwarded. You might be able to verify that by setting something up on Windows using the provider VPN app to check the port is forwarded.
Once you know the port is forwarded consider what you have, an external address and port, <unknown, 11167>, connecting to the VPN provider with packets being sent to the VPN internal address (provider) end to port 11167, forwarded to the VPN client end to port 51413.
I guess that should get you packets on the router bridge with a destination port of 51413 so forwarding <anything, 51413> to the IP of the Transmission client should work assuming the VPN client config incoming firewall has been disabled.
Whether you need to forward from the VPN network device (or IP, that you probably can't rely on the always be the same) to br0 at all is another question, not sure about. It looks like you should but the VPN client should setup routes to do that and this is internal to the router, not external, so firewall rules probably shouldn't be needed at all. So check the route table after establishing the VPN and ensure your getting packets, maybe your VPN configuration is not pulling all the configuration or is not setting everything up properly.
Perhaps a host route from the VPN client device to the IP address of the Transmission client is enough.
iptables rules to ensure the incoming port traffic can't go anywhere else would probably be wise but, AFAICS, shouldn't be needed.
But I also couldn't get this to work and ended up setting up a container that included the torrent client and openvpn so that everything is local to that instance, and the container developer sets up the (fairly simple) iptable rules needed and that worked. The one I used was based on one originally done for Transmission.
Ian