PPPOE WAN not working on OPNsense but works on router

[CntrlAltDel]

Hi there,

I have a ESXi host with OPNsense installed serving as my router connected to an Asus RT-AC88U in AP mode.
The issue I'm having is I can connect to my GPON/ONT with WAN set in DHCP mode fine but it provisions a CG-NAT'd IP.
When I switch to PPPOE and use my PPPOE credentials that work on my Asus router (not ISP provided router) it connects fine.
But when I do the same on OPNsense I just get "connecting..." from the logs.
Strange thing is, I had OPNsense running in this exact scenario for 2 years prior, just about 4 months ago I decided to switch back to my regular router and decomission my ESXi host, I've now redeployed everything but seem to be stuck here.
Has anyone else dealt with this issue?
ISP is saying PPPOE connectivity is being rejected from their end but like I said I've gotten this to work before and it does work on my regular Asus router.

 

[Digilog]

Looking around on the Internet, I see others that had expirienced that bug. There is even a bug report raised at its github. So if you have that older version of OPNsense, you need to use that instead of the two newer versions that have the VM bugs with the kernel. But to be fair, this software wasn't really designed to run inside a VM environment and there will be a small deterioration of performance with PPPoe connections.

 

[sfx2000]

ISP is saying PPPOE connectivity is being rejected from their end but like I said I've gotten this to work before and it does work on my regular Asus router.

Authentication?

I would start looking at the OpnSense system logs and see what's happening there...

 

[Digilog]

Its the setting choices in the OS and kernel code, what I see on my scratchpad computer I installed it on to look at the code.

the module MSIX has bad IRQ timing statements. Which is just one of several of the issues I see, however, I don't have time today to further go into detail of the other things I see. Use Legacy settings (MSI)

Bios:
Disable all c-states.
Disable all onboard Audio
Disable CPU scaling

inside OPNsense
No MAC spoofing,No promiscuous,
set

hw.pci.enable_msix: 0
hw.pci.enable_msi: 1

 

[coxhaus]

You should be running with pfsense as it is on the latest FreeBSD version. It will have newer drivers in some cases and the latest code. Forget OPNsense until they catch up.

 

[sfx2000]

You should be running with pfsense as it is on the latest FreeBSD version. It will have newer drivers in some cases and the latest code. Forget OPNsense until they catch up.

Deep breath here - one should not do pfSense due to all the shenanigans of the management team.

 

[coxhaus]

Deep breath here - one should not do pfSense due to all the shenanigans of the management team.

I am a techy, so I want the best tech. OPNsense is lacking.

 

[sfx2000]

Going back to OP's issue - PPPoE has additional overhead because of the PPP framing... this could cause issues with the PPPoE session setup...

If PathMTU discovery isn't working, the default MaxMTU of 1500 can get fragmented - try rolling back the MaxMTU size from 1500 down to 1492, and see if that helps...

Other concerns here is the authentication methods - there are more than a few with PPPoE, and discovering the correct method could take some time.

Challenge is that OP is running a pfSense VM inside ESXi - so where to put that limit on MaxMTU could be, perhaps, complicated...

Additional insights here from the network side perhaps - might be a reasonable solution inside...

Configuring Point-to-Point Protocol over Ethernet | Junos OS | Juniper Networks

 

[Digilog]

Challenge is that OP is running a pfSense VM inside ESXi - so where to put that limit on MaxMTU could be, perhaps, complicated...

Timing and bandwidth constrains are the reasons why a router OS should not be ran in a VM. Linux is better at running a router program in a VM compared to BSD and I'll be surprised if no one out there has said anything about this nor the drawbacks of doing this.