I didn't mean to belittle or imply that you were "silly" as I can completely empathize with most networking frustrations.
I've had my own share of networking problems & they can be challenging & sometimes darn-near impossible to resolve.
But to basically mirror what
Skiron pointed out...
You can only eliminate the additional ping spikes & delays which are introduced via "you're own networking hardware & devices".
Obviously any delays outside your network infrastructure will be unavoidable.
An inconsistant or unreliable ISP & your problem will always remain.
Tech9 had suggested you could upgrade your router hardware to a RT-AX68U & minimize the ping spikes.
And according to the following review... he is most likely correct.
Note: Pay special attention to the section:
The leader in latency & the circular animated Graph.
Latency is king when it comes to gaming routers, and the RT-AX86U has the lowest lag of any router I've ever tasted.
www.cnet.com
Myself I was somewhat reluctant to recommend you spend more money because as you can see from the latency graphs...
Your 40ms ping is already close to about as good as you'll ever see.
I'm not really a gamer myself so... I'll let someone more experienced assist you regarding QOS.
In my household I have a part-time step son who has been the BIG gamer in the family.
However, he's currently 17 & has always had an affinity for "one word answers & grunts etc" so obtaining any substantial quality of network latency is difficult.
I get... GOOD or FINE.
I've even tried pretending to be hip & asked if he ever "laggs-out while gaming online"
Anyways I've made his PS-5 use a static IP & use a custom FlexQOS rule to basically give that device the highest priority.
However just as others have said...
Plug in the game consoles with Ethernet if possible.
WiFi will always introduce some additive latency it has too + it's much more subject to noise interference which will obviously result in additional latency.
If you have to utilize wifi make sure it's at least WiFi 5 or newer because 2.4 is even subject to microwaves & they tend to be used somewhat frequently.
Good Luck,
I think I'll go microwave some PIZZA & watch my WYZE-CAM streams disconnect... LOL