Random packet losses sound more like a networking issue on the ISP's side. Did you check with traceroute if you could pinpoint at which hop packets were typically being lost?
I am not great at analyzing traceroutes, but it looks like the second hop is where it is timing out:
C:\Users\Geran>tracert google.com
Tracing route to google.com [2607:f8b0:4005:800::1005]
over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms 2601:c:a380:d16:beee:7bff:feef:51a8
2 * * * Request timed out.
3 8 ms 8 ms 9 ms 2001:558:1a2:200a::1
4 17 ms 14 ms 14 ms 2001:558:210:5b::1
5 12 ms 53 ms 15 ms te-0-4-0-7-ar03.sacramento.ca.sacra.comcast.net [2001:558:210:13::1]
6 23 ms 20 ms 19 ms he-3-6-0-0-11-cr01.sanjose.ca.ibone.comcast.net [2001:558:0:f6a1::1]
7 19 ms 17 ms 16 ms be-22-pe02.529bryant.ca.ibone.comcast.net [2001:558:0:f8a6::2]
8 70 ms 69 ms 79 ms 2001:559:0:1c::6
9 17 ms 47 ms 18 ms 2001:4860::1:0:7ea
10 19 ms 28 ms 22 ms 2001:4860:0:1::691
11 21 ms 28 ms 24 ms nuq05s01-in-x05.1e100.net [2607:f8b0:4005:800::1005]
Trace complete.
Edit: I will add that currently I am not seeing packet loss, so I will leave IPv6 on for a bit and see if I can break it.