Aggregating 2 or more WAN links?

[L&LD]

Ignore please - Content deleted because post can't be deleted without deleting thread.
Tim

 

[jalyst]

I started this in the wrong thread, going to get the mod to move some of the earlier posts into this;

Just interested in thoughts on if this is possible, I currently have 1 link at 100/2.5, from my manual testing so far (would love some tips on how I might automate this?), it seems to fluctuate wildly, though I'm not sure if I'm testing properly!

Using speedtest.net I sometmes get well over 100mbps, but more often than not it's usually 50-80mbps*, with a latency of ~8ms. And if I use speedofme or some of the other testing services, it can even be way worse than 50mbps! (I'm guessing they're using the wrong route/server?) IIRC, so far upload speed doesn't seem to vary wildly -no matter which testing service I use.

I have 2 major DOCSIS 3x Cable ISPs in my area, they both use completely different HFC infrastructure. The one I'm currently with (Telstra) has underground cable infra, the other (Optus) runs it along street posts.

If 2 cable links is out, then perhaps 2 other WAN links is an option???

I'm of course interested in fail-over & load balancing longer-term, but the sole focus for this thread, is to rule-out the possibility of aggregation.

Thank-you.
*though I still have a very small sample to go by!

 

[System Error Message]

if it is from 2 different ISPs and speeds arent consistent than use load balancing and fail over.

Your latency of your current WAN is good but its speed fluctuates so it doesnt sound very reliable. If you could see your sync speeds and do a longer term test that could help.

Alternatively try dslreports and select fiber to test to see what goes on.

 

[jalyst]

As explained in prior posts, I have 2 options; 2 links from 2 ISPs, or 2 links from one ISP (the one I'm currently with), if it truly is impossible to use a separate ISP for the 2nd link when aggregating, then ofc I'd only consider 2 links from 1 ISP.

If you could see your sync speeds and do a longer term test that could help.

Exactly, which is why I'm asking for the best way to do this, I don't have a fw/router capable of this yet, so it'd prolly have to be done from my Macbook, which -as mentioned in the other thread- is directly connected to my Netgear CG3100D-2BPAUS. If I could automate logging/monitoring over say 48hrs, & get an average from that, I'd have a much better picture of speed down/up, latency, & hence if aggregation's feasible. If it's really bad, then I'd be hardly pressing the ISP to investigate, & improve if/where possible.

Cheers.

 

[System Error Message]

So your router is also your modem and doesnt show your speeds? Speedtest is totally viable for testing your link if your ISP has a server. Call your ISP up and see about doing speed tests with them during some times of the day, they may be able to see your sync speeds though.

Speed tests arent entirely accurate. Does your router let you see stats? For example my mikrotik router shows me a graph of bandwidth and also shows it in text on the current bandwidth used. My connection syncs at 64/20Mb/s and when i do a speed test i get much less than that but my router shows the interface speeds (the ethernet port connected to modem) the same speed as the sync speed so there are overheads. Sometimes ISPs can have lots of traffic that you dont get the full speed.

What you need to know is your standard speed (speed during normal times). dslreports will do a bufferbloat tests that does help to see latency too because latency on a fully used connection is higher. Different types of mediums are different in that DOCSIS and DSL react differently. Fiber optics or ethernet is the most stable and reliable link in terms of speeds but DSL and DOCSIS are less reliable in speeds.

For your needs you can use either 2 different ISPs or 2 same ones. Bonding/LACP can only be done over the same ISP while load balancing doesnt. layer 4 bonding can be done over different ISPs but isnt as great as with load balancing for that case. Linus tech tips has made a video about layer 4 bonding before by having 2 links from his ISP but his ISP doesnt support the feature so he uses a layer 4 provider to perform the bonding with. He is mistaken about the performance over different links because some providers can adjust the weightage depending on the link. For layer 4 bonding the same ISP is the best but not so great with different ISPs.

 

[jalyst]

Here's some info I got from my gw device...
https://www.dropbox.com/s/far4ab3qut00rc8/Screenshot 2016-10-01 16.29.58.png?dl=0
https://www.dropbox.com/s/ilw5ip1ia3i8s2w/Screenshot 2016-10-01 16.30.37.png?dl=0
https://www.dropbox.com/s/d4rrzyv8kzpwctj/Screenshot 2016-10-01 16.31.28.png?dl=0

 

[System Error Message]

ah i see you're using cable. Cable isnt consistent. Layer 2 bonding would be terrible. Layer 4 would work with a buffer but can occur additional lag while load balancing will work but you will have to set your limits and do a more complicated ruleset. Since you are using pfsense you should be able to tweak it.

With cable you are already bonding multiple channels but cable companies are inherently known for their terrible network architecture which basically means they dont have enough bandwidth for their backbone and cabinets. So trying to do multiwan to use all the WANs simultaneously with cable is difficult. Its not impossible, it just means you have to set it up with more complication.

You can use whichever ISP you like for your 2nd link but you wont be able to reliably do layer 2 or layer 4 LACP. My suggestion is do a PCC based load balancing with QoS and fail over script. The PCC load balancing will use both WANs but with the first set with limits. Pfsense will have it preconfigured as profiles when you enable dual WAN but you will have to manually tweak it if you run into issues.

 

[jalyst]

Hmm, think I'll just focus on getting the current link improved, but 1st I need to be sure it's currently well short of optimal;

Once I know it's sub-par, I'll test to see if there's anything on or near my premises that's contributing, & hence can be fixed, if not I'll push my ISP to make any needed fixes to their infra, to ensure it's meeting the minimum legal standard here!

After I've gotten the link marginally<--->greatly improved, I'll go ahead & order one of the hw configs we discussed in this thread, then I'll add a 2nd link from the same ISP for a ~mth, & experiment with aggregating & LB.

Surely that's the best logical approach, yeah?

 

[System Error Message]

well once you get the ISPs you want you can do load balancing. With pfsense layer 4 LACP is out of the question as the providers usually send you their box. Layer 2 bonding on the other hand depends on both ISP and pfsense support. So i strongly suggest you use load balancing if your speeds arent predictable.

 

[System Error Message]

I understood what you wrote and i've explained the different actions you can take. With unequal ISPs you cannot do LACP on them on layer 2, you can on layer 4 but may not be so good With unpredictable speeds you can do bonding on layer 2 and 4 but it may not work great. In these cases you should do load balancing.

If you want to do layer 2 LACP you have to call up your ISP to do it. Essentially this means 2 lines to your house from the same ISP and a modem that supports it. Router only sees 1 WAN.
If you want to do layer 4 LACP you have to find a provider that basically gives you a box to connect to your modems which will than route all your traffic through the provider's server at a datacenter somewhere which increases latency. Can be done over 2 different ISPs but is best done over 2 lines from same ISP when ISP doesnt support layer 2 LACP.

For everything else theres load balancing which is done by router.

All 3 methods support fail over.

My understanding is that you're asking more about the different bondings and balancing in choosing an ISP and i have given you that information. Its up to you to select which and ask your ISP.

 

[L&LD]

layer 2 bonding rr mode.

And this is what is needed to get a single faster connection? There is no consumer hardware that will do this (without taking a deep dive into scripting, hardware and ISP support).

You are the only one I have seen talk about this. Every other discussion on this has been what I've stated, 'it can't be done on the users end without the stipulations stated above'.

 

[jalyst]

There is no consumer hardware that will do this (without taking a deep dive into scripting, hardware and ISP support).

You sound like you're excluding me from being able to do this 'right off the bat', bit presumptuous, no? How about trying to help towards my aim, instead pooh-poohing & saying: "No noob, don't waste your time, you're too clueless."

Ofc if you don't want to you're free not to, but please don't derail this into an intellectual debate (or past baggage about older threads). If there's the slightest possibility that it's doable, then that's the focus of this thread, please re-read this post: http://www.snbforums.com/threads/aggregating-2-or-more-wan-links.34986/#post-283352

Oh & SEM, I fully intend to address your last post, but need to get some shut-eye first, ~12:30am here! Thanks to those trying to help, it really is appreciated, & L&LD, I know you try to help in your own way, so TY too.

GN.

 

[System Error Message]

And this is what is needed to get a single faster connection? There is no consumer hardware that will do this (without taking a deep dive into scripting, hardware and ISP support).

You are the only one I have seen talk about this. Every other discussion on this has been what I've stated, 'it can't be done on the users end without the stipulations stated above'.

Most bonding is done using 802.3ad which doesnt allow splitting of packet streams. With balance rr mode streams are split so the packets are sent through each link equally. Its not quite as safe as 802.3ad in that the protcols used must support out of order packets or there may be issues.

http://wiki.mikrotik.com/wiki/Manual:Interface/Bonding
Bonding modes, it is explained there.

 

[L&LD]

Hmm, think I'll just focus on getting the current link improved, but 1st I need to be sure it's currently well short of optimal;

Once I know it's sub-par, I'll test to see if there's anything on or near my premises that's contributing, & hence can be fixed, if not I'll push my ISP to make any needed fixes to their infra, to ensure it's meeting the minimum legal standard here!

After I've gotten the link marginally<--->greatly improved, I'll go ahead & order one of the hw configs we discussed in this thread, then I'll add a 2nd link from the same ISP for a ~mth, & experiment with aggregating & LB.

Surely that's the best logical approach, yeah?

The reason your connection is well short of optimal is because you have a cable ISP connection. As a shared medium, cable can give speeds that are spectacular to dismal, depending on the time of day, number of users, etc.

From what you've provided so far, the connection you have is running as optimal as possible, ime.

I don't know why you insist on using the speedofme site for testing your connection with? It may give the most real world results, but that isn't what the ISP is selling you. ISP's sell 'up to' connections and while you can briefly go higher for a few seconds or minutes at a time, they will throttle you back to what you're paying for. Speedofme shows the opposite of what your ISP is providing, imo. And for that reason it shouldn't be used to 'prove' your current speeds as 'good' or 'bad' at all.

For example, my nominal '150/150'Mbps Fibre service shows the following at this minute:
Ookla speedtest: 165/165 Mbps @ 1ms latency.
DSLReports: 155/156 Mbps with scores of A+, A+ and A+ for overall, BufferBloat and Quality.
Speedofme IE: 47/11 Mbps @ 34ms latency.
Speedofme Edge: 35/8.5 Mbps @ 44ms latency.
Speedofme Firefox: 71/13 Mbps @ 45ms latency.

What is also interesting to me was with any of the browsers on the speedofme website, the computer was working very hard. IE had an average cpu usage of around 15%. Edge had an average cpu usage of around 28% and FF had an average cpu usage of around 90% (not a misprint) on my i5 desktop with 16GB ram. Not only during testing of the ISP's connection, but also just sitting there idle afterwards too.

Can speedofme be coded so inefficiently? It seems to be in my testing today.

In conclusion, I would not be using that site to test for a certain level of service or not (even if I've bookmarked it to test the 'web' in general).

You are free to pursue this to see if it's meeting the minimum legal standard*. But I know you can spend your time better though.


* I have no doubt 'their' lawyers are better funded than 'your' lawyers (but if you can bring a change to the miserable service cable providers offer (particularly upload speeds), more power to you).

 

[jalyst]

No speedofme I found to be ineffectual, which is why I was using speedtest.net/other.

TY/Bedtime/GN.

 

[L&LD]

You sound like you're excluding me from being able to do this 'right off the bat', bit presumptuous, no? How about trying to help towards my aim, instead pooh-poohing & saying: "No noob, don't waste your time, you're too clueless."

Ofc if you don't want to you're free not to, but please don't derail this into an intellectual debate (or past baggage about older threads). If there's the slightest possibility that it's doable, then that's the focus of this thread, please re-read this post: http://www.snbforums.com/threads/aggregating-2-or-more-wan-links.34986/#post-283352

Oh & SEM, I fully intend to address your last post, but need to get some shut-eye first, ~12:30am here! Thanks to those trying to help, it really is appreciated, & L&LD, I know you try to help in your own way, so TY too.

GN.

I'm not writing you off at all. Just trying to get some basic information straight which SEM doesn't seem to want to make clear. All the information I points to the fact that with the caveats I have presented already, merely wanting this to happen won't make it so. ISP support is 'required' and in a big way.

Let me be clear; if what you want from bonding two ISP links is to have a faster (combined) service, then this is what I'm saying is not possible with consumer equipment.

If you simply want two or more devices on the same network to share two ISP connections (with any one client limited to the maximum speed of either of the two ISP), then sure, you can do that. That is not bonding though. That is called Dual WAN (in balanced mode, for Asus routers).

I really am trying to help and if I end up learning something new, that will be a good thing too!

You're welcome and Good Night.

 

[L&LD]

Most bonding is done using 802.3ad which doesnt allow splitting of packet streams. With balance rr mode streams are split so the packets are sent through each link equally. Its not quite as safe as 802.3ad in that the protcols used must support out of order packets or there may be issues.

http://wiki.mikrotik.com/wiki/Manual:Interface/Bonding
Bonding modes, it is explained there.

SEM, thanks for the link. That is well over my head, I'll admit.

And I don't think a 'regular' consumer would be able to implement this without issues either?

What kind of hardware are we talking about to do this and what kind of investment is required?

Is this 'bulletproof'? Or will the issues be insurmountable that you hint at?

My direct experience with bonding? My ISP, a few years ago, offered bonded DSL to get to 25Mbps download and 5Mbps upload speeds. After almost half a dozen visits to the premises by the technicians, the 'bonded' modem was replaced with a single unit (can't remember it now) which offered 50Mbps/10Mbps service at my location (I actually tested it with my own computer at that time) which I passed on at that time (because of price issues). They left the latest modem but at my lower ISP rates and the technicians were never needed to return again.

When they were there, they showed me graphs of what my service was like with the bonded modem. At some points (sometimes for days at a time), the speeds were at 'the level of dialup' service, paraphrasing the tech.

A few weeks afterwards, when a customer asked for higher speeds (we had the same ISP), I suggested they ask about a 'bonded' modem. By that point, the ISP had given up on bonded service and were not offering it anymore. That is telling, no?

Depending on the cost (time and money) to implement this (if it is 'bulletproof' and a one shot setup), I know of a few customers that may go this way too.

Eagerly awaiting your reply.

 

[sfx2000]

Most bonding is done using 802.3ad which doesnt allow splitting of packet streams. With balance rr mode streams are split so the packets are sent through each link equally. Its not quite as safe as 802.3ad in that the protcols used must support out of order packets or there may be issues.

http://wiki.mikrotik.com/wiki/Manual:Interface/Bonding
Bonding modes, it is explained there.

In my experience, Bonding is more for reliability and redundancy - not speed... whether one does this at the ethernet layer (IEEE802) or using the Linux bonding driver (which can be easier for some to work with), there is still a lot of benefit, but again, more for reliability... whether it is active/active or active/standby - both have fail over if one of the two (or more) links is lost...

Some more insight here... it's older info (linux 2.4), but the general guidance is still good stuff...

https://www.kernel.org/pub/linux/ke...inux-2.4/Documentation/networking/bonding.txt

There are newer things out there - OpenStack has a lot of good stuff related to load-balancing and multiple links, and what's nice there is that it can be easily automated...

 

[Mokers]

SEM, thanks for the link. That is well over my head, I'll admit.

And I don't think a 'regular' consumer would be able to implement this without issues either?

What kind of hardware are we talking about to do this and what kind of investment is required?

Is this 'bulletproof'? Or will the issues be insurmountable that you hint at?

My direct experience with bonding? My ISP, a few years ago, offered bonded DSL to get to 25Mbps download and 5Mbps upload speeds. After almost half a dozen visits to the premises by the technicians, the 'bonded' modem was replaced with a single unit (can't remember it now) which offered 50Mbps/10Mbps service at my location (I actually tested it with my own computer at that time) which I passed on at that time (because of price issues). They left the latest modem but at my lower ISP rates and the technicians were never needed to return again.

When they were there, they showed me graphs of what my service was like with the bonded modem. At some points (sometimes for days at a time), the speeds were at 'the level of dialup' service, paraphrasing the tech.

A few weeks afterwards, when a customer asked for higher speeds (we had the same ISP), I suggested they ask about a 'bonded' modem. By that point, the ISP had given up on bonded service and were not offering it anymore. That is telling, no?

Depending on the cost (time and money) to implement this (if it is 'bulletproof' and a one shot setup), I know of a few customers that may go this way too.

Eagerly awaiting your reply.

Like you said, the ISP can offer the bonded service. Many ISPs do it because there infrastructure doesn't allow for higher speeds. So for example your DSL tops out at 25 Megabits. The reason for the technician visit is because the bonded connection requires another physical connection to the ISP. Going back to a single connection doesn't require another visit because at that point, the only changes they make are in firmware. In fact, if you had purchased new service and the technician had come to install the single service, it is likely you would have had the same modem.

However, all of this depends on the ISP knowing what is going on at both ends. Once you start going past a certain point, bonding doesn't make much sense to the ISP and they see better return on upgrading the infrastructure and offering different products. Bonding helps get around bandwidth limits that are largely defined by the method of transport. So when we had nothing but copper pairs to work with and no compression methods, etc, it was more prevalent. But new technologies scale much better and don't need bonding to get higher speeds.

Lots of places employ multiple ISPs and it can improve throughput, but although you gain benefits because you have additional total throughput, it largely depends on how its setup on your network. Some places just use generic failover, some dedicate certain applications to a particular gateway or just do on the fly or metered load balancing. Bonding like your original post, two different ISPs even if they are a similar product doesn't really happen because like @sfx200o said, you would normally have two ISPs for redundancy. If you wanted a faster product, you would buy a higher tier service from the one ISP.

Not sure if I added to or just muddled the conversation here.

 

[sfx2000]

Not sure if I added to or just muddled the conversation here.

It's good input...