Split a FTTP Connection between businesses

[delid4ve]

Hi All

We are a small family business currently restricted to a 2mb 20:1 adsl connection which we have severly outgrown.
We have just placed an order for a leased line 100mb 1:1 FTTP connection with multiple /30 ip addresses.
Since the cost of this connection is high we have arranged with other businesses on the estate to split this connection for cost sharing.

Although highly computer/network literate on a home basis, and also managing our work domain, this networking falls out of my scope, yet i am tasked with the job, which I believe im more than capable of achieving.
I have read various discussions on the scenario but cannot pinpoint a 'best' method as it seems there are multiple ways of achieving this.

The way we would like this to work is as follows:
Main connection into our premesis
Every business to have a cat5e/6 cable to their premises with a static Public facing IP address
Every contributing business to supply their own router to set up their own internal LAN
Each company should be able to port forward within their own router / no double NAT
Each company should not be able to see/get access to another companies IP.
Be able to limit each companies connection to a set speed

We need to keep cost to a minimum for this and also complexity, utilisation of extisting hardware would be preferred.

We currently have:
Netgear Prosafe GS724Tv3 managed smart switch
Asus N66u router (for our internal connection/network)
Still to purchase router for incoming connection

Any help would be most appreciated.

 

[Cloud200]

First off, every company SHOULD be able to see each other's external IP address.
At that point, they are "on the internet"

Second, how are you getting the IPs?
1 IP address from the ISP to connect to the default gateway and multiple secondary routed IP blocks?
Are you sure its not 1 Routed IP and say a /29 or /28 for the secondary IPs?

 

[delid4ve]

Hi Cloud thanks for the reply,

Having seen different ways of doing this via search engines, none of which were very clear, some discussion was based upon what type of IPs were provided, hence my question to them was:

Question:
Could you check with your tech team the type of IP addresses that are allocated. Are they /29 or /30?
Answer back:
The ones we supply you with will be /30, however we can request /30 and would do that using the RIPE form

With the initial setup we are allocated 2 x IP address's, to which we will apply for more.
Obviously my question was very vague to them, what do i really need to ask? I'm going to be teaching myself sub-netting over the next couple weeks to understand the /28 /29 /30 in depth, as at present im only used to a standard adsl connection dhcp assigned by ISP and a single internal subnet.

Yes I was wrong in my original, of course external should be able to see external, it should have been more along the lines of company A's internal and external should not be able to see Company B's internal. Excuse my noob description

 

[Cloud200]

OK, so whats going to happen is one of the following:
A. The ISP gives you a router pre-programmed with all the information to connect it to the internet.
It will have multiple sub-interfaces that they program in for you to connect over layer 2.
Any dumb switch will work in getting all the offices connected as you wish.

B. The ISP gives you an external IP address and gateway along with a list of IP blocks.
You need to set up a router (or a layer 3 switch) to do the routing from one set of addresses to the other.

 

[delid4ve]

There was an option on the line to have router included, or not, we have opted for not in order to source exactly what is required. The router they would supply was a cisco 891.

So what would be the correct terminology for me to ask in order for me to find out the correct information?

So if im right in thinking, option B is what I really require in order to set up VLANS for each business to have their own static public facing IP address.

 

[System Error Message]

Its not hard to set it up but the best would be to divide everything equally (bandwidth, IPs, etc). You could just set the minimum to the equally assigned bandwidth so during worst case scenario everyone would get their fair share of bandwidth.

you have to think of it as distributing not only the link but also how everything would connect and be set up. A /30 Ip block is quite a lot of IPs to go around. Since you already have a netgear switch will you be using that to distribute the internet across all establishments involved?

From how i see it the path would be ISP-- router -- switch -- router-- internal network. The first router can be used to distribute IPs but the main task is for user based QoS so everyone gets their fair share of bandwidth during worst cases. Normally i would suggest mikrotik or pfsense as they have good QoS capability and can also provide other interesting features (pfsense having snort and other easy to use firewall stuff). Using pfsense also means you can scavange a PC for use. Normally a fast MIPS based router from mikrotik or even a dual core ARM based router will handle the QoS task without an issue at your internet speed.

If you want to add security to prevent someone from just plugging in and stealing internet or hacking in just by connecting stuff you can use the first router with authentication for other routers. VLANs can be used to prevent snooping and to isolate all the ports.

To summarise you want a router before the switch that:
Performs user based QoS (so everyone gets their fair share of bandwidth during worst cases)
Supports lots of VLANs/isolation
Authentication, preferably RADIUS. Perhaps RADIUS with a mac to user binding
gigabit ethernet and WAN ports to make full use of 100Mb/s
Support the routing method used by ISP and inter routing if businesses want to share resources

Some routers from non consumer sources like mikrotik have SFP ports so you can just plug the fibre optic module into it but its not necessary. You can use your switch to set up VLANs and route the fibre optic from ISP into SFP and from SFP to the WAN port of the router. Basically you could even use a consumer router running tomato firmware.

You also have to secure the router and switch (such as disabling unused ports) and keep them under lock where they wont be easy to get to.

So you will need to discuss with the others,
bandwidth and IP division (how much bandwidth (during worse case) and how many IPs does one get)
Who is responsible to keep the hardware operating and secured

The GS724T is a layer 2 switch so it will not perform any layer 3 routing for sharing resources at wirespeed but it will support LAGG which can be done with some routers to allow for more transfers. A sophisticated router can be used for any sort of situation whether your ISP performs NAT or not. Your choice for routers would be either a fast recent MIPS based one from mikrotik or ubiquiti (dont be fooled by their wirespeed claim, their true QoS throughput is very low), a multicore ARM based router running tomato or openwrt firmware or an x86 solution such as scavaging a PC to use as a router and installing a router orientated OS or a server OS (microsoft does very poorly in networking so dont use windows/windows server). In the case of the non x86 choices, basically embedded you want to make sure that all the ports are CPU connected.

 

[delid4ve]

Thanks for the detail SEM,

When the initial connection is made it will just be our business utilising it until I get to grips with it, find out how the ISP is goin to assign the IPs. I've done a little reading now on sub netting so I understand this more, although no real world experience yet.

The way I think I'll do it first off, without buying any additional hardware:

Load wrt onto asus n66(currently running merlin)
Bring incoming fibre into gs724t sfp and out to n66 wan port.
Set up 2x vlans( for the 2 external IPs on 2 of the lan ports) (4 lan ports on n66 that I've already read can be separated using wrt on a port basis and untagged
If I'm right in thinking I should then be able to use the other two lan ports for internal routing, have to put some static routes in somewhere I believe and output the cables from both ports to the two separate vlans set up in the gs724t also.

If I'm right in thinking if the Isp is giving me /30 x2 IPs then this is one block, so will have a gateway address, sub mask, 2x useable IPs and the broadcast address. So the Isp has routing enabled their end to say that the two IPs I'm assigned are on my connection through the gateway address?

If, and excuse my poor description, this looks possible then I'll set this up to get my head around things as I can effectively have 2 separate networks with their own external IP before I ask for more IPs and buy some more hardware, probably looking at another 6 businesses jumping in on it which will make it cost effective for everyone as the connection is on a 100mb header that we can upgrade our speed at anytime.

 

[delid4ve]

Just been looking at the mikrotik routers, cheap compared to the ciscos

 

[delid4ve]

From my understanding one of these should be able to do everything I ask:

http://routerboard.com/CRS125-24G-1S-IN

At £145 that's a bargain if it is

 

[Cloud200]

A /30 in your situation is only a single usable IP.
You assign one to your router. This becomes the tenant's default gateway. The tenant gets the other IP in the block.
The CRS series is far too slow of a Layer3 device for your purposes. Only use them for what they were designed as, switches. At best it is good for 116mb/s. Reality will be far less.

 

[sfx2000]

A /30 in your situation is only a single usable IP.

a chart that some folks might want to keep handy...

32 = X-network-bits + Y-host-bits
Addresses = 2 ^ Y-host-bits
--------------------------------------------------------------
CIDR Total number Network Description:
Notation: of addresses: Mask:
--------------------------------------------------------------
/0 4,294,967,296 0.0.0.0 Every Address
/1 2,147,483,648 128.0.0.0 128 /8 nets
/2 1,073,741,824 192.0.0.0 64 /8 nets
/3 536,870,912 224.0.0.0 32 /8 nets
/4 268,435,456 240.0.0.0 16 /8 nets
/5 134,217,728 248.0.0.0 8 /8 nets
/6 67,108,864 252.0.0.0 4 /8 nets
/7 33,554,432 254.0.0.0 2 /8 nets
/8 16,777,214 255.0.0.0 1 /8 net
--------------------------------------------------------------
/9 8,388,608 255.128.0.0 128 /16 nets
/10 4,194,304 255.192.0.0 64 /16 nets
/11 2,097,152 255.224.0.0 32 /16 nets
/12 1,048,576 255.240.0.0 16 /16 nets
/13 524,288 255.248.0.0 8 /16 nets
/14 262,144 255.252.0.0 4 /16 nets
/15 131.072 255.254.0.0 2 /16 nets
/16 65,536 255.255.0.0 1 /16
--------------------------------------------------------------
/17 32,768 255.255.128.0 128 /24 nets
/18 16,384 255.255.192.0 64 /24 nets
/19 8,192 255.255.224.0 32 /24 nets
/20 4,096 255.255.240.0 16 /24 nets
/21 2,048 255.255.248.0 8 /24 nets
/22 1,024 255.255.252.0 4 /24 nets
/23 512 255.255.254.0 2 /24 nets
/24 256 255.255.255.0 1 /24
--------------------------------------------------------------
/25 128 255.255.255.128 Half of a /24
/26 64 255.255.255.192 Fourth of a /24
/27 32 255.255.255.224 Eighth of a /24
/28 16 255.255.255.240 1/16th of a /24
/29 8 255.255.255.248 5 Usable addresses
/30 4 255.255.255.252 1 Usable address
/31 2 255.255.255.254 Unusable
/32 1 255.255.255.255 Single host
--------------------------------------------------------------

In networks larger than a /31, one address is used for the network number, another for the broadcast address, and generally another as the default gateway for routing to other networks. A /29 may cover a range of 8 addresses, but only 5 of them can be used as host endpoints. A /30 has only 1 usable address.

Network: 192.168.1.0/30
Gateway: 192.168.1.1
Usable: 192.168.1.2
Broadcast: 192.168.1.3

Network: 192.168.1.0/29
Gateway: 192.168.1.1
Usable: 192.168.1.2-6
Broadcast: 192.168.1.7

 

[System Error Message]

The CRS is an actual layer 3 switch meaning you can perform layer 3 routing on the switch layer just like ubiquiti's layer 3 switches too.
However you already have a layer 2 switch and assuming you treat the ip block as a LAN than you dont need a layer 3 switch.

Use the hardware you already have and just get a good router. The RB2011 is fast enough for your needs.

Its bad to use the CRS as both a router and switch. It lacks RAM and is limited by its link to the CPU so it limits other LAN functionalities you want to add.

 

[Cloud200]

Also very useful:
http://www.subnet-calculator.com/cidr.php

PS. a /31 is sometimes a valid network for P2P communication. I use them many times at work for link up sites via wireless links.
See here for more info:
http://tools.ietf.org/html/rfc3021

 

[Cloud200]

I drew up a little diagram to make sure you understand what it is that needs to be done.
Here is the end result:

If you are planning on upping the speed past 100mb/s you will want a router that can do sub-interfaces via 802.1q at a minimum of 200mb/s with a few rules already in place with a packet size of 64 bytes.
Some examples of Routers that will work:
Cisco 2921 or 1941 (see this)
Mikrotik RB3011
Mikrotik RB1100
Mikrotik CCR1009
pFsense/Netgate SG-2220
Ubiquiti Edgerouter (not the ER-X) series
Zyxel Zywall 110

Some examples of switches that will work:
HP 1920
Cisco SMB SG300-10

One of the benefits of having more "real" physical interfaces on a router or layer 3 switch is you can just assign an interface to the port and hand-off to the tenant.

Edit:
The main difference between using a layer 3 switch vs router as your primary edge router is in creating rules to firewall, shape, limit traffic. Treat yourself as another tenant for ease of management.

 

[System Error Message]

The ubiquiti edgerouter isnt the best for the job because with QoS it can slow down quite a lot.

The only QoS the edge router has to do is user based. If you have 4 users than you should set the ensured bandwidth between 20-25Mb/s and the max/burst to 100Mb/s per user (you can also see this as per vlan or so on instead of just user).

With mikrotik there are a few variations of the RB1100 series with the most recent being the RB1100AHx2 and there are some very old ones only called the RB1100 that are only single core.

I strongly suggest against mikrotik CRS switches at the moment because they currently lack some important features in firmware such as STP and they use very different terms. In terms of layer 3 routing routers are able to do that very fast and mikrotik has a very good performance chart about how the router performs if given some QoS or firewall filter rules. It gives a good expectation of performance that ubiquiti doesnt give any details about. Using mikrotik would be the hard path but if you really understand networking than it is easier than the others to use whereas. The other solutions are more user friendly than mikrotik and pfsense can be used from scavaging PCs. Even an old core2 can run pfsense very well for your config.

Any solution you use it is important that you are able to protect the router and the switch as both the router and switch will be completely exposed to the internet on all interfaces. I know the netgear switch you use has a mac address whitelist for being allowed to access the management of the switch. Port whitelisting would be better if you could and make sure to take the best practices you can in protecting it.

This may be minor but you should also limit broadcast and multicast traffic rates on your switch too and i know the netgear switch can do it because i have the same switch and it will only let me access it from connecting directly (you need to check if yours does this and not through a router).

 

[Cloud200]

The ubiquiti edgerouter isnt the best for the job because with QoS it can slow down quite a lot.

Well . . . that is sometimes true.
It really depends on the type of QOS you are doing as well as the model of router.
If you are splurging for an ER8-Pro, you can easily do simple class based rate limiting with bursts and have the traffic flow at 200+ mbps. The RB1100 is pretty much the same speed under the same circumstances (talking about 64byte packets here).

fq_codel on the other hand . . . yeah that protocol is insanely hard on any CPU, even an x86.

The only QoS the edge router has to do is user based. If you have 4 users than you should set the ensured bandwidth between 20-25Mb/s and the max/burst to 100Mb/s per user (you can also see this as per vlan or so on instead of just user).

I absolutely agree on this.
Per VIF classes would be the easiest way to shape/rate limit traffic.
Technically though a "user" is just the tenant's router as far as the core is concerned.

Any solution you use it is important that you are able to protect the router and the switch as both the router and switch will be completely exposed to the internet on all interfaces. I know the netgear switch you use has a mac address whitelist for being allowed to access the management of the switch. Port whitelisting would be better if you could and make sure to take the best practices you can in protecting it.

One of the best ways of doing this is to restrict all management traffic to a single dedicated interface on the router/switch that is not routed directly to the internet.

This may be minor but you should also limit broadcast and multicast traffic rates on your switch too and i know the netgear switch can do it because i have the same switch and it will only let me access it from connecting directly (you need to check if yours does this and not through a router).

Why would you limit broadcast traffic?
The only devices within the tenant's routed subnets are the edge router and the tenant router . . .
Multicast traffic . . . yeah I can see a some rare cases for limiting that. If the speed you are giving each tenant is 25mb/s just set the rate limit for multicast at that. If the tenant wants to use all the bandwidth allocated on that for some reason, why should you care and stop them?

 

[System Error Message]

Well . . . that is sometimes true.
One of the best ways of doing this is to restrict all management traffic to a single dedicated interface on the router/switch that is not routed directly to the internet.


Why would you limit broadcast traffic?
The only devices within the tenant's routed subnets are the edge router and the tenant router . . .
Multicast traffic . . . yeah I can see a some rare cases for limiting that. If the speed you are giving each tenant is 25mb/s just set the rate limit for multicast at that. If the tenant wants to use all the bandwidth allocated on that for some reason, why should you care and stop them?

The netgear gs724T(s) doesnt have the ability to let you set a management interface or vlan.
The reason to limit broadcast and multicast traffic is that the netgear switch gets overburdened with it so you should limit them to reasonable rates. Otherwise it is possible for the netgear switch to overheat depending on the variant and how many fans on it you have.

Also regarding QoS, the limit should be set per direction and not for the entirety of the bandwidth as you actually have 200Mb/s of internet bandwidth if you upload and download at the same time.

 

[delid4ve]

Hi guys

Thanks for all the help.

It's becoming much clearer in my head now.
Got a 45 day lead time before the line is installed and ready to go so a lot of time to digest and set up the switch ready.

I've just checked the switch(have the same one at home) I can set a management vlan ID as per the attached, so if I set this as my businesses vlan that would restrict access to ourselves over just password authentification for the entire system if I'm correct?
(See attached)

The router and switch will be based on our premises with our servers etc so no one else would have access to these physically(these are small businesses not large companies /corporations, although I agree with the other bits Ie multicast traffic as although I don't think anyone within these businesses has the knowledge for this(I know most of them personally and they have no real I.T experience) I still see the need just in case.

I did read on my quest for this:
I give business A: Public IP, Gateway, DNS addresses
Say user makes a typo when putting this info into their wan details on their local router and instead of 192.168.1.3 they put in 192.168.1.4 as the IP and this falls within the block I'm given (remember I don't know what the ISP does when I request more IPs, they may assign me a /29 /28 or more /30s.

would I now need a layer 3 switch in order to route specific IPs to specific ports/vlans effectively isolating business A from using business Bs IP due to the typo.

On the same topic, if I'm assigned multiple /30 blocks from the ISP How would this work in the router, are the ones you suggested capable of this? Although I can get my head around the internal side of things I'm guessing on the ISP side.
From my thinking I would say they will give me a /29 /28 block as multiple /30 blocks doesn't play nice in my head from their end with routing say 16 different subnets down my pipe.

One other thing I'm still a little unsure about is the termination that they provide, from sales I've been told:
Faceplate on wall, Ethernet.
So from my understanding of that is standard cat5e,cat6 rj45. However as they are probably used to dealing with 'proper' I.T techs then in fact this could be SFP?

Thanks again guys for all your help, sorry I'm such a noob in this area

 

[System Error Message]

The router would just be routing rather than performing NAT.

You dont need to have everyone configure static IPs, you can use DHCP from the router and you can isolate them via layer 2 but allow layer 3 routing so servers could be shared without going through internet.

 

[Cloud200]

The netgear gs724T(s) doesnt have the ability to let you set a management interface or vlan.
The reason to limit broadcast and multicast traffic is that the netgear switch gets overburdened with it so you should limit them to reasonable rates. Otherwise it is possible for the netgear switch to overheat depending on the variant and how many fans on it you have.

LOL that sounds like a horrible switch then I guess i knew there was a reason I usually don't recommend Netgear, D-Link or any of the other "Best Buy specials".

I've just checked the switch(have the same one at home) I can set a management vlan ID as per the attached, so if I set this as my businesses vlan that would restrict access to ourselves over just password authentification for the entire system if I'm correct?

Correct. As long as that is not in a routed range. IE. keep it behind your office's gateway, not the edgerouter. The router however will usually need some form of ACL/firewall rule/policy to prevent access from the tenants.

I give business A: Public IP, Gateway, DNS addresses
Say user makes a typo when putting this info into their wan details on their local router and instead of 192.168.1.3 they put in 192.168.1.4 as the IP and this falls within the block I'm given (remember I don't know what the ISP does when I request more IPs, they may assign me a /29 /28 or more /30s.

would I now need a layer 3 switch in order to route specific IPs to specific ports/vlans effectively isolating business A from using business Bs IP due to the typo.

A layer 3 switch would take the place of the edgerouter.
A VLAN is a layer2 construct essentially creating virtual switches and router interfaces.
If the user has a /30 and they go outside the range, it will just not work. There will be no effect on the other tenants. The reason for this is you need to imagine that every VIF you create on the router on a dedicated VLAN per tenant is a physically distinct entity. As far as that portion of the network is concerned . . . that's all there is.

If you get say a /29 or /28 however . . . and are trying to split up the blocks further between the tenants . . . well theres a few ways to do it.
You can cut up a /29 into 4x /30 networks, or a /28 into 4x /30. You really need to start learning about subnetting for this to properly utilize it.
Another way to do it is to have an ACL on a VLAN or port that prevents anything but 1 designated IP from communicating.

One other thing I'm still a little unsure about is the termination that they provide, from sales I've been told:
Faceplate on wall, Ethernet.
So from my understanding of that is standard cat5e,cat6 rj45. However as they are probably used to dealing with 'proper' I.T techs then in fact this could be SFP?

It could be . . . or they could be giving you an ethernet converter. Just remember with ethernet, going from fiber to copper does not actually require any processing of the packets so it requires a relatively simple and cheap device to do it. OTOH if the ISP uses something like ATM, GPON, Fractional T-carrier, etc. you will need them to provide equipment to convert.

Just make sure that there is both a nice clean supply of AC power as well as an isolated ground. You do have an isolated ground for your equipment rack . . . right? Right?