We are back with our BGP configuration. We have set up our ECMP route or the
Equal Cost Multiple Path route, so our upload will get balanced,
because the OSPF will distribute the default route in our local network as
it receives it from BGP. But the download will still use a single, most optimal ISP connection.
The BGP peering to our ISP is running and we are advertising our network. What are we doing next?
Our third router is going to be our local gateway to the servers, so now we will
assign one public IP to a dummy interface that will represent or simulate a server on our
side and make it reachable from the internet. So if we run a traceroute from the ISP side,
we can see that our server address is reachable. As a sidenote, if we run traceroute
to a non-existing address from /25 range we just created then traffic will go through our whole
network. But if we pick address from the rest of the range then edge router is dropping the packet.
So why is this happening if there is a blackhole route?
So let's look at routing table on edge1. So as you can see we have a /25 route,
which is more preferred than /24, so when routing traffic this /24 route will be ignored. People
sometimes seem to forget that additional care must be taken to protect the router subnet in
local network. For example, if you have servers that need to be accessed from internet and are
scattered around your local network then better approach is to use /32 addresses for the servers.
Alright, so our server is reachable from the internet, but let's say that one of our ISP
connections is preferable. Can we improve our configuration? Well, we cannot influence it
directly, but we can steer incoming traffic path by trying to manipulate BGP attributes and hope
for the best. But to understand how to do that we need to look at the best path selection. So if
you look at the hierarchy, first one is weight which is always local to the router, the next
one is LOCAL_PREF which can be used only within AS, so our best bet is on AS_PATH manipulation.
Let's see what's happened on ISP side. We can see that AS path is prepended and the route with
the shortest AS path is more preferred. Is there anything else that we can do
in regards of load balancing in this case? For this particular network no, but if you
have large enough network, we can try to split it in half and set load sharing. But typically
ISPs do not accept smaller networks than /24. So this configuration is optimal for our setup,
but let's pretend that we do have another network and further set up the load sharing.
Now run pings to our servers. And check what is going on on our Edge routers. If we look at torch,
you can see that traffic from the ISP is balanced. Traffic from our side is also balanced by ISP,
but it is also possible that it will flow over one link, and because of this there will be asymmetric
routing, and in case of asymmetric routing you must be very careful when setting up the firewall.
Alright, because asymmetric routing can take place in this setup, make sure you disable
connection-tracking on the edge routers, as the stateful firewall will no longer be able to
correctly track connections. Because a connection can be established through one ISP link and then
continue through the other one, and in that case those connections will be considered as invalid.
Our current setup works great in a controlled environment, but in real life ISPs have their
own policies and can even strip all your prepends. For some of the ISPs you can
influence decision based on policies they have set up. This is typically done by advertising
specific BGP communities. But more on that later, let's complicate things a bit more.
Let's assume that edge1 and edge2 are connected to different ISP branches and edge1 connection
can reach one specific network faster than when using edge2 connection. ISP will send
both these networks to edge1 router. So we enable networks on ISP and check the
routing table on edge1 and edge2. And we see received Networks.
Now we have the routes on edge1 and one route on edge2,
but if the traffic from local servers to the second network will reach the edge2
first, because of the ECMP, edge2 upstream link will be used anyway,
because edge2 router does not know anything about the network and default route is used.
Okay and how could we fix this? Well there's an option to redistribute
BGP routes with OSPF, but it is a bad idea. It can end very badly in setups where upstream is
sending the full feed. Better approach is to set up internal BGP session between edge routers.
So I see you added loopback addresses on both of the edge routers. How so?
Loopbacks are best practice when setting connections in local network. This allows
to keep iBGP connection up even if some of the local links fail. OSPF will find and
re-route connection to other accessible links. Don't forget to set up correct local role.
Now we can see that session is up. And if we look at the routing table,
we can see that nothing is sent. Remember how I talked about router
IDs and BGP instances? I said different IDs intentionally to fix this on edge1
router I will set the correct router ID. Now, second network from edge2 will be
routed to edge1 and then out to the upstream. However, we want to re-route the first network
also over the edge1. We are setting higher local pref attribute to set higher preference on edge1.
As you can see local pref is now set to 200. If we look at the routing table we
can see that this didn't work. This is because the best path selection works only on routers
received by the same BGP instance. And to fix that, we need to change router ID. And now
we check the routing table again, and we can see that it's routed properly, as we wanted.
So now let's see what we can do if ISP is stripping our prepend.
Okay, now we can see that on ISP side our prepend is removed. Typically service providers are using
BGP community's attribute to allow client to change routing policy, for example,
prepend once, prepend twice, or prefer locally, and so on. Community is 4 byte
long attribute split in two 2 by sections. First part is typically the AS number. Let's
say that our ISP has the last community to change local preference in in its network.
We are setting a filter to match if community sent and adjust local pref value.
Now change your edge routers to send community instead of prepend.
As you can see, best route is now selected with the highest local pref,
and the same for other network. Our setup is looking pretty good,
but currently we're using two networks and a default route with our BGP. What if we wanted
to do do some sort of advanced setup and could we receive the whole BGP feed in our setup?
Okay, sure, let's try it. Now you can see we are receiving the whole
routing table on both edge routers and this table is also exchanged between both of them. Edge2
router loaded 1.9 million routes and, as you can see from system resources, it barely used any RAM.
Cool, we should test our routers a little further. Can we generate some traffic?
Sure.
Alright, so the traffic is now running through both of the 10G ISP connections. We're going
to wrap this up right here. Māris, thank you for this massive configuration. We're
going to export the configuration from all of these routers and attach it below the video,
I hope you find this useful, and we'll see you next time.