Build your own load balancer for your cluster

Load Balancers

Metallb

Kubernetes is wonderful for orchestrating your workloads, but out of the box exposing those workloads to the outside world isn't currently elegant.  There are 2 ways to expose the workloads.  Load balancers and ingress controllers.  On this page, we will discuss one load balancer solution.  Metallb is a Kubernetes load balancer solution.  It runs in 2 modes layer 2 and layer 3.  For the use in this cluster, we will be running it in layer 3 mode.  It uses layer 3 to advertise routes to upstream BGP routers.   In our case, we will be peering to the OpenWRT core router we discussed on another page.    

Metallb is configured using a Kubernetes config map. 

apiVersion: v1
kind: ConfigMap
metadata: 
  namespace: metallb-system
  name: config
data: 
  config: |
    peers:
    - peer-address: 192.168.200.1
      peer-asn: 64501
      my-asn: 64500
    address-pools:
    - name: default
      protocol: bgp
      addresses:
      - 192.168.197.0/24

This is the config map I'm using for metallb in this demo cluster.  Notice the different ASN numbers, this means that metallb and the router are speaking eBGP.   The address space that's being used by metallb to advertise is 192.168.197.0/24.  64500 and 64501 are in the ARIN assigned numbers range for private ASN numbers which is perfectly legal in this situation where these (also private) routes will never be advertised outside this network.  

Quagga/Zebra

Quagga/Zebra has a configuration that looks Cisco "IOSish" 

router bgp 64501
 bgp router-id 192.168.198.254
 neighbor 192.168.201.11 remote-as 64500
 neighbor 192.168.201.12 remote-as 64500

Since I have 3 nodes in my Kubernetes cluster, but one is a master node configured with fewer resources than the service/worker nodes and the master node will never be running any workloads with services to be exposed, I only peer with the 2 service nodes in the cluster.

show bgp summary
IPv4 Unicast Summary:
---------------------
BGP router identifier 192.168.198.254, local AS number 64501
RIB entries 2, using 216 bytes of memory
Peers 2, using 14 KiB of memory
Neighbor        V         AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
192.168.201.11  4 64500   10158   10157        0    0    0 3d12h38m        1
192.168.201.12  4 64500   10158   10157        0    0    0 3d12h38m        2

show ip route bgp
cr1.cluster.sysnetinc.com# show ip route bgp
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel,
       > - selected route, * - FIB route
B>* 192.168.197.0/32 [20/0] via 192.168.201.11, br-LAN201, 3d13h00m
B>* 192.168.197.1/32 [20/0] via 192.168.201.12, br-LAN201, 3d13h00m
B>* 192.168.197.254/32 [20/0] via 192.168.201.12, br-LAN201, 00:00:12

In this simple configuration, metallb advertises each route as a /32.  This is good internally from the point of minimizing the number of hops to get to the actual deployment workload.  In our case, since we're only using   /24 which would be a maximum of 255 routes it's not really something to worry about.  Even the little OpenWRT router like the one in use will handle a few hundred routes.  However, in more complex environments attention would have to be paid to route aggregation techniques.  There are both mechanisms in the upstream routers as well as mechanisms within the metallb config itself to tune the BGP advertisements.

Kubernetes Manifests

The manifests that created 2 of these load balancers are below.

apiVersion: v1
kind: Service
metadata:
  name: test-service
spec:
  selector:
    app: test-deployment
  type: LoadBalancer
  externalTrafficPolicy: Local
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 8000

The above manifest creates a load balancer service named "test-service".   It requests the automatic assignment of an "external" IP address.  It also maps the service port on the deployment from tcp/8000 to the external port tcp/80 on the assigned load balancer.  Finally, it requests that metallb advertises the assigned address directly from the service node using the "spec.externalTrafficPolicy: Local" key. 

apiVersion: v1
kind: Service
metadata:
  name: test-service-3
spec:
  selector:
    app: test-deployment
  type: LoadBalancer
  externalTrafficPolicy: Local
  loadBalancerIP: 192.168.197.254
  ports:
    - name: http
      protocol: TCP
      port: 8080
      targetPort: 8000

The above manifest creates a load balancer service named "test-service-3".  It exposes the same deployment as "test-service" however it exposes it on a different external port and address.  This manifest requests that an explicit IP address be assigned to it using the "spec.loadBalancerIP: 192.168.197.254" key.

Kubernetes Services

kubectl get svc 
NAME             TYPE           CLUSTER-IP       EXTERNAL-IP       PORT(S)          AGE
kubernetes       ClusterIP      10.96.0.1        <none>            443/TCP          47d
nginx            LoadBalancer   10.101.210.28    192.168.197.0     80:30894/TCP     12d
test-service     LoadBalancer   10.101.128.104   192.168.197.1     80:30384/TCP     7d4h
test-service-1   NodePort       10.97.204.46     <none>            80:30614/TCP     5d13h
test-service-3   LoadBalancer   10.101.244.47    192.168.197.254   8080:32336/TCP   6m14s

The above table is from the cluster being used in these examples.  Notice that only the LoadBalancer type services actually have an external IP listed.  The  "nginx" and "test-service" load balancers were automatically assigned the IP addresses in use.  So far, it appears that metallb just works through the pool linearly for auto assignments.   "test-service-3" has an address explicitly assigned via the manifest shown above.