K3s/loxilb in-cluster mode

Quick Start Guide with K3s and LoxiLB in-cluster mode

This document will explain how to install a K3s cluster with loxilb as a serviceLB provider running in-cluster mode.

Topology

For quickly bringing up loxilb in-cluster and K3s, we will be deploying all components in a single node :

loxilb topology

loxilb and kube-loxilb components run as pods managed by kubernetes in this scenario.

Setup K3s

# K3s installation
$ curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --disable traefik --disable servicelb --disable-cloud-controller --kube-proxy-arg metrics-bind-address=0.0.0.0 --kubelet-arg cloud-provider=external" K3S_KUBECONFIG_MODE="644" sh -

# Remove taints in k3s if any (usually happens if started without cloud-manager)
$ sudo kubectl taint nodes --all node.cloudprovider.kubernetes.io/uninitialized=false:NoSchedule-

How to deploy loxilb ?

loxilb can be deloyed by using the following command in the K3s node

sudo kubectl apply -f https://raw.githubusercontent.com/loxilb-io/loxilb/main/cicd/k3s-incluster/loxilb.yml

The description of important loxilb args in the manifest file are as follows -

--bgp : Indicates that loxilb will be running with bgp instance and will be advertising the service IP to the external peer or loxilb-peer.
--egr-hooks: It is required for those cases in which workloads can be scheduled in the same nodes as loxilb nodes (most of the cases).
--blacklist=cni[0-9a-z]|veth.|flannel.|cali.|tunl.|vxlan[.]calico: It is mandatory for running in in-cluster mode. As loxilb attaches it's ebpf programs on all the interfaces but since we running it in the default namespace then all the interfaces including CNI interfaces will be exposed and loxilb will attach it's ebpf program in those interfaces which is definitely not desired. So, user needs to mention a regex for  excluding all those interfaces.
--fallback: It is an optional argument and ensures that egress traffic from the cluster to the internet or external destinations defaults to system masquerade rules if no other rules apply. LoxiLB can also be used as a HA capable egress on its own.

How to deploy kube-loxilb ?

kube-loxilb is used as an operator to manage loxilb.

wget https://raw.githubusercontent.com/loxilb-io/loxilb/main/cicd/k3s-incluster/kube-loxilb.yml

kube-loxilb.yaml

       args:
         #- --loxiURL=http://172.17.0.2:11111
         - --externalCIDR=192.168.82.100/32
         - --setRoles=0.0.0.0
         #- --monitor
         #- --setBGP

In the above snippet, loxiURL is commented out which denotes to utilize in-cluster mode to discover loxilb pods automatically. External CIDR represents the IP pool from where serviceLB VIP will be allocated.

Apply after making changes (if any) :

sudo kubectl apply -f kube-loxilb.yaml

Create the service

sudo kubectl apply -f https://raw.githubusercontent.com/loxilb-io/loxilb/main/cicd/k3s-incluster/tcp-svc-lb.yml

Check the status

In k3s node:

## Check the pods created
$ sudo kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE
kube-system   local-path-provisioner-6c86858495-snvcm   1/1     Running   0          4m37s
kube-system   coredns-6799fbcd5-cpj6x                   1/1     Running   0          4m37s
kube-system   metrics-server-67c658944b-42ptz           1/1     Running   0          4m37s
kube-system   loxilb-lb-8l85d                           1/1     Running   0          3m40s
kube-system   kube-loxilb-6f44cdcdf5-5fdtl              1/1     Running   0          2m19s
default       tcp-onearm-test                           1/1     Running   0          88s


## Check the services created
$ sudo kubectl get svc
NAME            TYPE           CLUSTER-IP    EXTERNAL-IP          PORT(S)           AGE
kubernetes      ClusterIP      10.43.0.1     <none>               443/TCP           5m12s
tcp-lb-onearm   LoadBalancer   10.43.47.60   llb-192.168.82.100   56002:30001/TCP   108s

In loxilb pod, we can check internal LB rules:

$ sudo kubectl exec -it -n kube-system loxilb-lb-8l85d -- loxicmd get lb -o wide
|     EXT IP     | SEC IPS | PORT  | PROTO |         NAME          | MARK | SEL |  MODE  | ENDPOINT  | EPORT | WEIGHT | STATE  | COUNTERS |
|----------------|---------|-------|-------|-----------------------|------|-----|--------|-----------|-------|--------|--------|----------|
| 192.168.82.100 |         | 56002 | tcp   | default_tcp-lb-onearm |    0 | rr  | onearm | 10.0.2.15 | 30001 |      1 | active | 39:2874  |

Connect from host/client

$ curl http://192.168.82.100:56002
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

For more detailed information on incluster deployment of loxilb with bgp in a full-blown cluster, kindly follow this blog.