Skip to content

K3s/loxilb service-proxy with calico

Quick Start Guide - K3s, LoxiLB "service-proxy" and Calico

This document will explain how to install a K3s cluster with loxilb in "service-proxy" mode alongside calico networking.

What is service-proxy mode?

service-proxy mode is where kubernetes kube-proxy services are entirely replaced by loxilb for better performance. Users can continue to use their existing networking providers while enjoying streamlined performance and superior feature-set provided by loxilb.

service-proxy

Looking at the left side of the image, you will notice the traffic flow of the packet as it enters the Kubernetes cluster. Kube-proxy, the de-facto networking agent in the Kubernetes which runs on each node of the cluster which monitors the services and translates them to either iptables or IPVS tangible rules. If we talk about the functionality or a cluster with low volume traffic then kube-proxy is fine but when it comes to scalability or a high volume traffic then it acts as a bottle-neck. loxilb "service-proxy" mode works with Flannel/Calico and kube-proxy in IPVS mode only as of now. It inherits the IPVS rules and imports these in it's in-kernel eBPF implementation. Traffic will reach at the interface, will be processed by eBPF and sent directly to the pod or to the other node, bypassing all the layers of Linux networking. This way, all the services, be it External, NodePort or ClusterIP, can be managed through LoxiLB and provide optimal performance for the users. The added benefit for the user's is the fact that there is no need to rip and replace their current networking provider (e.g flannel or calico). Kindly note that Kubernetes network policies can't be supported in this miode currently.

Topology

For quickly bringing up loxilb "service-proxy" in K3s with Calico, we will be deploying a single node k3s cluster (v1.29.3+k3s1) :  

loxilb topology

loxilb and kube-loxilb components run as pods managed by kubernetes in this scenario.

Setup K3s

Configure K3s node

$ curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable traefik \
  --disable servicelb --disable-cloud-controller --kube-proxy-arg proxy-mode=ipvs \
  cloud-provider=external --flannel-backend=none --disable-network-policy --cluster-cidr=10.42.0.0/16 \
  --node-ip=${MASTER_IP} --node-external-ip=${MASTER_IP} \
  --bind-address=${MASTER_IP}" sh -

Deploy calico

K3s uses by default flannel for networking but here we are using calico to provide the same:

sudo kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/tigera-operator.yaml
sudo kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/custom-resources.yaml

Deploy kube-loxilb and loxilb ?

kube-loxilb is used as an operator to manage loxilb. We need to deploy both kube-loxilb and loxilb components in your kubernetes cluster

sudo kubectl apply -f https://raw.githubusercontent.com/loxilb-io/kube-loxilb/main/manifest/service-proxy/kube-loxilb.yml
sudo kubectl apply -f https://raw.githubusercontent.com/loxilb-io/kube-loxilb/main/manifest/service-proxy/loxilb-service-proxy.yml

Check the status

In k3s node:

## Check the pods created
$ sudo kubectl get pods -A
NAMESPACE          NAME                                      READY   STATUS    RESTARTS        AGE
tigera-operator    tigera-operator-689d868448-wwvts          1/1     Running   0               2d23h
calico-system      calico-typha-67d4484996-2cmzs             1/1     Running   0               2d23h
calico-system      calico-node-l8r8b                         1/1     Running   0               2d23h
kube-system        local-path-provisioner-6c86858495-mrtzv   1/1     Running   0               2d23h
calico-system      csi-node-driver-ssbnf                     2/2     Running   0               2d23h
calico-apiserver   calico-apiserver-7dccc79b59-txnl5         1/1     Running   0               2d10h
calico-apiserver   calico-apiserver-7dccc79b59-vk68t         1/1     Running   0               2d10h
calico-system      calico-node-glm64                         1/1     Running   0               2d23h
calico-system      calico-node-hs7pw                         1/1     Running   0               2d23h
calico-system      csi-node-driver-xqjcd                     2/2     Running   0               2d23h
calico-system      calico-typha-67d4484996-wctwv             1/1     Running   0               2d23h
kube-system        kube-loxilb-5fb5566999-4vvls              1/1     Running   0               38h
calico-system      csi-node-driver-hz87c                     2/2     Running   0               2d23h
kube-system        coredns-6799fbcd5-mhgwg                   1/1     Running   0               2d8h
calico-system      calico-kube-controllers-f5c6cdbdc-vztls   1/1     Running   0               32h
calico-system      calico-node-mjjs5                         1/1     Running   0               2d23h
calico-system      csi-node-driver-l5r75                     2/2     Running   0               2d23h
default            iperf1                                    1/1     Running   0               32h
kube-system        metrics-server-54fd9b65b-78mwr            1/1     Running   0               2d23h
kube-system        loxilb-lb-px6th                           1/1     Running   0               20h
In loxilb pod, we can check internal LB rules:
$ sudo kubectl exec -it -n kube-system loxilb-lb-px6th -- loxicmd get lb -o wide
|     EXT IP     | SEC IPS | PORT  | PROTO |             NAME              | MARK | SEL |  MODE   |    ENDPOINT     | EPORT | WEIGHT | STATE  | COUNTERS |
|----------------|---------|-------|-------|-------------------------------|------|-----|---------|-----------------|-------|--------|--------|----------|
| 10.0.2.15      |         | 32598 | tcp   | ipvs_10.0.2.15:32598-tcp      |    0 | rr  | fullnat | 192.168.235.161 |  5001 |      1 | active | 0:0      |
| 10.43.0.10     |         |    53 | tcp   | ipvs_10.43.0.10:53-tcp        |    0 | rr  | default | 192.168.182.39  |    53 |      1 | -      | 0:0      |
| 10.43.0.10     |         |    53 | udp   | ipvs_10.43.0.10:53-udp        |    0 | rr  | default | 192.168.182.39  |    53 |      1 | -      | 6:1149   |
| 10.43.0.10     |         |  9153 | tcp   | ipvs_10.43.0.10:9153-tcp      |    0 | rr  | default | 192.168.182.39  |  9153 |      1 | -      | 0:0      |
| 10.43.0.1      |         |   443 | tcp   | ipvs_10.43.0.1:443-tcp        |    0 | rr  | default | 192.168.80.10   |  6443 |      1 | -      | 0:0      |
| 10.43.182.250  |         |   443 | tcp   | ipvs_10.43.182.250:443-tcp    |    0 | rr  | default | 192.168.182.14  |  5443 |      1 | -      | 0:0      |
|                |         |       |       |                               |      |     |         | 192.168.189.75  |  5443 |      1 | -      | 0:0      |
| 10.43.184.155  |         | 55001 | tcp   | ipvs_10.43.184.155:55001-tcp  |    0 | rr  | default | 192.168.235.161 |  5001 |      1 | -      | 0:0      |
| 10.43.78.171   |         |  5473 | tcp   | ipvs_10.43.78.171:5473-tcp    |    0 | rr  | default | 192.168.80.10   |  5473 |      1 | -      | 0:0      |
|                |         |       |       |                               |      |     |         | 192.168.80.102  |  5473 |      1 | -      | 0:0      |
| 10.43.89.40    |         |   443 | tcp   | ipvs_10.43.89.40:443-tcp      |    0 | rr  | default | 192.168.219.68  | 10250 |      1 | -      | 0:0      |
| 192.168.219.64 |         | 32598 | tcp   | ipvs_192.168.219.64:32598-tcp |    0 | rr  | fullnat | 192.168.235.161 |  5001 |      1 | active | 0:0      |
| 192.168.80.10  |         | 32598 | tcp   | ipvs_192.168.80.10:32598-tcp  |    0 | rr  | fullnat | 192.168.235.161 |  5001 |      1 | active | 0:0      |
| 192.168.80.20  |         | 32598 | tcp   | ipvs_192.168.80.20:32598-tcp  |    0 | rr  | fullnat | 192.168.235.161 |  5001 |      1 | active | 0:0      |
| 192.168.80.20  |         | 55001 | tcp   | default_iperf-service         |    0 | rr  | onearm  | 192.168.80.101  | 32598 |      1 | -      | 0:0      |

Deploy a sample service

To deploy a sample service, we can create service as usual in Kubernetes with few extra annotations as follows :

sudo kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: iperf-service
  annotations:
   loxilb.io/lbmode: "onearm" 
   loxilb.io/staticIP: "192.168.80.20"
spec:
  externalTrafficPolicy: Local
  loadBalancerClass: loxilb.io/loxilb
  selector:
    what: perf-test
  ports:
    - port: 55001
      targetPort: 5001
  type: LoadBalancer
---
apiVersion: v1
kind: Pod
metadata:
  name: iperf1
  labels:
    what: perf-test
spec:
  containers:
    - name: iperf
      image: ghcr.io/nicolaka/netshoot:latest
      command:
        - iperf
        - "-s"
      ports:
        - containerPort: 5001
EOF

Check the service created :

$ sudo kubectl get svc
NAME            TYPE           CLUSTER-IP      EXTERNAL-IP         PORT(S)           AGE
kubernetes      ClusterIP      10.43.0.1       <none>              443/TCP           3d1h
iperf-service   LoadBalancer   10.43.131.107   llb-192.168.80.20   55001:31181/TCP   9m14s

Test the service created (from a host outside the cluster) :

## Using service VIP
$ iperf -c 192.168.80.20 -p 55001  -i1 -t3
------------------------------------------------------------
Client connecting to 192.168.80.20, TCP port 55001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  1] local 192.168.80.80 port 58936 connected with 192.168.80.20 port 55001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-1.0000 sec   282 MBytes  2.36 Gbits/sec
[  1] 1.0000-2.0000 sec   276 MBytes  2.31 Gbits/sec
[  1] 2.0000-3.0000 sec   279 MBytes  2.34 Gbits/sec

## Using node-port
$ iperf -c 192.168.80.100 -p 31181  -i1 -t10
------------------------------------------------------------
Client connecting to 192.168.80.100, TCP port 31181
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  1] local 192.168.80.80 port 43208 connected with 192.168.80.100 port 31181
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-1.0000 sec   612 MBytes  5.14 Gbits/sec
[  1] 1.0000-2.0000 sec   598 MBytes  5.02 Gbits/sec
[  1] 2.0000-3.0000 sec   617 MBytes  5.17 Gbits/sec
[  1] 3.0000-4.0000 sec   600 MBytes  5.04 Gbits/sec
[  1] 4.0000-5.0000 sec   630 MBytes  5.28 Gbits/sec
[  1] 5.0000-6.0000 sec   699 MBytes  5.86 Gbits/sec
[  1] 6.0000-7.0000 sec   682 MBytes  5.72 Gbits/sec

For more detailed performance comparison with other solutions, kindly follow this blog and for more detailed information on incluster deployment of loxilb with bgp in a full-blown cluster, kindly follow this blog.