K8s: Introducing SCTP Multihoming functionality with LoxiLB

SCTP (Stream Control Transmission Protocol) multi-homing refers to the capability of a SCTP-enabled endpoint to establish multiple network connections with different IP addresses. SCTP is a transport layer protocol that provides reliable, message-oriented communication between two endpoints. Unlike TCP, which supports only a single connection between two hosts, SCTP allows for the establishment of multiple simultaneous connections, known as associations.

Multi-homing in SCTP enables an endpoint to maintain redundant network paths and IP addresses. This redundancy can improve the overall reliability and fault tolerance of the communication. If one network path or IP address becomes unavailable or experiences issues, the endpoint can switch to an alternate path or address without disrupting the communication.

Importance of SCTP multi-homing in 5G /Telco networks

SCTP multi-homing in 5G networks enhances reliability, enables load balancing, leverages path diversity, facilitates seamless handovers, and provides redundancy and resilience. These features are vital for delivering the performance and reliability required by the diverse range of applications and services supported by 5G technology. Here are some key reasons why SCTP multi-homing is important in 5G:

Improved Reliability: 5G networks are expected to support a wide range of applications and services with diverse requirements. By establishing multiple network connections through SCTP multi-homing, 5G devices can ensure reliable communication even in the presence of network failures or disruptions. If one network path or IP address becomes unavailable, the device can seamlessly switch to an alternative path, ensuring continuous connectivity.

Load Balancing: With the proliferation of data-intensive applications and the growing number of connected devices in 5G networks, distributing network traffic efficiently becomes crucial. SCTP multi-homing allows for load balancing by enabling devices to distribute data across multiple available paths. This helps in optimizing network resource utilization and avoiding congestion on any single path.

Path Diversity: 5G networks aim to provide low-latency and high-bandwidth connections to support applications like autonomous vehicles, remote surgery, and augmented reality. SCTP multi-homing enables devices to leverage path diversity by selecting the most suitable network path based on factors like latency, bandwidth, and network conditions. This ensures that applications receive the best possible performance by dynamically adapting to the available network resources.

Seamless Handovers: 5G networks support mobility, allowing devices to switch between different base stations or network slices while maintaining continuous connectivity. SCTP multi-homing facilitates seamless handovers by enabling devices to establish and maintain multiple connections simultaneously. This ensures uninterrupted communication during handover events, reducing latency and minimizing service disruptions.

Redundancy and Resilience: 5G networks are designed to provide high availability and fault tolerance. SCTP multi-homing contributes to this by offering redundancy through multiple network paths and IP addresses. In the event of network failures, such as link or node outages, 5G devices can switch to alternative paths or addresses, ensuring uninterrupted communication and improved network resilience.

SCTP Multihoming can be deployed in various scenarios, but its bespoke use case is SCTP connection between gNB (RAN) and AMF in 5G Core running in the cloud.

SCTP multi-homing for micro-services

Due to the presence of various layers of service abstractions in k8s, NAT’ing and Proxy’ing is inevitable in almost all k8s networking scenarios. Hence as a first step to bring multi-homing to cloud-native micro-service world, we need to repurpose existing Kubernetes services abstractions and introduce this feature in the most non-intrusive way possible.

SCTP Multihoming with NAT can be deployed in various ways. There are many general discussions and proposals on how to implement SCTP Multihoming with NAT, but none of them address the cloud-native aspect. LoxiLB community has been busy attempting to solve this issue. Mainly, two requirements have come out of this :

1. Client <-> Multi-homed LB/NAT, and

2. Client<->LB/NAT<->Endpoint end-to-end multi-homed scenario

In this blog, we will discuss and focus on how LoxiLB implements client to LB multi-homing. The LB<->Endpoint multi-homing is not particularly critical in k8s due to the fact that the LB and end-points are usually located in the same cluster and k8s internal networking implementations are robust enough to deal with any failovers. Plus there is an added complexity to enable multi-homing at endpoint pods. Since we need to have multiple interfaces per pod, this would require adding meta-plugins like Multus into the complexity mix. Hence, gains from LB <-> endpoint multi-homing part are seemingly non-existent.

Client to multi-homed LB

Considerations and Limitations (for LoxiLB’s implementation)

• The backend endpoint will be single homed.

• LB Multi-homing will work in fullNAT mode only.

• A maximum of 3 secondary IPs can be configured with service rules.

Client to multi-homed LB means that LB will have a primary IP address and may have multiple secondary IP addresses for a SCTP association. A client will use primary service IP addresses to start a SCTP association.

Service rule for LB SCTP Multi-homing

For hosting Multi-homed SCTP service, LB rules need to be created using kube-loxilb. A LB rule will contain primary service IP and a list of secondary service IP addresses, along with the other parameters such as endpoints and weights. In case there is a failure to reach the primary service IP, the client can use secondary service IP for communication without any service interruption as per design of SCTP protocol.

Configuring SCTP Multihoming service with kube-loxilb

kube-loxilb implements the service LB spec of Kubernetes. It has also been enhanced to fully support deployment of multi-homed SCTP services in Kubernetes clusters. To run kube-loxilb daemonset as part of kube-system, we need to do the following:

• Fetch the kube-loxilb yaml

$ wget https://github.com/loxilb-io/kube-loxilb/raw/main/manifest/kube-loxilb.yaml

• Edit (or uncomment) the contents related to multi-homing (as per need)

args:
        - --externalCIDR=123.123.123.1/24
        - --externalSecondaryCIDRs=124.124.124.1/24,125.125.125.1/24
        - --setBGP

where,

externalCIDR : Denotes primary IP address for the service

externalSecondaryCIDRs : Denotes secondary/backup IP addresses for the services

setBGP : Advertise the service using BGP

• Apply the modified yaml

$ kubectl apply -f kube-loxilb.yaml

• Check kube-loxilb is working fine

$ kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE
kube-system   local-path-provisioner-84bb864455-qr2l5   1/1     Running   0          50m
kube-system   coredns-7796b77cd4-nd2p2                  1/1     Running   0          50m
kube-system   metrics-server-ff9dbcb6c-8sgvd            1/1     Running   0          50m
kube-system   kube-loxilb-f9c5cd878-25ppx               1/1     Running   0          49m
kube-system   kube-multus-ds-amd64-7kbms                1/1     Running   0          20m
default       pod-01                                    1/1     Running   0          6m41s

Now that the kube-loxilb component is up and running, it is a breeze to spin up the actual multi-homed SCTP service. The following is a sample config for such a service :

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Service
metadata:
  name: sctp-lb1
  annotations:
    loxilb.io/num-secondary-networks: "2"
    loxilb.io/liveness: "yes"
    loixlb.io/lbmode: "fullnat"
spec:
  loadBalancerClass: loxilb.io/loxilb
  selector:
    what: sctp-test
  ports:
    - port: 55002
      protocol: SCTP
      targetPort: 9999
  type: LoadBalancer
---
apiVersion: v1
kind: Pod
metadata:
  name: sctp-test
  labels:
    what: sctp-test
spec:
  containers:
    - name: sctp-test
      image: alpine/socat
      command: [ "sh", "-c"]
      args:
      - while true; do
          socat -v -T2 sctp-l:9999,reuseaddr,fork system:"echo 'server1'; cat";
          sleep 20;
        done;
      ports:
        - containerPort: 9999
EOF

It is a pretty straightforward affair but the annotation loxilb.io/num-secondary-networks: "2" needs special attention. This denotes the number of secondary networks to use as backup of a multi-homed service. Based on this kube-loxilb will do secondary IP address allocation (IPAM) from the CIDR ranges defined as “externalSecondaryCIDRs” in kube-loxilb.yaml

Verifying created services in K8s

$ kubectl get svc
NAME            TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)            AGE
kubernetes      ClusterIP      10.43.0.1       <none>          443/TCP            3m53s
nginx-lb1       LoadBalancer   10.43.60.79     123.123.123.1   55002:31706/TCP    2m23s
sctp-lb1        LoadBalancer   10.43.77.241    123.123.123.1   55002:32153/SCTP   49s

Verifying service LB Rules with loxicmd

$ docker exec -it llb1 loxicmd get lb  -o wide
|  EXTERNAL IP  |        SECONDARY IPS         | PORT  | PROTOCOL | BLOCK | SELECT |  MODE   | ENDPOINT IP  | TARGET PORT | WEIGHT | STATE  |
|---------------|------------------------------|-------|----------|-------|--------|---------|--------------|-------------|--------|--------|
| 123.123.123.1 |                              |    80 | tcp      |     0 | rr     | default | 172.30.1.177 |       30820 |     10 | -      |
| 123.123.123.1 | 124.124.124.1, 125.125.125.1 | 55002 | sctp     |     0 | rr     | fullnat | 172.30.1.177 |       32153 |     10 | active |
| 123.123.123.1 |                              | 55002 | tcp      |     0 | rr     | default | 172.30.1.177 |       31706 |     10 | -      |
| 20.20.20.1    |                              |  2020 | tcp      |     0 | rr     | fullnat | 31.31.31.1   |        8080 |      1 | -      |
|               |                              |       |          |       |        |         | 32.32.32.1   |        8080 |      1 | -      |
|               |                              |       |          |       |        |         | 33.33.33.1   |        8080 |      1 | -      |

Deploying multiple LoxiLB instances

Usually load-balancers/proxies are deployed as clusters themselves to eliminate any single-point of failure. LoxiLB supports various cluster modes e.g. maglev clustering or state-aware cluster set. For SCTP-multihoming, as we will see in the forthcoming sections, it would be practically impossible to do maglev clustering due to multiple 5-tuple-sets belonging to the same service. Hence, LoxiLB instances should be deployed as a state-aware cluster set. SCTP multi-homing states are shared across LoxiLB nodes with strong eventual consistency.

Secondary Service IP advertisement through BGP

LoxiLB will also advertise secondary service IP addresses along with primary service IP address, if LB rule is configured with the “bgp” flag.

When LoxiLB is deployed as a cluster, LoxiLB instances sync all the tracked connections within the cluster nodes and all the multihoming session information will also be synced. Users can configure the cluster in a way to use different LoxiLB instances for primary or secondary network paths also. In that case, LoxiLB instances will advertise primary and secondary IP addresses with different priorities so that different network paths are used.

SCTP Connection establishment in fullNAT mode

Currently, SCTP multi-homing is supported in fullNAT mode only. We would encourage the readers to get acquainted with all the modes in which LoxiLB works here.

A client will initiate the SCTP association by sending the INIT packet to LoxiLB. In LoxiLB fullNAT mode, LoxiLB will replace destination IP addresses in the INIT packet with the primary service IP, add a list of secondary service IPs in the INIT packet and then forward the INIT packet to the endpoint. In response, Endpoint will send INIT ACK to LoxiLB with a primary service IP address. LoxiLB will replace primary service IP with client’s primary IP address, append list of secondary IP addresses in the INIT ACK packet and will pass it to the client.

At this point, client, backend endpoint and LoxiLB, all are aware of primary and secondary IP addresses related to the SCTP association.

After the connection has been established, LoxiLB will be ready to receive SCTP multi-homed traffic as it will keep track of all primary and secondary network paths.

DATA Traffic flow in SCTP multi-homing

Initially, Client will send all the data traffic using the primary path. The traffic flow will be the same as it flows in the SCTP single-homed case with fullNAT. No special handling is required for DATA packets. Below Diagram shows the DATA and SACK traffic flow with SACK.

HEARTBEAT Management in SCTP multi-homing

Client and endpoints will send HEARTBEAT messages using primary and secondary IP addresses to check the path availability.

When the HEARTBEAT message comes then the endpoint expects HEARTBEAT ACK to come back from the same IP address it had sent the HEARTBEAT to. LoxiLB will handle the HEARTBEAT message coming from the client or endpoint as other packets. HEARTBEAT messages need no special handling. It means that HEARTBEAT messages will be forwarded to the endpoint by LoxiLB after doing fullNAT. LoxiLB fullNAT mode will make sure to know about the destination IP address client has used in the HEARTBEAT message when it will receive the HEARTBEAT ACK from the endpoint. HEARTBEAT messages from the endpoints to the clients also need no special handling. They will be handled the same way.

PATH failover in SCTP multi-homing

As explained in the above section, SCTP endpoints exchange HEARTBEAT messages for all network paths to check their health. If an endpoint fails to respond to the HEARTBEAT messages or SACK for the DATA messages (including re-transmissions), then that path is considered inactive. If the inactive path is the primary path, then initiating the SCTP endpoint chooses one of the secondary paths as the primary path.

The diagram below shows how LoxiLB will take care of the path failover.

SHUTDOWN in SCTP multi-homing

Any of the SCTP endpoints can initiate SCTP SHUTDOWN. SCTP endpoint sends the SHUTDOWN packet using the primary path only. It may happen, that primary IP address now was a secondary IP address then. As LoxiLB has kept track of all the possible network paths, the SHUTDOWN packet will be forwarded to the other endpoint. LoxiLB does not need to validate if SHUTDOWN was initiated with the then primary IP address or not, only other endpoints will do that kind of validation. It will be handled the same way as DATA packets or HEARTBEAT packets are handled.

Conclusion

LoxiLB is one of the first to bring SCTP multi-homing features to k8s. It will no doubt add to the resilience, stability and scalability of SCTP based micro-services in cloud-native environments (not restricted to 5G apps) . In a future blog, we will further explore how to use SCTP (w/ multi-homing) using Multus as well as its usage on Amazon EKS. Interested readers can also dig deeper by going though LoxiLB's sctp-ci testing scripts which creates a complete k3s test-bed sctp topology in a single node.

** If you like this article and would love to see more new k8s content/features, kindly *star* our github repo.