KubeGateway: A customized seven-layer Load Balancer for kube-apiserver
输入“/”快速插入
KubeGateway: A customized seven-layer Load Balancer for kube-apiserver
KubeGateway: A customized seven-layer Load Balancer for kube-apiserver
Blog Author: Jun Zhang
KubeGateway is a seven-layer load balancer specially customized by ByteDance for kube-apiserver traffic characteristics. It completely solves the problem of kube-apiserver load imbalance. For the first time in the community, it realizes complete governance of kube-apiserver requests, including request routing, shunting, rate limiting, downgrade etc, which significantly improves the availability of Kubernetes cluster.
1.
Background
In the Kubernetes cluster, kube-apiserver is the entry of the entire cluster. Any user or program to add, delete and change the resources needs to communicate with the kube-apiserver. So the high availability of the entire cluster depends on the availability of kube-apiserver. In order to achieve its high availability, multiple kube-apiserver instances are usually deployed behind a Load Balancer(hereinafter referred to as LB) because kube-apiserver is essentially a stateless server.
In order to ensure the security of the cluster, kube-apiserver authenticates and authorizes the API requests, where authentication is to identify the user and authorization is to grant permission to access. Kubernetes supports multiple authentication strategies , such as Bootstrap Token, Service Account Token, OpenID Connect Token , mTLS authentication, etc.
Currently, mTLS is the most commonly used authentication policy. TLS authentication requires client certificate in request correctly pass through LB to kube-apiserver, but the traditional seven-layer LB can not do this, certificate will be dropped in the forwarding process, resulting in kube-apiserver authentication failing .
Therefore , kube-apiserver load balance is generally a four-layer solution, such as LVS , public cloud SLB or four-layer load balance of nginx/haproxy.
Four-layer load balancer works in the fourth OSI layer - transport layer, using NAT technology for proxy forwarding
Seven-Layer load balancer works in the seventh OSI layer - application layer, generally based on the request URL the address of the proxy forwarding.
But using four layers LB raises additional problems, as follows:
1.
Load imbalance: since kube-apiserver and client are communicating with each others by the HTTP2 protocol, multiple requests will reuse the same underlying TCP connection and not disconnect for a long time. When a kube-apiserver instance is restarted, it is easy to cause load imbalance. The late startup kube-apiserver instance will have only a small number of requests over a long period of time . In extreme cases, instances with higher loads will OOM and even cause avalanches.
2.
Lack of flexibility in request governance: four-layer load balance working at the transport layer, it is only responsible for message delivery, but cannot handle the information of the application layer's HTTP protocol, so it lacks "flexibility" and "intelligence" for request governance compared to seven-layer load balance. It can not develop flexible load balance and routing policies based on the request content, such as verb , url and other fields, nor can set a rate limit on the request level.
There are some related efforts in the community to try to solve the above problems, but none of them have cured the problem :
With Cloud Native technology development, more than 95% of ByteDance's business is currently running on Kubernetes clusters, which put forward higher requirements for high availability. In the production environment, we have also encountered many incidents caused by kube-apiserver load imbalance or lack of request governance capabilities. Facing the above problems, we developed a seven-layer load balancer specially customized for kube-apiserver traffic characteristics named KubeGateway.
2.
Architecture Design
KubeGateway is a seven-layer load balancer to forward kube-apiserver requests. It has the following characteristics:
1.
Completely transparent for clients : Client can access kube-apiserver through KubeGateway without any modification.
2.
Support proxy multiple K8S clusters of requests at the same time: different K8S clusters are distinguished by domain name or virtual address.
3.
Request level load balance achieves fast and efficient load balance, completely solving the problem of kube-apiserver load imbalance.
4.
Flexible routing policies: KubeGateway routes according to request information, including but not limited to resource/verb/user/namespace/groupversion, etc. Provides basic capabilities for kube- apiserver grouping and implements it at low operation and maintenance cost, improves cluster stability.
5.
Cloud Native configuration management: It provides configuration in K8s standard API form and supports hot updating.
6.
Support rate limit, downgrade, dynamic service discovery, elegant exit, upstream anomaly detection and other gateway and general capabilities.
KubeGateway provides configuration management in K8s standard API form, mainly providing management for routing forwarding rules , upstream cluster kube-apiserver address , cluster certificates , flow control and other request governance configuration information. Its proxy flow, as shown in the figure below, is mainly divided into five steps: request parsing, route matching, authentication, traffic governance and request forwarding.