ByteDance飞书用户9427
KubeAdmiral: Next-Generation Multi-Cluster Orchestration Engine Based on Kubernetes
输入“/”快速插入
KubeAdmiral: Next-Generation Multi-Cluster Orchestration Engine Based on Kubernetes
KubeAdmiral: Next-Generation Multi-Cluster Orchestration Engine Based on Kubernetes
飞书用户9427
飞书用户11738月16日修改
Author: Gary Liu
Since its release in 2014, Kubernetes has become the de facto standard for cloud native orchestration and scheduling systems, delivering substantial value to infrastructure developers around the world. As an increasing number of corporations embrace cloud native technologies and migrate their workloads to Kubernetes, the scale of their clusters grows rapidly. The community edition of Kubernetes, capped at 5000 nodes per cluster, is no longer able to keep up with the scale requirements of large-scale enterprise applications. Moreover, many companies are adopting multi-cloud architectures to achieve cost reduction, increased resource and operational efficiency, geographical disaster recovery, and environment isolation. As a result, the demand for multi-cluster orchestration and scheduling tools is on the rise.
Brief History of Kubernetes at ByteDance
Dedicated Clusters
In the early years of ByteDance's cloud native adoption, each business line operated in separate dedicated clusters due to isolation concerns. However, this led to low resource elasticity and efficiency, observed in several ways:
•
Each business line had to maintain independent resource buffers for scaling and upgrading.
•
Applications were tightly coupled to specific clusters, and manual resource transfer was required to balance resource utilization as applications scale.
•
SRE teams had to deeply understand both the businesses and the clusters in order to manage resources efficiently.
Consequently, this resulted in inefficient resource management and suboptimal overall deployment rate.
KubeFed v2
To address these challenges, the technical infrastructure team at ByteDance started exploring cluster federation based on KubeFed v2 in 2019. The goal is to pool resources across business lines, reduce unnecessary buffers, and improve the efficiency of resource management.
KubeFed v2 introduces the concept of host and member clusters. Users create federated workloads (e.g. FederatedDeployment) in the host cluster, and KubeFed schedules and dispatches workloads in the member clusters based on these federated workloads. Each federated workload contains three primary fields: Template (specifying the workload to be dispatched to member clusters), Placement (designating target member clusters), and Overrides (indicating how the template should be varied in some clusters). For example, the following FederatedDeployments instructs KubeFed to create a Deployment in cluster1 and cluster2 with 2 and 3 replicas respectively.
apiVersion: types.kubefed.k8s.io/v1beta1
kind: FederatedDeployment
metadata:
name: test-deployment
spec:
template:
metadata:
labels:
app: nginx
spec:
replicas: 5
# more Deployment fields...
placement:
clusters:
- name: cluster1
- name: cluster2
overrides:
- clusterName: cluster1
clusterOverrides:
- path: /spec/replicas
value: 2
- clusterName: cluster2
clusterOverrides:
- path: /spec/replicas
value: 3
For Deployment and ReplicaSet, KubeFed supports dividing the desired replicas across multiple clusters based on ReplicaSchedulingPreference (RSP). Users can configure the weights, minimum replicas, and maximum replicas for each cluster, and the RSP controller computes a valid replica distribution and updates the Placement and Overrides fields of FederatedDeployment or FederatedReplicaSet.