Cài đặt k8s cluster sử dụng kubeadm và HAproxy

I. Cài đặt HAproxy, cấu hình cho Api-server

Đối với ubuntu sử dụng command: apt install haproxy -y

Sửa config của HAproxy như sau:

global
#log /dev/log local0 warning
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats
defaults
log global
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
frontend kube-apiserver
bind *:6443
mode tcp
option tcplog
default_backend kube-apiserver
backend kube-apiserver
mode tcp
option tcplog
option tcp-check
balance roundrobin
default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
server master2 192.168.200.100:6443 check # Replace the IP address with your own.
server master1 192.168.200.101:6443 check # Replace the IP address with your own.

II. Cài đặt k8s cluster

1. Enable iptables Bridged Traffic on all the Nodes

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# sysctl params required by setup, params persist across reboots
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF

# Apply sysctl params without reboot
sudo sysctl --system

 

2. Disable swap on all the Nodes

sudo swapoff -a
(crontab -l 2>/dev/null; echo "@reboot /sbin/swapoff -a") | crontab - || true

 

3. Install kubeadm – kubelet – kubectl

#!/bin/bash

sudo apt-get update

sudo apt-get install -y apt-transport-https ca-certificates curl

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update

sudo apt-get install -y kubelet kubeadm kubectl

sudo apt-mark hold kubelet kubeadm kubectl

 

4. Install CRI, cụ thể ở đâu sẽ cài đặt containerd

#!/bin/bash

cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf

overlay

br_netfilter

EOF

sudo modprobe overlay

sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf

bridge.bridge-nf-call-iptables  = 1

ipv4.ip_forward                 = 1

bridge.bridge-nf-call-ip6tables = 1

EOF

sudo sysctl --system

sudo apt install containerd -y

mkdir /etc/containerd

containerd config default > /etc/containerd/config.toml

sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml

systemctl restart containerd

 

5. Tạo cluster trên 1 master, trong command sẽ phải gọi đến server haproxy vừa thiết lập ở trên.

kubeadm init --control-plane-endpoint=haproxy:6443 --upload-certs --pod-network-cidr=192.168.200.0/24

export KUBECONFIG=/etc/kubernetes/admin.conf

kubectl get pod -n kube-system

  • Để Join server master khác và các server worker còn lại vào trong cluster:

+ check token join: kubeadm token list

+ lấy token: kubeadm token create

+ lấy cert: kubeadm init phase upload-certs --upload-certs

+ lấy discovery-token-ca-cert-hash:openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \

+ Lấy control-plane –certificate-key: openssl dgst -sha256 -hex | sed 's/^.* //'

Ví dụ:

  • Join master:

kubeadm join haproxy:6443 –token oylvmu.12pwimke0blaaeji –discovery-token-ca-cert-hash sha256:303a791ef0bdaeb3a3b54ca80f8f4831dff6d0bb1c43c664d9102c9ec569ef61 –control-plane –certificate-key 3b4da12cd25d1c1e7a47abcb908c73405c4abd5e542f99692d8f1b9d368d307a

  • Join worker

kubeadm join haproxy:6443 –token t7a9sr.yamaink129rswuix –discovery-token-ca-cert-hash sha256:303a791ef0bdaeb3a3b54ca80f8f4831dff6d0bb1c43c664d9102c9ec569ef61

  • mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

 

6. Cài đặt cni

Kubeadm không cấu hình bất kỳ plugin mạng nào. Bạn cần cài đặt plugin mạng mà bạn chọn cho network của kubernetes pod và kích hoạt chính sách mạng. Tôi đang sử dụng plugin mạng cilium cho thiết lập này.

helm repo add cilium https://helm.cilium.io/

helm install cilium cilium/cilium –version 1.11.6 –namespace kube-system