I. Cài đặt HAproxy, cấu hình cho Api-server
Đối với ubuntu sử dụng command: apt install haproxy -y
Sửa config của HAproxy như sau:
global
#log /dev/log local0 warning
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats
defaults
log global
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
frontend kube-apiserver
bind *:6443
mode tcp
option tcplog
default_backend kube-apiserver
backend kube-apiserver
mode tcp
option tcplog
option tcp-check
balance roundrobin
default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
server master2 192.168.200.100:6443 check # Replace the IP address with your own.
server master1 192.168.200.101:6443 check # Replace the IP address with your own.
II. Cài đặt k8s cluster
1. Enable iptables Bridged Traffic on all the Nodes
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter
# sysctl params required by setup, params persist across reboots
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
# Apply sysctl params without reboot
sudo sysctl --system
2. Disable swap on all the Nodes
sudo swapoff -a
(crontab -l 2>/dev/null; echo "@reboot /sbin/swapoff -a") | crontab - || true
3. Install kubeadm – kubelet – kubectl
#!/bin/bash
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl
mkdir -p /etc/apt/keyrings/
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
4. Install CRI, cụ thể ở đâu sẽ cài đặt containerd
#!/bin/bash
cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
bridge.bridge-nf-call-iptables = 1
ipv4.ip_forward = 1
bridge.bridge-nf-call-ip6tables = 1
EOF
sudo sysctl --system
sudo apt install containerd -y
mkdir /etc/containerd
containerd config default > /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml
systemctl restart containerd
5. Tạo cluster trên 1 master, trong command sẽ phải gọi đến server haproxy vừa thiết lập ở trên.
kubeadm init --control-plane-endpoint=haproxy:6443 --upload-certs --pod-network-cidr=192.168.200.0/24
export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl get pod -n kube-system
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
6. Cài đặt cni
Kubeadm không cấu hình bất kỳ plugin mạng nào. Bạn cần cài đặt plugin mạng mà bạn chọn cho network của kubernetes pod và kích hoạt chính sách mạng. Tôi đang sử dụng plugin mạng cilium cho thiết lập này.
- Trước đó phải cài đặt helm: https://helm.sh/docs/intro/install/
- Sau đó sử dụng helm để cài đặt cilium:
helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium –version 1.11.6 –namespace kube-system
7. Để Join server master khác và các server worker còn lại vào trong cluster:
+ check token join: kubeadm token list
+ lấy token: kubeadm token create
+ lấy cert: kubeadm init phase upload-certs --upload-certs
+ lấy discovery-token-ca-cert-hash:openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \
+ Lấy control-plane –certificate-key: openssl dgst -sha256 -hex | sed 's/^.* //'
Ví dụ:
- Join master:
kubeadm join haproxy:6443 –token oylvmu.12pwimke0blaaeji –discovery-token-ca-cert-hash sha256:303a791ef0bdaeb3a3b54ca80f8f4831dff6d0bb1c43c664d9102c9ec569ef61 –control-plane –certificate-key 3b4da12cd25d1c1e7a47abcb908c73405c4abd5e542f99692d8f1b9d368d307a
- Join worker
kubeadm join haproxy:6443 –token t7a9sr.yamaink129rswuix –discovery-token-ca-cert-hash sha256:303a791ef0bdaeb3a3b54ca80f8f4831dff6d0bb1c43c664d9102c9ec569ef61