5 台服务器（二）：etcd 与控制面高可用

5 台服务器系列第二篇：etcd 三成员共识、控制平面三件套、haproxy + keepalived 提供 apiserver 的唯一入口与高可用。

技术文章2026 年 9 月 9 日约 12 分钟1 次阅读

系列导航：系列总览 · 上一篇：5 台服务器（一）：系统初始化与证书底座 · 下一篇：5 台服务器（三）：工作节点、CNI 与存储

项目	说明
系列	5 台服务器二进制 K8s（第 2/5 篇）
本篇	etcd 三成员、apiserver/scheduler/controller-manager、VIP 高可用
前置	建议先读第 1 篇：系统初始化与证书底座
版本	Kubernetes v1.36.4 / etcd v3.6.13 / containerd v2.3.3 LTS / Calico v3.32.2

#一、etcd：先有共识，后有入口

三台 master 下载并在本机启动 etcd。三份 unit 只有 --name、--listen-*、--initial-advertise-peer-urls 三处不同，脚本用变量生成：

ETCD_VER=v3.6.13
wget -q https://github.com/etcd-io/etcd/releases/download/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz
tar -xzf etcd-${ETCD_VER}-linux-amd64.tar.gz
cp etcd-${ETCD_VER}-linux-amd64/etcd{,-ctl} /usr/local/bin/
etcd --version

cat > /root/install-etcd.sh <<'SCRIPT'
#!/bin/bash
set -e
NODE_IP=$1
NODE_NAME=$(hostname)

cat > /usr/lib/systemd/system/etcd.service <<EOF
[Unit]
Description=Etcd Server
Documentation=https://etcd.io
After=network.target

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd \\
  --name=${NODE_NAME} \\
  --data-dir=/var/lib/etcd \\
  --wal-dir=/var/lib/etcd/wal \\
  --listen-peer-urls=https://${NODE_IP}:2380 \\
  --listen-client-urls=https://${NODE_IP}:2379,https://127.0.0.1:2379 \\
  --advertise-client-urls=https://${NODE_IP}:2379 \\
  --initial-advertise-peer-urls=https://${NODE_IP}:2380 \\
  --initial-cluster=k8s-m1=https://10.0.0.11:2380,k8s-m2=https://10.0.0.12:2380,k8s-m3=https://10.0.0.13:2380 \\
  --initial-cluster-state=new \\
  --initial-cluster-token=etcd-k8s-cluster \\
  --cert-file=/etc/etcd/ssl/etcd-server.pem \\
  --key-file=/etc/etcd/ssl/etcd-server-key.pem \\
  --peer-cert-file=/etc/etcd/ssl/etcd-server.pem \\
  --peer-key-file=/etc/etcd/ssl/etcd-server-key.pem \\
  --trusted-ca-file=/etc/etcd/ssl/etcd-ca.pem \\
  --peer-trusted-ca-file=/etc/etcd/ssl/etcd-ca.pem \\
  --client-cert-auth=true \\
  --peer-client-cert-auth=true \\
  --snapshot-count=10000 \\
  --quota-backend-bytes=8589934592 \\
  --auto-compaction-mode=revision \\
  --auto-compaction-retention=1000 \\
  --v2-deprecation=write-only \\
  --log-level=info \\
  --logger=zap
Restart=always
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now etcd
sleep 3
systemctl is-active etcd
SCRIPT

chmod +x /root/install-etcd.sh
/root/install-etcd.sh 10.0.0.11   # m1

m2、m3 上把 IP 换成自己的。三台启动间隔别超过 30 秒：先起的那台在 initial-cluster-state=new 下会等 peer，等太久会自己退出。验证必须三行都是 true 且只有一个 leader：

export ETCDCTL_API=3
etcdctl --endpoints=https://10.0.0.11:2379,https://10.0.0.12:2379,https://10.0.0.13:2379 \
  --cacert=/etc/etcd/ssl/etcd-ca.pem \
  --cert=/etc/etcd/ssl/etcd-client.pem \
  --key=/etc/etcd/ssl/etcd-client-key.pem \
  endpoint health --write-out=table

etcdctl --endpoints=https://10.0.0.11:2379 \
  --cacert=/etc/etcd/ssl/etcd-ca.pem \
  --cert=/etc/etcd/ssl/etcd-client.pem \
  --key=/etc/etcd/ssl/etcd-client-key.pem \
  member list --write-out=table

顺手在 /root/.bashrc 里配个别名，后面天天用：

cat >> /root/.bashrc <<'EOF'
export ETCDCTL_API=3
alias etcdctl-prod='etcdctl --endpoints=https://10.0.0.11:2379,https://10.0.0.12:2379,https://10.0.0.13:2379 --cacert=/etc/etcd/ssl/etcd-ca.pem --cert=/etc/etcd/ssl/etcd-client.pem --key=/etc/etcd/ssl/etcd-client-key.pem'
EOF
source /root/.bashrc

etcd 3.6 有两个值得记的变化：--v2-deprecation 默认变成 write-only（v2 API 只能写不能读，老的 v2 数据已经在被清理的路上）；--snapshot-count 默认从 100000 降到 10000，快照更频繁、磁盘 IO 略增但恢复更快。

#二、控制平面三件套

下载 k8s 二进制，只需要 server 包：kubernetes-server-linux-amd64.tar.gz 里已包含 kube-apiserver、kube-controller-manager、kube-scheduler、kubelet、kube-proxy、kubectl：

K8S_VER=v1.36.4
wget -q https://dl.k8s.io/${K8S_VER}/kubernetes-server-linux-amd64.tar.gz
tar -xzf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin
cp kube-apiserver kube-controller-manager kube-scheduler kubelet kube-proxy kubectl /usr/local/bin/
chmod +x /usr/local/bin/kube*
kube-apiserver --version

先造 kubeconfig。kubeconfig 的本质是"用哪张证书、连哪个地址、以什么身份"的打包，在 m1 上统一生成再分发。server 地址走 VIP：https://10.0.0.10:16443（haproxy 起在这个端口）。kubelet 特殊，用 bootstrap 机制而不走证书直配：

cd /root/ssl
KUBE_APISERVER="https://10.0.0.10:16443"

gen_kubeconfig () {
  local NAME=$1 CN=$2
  kubectl config set-cluster kubernetes \
    --certificate-authority=k8s-ca.pem \
    --embed-certs=true \
    --server=${KUBE_APISERVER} \
    --kubeconfig=${NAME}.kubeconfig
  kubectl config set-credentials ${CN} \
    --client-certificate=${NAME}.pem \
    --client-key=${NAME}-key.pem \
    --embed-certs=true \
    --kubeconfig=${NAME}.kubeconfig
  kubectl config set-context ${CN}@kubernetes \
    --cluster=kubernetes --user=${CN} \
    --kubeconfig=${NAME}.kubeconfig
  kubectl config use-context ${CN}@kubernetes --kubeconfig=${NAME}.kubeconfig
}

gen_kubeconfig kube-controller-manager system:kube-controller-manager
gen_kubeconfig kube-scheduler           system:kube-scheduler
gen_kubeconfig admin                    admin
gen_kubeconfig kube-proxy               system:kube-proxy

TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' \n' | head -c 6).$(head -c 24 /dev/urandom | base64 | tr -d '=+/' | head -c 16)
echo "bootstrap token: ${TOKEN}"

cat > token.csv <<EOF
${TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF

kubectl config set-cluster kubernetes \
  --certificate-authority=k8s-ca.pem --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=bootstrap-kubelet.kubeconfig
kubectl config set-credentials kubelet-bootstrap \
  --token=${TOKEN} \
  --kubeconfig=bootstrap-kubelet.kubeconfig
kubectl config set-context kubelet-bootstrap@kubernetes \
  --cluster=kubernetes --user=kubelet-bootstrap \
  --kubeconfig=bootstrap-kubelet.kubeconfig
kubectl config use-context kubelet-bootstrap@kubernetes \
  --kubeconfig=bootstrap-kubelet.kubeconfig

for h in k8s-m1 k8s-m2 k8s-m3; do
  scp token.csv bootstrap-kubelet.kubeconfig \
      kube-controller-manager.kubeconfig kube-scheduler.kubeconfig \
      admin.kubeconfig kube-proxy.kubeconfig $h:/etc/kubernetes/
done

kube-apiserver 的 unit 是控制面里最长的一个。注意 --advertise-address 必须是本机 IP，不能写 0.0.0.0，三台 master 各不相同，用 sed 替换占位符：

cat > /usr/lib/systemd/system/kube-apiserver.service <<'EOF'
[Unit]
Description=Kubernetes API Server
Documentation=https://kubernetes.io
After=network.target etcd.service

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
  --advertise-address=NODE_IP_PLACEHOLDER \
  --bind-address=0.0.0.0 \
  --secure-port=6443 \
  --allow-privileged=true \
  --authorization-mode=Node,RBAC \
  --enable-admission-plugins=NodeRestriction \
  --enable-bootstrap-token-auth=true \
  --token-auth-file=/etc/kubernetes/token.csv \
  --service-cluster-ip-range=10.96.0.0/12 \
  --service-node-port-range=30000-32767 \
  --etcd-servers=https://10.0.0.11:2379,https://10.0.0.12:2379,https://10.0.0.13:2379 \
  --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \
  --etcd-certfile=/etc/etcd/ssl/etcd-client.pem \
  --etcd-keyfile=/etc/etcd/ssl/etcd-client-key.pem \
  --client-ca-file=/etc/kubernetes/pki/k8s-ca.pem \
  --tls-cert-file=/etc/kubernetes/pki/apiserver.pem \
  --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \
  --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.pem \
  --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client-key.pem \
  --service-account-key-file=/etc/kubernetes/pki/sa.pub \
  --service-account-signing-key-file=/etc/kubernetes/pki/sa.key \
  --service-account-issuer=https://kubernetes.default.svc.cluster.local \
  --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
  --requestheader-username-headers=X-Remote-User \
  --requestheader-group-headers=X-Remote-Group \
  --requestheader-extra-headers-prefix=X-Remote-Extra- \
  --requestheader-allowed-names=front-proxy-client \
  --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \
  --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \
  --enable-aggregator-routing=true \
  --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
  --max-mutating-requests-inflight=2000 \
  --max-requests-inflight=4000 \
  --default-not-ready-toleration-seconds=30 \
  --default-unreachable-toleration-seconds=30 \
  --audit-log-maxage=15 --audit-log-maxbackup=5 --audit-log-maxsize=200 \
  --audit-log-path=/var/log/kubernetes/apiserver-audit.log \
  --v=2
Restart=always
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

sed -i "s/NODE_IP_PLACEHOLDER/$(hostname -i | awk '{print $1}')/" /usr/lib/systemd/system/kube-apiserver.service
systemctl daemon-reload && systemctl enable --now kube-apiserver

kube-controller-manager：--cluster-cidr=172.16.0.0/16 这个值和后面 Calico 的 IP 池必须一字不差，不一致的话节点能就绪，但 Pod 会一直 ContainerCreating：

cat > /usr/lib/systemd/system/kube-controller-manager.service <<'EOF'
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://kubernetes.io
After=network.target kube-apiserver.service

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
  --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \
  --authentication-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \
  --authorization-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \
  --client-ca-file=/etc/kubernetes/pki/k8s-ca.pem \
  --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
  --root-ca-file=/etc/kubernetes/pki/k8s-ca.pem \
  --service-account-private-key-file=/etc/kubernetes/pki/sa.key \
  --cluster-signing-cert-file=/etc/kubernetes/pki/k8s-ca.pem \
  --cluster-signing-key-file=/etc/kubernetes/pki/k8s-ca-key.pem \
  --cluster-signing-duration=87600h \
  --cluster-name=kubernetes \
  --cluster-cidr=172.16.0.0/16 \
  --service-cluster-ip-range=10.96.0.0/12 \
  --allocate-node-cidrs=true \
  --controllers=*,bootstrapsigner,tokencleaner \
  --use-service-account-credentials=true \
  --node-monitor-period=5s \
  --node-monitor-grace-period=40s \
  --terminated-pod-gc-threshold=100 \
  --leader-elect=true \
  --leader-elect-lease-duration=15s \
  --leader-elect-renew-deadline=10s \
  --leader-elect-retry-period=2s \
  --v=2
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload && systemctl enable --now kube-controller-manager

kube-scheduler 用配置文件方式启动，API 版本是 kubescheduler.config.k8s.io/v1：

cat > /etc/kubernetes/kube-scheduler.yaml <<'EOF'
apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: /etc/kubernetes/kube-scheduler.kubeconfig
leaderElection:
  leaderElect: true
  leaseDuration: 15s
  renewDeadline: 10s
  retryPeriod: 2s
percentageOfNodesToScore: 100
EOF

cat > /usr/lib/systemd/system/kube-scheduler.service <<'EOF'
[Unit]
Description=Kubernetes Scheduler
Documentation=https://kubernetes.io
After=network.target kube-apiserver.service

[Service]
ExecStart=/usr/local/bin/kube-scheduler \
  --config=/etc/kubernetes/kube-scheduler.yaml \
  --v=2
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload && systemctl enable --now kube-scheduler

配置 kubectl 并处理 RBAC。二进制集群里 system:kube-apiserver-to-kubelet 这个 ClusterRole 默认不存在，必须自己建，否则 kubectl logs/exec/top 全部报 Unauthorized。下面这份清单同时覆盖：bootstrap token 发起 CSR、首次证书自动批准、kubelet 客户端续期、kubelet server 证书自动批准，以及 apiserver 访问 kubelet 的 logs/exec/metrics 授权：

mkdir -p ~/.kube
cp /etc/kubernetes/admin.kubeconfig ~/.kube/config
kubectl completion bash > /etc/bash_completion.d/kubectl
source /etc/bash_completion.d/kubectl

kubectl get --raw='/readyz?verbose' | grep -E 'etcd|readyz'

# /root/rbac-bootstrap.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubelet-bootstrap
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node-bootstrapper
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:kubelet-bootstrap
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: auto-approve-csrs-for-group
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:kubelet-bootstrap
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: auto-approve-renewals-for-nodes
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:nodes
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: auto-approve-server-renewals-for-nodes
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:nodes
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:kube-apiserver-to-kubelet
rules:
- apiGroups: [""]
  resources: ["nodes/proxy", "nodes/stats", "nodes/log", "nodes/spec", "nodes/metrics"]
  verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: kube-apiserver

kubectl apply -f /root/rbac-bootstrap.yaml

#三、apiserver 高可用：haproxy + keepalived

三台 apiserver 都在跑，但 kubelet 和 kubectl 需要唯一一个地址。做法：三个 master 各跑一份 haproxy（本机 16443 转发到三台 6443）+ keepalived（VIP 漂移）。关键点：keepalived 检测的是本机 haproxy 的 16443，而不是本机 apiserver 的 6443——这样即使本机 apiserver 挂了，只要 haproxy 能转发到其他两台，VIP 就不用漂，避免无谓抖动：

dnf install -y haproxy keepalived

cat > /etc/haproxy/haproxy.cfg <<'EOF'
global
    log         127.0.0.1 local2
    maxconn     32768
    user        haproxy
    group       haproxy
    daemon

defaults
    mode                    tcp
    log                     global
    option                  tcplog
    timeout connect         10s
    timeout client          1m
    timeout server          1m

frontend k8s-apiserver
    bind *:16443
    mode tcp
    default_backend k8s-apiserver

backend k8s-apiserver
    mode tcp
    balance roundrobin
    option tcp-check
    server k8s-m1 10.0.0.11:6443 check inter 2000 fall 3 rise 2
    server k8s-m2 10.0.0.12:6443 check inter 2000 fall 3 rise 2
    server k8s-m3 10.0.0.13:6443 check inter 2000 fall 3 rise 2
EOF

cat > /etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived
global_defs {
    router_id LVS_K8S_$(hostname | tr 'a-z' 'A-Z')
    script_user root
    enable_script_security
}

vrrp_script check_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 3
    weight -30
    fall 2
    rise 2
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0              # 改成你的网卡名
    virtual_router_id 51
    priority 100                # m1=100 m2=99 m3=98
    advert_int 1
    nopreempt
    authentication {
        auth_type PASS
        auth_pass k8s_vip_2026
    }
    virtual_ipaddress {
        10.0.0.10/24 dev eth0
    }
    track_script {
        check_apiserver
    }
}
EOF

cat > /etc/keepalived/check_apiserver.sh <<'EOF'
#!/bin/bash
if curl -sk --max-time 2 https://127.0.0.1:16443/readyz >/dev/null 2>&1; then
  exit 0
fi
exit 1
EOF
chmod +x /etc/keepalived/check_apiserver.sh

systemctl enable --now haproxy keepalived

验证：

ip addr show | grep 10.0.0.10        # 只在某台上出现
curl -sk https://10.0.0.10:16443/readyz
systemctl stop kube-apiserver        # 在持有 VIP 的机器上停掉，看 VIP 是否漂走

下一篇：5 台服务器（三）：工作节点、CNI 与存储 —— kubelet、kube-proxy、Calico 与 NFS。

#Kubernetes #etcd #高可用 #控制面

留言讨论

0 条

还没有留言，来说点什么吧。

#一、etcd：先有共识，后有入口

#二、控制平面三件套

#三、apiserver 高可用：haproxy + keepalived

相关阅读

留言讨论