LXH·BLOG

5 台服务器（三）：工作节点、CNI 与存储

5 台服务器系列第三篇：kubelet 与 kube-proxy、Calico CNI、Ingress-NGINX 与 NFS 存储，让工作节点真正可调度。

技术文章约 9 分钟0 次阅读

系列导航：系列总览 · 上一篇：5 台服务器（二）：etcd 与控制面高可用 · 下一篇：5 台服务器（四）：DevOps 工具链

项目说明
系列5 台服务器二进制 K8s（第 3/5 篇）
本篇kubelet、kube-proxy、Calico、Ingress-NGINX、NFS 存储
前置建议先读 第 2 篇：etcd 与控制面高可用
版本Kubernetes v1.36.4 / etcd v3.6.13 / containerd v2.3.3 LTS / Calico v3.32.2

#一、kubelet 与 kube-proxy

全部 5 台都要装 kubelet；worker 节点先把 bootstrap-kubelet.kubeconfig 和 CA 公钥从 m1 拷过去。kubelet 配置里 clusterDNS: 10.96.0.10 必须和后面 CoreDNS 的 Service IP 一致；serverTLSBootstrap: true 让 kubelet 自己申请 serving 证书（依赖前面 auto-approve-server-renewals-for-nodes 绑定）：

cat > /etc/kubernetes/kubelet-config.yaml <<'EOF'
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 0
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
    cacheTTL: 2m0s
  x509:
    clientCAFile: /etc/kubernetes/pki/k8s-ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerLogMaxFiles: 5
containerLogMaxSize: 100Mi
maxPods: 110
failSwapOn: true
serializeImagePulls: false
registryPullQPS: 10
registryBurst: 20
rotateCertificates: true
serverTLSBootstrap: true
evictionHard:
  memory.available: 500Mi
  nodefs.available: 10%
  imagefs.available: 15%
  nodefs.inodesFree: 5%
evictionSoft:
  memory.available: 1Gi
evictionSoftGracePeriod:
  memory.available: 1m30s
kubeReserved:
  cpu: 200m
  memory: 1Gi
  ephemeral-storage: 1Gi
systemReserved:
  cpu: 200m
  memory: 1Gi
  ephemeral-storage: 1Gi
EOF

cat > /usr/lib/systemd/system/kubelet.service <<'EOF'
[Unit]
Description=Kubernetes Kubelet
Documentation=https://kubernetes.io
After=containerd.service
Requires=containerd.service

[Service]
ExecStart=/usr/local/bin/kubelet \
  --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig \
  --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
  --config=/etc/kubernetes/kubelet-config.yaml \
  --cert-dir=/etc/kubernetes/pki \
  --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
  --node-ip=NODE_IP_PLACEHOLDER \
  --hostname-override=HOSTNAME_PLACEHOLDER \
  --pod-infra-container-image=registry.k8s.io/pause:3.10 \
  --root-dir=/var/lib/kubelet \
  --v=2
Restart=always
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

sed -i "s/NODE_IP_PLACEHOLDER/$(hostname -i | awk '{print $1}')/" /usr/lib/systemd/system/kubelet.service
sed -i "s/HOSTNAME_PLACEHOLDER/$(hostname)/" /usr/lib/systemd/system/kubelet.service

systemctl daemon-reload && systemctl enable --now kubelet

worker 节点先拷文件再启动：

# m1 上执行
scp /etc/kubernetes/bootstrap-kubelet.kubeconfig /etc/kubernetes/pki/k8s-ca.pem k8s-n1:/etc/kubernetes/
scp /etc/kubernetes/bootstrap-kubelet.kubeconfig /etc/kubernetes/pki/k8s-ca.pem k8s-n2:/etc/kubernetes/
# n1/n2 上把 k8s-ca.pem 放到 /etc/kubernetes/pki/

kubelet 起来后会自己申请证书。在 m1 上看 kubectl get csr，会出现 Pending 变成 Approved,Issued；开了 serverTLSBootstrap 还会多一个 selfnodeserver 的 CSR，同样自动批准。如果一直 Pending，去看 journalctl -u kubelet -f，十有八九是 bootstrap token 不对或者 kubeconfig 里的 apiserver 地址不通。

kube-proxy 用 ipvs 模式，依赖前面加载的 ip_vs 内核模块：

cat > /etc/kubernetes/kube-proxy-config.yaml <<'EOF'
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
bindAddress: 0.0.0.0
clientConnection:
  kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
clusterCIDR: 172.16.0.0/16
mode: ipvs
healthzBindAddress: 0.0.0.0:10256
metricsBindAddress: 0.0.0.0:10249
ipvs:
  scheduler: rr
  strictARP: true
  tcpTimeout: 0s
  tcpFinTimeout: 0s
  udpTimeout: 0s
conntrack:
  maxPerCore: 32768
  min: 131072
  tcpEstablishedTimeout: 0s
  tcpCloseWaitTimeout: 0s
EOF

cat > /usr/lib/systemd/system/kube-proxy.service <<'EOF'
[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://kubernetes.io
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-proxy \
  --config=/etc/kubernetes/kube-proxy-config.yaml \
  --hostname-override=HOSTNAME_PLACEHOLDER \
  --v=2
Restart=always
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

sed -i "s/HOSTNAME_PLACEHOLDER/$(hostname)/" /usr/lib/systemd/system/kube-proxy.service
systemctl daemon-reload && systemctl enable --now kube-proxy

worker 节点需要 kube-proxy.kubeconfig：

scp /etc/kubernetes/kube-proxy.kubeconfig k8s-n1:/etc/kubernetes/
scp /etc/kubernetes/kube-proxy.kubeconfig k8s-n2:/etc/kubernetes/

给 master 打污点和角色标签，之后业务 Pod 不会调度到 master：

kubectl label node k8s-m1 node-role.kubernetes.io/control-plane=
kubectl label node k8s-m2 node-role.kubernetes.io/control-plane=
kubectl label node k8s-m3 node-role.kubernetes.io/control-plane=
kubectl label node k8s-n1 node-role.kubernetes.io/worker=
kubectl label node k8s-n2 node-role.kubernetes.io/worker=

kubectl taint nodes k8s-m1 node-role.kubernetes.io/control-plane:NoSchedule
kubectl taint nodes k8s-m2 node-role.kubernetes.io/control-plane:NoSchedule
kubectl taint nodes k8s-m3 node-role.kubernetes.io/control-plane:NoSchedule

kubectl get nodes -o wide

此时 kubectl get nodes 应该 5 台都是 Ready。到这里，一个"能跑 Pod 但没网络"的集群就立起来了。

#二、CNI 与核心插件

Calico 提供 Pod 网络。 用 release tgz 里的 manifests，IP 池必须和 controller-manager 的 --cluster-cidr 一致：

CALICO_VER=v3.32.2
wget -q https://github.com/projectcalico/calico/releases/download/${CALICO_VER}/release-${CALICO_VER}.tgz
tar -xzf release-${CALICO_VER}.tgz
cd release-${CALICO_VER}/manifests/

sed -i 's|# - name: CALICO_IPV4POOL_CIDR|  - name: CALICO_IPV4POOL_CIDR|; s|#   value: "192.168.0.0/16"|  value: "172.16.0.0/16"|' calico.yaml
grep -A1 'CALICO_IPV4POOL_CIDR' calico.yaml

kubectl apply -f calico.yaml
kubectl -n kube-system get pod -l k8s-app=calico-node -w

默认单网卡环境不用改网卡发现规则；多网卡机器（比如同时有管理网和业务网）加一条明确指定网卡：

sed -i '/# - name: IP_AUTODETECTION_METHOD/{n; s|#   value: "first-found"|  value: "interface=eth0"|}' calico.yaml
sed -i 's|# - name: IP_AUTODETECTION_METHOD|  - name: IP_AUTODETECTION_METHOD|' calico.yaml

装 calicoctl 备用：

wget -q -O /usr/local/bin/calicoctl https://github.com/projectcalico/calico/releases/download/${CALICO_VER}/calicoctl-linux-amd64
chmod +x /usr/local/bin/calicoctl
calicoctl node status

顺序问题很关键：集群还没有可用 CNI 时，CoreDNS 的 Pod 拿不到 IP 会一直重启，报 no endpoints available for service "kube-dns"。所以必须先 Calico，等 calico-node 全部 Running，再上 CoreDNS。

CoreDNS 提供集群 DNS。 核心是回答 <service>.<namespace>.svc.cluster.local 的查询。两个必须对齐的值：Service 的 clusterIP: 10.96.0.10（落在 Service CIDR 内）和 kubelet 配置里的 clusterDNS。完整清单如下（ServiceAccount、RBAC、反亲和、探针都包含在内），保存为 /root/coredns.yaml 后 kubectl apply -f：

apiVersion: v1
kind: ServiceAccount
metadata:
  name: coredns
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:coredns
rules:
- apiGroups: [""]
  resources: ["endpoints", "services", "pods", "namespaces"]
  verbs: ["list", "watch"]
- apiGroups: ["discovery.k8s.io"]
  resources: ["endpointslices"]
  verbs: ["list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:coredns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:coredns
subjects:
- kind: ServiceAccount
  name: coredns
  namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
          lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
          ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
          max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: coredns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  selector:
    matchLabels:
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        k8s-app: kube-dns
    spec:
      priorityClassName: system-cluster-critical
      serviceAccountName: coredns
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: k8s-app
                operator: In
                values: ["kube-dns"]
            topologyKey: kubernetes.io/hostname
      containers:
      - name: coredns
        image: registry.k8s.io/coredns/coredns:v1.13.2
        imagePullPolicy: IfNotPresent
        args: ["-conf", "/etc/coredns/Corefile"]
        resources:
          limits:
            memory: 512Mi
          requests:
            cpu: 100m
            memory: 128Mi
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        - containerPort: 9153
          name: metrics
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          periodSeconds: 10
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /ready
            port: 8181
            scheme: HTTP
          periodSeconds: 2
          timeoutSeconds: 3
        volumeMounts:
        - name: config-volume
          mountPath: /etc/coredns
          readOnly: true
      volumes:
      - name: config-volume
        configMap:
          name: coredns
          items:
          - key: Corefile
            path: Corefile
---
apiVersion: v1
kind: Service
metadata:
  name: kube-dns
  namespace: kube-system
  annotations:
    prometheus.io/port: "9153"
    prometheus.io/scrape: "true"
  labels:
    k8s-app: kube-dns
spec:
  selector:
    k8s-app: kube-dns
  clusterIP: 10.96.0.10
  ports:
  - name: dns
    port: 53
    protocol: UDP
    targetPort: 53
  - name: dns-tcp
    port: 53
    protocol: TCP
    targetPort: 53
  - name: metrics
    port: 9153
    protocol: TCP
    targetPort: 9153

metrics-server 提供 kubectl top 的数据，HPA 依赖它。二进制集群里 kubelet 的 serving 证书不是集群 CA 签发的场景，要加 --kubelet-insecure-tls：

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

kubectl -n kube-system patch deployment metrics-server --type='json' -p='[
  {"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"},
  {"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname"}
]'

kubectl top nodes
kubectl top pod -A

Ingress-NGINX：裸金属没有云 LoadBalancer，用 DaemonSet + hostNetwork 直接占住 worker 的 80/443；master 有 NoSchedule 污点，所以只落在 n1/n2 上：

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \
  -n ingress-nginx --create-namespace \
  --set controller.kind=DaemonSet \
  --set controller.hostNetwork=true \
  --set controller.dnsPolicy=ClusterFirstWithHostNet \
  --set controller.service.type=ClusterIP \
  --set controller.ingressClassResource.default=true \
  --set controller.metrics.enabled=true \
  --set controller.config.proxy-body-size=100m \
  --set controller.config.use-forwarded-headers=true

kubectl -n ingress-nginx get pod -o wide

外网流量怎么进来：在内网 DNS 上把 *.apps.lxhhub.top 做成两条 A 记录分别指向 n1、n2（DNS 轮询），或者在 n1/n2 上再跑一组 keepalived 提供第二个 VIP——前者简单，后者能摘掉故障节点，看你对可用性要求到哪一步。顺带一句，Ingress-NGINX 社区近两年维护节奏放缓，Gateway API 是明确方向；新集群如果不想中途迁移，可以一开始就用 Nginx Gateway Fabric 或 Traefik，本文为了贴国内主流做法还是用 Ingress-NGINX。

#三、存储：给有状态服务一个落脚点

5 台机器没有独立存储节点，Harbor、Jenkins、Nacos、MySQL 全都要 PVC。做法：m1 上起 NFS Server，用 nfs-subdir-external-provisioner 提供动态 PV。这是明显单点——m1 挂了新 PVC 创建不了，但已有 PV 上的数据还在，正在跑的服务不受影响；对 5 台机器规模可以接受，别拿它当长期方案：

# m1 上
mkdir -p /data/nfs && chmod 777 /data/nfs
dnf install -y nfs-utils
systemctl enable --now nfs-server

cat > /etc/exports <<'EOF'
/data/nfs 10.0.0.0/24(rw,sync,no_root_squash,no_subtree_check)
EOF
exportfs -arv
showmount -e 10.0.0.11

# n1/n2 上验证
showmount -e 10.0.0.11
helm repo add nfs-subdir-external-provisioner \
  https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/

helm upgrade --install nfs-client nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
  -n kube-system \
  --set nfs.server=10.0.0.11 \
  --set nfs.path=/data/nfs \
  --set storageClass.name=nfs-client \
  --set storageClass.defaultClass=true \
  --set storageClass.reclaimPolicy=Retain \
  --set storageClass.archiveOnDelete=false \
  --set replicaCount=1

kubectl get sc

reclaimPolicy=Retain 是有意为之：PVC 误删时数据不会跟着没。需要高性能本地盘的东西（Nacos 的 JRaft 日志、MySQL）可以再准备一个 local 类型 StorageClass；真上生产换 Rook-Ceph 或 Longhorn，但那至少再要 3 块独立磁盘，5 台机器吃不消。

下一篇：5 台服务器（四）：DevOps 工具链 —— Harbor、Gitea、Nexus、Jenkins、Argo CD 与可观测性。

#Kubernetes#CNI#网络#存储

相关阅读

Related

留言讨论

0 条

还没有留言，来说点什么吧。

5 台服务器（三）：工作节点、CNI 与存储 · LXH·BLOG