5 台服务器（四）：DevOps 工具链

5 台服务器系列第四篇：Harbor、Gitea、Nexus、Jenkins、Argo CD 与 Prometheus/Loki/SkyWalking，把构建、发布与可观测性串起来。

技术文章2026 年 9 月 9 日约 10 分钟1 次阅读

系列导航：系列总览 · 上一篇：5 台服务器（三）：工作节点、CNI 与存储 · 下一篇：5 台服务器（五）：Spring Cloud 上线与验收

项目	说明
系列	5 台服务器二进制 K8s（第 4/5 篇）
本篇	Harbor、Gitea、Nexus、Jenkins、Argo CD、监控与日志
前置	建议先读第 3 篇：工作节点、CNI 与存储
版本	Kubernetes v1.36.4 / etcd v3.6.13 / containerd v2.3.3 LTS / Calico v3.32.2

#一、DevOps 工具链：从镜像仓库到可观测性

装 Helm 客户端（在 m1 上），一次性建好命名空间：

HELM_VER=v3.17.3
wget -q https://get.helm.sh/helm-${HELM_VER}-linux-amd64.tar.gz
tar -xzf helm-${HELM_VER}-linux-amd64.tar.gz
mv linux-amd64/helm /usr/local/bin/
helm version

for ns in devops harbor git cicd observability logging middleware prod; do
  kubectl create ns $ns --dry-run=client -o yaml | kubectl apply -f -
done

#Harbor 镜像仓库

内网用 HTTP，所以每台节点的 containerd 都要信任它（这一步在第一节已配好）。values 里注意 resourcePolicy: "keep" 和各类存储都落到 nfs-client：

helm repo add harbor https://helm.goharbor.io
helm repo update

# /root/harbor-values.yaml
expose:
  type: ingress
  tls:
    enabled: false
  ingress:
    className: nginx
    hosts:
      core: harbor.apps.lxhhub.top
    annotations:
      nginx.ingress.kubernetes.io/proxy-body-size: "0"
      nginx.ingress.kubernetes.io/proxy-read-timeout: "900"
      nginx.ingress.kubernetes.io/proxy-send-timeout: "900"

externalURL: http://harbor.apps.lxhhub.top
harborAdminPassword: "Harbor@2026!ChangeMe"

persistence:
  enabled: true
  resourcePolicy: "keep"
  persistentVolumeClaim:
    registry:
      storageClass: "nfs-client"
      size: 200Gi
    jobservice:
      jobLog:
        storageClass: "nfs-client"
        size: 10Gi
    database:
      storageClass: "nfs-client"
      size: 10Gi
    redis:
      storageClass: "nfs-client"
      size: 10Gi
    trivy:
      storageClass: "nfs-client"
      size: 10Gi

notary:
  enabled: false
trivy:
  enabled: true

helm upgrade --install harbor harbor/harbor \
  -n harbor -f /root/harbor-values.yaml --wait --timeout 15m

起来后在 Harbor 建两个项目：library（公共基础镜像）和 shop（业务镜像）。Helm chart 现在走 OCI，不用 ChartMuseum：

helm package ./charts/shop && helm push shop-0.1.0.tgz oci://harbor.apps.lxhhub.top/shop/charts

再建机器人账号 robot$shop-pusher 给 Jenkins 用、robot$shop-puller 给集群拉镜像用——机器人账号权限最小化，比拿 admin 到处配安全。集群拉镜像的 secret（每个业务命名空间都要建），密码用 Harbor 控制台生成的 token：

# 先执行 export ROBOT_TOKEN='在 Harbor 控制台生成 robot 账号后复制的 token'
kubectl create secret docker-registry harbor-pull \
  -n prod \
  --docker-server=harbor.apps.lxhhub.top \
  --docker-username='robot$shop-puller' \
  --docker-password="$ROBOT_TOKEN" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl -n prod patch serviceaccount default \
  -p '{"imagePullSecrets":[{"name":"harbor-pull"}]}'

#Gitea 代码仓库

GitLab 太重，5 台机器跑不动；Gitea 一个二进制搞定，带 Web 界面、PR、Actions，够用。DISABLE_REGISTRATION: true 建议开着，内网仓库别让人随便注册：

helm repo add gitea https://dl.gitea.com/charts
helm repo update

# /root/gitea-values.yaml
ingress:
  enabled: true
  className: nginx
  hosts:
  - host: git.apps.lxhhub.top
    paths:
    - path: /
      pathType: Prefix
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "512m"

persistence:
  enabled: true
  storageClass: nfs-client
  size: 50Gi

gitea:
  admin:
    username: gitea-admin
    password: "Gitea@2026!ChangeMe"
    email: admin@lxhhub.top
  config:
    server:
      ROOT_URL: http://git.apps.lxhhub.top/
      SSH_DOMAIN: git.apps.lxhhub.top
    service:
      DISABLE_REGISTRATION: true
    actions:
      ENABLED: true

redis-cluster:
  enabled: false
postgresql:
  enabled: true
  persistence:
    enabled: true
    storageClass: nfs-client
    size: 10Gi
  auth:
    database: gitea

helm upgrade --install gitea gitea/gitea -n git -f /root/gitea-values.yaml --wait --timeout 10m

#Nexus Maven 私服

官方 chart 维护一般，直接写清单：PVC + Deployment + Service + Ingress。初始化密码在 /nexus-data/admin.password，用 kubectl exec 进去拿；之后建 maven-proxy 代理 https://maven.aliyun.com/repository/public（国内速度快很多）、maven-releases / maven-snapshots 两个 hosted 仓库，再用 maven-public group 聚合，Maven 的 settings.xml 只配这一个镜像地址。Nexus 起得慢，ready 探针给 90 秒初始延迟和 20 次失败容忍，别改小：

# /root/nexus.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nexus-data
  namespace: devops
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: nfs-client
  resources:
    requests:
      storage: 200Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nexus
  namespace: devops
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nexus
  template:
    metadata:
      labels:
        app: nexus
    spec:
      initContainers:
      - name: fix-perm
        image: busybox:1.37
        command: ["sh", "-c", "chown -R 200:200 /nexus-data"]
        volumeMounts:
        - name: data
          mountPath: /nexus-data
      containers:
      - name: nexus
        image: sonatype/nexus3:3.83.1
        ports:
        - containerPort: 8081
        env:
        - name: INSTALL4J_ADD_VM_PARAMS
          value: "-Xms2g -Xmx2g -XX:MaxDirectMemorySize=2g -Djava.util.prefs.userRoot=/nexus-data/javaprefs"
        readinessProbe:
          httpGet: { path: /service/rest/v1/status, port: 8081 }
          initialDelaySeconds: 90
          periodSeconds: 15
          failureThreshold: 20
        livenessProbe:
          httpGet: { path: /service/rest/v1/status, port: 8081 }
          initialDelaySeconds: 180
          periodSeconds: 30
        resources:
          requests: { cpu: 1, memory: 4Gi }
          limits:   { cpu: 2, memory: 5Gi }
        volumeMounts:
        - name: data
          mountPath: /nexus-data
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: nexus-data
---
apiVersion: v1
kind: Service
metadata:
  name: nexus
  namespace: devops
spec:
  selector: { app: nexus }
  ports:
  - port: 8081
    targetPort: 8081
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nexus
  namespace: devops
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "1g"
spec:
  ingressClassName: nginx
  rules:
  - host: nexus.apps.lxhhub.top
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nexus
            port: { number: 8081 }

#Jenkins CI

重点是让它能动态起 Pod 当构建节点，numExecutors: 0 把构建全丢给 agent pod：

helm repo add jenkins https://charts.jenkins.io
helm repo update

# /root/jenkins-values.yaml
controller:
  image: "jenkins/jenkins"
  tag: "lts-jdk21"
  numExecutors: 0
  adminUser: "admin"
  adminPassword: "Jenkins@2026!ChangeMe"
  ingress:
    enabled: true
    className: nginx
    hostName: jenkins.apps.lxhhub.top
  additionalPlugins:
  - kubernetes:latest
  - workflow-aggregator:latest
  - git:latest
  - configuration-as-code:latest
  - credentials-binding:latest
  - sonar:latest
  - docker-workflow:latest
  - blueocean:latest
  installLatestPlugins: false
  persistence:
    enabled: true
    storageClass: nfs-client
    size: 50Gi
  resources:
    requests: { cpu: 500m, memory: 2Gi }
    limits:   { cpu: 2,    memory: 4Gi }
  serviceType: ClusterIP
  healthProbes:
    livenessFailureThreshold: 12

agent:
  enabled: true
  image: "jenkins/inbound-agent"
  tag: "latest-jdk21"

helm upgrade --install jenkins jenkins/jenkins -n cicd -f /root/jenkins-values.yaml --wait --timeout 15m

装完做三件事：第一，系统管理 → Clouds → New cloud，Kubernetes 类型，命名空间 cicd，Jenkins 地址填 http://jenkins.cicd.svc.cluster.local:8080，不用填 kubeconfig，Pod 里的 ServiceAccount 够用（chart 自动建了 RBAC）；第二，配 Maven 缓存 PVC，每个构建 Pod 都重新下依赖会慢到让人放弃——kubectl -n cicd create pvc maven-repo --storage-class=nfs-client --access-mode=ReadWriteMany，注意要 RWX，因为可能同时跑多个构建，这正是选 NFS 而不是 local-path 的原因之一；第三，把 Gitea 账号、Harbor 机器人账号、Sonar token 加进 Jenkins credentials。

#SonarQube（可选）

4G 内存起步，机器紧张可以跳过，不影响主线：

helm repo add sonarqube https://SonarSource.github.io/helm-chart-sonarqube
helm repo update

helm upgrade --install sonarqube sonarqube/sonarqube -n devops \
  --set edition=community \
  --set ingress.enabled=true \
  --set ingress.className=nginx \
  --set ingress.hosts[0].name=sonar.apps.lxhhub.top \
  --set persistence.enabled=true \
  --set persistence.storageClass=nfs-client \
  --set persistence.size=20Gi \
  --set postgresql.persistence.enabled=true \
  --set postgresql.persistence.storageClass=nfs-client \
  --wait --timeout 15m

#Argo CD（GitOps CD）

发布环节用它，Jenkins 只负责"构建镜像 + 改 GitOps 仓库里的 tag"，不直接碰集群。这样集群里跑了什么，Git 仓库里一清二楚，回滚就是一次 Git revert：

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

# /root/argocd-values.yaml
configs:
  params:
    server.insecure: true
  repositories:
    gitea-shop:
      url: http://git.apps.lxhhub.top/gitea-admin/shop-gitops.git

server:
  ingress:
    enabled: true
    ingressClassName: nginx
    hostname: argocd.apps.lxhhub.top
    extraAnnotations:
      nginx.ingress.kubernetes.io/ssl-passthrough: "false"
      nginx.ingress.kubernetes.io/backend-protocol: "HTTP"

helm upgrade --install argocd argo/argo-cd -n argocd --create-namespace \
  -f /root/argocd-values.yaml --wait --timeout 10m

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d; echo

#监控：Prometheus + Grafana

用 kube-prometheus-stack，关键是把存储落到 nfs-client：

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# /root/prom-values.yaml
prometheus:
  prometheusSpec:
    retention: 15d
    retentionSize: 45GiB
    scrapeInterval: 30s
    evaluationInterval: 30s
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: nfs-client
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 100Gi
    resources:
      requests: { cpu: 500m, memory: 2Gi }
      limits:   { cpu: 2,    memory: 6Gi }

grafana:
  enabled: true
  adminPassword: "Grafana@2026!ChangeMe"
  ingress:
    enabled: true
    ingressClassName: nginx
    hosts: ["grafana.apps.lxhhub.top"]
  persistence:
    enabled: true
    storageClassName: nfs-client
    size: 10Gi

alertmanager:
  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: nfs-client
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 10Gi

kubeStateMetrics:
  enabled: true
nodeExporter:
  enabled: true
prometheusOperator:
  enabled: true

helm upgrade --install kube-prom prometheus-community/kube-prometheus-stack \
  -n observability -f /root/prom-values.yaml --wait --timeout 15m

二进制部署的 etcd 不在集群里，默认抓不到。把 etcd 客户端证书做成 Secret，Prometheus 就能通过节点 IP 抓它：

kubectl -n observability create secret generic etcd-client-certs \
  --from-file=ca.crt=/etc/etcd/ssl/etcd-ca.pem \
  --from-file=tls.crt=/etc/etcd/ssl/etcd-client.pem \
  --from-file=tls.key=/etc/etcd/ssl/etcd-client-key.pem

# /root/prom-etcd.yaml
prometheus:
  prometheusSpec:
    secrets: ["etcd-client-certs"]
    additionalScrapeConfigs:
    - job_name: 'etcd'
      scheme: https
      tls_config:
        ca_file: /etc/prometheus/secrets/etcd-client-certs/ca.crt
        cert_file: /etc/prometheus/secrets/etcd-client-certs/tls.crt
        key_file: /etc/prometheus/secrets/etcd-client-certs/tls.key
      static_configs:
      - targets: ['10.0.0.11:2379','10.0.0.12:2379','10.0.0.13:2379']

helm upgrade kube-prom prometheus-community/kube-prometheus-stack \
  -n observability -f /root/prom-values.yaml -f /root/prom-etcd.yaml

告警推送到企业微信/钉钉，改 alertmanager.config 即可。

#日志：Loki + Alloy

Promtail 已进入维护尾声，新部署直接用 Grafana Alloy 采集：

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# /root/loki-values.yaml
deploymentMode: SingleBinary
loki:
  auth_enabled: false
  commonConfig:
    replication_factor: 1
  storage:
    type: filesystem
  limits_config:
    retention_period: 15d
  schemaConfig:
    configs:
    - from: "2024-01-01"
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

singleBinary:
  replicas: 1
  persistence:
    enabled: true
    storageClass: nfs-client
    size: 100Gi
  resources:
    requests: { cpu: 500m, memory: 1Gi }
    limits:   { cpu: 2,    memory: 3Gi }

gateway:
  enabled: true

helm upgrade --install loki grafana/loki -n logging -f /root/loki-values.yaml --wait --timeout 10m

# /root/alloy-values.yaml
alloy:
  configMap:
    create: true
    content: |
      logging {
        level = "info"
      }

      discovery.kubernetes "pods" {
        role = "pod"
      }

      discovery.relabel "pod_logs" {
        targets = discovery.kubernetes.pods.targets

        rule {
          source_labels = ["__meta_kubernetes_namespace"]
          action        = "replace"
          target_label  = "namespace"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_name"]
          action        = "replace"
          target_label  = "pod"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_container_name"]
          action        = "replace"
          target_label  = "container"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_uid", "__meta_kubernetes_pod_container_name"]
          action        = "replace"
          separator     = "/"
          replacement   = "/var/log/pods/*$1/*.log"
          target_label  = "__path__"
        }
        rule {
          action       = "labelmap"
          regex        = "__meta_kubernetes_pod_label_(.+)"
        }
      }

      loki.source.file "pod_logs" {
        targets    = discovery.relabel.pod_logs.output
        forward_to = [loki.write.default.receiver]
      }

      loki.write "default" {
        endpoint {
          url = "http://loki-gateway.logging.svc.cluster.local/loki/api/v1/push"
        }
      }

  mounts:
    varlog: true

controller:
  type: daemonset

helm upgrade --install alloy grafana/alloy -n logging -f /root/alloy-values.yaml --wait --timeout 10m

最后在 Grafana 里加 Loki 数据源 http://loki-gateway.logging.svc.cluster.local，就能用 LogQL 查日志：

{namespace="prod", app="shop-order"} |= "ERROR"

#SkyWalking（可选）

Spring Cloud 项目在国内基本绕不开它；它吃资源（OAP + ES），机器紧张可以先只装 OAP + UI，存储用已有 ES 或者直接用 H2 做验证。OAP 的 gRPC 端口 11800 暴露出去（NodePort 或 Ingress），Java 应用通过 agent 接入：

helm repo add skywalking https://apache.jfrog.io/artifactory/skywalking-helm
helm repo update

helm upgrade --install skywalking skywalking/skywalking -n observability \
  --set oap.image.tag=10.2.0 \
  --set oap.storageType=elasticsearch \
  --set oap.replicas=1 \
  --set ui.image.tag=10.2.0 \
  --set elasticsearch.enabled=true \
  --set elasticsearch.imageTag=8.15.2 \
  --set elasticsearch.persistence.enabled=true \
  --set elasticsearch.persistence.storageClass=nfs-client \
  --wait --timeout 20m

下一篇：5 台服务器（五）：Spring Cloud 上线与验收 —— 中间件、微服务上线、发布回滚与验收。

#DevOps #CI/CD #可观测性 #Kubernetes

留言讨论

0 条

还没有留言，来说点什么吧。