部署云原生监控平台Prometheus 十八岁 2026-08-13 2026-08-13 一、部署基于Prometheus的全方位监控平台 Prometheus的安装方式有多种，该笔记中选择Kube-Prometheus Stack的方式在k8s集群中安装部署。
Kube-Prometheus项目地址：https://github.com/prometheus-operator/kube-prometheus/
首先需要通过该项目地址，找到和自己Kubernetes版本对应的Kube Prometheus Stack的版本：(我的k8s集群版本是v1.31.4，选择release-0.14、release-0.15、release-0.16都可以，我这里选择release-0.14版本)
1.下载对应的release-0.14版本： 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 # 创建目录：17-prometheus是学习本章节目录，监控章节的所有文件都在该目录中完成 [root@k8s-master01 pra] mkdir -p /root/pra/17-prometheus # 进入目录 [root@k8s-master01 pra]# cd 17-prometheus/ # 拉取对应版本的文件包 [root@k8s-master01 17-prometheus]# git clone -b release-0.14 https://github.com/prometheus-operator/kube-prometheus.git Cloning into 'kube-prometheus'... remote: Enumerating objects: 20464, done. remote: Counting objects: 100% (3886/3886), done. remote: Compressing objects: 100% (262/262), done. remote: Total 20464 (delta 3715), reused 3700 (delta 3605), pack-reused 16578 (from 1) Receiving objects: 100% (20464/20464), 12.32 MiB | 3.03 MiB/s, done. Resolving deltas: 100% (14089/14089), done. # -b release-0.14 表示告诉 Git 在克隆仓库的同时，切换到指定的release-0.14分支（而不是默认的 main 或 master 分支） # 就不需要再额外执行 git checkout release-0.14 切换分支了 [root@k8s-master01 test]# ll total 4 drwxr-xr-x 12 root root 4096 Jul 9 18:40 kube-prometheus [root@k8s-master01 test]# cd kube-prometheus/ [root@k8s-master01 kube-prometheus]# ll total 132 -rwxr-xr-x 1 root root 679 Jul 9 18:40 build.sh -rw-r--r-- 1 root root 16162 Jul 9 18:40 CHANGELOG.md -rw-r--r-- 1 root root 2020 Jul 9 18:40 code-of-conduct.md -rw-r--r-- 1 root root 3782 Jul 9 18:40 CONTRIBUTING.md drwxr-xr-x 5 root root 69 Jul 9 18:40 developer-workspace drwxr-xr-x 4 root root 4096 Jul 9 18:40 docs -rw-r--r-- 1 root root 2273 Jul 9 18:40 example.jsonnet drwxr-xr-x 7 root root 4096 Jul 9 18:40 examples drwxr-xr-x 3 root root 28 Jul 9 18:40 experimental -rw-r--r-- 1 root root 2242 Jul 9 18:40 go.mod -rw-r--r-- 1 root root 15676 Jul 9 18:40 go.sum drwxr-xr-x 3 root root 29 Jul 9 18:40 jsonnet -rw-r--r-- 1 root root 400 Jul 9 18:40 jsonnetfile.json -rw-r--r-- 1 root root 6580 Jul 9 18:40 jsonnetfile.lock.json -rw-r--r-- 1 root root 1807 Jul 9 18:40 kubescape-exceptions.json -rw-r--r-- 1 root root 4902 Jul 9 18:40 kustomization.yaml -rw-r--r-- 1 root root 11325 Jul 9 18:40 LICENSE -rw-r--r-- 1 root root 3450 Jul 9 18:40 Makefile drwxr-xr-x 3 root root 8192 Jul 9 18:40 manifests # 所有的安装文件都在manifests目录中 -rw-r--r-- 1 root root 8970 Jul 9 18:40 README.md -rw-r--r-- 1 root root 3922 Jul 9 18:40 RELEASE.md drwxr-xr-x 2 root root 4096 Jul 9 18:40 scripts drwxr-xr-x 3 root root 17 Jul 9 18:40 tests # 进入manifests目录，所有的安装文件都在manifests目录中 [root@k8s-master01 17-prometheus]# cd kube-prometheus/manifests
如果上述clone命令无法拉取也可以下载压缩包，上传到服务器的目录中：
1 2 3 4 [root@k8s-master01 17-prometheus]# git clone -b release-0.14 https://github.com/prometheus-operator/kube-prometheus.git Cloning into 'kube-prometheus'... fatal: unable to access 'https://github.com/prometheus-operator/kube-prometheus.git/': Failed connect to github.com:443; Connection timed out # 如果遇到网络问题无法在线拉取可以参考下图解决
进入目标目录：
1 2 3 4 5 6 7 [root@k8s-master01 17-prometheus]# ll total 4 drwxr-xr-x 12 root root 4096 Nov 12 17:38 kube-prometheus # 进入manifests目录，所有的安装文件都在manifests目录中 [root@k8s-master01 17-prometheus]# cd kube-prometheus/manifests
2.安装Prometheus Operator CRD： 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 # 进入到manifests目录中执行下面命令 [root@k8s-master01 manifests]# kubectl create -f setup/ customresourcedefinition.apiextensions.k8s.io/alertmanagerconfigs.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/probes.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/prometheusagents.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/scrapeconfigs.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com created namespace/monitoring created # 会在monitoring命令空间中安装很多CRD资源 [root@k8s-master01 kube-prometheus]# kubectl get crd |grep monitoring alertmanagerconfigs.monitoring.coreos.com 2025-09-21T12:12:07Z alertmanagers.monitoring.coreos.com 2025-09-21T12:12:07Z podmonitors.monitoring.coreos.com 2025-09-21T12:12:07Z probes.monitoring.coreos.com 2025-09-21T12:12:08Z prometheusagents.monitoring.coreos.com 2025-09-21T12:12:08Z prometheuses.monitoring.coreos.com 2025-09-21T12:12:08Z prometheusrules.monitoring.coreos.com 2025-09-21T12:12:08Z scrapeconfigs.monitoring.coreos.com 2025-09-21T12:12:09Z servicemonitors.monitoring.coreos.com 2025-09-21T12:12:09Z thanosrulers.monitoring.coreos.com 2025-09-21T12:12:09Z
安装 Prometheus Operator 及核心组件：
需要修改镜像地址的文件包含下面所有文件：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 路径：~/kube-prometheus-release-0.14/manifests [root@k8s-master01 manifests]# grep "image:" *.yaml alertmanager-alertmanager.yaml: image: quay.io/prometheus/alertmanager:v0.27.0 blackboxExporter-deployment.yaml: image: quay.io/prometheus/blackbox-exporter:v0.25.0 blackboxExporter-deployment.yaml: image: ghcr.io/jimmidyson/configmap-reload:v0.13.1 blackboxExporter-deployment.yaml: image: quay.io/brancz/kube-rbac-proxy:v0.18.1 grafana-deployment.yaml: image: grafana/grafana:11.2.0 kubeStateMetrics-deployment.yaml: image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.13.0 kubeStateMetrics-deployment.yaml: image: quay.io/brancz/kube-rbac-proxy:v0.18.1 kubeStateMetrics-deployment.yaml: image: quay.io/brancz/kube-rbac-proxy:v0.18.1 nodeExporter-daemonset.yaml: image: quay.io/prometheus/node-exporter:v1.8.2 nodeExporter-daemonset.yaml: image: quay.io/brancz/kube-rbac-proxy:v0.18.1 prometheusAdapter-deployment.yaml: image: registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.12.0 prometheusOperator-deployment.yaml: image: quay.io/prometheus-operator/prometheus-operator:v0.76.2 prometheusOperator-deployment.yaml: image: quay.io/brancz/kube-rbac-proxy:v0.18.1 prometheus-prometheus.yaml: image: quay.io/prometheus/prometheus:v2.54.1 # 修改下面三个yaml文件中的镜像地址，像ghcr.io和quay.io中的镜像地址应该可以正常拉取，可以不改 kubeStateMetrics-deployment.yaml prometheusAdapter-deployment.yaml grafana-deployment.yaml
修改alertmanager的副本数和镜像下载地址：
1 2 3 4 5 # 在创建之前需要决定你集群的副本数，假设我这里是测试环境，所以只创建一个副本，如果是生产环境建议至少3个副本 # alertmanager副本数在alertmanager-alertmanager.yaml配置文件中修改 # 主要是副本数和镜像下载地址的配置： image: quay.io/prometheus/alertmanager:v0.27.0 #如果自己有镜像仓库可以改为自己的仓库地址，下图已经修改为我自己的镜像仓库地址了 replicas: 3 #根据实际情况修改副本的数量，我配置的是1
修改prometheus的副本数和镜像下载地址：
1 2 3 4 5 # 在创建之前需要决定你集群的副本数，假设我这里是测试环境，所以只创建一个副本，如果是生产环境建议至少3个副本 # prometheus副本数在 prometheus-prometheus.yaml配置文件中修改 # 主要是副本数和镜像下载地址的配置： image: quay.io/prometheus/alertmanager:v0.27.0 #如果自己有镜像仓库可以改为自己的仓库地址，下图已经修改为我自己的镜像仓库地址了 replicas: 3 #根据实际情况修改副本的数量，我配置的是1
修改kubeStateMetrics-deployment.yaml配置文件中镜像的下载地址：
1 # kubeStateMetrics-deployment.yaml配置文件中的镜像地址我改为我自己的仓库地址，事先已经将镜像同步到自己的仓库中
接下来安装prometheus的技术栈(安装需要至少3个节点，配个节点配置最低要4核4G)：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 # 注意所在目录manifests [root@k8s-master01 manifests]# pwd /root/pra/17-prometheus/kube-prometheus/manifests # 安装prometheus技术栈 [root@k8s-master01 manifests]# kubectl create -f . # 查看prometheus容器状态 [root@k8s-master01 manifests]# kubectl get po -n monitoring NAME READY STATUS RESTARTS AGE alertmanager-main-0 0/2 Init:0/1 0 9m54s blackbox-exporter-75c7985cb8-xkxrs 0/3 ContainerCreating 0 11m grafana-664dd67585-mh7tb 0/1 ImagePullBackOff 0 11m kube-state-metrics-bbcd9ff4f-8c5vq 0/3 ContainerCreating 0 11m node-exporter-7b428 2/2 Running 0 11m node-exporter-m2g8z 2/2 Running 0 11m node-exporter-nvpz5 2/2 Running 0 11m node-exporter-w7lrs 2/2 Running 0 11m node-exporter-wwtnm 0/2 ContainerCreating 0 11m prometheus-adapter-77f8587965-5x7bn 0/1 ImagePullBackOff 0 11m prometheus-adapter-77f8587965-wgp55 0/1 ImagePullBackOff 0 11m prometheus-k8s-0 0/2 Init:0/1 0 9m54s prometheus-operator-6f9479b5f5-4zpfm 2/2 Running 0 11m
虽然上述三个配置文件修改了镜像的下载地址，但是查看pod会发现还是会有一些pod由于镜像地址的问题无法启动pod，接下继续修改无法启动pod的那些镜像地址：
思路一：通过describe查看无法启动的pod使用的镜像是什么，然后利用github同步到我自己的阿里云仓库中，在把配置文件中的镜像地址修改为我自己的阿里云仓库地址。（这里就不演示了）
1 2 3 4 5 # 下面是几个pod的资源配置所在的文件 blackbox-exporter-75c7985cb8-xkxrs的镜像在blackboxExporter-deployment.yaml文件中 grafana-664dd67585-mh7tb 的镜像在grafana-deployment.yaml文件中 prometheus-adapter-77f8587965-5x7bn的镜像在prometheusAdapter-deployment.yaml文件中 node-exporter-w7lrs的镜像在nodeExporter-daemonset.yaml文件中
思路二：也可以通过在线编辑的方法直接修改镜像地址
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 # 用prometheus-adapter-77f8587965-5x7bn举例 [root@k8s-master01 manifests]# kubectl edit pod -n monitoring prometheus-adapter-77f8587965-5x7bn # 找到spec.containers.image参数，修改为自己的镜像仓库（前提是已经将镜像同步到自己的仓库中了） 42 image: registry.cn-beijing.aliyuncs.com/k8s-liujunwei/prometheus-adapter:v0.12.0 # 修改后保存退出，程序会自动更新配置，再次检查pod，已经正常运行 [root@k8s-master01 manifests]# kubectl get pod -n monitoring NAME READY STATUS RESTARTS AGE alertmanager-main-0 2/2 Running 0 15h blackbox-exporter-75c7985cb8-xkxrs 3/3 Running 0 15h grafana-664dd67585-mh7tb 1/1 Running 0 15h kube-state-metrics-bbcd9ff4f-8c5vq 3/3 Running 0 15h node-exporter-7b428 2/2 Running 0 15h node-exporter-m2g8z 2/2 Running 0 15h node-exporter-nvpz5 2/2 Running 0 15h node-exporter-w7lrs 2/2 Running 0 15h node-exporter-wwtnm 2/2 Running 0 15h prometheus-adapter-77f8587965-5x7bn 1/1 Running 0 15h prometheus-adapter-77f8587965-wgp55 1/1 Running 0 15h prometheus-k8s-0 2/2 Running 0 12h prometheus-operator-6f9479b5f5-4zpfm 2/2 Running 0 15h # 让我们来详细解释一下这些 Pod 的功能和作用： 1.prometheus-operator-6f9479b5f5-4zpfm：这是整个 Prometheus 监控栈的控制器，负责管理和维护 Prometheus 相关的自定义资源(CRD)，自动化部署和管理 Prometheus 实例、配置等 2.prometheus-k8s-0：核心的 Prometheus 服务器实例，负责收集、存储时序数据，执行数据查询和告警规则评估 3.alertmanager-main-0：负责告警管理和通知，处理来自 Prometheus 的告警，提供告警分组、抑制、静默等功能，支持多种通知方式(邮件、Slack、webhook等) 4.grafana-664dd67585-mh7tb：可视化平台，提供丰富的数据可视化界面，支持创建各种监控面板，可以展示 Prometheus 采集的数据 5.node-exporter-*（多个实例）：部署在每个节点上的数据采集器，收集主机级别的监控指标，如：CPU、内存、磁盘、网络等系统层面的数据 6.kube-state-metrics-bbcd9ff4f-8c5vq：收集 Kubernetes 对象的状态指标，监控 Pod、Deployment、StatefulSet 等资源的状态，提供集群层面的监控数据 7.blackbox-exporter-75c7985cb8-xkxrs：用于探测服务的可用性，支持 HTTP、HTTPS、DNS、TCP、ICMP 等协议探测，可以监控服务的响应时间、可用性等 8.prometheus-adapter-77f8587965-*（两个实例）：实现了自定义指标 API，使 Kubernetes HPA 可以使用 Prometheus 指标进行自动扩缩容，将 Prometheus 指标转换为 Kubernetes 可用的指标格式 # 这些组件协同工作形成了一个完整的监控体系： node-exporter 收集每个节点的系统级别资源的数据。 kube-state-metrics 采集 Kubernetes 资源的状态数据。 blackbox-exporter 用于探测外部服务的可用性。 Prometheus ：采集和存储指标数据。 Alertmanager 处理和分发告警。 Grafana 提供指标的可视化展示。 Prometheus Operator 管理整个监控栈，管理和部署 Prometheus 及其相关组件。 Prometheus Adapter 将 Prometheus 指标数据暴露给 Kubernetes HPA，用于自动伸缩。使监控指标可用于集群的自动化运维。 # 这样的架构既能监控基础设施，也能监控应用服务，同时支持告警和可视化，是一个非常完整的监控解决方案。
3.访问Grafana和Prometheus 新版本添加了NetworkPolicy，导致无法访问Grafana和Prometheus等服务，执行删除即可：
1 2 3 4 5 6 7 8 9 10 11 12 13 [root@k8s-master01 manifests]# kubectl delete networkpolicy --all -n monitoring networkpolicy.networking.k8s.io "alertmanager-main" deleted networkpolicy.networking.k8s.io "blackbox-exporter" deleted networkpolicy.networking.k8s.io "grafana" deleted networkpolicy.networking.k8s.io "kube-state-metrics" deleted networkpolicy.networking.k8s.io "node-exporter" deleted networkpolicy.networking.k8s.io "prometheus-adapter" deleted networkpolicy.networking.k8s.io "prometheus-k8s" deleted networkpolicy.networking.k8s.io "prometheus-operator" deleted # 再次查看已经没有相关资源 [root@k8s-master01 manifests]# kubectl get networkpolicy -n monitoring No resources found in monitoring namespace.
将alertmanager-main、grafana、prometheus-k8s三个Service改成NodePort类型：
1 2 3 4 5 6 7 8 9 10 11 12 [root@k8s-master01 manifests]# kubectl edit svc -n monitoring grafana apiVersion: v1 kind: Service ... selector: app.kubernetes.io/component: grafana app.kubernetes.io/name: grafana app.kubernetes.io/part-of: kube-prometheus sessionAffinity: None type: NodePort #将type的原值ClusterIP改为NodePort status: loadBalancer: {}
查看Grafana的Service暴露的端口，通过该端口访问grafana：
1 2 3 4 # grafana的NodePort为30391 [root@k8s-master01 manifests]# kubectl get svc -n monitoring grafana NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE grafana NodePort 10.96.250.130 <none> 3000:30391/TCP 20h
之后可以通过任意一个安装了kube-proxy服务的节点IP+32293 端口即可访问到Grafana,如果浏览器无法访问执行删除网络策略：
1 2 3 4 5 6 7 8 9 10 # 删除网络策略 [root@k8s-master01 manifests]# kubectl delete networkpolicy --all -n monitoring networkpolicy.networking.k8s.io "alertmanager-main" deleted networkpolicy.networking.k8s.io "blackbox-exporter" deleted networkpolicy.networking.k8s.io "grafana" deleted networkpolicy.networking.k8s.io "kube-state-metrics" deleted networkpolicy.networking.k8s.io "node-exporter" deleted networkpolicy.networking.k8s.io "prometheus-adapter" deleted networkpolicy.networking.k8s.io "prometheus-k8s" deleted networkpolicy.networking.k8s.io "prometheus-operator" deleted
Prometheus控制台：
Grafana控制台： Grafana默认登录的账号密码为admin/admin
alertmanager控制台：
4.数据持久化 4.1grafana数据持久化 kube-prometheus 项目中 Grafana 的默认部署（grafana-deployment.yaml）里，/var/lib/grafana 挂载的是 **emptyDir**，而不是 PVC。所以 Pod/Deployment 重建后，Grafana 里手动创建的 dashboard、数据源、用户等都会丢失。
我是直接拿 manifests/ 目录 kubectl apply -f 的方式部署（不是自己走 jsonnet 编译流程），最直接、改动最小的办法就是：手动创建一个 PVC，然后修改 grafana-deployment.yaml，把 volume 从 emptyDir 换成 PVC。
信息确认：
1 2 3 4 5 6 7 我的Grafana 部署在 monitoring 命名空间中（kube-prometheus 默认）； 我的Grafana副本数是1， replicas: 1（如果副本是 1，PVC 用 ReadWriteOnce 没问题；如果是多副本，必须用ReadWriteMany）； PVC 大小先给10Gi，可以自行调整； 我的集群中已经部署好了存储CubeFS： [root@k8s-master01 release-0.14]# kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE cfs-sc (default) csi.cubefs.com Delete Immediate true 195d
第1步：创建 Grafana 需要的 PVC
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 vim grafana-pvc.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: grafana-storage namespace: monitoring labels: app.kubernetes.io/name: grafana app.kubernetes.io/component: grafana app.kubernetes.io/part-of: kube-prometheus spec: accessModes: - ReadWriteMany storageClassName: cfs-sc resources: requests: storage: 10Gi
创建该pvc资源：
1 2 3 4 5 [root@k8s-master01 manifests]# kubectl create -f grafana-pvc.yaml persistentvolumeclaim/grafana-storage created [root@k8s-master01 manifests]# kubectl get pvc -n monitoring NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE grafana-storage Bound pvc-6092c93c-07ad-41f7-9d3c-14e7a0f54d66 10Gi RWX cfs-sc <unset> 15s
第 2 步：修改 grafana-deployment.yaml，把 emptyDir 换成 PVC
先看一下 manifests/grafana-deployment.yaml目录里现在的定义：
1 2 3 4 5 6 7 8 9 10 160 volumes: 161 - emptyDir: {} 162 name: grafana-storage volumes: - name: grafana-storage persistentVolumeClaim: claimName: grafana-storage
第3步：更新应用
1 2 [root@k8s-master01 kube-prometheus-release-0.14]# kubectl replace -f manifests/grafana-deployment.yaml deployment.apps/grafana replaced
第4步：检查配置
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 # 等待pod重启后检查挂载是否成效 [root@k8s-master01 kube-prometheus-release-0.14]# kubectl get po -n monitoring |grep grafana grafana-765f8849f-6c9d5 0/1 Running 0 33s grafana-77857544b8-lsqgn 1/1 Running 0 90m # grafana-765f8849f-6c9d5 新pod启动完成后就会销毁旧pod(grafana-77857544b8-lsqgn) # 验证挂载 [root@k8s-master01 kube-prometheus-release-0.14]# kubectl exec -ti -n monitoring grafana-765f8849f-6c9d5 -- sh /usr/share/grafana $ df -h Filesystem Size Used Available Use% Mounted on overlay 98.9G 15.6G 83.4G 16% / tmpfs 64.0M 0 64.0M 0% /dev tmpfs 200.0M 0 200.0M 0% /tmp tmpfs 200.0M 4.0K 200.0M 0% /etc/grafana /dev/mapper/rl-root 98.9G 15.6G 83.4G 16% /etc/hosts /dev/mapper/rl-root 98.9G 15.6G 83.4G 16% /dev/termination-log /dev/mapper/rl-root 98.9G 15.6G 83.4G 16% /etc/hostname /dev/mapper/rl-root 98.9G 15.6G 83.4G 16% /etc/resolv.conf shm 64.0M 0 64.0M 0% /dev/shm cubefs-pvc-6092c93c-07ad-41f7-9d3c-14e7a0f54d66 10.0G 1.8M 10.0G 0% /var/lib/grafana # 已经成功将cubefs-pvc-6092c93c-07ad-41f7-9d3c-14e7a0f54d66挂载到/var/lib/grafana # 此时在登录 Grafana，修改用户密码，配置监控模板，删除pod后重新登录数据都还在就说明持久化配置成功。
将密码重置为asdqwe123@ ,将pod删除重建后还是用修改后的密码登录，默认密码admin就失效了证明持久化成功
4.2Prometheus数据持久化 Prometheus ：存的是所有的时序监控数据（metrics 历史数据），emptyDir 意味着 Pod 重启/重新调度、节点故障、版本升级都会导致监控数据全部丢失，kubectl top、历史图表、告警回溯全部没了。这个必要性其实比 Grafana 更高。
Prometheus 和 Alertmanager 的持久化是 Prometheus Operator 官方支持的标准字段 （不像 Grafana 那样需要手动改 emptyDir），只需要在对应的 CR（Prometheus/Alertmanager 资源）里加 spec.storage.volumeClaimTemplate 即可，Operator 会自动帮你把底层 StatefulSet 换成用 PVC。
第 1 步：查看当前 CR 名称
kube-prometheus 默认部署的资源名一般是 k8s（Prometheus）和 main（Alertmanager），确认一下：
1 2 3 4 5 6 [root@k8s-master01 manifests]# kubectl -n monitoring get prometheus NAME VERSION DESIRED READY RECONCILED AVAILABLE AGE k8s 2.54.1 1 1 True True 292d [root@k8s-master01 manifests]# kubectl -n monitoring get alertmanager NAME VERSION REPLICAS READY RECONCILED AVAILABLE AGE main 0.27.0 1 1 True True 292d
第 2 步：给 Prometheus 加持久化
编辑 manifests/prometheus-prometheus.yaml（如果是改本地 manifests 后 apply 的话），在 spec: 下加 storage：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 vim manifests/prometheus-prometheus.yaml apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: labels: app.kubernetes.io/component: prometheus app.kubernetes.io/instance: k8s app.kubernetes.io/name: prometheus app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 2.54 .1 name: k8s namespace: monitoring spec: storage: volumeClaimTemplate: spec: storageClassName: cfs-sc accessModes: - ReadWriteOnce resources: requests: storage: 100Gi alerting: alertmanagers: - apiVersion: v2 name: alertmanager-main namespace: monitoring port: web enableFeatures: [] externalLabels: {} image: quay.io/prometheus/prometheus:v2.54.1 nodeSelector: kubernetes.io/os: linux podMetadata: labels: app.kubernetes.io/component: prometheus app.kubernetes.io/instance: k8s app.kubernetes.io/name: prometheus app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 2.54 .1 podMonitorNamespaceSelector: {} podMonitorSelector: {} probeNamespaceSelector: {} probeSelector: {} replicas: 1 resources: requests: memory: 400Mi ruleNamespaceSelector: {} ruleSelector: {} scrapeConfigNamespaceSelector: {} scrapeConfigSelector: {} securityContext: fsGroup: 2000 runAsNonRoot: true runAsUser: 1000 serviceAccountName: prometheus-k8s serviceMonitorNamespaceSelector: {} serviceMonitorSelector: {} version: 2.54 .1
添加13行到21行的配置，配置后的结果如图所示：
第3步：更新应用即可
1 2 [root@k8s-master01 manifests]# kubectl replace -f prometheus-prometheus.yaml prometheus.monitoring.coreos.com/k8s replaced
第4步：验证挂载
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 # 等待pod重启后查看容器内的挂载 [root@k8s-master01 manifests]# kubectl exec -ti -n monitoring prometheus-k8s-0 sh -- df -h Filesystem Size Used Available Use% Mounted on overlay 98.9G 15.6G 83.3G 16% / tmpfs 64.0M 0 64.0M 0% /dev cubefs-pvc-581d53e3-5dc2-4bd5-938c-35eb8af75aa9 100.0G 31.8M 100.0G 0% /prometheus #这里是配置的持久化存储 /dev/mapper/rl-root 98.9G 15.6G 83.3G 16% /etc/hosts /dev/mapper/rl-root 98.9G 15.6G 83.3G 16% /dev/termination-log /dev/mapper/rl-root 98.9G 15.6G 83.3G 16% /etc/hostname /dev/mapper/rl-root 98.9G 15.6G 83.3G 16% /etc/resolv.conf shm 64.0M 0 64.0M 0% /dev/shm tmpfs 7.4G 72.0K 7.4G 0% /etc/prometheus/config_out tmpfs 7.4G 0 7.4G 0% /etc/prometheus/certs /dev/mapper/rl-root 98.9G 15.6G 83.3G 16% /etc/prometheus/rules/prometheus-k8s-rulefiles-0 tmpfs 7.4G 0 7.4G 0% /etc/prometheus/web_config/web-config.yaml tmpfs 7.4G 12.0K 7.4G 0% /var/run/secrets/kubernetes.io/serviceaccount tmpfs 3.7G 0 3.7G 0% /proc/acpi tmpfs 64.0M 0 64.0M 0% /proc/kcore tmpfs 64.0M 0 64.0M 0% /proc/keys tmpfs 64.0M 0 64.0M 0% /proc/timer_list tmpfs 3.7G 0 3.7G 0% /proc/scsi tmpfs 3.7G 0 3.7G 0% /sys/firmware
4.3Alertmanager 数据持久化 同样编辑 manifests/alertmanager-alertmanager.yaml：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 vim manifests/alertmanager-alertmanager.yaml apiVersion: monitoring.coreos.com/v1 kind: Alertmanager metadata: labels: app.kubernetes.io/component: alert-router app.kubernetes.io/instance: main app.kubernetes.io/name: alertmanager app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 0.27.0 name: main namespace: monitoring spec: # image: quay.io/prometheus/alertmanager:v0.27.0 image: registry.cn-beijing.aliyuncs.com/k8s-liujunwei/alertmanager:v0.27.0 nodeSelector: kubernetes.io/os: linux # 添加storage配置 storage: volumeClaimTemplate: spec: storageClassName: cfs-sc accessModes: - ReadWriteOnce resources: requests: storage: 2Gi # alertmanager 数据量很小，几个 G 足够 podMetadata: labels: app.kubernetes.io/component: alert-router app.kubernetes.io/instance: main app.kubernetes.io/name: alertmanager app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 0.27.0 replicas: 1 resources: limits: cpu: 100m memory: 100Mi requests: cpu: 4m memory: 100Mi secrets: [] securityContext: fsGroup: 2000 runAsNonRoot: true runAsUser: 1000 serviceAccountName: alertmanager-main version: 0.27.0
添加17行到25行的内容，修改后的配置如图所示：
重新应用配置即可：
1 kubectl apply -f manifests/alertmanager-alertmanager.yaml
验证参考上述4.2 中的第四步即可。
