APP
prometheus監(jiān)控JMX需要向Tomcat內(nèi)添加jmx_prometheus_javaagent包，并暴露一個端口給外部訪問來獲取數(shù)據(jù)。
本文采用0.3.1版本：??jmx_prometheus_javaagent-0.3.1.jar??
Tomcat采用docker部署，生產(chǎn)環(huán)境建議做成鏡像用k8s啟動。
1、前置準備
本文采用docker方式部署Tomcat
1.1、創(chuàng)建install_tomcat腳本
# cat install_tomcat.sh
docker run -d
--name tomcat-1
-v /root/manifests/jvm/prom-jvm-demo:/jmx-exporter
-e CATALINA_OPTS="-Xms64m -Xmx128m -javaagent:/jmx-exporter/jmx_prometheus_javaagent-0.3.1.jar=6060:/jmx-exporter/simple-config.yml"
-p 6060:6060
-p 8080:8080
tomcat:latest
1.2、創(chuàng)建prometheus-serviceMonitorJvm.yaml，用于向kube-prometheus內(nèi)添加serviceMonitor
# cat prometheus-serviceMonitorJvm.yaml
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: jmx-metrics
namespace: monitoring
labels:
k8s-apps: jmx-metrics
spec:
jobLabel: metrics
selector:
matchLabels:
metrics: jmx-metrics # 根據(jù)label中有metrics: jmx-metrics 的service
namespaceSelector:
any: true # 所有名稱空間
endpoints:
- port: http-metrics # 拉取的端口和下邊對應(yīng)
interval: 15s # 拉取metric的時間間隔
---
apiVersion: v1
kind: Service
metadata:
annotations:
prometheus.io/scrapetrue: "true"
labels:
metrics: jmx-metrics # 和上邊一致
name: kube-jmx
namespace: monitoring
spec:
ports:
- name: http-metrics
port: 6060 # service的端口，供上邊獲取數(shù)據(jù)
protocol: TCP
targetPort: 6060 # 綁定宿主機的端口，也就是jmx_prometheus_javaagent的端口
# selector: # 因為是手動指定endpoint所以不能添加。否則過一段時間查看service的endpoint會為空
# k8s-app: kube-jmx # 如果是k8s部署的就需要寫標簽，去綁定tomcat pod的service
---
apiVersion: v1
kind: Endpoints
metadata:
name: kube-jmx
namespace: monitoring
subsets:
- addresses:
- ip: 宿主機IP
ports:
- name: http-metrics
port: 6060 # jmx_prometheus_javaagent的端口
protocol: TCP
1.3、在當前目錄創(chuàng)建prom-jvm-demo，下載jmx_prometheus_javaagent至prom-jvm-demo目錄下
# ls
docker.yaml prometheus-serviceMonitorJvm.yaml
# mkdir prom-jvm-demo
# wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar -O prom-jvm-demo/jmx_prometheus_javaagent-0.3.1.jar
1.4、進入prom-jvm-demo目錄內(nèi)，創(chuàng)建jmx_prometheus_javaagent的配置文件simple-config.yml
# cd prom-jvm-demo
# cat simple-config.yml
---
lowercaseOutputLabelNames: true
lowercaseOutputName: true
whitelistObjectNames: ["java.lang:type=OperatingSystem"]
rules:
- pattern: 'java.lang<type=OperatingSystem><>((?!process_cpu_time)w+):'
name: os_$1
type: GAUGE
attrNameSnakeCase: true
2、啟動Tomcat
2.1、執(zhí)行install_tomcat.sh
# sh install_tomcat.sh
2.2、測試訪問6060端口
# curl 127.0.0.1:6060
# HELP jmx_config_reload_success_total Number of times configuration have successfully been reloaded.
# TYPE jmx_config_reload_success_total counter
jmx_config_reload_success_total 0.0
# HELP os_free_swap_space_size FreeSwapSpaceSize (java.lang<type=OperatingSystem><>FreeSwapSpaceSize)
# TYPE os_free_swap_space_size gauge
os_free_swap_space_size 0.0
# HELP os_free_physical_memory_size FreePhysicalMemorySize (java.lang<type=OperatingSystem><>FreePhysicalMemorySize)
# TYPE os_free_physical_memory_size gauge
os_free_physical_memory_size 3.68160768E8
# HELP os_max_file_descriptor_count MaxFileDescriptorCount (java.lang<type=OperatingSystem><>MaxFileDescriptorCount)
# TYPE os_max_file_descriptor_count gauge
os_max_file_descriptor_count 1048576.0
# HELP os_system_load_average SystemLoadAverage (java.lang<type=OperatingSystem><>SystemLoadAverage)
# TYPE os_system_load_average gauge
os_system_load_average 1.74
# HELP os_total_physical_memory_size TotalPhysicalMemorySize (java.lang<type=OperatingSystem><>TotalPhysicalMemorySize)
# TYPE os_total_physical_memory_size gauge
os_total_physical_memory_size 3.974213632E9
# HELP os_committed_virtual_memory_size CommittedVirtualMemorySize (java.lang<type=OperatingSystem><>CommittedVirtualMemorySize)
# TYPE os_committed_virtual_memory_size gauge
os_committed_virtual_memory_size 3.71601408E9
# HELP os_system_cpu_load SystemCpuLoad (java.lang<type=OperatingSystem><>SystemCpuLoad)
# TYPE os_system_cpu_load gauge
os_system_cpu_load 0.10213187902825979
# HELP os_available_processors AvailableProcessors (java.lang<type=OperatingSystem><>AvailableProcessors)
# TYPE os_available_processors gauge
os_available_processors 4.0
# HELP os_process_cpu_load ProcessCpuLoad (java.lang<type=OperatingSystem><>ProcessCpuLoad)
# TYPE os_process_cpu_load gauge
os_process_cpu_load 0.0
......省略
3、創(chuàng)建serviceMonitor
3.1、apply?prometheus-serviceMonitorJvm.yaml
# kubectl apply -f prometheus-serviceMonitorJvm.yaml
3.2、查看創(chuàng)建的serviceMonitorJvm、service、endpoint
# kubectl get servicemonitors,svc,endpoints -n monitoring | grep jmx
servicemonitor.monitoring.coreos.com/jmx-metrics 4d20h
service/kube-jmx ClusterIP 10.0.0.157 <none> 6060/TCP 4d20h
endpoints/kube-jmx xxx.xx.x.xxx:6060 4d20h
4、查看prometheus WEB
4.1、訪問prometheus web頁面
5、添加grafana展示
5.1、在grafana內(nèi)添加dashboards:8878
6、添加告警規(guī)則
6.1、編寫prometheus-rules-add-jvm.yaml，也可在prometheus-rules.yaml內(nèi)追加
# cat prometheus-rules-add-jvm.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
labels:
prometheus: k8s
role: alert-rules
name: jvm-metrics-rules
namespace: monitoring
spec:
groups:
- name: jvm-metrics-rules
rules:
# 在5分鐘里，GC花費時間超過10%
- alert: GcTimeTooMuch
expr: increase(jvm_gc_collection_seconds_sum[5m]) > 30
for: 5m
labels:
severity: red
annotations:
summary: "{{ $labels.app }} GC時間占比超過10%"
message: "ns:{{ $labels.namespace }} pod:{{ $labels.pod }} ip:{{ $labels.instance }} GC時間占比超過10%，當前值({{ $value }}%)"
# GC次數(shù)太多
- alert: GcCountTooMuch
expr: increase(jvm_gc_collection_seconds_count[1m]) > 30
for: 1m
labels:
severity: red
annotations:
summary: "{{ $labels.app }} 1分鐘GC次數(shù)>30次"
message: "ns:{{ $labels.namespace }} pod:{{ $labels.pod }} ip:{{ $labels.instance }} 1分鐘GC次數(shù)>30次，當前值({{ $value }})"
# FGC次數(shù)太多
- alert: FgcCountTooMuch
expr: increase(jvm_gc_collection_seconds_count{gc="ConcurrentMarkSweep"}[1h]) > 3
for: 1m
labels:
severity: red
annotations:
summary: "{{ $labels.app }} 1小時的FGC次數(shù)>3次"
message: "ns:{{ $labels.namespace }} pod:{{ $labels.pod }} ip:{{ $labels.instance }} 1小時的FGC次數(shù)>3次，當前值({{ $value }})"
# 非堆內(nèi)存使用超過80%
- alert: NonheapUsageTooMuch
expr: jvm_memory_bytes_used{job="jmx-metrics", area="nonheap"} / jvm_memory_bytes_max * 100 > 80
for: 5m
labels:
severity: red
annotations:
summary: "{{ $labels.app }} 非堆內(nèi)存使用>80%"
message: "ns:{{ $labels.namespace }} pod:{{ $labels.pod }} ip:{{ $labels.instance }} 非堆內(nèi)存使用率>80%，當前值({{ $value }}%)"
# 內(nèi)存使用預警
- alert: HeighMemUsage
expr: process_resident_memory_bytes{job="jmx-metrics"} / os_total_physical_memory_bytes * 100 > 85
for: 5m
labels:
severity: red
annotations:
summary: "{{ $labels.app }} rss內(nèi)存使用率大于85%"
message: "ns:{{ $labels.namespace }} pod:{{ $labels.pod }} ip:{{ $labels.instance }} rss內(nèi)存使用率大于85%，當前值({{ $value }}%)"
# 堆內(nèi)存使用超過85%
- alert: heapUsageTooMuch
expr: jvm_memory_bytes_used{area="heap"} / jvm_memory_bytes_max * 100 > 95
for: 5m
labels:
severity: red
annotations:
summary: "{{ $labels.app }} 堆內(nèi)存使用>85%"
message: "ns:{{ $labels.namespace }} pod:{{ $labels.pod }} ip:{{ $labels.instance }} 堆內(nèi)存使用率>85%，當前值({{ $value }}%)"
6.2、執(zhí)行并在prometheus Alerts內(nèi)查看
# kubectl apply -f prometheus-rules-add-jvm.yaml
本文摘自 ：https://blog.51cto.com/u
