Prometheus Consul Blackbox | export 监控实现

新钛云服已累计为您分享711篇技术干货

前言：

• blackbox_exporter

是Prometheus 官方提供的 exporter 之一,主要提供http、dns、tcp、icmp 的监控数据采集。

• Consul

主要提供，服务发现，健康检查，等功能，本次集成主要使用到服务发现功能。

本文主要实现，基于consul_sd_config & consul 的 prometheus 服务发现，实现网路设备ping监控，站点可用行监控，以及证书相关信息监控。

安装环境：

• k8s

• consul

• Prometheus

• blackbox_exporter

1: Consul 安装

1.1：使用helm 安装 consul

Bash

# 添加 consul helm 源

helm repo add hashicorp https://helm.releases.hashicorp.com

# 安装consul

helm -n consul install \

set

storageClass=alicloud-disk-efficiency \

consul hashicorp/consul \

--version=0.32.1

1.2：查看服务安装状态

Bash

[root@xxxxxxxx consul_install]

# kubectl -n consul get pods

NAME READY STATUS RESTARTS AGE

consul-consul-9lxfc 1/1 Running 0 6d1h

consul-consul-ntqcf 1/1 Running 0 6d1h

consul-consul-q7c6f 1/1 Running 0 6d1h

consul-consul-server-0 1/1 Running 0 6d1h

consul-consul-server-1 1/1 Running 0 6d1h

consul-consul-server-2 1/1 Running 0 6d1h

1.3：nginx-ingress consul

• consul_ingress.yml

Bash

# consul.xxxxxx.cn -----> 替换为正确域名

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

namespace: consul

annotations:

kubernetes.io/ingress.class: nginx

nginx.ingress.kubernetes.io/rewrite-target: /

spec:

rules:

- host: consul.xxxxxx.cn

http:

paths:

- path: /

pathType: Prefix

backend:

service:

port:

number: 80

• 执行部署

Bash

kubectl apply -f consul_ingress.yml

1.4：访问测试

2: Blackbox_export

2.1：blackbox 安装

• blackbox-exporter-config.yaml

Bash

apiVersion: v1

kind: ConfigMap

metadata:

labels:

app: blackbox-exporter

data:

blackbox.yml: |-

modules:

## ----------- DNS 检测配置 -----------

dns_tcp:

prober: dns

dns:

transport_protocol:

"tcp"

preferred_ip_protocol:

"ip4"

query_name:

"kubernetes.default.svc.cluster.local"# 用于检测域名可用的网址

query_type:

"A"
## ----------- TCP 检测模块配置 -----------

tcp_connect:

prober: tcp

timeout: 5s

## ----------- ICMP 检测配置 -----------

ping:

prober: icmp

timeout: 5s

icmp:

preferred_ip_protocol:

"ip4"
## ----------- HTTP GET 2xx 检测模块配置 -----------

http_get_2xx:

prober: http

timeout: 10s

http:

method: GET

preferred_ip_protocol:

"ip4"

valid_http_versions: [

"HTTP/1.1"

"HTTP/2"

]

valid_status_codes: [200]

# 验证的HTTP状态码,默认为2xx

no_follow_redirects:

false# 是否不跟随重定向
## ----------- HTTP GET 3xx 检测模块配置 -----------

http_get_3xx:

prober: http

timeout: 10s

http:

method: GET

preferred_ip_protocol:

"ip4"

valid_http_versions: [

"HTTP/1.1"

"HTTP/2"

]

valid_status_codes: [301,302,304,305,306,307]

# 验证的HTTP状态码,默认为2xx

no_follow_redirects:

false# 是否不跟随重定向
## ----------- HTTP POST 监测模块 -----------

http_post_2xx:

prober: http

timeout: 10s

http:

method: POST

preferred_ip_protocol:

"ip4"

valid_http_versions: [

"HTTP/1.1"

"HTTP/2"

]

#headers:                             # HTTP头设置
#  Content-Type: application/json
#body: '{}'                           # 请求体设置

• blackbox-exporter-deploy.yaml

Bash

apiVersion: v1

kind: Service

metadata:

labels:

k8s-app: blackbox-exporter

spec:

type

: ClusterIP

ports:

- name: http

port: 9115

targetPort: 9115

selector:

k8s-app: blackbox-exporter

---

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

k8s-app: blackbox-exporter

spec:

replicas: 1

selector:

matchLabels:

k8s-app: blackbox-exporter

template:

metadata:

labels:

k8s-app: blackbox-exporter

spec:

containers:

- name: blackbox-exporter

image: prom/blackbox-exporter:v0.19.0

args:

- --config.file=/etc/blackbox_exporter/blackbox.yml

- --web.listen-address=:9115

- --log.level=info

ports:

- name: http

containerPort: 9115

resources:

limits:

cpu: 3

memory: 6000Mi

requests:

cpu: 100m

memory: 50Mi

livenessProbe:

tcpSocket:

port: 9115

initialDelaySeconds: 5

timeoutSeconds: 5

periodSeconds: 10

successThreshold: 1

failureThreshold: 3

readinessProbe:

tcpSocket:

port: 9115

initialDelaySeconds: 5

timeoutSeconds: 5

periodSeconds: 10

successThreshold: 1

failureThreshold: 3

volumeMounts:

- name: config

mountPath: /etc/blackbox_exporter

volumes:

- name: config

configMap:

defaultMode: 420

• 执行安装

Bash

kubectl apply -f blackbox-exporter-deploy.yaml

kubectl apply -f blackbox-exporter-config.yaml

2.2：nginx ingress blackbox-exporter • blackbox_ingress.yml

Bash

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

namespace: monitoring

annotations:

kubernetes.io/ingress.class: nginx

nginx.ingress.kubernetes.io/rewrite-target: /

spec:

rules:

- host: blackbox-devops.lululemon.cn

http:

paths:

- path: /

pathType: Prefix

backend:

service:

port:

number: 9115

• 执行安装

Bash

kubectl apply -f blackbox_ingress.yml

3: rometheus 添加服务动态发现

Bash

##### http_get_2xx 数据获取

- job_name: http_get_2xx

params:

module:

- http_get_2xx

scrape_interval: 2s

scrape_timeout: 2s

metrics_path: /probe

consul_sd_configs:

# consul 服务地址

- server: consul-consul-server.consul.svc.cluster.local:8500

tag_separator:

','

services:

- http_get_2xx

relabel_configs:

- source_labels: [

'__meta_consul_service_address'

]

target_label: __param_target

- source_labels: [

'__meta_consul_service_address'

]

target_label: instance

- target_label: __address__

## blackbox-export 地址

replacement: blackbox-exporter.monitoring.svc.cluster.local:9115

####### icmp 配置

- job_name: blackbox_icmp

params:

module:

- ping

scrape_interval: 2s

scrape_timeout: 2s

metrics_path: /probe

consul_sd_configs:

# consul 服务地址

- server: consul-consul-server.consul.svc.cluster.local:8500

tag_separator:

','

services:

- ping

relabel_configs:

- source_labels: [

'__meta_consul_service_address'

]

target_label: __param_target

- source_labels: [

'__meta_consul_service_address'

]

target_label: instance

- target_label: __address__

## blackbox-export 地址

replacement: blackbox-exporter.monitoring.svc.cluster.local:9115

4：添加 icmp 监控

4.1：添加监控地址到consul

• icmp_list

Bash

192.168.1.1

192.168.1.2

• add_consul_service_icmp.sh

Bash

#!/usr/bin/env bash

ip_addr=

$1

iftest"$ip_addr"

;

then

curl -X PUT -d

"id": "icmp_'${ip_addr}'",

"name": "ping",

"address": "'${ip_addr}'",

"port": 443,

"Meta": {

"env": "prod",

"team": "network",

"project": "network",

"owner": "Mike"

"tags": ["node"],

"checks": [{"http": "http://blackbox-exporter.monitoring.svc.cluster.local:9115/","interval": "15s"}]}'

http://consul-consul-server:8500/v1/agent/service/register

else
echo"请输入参数"
fi

• 添加service ping

Bash

for

`cat icmp_list`;

bash add_consul_service_icmp.sh

;

done

4.2：查看consul 服务

4.3：删除ping 监控地址脚本

Bash

#!/usr/bin/env bash

ip_addr=

curl -X PUT http://consul-consul-server:8500/v1/agent/service/deregister/icmp_

${ip_addr}

5: 添加http_get_2xx

5.1：添加监控域名

• domain_name_list

Bash

wwww.baidu.com

wwww.1111.com

wwww.2222.com

• add_consul_service_http_get_2xx.sh

Bash

#!/usr/bin/env bash

service_name=

$1

iftest"$service_name"

;

then

curl -X PUT -d

"id": "http_get_2xx_'${service_name}'",

"name": "http_get_2xx",

"address": "https://'${service_name}'",

"port": 443,

"Meta": {

"env": "prod",

"team": "web",

"project": "web",

"owner": "Devops"

"tags": ["node"],

"checks": [{"http": "http://blackbox-exporter.monitoring.svc.cluster.local:9115/","interval": "15s"}]}'

http://consul-consul-server:8500/v1/agent/service/register

else
echo"请输入参数"
fi

• 添加 service http_get_2xx

Bash

for

`cat domain_name_list`;

bash add_consul_service_http_get_2xx.sh

;

done

5.2：查看consul 服务

5.3：删除域名监控脚本

• del_consul_service_http_get_2xx.sh

Bash

#!/usr/bin/env bash

ip_addr=

curl -X PUT http://consul-consul-server:8500/v1/agent/service/deregister/http_get_2xx_

${ip_addr}

6：查看prometheus 监控

总结：

使用上述方案，黑盒监控与自建cmdb 平台很容易进行集成，使其监控自动化，不需要过多的人工干预，可以省去大量的人工成本，grafana 的配置这里就不进行过多介绍，自行通过谷歌完成。

推荐阅读

推荐视频

继续阅读

阅读原文

关键词

服务发现

功能

信息

平台

方案