Prometheus+Grafana监控告警实战Demo

DevOps

2020-04-03

119

0

1.准备环境:

go环境的准备:
wget https://studygolang.com/dl/golang/go1.14.1.linux-amd64.tar.gz
tar zxvf go1.14.1.linux-amd64.tar.gz
配置环境变量
vi /etc/profile
source /etc/profile
验证是否安装成功
go version
 
 

2.下载prometheus:

tar zxvf prometheus-2.17.1.linux-amd64.tar.gz
配置文件:
[root@bd-test ~]# cat /opt/prometheus/prometheus-2.17.1.linux-amd64/prometheus.yml
# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).
  external_labels:
    monitor: 'codelab-monitor'

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']
运行:
./prometheus --config.file=prometheus.yml
打开网站:localhost:9090
 
封装系统进程:
vi /usr/lib/systemd/system/prometheus.service
[Unit]
Description=prometheus services
After=network.target
[Service]
Type=sample
User=root
ExecStart=/usr/bin/prometheus --config.file=/opt/soft/prometheus-2.17.1.linux-amd64/prometheus.yml
ExecStop=/bin/kill -s HUP $MAINPID
[Install]
WantedBy=multi-user.target
#######################
保存并执行, systemctl daemon-reload
ps -ef|grep prometheus
systemctl enable prometheus.service
systemctl start prometheus.service
systemctl stop prometheus.service
使用表达式进行查看图标
 
 
 

3.启动一些样本目标

启动一些样本目标
让我们变得更加有趣,并为Prometheus抓取一些示例目标。
Go客户端库包含一个示例,该示例导出具有不同延迟分布的三个服务的虚拟RPC延迟。
确保您已安装Go编译器,并设置了可正常运行的Go构建环境(带有正确的GOPATH)。
下载Prometheus的Go客户端库,并运行以下三个示例过程:
# Fetch the client library code and compile example.
git clone https://github.com/prometheus/client_golang.git
cd client_golang/examples/random
go get -d
go build
# Start 1 example targets in separate terminals:
./random -listen-address=:8080 
查看
 
以上说明已经对图表监控有一些认识了。。
接下来我们可以进行对mysql也做一些监控,需要对相应服务器安装mysql_export
 

4.部署mysql_export

在mysql服务器创建用户并授权
CREATE USER 'exporter'@'localhost' IDENTIFIED BY 'p@ssW0rd' WITH MAX_USER_CONNECTIONS 3;
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost';
flush privileges;
 
安装install:
tar zxvf mysqld_exporter-0.10.0.linux-amd64.tar.gz
cd mysqld_exporter-0.10.0.linux-amd64/
配置:
vi .my.cnf
$ vim .my.cnf
[client]
user=exporter
password= p@ssW0rd
 
启动mysql_exporter
./mysqld_exporter -config.my-cnf=".my.cnf"
 
 
在promotheus中添加配置
vim prometheus.yml
- job_name: mysql
static_configs:
- targets: ['localhost:9104']
labels:
instance: db1
 
重启prometheus,启动之后在status-查看targets
 
 对mysql进行简单的监控
 
 
以上的图表监控看起来不是很直观,没有那么炫酷,接下来我们引入granfa进行整合。
 
 

5.Grafana的部署

安装的两种方式:
tar -zxvf grafana-6.7.1.linux-amd64.tar.gz
cd grafana-6.7.1
./bin/grafana-server web
建议使用rpm安装:(省去自己封装systemd)
sudo yum install  grafana-6.7.1-1.x86_64.rpm
 
启动:
systemctl start grafana-server
进入web:
http://localhost:3000 
 设置数据源Prometheus,可以导入默认的三个配置模板
 
查看Prometheus的监控视图,是不是很美观,能读取到很多metrics:
设置mysql数据库,即可读属mysql_export
 
查看mysql_exporter
 

6.服务器信息监控node_exporter的部署

修改Prometheus的配置,在scrape_configs增加配置
scrape_configs:
    - job_name: 'node'
    static_configs:
    - targets: ['localhost:9100']
重启Prometheus,查看targets
 

 7.监控JAVA程序JVM之tomcat

jmx_exporter-parent-0.12.0
下载
进行编译,进入jmx_exporter-parent-0.12.0/jmx_prometheus_javaagent/
mvn package
(maven安装完成,并修改阿里镜像)
拷贝 tartget下的 jmx_prometheus_javaagent-0.12.0.jar和案列的tocat.yml(默认的配置文件)到tomcat/bin目录下
ll /usr/local/apache-tomcat/bin/
 
再tomcat/bin/catalina.sh添加如下配置
JAVA_OPTS=" -javaagent:/usr/local/apache-tomcat/bin/jmx_prometheus_javaagent-0.12.0.jar=5556:/usr/local/apache-tomcat/bin/tomcat.yml"
启动tomcat.
普罗米修斯中targets即可见,配置和node类似添加配置更改端口和job名,重启Prometheus查看
grafana中导入jvm overview的视图。设置数据源为Prometheus。即可看见程序的
 

8.告警邮件之node节点服务器内存监控

 
单独创建一个视图
查询设置:
(1 - (node_memory_MemAvailable_bytes{instance=~"122.51.71.194:9100"} / (node_memory_MemTotal_bytes{instance=~"122.51.71.194:9100"})))* 100
通用设置:
视图设置:
等待告警产生并发送邮件
-----------------------------------------------
Insert配置grafana配置smtp:
#################################### SMTP / Emailing ##########################
[smtp]
enabled = true
host = smtp.163.com:25
user = alert
# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
password = P@ssW0rd   #在163开启smtp,设置密码或者安全码
;cert_file =
;key_file =
;skip_verify = true
from_address = alert@163.com
from_name = aleart
# EHLO identity in SMTP dialog (defaults to instance_name)
;ehlo_identity = dashboard.example.com
--------------------------------------------------------------------------------------
以上就是监控的一些草稿,具体不懂的或者不清楚的欢迎大家留言,邮箱交流~~~

 

 

发表评论

全部评论:0条

lmy233

努力工作学习生活的人呐~~

联系方式

神圣之子仅供学习交流
E-mail:limengyu233@163.com
github:https://github.com/lmy01