logstash采集springboot微服务日志

前面一篇文章介绍了logstash采集nginx日志，这次介绍一下通过logstash采集springboot微服务中指定格式的日志。

需求

现有日志示例格式如下：

2025-07-21 11:16:37.629 INFO 8 --- [reactor-http-nio-1] c.test.authgateway.filter.GateWayFilter : logstashVO:{\"body\":\"{}\",\"fullName\":\"TEST\",\"time\":1753067797629,\"url\":\"/test/findUserInfo.do\",\"userId\":1088479580471164928}2025-07-21 11:16:37.629 INFO 8 --- [reactor-http-nio-1] c.test.authgateway.filter.GateWayFilter : logstashVO:{\"body\":\"{}\",\"fullName\":\"TEST\",\"time\":1753067797629,\"url\":\"/test/findUserInfo.do\",\"userId\":1088479580471164928}2025-07-21 11:16:37.629 INFO 8 --- [reactor-http-nio-1] c.test.authgateway.filter.GateWayFilter : logstashVO:{\"body\":\"{}\",\"fullName\":\"TEST\",\"time\":1753067797629,\"url\":\"/test/findUserInfo.do\",\"userId\":1088479580471164928}

需要采集“logstashVO:”之后的JSON格式字符串，该功能主要通过网关服务收集用户访问行为，当然根据用户需求可以做的事情还可以很多，这里就不一一列举。

logstash配置

input { file { path => [\"/data/logs/spring.log\"] type => \"test\" start_position => \"beginning\" }}filter { if \"c.test.authgateway.filter.GateWayFilter : logstashVO:\" not in [message] { drop {} } ruby { code => \" event.set(\'@timestamp\', LogStash::Timestamp.at(event.get(\'@timestamp\').time.localtime + 8*60*60)) \" } grok { match => { \"message\" => \"c\\.test\\.authgateway\\.filter\\.GateWayFilter \\: logstashVO\\:%{GREEDYDATA:logstashVO}\" } } json { source => \"logstashVO\" } json { source => \"body\" target => \"body\" } mutate { remove_field => [\"@version\",\"event\",\"host\",\"logstashVO\"] } }output { stdout { codec => rubydebug } kafka { bootstrap_servers => \"10.16.39.121:9092\" topic_id => \"test2-logs\" codec => json acks => \"all\" retries => 5 batch_size => 16384 }}

注意这里的输入input.file.path对应的路径一定要是镜像中的路径/data/logs/，而不能为宿主机里的路径/data/public/logstash/logs

启动镜像

docker run --log-driver json-file --log-opt max-size=100m --log-opt max-file=2 \\--privileged=true --network=host --restart=always --name logstash \\-v /data/public/logstash/config/:/usr/share/logstash/config/ \\-v /data/public/logstash/logs/:/data/logs/ \\-d logstash:8.4.0

检查日志采集

启动后，可以通过Offset Explorer连接Kafka检查是否正确采集到了相关的日志。

测试

echo \'2025-07-21 11:16:37.629 INFO 8 --- [reactor-http-nio-1] c.test.authgateway.filter.GateWayFilter : logstashVO:{\"body\":\"{}\",\"fullName\":\"TEST\",\"time\":1753067797629,\"url\":\"/test/findUserInfo.do\",\"userId\":1088479580471164928}\' >> /data/public/logstash/logs/spring.log

执行一下，可以看到新日志进入到Kafka中

写入Kafka中日志格式如下：

{ \"@timestamp\": \"2025-07-26T17:48:58.874280406Z\", \"body\": {}, \"url\": \"/test/findUserInfo.do\", \"fullName\": \"TEST\", \"log\": { \"file\": { \"path\": \"/data/logs/spring.log\" } }, \"type\": \"test\", \"time\": 1753067797629, \"userId\": 1088479580471164928, \"message\": \"2025-07-21 11:16:37.629 INFO 8 --- [reactor-http-nio-1] c.test.authgateway.filter.GateWayFilter : logstashVO:{\\\"body\\\":\\\"{}\\\",\\\"fullName\\\":\\\"TEST\\\",\\\"time\\\":1753067797629,\\\"url\\\":\\\"/test/findUserInfo.do\\\",\\\"userId\\\":1088479580471164928}\"}

健康管理