求分析:logstash中使用codec multiline合并行,使用filter grok匹配所需字段信息出错

zhang506281581 2017-11-27 04:21:29
使用logstash采集日志文件中的日志信息,
logstash启动配置文件logstash.conf配置如下:
1 input {
2 file{
3 path => "/mnt/DShare/nginx/access.log.1"
4 type => "nginx"
5 start_position => "beginning"
6 codec => multiline {
7 pattern => "[=]"
8 negate => "false"
9 what => "previous"
10 }
11 }
12 }
13 filter {
14 grok {
15 match => { "message" => "%{ERRORID:ErrorId} %{BAYID:bayId}"
16 }
17 }
18 }
19
20 output {
21 stdout{
22 codec => rubydebug
23 }
24}
file中使用codec的multiline插件进行行的合并,filter中使用grok正则匹配所需字段,问题是:单独匹配%{ERRORID:ErrorId} 或%{BAYID:bayId}时均能正常匹配数据,但当两个字段同时进行匹配时报"_grokparsefailure"错误。

输出结果
正常输出:
匹配pattern:ERRORID
"path" => "/mnt/DShare/nginx/access.log.1",
"ErrorId" => "Error005",
"@timestamp" => 2017-11-27T07:57:27.352Z,
"@version" => "1",
"host" => "ubuntu-VirtualBox",
"message" => "[2017-11-23 13:20:48,996],travelRecord convert failed, bayId:bay005, orientation:WEST, validate failed for CaptureDirection value : aaaa\r\n[2017-11-23 13:20:49,996],travelRecord convert failed, ErrorId=Error005\r",
"type" => "nginx",
"tags" => [
[0] "multiline"
]
}

匹配模式BAYID:
{
"path" => "/mnt/DShare/nginx/access.log.1",
"@timestamp" => 2017-11-27T07:45:31.364Z,
"bayId" => "bay007",
"@version" => "1",
"host" => "ubuntu-VirtualBox",
"message" => "[2017-11-23 13:20:40,996],travelRecord convert failed, bayId:bay007, orientation:SOUTH, validate failed for CaptureDirection value : aaaa\r\n[2017-11-23 13:20:49,996],travelRecord convert failed, ErrorId=Error007\r",
"type" => "nginx",
"tags" => [
[0] "multiline"
]
}

当同时匹配ERRORID和BAYID时解析错误,输出结果:
{
"path" => "/mnt/DShare/nginx/access.log.1",
"@timestamp" => 2017-11-27T08:00:53.351Z,
"@version" => "1",
"host" => "ubuntu-VirtualBox",
"message" => "[2017-11-23 13:20:40,996],travelRecord convert failed, bayId:bay007, orientation:SOUTH, validate failed for CaptureDirection value : aaaa\r\n[2017-11-23 13:20:49,996],travelRecord convert failed, ErrorId=Error007\r",
"type" => "nginx",
"tags" => [
[0] "multiline",
[1] "_grokparsefailure"
]
}
...全文
1069 回复 打赏 收藏 转发到动态 举报
写回复
用AI写文章
回复
切换为时间正序
请发表友善的回复…
发表回复

2,760

社区成员

发帖
与我相关
我的任务
社区描述
搜索引擎的服务器通过网络搜索软件或网络登录等方式,将Internet上大量网站的页面信息收集到本地,经过加工处理建立信息数据库和索引数据库。
社区管理员
  • 搜索引擎技术社区
加入社区
  • 近7日
  • 近30日
  • 至今
社区公告
暂无公告

试试用AI创作助手写篇文章吧