logstash 如果是新条目或前一条目的延续,则使用Grok语法和distinct

omqzjyyz  于 9个月前  发布在  Logstash
关注(0)|答案(1)|浏览(83)

我使用Grok作为默认插件来过滤我的日志,所以让我说我有一个简单的3个日志条目:

2023-08-17 10:10:50.751 +02:00 [WARNING] [Provider] Failed to collect
2023-08-17 10:10:50.751 +02:00 [Error] [Provider] Failed to collect
AdapterException: Connection from Adapter to turbine could not be established
   at IsReadyAsync(CancellationToken token) in C:\server\Connection.cs:line 403
   at AlarmsAsync(CancellationToken token) in C:\server\Connection.cs:line 242
   at AlarmsAsync(CancellationToken token) in C:\server\Connection.cs:line 256
   at EventsAsync(Unit unit, LiveEventSubscriptionData eventData, CancellationToken token) in C:\server\Events.cs:line 55
2023-08-17 10:10:50.751 +02:00 [WARNING] [Provider] Failed to collect

所以为了支持多行异常读取阅读我创建了这个Grok表达式:

filter {
      grok {
        match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{ISO8601_TIMEZONE:timezone} \[%{WORD:level}\] \[%{GREEDYDATA:source}\] %{GREEDYDATA:message}(?<message>(.|\r|\n)*)" }
      }
}

但现在这里存在的问题,它读3 entrires作为一个像:

{
  "source": "Provider",
  "message": [
    "Failed to collect",
    "\n2023-08-17 10:10:50.751 +02:00 [Error] [Provider] Failed to collect\nAdapterException: Connection from Adapter to turbine could not be established\n   at IsReadyAsync(CancellationToken token) in C:\\server\\Connection.cs:line 403\n   at AlarmsAsync(CancellationToken token) in C:\\server\\Connection.cs:line 242\n   at AlarmsAsync(CancellationToken token) in C:\\server\\Connection.cs:line 256\n   at EventsAsync(Unit unit, LiveEventSubscriptionData eventData, CancellationToken token) in C:\\server\\Events.cs:line 55\n2023-08-17 10:10:50.751 +02:00 [WARNING] [Provider] Failed to collect"
  ],
  "level": "WARNING",
  "timezone": "+02:00",
  "timestamp": "2023-08-17 10:10:50.751"
}

还尝试添加多行编解码器,但没有成功:

input {
 file {
   mode => "tail"
   path => "/usr/share/logstash/ingest_data/*"
   codec => multiline {
       pattern => "%{TIMESTAMP_ISO8601}"
       negate => true
       what => "previous"
   }
 }
}

因此,当我卡住了,有没有一种方法来通知表达式女巫行是配置,并与一个是新的日志条目?

disho6za

disho6za1#

所以我想出了这个方法来支持Logstash中的多行配置,在输入文件配置中需要codex,看起来像这样:

codec => multiline {
   pattern => "^%{TIMESTAMP_ISO8601}" <- pattern defining how new line starts
   negate => true
   what => "previous"
}

所以这个过滤器看起来像这样:

%{TIMESTAMP_ISO8601:event_time} %{ISO8601_TIMEZONE:timezone} \[%{WORD:level}\] \[%{GREEDYDATA:source}\] %{GREEDYDATA:message}

相关问题