用于http_forwared_for的Logstash nginx解析器

eoxn13cs 于 2022-11-02 发布在 Nginx

关注(0)|答案(1)|浏览(114)

我正在使用filebeat和logstash将nginx日志发送到elasticsearch。我的日志具有以下格式：

000.000.000.000 - - [17/Oct/2022:08:25:18 +0000] "OPTIONS /favicon.svg HTTP/1.1" 405 559 "https://example.net/auth/login" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36" "111.111.111.111, 222.222.222.222"

我有以下logstash的配置文件：

input {
    beats {
        port => 5035
    }
}

filter {
    grok {
        match => [ "message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:http_x_forwarded_for}"]
    }
    mutate {
        convert => ["response", "integer"]
        convert => ["bytes", "integer"]
        convert => ["responsetime", "float"]
    }
    geoip {
        source => "clientip"
        target => "geoip"
        add_tag => [ "nginx-geoip" ]
    }
    date {
        match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
    }
    useragent {
        source => "message"
    }
}

output {
    elasticsearch {
        hosts => "elasticsearch:9200"
        index => "weblogs-%{+YYYY.MM.dd}"
        document_type => "nginx_logs"
        user => "elastic"
        password => "changeme"
    }
    stdout { codec => rubydebug }
}

此管道以以下形式将日志保存到elasticsearch：

"response" : 405,
          "timestamp" : "17/Oct/2022:08:25:18 +0000",
          "os_version" : "10",
          "auth" : "-",
          "verb" : "OPTIONS",
          "clientip" : "000.000.000.000",
          "httpversion" : "1.1",
          "referrer" : "\"https://example.net/auth/login\"",
          "geoip" : { },
          "os" : "Windows",
          "os_name" : "Windows",
          "agent" : {
            "version" : "7.17.6",
            "hostname" : "0242869f2486",
            "type" : "filebeat",
            "id" : "4de3a108-35bf-4bd9-8b18-a5d8f9f2bc83",
            "ephemeral_id" : "3a5f78b5-bae0-41f6-8d63-eea700df6c3c",
            "name" : "0242869f2486"
          },
          "log" : {
            "file" : {
              "path" : "/var/log/nginx/access.log"
            },
            "offset" : 1869518
          },
          "bytes" : 559,
          "ident" : "-",
          "http_x_forwarded_for" : " \"111.111.111.111, 222.222.222.222\"",
          "os_full" : "Windows 10",
          "@timestamp" : "2022-10-17T08:25:18.000Z",
          "request" : "/favicon.svg",
          "device" : "Spider",
          "name" : "favicon",
          "input" : {
            "type" : "log"
          },
          "host" : {
            "name" : "0242869f2486"
          },
          "os_major" : "10",
          "@version" : "1",
          "message" : "000.000.000.000 - - [17/Oct/2022:08:25:18 +0000] \"OPTIONS /favicon.svg HTTP/1.1\" 405 559 \"https://example.net/auth/login\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36\" \"111.111.111.111, 222.222.222.222\"",
          "tags" : [
            "beats_input_codec_plain_applied",
            "_geoip_lookup_failure"
          ]

然而，我的目标是从http_forwared_for字段解析第一个IP，并添加一个名为real_client_ip的新字段，然后将其添加并保存到索引中。有没有办法实现这一点？

nginx

来源：https://stackoverflow.com/questions/74097417/logstash-nginx-parser-for-http-forwared-for

1条答案

按热度按时间

kzmpq1sx1#

您可以在第一个grok过滤器之后再添加一个grok过滤器到logstash管道。

filter {
    grok {
        match => [ "message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:http_x_forwarded_for}"]
    }
    grok {
        match => [ "http_x_forwarded_for" , "%{IP:real_client_ip}"]
    }
    mutate {
        convert => ["response", "integer"]
        convert => ["bytes", "integer"]
        convert => ["responsetime", "float"]
    }
    geoip {
        source => "clientip"
        target => "geoip"
        add_tag => [ "nginx-geoip" ]
    }
    date {
        match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
    }
    useragent {
        source => "message"
    }
}

PS：我已经在Kibana中验证了grok模式，但不是通过运行logstash管道。但这应该对你的用例有效。

赞(0）回复(0）举报 2022-11-02

我来回答

用于http_forwared_for的Logstash nginx解析器

1条答案

相关问题

热门标签

最新问答