efk(elasticsearch+fluentd-(td-agent)+kibana):kibana未显示正确的日志

0qx6xfy6  于 2021-06-15  发布在  ElasticSearch
关注(0)|答案(1)|浏览(1312)

我已经在redhat linux系统(7.6版)上安装了efk。fluentd分布稳定,即用td剂代替fluentd。日志文件(/mnt/log/startup.log)的路径在td-agent.conf文件中配置。但在kibana Jmeter 板上,它显示的是td-agent.log中的内容,而不是日志文件(startup.log)。
td-agent.log:

2020-09-04 16:02:16 +0530 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf"
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-elasticsearch' version '4.0.9'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-kafka' version '0.13.0'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-prometheus' version '1.8.0'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-prometheus_pushgateway' version '0.0.2'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.3.0'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-s3' version '1.3.2'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-systemd' version '1.0.2'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-td' version '1.1.0'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.4'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-webhdfs' version '1.2.5'
2020-09-04 16:02:16 +0530 [info]: gem 'fluentd' version '1.11.1'
2020-09-04 16:02:16 +0530 [info]: 'flush_interval' is configured at out side of <buffer>. 'flush_mode' is set to 'interval' to keep existing behaviour
2020-09-04 16:02:16 +0530 [debug]: 'host localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: localhost' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'index_name fluentd' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: fluentd' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2020-09-04 16:02:16 +0530 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: Need substitution: false
2020-09-04 16:02:16 +0530 [debug]: 'host_placeholder localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: localhost' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2020-09-04 16:02:16 +0530 [info]: using configuration file: <ROOT>
  <system>
    log_level debug
  </system>
  <source>
    @type tail
    path "/mnt/Log/Startup.log"
    pos_file "/mnt/Log/Startup.log.pos"
    format multiline
    format_firstline /\d{4}-\d{1,2}-\d{1,2}/
    format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/
    tag "log"
    <parse>
      format_firstline /\d{4}-\d{1,2}-\d{1,2}/
      @type multiline
      format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/
      unmatched_lines
    </parse>
  </source>
  <match *.**>
    @type elasticsearch
    host "localhost"
    port 9200
    include_tag_key true
    tag_key "@log_name"
    logstash_format true
    flush_interval 10s
    <buffer>
      flush_interval 10s
    </buffer>
  </match>
</ROOT>
2020-09-04 16:02:16 +0530 [info]: starting fluentd-1.11.1 pid=67918 ruby="2.4.10"
2020-09-04 16:02:16 +0530 [info]: spawn command to main:  cmdline=["/opt/td-agent/embedded/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/td-agent/embedded/bin/fluentd", "--log", "/var/log/td-agent/td-agent.log", "--daemon", "/var/run/td-agent/td-agent.pid", "--under-supervisor"]
2020-09-04 16:02:17 +0530 [info]: adding match pattern="*.**" type="elasticsearch"
2020-09-04 16:02:17 +0530 [info]: #0 'flush_interval' is configured at out side of <buffer>. 'flush_mode' is set to 'interval' to keep existing behaviour
2020-09-04 16:02:17 +0530 [debug]: #0 'host localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: localhost' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'index_name fluentd' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: fluentd' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2020-09-04 16:02:17 +0530 [debug]: #0 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 Need substitution: false
2020-09-04 16:02:17 +0530 [debug]: #0 'host_placeholder localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: localhost' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [warn]: #0 Detected ES 7.x: `_doc` will be used as the document `_type`.
2020-09-04 16:02:17 +0530 [info]: adding source type="tail"
2020-09-04 16:02:17 +0530 [warn]: #0 define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2020-09-04 16:02:17 +0530 [info]: #0 starting fluentd worker pid=67935 ppid=67930 worker=0
2020-09-04 16:02:17 +0530 [debug]: #0 buffer started instance=70139276565080 stage_size=0 queue_size=0
2020-09-04 16:02:17 +0530 [debug]: #0 enqueue_thread actually running
2020-09-04 16:02:17 +0530 [debug]: #0 tailing paths: target = /mnt/Log/Startup.log | existing =
2020-09-04 16:02:17 +0530 [info]: #0 following tail of /mnt/Log/Startup.log
2020-09-04 16:02:17 +0530 [info]: #0 fluentd worker is now running worker=0
2020-09-04 16:02:17 +0530 [debug]: #0 flush_thread actually running

尽管上面的日志显示它跟在startup.log后面。它仍然显示kibana Jmeter 板上td-agent.log的内容,而不是配置的日志文件中的内容。我也能找到kibana的默认索引
td-agent.conf格式:

<system>
log_level debug
</system>

<source>
@type tail
path /mnt/Log/Startup.log
pos_file /mnt/Log/Startup.log.pos
format multiline
format_firstline /\d{4}-\d{1,2}-\d{1,2}/
format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/
tag log
</source>

<match *.**>
   @type elasticsearch
    host localhost
    port 9200
    include_tag_key true
    tag_key @log_name
    logstash_format true
    flush_interval 10s
</match>

版本详细信息:
elasticsearch 7.8.0版fluentd(td代理)3.8.0版kibana 7.8.0版
我已使用RPM安装了上述所有工具:
elasticsearch-7.8.1-x86_64.rpm td-agent-3.8.0-0.el7.x86_64.rpm kibana-7.8.1-x86_64.rpm
curl -x得到“localhost:9200/?pretty“给予:

{
  "name" : "ncnsidapp2",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "MnfFXTyZT0ahp9u4eLSW2A",
  "version" : {
    "number" : "7.8.1",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "b5ca9c58fb664ca8bf9e4057fc229b3396bf3a89",
    "build_date" : "2020-07-21T16:40:44.668009Z",
    "build_snapshot" : false,
    "lucene_version" : "8.5.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

我可以在kibana上找到默认索引。
有什么问题吗?请帮忙。

ttygqcqt

ttygqcqt1#

尝试设置 logstash_prefix 在你的区域。当您将logstash\u format设置为true时,它将覆盖您设置为索引名的内容。如果设置logstash\u prefix,则默认情况下索引名的格式为{logstash\u prefix}-%y.%m.%d。

相关问题