kubernetes FluentD日志不可读,已排除,下次将进行检查

sd2nnvve  于 5个月前  发布在  Kubernetes
关注(0)|答案(7)|浏览(123)

Facing:fluentd log unreadable. it is excluded and would be examined next time

我有一个在kubernetes设置中运行的fluentD守护进程集的简单配置。
Fluentd版本:fluentd-0.12.43
下面是我的配置。

<source>
    @type tail
    path /var/log/containers/sample*.log
    time_format %Y-%m-%dT%H:%M:%S.%NZ
    tag sample.*
    format json
    read_from_head true
  </source>
  <match sample.**>
    @type forward
    heartbeat_type tcp
    send_timeout 60s
    recover_wait 10s
    hard_timeout 60s
    <server>
      name worker-node2
      host 10.32.0.15
      port 24224
      weight 60
    </server>
  </match>

字符串
低于警告,没有转发日志
2018-08-03 06:36:53 +0000 [warn]:/var/log/containers/samplelog-79bd66868b-t7xn9_logging1_fluentd-70e85c5d6328e7d.log不可读。已排除,下次将进行检查。
2018-08-03 06:37:53 +0000 [warn]:/var/log/containers/samplelog-79bd66868b-t7xn9_logging1_fluentd-70e85c5bc89ab24.log不可读。已排除,下次将进行检查。

日志文件权限:

[root@k8s-master fluentd-daemonset]# ls -lrt **/var/log/containers/**

**lrwxrwxrwx** Jun 25 06:25 sample-77g68_kube-system_kube-proxy-9f3c3951c32ee.log 
-> /var/log/pods/aa1f8d5b-746f-11e8-95c0-005056b9ff3a/sample/7.log

守护进程集的YAML文件有挂载指令:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: logging1
  labels:
    k8s-app: fluentd-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  template:
    -----
    -----
    -----

        volumeMounts:
        - name: fluentd-config
          mountPath: /fluentd/etc/ 
        - name: varlog
          mountPath: /var/log
          readOnly: true
        - name: varlogpods
          mountPath: /var/log/pods
          readOnly: true
        - name: varlogcontainers
          mountPath: /var/log/containers
          readOnly: true
        - name: varlibdocker
          mountPath: /var/lib/docker
          readOnly: true
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: fluentd-config
        configMap:
          name: fluentd-config
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlogpods
        hostPath:
          path: /var/log/pods
      - name: varlogcontainers
        hostPath:
          path: /var/log/containers
      - name: varlibdocker
        hostPath:
          path: /var/lib/docker
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers


即使权限正确fluentD版本正确挂载指令在kubernetes daemonset中,也没有线索,为什么我会收到这个警告。

vsmadaxz

vsmadaxz1#

我也遇到过类似的问题。所以,实际上-

  1. Fluentd在/var/log/containers/中创建了一个符号链接,它是/var/log/pods/中文件的符号链接-
root@fluentd-forwarders-5bfzm:/home/fluent# ls -ltr /var/log/containers/consul-0_default_consul-c4dbf47bf46b4cacfb0db67885fdba73835e05b45b14ec7dc746cc2d5ed92ea3.log 
lrwxrwxrwx. 1 root root 83 Oct 30 07:42 /var/log/containers/consul-0_default_consul-c4dbf47bf46b4cacfb0db67885fdba73835e05b45b14ec7dc746cc2d5ed92ea3.log -> /var/log/pods/default_consul-0_2a206546-73b3-4d05-bd7a-0b307c8b24d1/consul/1628.log

字符串
2./var/log/pods是挂载在host上的日志文件的符号链接。在我的设置中,我使用host/node的/data/目录来存储docker数据。

root@fluentd-forwarders-5bfzm:/home/fluent# ls -ltr /var/log/pods/default_consul-0_2a206546-73b3-4d05-bd7a-0b307c8b24d1/consul/1629.log 
lrwxrwxrwx. 1 root root 162 Oct 30 07:47 /var/log/pods/default_consul-0_2a206546-73b3-4d05-bd7a-0b307c8b24d1/consul/1629.log -> /data/docker/containers/478642a56a6e15e7398391a2526ec52ad1aa24341e95aa32063163da11f4cc8b/478642a56a6e15e7398391a2526ec52ad1aa24341e95aa32063163da11f4cc8b-json.log


因此,在我的deployment.yaml中,我必须挂载/data/docker/containers而不是/var/lib/containers/来解决这个问题,即

volumeMounts:
        - mountPath: /var/log
          name: varlog
        - mountPath: /data/docker/containers
          name: datadockercontainers
          readOnly: true
        - mountPath: /fluentd/etc
          name: config-path

mm9b1k5b

mm9b1k5b2#

colachg建议可以帮助您:
我认为kubelet在“/var/log/containers”中创建了一些符号链接(只是链接而不是真实的文件),所以你必须同时挂载链接和真实的文件,或者只挂载真实的文件。

628mspwn

628mspwn3#

当您在列表中定义/var/log时,其他/var/log/...是重复的。
删除/var/log
使用kubectl describe pod fluentd-...检查所有卷是否已正确装载。

dba5bblo

dba5bblo4#

要在spec.containers下添加securityContext字段,可以使用以下YAML代码:

spec:
  containers:
  - name: fluentd
    image: fluent/fluentd:v1.16.3-debian-amd64-1.0
    securityContext:    # this
      runAsUser: 0
    # Rest of the configuration...

字符串
通过使用runAsUser: 0添加securityContext字段,您将fluentd容器的用户ID设置为0,即root用户。这允许fluentd在容器中以root用户身份运行。

f1tvaqid

f1tvaqid5#

我们需要设置以下环境变量:FLUENT_UID为0

bgtovc5b

bgtovc5b6#

/var/log/containers/*.log  unreadable.

字符串
最直接的方法是改变模式:

chmod 777 /var/log/containers/*.log


但最好的方法是:将fluent user更改为root(在docker/kubernetes配置中将FLUENT_UID环境变量设置为0);
将--env FLUENT_UID=0添加到docker命令中,例如:

docker run -it -d   -p 24224:24224   -v /path/to/conf:/fluentd/etc   -v /var:/var --env FLUENT_UID=0 fluent/fluentd:latest


或添加到Kubernetes yaml文件:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
  # namespace: default
  labels:
    k8s-app: fluentd-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      serviceAccount: fluentd
      serviceAccountName: fluentd
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1.4-debian-elasticsearch
        env:
          - name:  FLUENT_ELASTICSEARCH_HOST
            value: "elasticsearch.logging"
          - name:  FLUENT_ELASTICSEARCH_PORT
            value: "9200"
          - name: FLUENT_ELASTICSEARCH_SCHEME
            value: "http"
          - name: FLUENT_UID  # change this place
            value: "0"

bnlyeluc

bnlyeluc7#

你必须设置这个env vars:

- name: FLUENT_CONTAINER_TAIL_EXCLUDE_PATH
  value: '["/var/log/containers/fluentd-*"]'
- name: FLUENT_CONTAINER_TAIL_PARSER_TYPE
  value: "/^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/"

字符串

相关问题