Reputation: 337
Facing: fluentd log unreadable. it is excluded and would be examined next time
I have a simple configuration for fluentD daemon set running in kubernetes setup.
Fluentd version: fluentd-0.12.43
Below is my configuration.
<source>
@type tail
path /var/log/containers/sample*.log
time_format %Y-%m-%dT%H:%M:%S.%NZ
tag sample.*
format json
read_from_head true
</source>
<match sample.**>
@type forward
heartbeat_type tcp
send_timeout 60s
recover_wait 10s
hard_timeout 60s
<server>
name worker-node2
host 10.32.0.15
port 24224
weight 60
</server>
</match>
Getting below warning and NO logs are forwarded
2018-08-03 06:36:53 +0000 [warn]: /var/log/containers/samplelog-79bd66868b-t7xn9_logging1_fluentd-70e85c5d6328e7d.log unreadable. It is excluded and would be examined next time.
2018-08-03 06:37:53 +0000 [warn]: /var/log/containers/samplelog-79bd66868b-t7xn9_logging1_fluentd-70e85c5bc89ab24.log unreadable. It is excluded and would be examined next time.
Permission for log file:
[root@k8s-master fluentd-daemonset]# ls -lrt **/var/log/containers/**
**lrwxrwxrwx** Jun 25 06:25 sample-77g68_kube-system_kube-proxy-9f3c3951c32ee.log
-> /var/log/pods/aa1f8d5b-746f-11e8-95c0-005056b9ff3a/sample/7.log
YAML file for daemon set have mount instructions:
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: fluentd
namespace: logging1
labels:
k8s-app: fluentd-logging
version: v1
kubernetes.io/cluster-service: "true"
spec:
template:
-----
-----
-----
volumeMounts:
- name: fluentd-config
mountPath: /fluentd/etc/
- name: varlog
mountPath: /var/log
readOnly: true
- name: varlogpods
mountPath: /var/log/pods
readOnly: true
- name: varlogcontainers
mountPath: /var/log/containers
readOnly: true
- name: varlibdocker
mountPath: /var/lib/docker
readOnly: true
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
terminationGracePeriodSeconds: 30
volumes:
- name: fluentd-config
configMap:
name: fluentd-config
- name: varlog
hostPath:
path: /var/log
- name: varlogpods
hostPath:
path: /var/log/pods
- name: varlogcontainers
hostPath:
path: /var/log/containers
- name: varlibdocker
hostPath:
path: /var/lib/docker
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
Getting no clue even when permission is correct, fluentD version is correct, Mount instruction are their in kubernetes daemonset, why I am getting this warning.
Upvotes: 10
Views: 11335
Reputation: 11
To add a securityContext field under spec.containers, you can use the following YAML code:
spec:
containers:
- name: fluentd
image: fluent/fluentd:v1.16.3-debian-amd64-1.0
securityContext: # this
runAsUser: 0
# Rest of the configuration...
By adding the securityContext
field with runAsUser: 0
, you are setting the user ID for the fluentd container to 0, which is the root user. This allows fluentd to run as the root user within the container.
Upvotes: 1
Reputation: 1
You have to set this env vars:
- name: FLUENT_CONTAINER_TAIL_EXCLUDE_PATH
value: '["/var/log/containers/fluentd-*"]'
- name: FLUENT_CONTAINER_TAIL_PARSER_TYPE
value: "/^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/"
Upvotes: 0
Reputation: 51
/var/log/containers/*.log unreadable.
The most direct way is to change mode:
chmod 777 /var/log/containers/*.log
but the best way is: change fluent user to root (set FLUENT_UID environment variable to 0 in your docker/kubernetes configuration);
add --env FLUENT_UID=0 to docker command, for example:
docker run -it -d -p 24224:24224 -v /path/to/conf:/fluentd/etc -v /var:/var --env FLUENT_UID=0 fluent/fluentd:latest
or add to Kubernetes yaml file:
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: fluentd
namespace: kube-system
# namespace: default
labels:
k8s-app: fluentd-logging
version: v1
kubernetes.io/cluster-service: "true"
spec:
template:
metadata:
labels:
k8s-app: fluentd-logging
version: v1
kubernetes.io/cluster-service: "true"
spec:
serviceAccount: fluentd
serviceAccountName: fluentd
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
containers:
- name: fluentd
image: fluent/fluentd-kubernetes-daemonset:v1.4-debian-elasticsearch
env:
- name: FLUENT_ELASTICSEARCH_HOST
value: "elasticsearch.logging"
- name: FLUENT_ELASTICSEARCH_PORT
value: "9200"
- name: FLUENT_ELASTICSEARCH_SCHEME
value: "http"
- name: FLUENT_UID # change this place
value: "0"
Upvotes: 0
Reputation: 687
I faced a similar issue.
So, what actually happens is -
1. Fluentd creates a symbolic links in /var/log/containers/ which are further a symbolic link of files that are in /var/log/pods/ like -
root@fluentd-forwarders-5bfzm:/home/fluent# ls -ltr /var/log/containers/consul-0_default_consul-c4dbf47bf46b4cacfb0db67885fdba73835e05b45b14ec7dc746cc2d5ed92ea3.log
lrwxrwxrwx. 1 root root 83 Oct 30 07:42 /var/log/containers/consul-0_default_consul-c4dbf47bf46b4cacfb0db67885fdba73835e05b45b14ec7dc746cc2d5ed92ea3.log -> /var/log/pods/default_consul-0_2a206546-73b3-4d05-bd7a-0b307c8b24d1/consul/1628.log
2. /var/log/pods are symbolic links of the log files mounted at host. In my setup I am using /data/ directory of host/node to store docker data.
root@fluentd-forwarders-5bfzm:/home/fluent# ls -ltr /var/log/pods/default_consul-0_2a206546-73b3-4d05-bd7a-0b307c8b24d1/consul/1629.log
lrwxrwxrwx. 1 root root 162 Oct 30 07:47 /var/log/pods/default_consul-0_2a206546-73b3-4d05-bd7a-0b307c8b24d1/consul/1629.log -> /data/docker/containers/478642a56a6e15e7398391a2526ec52ad1aa24341e95aa32063163da11f4cc8b/478642a56a6e15e7398391a2526ec52ad1aa24341e95aa32063163da11f4cc8b-json.log
So, in my deployment.yaml I had to mount /data/docker/containers rather /var/lib/containers/ to solve the issue i.e
volumeMounts:
- mountPath: /var/log
name: varlog
- mountPath: /data/docker/containers
name: datadockercontainers
readOnly: true
- mountPath: /fluentd/etc
name: config-path
Upvotes: 11
Reputation: 337
We need to set the below environment variable: FLUENT_UID to 0
Upvotes: 0
Reputation: 2270
May colachg suggestion help you:
I think that kubelet create some symbolic links in '/var/log/containers'(just links not real file), so you must mount both links and real files or only mount real file with right fluentd.conf.
Upvotes: 3
Reputation: 11377
As you defined /var/log
in the list, the others /var/log/...
are duplicated.
Remove /var/log
Check with kubectl describe pod fluentd-...
whether all volumes were mounted properly.
Upvotes: 1