Logging
Its important to have and to analyze logs for both the nodes and the applications.
Pre-Req's
- This assumes you have a log server already setup, for example Graylog* That you know how to setup inputs on on your log server of choice.
Node logs
On each node, open the rsyslog config file -
sudo nano /etc/rsyslog.conf
Then add to the bottom
# Log to Graylog
*.* @@logserver:1516
Restart rsyslog -
systemctl restart rsyslog
You should now see you logs start to come into your logging solution.
Kubernetes Logs
Next we need to get the logs from Kubernetes itself.
For this, we will be using an app called Fluent-bit, which is a lightweight derivative of FluentD.
As a side note, it also has a VERY helpful community in its Slack channel.
For this we will be creating 3x config YAML files.
- Role, Role Binding & Service Account
- Config Map
- Daemon Set.
fluent-bit_roles.yaml
No edits needed on this one, but note the aforementioned ---
usage to combine several configs into one file.
We can technically combine all of these YAMLs into one, however for the purpose of this, we will be separating so its easier to breakdown the functions of each.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: fluent-bit-read
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: fluent-bit-read
subjects:
- kind: ServiceAccount
name: fluent-bit
namespace: logging-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: fluent-bit-read
rules:
- apiGroups: [""]
resources:
- namespaces
- pods
verbs: ["get", "list", "watch"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: fluent-bit
namespace: logging-system
fluent-bit_configmap.yaml
This is what gives Fluent-bit its configuration.
In this section, you will need to edit the variables for host and port, under the output-graylog.conf
:` section, updating it to the IP of your log server, and the port setup for the GELF connection.
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-config
namespace: logging-system
labels:
k8s-app: fluent-bit
data:
# Configuration files: server, input, filters and output
# ======================================================
fluent-bit.conf: |
[SERVICE]
Flush 1
Log_Level info
Daemon off
Parsers_File parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
@INCLUDE input-kubernetes.conf
@INCLUDE filter-kubernetes.conf
@INCLUDE output-graylog.conf
input-kubernetes.conf: |
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
Parser docker
DB /var/log/flb_graylog.db
DB.Sync Normal
Docker_Mode On
Buffer_Chunk_Size 512KB
Buffer_Max_Size 5M
Rotate_Wait 30
Mem_Buf_Limit 30MB
Skip_Long_Lines On
Refresh_Interval 10
filter-kubernetes.conf: |
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Merge_Log_Key log
Keep_Log Off
K8S-Logging.Parser On
K8S-Logging.Exclude Off
Annotations Off
Labels On
output-graylog.conf: |
[OUTPUT]
Name gelf
Match *
Host IP_OF_YOUR_SERVER
Port 12204
Mode tcp
Gelf_Short_Message_Key log
parsers.conf: |
[PARSER]
Name apache
Format regex
Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name apache2
Format regex
Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name apache_error
Format regex
Regex ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$
[PARSER]
Name nginx
Format regex
Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name json
Format json
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
[PARSER]
Name syslog
Format regex
Regex ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
Time_Key time
Time_Format %b %d %H:%M:%S
Make sure you updated IP_OF_YOUR_SERVER
with...the IP of your log server.
Right now, it doesn't appear to play very nicely, or consistently, with DNS names.
I am not sure if this is a syntax issue, or a limitation/bug of the graylog parser. Once i get some time to investigate, i will update this article accordingly.
fluent-bit_daemonset.yaml
This is the "Application" itself, ran as a Daemon Set. No changes are needed here.
The image version in this is set to 1.9 - deliberately.
The below config does not work on versions greater than 1.9.x, it will be updated at a later date to allow 2.0+ to be used.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluent-bit
namespace: logging-system
labels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
spec:
selector:
matchLabels:
k8s-app: fluent-bit-logging
template:
metadata:
labels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "2020"
prometheus.io/path: /api/v1/metrics/prometheus
spec:
containers:
- name: fluent-bit
image: fluent/fluent-bit:1.9
imagePullPolicy: Always
ports:
- containerPort: 2020
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: fluent-bit-config
mountPath: /fluent-bit/etc/
terminationGracePeriodSeconds: 10
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: fluent-bit-config
configMap:
name: fluent-bit-config
serviceAccountName: fluent-bit
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
- operator: "Exists"
effect: "NoExecute"
- operator: "Exists"
effect: "NoSchedule"
Apply all three YAML files -
kubectl apply -f fluent-bit_roles.yaml -f fluent-bit_configmap.yaml -f fluent-bit_daemonset.yaml
After a minute or two, you should see the logs start to stream into Graylog.
Application Logs
Coming soon!
Next Step
Next, go to the next step, Dashboard - Traefik.
Or,
Go back to the index page.