Centralized Logging for Kubernetes: Fluentd + Elasticsearch

Disclosure: We are reader supported, and earn affiliate commissions when you buy through us.

Tired of hopping between pods with kubectl logs to find that elusive error? Centralized logging for Kubernetes puts every container log in one searchable view, so you can troubleshoot fast.

I used to waste minutes—sometimes hours—chasing fragments across pods, only to miss the critical timeout that bubbled up from the database layer. That frustration showed me I needed a single timeline of all logs, not a scattered puzzle.

The fix was straightforward: a Fluentd DaemonSet collects logs from each node, ships them to an Elasticsearch cluster, and Kibana provides a unified UI. Below is the exact, copy‑paste‑ready process I followed, preserving every technical step.

Why Centralized Logging for Kubernetes Is Essential

When logs stay trapped inside individual pods, correlating events across services feels like guesswork. By aggregating logs, you gain:

• Instant visibility into errors that span multiple containers.
• Fast searches across namespaces, pods, and time ranges without juggling terminals.
• Easy sharing of Kibana links with teammates for collaborative debugging.

These benefits turn a frustrating log‑hunt into a quick lookup, letting you return to building rather than hunting.

Step‑by‑Step: Deploy Fluentd DaemonSet

First, add a Fluentd DaemonSet so a pod runs on every node, reading container logs from /var/log/containers. The manifest mounts the node’s log directory and tags each line with its namespace and pod name.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  labels:
    app: fluentd
spec:
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1.14-debian-elasticsearch-1.1
        env:
        - name: FLUENT_ELASTICSEARCH_HOST
          value: "elasticsearch"
        - name: FLUENT_ELASTICSEARCH_PORT
          value: "9200"
        - name: FLUENT_ELASTICSEARCH_SCHEME
          value: "http"
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

Apply it with kubectl apply -f fluentd-daemonset.yaml. Fluentd will now tail logs and forward them to the Elasticsearch service you’ll create next.

Deploy Elasticsearch with Helm

Next, spin up a minimal Elasticsearch stack using the official Helm chart. This setup provides enough storage for a few days of logs and a single node—perfect for testing.

helm repo add elastic https://helm.elastic.co
helm install elasticsearch elastic/elasticsearch \
  --set replicas=1 \
  --set minimumMasterNodes=1 \
  --set esJavaOpts="-Xmx512m -Xms512m"

Once the chart finishes, expose a Kubernetes service named elasticsearch so Fluentd can reach it by that hostname. Verify the service is ready with kubectl get svc elasticsearch.

Visualize Logs in Kibana

With data flowing into Elasticsearch, launch Kibana via Helm and point it at the Elasticsearch service.

helm install kibana elastic/kibana \
  --set elasticsearch.hosts="http://elasticsearch:9200"

After Kibana pods are running, access the UI (typically via kubectl port-forward svc/kibana 5601:5601). Create an index pattern fluentd-* and you’ll instantly see a unified log view. Search by namespace, pod, or time range, and drill down to the exact error trace—all from one search bar.

Wrap Up & Next Steps

Having all logs in one place transforms a frustrating guessing game into a rapid lookup. You can spot issues faster, share a Kibana link with teammates, and get back to building instead of log‑hunting.

If you found this guide useful, consider applying the same Fluentd‑Elasticsearch pattern to other clusters or exploring additional outputs like Loki or Splunk. The core idea remains: centralize, search, and act.