EFK/Loki Stack

Why Centralize and Which Stack

Node-local logs die with the node, so clusters ship them to a central store. Two families dominate:

• EFK: Fluent Bit (or Fluentd) as a DaemonSet tails /var/log/containers/*.log, enriches lines with Kubernetes metadata from the API, and indexes them into Elasticsearch/OpenSearch, queried in Kibana.
• Loki: Promtail/Grafana Agent does the same collection, but Loki indexes only labels (namespace, pod, stream) and stores log bodies in object storage — cheaper to run, queried with LogQL that mirrors PromQL.

Both agents run per node; neither reads logs through kubectl logs.

Configuring the Shipper

Fluent Bit tails node files and attaches Kubernetes context:

[INPUT]
    Name              tail
    Path              /var/log/containers/*.log
    Parser            docker
    Mem_Buf_Limit     5MB
[FILTER]
    Name              kubernetes
    Match             kube.*
    Kube_URL          https://kubernetes.default.svc:443
[OUTPUT]
    Name              es
    Match             kube.*
    Host              elasticsearch.logging
    Logstash_Prefix   k8s

Multiline handling matters most: without a multiline parser, every line of a Java stack trace becomes a separate log record. Match continuations by a timestamp or ^s+(at|...) pattern and flush after a timeout so partial traces still emit.

Pitfalls and Interview Angles

Lost lines under load. Tail inputs buffer in memory; without Mem_Buf_Limit and an offset checkpoint file, backpressure drops data or re-reads files after restart.

Index growth. Elasticsearch is disk-heavy: apply ILM/rollover with a retention window, shard by day, and keep mapping templates fixed — dynamic mapping on message fields explodes the cluster. Loki sidesteps this by indexing labels only.

Secrets in logs. Agents forward everything; redact at the source or with a filter before it leaves the node.

Interview angle: contrast indexing strategies (full-text index vs label index), place the agent (DaemonSet, never in-app shippers per pod), and explain how multiline reassembly works.