Step‑by‑Step Guide to Building a Scalable Log Management Pipeline with Open‑Source Tools

You’ve probably felt the pain of a noisy log file that grows faster than your coffee budget. In a world where every microservice spits out its own lines, you need a way to keep things tidy, searchable, and cheap. That’s why today’s post on LogCraft is all about building a log pipeline that can grow with you—using only free tools you can spin up on a weekend.

Why a Log Pipeline Matters Right Now

Logs are the first clue when something breaks. If you can’t find the right line fast, you waste hours hunting down the problem. A good pipeline does three things:

1. Collect – pull logs from many places.
2. Store – keep them safe and searchable.
3. Visualize – let you see patterns and alerts.

Doing this with open‑source software means you stay in control of costs and can tweak each piece to fit your team’s style. Let’s walk through a simple, scalable setup that works for small teams and can be expanded for big ones.

The Building Blocks We’ll Use

All of these are open source, have good docs, and play nicely together. If you already have Docker, you can spin them up in minutes.

Step 1 – Collect Logs with Filebeat

Filebeat is a tiny agent you install on each server. It watches the files you point it at and sends each new line to Kafka.

Install Filebeat

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.10.0-amd64.deb
sudo dpkg -i filebeat-8.10.0-amd64.deb

Basic Config

Create /etc/filebeat/filebeat.yml with just a few lines:

filebeat.inputs:
- type: log
  paths:
    - /var/log/*.log
output.kafka:
  hosts: ["kafka:9092"]
  topic: "logs"

That’s it. Filebeat will now tail any file in /var/log and push each line to the Kafka topic logs. On LogCraft we love keeping the config short—less to break.

Step 2 – Buffer with Kafka

Kafka sits in the middle and smooths out bursts. If one service suddenly spits out a million lines, Kafka will hold them until Elasticsearch can catch up.

Spin Up Kafka with Docker

docker run -d --name kafka \
  -p 9092:9092 \
  -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \
  -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \
  wurstmeister/kafka

You’ll also need a Zookeeper container, but the wurstmeister/kafka image can start one for you if you add -e KAFKA_CREATE_TOPICS="logs:1:1".

Test the Flow

Send a test line:

echo "test log line from LogCraft" | kafka-console-producer.sh --broker-list localhost:9092 --topic logs

Then check the topic:

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic logs --from-beginning

If you see the line, the pipeline is alive.

Step 3 – Store and Index with Elasticsearch

Elasticsearch is the heart of the pipeline. It stores each log entry as a document, making it easy to search by time, service, or error code.

Run Elasticsearch

docker run -d --name es \
  -p 9200:9200 -p 9300:9300 \
  -e "discovery.type=single-node" \
  elasticsearch:8.10.0

Connect Kafka to Elasticsearch

We’ll use Logstash as a bridge. It reads from Kafka, transforms if needed, and writes to Elasticsearch.

Create logstash.conf:

input {
  kafka {
    bootstrap_servers => "kafka:9092"
    topics => ["logs"]
  }
}
output {
  elasticsearch {
    hosts => ["http://es:9200"]
    index => "logcraft-%{+YYYY.MM.dd}"
  }
}

Run Logstash:

docker run -d --name logstash \
  -v $(pwd)/logstash.conf:/usr/share/logstash/pipeline/logstash.conf \
  --link kafka:kafka --link es:es \
  logstash:8.10.0

Now every line that Filebeat sent to Kafka ends up in an Elasticsearch index named like logcraft-2024.06.24. Notice the logcraft prefix? That makes it easy to separate our logs from any other data you might store.

Step 4 – Visualize with Kibana

Kibana gives you a web UI to search and chart logs. It’s the friendly face of the pipeline.

Start Kibana

docker run -d --name kibana \
  -p 5601:5601 \
  --link es:es \
  kibana:8.10.0

Open http://localhost:5601 in your browser. The first time you log in, set up an index pattern: logcraft-*. Now you can type queries like message:"error" and see a timeline of all error lines.

On LogCraft we often create a simple dashboard that shows:

• Log volume per minute (helps spot spikes)
• Top error messages
• Slowest request paths (if you include latency in logs)

All of this can be built with a few clicks—no code required.

Step 5 – Keep an Eye on the Pipeline Itself

A pipeline that’s down is worse than noisy logs. We’ll use Prometheus to scrape metrics from each component and Grafana to draw alerts.

Export Metrics

• Filebeat has a built‑in metrics endpoint at http://localhost:5066/metrics.
• Kafka and Elasticsearch both expose Prometheus‑compatible metrics if you enable the right plugins (the Docker images have them on by default).

Add these targets to prometheus.yml:

scrape_configs:
  - job_name: 'filebeat'
    static_configs:
      - targets: ['filebeat:5066']
  - job_name: 'kafka'
    static_configs:
      - targets: ['kafka:9090']
  - job_name: 'elasticsearch'
    static_configs:
      - targets: ['es:9200']

Run Prometheus:

docker run -d --name prometheus \
  -p 9090:9090 \
  -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus

Grafana Dashboard

docker run -d --name grafana \
  -p 3000:3000 \
  --link prometheus:prometheus \
  grafana/grafana

Add Prometheus as a data source in Grafana, then import a simple dashboard that shows:

• Kafka lag (how many messages are waiting)
• Elasticsearch indexing rate
• Filebeat error count

Set alerts to email you if any metric crosses a threshold. That way LogCraft’s pipeline will tell you before a log storm knocks it offline.

Scaling Tips from LogCraft

1. Add More Kafka Partitions – When traffic grows, increase partitions and add more broker nodes. This spreads the load.
2. Use ILM (Index Lifecycle Management) – Tell Elasticsearch to roll over old indices and delete after 30 days. Keeps storage cheap.
3. Run Filebeat as a DaemonSet in Kubernetes – If you’re in k8s, this makes sure every pod’s logs get collected automatically.
4. Separate Hot and Cold Data – Keep recent logs on fast SSDs, move older ones to cheaper storage. Elasticsearch supports this with “cold” nodes.
5. Monitor Disk Space – Logs can fill disks fast. Grafana alerts on disk usage can save you from a nasty outage.

A Quick Recap (No Fancy Bullets)

• Install Filebeat on each server, point it at your log files, and send to Kafka.
• Run Kafka as a buffer; it smooths spikes.
• Use Logstash to pull from Kafka and push into Elasticsearch, naming indices with a logcraft- prefix.
• Spin up Kibana, set an index pattern, and start searching.
• Add Prometheus and Grafana to watch the health of the whole thing.
• Follow the scaling tips as you grow.

That’s the whole pipeline in a nutshell. It’s built with tools you can run for free, and it scales from a single VM to a full‑blown cluster. On LogCraft we’ve used this exact stack for months, and it’s saved us countless late‑night debugging sessions.

Give it a try, tweak the parts that don’t fit your environment, and enjoy a cleaner, faster way to see what’s happening inside your systems.