Skip to content
Houssam
EnglishFrançais
← All work

Data platform · Observability

ELK Log Analytics

A log analytics pipeline that turns raw application logs into searchable data, deeper analysis, dashboards and alerts.

logs → index → alert
Illustration

Raw logs

  • 12:04:31INFOapiGET /health 200 4ms
  • 12:04:31WARNworkerretry job=881 attempt=2
  • 12:04:32ERRORapiPOST /checkout 502 184ms
  • 12:04:32INFOauthlogin ok user=•••
  • 12:04:33INFOapiGET /orders 200 21ms
  • 12:04:33ERRORapiPOST /checkout 502 201ms
  • 12:04:34INFOworkerjob=882 done 1.2s
  • 12:04:35WARNdbslow query 840ms
  • 12:04:31INFOapiGET /health 200 4ms
  • 12:04:31WARNworkerretry job=881 attempt=2
  • 12:04:32ERRORapiPOST /checkout 502 184ms
  • 12:04:32INFOauthlogin ok user=•••
  • 12:04:33INFOapiGET /orders 200 21ms
  • 12:04:33ERRORapiPOST /checkout 502 201ms
  • 12:04:34INFOworkerjob=882 done 1.2s
  • 12:04:35WARNdbslow query 840ms

Structured event

{
  "level": "error",
  "service": "api",
  "route": "/checkout",
  "status": 502,
  "latency_ms": 184
}

Alert rule

WHEN
status ≥ 500
RATE
above threshold
THEN
notify on-call

Evaluating

Illustration of raw log lines being parsed into structured documents and evaluated by an alert rule.

The problem

Application logs are high-volume and unstructured. Without a pipeline they are hard to search, analyse or act on.

The approach

  1. 01Logstash ingests and parses application logs.
  2. 02Elasticsearch indexes events for search and aggregation.
  3. 03Spark runs heavier analysis over the data.
  4. 04Kibana dashboards surface KPIs and drive alerts.

Architecture

How ELK Log Analytics fits together.

Each step, in order. Select a node to see what it is responsible for.

  1. Raw log events emitted by running applications.

Stack

  • Logstash
  • Elasticsearch
  • Spark
  • Kibana

Next project

Document RAG Pipeline

A retrieval-augmented generation pipeline that answers questions from a document collection using vector search in PostgreSQL.