Skip to content
← All work

Data platform · End-of-studies

Authentication log analytics

An end-of-studies project: a data pipeline and dashboards that turn raw authentication logs into searchable events and reporting views.

Completed projectPipeline and dashboards completed as an end-of-studies project. Not presented as a lasting production deployment.

logs → index → alert
Illustration

Raw logs

  • 12:04:31INFOauthlogin success user=•••
  • 12:04:31WARNauthlogin failed user=•••
  • 12:04:32ERRORauthtoken rejected
  • 12:04:32INFOauthlogout user=•••
  • 12:04:33INFOauthsession refresh
  • 12:04:33WARNauthlogin failed user=•••
  • 12:04:34INFOauthlogin success user=•••
  • 12:04:35ERRORauthmfa challenge timeout
  • 12:04:31INFOauthlogin success user=•••
  • 12:04:31WARNauthlogin failed user=•••
  • 12:04:32ERRORauthtoken rejected
  • 12:04:32INFOauthlogout user=•••
  • 12:04:33INFOauthsession refresh
  • 12:04:33WARNauthlogin failed user=•••
  • 12:04:34INFOauthlogin success user=•••
  • 12:04:35ERRORauthmfa challenge timeout

Structured event

{
  "level": "warn",
  "service": "auth",
  "event": "login_failed",
  "reason": "invalid_credentials"
}

Alert rule

WHEN
login_failed
RATE
above threshold
THEN
flag in the dashboard

Evaluating

Illustration of raw authentication log lines being parsed into structured documents and reviewed in a dashboard.

The problem

Authentication logs are high-volume and unstructured. Without a pipeline they are hard to search, analyse or review.

Context

An end-of-studies project: ingest authentication events, structure them, search them, then expose dashboards. No lasting production deployment is claimed.

My role

I designed and built the pipeline from ingest to dashboards: Logstash parsing, Elasticsearch indexing, Spark analysis, and Kibana views.

What I built

Built features

  1. 01Ingest and parsing of authentication logs
  2. 02Indexed search and aggregation
  3. 03Batch analysis over the same events
  4. 04Dashboards built for the project

Design decisions

  • Parse and normalise events at ingest so later search and aggregation run on structured records.
  • Index in Elasticsearch for interactive search; reserve Spark for heavier, batch-style analysis.
  • Expose the processed events in Kibana dashboards instead of leaving them in the index only.
  1. 01Logstash ingests and parses authentication logs.
  2. 02Elasticsearch indexes events for search and aggregation.
  3. 03Spark runs heavier analysis over the data.
  4. 04Kibana dashboards surface the processed events.

Outcome

A data pipeline and dashboards were completed as an end-of-studies project. This is not presented as a lasting production deployment.

Architecture

How Authentication log analytics fits together.

Each step, in order. Select a node to see what it is responsible for.

  1. Raw authentication events emitted by the systems under study.

  2. Ingests, parses and normalises log events into structured records.

    • Logstash

  3. Indexes structured events for fast search and aggregation.

    • Elasticsearch

  4. Runs heavier, batch-style analysis over the log data.

    • Spark

  5. Dashboards and exploration on top of the indexed data.

    • Kibana

  6. Reporting views built on top of the processed authentication events.

Stack

  • Logstash
  • Elasticsearch
  • Spark
  • Kibana

Next project

Document question-answering pipeline

A retrieval-augmented generation pipeline that answers questions from a document collection using vector search in PostgreSQL. POC: the work was not pursued to deployment.