Skip to project
pedromartins.tech
Delivered
Data Engineering2023

Topology and operational details are generalized to preserve enterprise boundaries.

JBS Data Pipeline

Designed a streaming path that decouples production events from analytics and preserves processing continuity during load peaks.

Role
Data pipeline engineer
Environment
Sanitized data-engineering case study
Ownership
Ingestion design · Kafka consumer behavior · Transformation · Durable writes · Operational visibility
JBS Data Pipeline project visual
jbs-stream-console5 system boundaries
Production Systems
Kafka
Python Consumer
Data Lake
Kafkadecoupled ingestion backbone
Manualoffset commit after durable write
TLSauthenticated and encrypted transport pattern
Lakeanalytics destination
01 / Interactive system

Move through the system layer by layer

jbs-stream-consoleSanitized data-engineering case study · Read-only
Delivered
System map

Boundaries and responsibilities

Public or untrustedAuthenticated boundaryInternal-only dependency
READ-ONLYSanitized data-engineering case studytool: systemselection: production

Production Systems selected.

02 / Context

The problem behind the system

Production metrics arrive unevenly, but executive and operational analysis still depends on a complete, ordered-enough history.

The pipeline separates event production from downstream processing so a load peak or consumer interruption does not automatically become data loss.

Read the full project overview

A high-throughput data pipeline architecture designed to ingest and process production metrics in real-time.

Built on Apache Kafka, the system decouples data production from analysis, preventing data loss during high load peaks. It feeds a Data Lake that drives executive dashboards, providing granular monitoring of operational efficiency.

03 / Operations

The unhappy path is part of the design

01

What if the consumer stops?

Kafka retains the backlog while producers continue publishing.

Signal
Consumer lag grows by partition.
Response
Restore processing, watch catch-up rate, and investigate poison events.
02

What if a write succeeds before a crash?

The uncommitted event is delivered again.

Signal
The same event ID appears in a replay attempt.
Response
Idempotent upsert preserves one analytical record, then commit.
04 / Decisions

The trade-offs that shaped the build

Decision

Commit after durable write

Prefer replay over silent loss.

Reason
The offset represents completed work, not merely received work.
Trade-off
Consumers must make reprocessing safe.
Revisit when
Only with a transactional end-to-end processing model.
Decision

Decouple analytics

Keep production systems independent from dashboard latency.

Reason
Operational event generation should not block on downstream analysis.
Trade-off
Dashboards become eventually consistent.
Revisit when
When a use case explicitly requires synchronous feedback.
Decision

Idempotent lake writes

Give replay an exactly-once effect.

Reason
At-least-once delivery is practical only when repeated work is harmless.
Trade-off
Event identity quality becomes a core contract.
Revisit when
As schemas and partition strategy evolve.
05 / Security

Threats, controls, and what remains

No control is presented as total risk elimination.

Integrity

Replay double-counts a metric

Control
Idempotent upsert keyed by event identity
Residual risk
A bad or reused producer ID can still merge distinct events.
Loss

Offset commits before durable write

Control
Manual synchronous commit after the lake write
Residual risk
A permanently poison event needs a governed dead-letter path.
Transport

Stream credentials or data intercepted

Control
Authenticated encrypted Kafka transport pattern
Residual risk
Broker and client certificate operations remain critical.
06 / Implementation

How the decisions appear in the build

pipeline/consumer.py
01# Decouple production from analysis; survive load peaks without losing events.02from confluent_kafka import Consumer03 04consumer = Consumer({05    "bootstrap.servers": BROKERS,06    "group.id": "metrics-ingest",07    "enable.auto.commit": False,             # commit only AFTER a durable write08    "security.protocol": "SASL_SSL",         # encrypted + authenticated transport09    "sasl.mechanism": "SCRAM-SHA-512",10    "ssl.ca.location": "/etc/kafka/ca.pem",11})12consumer.subscribe(["production.metrics"])13 14while True:15    msg = consumer.poll(1.0)16    if msg is None:17        continue18    if msg.error():19        log.error("kafka.error", err=str(msg.error()))20        continue21 22    event = decode(msg.value())23    # Idempotent upsert keyed on event id - reprocessing never double-counts.24    lake.upsert(key=event.id, payload=event, partition=msg.partition())25    consumer.commit(msg, asynchronous=False)  # at-least-once, exactly-once effect
Representative / sanitized excerpt

Encrypted + authenticated transport, manual commits, idempotent upserts.

PythonApache KafkaPandasSQL
07 / Evidence

Proof, source, and inspectable outcomes

08 / Results

What the project demonstrates

Production event generation is decoupled from analytics latency

Replay can recover work without double-counting the lake record

Durable history supports granular operational views

What worked
  • Kafka isolates burst handling from analysis.
  • Commit order gives failure behavior a clear rule.
Next iteration
  • Schema registry governance
  • Dead-letter workflow
  • Lag-based capacity forecasting
Professional signal
  • Streaming architecture
  • Data integrity reasoning
  • Operational pipeline design