;
;

DZone Big Data Zone

Recent posts in Big Data on DZone.com

You Don't Have to Pick One: A Hybrid Approach to Enterprise Data Refresh

If you are a data engineer, you have probably come across this situation where you have to decide between two approaches for refreshing enterprise ...
Posted on 15 October 2026 | 2:00 pm

Consumer Lag Is the Wrong Finish Line

A Kafka recovery often looks successful before it actually is. The replay job exits. Consumer lag drops. Error rate returns to normal. The dashboar...
Posted on 13 October 2026 | 2:00 pm

Kafka With Spring Boot: Introduction

Kafka is a distributed event streaming platform that enables real-time data ingestion, storage, and processing of continuous event streams. To do t...
Posted on 12 October 2026 | 4:00 pm

From Legacy ETL to Modern ELT: Engineering a Governed dbt and Databricks Architecture

A few years back, I sat through a production incident review where the root cause turned out to be a single stored procedure nobody on the team had...
Posted on 12 October 2026 | 12:00 pm

OpenAI Watermarks ChatGPT and Codex: What Changes for EU Users

ChatGPT responses in Europe may soon carry a signal users cannot see. OpenAI is rolling out invisible text watermarks for eligible ChatGPT and Code...
Posted on 7 October 2026 | 8:40 pm

Building IoT Time-Series Applications With Java and Apache IoTDB

Apache IoTDB is well-suited for environments where time-series data from connected devices and industrial systems is generated continuously and req...
Posted on 6 October 2026 | 4:00 pm

Beyond @Transactional: Solving the Dual-Write Problem in Distributed Microservices

The Illusion of a Single Database In a traditional monolithic application, maintaining data consistency is straightforward. If you need to create a...
Posted on 6 October 2026 | 3:00 pm

Why Databricks and Snowflake Speak the Kafka Protocol: Ingestion vs Architecture

Kafka-compatible ingestion is showing up everywhere now, including in analytics platforms that have nothing to do with running Kafka. Databricks ad...
Posted on 30 September 2026 | 3:00 pm

OpenAI ‘o’ Leak: What We Know About ChatGPT’s Always-On Assistant Before DevDay

References to an unannounced OpenAI assistant called “o” have surfaced in ChatGPT, suggesting the company is testing an always-on assistant that ma...
Posted on 29 September 2026 | 9:31 pm

Architecting for <1s Latency: Managing Eventual Consistency in Distributed Search Platforms

At a logistics visibility company, the shipment state can change thousands of times per second due to the massive scale of operations. Such changes...
Posted on 22 September 2026 | 3:00 pm

When an iOS Retry Executes an Agent Twice: Building Effectively-Once Tool Workflows With LangGraph, MCP Tasks, Kafka, and App Attest

A mobile request can fail without the server-side work failing. An iOS app may time out, lose the response after a POST has reached the service, or...
Posted on 21 September 2026 | 5:00 pm

MCP vs REST/HTTP API vs Kafka: The Architect's Guide to Agentic AI Integration

Every major AI vendor now supports the Model Context Protocol. The framing is almost always the same: MCP is the universal connector for AI agents ...
Posted on 18 September 2026 | 7:00 pm

Event-Driven AI Systems With Kafka and Autonomous Agents

Enterprise AI is moving beyond isolated prompt-response calls and toward systems that observe events, preserve state, invoke tools, and publish dec...
Posted on 16 September 2026 | 4:00 pm

Data Governance for the Agentic Era

The modern enterprise generates and consumes unprecedented volumes of data across operational systems, customer interactions, partner ecosystems, c...
Posted on 14 September 2026 | 5:00 pm

Improving Repeated Analytics Workloads With Databricks Disk Cache

In many analytics platforms, there are performance issues that do not always come from complex transformations. Sometimes the bottleneck is much si...
Posted on 11 September 2026 | 1:00 pm

Stream Processing on the Mainframe With Apache Flink: Genius or a Glitch in the Matrix?

Running Apache Flink on a mainframe sounds odd at first. A modern stream processing engine on a platform most people call legacy? But take a closer...
Posted on 10 September 2026 | 7:00 pm

Dashboards and Queries for Apache Kafka

Dashboards are everywhere. Business and IT teams use them to track metrics, visualize trends, and make decisions. But when working with real-time d...
Posted on 10 September 2026 | 4:00 pm

From ETL, ELT, and EtLT to Agent: What Is Changing in Enterprise Data Engineering?

For the past two decades, most enterprise data engineering systems have been built on one default assumption: People understand the system. The sys...
Posted on 8 September 2026 | 6:00 pm

Handling Large API Responses Without Freezing the Client: A Practical Architecture With Temporal, Kafka, and RAG

A large API response becomes a client problem long before it becomes a network problem. A browser can receive hundreds of megabytes and still becom...
Posted on 4 September 2026 | 5:00 pm

Your Spark Job Isn't Slow Because of Bad Code. It's Slow Because of the Wrong Join

I learned this lesson the hard way. We had a critical data pipeline running for over 3 hours every single day. The logic was perfectly clean. The o...
Posted on 3 September 2026 | 3:00 pm