;
;

DZone Big Data Zone

Recent posts in Big Data on DZone.com

Delta Lake UniForm to Snowflake: Cross-Platform Table Access Without a Second Copy

Making a Databricks-owned Delta table queryable from Snowflake without copying it is one of those things where every individual piece — Delta Lake ...
Posted on 22 October 2026 | 6:00 pm

Rebuilding Elasticsearch Under Load: Unsafe Watermarks and a Sampling Blind Spot

Rebuilding an Elasticsearch index from PostgreSQL sounds simple: Create a new index. Copy the rows that belong in search. Replay anything that ...
Posted on 22 October 2026 | 5:00 pm

Idempotency Under Concurrency: Why Your Application Check Isn't the Last Line of Defense

Every Kafka tutorial covers idempotency the same way: "Store the event ID. Before processing, check if it exists. If it does, skip. Done."
Posted on 22 October 2026 | 4:00 pm

How to Migrate From C/C++ to Rust: A Practical Guide for Embedded IoT Systems

Embedded and IoT devices operate in environments where every kilobyte of memory and every millisecond of response time matters. Yet most of them st...
Posted on 22 October 2026 | 1:00 pm

How Kestra Unifies Enterprise Workflow Orchestration and Agentic AI

Kestra is an open-source workflow orchestration platform that unifies IT scheduling, data pipelines, business processes, infrastructure automation,...
Posted on 21 October 2026 | 2:30 pm

You Don't Have to Pick One: A Hybrid Approach to Enterprise Data Refresh

If you are a data engineer, you have probably come across this situation where you have to decide between two approaches for refreshing enterprise ...
Posted on 15 October 2026 | 2:00 pm

Consumer Lag Is the Wrong Finish Line

A Kafka recovery often looks successful before it actually is. The replay job exits. Consumer lag drops. Error rate returns to normal. The dashboar...
Posted on 13 October 2026 | 2:00 pm

Kafka With Spring Boot: Introduction

Kafka is a distributed event streaming platform that enables real-time data ingestion, storage, and processing of continuous event streams. To do t...
Posted on 12 October 2026 | 4:00 pm

From Legacy ETL to Modern ELT: Engineering a Governed dbt and Databricks Architecture

A few years back, I sat through a production incident review where the root cause turned out to be a single stored procedure nobody on the team had...
Posted on 12 October 2026 | 12:00 pm

OpenAI Watermarks ChatGPT and Codex: What Changes for EU Users

ChatGPT responses in Europe may soon carry a signal users cannot see. OpenAI is rolling out invisible text watermarks for eligible ChatGPT and Code...
Posted on 7 October 2026 | 8:40 pm

Building IoT Time-Series Applications With Java and Apache IoTDB

Apache IoTDB is well-suited for environments where time-series data from connected devices and industrial systems is generated continuously and req...
Posted on 6 October 2026 | 4:00 pm

Beyond @Transactional: Solving the Dual-Write Problem in Distributed Microservices

The Illusion of a Single Database In a traditional monolithic application, maintaining data consistency is straightforward. If you need to create a...
Posted on 6 October 2026 | 3:00 pm

Why Databricks and Snowflake Speak the Kafka Protocol: Ingestion vs Architecture

Kafka-compatible ingestion is showing up everywhere now, including in analytics platforms that have nothing to do with running Kafka. Databricks ad...
Posted on 30 September 2026 | 3:00 pm

OpenAI ‘o’ Leak: What We Know About ChatGPT’s Always-On Assistant Before DevDay

References to an unannounced OpenAI assistant called “o” have surfaced in ChatGPT, suggesting the company is testing an always-on assistant that ma...
Posted on 29 September 2026 | 9:31 pm

Architecting for <1s Latency: Managing Eventual Consistency in Distributed Search Platforms

At a logistics visibility company, the shipment state can change thousands of times per second due to the massive scale of operations. Such changes...
Posted on 22 September 2026 | 3:00 pm

When an iOS Retry Executes an Agent Twice: Building Effectively-Once Tool Workflows With LangGraph, MCP Tasks, Kafka, and App Attest

A mobile request can fail without the server-side work failing. An iOS app may time out, lose the response after a POST has reached the service, or...
Posted on 21 September 2026 | 5:00 pm

MCP vs REST/HTTP API vs Kafka: The Architect's Guide to Agentic AI Integration

Every major AI vendor now supports the Model Context Protocol. The framing is almost always the same: MCP is the universal connector for AI agents ...
Posted on 18 September 2026 | 7:00 pm

Event-Driven AI Systems With Kafka and Autonomous Agents

Enterprise AI is moving beyond isolated prompt-response calls and toward systems that observe events, preserve state, invoke tools, and publish dec...
Posted on 16 September 2026 | 4:00 pm

Data Governance for the Agentic Era

The modern enterprise generates and consumes unprecedented volumes of data across operational systems, customer interactions, partner ecosystems, c...
Posted on 14 September 2026 | 5:00 pm

Improving Repeated Analytics Workloads With Databricks Disk Cache

In many analytics platforms, there are performance issues that do not always come from complex transformations. Sometimes the bottleneck is much si...
Posted on 11 September 2026 | 1:00 pm