Processing & streaming

Amazon Kinesis is AWS's managed streaming family: Data Streams (a sharded log), Firehose (managed delivery), and Managed Service for Apache Flink.

Kafka is a distributed, append-only log you can publish events to and replay later. Not a queue and not a database — a durable commit log that decouples the systems producing data from the ones consuming it. How the log, partitions, and consumer groups fit together.

Apache Beam is a unified model for batch and streaming: write one pipeline and run it on a pluggable runner like Dataflow, Flink, or Spark.

Apache Flink is a distributed engine for stateful stream processing: true record-at-a-time streaming, event-time with watermarks, exactly-once state.

Spark is a distributed compute engine that runs one logical query across a cluster of machines. It builds a lazy plan, splits it into parallel tasks, and shuffles data between stages. How the driver, executors, and DAG actually fit together.

Apache Pulsar is a distributed messaging and streaming platform that splits serving (brokers) from storage (BookKeeper), with multi-tenancy built in.

Hadoop launched the big-data era with HDFS, MapReduce, and YARN. Foundational, but largely superseded by Spark, S3, and cloud warehouses.

RabbitMQ is a smart message broker that routes messages through exchanges to queues and deletes them once consumed — unlike Kafka's replayable log.

Redpanda is a Kafka-API-compatible streaming platform in C++ with no JVM and no ZooKeeper, built for lower latency and simpler operations.