What is Amazon Kinesis?

Amazon Kinesis is AWS's family of managed services for moving and processing streaming data without running your own cluster. It is not one product but three that get used together: Kinesis Data Streams, a Kafka¹-like sharded append-only log; Kinesis Data Firehose, a managed pipe that buffers records and delivers them to destinations like S3, Redshift², or OpenSearch; and Amazon Managed Service for Apache Flink³ (formerly Kinesis Data Analytics), which runs Apache Flink jobs for stateful stream processing⁴. Most of the interesting design decisions live in Data Streams, so that is where to start.¹

🔗 Learn more — ¹ What is Apache Kafka?

🔗 Learn more — ² What is Amazon Redshift?

🔗 Learn more — ³ What is Apache Flink?

🔗 Learn more — ⁴ Batch vs stream processing

🔗 Learn more — ¹ Amazon Kinesis Data Streams documentation

Shards, records, and ordering

A Kinesis data stream is divided into shards, and the shard is the unit of everything that matters: throughput, ordering, and cost. You write records to the stream, and every record carries a partition key — an arbitrary string you choose. Kinesis hashes that key to decide which shard the record lands in, exactly the way a hash decides a Kafka partition. Records sharing a partition key go to the same shard and are read back in the order they were written; ordering is guaranteed within a shard, never across the whole stream. Pick a partition key that matches your ordering needs (a user ID, a device ID) rather than something random, or you lose the per-entity ordering you probably wanted.

Each shard gives you a fixed slice of capacity for ingest and for read fan-out, so a stream's total throughput is just shard count times per-shard capacity. That makes scaling explicit: you grow a stream by splitting shards and shrink it by merging them. There is no surprise here, but there is also no autopilot in the basic mode — shard count is a knob you turn.

Data Streams vs Firehose

The two ingest services solve different problems, and confusing them is the most common Kinesis mistake.

Data Streams is a durable log you read from. Records are retained for a configurable window (extendable well beyond the default), and multiple independent consumers can each read the whole stream at their own pace and re-read history within the retention period. You own the consumer code, the checkpointing, and the scaling. Use it when you need stream processing, replay, or several downstream consumers of the same events — it is the building block other systems sit on top of.

Firehose is fire-and-forget delivery. You point producers at a Firehose stream and it buffers records by size or time and then writes them to a destination, optionally converting format (to Parquet⁵, say) or invoking a Lambda to transform en route. There is no consumer to write and no shards to manage; throughput scales automatically. The tradeoff is that Firehose is a one-way pipe to a sink, not a re-readable log — you cannot replay or attach a second arbitrary consumer. Reach for Firehose when the job is simply "land these events in S3/Redshift/OpenSearch reliably," and for Data Streams when events need to be processed or fanned out.

🔗 Learn more — ⁵ How Parquet works: columnar storage explained

flowchart TD
    PROD["Producers"] --> DS["Data Streams: sharded, re-readable log"]
    PROD --> FH["Firehose: buffered one-way delivery"]
    DS --> CONS["Your consumers / Flink — replay, fan-out"]
    FH --> SINK["S3 / Redshift / OpenSearch — no replay"]

    %% color = role: green re-readable log, amber delivery-only, grey shared
    classDef log stroke:#a3be8c,stroke-width:2.5px
    classDef sink stroke:#ebcb8b,stroke-width:2.5px
    classDef plain stroke:#7b88a1,stroke-width:2.5px
    class DS,CONS log
    class FH,SINK sink
    class PROD plain

The honest tradeoff

The pitch for Kinesis is real: if you are already on AWS, it is genuinely low-ops. There is no broker fleet to patch, no ZooKeeper or KRaft quorum to babysit, IAM handles auth, and it wires into the rest of AWS with almost no glue. For a team that wants streaming without running a streaming platform, that is a lot of operational pain avoided.

The catches are equally real. Shard-based capacity means scaling is a manual provisioning decision in the basic mode — over-provision and you pay for idle shards, under-provision and producers get throttled. There are fixed per-shard throughput and retention limits you have to design around, and pricing is largely per-shard-hour plus data volume, so a high-shard-count stream is a standing bill whether or not it is busy. And every record you write is AWS lock-in: the API, the partitioning, and the consumer libraries are proprietary, so moving off later means rewriting producers and consumers.

That is the trade against open alternatives. Apache Kafka and Redpanda⁶ give you the same sharded-log model without the lock-in and, run well, often at lower cost at scale — but you (or a vendor) carry the operational weight Kinesis hides. The Flink side of Kinesis is the least proprietary piece, since Apache Flink jobs are portable across runtimes. If your data pipeline⁷ lives entirely inside AWS and you value not running infrastructure over portability, Kinesis is a reasonable default; if you expect multi-cloud, cost-sensitive scale, or want to avoid betting your stream layer on one vendor, weigh it carefully before committing.

🔗 Learn more — ⁶ What is Redpanda?

🔗 Learn more — ⁷ What is a data pipeline?