Amazon Kinesis Overview

• Purpose: Collect, process, and analyze real-time, streaming data for timely insights.
• A collection of services for processing streams of various data types.
• Data processed in "shards," each ingesting 1000 records/second.
• Default limit of 500 shards (can be increased).
• A record consists of a partition key, sequence number, and data blob (up to 1 MB).
• Transient Data Store: Default retention of 24 hours, configurable up to 7 days.
• Four main services: Video Streams, Data Streams, Data Firehose, Data Analytics.

Kinesis Video Streams

• Purpose: Securely stream video from connected devices to AWS for analytics, ML, and other processing.
• Durably stores, encrypts, and indexes video data streams.
• Producers provide data streams; Consumers receive and process.
• Stores data for 24 hours (default), up to 7 days.
• Supports encryption at rest with server-side encryption (KMS).
• Exam Relevance: Does not appear much on AWS exams.

Kinesis Data Streams

Basics & Use Cases

• Purpose: Build custom applications to process or analyze streaming data for specialized needs.
• Enables real-time processing of streaming big data.
• Useful for rapidly moving data off data producers and continuously processing it.
• Key Difference: Stores data for later processing by applications (unlike Firehose, which delivers directly).
• Common Use Cases: Accelerated log/data feed intake, real-time metrics/reporting, real-time data analytics, complex stream processing.

Producers

• Continually push data to Kinesis Data Streams.
• Methods: Kinesis Streams API, Kinesis Producer Library (KPL), Kinesis Agent.

Records

• Unit of data stored in a Kinesis Data Stream.
• Composed of: sequence number, partition key, and data blob.
• Data Blob: The actual data payload (max 1 MB before Base64-encoding).
• Retention: Default 24 hours, can be extended to 7 days.

Shards

• Base throughput unit of a Kinesis Data Stream.
• Capacity per Shard: 1 MB/sec data input, 2 MB/sec data output.
• Supports up to 1000 PUT records/second.
• A stream is composed of one or more shards; total capacity is sum of shard capacities.

Resharding (Split/Merge)

• Adjusts the number of shards to adapt to changes in data flow rate.
• Shard Split: Divides a single shard into two. Increases shard count, data capacity, and cost.
• Shard Merge: Combines two shards into one. Reduces shard count, data capacity, and cost.

Consumers

• EC2 instances (or other compute) that analyze data received from a stream.
• Known as Amazon Kinesis Streams Applications.
• Scale consumers by adding/removing shards when data rate increases/decreases.
• Partition Keys: Used to group data by shard within a stream. Ensures all records with the same partition key go to the same shard. (Exam Tip!)

Security & Durability

• Replicates synchronously across three AZs for high durability.
• Supports encryption at rest using KMS master keys. Producers/consumers need KMS key access.

Kinesis Data Firehose

Basics & Features

• Definition: Easiest way to load streaming data into data stores and analytics tools.
• Captures, transforms, and loads streaming data.
• Enables near real-time analytics with existing BI tools.
• Can use Kinesis Data Streams as a source.
• Key Difference: Delivers data directly to AWS services (no need to write an application or manage resources).
• Can batch, compress, and encrypt data before loading.
• Synchronously replicates data across three AZs during transport.
• Stores data records for up to 24 hours.
• No shards: Totally automated and managed service. (Exam Tip!)

Components (Source, Stream, Record, Destination)

• Source: Where streaming data is generated/captured.
• Delivery Stream: Underlying entity of Firehose.
• Record: Data sent to a delivery stream (max 1000 KB before Base64-encoding).
• Destination: Data store where data is delivered.

Destinations

• Amazon S3 (common for raw data backup).
• Amazon Redshift (data loaded via S3 COPY command).
• Amazon Elasticsearch Service.
• Splunk.

Data Transformation

• Can invoke an AWS Lambda function to transform incoming data before delivery.
• Optional backup of source data to another S3 bucket if transformation is enabled.

Security

• Can encrypt data with an existing KMS key.
• Server-side encryption can be used if Kinesis Data Streams is the source.

Kinesis Data Analytics

Basics & Use Cases

• Definition: Easiest way to process and analyze real-time, streaming data.
• Can use standard SQL queries to process Kinesis data streams.
• Provides real-time analysis.
• Use Cases: Generate time-series analytics, feed real-time dashboards, create real-time alerts/notifications.
• Sits over Kinesis Data Streams and Kinesis Data Firehose.

Application Components

• Input: Streaming source.
• Application Code: SQL statements for processing.
• Output: In-application streams for intermediate results.

Input & Output

• Input Types: Streaming data sources (continuous) and Reference data sources (static data for enrichment).
• Destinations: Persist results to S3, Redshift, Elasticsearch, and Kinesis Data Streams.
• IAM provides permissions to read from sources and write to destinations.

Kinesis Client Library (KCL)

• Definition: A Java library (also available for other languages) that helps read records from a Kinesis Data Stream with distributed applications.
• Helps consumers process data in a distributed, fault-tolerant manner.
• Abstraction Layer: Provides an abstraction specifically for processing data in a consumer role, different from the raw Kinesis Data Streams API.
• How it works:
  • Instantiates a worker, connects to stream, enumerates shards.
  • Coordinates shard associations with other workers.
  • Instantiates a record processor for every shard it manages.
  • Pulls records, pushes to processor, checkpoints processed records to DynamoDB. (Exam Tip!)
  • Balances shard-worker associations during scaling (split/merge).
• Scaling Consumers:
  • Ensures exactly one KCL worker/record processor per shard.
  • Number of instances should generally not exceed number of shards (except for standby).
  • One worker can process multiple shards.
• Can run on EC2, Elastic Beanstalk, on-premises servers.
• Records are read in order at the shard level.

Kinesis Security (General)

• Authorization: Control access using IAM policies.
• Encryption in Flight: HTTPS endpoints.
• Encryption at Rest: Server-Side Encryption (SSE) using KMS.
• Possible to encrypt/decrypt data on the client side.
• Network: VPC endpoints available for Kinesis to access within a VPC.

SQS vs SNS vs Kinesis Comparison

Important Exam Tips & Tricks

• Managed vs. Unmanaged:
  • Data Streams: You manage shards (scaling).
  • Firehose & Analytics: Fully managed (no shards to manage for Firehose, SQL-based for Analytics).
• Key Differentiator (Data Streams vs. Firehose):
  • Data Streams: For custom applications that need to process streaming data in real-time and potentially store it for a short period, allowing data replay. You write the consumer application.
  • Firehose: For loading streaming data into specific AWS destinations (S3, Redshift, ES, Splunk) with minimal effort. It's fully managed and handles batching, compression, and encryption automatically. You don't write a consumer application.
• Partition Key: Crucial for Kinesis Data Streams. It determines which shard a record goes into, ensuring ordering within a shard and even distribution of data. (Frequent Exam Question!)
• KCL & DynamoDB: Remember KCL uses DynamoDB for checkpointing (tracking processed records) and coordinating workers. (Key Exam Point!)
• KCL Scaling: Understand that the number of KCL instances should ideally match the number of shards for optimal performance, as each shard is processed by one KCL instance.
• Record Size: Remember the 1MB limit for Kinesis records.
• Retention Period: Know the default (24 hours) and max (7 days) retention for Data Streams.
• Replication: Kinesis Data Streams replicates across 3 AZs for high durability.
• Encryption: KMS is used for encryption at rest across Kinesis services.
• Use Cases: Be able to identify which Kinesis service fits a given scenario (e.g., "real-time dashboards" -> Analytics, "loading logs to S3" -> Firehose, "custom real-time processing" -> Data Streams).