AWS Application Integration Services Overview

• Suite of services enabling decoupled communication between application components.
• Ideal for microservices, distributed systems, and serverless applications.
• Connects apps without custom interoperability code.
• Decoupled apps are resilient to individual component failures/overloads.

Amazon Simple Queue Service (SQS)

SQS Basics

• Definition: Distributed queue system for sending, storing, and receiving messages between software components.
• Acts as a temporary repository/buffer for messages.
• Resolves issues of producer/consumer speed mismatch or intermittent connectivity.
• Enables decoupling/loose coupling and elasticity.
• Pull-based: Consumers pull messages from the queue (unlike SNS, which is push-based).
• Message Size: Up to 256KB.
• Message Retention: 1 minute to 14 days (default: 4 days).
• Delivery Guarantee: Messages processed at least once. (Exam Tip!)

Standard vs. FIFO Queues

• Queue names must be unique within a region.
• Scalability: Achieved by creating more queues.
• Durability: Messages stored redundantly across multiple AZs.

Visibility Timeout

• Time a message is invisible in the queue after a consumer picks it up.
• If not processed/deleted within timeout, message becomes visible again and another consumer can process it (potential duplicate processing).
• Default: 30 seconds.
• Maximum: 12 hours.
• Increase if processing takes longer than default.

SQS Polling (Short vs. Long)

• Short Polling (Default):
  • Returns immediately, even if queue is empty.
  • Queries only a subset of servers for messages.
  • More requests, potentially higher cost.
  • ReceiveMessageWaitTime = 0.
• Long Polling:
  • Waits for a message to arrive or for the poll to time out (up to 20 seconds).
  • Queries all servers, eliminates false empty responses.
  • Fewer requests, reduces cost.
  • ReceiveMessageWaitTime > 0 (up to 20 seconds).
  • Shouldn't be used if immediate response is expected.

Dead-Letter Queue (DLQ)

• Handles message failures.
• Isolates messages that cannot be processed correctly.
• Not a queue type itself; a standard or FIFO queue designated as a DLQ for another queue.
• Messages move to DLQ when ReceiveCount exceeds maxReceiveCount.
• DLQs should not be used with standard queues if application retries transmission (can break order).
• DLQs will break the order of messages in FIFO queues if messages are moved.
• DLQ must be the same type (Standard or FIFO) as the source queue.

Delay Queues

• Postpones delivery of new messages to a queue for a specified duration.
• Messages remain invisible to consumers during the delay period.
• Default delay: 0 seconds; Maximum: 900 seconds (15 minutes).
• For standard queues, setting only affects new messages; for FIFO, it affects messages already in queue.
• Use for large distributed apps needing processing delays (e.g., allowing DB updates before sending notifications).

Lambda with SQS

• SQS can be configured as an event source for AWS Lambda functions.
• Lambda polls the SQS queue and invokes your function synchronously with batches of messages.
• If function successfully processes a batch, Lambda deletes messages from the queue.
• Supports both standard and FIFO queues.

SQS Extended Client

• Used with Amazon S3 to manage SQS messages larger than 256KB (up to 2GB).
• Stores message payload in S3 and sends a small message (reference to S3 object) to SQS.
• Client library handles S3 interaction (store, retrieve, delete object).

Monitoring & Security

• Monitoring: Integrated with CloudWatch (metrics collected every 5 mins, no charge for standard monitoring). CloudWatch considers queue active if messages or API access within 6 hours.
• Auditing: CloudTrail captures SQS API calls.
• Security:
  • IAM policies control read/write access.
  • In-flight security via HTTPS.
  • Server-Side Encryption (SSE) using KMS (encrypts message body, not attributes).
  • Queue access policies provide finer-grained control.
  • PCI DSS Level 1 compliant and HIPAA eligible.

Key SQS APIs (for Developer Associate Exam)

• CreateQueue: Creates a new standard or FIFO queue.
• DeleteQueue: Deletes a queue (successful even if queue doesn't exist).
• PurgeQueue: Deletes all messages in a queue.
• SendMessage: Delivers a message to a queue.
• ReceiveMessage: Retrieves 1-10 messages from a queue. Use WaitTimeSeconds for long polling.
• DeleteMessage: Deletes a specific message using its ReceiptHandle (NOT MessageId!).
• ChangeMessageVisibility: Changes the visibility timeout of a message.
• Batch operations: SendMessageBatch, DeleteMessageBatch, ChangeMessageVisibilityBatch (to reduce costs/requests).

Amazon Simple Notification Service (SNS)

SNS Basics

• Definition: Fully managed messaging service for Application-to-Application (A2A) and Application-to-Person (A2P) communication.
• Provides pub/sub functionality for high-throughput, push-based, many-to-many use cases.
• Used for sending notifications between distributed systems, microservices, and event-driven serverless applications.
• Push-based: Publishes messages to subscribers (unlike SQS, which is pull-based).
• Inexpensive, pay-as-you-go.

SNS Topics & Endpoints

• SNS Topics: Group multiple recipients (subscribers) to receive identical copies of the same notification. An "access point" for subscriptions.
• Messages stored redundantly across multiple AZs.
• Provides instantaneous, push-based delivery.
• Flexible message delivery over multiple transport protocols.
• Supported Endpoint Types (Subscribers):
  • HTTP/HTTPS
  • Email/Email-JSON
  • Amazon Kinesis Data Firehose
  • Amazon SQS queues (common for Fanout)
  • AWS Lambda functions (common for Fanout)
  • Platform application endpoint (mobile push: Apple, Google, Fire OS, Windows, Baidu)
  • SMS

SNS Fanout

• Publisher systems can fanout messages to many subscriber systems for parallel processing.
• Common fanout patterns: SNS Topic → multiple SQS Queues, Lambda functions, HTTP/S endpoints, Kinesis Data Firehose delivery streams.
• SQS manages the subscription and necessary permissions when subscribing an SQS queue to an SNS topic.

AWS Step Functions

• Definition: Coordinates components of distributed applications as a series of steps in a visual workflow (state machine).
• Quickly build and run state machines for reliable, scalable application execution.
• How it works:
  • Define workflow steps in JSON-based Amazon States Language.
  • Visual console graphs execution flow and highlights real-time status.
  • Operates and scales application steps and underlying compute automatically.
• Features:
  • Built-in Error Handling: Tracks state, retries failed/timed-out tasks, catches specific errors, recovers gracefully.
  • Automatic Scaling: Scales operations and compute in response to changing workloads.
  • Pay-per-use: Billed per state transition (up to one year duration).
  • Execution Event History: Detailed logs for troubleshooting.
  • High Availability: Built-in fault tolerance across multiple AZs.
  • Administrative Security: Integrated with IAM for API access control.
• Recommendation: AWS recommends Step Functions for new applications over SWF. (Exam Tip!)

Amazon Simple Workflow Service (SWF)

• Definition: Web service for coordinating work across distributed application components.
• Used for processing background jobs with parallel or sequential steps.
• Acts as a fully managed state tracker and task coordinator.
• Use if app steps take >500ms, need state tracking, or require recovery/retry on task failure.
• Best suited for human-enabled workflows (e.g., order fulfillment, procedural requests).
• Workflow executions can have a completion time of up to 1 year.
• Uses a task-oriented API.
• Ensures tasks are assigned once and never duplicated.
• Components: Domains, Workflows, Activities, Task Lists, Workers, Decider.
• Recommendation: AWS recommends Step Functions for new applications instead of SWF. (Exam Tip!)

Amazon MQ

• Definition: Managed message broker service for Apache ActiveMQ and RabbitMQ.
• Supports industry-standard APIs and protocols (JMS, NMS, MQTT, WebSockets).
• Purpose: Designed as a drop-in replacement for on-premises message brokers to ease migration without rewriting code. (Exam Tip!)
• Cost-efficient, flexible, manages administration and maintenance.
• Automatically provisions infrastructure for high availability (active/standby brokers across AZs).
• Comparison with SQS:
  • Use SQS if creating a new application from scratch (cloud-native queue).
  • Use MQ if migrating existing applications that rely on standard message broker APIs.
• Security: Encryption at rest and in transit (SSL), VPC private endpoints, security groups.
• Monitoring: Integrated with CloudWatch and CloudTrail.

Amazon Kinesis (Brief)

• Service for collecting, processing, and analyzing streaming data in real-time.
• Example use case: Collecting data from IoT devices for later processing.

Important Exam Tips & Tricks

• Decoupling is Key: Understand that all these services primarily aim to decouple application components, improving resilience, scalability, and maintainability.
• SQS vs. SNS (Crucial!):
  • SQS: Pull-based, message queue, buffer, consumers pull.
  • SNS: Push-based, pub/sub, notifications, publishers push to subscribers.
  • They often work together in a "Fanout" pattern (SNS → SQS).
• SQS Standard vs. FIFO:
  • Standard: High throughput, at-least-once delivery, best-effort ordering.
  • FIFO: Exactly-once processing, strict ordering, deduplication, message groups (lower throughput).
  • Know when to use each based on ordering/deduplication requirements.
• SQS Visibility Timeout: Important for preventing duplicate processing. Know its default and max values.
• SQS Polling: Differentiate between short (immediate, more requests) and long (waits, fewer requests, cost-efficient).
• SQS Dead-Letter Queues: For handling failed messages. Understand their impact on FIFO order.
• Step Functions vs. SWF:
  • Step Functions: Recommended for new apps, visual workflows, state machines, built-in error handling, pay-per-transition.
  • SWF: Older, more complex, for human-enabled workflows, long-running processes (up to 1 year).
• Amazon MQ vs. SQS:
  • MQ: For lift-and-shift of existing applications using standard messaging protocols (ActiveMQ, RabbitMQ).
  • SQS: For new cloud-native applications.
• Kinesis: Focus on real-time streaming data processing.
• Message Size: Remember SQS 256KB limit and the SQS Extended Client for larger messages (up to 2GB via S3).
• Security: All services integrate with IAM for access control, and many support SSE (KMS).