Overview of CloudWatch Logs

CloudWatch Logs is a service designed for monitoring, storing, and accessing your log files from Amazon EC2 instances, AWS CloudTrail, and other sources. Below, we summarize the key aspects discussed in the lecture:

• Log Groups and Streams:
  • Log Groups: Collections named by the user, typically representing an application.
  • Log Streams: Instances within a Log Group, representing specific log files or containers in a cluster.
• Log Expiration Policy:
  • Options range from indefinite retention to a specific period between one day and ten years.
• Log Destinations:
  • Logs can be exported to Amazon S3, streamed to AWS services like Kinesis Data Streams, Kinesis Data Firehose, AWS Lambda, and Amazon OpenSearch.
  • All logs are encrypted by default, with the option for user-managed KMS encryption.

Types of Logs and Integration

Logs can be ingested from various sources using different methods:

• Sources: Elastic Beanstalk, ECS, AWS Lambda, VPC Flow Logs, API Gateway, CloudTrail, and Route53.
• Ingestion Methods: SDK, CloudWatch Logs Agent (deprecated), and CloudWatch Unified Agent.

Querying Logs with CloudWatch Logs Insights

CloudWatch Logs Insights provides a powerful querying and visualization tool for log data:

• Features: Allows users to write queries, specify timeframes, and visualize results.
• Capabilities: Supports saving queries, adding them to dashboards, and querying multiple log groups across different accounts.
• Query Language: Automatically detects fields from logs for building queries, filtering, calculating statistics, and more.

Exporting and Streaming Log Data

CloudWatch Logs supports various methods for exporting and streaming data:

• Batch Export to Amazon S3: Using the CreateExportTask API, exports can take up to 12 hours.

• Real-Time Streaming: Through subscription filters, log data can be streamed in real-time to destinations like Kinesis Data Streams, Kinesis Data Firehose, or AWS Lambda for further processing or analysis.