Introduction to Amazon CloudWatch Logs Insights

Amazon CloudWatch Logs Insights is a powerful tool designed to help developers, DevOps engineers, and cloud administrators extract actionable intelligence from their log data. Whether you’re troubleshooting application errors, monitoring system health, or auditing security events, CloudWatch Logs Insights enables you to query logs in real time using a purpose-built query language.

Query Methods: CloudWatch Logs Insights uses a SQL-like syntax with support for commands like:

  • fields to select specific log fields.
  • filter to apply conditions (e.g., filter @message LIKE “ERROR”).
  • stats to aggregate data (e.g., stats count() by @logStream).
  • sort and limit to refine results.

Key Use Cases:

  • Troubleshooting: Quickly identify errors, exceptions, or latency spikes.
  • Operational Monitoring: Track metrics like request rates, response times, or resource utilization.
  • Security Analysis: Detect suspicious patterns, unauthorized access, or compliance violations.
  • Cost Optimization: Pinpoint underutilized resources or inefficient processes.

Field-Level Indexing: A Game-Changer for Query Performance

AWS recently launched field-level indexing for CloudWatch Logs, a feature that dramatically improves query speed and cost efficiency. By indexing specific log fields, you enable CloudWatch to skip full log scans and retrieve data directly from optimized indexes.

How Indexing Works

Traditionally, querying logs required scanning every log entry in the specified time range—a process that could be slow and expensive for large datasets. With field-level indexing, CloudWatch creates structured metadata for selected fields (e.g., requestId, statusCode, userId), allowing the engine to:

  1. Locate Data Faster: Directly access indexed fields without scanning irrelevant logs.
  2. Reduce Scanned Data Volume: Minimize the amount of data processed per query.

Benefits of Field-Level Indexing

  • Improved Query Performance: Indexed queries execute up to 10x faster by bypassing full log scans. For example, filtering logs by a high-cardinality field like requestId becomes instantaneous.
  • Lower Costs: CloudWatch charges based on the amount of data scanned. Indexing reduces scanned data, directly lowering costs—especially for frequent or complex queries.
  • Lower Latency for Critical Workloads: Real-time applications (e.g., fraud detection, live monitoring) benefit from near-instantaneous results

Implementing Field-Level Indexing

1. Choosing Fields to Index

Not all fields need indexing. Prioritize:

  • High-Cardinality Fields (e.g., unique IDs, transaction IDs).
  • Frequently Queried Fields (e.g., statusCode, userId).

2. Creating Indexes

You create field indexes by creating field index policies. You can create account-level index policies that apply to your whole account, and you can also create policies that apply to only a single log group. For account-wide index policies, you can have one that applies to all log groups in the account. You can also create account-level index policies that apply to a subset of log groups in the account, selected by the prefixes of their log group names. If you have multiple account-level policies in the same account, the log group name prefixes for these policies can’t overlap.

By default Cloudwatch create indexes for system generated logs provided below

  • @logStream
  • @ingestionTime
  • @requestId
  • @type
  • @initDuration
  • @duration
  • @billedDuration
  • @memorySize
  • @maxMemoryUsed
  • @xrayTraceId
  • @xraySetmentId

3. How to query the Indexed Fields

{
  "eventName": "OrderPlaced",
  "sourceIPAddress": "203.0.113.42",
  "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
  "requestParameters": {
    "orderId": "ORD123456789",
    "userId": "USR987654321",
    "orderStatus": "Processing",
    "paymentDetails": {
      "paymentMethod": "Credit Card",
      "transactionId": "TXN654321ABC",
      "amount": 129.99,
      "currency": "USD"
    },
    "items": [
      {
        "itemId": "ITM12345",
        "productName": "Wireless Headphones",
        "quantity": 1,
        "price": 99.99
      },
      {
        "itemId": "ITM67890",
        "productName": "USB-C Charging Cable",
        "quantity": 2,
        "price": 15.00
      }
    ],
    "shippingAddress": {
      "name": "Alice Johnson",
      "street": "123 Main St",
      "city": "New York",
      "state": "NY",
      "zipCode": "10001",
      "country": "USA"
    }
  }
}

Take the example above,

Here we can create index on root level field, nested json fields, also array fields.

Root Field

Create an index with route path eventName

fields @timestamp, @message 
| filterIndex eventName = 'eventName'
Nested JSON field

Create an index with route path requestParameters.orderStatus

fields @timestamp, @message 
| filterIndex requestParameters.orderStatus = 'pending'
Array fields

Create an index with route path requestParameters.orderStatus.items.0.itemId

fields @timestamp, @message 
| filterIndex requestParameters.orderStatus.items.0.itemId = '1234'

Conclusion

Field-level indexing transforms CloudWatch Logs Insights into a faster, more cost-effective tool for log analysis. By strategically indexing high-value fields, teams can accelerate troubleshooting, optimize costs, and scale their monitoring workflows.

Ready to Get Started?

Unlock the full potential of your logs today—AWS has just made it faster and cheaper than ever! 🚀