AWS Secure Document Pipeline - Part 3: Advanced Security and Monitoring

Learn how to implement enterprise-grade security and monitoring for your document processing pipeline. Complete setup with KMS encryption, CloudTrail logging, CloudWatch alarms, and SNS notifications.

AWS Services

October 26, 2025

AWS Secure Document Pipeline - Part 3: Advanced Security and Monitoring

Share This Post

Twitter LinkedIn Copy Link

AWS Secure Document Pipeline - Part 3: Advanced Security and Monitoring

Introduction

Implementing enterprise-grade security and monitoring is crucial for production document processing pipelines. This comprehensive guide walks you through adding advanced security controls, comprehensive audit logging, real-time monitoring, and automated alerting to your existing infrastructure.

What You’ll Learn

How to implement custom KMS encryption for all S3 buckets
Setting up comprehensive CloudTrail logging for audit compliance
Configuring CloudWatch alarms for operational monitoring
Creating SNS notifications for real-time alerting
Building operational dashboards for visibility
Implementing advanced compliance features

Prerequisites

Before starting Phase 3, ensure:

Phase 1 is completed (S3 infrastructure)
Phase 2 is completed (Lambda function)
You have email access for SNS subscriptions
AWS CLI configured with appropriate permissions

Architecture

┌──────────────────────────────────────────────────────────────────┐
│                     Security & Monitoring Layer                  │
└──────────────────────────────────────────────────────────────────┘

┌─────────────────┐         ┌──────────────────┐
│  AWS KMS Keys   │────────▶│  S3 Buckets      │
│  (per bucket)   │         │  (Encrypted)     │
└─────────────────┘         └──────────────────┘
                                     │
                                     │ API Calls
                                     ↓
                            ┌──────────────────┐
                            │  AWS CloudTrail  │
                            │  (Audit Logs)    │
                            └──────────────────┘
                                     │
                                     ↓
┌──────────────────┐         ┌──────────────────┐         ┌─────────────┐
│ CloudWatch       │────────▶│  CloudWatch      │────────▶│  SNS Topic  │
│ Metrics          │         │  Alarms          │         │  (Alerts)   │
└──────────────────┘         └──────────────────┘         └──────┬──────┘
                                                                  │
                                                                  ↓
                                                            Email/SMS
                                                            Notifications

Prerequisites

Before starting Phase 3, ensure:

Phase 1 is completed (S3 infrastructure)
Phase 2 is completed (Lambda function)
You have email access for SNS subscriptions
AWS CLI configured with appropriate permissions

Architecture Overview

Our security and monitoring layer provides enterprise-grade protection and visibility:

┌──────────────────────────────────────────────────────────────────┐
│                     Security & Monitoring Layer                  │
└──────────────────────────────────────────────────────────────────┘

┌─────────────────┐         ┌──────────────────┐
│  AWS KMS Keys   │────────▶│  S3 Buckets      │
│  (per bucket)   │         │  (Encrypted)     │
└─────────────────┘         └──────────────────┘
                                     │
                                     │ API Calls
                                     ↓
                            ┌──────────────────┐
                            │  AWS CloudTrail  │
                            │  (Audit Logs)    │
                            └──────────────────┘
                                     │
                                     ↓
┌──────────────────┐         ┌──────────────────┐         ┌─────────────┐
│ CloudWatch       │────────▶│  CloudWatch      │────────▶│  SNS Topic  │
│ Metrics          │         │  Alarms          │         │  (Alerts)   │
└──────────────────┘         └──────────────────┘         └──────┬──────┘
                                                                  │
                                                                  ↓
                                                            Email/SMS
                                                            Notifications

Key Benefits

Enterprise Security: Customer-managed KMS encryption with automatic key rotation
Comprehensive Auditing: CloudTrail logging for all S3 operations and API calls
Real-time Monitoring: CloudWatch alarms for immediate issue detection
Automated Alerting: SNS notifications for operational awareness
Compliance Ready: Meets enterprise security and audit requirements
Operational Visibility: Dashboards for monitoring and troubleshooting

What We’ll Implement

1. Custom KMS Encryption

Create dedicated KMS keys for each critical bucket
Enable automatic key rotation
Configure key policies for service access
Update S3 buckets to use KMS encryption

2. AWS CloudTrail

Create a dedicated trail for S3 data events
Configure logging for all bucket operations
Enable log file validation
Set up log encryption

3. CloudWatch Monitoring

Create custom metrics for pipeline operations
Set up alarms for Lambda failures
Monitor S3 replication lag
Track processing times

Create SNS topic for operational alerts
Configure email subscriptions
Integrate with CloudWatch alarms
Set up dead-letter queues

5. Enhanced Compliance

Implement S3 Object Lock (optional)
Configure VPC Endpoint for private S3 access
Enable MFA delete protection
Set up AWS Config rules

Step 1: Custom KMS Encryption

Important Note on KMS with S3 Replication

When implementing KMS encryption with S3 cross-bucket replication, you need to:

Grant KMS permissions to the replication IAM role - Add kms:Encrypt, kms:Decrypt, kms:ReEncrypt*, kms:GenerateDataKey*, kms:DescribeKey
Update KMS key policies - Include the replication role in both source and destination key policies
Configure replication rules for KMS - Add source_selection_criteria and encryption_configuration blocks
Grant enhanced S3 service permissions - S3 service needs kms:Encrypt and kms:ReEncrypt* permissions
Grant third-party user encrypt permission - Add kms:Encrypt to the third-party user’s KMS permissions

These changes are critical for KMS-encrypted object replication to work properly.

Create KMS Keys with Terraform

Add the following to terraform/main.tf:

# ============================================
# KMS Keys for S3 Bucket Encryption
# ============================================

# KMS Key for uploads bucket
resource "aws_kms_key" "uploads_key" {
  description             = "KMS key for ${var.project_name} uploads bucket encryption"
  deletion_window_in_days = 10
  enable_key_rotation     = true

  tags = {
    Name = "${var.project_name}-uploads-kms-key"
  }
}

resource "aws_kms_alias" "uploads_key_alias" {
  name          = "alias/${var.project_name}-uploads"
  target_key_id = aws_kms_key.uploads_key.key_id
}

# KMS Key for internal-processing bucket
resource "aws_kms_key" "internal_processing_key" {
  description             = "KMS key for ${var.project_name} internal-processing bucket encryption"
  deletion_window_in_days = 10
  enable_key_rotation     = true

  tags = {
    Name = "${var.project_name}-internal-processing-kms-key"
  }
}

resource "aws_kms_alias" "internal_processing_key_alias" {
  name          = "alias/${var.project_name}-internal-processing"
  target_key_id = aws_kms_key.internal_processing_key.key_id
}

# KMS Key for processed-output bucket
resource "aws_kms_key" "processed_output_key" {
  description             = "KMS key for ${var.project_name} processed-output bucket encryption"
  deletion_window_in_days = 10
  enable_key_rotation     = true

  tags = {
    Name = "${var.project_name}-processed-output-kms-key"
  }
}

resource "aws_kms_alias" "processed_output_key_alias" {
  name          = "alias/${var.project_name}-processed-output"
  target_key_id = aws_kms_key.processed_output_key.key_id
}

# KMS Key for delivery bucket
resource "aws_kms_key" "delivery_key" {
  description             = "KMS key for ${var.project_name} delivery bucket encryption"
  deletion_window_in_days = 10
  enable_key_rotation     = true

  tags = {
    Name = "${var.project_name}-delivery-kms-key"
  }
}

resource "aws_kms_alias" "delivery_key_alias" {
  name          = "alias/${var.project_name}-delivery"
  target_key_id = aws_kms_key.delivery_key.key_id
}

# ============================================
# KMS Key Policy for S3 and Lambda Access
# ============================================

# Policy for uploads bucket KMS key
data "aws_iam_policy_document" "uploads_key_policy" {
  statement {
    sid    = "Enable IAM User Permissions"
    effect = "Allow"
    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"]
    }
    actions   = ["kms:*"]
    resources = ["*"]
  }

  statement {
    sid    = "Allow S3 to use the key"
    effect = "Allow"
    principals {
      type        = "Service"
      identifiers = ["s3.amazonaws.com"]
    }
    actions = [
      "kms:Encrypt",
      "kms:Decrypt",
      "kms:ReEncrypt*",
      "kms:GenerateDataKey*",
      "kms:DescribeKey"
    ]
    resources = ["*"]
  }

  statement {
    sid    = "Allow third party user to encrypt/decrypt"
    effect = "Allow"
    principals {
      type        = "AWS"
      identifiers = [aws_iam_user.third_party_user.arn]
    }
    actions = [
      "kms:Encrypt",
      "kms:Decrypt",
      "kms:GenerateDataKey",
      "kms:DescribeKey"
    ]
    resources = ["*"]
  }

  statement {
    sid    = "Allow S3 replication to decrypt"
    effect = "Allow"
    principals {
      type        = "AWS"
      identifiers = [aws_iam_role.replication_role.arn]
    }
    actions = [
      "kms:Decrypt",
      "kms:GenerateDataKey",
      "kms:DescribeKey"
    ]
    resources = ["*"]
  }
}

resource "aws_kms_key_policy" "uploads_key_policy" {
  key_id = aws_kms_key.uploads_key.id
  policy = data.aws_iam_policy_document.uploads_key_policy.json
}

# Policy for internal-processing bucket KMS key
data "aws_iam_policy_document" "internal_processing_key_policy" {
  statement {
    sid    = "Enable IAM User Permissions"
    effect = "Allow"
    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"]
    }
    actions   = ["kms:*"]
    resources = ["*"]
  }

  statement {
    sid    = "Allow S3 replication"
    effect = "Allow"
    principals {
      type        = "AWS"
      identifiers = [aws_iam_role.replication_role.arn]
    }
    actions = [
      "kms:Encrypt",
      "kms:Decrypt",
      "kms:ReEncrypt*",
      "kms:GenerateDataKey*",
      "kms:DescribeKey"
    ]
    resources = ["*"]
  }

  statement {
    sid    = "Allow Lambda to decrypt"
    effect = "Allow"
    principals {
      type        = "AWS"
      identifiers = [aws_iam_role.lambda_execution_role.arn]
    }
    actions = [
      "kms:Decrypt",
      "kms:DescribeKey"
    ]
    resources = ["*"]
  }
}

resource "aws_kms_key_policy" "internal_processing_key_policy" {
  key_id = aws_kms_key.internal_processing_key.id
  policy = data.aws_iam_policy_document.internal_processing_key_policy.json
}

# Policy for processed-output bucket KMS key
data "aws_iam_policy_document" "processed_output_key_policy" {
  statement {
    sid    = "Enable IAM User Permissions"
    effect = "Allow"
    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"]
    }
    actions   = ["kms:*"]
    resources = ["*"]
  }

  statement {
    sid    = "Allow Lambda to encrypt"
    effect = "Allow"
    principals {
      type        = "AWS"
      identifiers = [aws_iam_role.lambda_execution_role.arn]
    }
    actions = [
      "kms:Decrypt",
      "kms:GenerateDataKey"
    ]
    resources = ["*"]
  }

  statement {
    sid    = "Allow S3 replication"
    effect = "Allow"
    principals {
      type        = "AWS"
      identifiers = [aws_iam_role.replication_role.arn]
    }
    actions = [
      "kms:Decrypt",
      "kms:GenerateDataKey"
    ]
    resources = ["*"]
  }
}

resource "aws_kms_key_policy" "processed_output_key_policy" {
  key_id = aws_kms_key.processed_output_key.id
  policy = data.aws_iam_policy_document.processed_output_key_policy.json
}

# Policy for delivery bucket KMS key
data "aws_iam_policy_document" "delivery_key_policy" {
  statement {
    sid    = "Enable IAM User Permissions"
    effect = "Allow"
    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"]
    }
    actions   = ["kms:*"]
    resources = ["*"]
  }

  statement {
    sid    = "Allow third party user to decrypt"
    effect = "Allow"
    principals {
      type        = "AWS"
      identifiers = [aws_iam_user.third_party_user.arn]
    }
    actions = [
      "kms:Decrypt",
      "kms:DescribeKey"
    ]
    resources = ["*"]
  }

  statement {
    sid    = "Allow S3 replication"
    effect = "Allow"
    principals {
      type        = "AWS"
      identifiers = [aws_iam_role.replication_role.arn]
    }
    actions = [
      "kms:Decrypt",
      "kms:GenerateDataKey"
    ]
    resources = ["*"]
  }
}

resource "aws_kms_key_policy" "delivery_key_policy" {
  key_id = aws_kms_key.delivery_key.id
  policy = data.aws_iam_policy_document.delivery_key_policy.json
}

# ============================================
# Data Source for Current AWS Account
# ============================================
data "aws_caller_identity" "current" {}

# ============================================
# Update S3 Bucket Encryption with KMS
# ============================================

# IMPORTANT: Comment out the old AES256 encryption configuration from Phase 1
# Find and comment out this block in your main.tf:
#
# resource "aws_s3_bucket_server_side_encryption_configuration" "encryption" {
#   for_each = aws_s3_bucket.doc_buckets
#   bucket   = each.value.id
#
#   rule {
#     apply_server_side_encryption_by_default {
#       sse_algorithm = "AES256"
#     }
#     bucket_key_enabled = true
#   }
# }

# Now add the new KMS-based encryption configurations:

resource "aws_s3_bucket_server_side_encryption_configuration" "uploads_encryption" {
  bucket = aws_s3_bucket.doc_buckets["uploads"].id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.uploads_key.arn
    }
    bucket_key_enabled = true
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "internal_processing_encryption" {
  bucket = aws_s3_bucket.doc_buckets["internal_processing"].id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.internal_processing_key.arn
    }
    bucket_key_enabled = true
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "processed_output_encryption" {
  bucket = aws_s3_bucket.doc_buckets["processed_output"].id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.processed_output_key.arn
    }
    bucket_key_enabled = true
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "delivery_encryption" {
  bucket = aws_s3_bucket.doc_buckets["delivery"].id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.delivery_key.arn
    }
    bucket_key_enabled = true
  }
}

# Note: The compliance_logs bucket can keep AES256 encryption or you can create a KMS key for it too

Step 1.5: Update Replication Configuration for KMS

When using KMS encryption with S3 replication, you need to update both the replication configuration and IAM policies.

Update Replication IAM Policy

Add KMS permissions to the replication policy in terraform/main.tf:

resource "aws_iam_policy" "replication_policy" {
  name = "${var.project_name}-replication-policy"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "AllowS3GetReplicationConfiguration"
        Effect = "Allow"
        Action = [
          "s3:GetReplicationConfiguration",
          "s3:ListBucket"
        ]
        Resource = [
          aws_s3_bucket.doc_buckets["uploads"].arn,
          aws_s3_bucket.doc_buckets["processed_output"].arn
        ]
      },
      {
        Sid    = "AllowS3GetObjectVersions"
        Effect = "Allow"
        Action = [
          "s3:GetObjectVersionForReplication",
          "s3:GetObjectVersionAcl",
          "s3:GetObjectVersionTagging"
        ]
        Resource = [
          "${aws_s3_bucket.doc_buckets["uploads"].arn}/*",
          "${aws_s3_bucket.doc_buckets["processed_output"].arn}/*"
        ]
      },
      {
        Sid    = "AllowS3ReplicateObjects"
        Effect = "Allow"
        Action = [
          "s3:ReplicateObject",
          "s3:ReplicateDelete",
          "s3:ReplicateTags"
        ]
        Resource = [
          "${aws_s3_bucket.doc_buckets["internal_processing"].arn}/*",
          "${aws_s3_bucket.doc_buckets["delivery"].arn}/*"
        ]
      },
      {
        Sid    = "AllowKMSForReplication"
        Effect = "Allow"
        Action = [
          "kms:Decrypt",
          "kms:Encrypt",
          "kms:ReEncrypt*",
          "kms:GenerateDataKey*",
          "kms:DescribeKey"
        ]
        Resource = [
          aws_kms_key.uploads_key.arn,
          aws_kms_key.internal_processing_key.arn,
          aws_kms_key.processed_output_key.arn,
          aws_kms_key.delivery_key.arn
        ]
      }
    ]
  })
}

Update Replication Rules with KMS Configuration

Update your S3 replication rules to specify the destination KMS keys:

resource "aws_s3_bucket_replication_configuration" "uploads_to_internal" {
  role   = aws_iam_role.replication_role.arn
  bucket = aws_s3_bucket.doc_buckets["uploads"].id

  rule {
    id     = "ReplicateAllUploads"
    status = "Enabled"
    filter {}

    # IMPORTANT: Enable KMS encrypted object replication
    source_selection_criteria {
      sse_kms_encrypted_objects {
        status = "Enabled"
      }
    }

    destination {
      bucket        = aws_s3_bucket.doc_buckets["internal_processing"].arn
      storage_class = "STANDARD"

      # Specify destination KMS key
      encryption_configuration {
        replica_kms_key_id = aws_kms_key.internal_processing_key.arn
      }
    }

    delete_marker_replication {
      status = "Enabled"
    }
  }

  depends_on = [aws_s3_bucket_versioning.versioning, aws_iam_role_policy_attachment.replication_attach]
}

resource "aws_s3_bucket_replication_configuration" "processed_to_delivery" {
  role   = aws_iam_role.replication_role.arn
  bucket = aws_s3_bucket.doc_buckets["processed_output"].id

  rule {
    id     = "ReplicateProcessedFiles"
    status = "Enabled"
    filter {}

    # IMPORTANT: Enable KMS encrypted object replication
    source_selection_criteria {
      sse_kms_encrypted_objects {
        status = "Enabled"
      }
    }

    destination {
      bucket        = aws_s3_bucket.doc_buckets["delivery"].arn
      storage_class = "STANDARD"

      # Specify destination KMS key
      encryption_configuration {
        replica_kms_key_id = aws_kms_key.delivery_key.arn
      }
    }

    delete_marker_replication {
      status = "Enabled"
    }
  }

  depends_on = [aws_s3_bucket_versioning.versioning, aws_iam_role_policy_attachment.replication_attach]
}

Key Changes for KMS Replication:

source_selection_criteria - Tells S3 to replicate KMS-encrypted objects
encryption_configuration - Specifies the destination KMS key for re-encryption
IAM policy - Added KMS permissions for Decrypt, Encrypt, ReEncrypt operations
KMS key policies - Added replication role to both source and destination key policies

Why These Changes Are Needed:

When S3 replicates a KMS-encrypted object:

It decrypts the object using the source KMS key
Re-encrypts it with the destination KMS key
Stores it in the destination bucket

This requires the replication role to have permissions on both KMS keys.

Step 2: AWS CloudTrail Setup

Add CloudTrail configuration to terraform/main.tf:

# ============================================
# S3 Bucket for CloudTrail Logs
# ============================================
resource "aws_s3_bucket" "cloudtrail_logs" {
  bucket        = "${var.project_name}-cloudtrail-logs"
  force_destroy = var.force_destroy_buckets

  tags = {
    Name = "${var.project_name}-cloudtrail-logs"
  }
}

resource "aws_s3_bucket_public_access_block" "cloudtrail_pab" {
  bucket = aws_s3_bucket.cloudtrail_logs.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

# CloudTrail bucket policy
data "aws_iam_policy_document" "cloudtrail_bucket_policy" {
  statement {
    sid    = "AWSCloudTrailAclCheck"
    effect = "Allow"
    principals {
      type        = "Service"
      identifiers = ["cloudtrail.amazonaws.com"]
    }
    actions   = ["s3:GetBucketAcl"]
    resources = [aws_s3_bucket.cloudtrail_logs.arn]
  }

  statement {
    sid    = "AWSCloudTrailWrite"
    effect = "Allow"
    principals {
      type        = "Service"
      identifiers = ["cloudtrail.amazonaws.com"]
    }
    actions   = ["s3:PutObject"]
    resources = ["${aws_s3_bucket.cloudtrail_logs.arn}/*"]
    condition {
      test     = "StringEquals"
      variable = "s3:x-amz-acl"
      values   = ["bucket-owner-full-control"]
    }
  }
}

resource "aws_s3_bucket_policy" "cloudtrail_bucket_policy" {
  bucket = aws_s3_bucket.cloudtrail_logs.id
  policy = data.aws_iam_policy_document.cloudtrail_bucket_policy.json
}

# ============================================
# CloudTrail Trail
# ============================================
resource "aws_cloudtrail" "cloudtrail_trail" {
  name                          = "${var.project_name}-trail"
  s3_bucket_name                = aws_s3_bucket.cloudtrail_logs.id
  include_global_service_events = true
  is_multi_region_trail         = true
  enable_log_file_validation    = true

  event_selector {
    read_write_type           = "All"
    include_management_events = true

    # Log S3 data events for all our buckets
    data_resource {
      type   = "AWS::S3::Object"
      values = [
        "${aws_s3_bucket.doc_buckets[local.bucket_names.uploads].arn}/*",
        "${aws_s3_bucket.doc_buckets[local.bucket_names.internal_processing].arn}/*",
        "${aws_s3_bucket.doc_buckets[local.bucket_names.processed_output].arn}/*",
        "${aws_s3_bucket.doc_buckets[local.bucket_names.delivery].arn}/*"
      ]
    }
  }

  depends_on = [aws_s3_bucket_policy.cloudtrail_bucket_policy]

  tags = {
    Name = "${var.project_name}-trail"
  }
}

Add SNS configuration to terraform/main.tf:

# ============================================
# SNS Topic for Operational Alerts
# ============================================
resource "aws_sns_topic" "alerts" {
  name = "${var.project_name}-alerts"

  tags = {
    Name = "${var.project_name}-alerts"
  }
}

# SNS Topic Policy
data "aws_iam_policy_document" "sns_topic_policy" {
  statement {
    sid    = "Allow CloudWatch to publish to SNS"
    effect = "Allow"
    principals {
      type        = "Service"
      identifiers = ["cloudwatch.amazonaws.com"]
    }
    actions   = ["SNS:Publish"]
    resources = [aws_sns_topic.alerts.arn]
  }
}

resource "aws_sns_topic_policy" "alerts_policy" {
  arn    = aws_sns_topic.alerts.arn
  policy = data.aws_iam_policy_document.sns_topic_policy.json
}

# ============================================
# SNS Email Subscription (requires manual confirmation)
# ============================================
variable "alert_email" {
  description = "Email address for receiving alerts"
  type        = string
  default     = "your-email@example.com"
}

resource "aws_sns_topic_subscription" "email_alerts" {
  topic_arn = aws_sns_topic.alerts.arn
  protocol  = "email"
  endpoint  = var.alert_email
}

Add to terraform/terraform.tfvars:

alert_email = "your-email@example.com"  # Replace with your actual email

Step 4: CloudWatch Alarms

Add CloudWatch alarms to terraform/main.tf:

# ============================================
# CloudWatch Alarms
# ============================================

# Alarm: Lambda Function Errors
resource "aws_cloudwatch_metric_alarm" "lambda_errors" {
  alarm_name          = "${var.project_name}-lambda-errors"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "Errors"
  namespace           = "AWS/Lambda"
  period              = 300  # 5 minutes
  statistic           = "Sum"
  threshold           = 1
  alarm_description   = "Alert when Lambda function has errors"
  alarm_actions       = [aws_sns_topic.alerts.arn]

  dimensions = {
    FunctionName = aws_lambda_function.document_processor.function_name
  }

  tags = {
    Name = "${var.project_name}-lambda-errors"
  }
}

# Alarm: Lambda Function Throttles
resource "aws_cloudwatch_metric_alarm" "lambda_throttles" {
  alarm_name          = "${var.project_name}-lambda-throttles"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "Throttles"
  namespace           = "AWS/Lambda"
  period              = 300
  statistic           = "Sum"
  threshold           = 1
  alarm_description   = "Alert when Lambda function is throttled"
  alarm_actions       = [aws_sns_topic.alerts.arn]

  dimensions = {
    FunctionName = aws_lambda_function.document_processor.function_name
  }

  tags = {
    Name = "${var.project_name}-lambda-throttles"
  }
}

# Alarm: Lambda Duration (Performance)
resource "aws_cloudwatch_metric_alarm" "lambda_duration" {
  alarm_name          = "${var.project_name}-lambda-duration"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 2
  metric_name         = "Duration"
  namespace           = "AWS/Lambda"
  period              = 300
  statistic           = "Average"
  threshold           = 60000  # 60 seconds
  alarm_description   = "Alert when Lambda execution time exceeds 60 seconds"
  alarm_actions       = [aws_sns_topic.alerts.arn]

  dimensions = {
    FunctionName = aws_lambda_function.document_processor.function_name
  }

  tags = {
    Name = "${var.project_name}-lambda-duration"
  }
}

# Alarm: S3 Replication Lag (Custom Metric)
resource "aws_cloudwatch_metric_alarm" "replication_lag" {
  alarm_name          = "${var.project_name}-replication-lag"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 2
  metric_name         = "ReplicationLatency"
  namespace           = "AWS/S3"
  period              = 900  # 15 minutes
  statistic           = "Maximum"
  threshold           = 900  # 15 minutes in seconds
  alarm_description   = "Alert when S3 replication lag exceeds 15 minutes"
  alarm_actions       = [aws_sns_topic.alerts.arn]
  treat_missing_data  = "notBreaching"

  dimensions = {
    SourceBucket      = aws_s3_bucket.doc_buckets[local.bucket_names.uploads].id
    DestinationBucket = aws_s3_bucket.doc_buckets[local.bucket_names.internal_processing].id
    RuleId            = "ReplicateAllUploads"
  }

  tags = {
    Name = "${var.project_name}-replication-lag"
  }
}

# Alarm: S3 4xx Errors
resource "aws_cloudwatch_metric_alarm" "s3_4xx_errors" {
  alarm_name          = "${var.project_name}-s3-4xx-errors"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "4xxErrors"
  namespace           = "AWS/S3"
  period              = 300
  statistic           = "Sum"
  threshold           = 10
  alarm_description   = "Alert on excessive S3 4xx errors"
  alarm_actions       = [aws_sns_topic.alerts.arn]

  dimensions = {
    BucketName = aws_s3_bucket.doc_buckets[local.bucket_names.uploads].id
  }

  tags = {
    Name = "${var.project_name}-s3-4xx-errors"
  }
}

# Alarm: S3 5xx Errors
resource "aws_cloudwatch_metric_alarm" "s3_5xx_errors" {
  alarm_name          = "${var.project_name}-s3-5xx-errors"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "5xxErrors"
  namespace           = "AWS/S3"
  period              = 300
  statistic           = "Sum"
  threshold           = 1
  alarm_description   = "Alert on any S3 5xx errors"
  alarm_actions       = [aws_sns_topic.alerts.arn]

  dimensions = {
    BucketName = aws_s3_bucket.doc_buckets[local.bucket_names.uploads].id
  }

  tags = {
    Name = "${var.project_name}-s3-5xx-errors"
  }
}

Step 5: CloudWatch Dashboard

Add dashboard configuration to terraform/main.tf:

# ============================================
# CloudWatch Dashboard
# ============================================
resource "aws_cloudwatch_dashboard" "main" {
  dashboard_name = "${var.project_name}-dashboard"

  dashboard_body = jsonencode({
    widgets = [
      {
        type = "metric"
        properties = {
          metrics = [
            ["AWS/Lambda", "Invocations", { stat = "Sum", label = "Lambda Invocations" }],
            [".", "Errors", { stat = "Sum", label = "Lambda Errors" }],
            [".", "Duration", { stat = "Average", label = "Avg Duration (ms)" }]
          ]
          view    = "timeSeries"
          stacked = false
          region  = var.aws_region
          title   = "Lambda Function Metrics"
          period  = 300
        }
      },
      {
        type = "metric"
        properties = {
          metrics = [
            ["AWS/S3", "NumberOfObjects", { stat = "Average" }],
            [".", "BucketSizeBytes", { stat = "Average" }]
          ]
          view    = "timeSeries"
          stacked = false
          region  = var.aws_region
          title   = "S3 Storage Metrics"
          period  = 86400
        }
      },
      {
        type = "log"
        properties = {
          query   = "SOURCE '/aws/lambda/${aws_lambda_function.document_processor.function_name}' | fields @timestamp, @message | filter @message like /ERROR/ | sort @timestamp desc | limit 20"
          region  = var.aws_region
          title   = "Recent Lambda Errors"
        }
      }
    ]
  })
}

Step 6: Deploy Security Enhancements

Update variables and deploy

cd terraform
terraform init

# Update terraform.tfvars with your email
# alert_email = "your-actual-email@example.com"

# Validate configuration
terraform validate

# Preview changes
terraform plan

# Apply changes
terraform apply

Type yes when prompted.

Deployment time: 3-5 minutes

After deployment:

Check your email inbox
Look for email from “AWS Notifications”
Click “Confirm subscription” link
You’ll see a confirmation page

Verify subscription:

aws sns list-subscriptions-by-topic \
  --topic-arn $(terraform output -raw sns_topic_arn)

Step 8: Test Security and Monitoring

Test 1: Verify KMS Encryption

# Upload a file with third-party profile
echo "test encrypted data" > kms-test.pdf
aws s3 cp kms-test.pdf s3://secure-doc-pipeline-uploads/ --profile third-party-test

# Check encryption details
aws s3api head-object \
  --bucket secure-doc-pipeline-uploads \
  --key kms-test.pdf \
  --query 'ServerSideEncryption'

Expected output: "aws:kms"

Test 2: Verify CloudTrail Logging

# Wait 5-10 minutes after upload, then check CloudTrail logs
aws s3 ls s3://secure-doc-pipeline-cloudtrail-logs/ --recursive

# Look for recent log files
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=secure-doc-pipeline-uploads \
  --max-results 5

Test 3: Trigger Lambda Error Alarm

Create a test file that will cause Lambda to fail:

# Create an invalid PDF
echo "this is not a valid PDF" > invalid.pdf

# Upload it
aws s3 cp invalid.pdf s3://secure-doc-pipeline-uploads/ --profile third-party-test

# Check CloudWatch alarm status (wait 5 minutes)
aws cloudwatch describe-alarms \
  --alarm-names secure-doc-pipeline-lambda-errors \
  --query 'MetricAlarms[0].StateValue'

You should receive an email alert if the alarm triggers.

Test 4: View CloudWatch Dashboard

# Get dashboard URL
echo "https://console.aws.amazon.com/cloudwatch/home?region=ap-south-1#dashboards:name=secure-doc-pipeline-dashboard"

Open in browser to see real-time metrics.

Step 9: Advanced Compliance Features (Optional)

Enable S3 Object Lock for Compliance

Add to terraform/main.tf:

# Enable Object Lock on compliance-logs bucket
resource "aws_s3_bucket_object_lock_configuration" "compliance_lock" {
  bucket = aws_s3_bucket.doc_buckets[local.bucket_names.compliance_logs].id

  rule {
    default_retention {
      mode = "GOVERNANCE"  # or "COMPLIANCE" for stricter retention
      days = 365
    }
  }
}

Note: Object Lock must be enabled at bucket creation. You’ll need to recreate the bucket to enable this feature.

Enable MFA Delete Protection

# Enable versioning with MFA delete (requires MFA device)
aws s3api put-bucket-versioning \
  --bucket secure-doc-pipeline-uploads \
  --versioning-configuration Status=Enabled,MFADelete=Enabled \
  --mfa "arn:aws:iam::ACCOUNT_ID:mfa/USERNAME TOKEN_CODE"

Step 10: VPC Endpoint for Private S3 Access (Optional)

For enhanced security, access S3 privately without internet gateway:

# VPC Endpoint for S3 (if you have VPC infrastructure)
resource "aws_vpc_endpoint" "s3" {
  vpc_id       = var.vpc_id  # You need to provide this
  service_name = "com.amazonaws.${var.aws_region}.s3"

  tags = {
    Name = "${var.project_name}-s3-endpoint"
  }
}

resource "aws_vpc_endpoint_route_table_association" "s3_endpoint" {
  route_table_id  = var.route_table_id  # You need to provide this
  vpc_endpoint_id = aws_vpc_endpoint.s3.id
}

Troubleshooting

Issue: KMS Access Denied Errors

Symptoms: Third party or Lambda gets “KMS.AccessDeniedException”

Solutions:

Check KMS key policy:

aws kms get-key-policy \
  --key-id alias/secure-doc-pipeline-uploads \
  --policy-name default

Verify IAM permissions: Ensure users/roles have KMS permissions
Update key policy in Terraform and redeploy

Issue: S3 Replication Fails with KMS Encryption

Symptoms: Files uploaded to uploads bucket don’t replicate to internal-processing bucket after enabling KMS

Common Causes:

Missing KMS permissions in replication IAM policy
Missing replication role in KMS key policies
Replication rules not configured for KMS encryption

Solutions:

Verify replication IAM policy has KMS permissions:

aws iam get-policy-version \
  --policy-arn $(aws iam list-policies --query 'Policies[?PolicyName==`secure-doc-pipeline-replication-policy`].Arn' --output text) \
  --version-id $(aws iam list-policies --query 'Policies[?PolicyName==`secure-doc-pipeline-replication-policy`].DefaultVersionId' --output text)

Ensure it includes: kms:Decrypt, kms:Encrypt, kms:ReEncrypt*, kms:GenerateDataKey*, kms:DescribeKey

Verify KMS key policies include replication role:
Check both source (uploads) and destination (internal-processing) KMS key policies have the replication role with required permissions.
Verify replication configuration has KMS settings:
```
aws s3api get-bucket-replication \
  --bucket secure-doc-pipeline-uploads
```
Look for:
- SourceSelectionCriteria.SseKmsEncryptedObjects.Status: Enabled
- EncryptionConfiguration.ReplicaKmsKeyID with the destination KMS key ARN

Test with a fresh upload:

Remember: Replication is NOT retroactive. Only new uploads after fixing the configuration will replicate.

echo "test after KMS fix" > kms-replication-test.pdf
aws s3 cp kms-replication-test.pdf s3://secure-doc-pipeline-uploads/ --profile third-party-test

# Wait 1-2 minutes, then verify
aws s3 ls s3://secure-doc-pipeline-internal-processing/

Check replication metrics in CloudWatch:

aws cloudwatch get-metric-statistics \
  --namespace AWS/S3 \
  --metric-name ReplicationLatency \
  --dimensions Name=SourceBucket,Value=secure-doc-pipeline-uploads Name=DestinationBucket,Value=secure-doc-pipeline-internal-processing \
  --start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 300 \
  --statistics Maximum

Symptoms: No email confirmation received

Solutions:

Check spam folder
Verify SNS subscription:
```
aws sns list-subscriptions
```

Resend confirmation:

aws sns subscribe \
  --topic-arn arn:aws:sns:ap-south-1:ACCOUNT_ID:secure-doc-pipeline-alerts \
  --protocol email \
  --notification-endpoint your-email@example.com

Issue: CloudTrail Not Logging

Symptoms: No logs appearing in CloudTrail bucket

Solutions:

Check trail status:

aws cloudtrail get-trail-status \
  --name secure-doc-pipeline-trail

Verify bucket policy allows CloudTrail writes
Check for errors in CloudTrail console

Issue: CloudWatch Alarms Not Triggering

Symptoms: Alarms stay in “INSUFFICIENT_DATA” state

Solutions:

Generate test data: Upload files to trigger metrics
Wait for metric publication: Some metrics take 5-15 minutes

Check alarm configuration:

aws cloudwatch describe-alarms \
  --alarm-names secure-doc-pipeline-lambda-errors

Cost Estimation

Phase 3 Monthly Costs (Approximate)

KMS:

4 customer-managed keys: $4.00/month (4 × $1.00)
10,000 API requests: $0.30
Total KMS: ~$4.30/month

CloudTrail:

First trail: Free for management events
Data events: $0.10 per 100,000 events
10,000 events/month: ~$0.01
Total CloudTrail: ~$0.01/month

CloudWatch:

Alarms: 6 alarms × $0.10 = $0.60
Dashboard: 1 dashboard = $3.00
Logs: 100 MB ingested = $0.05
Metrics: Standard metrics (free)
Total CloudWatch: ~$3.65/month

SNS:

Email notifications: Free for first 1,000
Total SNS: ~$0.00/month

CloudTrail Log Storage (S3):

1 GB logs/month: ~$0.025
Total Storage: ~$0.03/month

Total Estimated Cost for Phase 3: ~$7.99/month

Combined All Phases: ~$8.57/month

Note: The majority of Phase 3 costs come from KMS keys ($4) and CloudWatch Dashboard ($3). These can be optimized:

Reduce number of KMS keys (share keys across buckets)
Remove dashboard (use console instead)
Optimized Phase 3 cost: ~$0.57/month

Security Best Practices Checklist

After Phase 3, your infrastructure includes:

Encryption at rest with customer-managed KMS keys
Automatic key rotation enabled
Encryption in transit (HTTPS enforced)
Comprehensive audit logging via CloudTrail
Log file validation enabled
Real-time alerting via CloudWatch and SNS
IP-based access control for third party
Least privilege IAM policies
Multi-layered security (S3, KMS, IAM, CloudTrail)
Monitoring dashboard for operational visibility
Automated error detection and notification

Additional Recommendations

Enable AWS Config for compliance monitoring
Implement AWS WAF if exposing API Gateway
Use AWS Security Hub for centralized security view
Enable AWS GuardDuty for threat detection
Implement backup strategy for critical data
Regular security audits using AWS Trusted Advisor
Implement disaster recovery plan
Document incident response procedures

Verification Checklist

Before considering the project complete, verify:

All KMS keys created and configured
Automatic key rotation enabled
S3 buckets using KMS encryption
CloudTrail logging all S3 data events
CloudTrail log files encrypted
SNS topic created and email confirmed
All CloudWatch alarms configured
CloudWatch dashboard displaying metrics
Test alarm triggers successfully
Email notifications received
KMS encryption working for uploads/downloads
CloudTrail logs appearing in S3 bucket
Dashboard shows real-time data
All phases working together end-to-end

Monitoring and Operational Tasks

Daily Tasks

# Check for alarm states
aws cloudwatch describe-alarms \
  --state-value ALARM

# View recent Lambda errors
aws logs tail /aws/lambda/secure-doc-pipeline-document-processor \
  --since 1h \
  --filter-pattern "ERROR"

# Check S3 replication status
aws s3api get-bucket-replication \
  --bucket secure-doc-pipeline-uploads

Weekly Tasks

# Review CloudTrail events
aws cloudtrail lookup-events \
  --start-time $(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%S) \
  --max-results 100

# Check storage usage
aws s3 ls s3://secure-doc-pipeline-uploads --recursive --summarize

# Review CloudWatch metrics
aws cloudwatch get-metric-statistics \
  --namespace AWS/Lambda \
  --metric-name Duration \
  --dimensions Name=FunctionName,Value=secure-doc-pipeline-document-processor \
  --start-time $(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 86400 \
  --statistics Average

Monthly Tasks

Review KMS key usage and costs
Audit IAM permissions
Check for security findings in Security Hub
Review and archive old logs
Update documentation

Next Steps

Congratulations! You’ve completed a production-grade secure document processing pipeline with:

Enterprise-level encryption (KMS)
Comprehensive audit logging (CloudTrail)
Real-time monitoring and alerting (CloudWatch + SNS)
Operational dashboard for visibility

Project Complete!

Your infrastructure now meets:

✅ Security compliance requirements
✅ Operational monitoring standards
✅ Audit and governance requirements
✅ Production-ready best practices

Ready for Cleanup?

When you’re done testing, proceed to the cleanup guide to safely tear down all resources and avoid AWS charges.

Proceed to: AWS Secure Document Pipeline - Part 4: Complete Resource Cleanup Guide Here is the Part 4, where we’ll clean up the source environment and temporary infrastructure!

Additional Resources

Share This Post

Twitter LinkedIn Copy Link

AWS Secure Document Pipeline - Part 3: Advanced Security and Monitoring

Table of Contents

Share This Post

AWS Secure Document Pipeline - Part 3: Advanced Security and Monitoring

Introduction

What You’ll Learn

Prerequisites

Architecture

Prerequisites

Architecture Overview

Key Benefits

What We’ll Implement

1. Custom KMS Encryption

2. AWS CloudTrail

3. CloudWatch Monitoring

4. SNS Notifications

5. Enhanced Compliance

Step 1: Custom KMS Encryption

Important Note on KMS with S3 Replication

Create KMS Keys with Terraform

Step 1.5: Update Replication Configuration for KMS

Update Replication IAM Policy

Update Replication Rules with KMS Configuration

Step 2: AWS CloudTrail Setup

Step 3: SNS Topic for Alerts

Step 4: CloudWatch Alarms

Step 5: CloudWatch Dashboard

Step 6: Deploy Security Enhancements

Update variables and deploy

Step 7: Confirm SNS Email Subscription

Step 8: Test Security and Monitoring

Test 1: Verify KMS Encryption

Test 2: Verify CloudTrail Logging

Test 3: Trigger Lambda Error Alarm

Test 4: View CloudWatch Dashboard

Step 9: Advanced Compliance Features (Optional)

Enable S3 Object Lock for Compliance

Enable MFA Delete Protection

Step 10: VPC Endpoint for Private S3 Access (Optional)

Troubleshooting

Issue: KMS Access Denied Errors

Issue: S3 Replication Fails with KMS Encryption

Issue: SNS Email Not Received

Issue: CloudTrail Not Logging

Issue: CloudWatch Alarms Not Triggering

Cost Estimation

Phase 3 Monthly Costs (Approximate)

Total Estimated Cost for Phase 3: ~$7.99/month

Combined All Phases: ~$8.57/month

Security Best Practices Checklist

Additional Recommendations

Verification Checklist

Monitoring and Operational Tasks

Daily Tasks

Weekly Tasks

Monthly Tasks

Next Steps

Project Complete!

Ready for Cleanup?

Additional Resources

Table of Contents

Share This Post