AWS Serverless Media Processing Pipeline - Part 3: Cleanup & Resource Management

Overview

This comprehensive guide will help you systematically delete all AWS resources created during the serverless media processing pipeline project to avoid ongoing charges. It’s critical to follow the deletion order to prevent errors caused by dependencies between resources.

What you’ll learn:

• Proper resource deletion order
• Cost verification techniques
• Billing alert setup
• Emergency cleanup procedures
• Resource monitoring strategies

Region: ap-south-1 (Mumbai)
Estimated Cleanup Time: 15-20 minutes
Prerequisites: Parts 1 & 2 completed

Table of Contents

1. Important Notes Before Starting
2. Deletion Order Summary
3. Step-by-Step Cleanup Process
4. Cost Verification
5. Emergency Procedures
6. Best Practices
7. Troubleshooting

Important Notes Before Starting

⚠️ Critical Warnings

1. Deletion is permanent - Once deleted, resources cannot be recovered
2. Follow the order - Some resources depend on others and must be deleted in sequence
3. Verify each step - Check that resources are fully deleted before moving to the next step
4. Check your AWS bill - After cleanup, verify no charges are accumulating in the Billing Console

📋 Pre-Cleanup Checklist

Before starting cleanup, ensure you have:

• Access to AWS Console with appropriate permissions
• List of all resources created (use the reference tables below)
• Understanding of your current AWS bill
• Backup of any important data (if needed)

Deletion Order Summary

The correct deletion order is crucial to avoid dependency errors:

1. S3 Event Notifications (stop triggers)
2. SQS Queue Messages (purge queues)
3. Lambda Functions (remove processing logic)
4. API Gateway API (remove public endpoints)
5. S3 Bucket Contents (empty buckets)
6. S3 Buckets (delete storage)
7. DynamoDB Table (remove data store)
8. SQS Queues (remove message queues)
9. CloudWatch Log Groups (clean up logs)
10. Lambda Layers (remove dependencies)
11. IAM Roles and Policies (clean up permissions)

Step-by-Step Cleanup Process

Step 1: Delete S3 Event Notifications

Why first: Prevents new Lambda invocations while we’re cleaning up

1.1 Navigate to S3

1. AWS Console → Search “S3” → Click S3

1.2 Open Uploads Bucket

1. Click on amodhbh-media-uploads

1.3 Delete Event Notification

1. Go to Properties tab
2. Scroll to Event notifications
3. Find trigger-dispatcher-on-upload
4. Click on it → Click Delete
5. Confirm deletion

1.4 Verify

• Event notifications section should be empty or show “No event notifications”

Step 2: Purge SQS Queues

Why before deleting Lambda: Prevents messages from being processed during cleanup

2.1 Purge Main Queue

1. Navigate to SQS:

   • AWS Console → Search “SQS” → Click Simple Queue Service

2. Purge processing queue:

   • Select media-processing-queue (checkbox)
   • Click Purge
   • Type purge to confirm
   • Click Purge
   • Wait for confirmation message

2.2 Purge Dead Letter Queue

1. Purge DLQ:

   • Select media-processing-dlq (checkbox)
   • Click Purge
   • Type purge to confirm
   • Click Purge

2. Verify:

   • Both queues should show “Messages available: 0”

Step 3: Delete Lambda Functions

Order matters: Delete in this sequence to avoid errors

3.1 Delete Worker Lambda

1. Navigate to Lambda:

   • AWS Console → Search “Lambda” → Click Lambda
   • Click Functions in left sidebar

2. Delete worker-lambda:

   • Select worker-lambda (checkbox)
   • Click Actions → Delete
   • Type delete to confirm
   • Click Delete

3.2 Delete Dispatcher Lambda

1. Delete dispatcher-lambda:
   • Select dispatcher-lambda (checkbox)
   • Click Actions → Delete
   • Type delete to confirm
   • Click Delete

3.3 Delete API Lambda Functions

1. Delete request-upload-lambda:

   • Select request-upload-lambda (checkbox)
   • Click Actions → Delete
   • Type delete to confirm
   • Click Delete

2. Delete get-job-status-lambda:

   • Select get-job-status-lambda (checkbox)
   • Click Actions → Delete
   • Type delete to confirm
   • Click Delete

3.4 Verify

• Lambda functions list should not show any of these functions:
  • worker-lambda
  • dispatcher-lambda
  • request-upload-lambda
  • get-job-status-lambda

Step 4: Delete API Gateway API

4.1 Navigate to API Gateway

1. AWS Console → Search “API Gateway” → Click API Gateway

4.2 Delete the API

1. Find media-processing-api
2. Select it (checkbox)
3. Click Actions → Delete
4. Type the API name to confirm: media-processing-api
5. Click Delete

4.3 Verify

• API should no longer appear in the APIs list

Step 5: Empty S3 Buckets

Critical: S3 buckets MUST be empty before they can be deleted

5.1 Empty Uploads Bucket

1. Navigate to S3:

   • AWS Console → Search “S3” → Click S3

2. Empty uploads bucket:

   • Find amodhbh-media-uploads
   • Select it (checkbox, NOT click on name)
   • Click Empty
   • Type permanently delete to confirm
   • Click Empty
   • Wait for “Successfully emptied” message
   • Click Exit

5.2 Empty Processed Bucket

1. Empty processed bucket:
   • Find amodhbh-media-processed
   • Select it (checkbox)
   • Click Empty
   • Type permanently delete to confirm
   • Click Empty
   • Wait for completion
   • Click Exit

5.3 Verify

• Both buckets should show “0 objects”

Step 6: Delete S3 Buckets

Now that buckets are empty, we can delete them.

6.1 Delete Uploads Bucket

1. In S3 console:
   • Select amodhbh-media-uploads (checkbox)
   • Click Delete
   • Type the bucket name to confirm: amodhbh-media-uploads
   • Click Delete bucket

6.2 Delete Processed Bucket

1. Delete processed bucket:
   • Select amodhbh-media-processed (checkbox)
   • Click Delete
   • Type the bucket name to confirm: amodhbh-media-processed
   • Click Delete bucket

6.3 Verify

• Neither bucket should appear in your S3 buckets list

Step 7: Delete DynamoDB Table

7.1 Navigate to DynamoDB

1. AWS Console → Search “DynamoDB” → Click DynamoDB
2. Click Tables in left sidebar

7.2 Delete the Table

1. Select media-processing-jobs (checkbox)
2. Click Delete
3. Deselect “Create a backup of this table” (to avoid charges)
4. Type delete to confirm
5. Click Delete table

7.3 Verify

• Table status will show “Deleting” then disappear from the list
• This may take 1-2 minutes

Step 8: Delete SQS Queues

8.1 Delete Main Queue

1. Navigate to SQS:

   • AWS Console → Search “SQS” → Click Simple Queue Service

2. Delete processing queue:

   • Select media-processing-queue (checkbox)
   • Click Delete
   • Type delete to confirm
   • Click Delete

8.2 Delete Dead Letter Queue

1. Delete DLQ:
   • Select media-processing-dlq (checkbox)
   • Click Delete
   • Type delete to confirm
   • Click Delete

8.3 Verify

• Both queues should no longer appear in the queues list

Step 9: Delete CloudWatch Log Groups

Lambda functions automatically create log groups in CloudWatch. These can accumulate charges over time.

9.1 Navigate to CloudWatch

1. AWS Console → Search “CloudWatch” → Click CloudWatch
2. Left sidebar → Logs → Log groups

9.2 Delete Log Groups

1. Delete log groups for each Lambda function:

   • Find and delete these log groups (one at a time):
     • /aws/lambda/worker-lambda
     • /aws/lambda/dispatcher-lambda
     • /aws/lambda/request-upload-lambda
     • /aws/lambda/get-job-status-lambda

2. For each log group:

   • Select it (checkbox)
   • Click Actions → Delete log group(s)
   • Click Delete

9.3 Verify

• None of these log groups should remain in the list

Step 10: Delete Lambda Layers

10.1 Navigate to Lambda

1. AWS Console → Search “Lambda” → Click Lambda
2. Left sidebar → Layers

10.2 Delete Pillow Layer

1. Click on pillow-layer
2. For each version listed:
   • Click the version number
   • Click Delete (if available)

Note: If delete is not available, this is fine. Layers with no associated functions don’t incur charges.

Step 11: Delete IAM Roles and Policies

Why last: Ensures no resources are still using these roles

11.1 Delete Worker Lambda Role

1. Navigate to IAM:

   • AWS Console → Search “IAM” → Click IAM
   • Left sidebar → Roles

2. Delete media-worker-lambda-role:

   • Search for media-worker-lambda-role
   • Select it (checkbox)
   • Click Delete
   • Type the role name to confirm
   • Click Delete

11.2 Delete API Lambda Role

1. Delete media-api-lambda-role:
   • Search for media-api-lambda-role
   • Select it (checkbox)
   • Click Delete
   • Type the role name to confirm
   • Click Delete

11.3 Delete Dispatcher Lambda Role

1. Delete dispatcher-lambda-role:
   • Search for dispatcher-lambda-role
   • Select it (checkbox)
   • Click Delete
   • Type the role name to confirm
   • Click Delete

11.4 Verify

• None of these roles should appear in your IAM roles list:
  • media-worker-lambda-role
  • media-api-lambda-role
  • dispatcher-lambda-role

Cost Verification

Step 1: Final Verification

1.1 Resource Checklist

Go through each service and verify no project resources remain:

S3:

• No amodhbh-media-uploads bucket
• No amodhbh-media-processed bucket

Lambda:

• No worker-lambda function
• No dispatcher-lambda function
• No request-upload-lambda function
• No get-job-status-lambda function
• pillow-layer layer deleted or orphaned (no charges)

API Gateway:

• No media-processing-api

DynamoDB:

• No media-processing-jobs table

SQS:

• No media-processing-queue
• No media-processing-dlq

CloudWatch Logs:

• No Lambda log groups for project functions

IAM:

• No media-worker-lambda-role
• No media-api-lambda-role
• No dispatcher-lambda-role

Step 2: Check AWS Billing

It’s important to verify that cleanup was successful by checking your AWS bill.

2.1 View Current Charges

1. Navigate to Billing:

   • Click your account name (top-right)
   • Click Billing and Cost Management

2. Check costs:

   • Click Bills in left sidebar
   • View current month charges
   • Look for these services (should be $0 or minimal):
     • Amazon S3
     • AWS Lambda
     • Amazon DynamoDB
     • Amazon API Gateway
     • Amazon SQS

2.2 Set Up Billing Alerts (Recommended)

To prevent unexpected charges in the future:

1. In Billing Console:

   • Left sidebar → Billing preferences
   • Enable Receive Billing Alerts
   • Save preferences

2. Create CloudWatch Alarm:

   • Go to CloudWatch console
   • Left sidebar → Alarms → All alarms
   • Click Create alarm
   • Select metric → Billing → Total Estimated Charge
   • Set threshold (e.g., $5)
   • Configure email notification
   • Create alarm

Step 3: Cost Verification Timeline

After completing cleanup:

• Immediate: Most resources stop incurring charges
• 24 hours: Check billing to ensure no new charges
• 7 days: Verify final bill for the month
• 30 days: Confirm no recurring charges on next month’s bill

Emergency Procedures

Emergency: Stop All Charges Immediately

If you’re seeing unexpected high charges:

1. Delete all S3 buckets immediately (empty them first)
2. Delete all Lambda functions
3. Delete API Gateway APIs
4. Purge and delete all SQS queues
5. Delete DynamoDB tables
6. Open a support ticket in AWS Console explaining the situation
7. Contact AWS Billing Support (available even on free tier)

Emergency Cleanup Script

For rapid cleanup, you can use AWS CLI (if configured):

# Delete Lambda functions
aws lambda delete-function --function-name worker-lambda
aws lambda delete-function --function-name dispatcher-lambda
aws lambda delete-function --function-name request-upload-lambda
aws lambda delete-function --function-name get-job-status-lambda

# Delete S3 buckets (must be empty first)
aws s3 rb s3://amodhbh-media-uploads --force
aws s3 rb s3://amodhbh-media-processed --force

# Delete DynamoDB table
aws dynamodb delete-table --table-name media-processing-jobs

# Delete SQS queues
aws sqs delete-queue --queue-url https://sqs.ap-south-1.amazonaws.com/YOUR_ACCOUNT/media-processing-queue
aws sqs delete-queue --queue-url https://sqs.ap-south-1.amazonaws.com/YOUR_ACCOUNT/media-processing-dlq

Best Practices

1. Resource Tagging

Always tag your resources for easy identification:

{
  "Project": "serverless-media",
  "Environment": "production",
  "Owner": "your-email@example.com",
  "CostCenter": "engineering"
}

2. Cost Monitoring

Set up comprehensive cost monitoring:

1. Billing Alerts: Multiple thresholds ($5, $25, $50)
2. Cost Explorer: Regular analysis of spending patterns
3. Resource Groups: Organize resources by project
4. CloudWatch Dashboards: Visual cost monitoring

3. Cleanup Automation

Consider using Infrastructure as Code (IaC) for easier cleanup:

• AWS CDK: TypeScript/Python infrastructure
• Terraform: Multi-cloud infrastructure
• CloudFormation: AWS-native templates

4. Resource Lifecycle Management

Implement lifecycle policies:

• S3 Lifecycle: Automatic deletion of old files
• DynamoDB TTL: Automatic cleanup of expired items
• Lambda Logs: Automatic log retention policies

Troubleshooting

Common Issues and Solutions

Cannot delete S3 bucket - “Bucket not empty”

Solution:

• Go back to Step 5 and empty the bucket again
• Check if versioning is enabled (if so, you need to delete all versions)
• In bucket → Properties → Bucket Versioning
• If enabled, go to bucket, toggle “Show versions”, select all, delete

Cannot delete SQS queue - timeout error

Solution:

• Wait 60 seconds after purging before attempting deletion
• Refresh the page and try again

Cannot delete IAM role - “Role is being used”

Solution:

• A resource is still using this role
• Go back and verify all Lambda functions are deleted
• Wait 5 minutes for AWS to propagate the deletion
• Try deleting the role again

Cannot delete Lambda function - “Function is being invoked”

Solution:

• Ensure S3 event notifications are deleted (Step 1)
• Wait a few minutes for in-flight executions to complete
• Try deletion again

CloudWatch Log Groups keep appearing

Solution:

• Log groups are recreated if Lambda functions are invoked
• Ensure Lambda functions are deleted first
• Delete log groups after Lambda deletion
• If they reappear, you may have another Lambda with the same name

Advanced Troubleshooting

Check for Hidden Resources

1. Check all regions:

   • Change region in AWS Console
   • Look for resources in other regions

2. Check CloudFormation stacks:

   • Look for any stacks that might have created resources
   • Delete stacks if found

3. Check AWS Config:

   • Review configuration history for any missed resources

Resource Dependencies

Some resources have hidden dependencies:

1. Lambda Layers: May be referenced by other functions
2. IAM Policies: May be attached to multiple roles
3. S3 Bucket Policies: May have complex access rules
4. VPC Endpoints: May be used by other services

Additional Cleanup (Optional but Recommended)

Delete CloudShell Files

If you used CloudShell to create the Pillow layer:

1. Open CloudShell
2. Run: rm -rf pillow-layer/
3. Close CloudShell

Check Other Regions

If you accidentally created resources in another region:

1. In AWS Console, change region (top-right dropdown)
2. Check each service in other regions
3. Delete any project resources found

Review Cost Explorer

1. Go to Cost Explorer in Billing Console
2. View costs by service for the past month
3. Identify any unexpected charges
4. Investigate and clean up if needed

What If I Want to Rebuild?

If you want to recreate this project later:

1. All the Phase guides (1-4) contain complete instructions
2. You can reuse the same bucket names (if globally available)
3. All code is preserved in the markdown files
4. The entire setup takes approximately 2-3 hours

Quick Rebuild Checklist

• Review Part 1: Infrastructure setup
• Review Part 2: API and pipeline completion
• Update bucket names if needed
• Follow the step-by-step guides
• Test thoroughly before production use

Summary

After completing this cleanup guide, you should have:

✅ Deleted all project-related AWS resources
✅ Verified $0 (or minimal) ongoing charges
✅ Learned the correct order for resource deletion
✅ Set up billing alerts to prevent future surprises

Estimated time to complete cleanup: 15-20 minutes

Final Checklist

Before considering cleanup complete:

• All Lambda functions deleted
• All S3 buckets deleted (after emptying)
• API Gateway API deleted
• DynamoDB table deleted
• SQS queues deleted
• CloudWatch log groups deleted
• IAM roles deleted
• Billing shows $0 or minimal charges
• No error messages during deletion
• Verified in AWS Console that all resources are gone

Questions?

If you encounter issues during cleanup:

1. Check the troubleshooting section above
2. Review AWS documentation for specific services
3. Use AWS re:Post (AWS community forum) for technical questions
4. Contact AWS Support if charges continue after cleanup

Cleanup Complete! 🎉

You’ve successfully removed all resources from your serverless media processing pipeline. Your AWS account is now clean, and you won’t incur ongoing charges for this project.

If you learned valuable skills from this project, consider building more serverless architectures - they’re cost-effective, scalable, and represent the future of cloud computing!

Resource Summary Table

Active costs based on 1000 operations/month estimate

This is Part 3 of a 4-part series on building a production-ready serverless media processing pipeline. You’ve now completed the entire series and have the knowledge to build, deploy, and clean up serverless architectures on AWS!