Implement EMF and Container Insights log cleanup (#147)

[subhobhai943]

Description

This PR addresses issue #147 by implementing comprehensive cleanup functionality for EMF (Embedded Metric Format) and Container Insights logs that persist after test execution.

Changes Made

1. New Log Cleanup Utility (util/awsservice/log_cleanup.go)

• Pattern-based identification: Automatically identifies EMF and Container Insights log groups using common patterns
• Flexible configuration: Supports custom include/exclude patterns, age constraints, and batch processing
• Safety features: Dry-run by default, production log exclusions, age-based protection
• Comprehensive logging: Detailed reporting of cleanup operations

2. Test Framework Integration

• BaseTestRunner: Added CleanupAfterTest() method to interface and default implementation
• ECSTestRunner: Specialized cleanup for ECS Container Insights logs
• EMFTestRunner: Targeted cleanup for EMF-specific log groups
• Graceful error handling: Cleanup failures don't cause test failures

3. Environment Variable Controls

• CWAGENT_SKIP_LOG_CLEANUP: Completely disable cleanup (useful for debugging)
• CWAGENT_FORCE_LOG_CLEANUP: Enable actual deletion (defaults to dry-run for safety)

4. Comprehensive Documentation (docs/log-cleanup.md)

• Usage instructions and best practices
• Configuration options and examples
• Troubleshooting guide
• Security considerations

Log Group Patterns Addressed

Container Insights:

• /aws/ecs/containerinsights/*/performance
• /aws/ecs/containerinsights/*/application
• /aws/eks/containerinsights/*/performance
• /aws/eks/containerinsights/*/application

EMF Logs:

• EMFECSNameSpace / EMFEKSNameSpace (from test configurations)
• *EMF* and *emf* patterns
• /aws/lambda/* (Lambda EMF logs)

ECS Task Logs:

• /ecs/* and /aws/ecs/*

Test-specific:

• *test*, cwagent-*, etc.

Safety Features

1. Default Dry-Run: All operations are dry-run by default to prevent accidental deletions
2. Age Protection: Only deletes logs older than 1-2 hours
3. Production Exclusions: Automatically excludes logs with "production" or "prod" patterns
4. Error Isolation: Cleanup errors don't fail tests
5. Detailed Logging: Comprehensive audit trail of all operations

Usage Examples

Default Behavior (Dry-Run)

# Shows what would be cleaned up without actually deleting
go test ./test/emf/

Enable Actual Cleanup

# Use in CI/CD pipelines to actually clean up logs
export CWAGENT_FORCE_LOG_CLEANUP=true
go test ./test/emf/

Disable for Debugging

# Keep logs for post-test analysis
export CWAGENT_SKIP_LOG_CLEANUP=true
go test ./test/emf/

Integration Points

• Automatic: Cleanup runs after every test via the test framework
• ECS-specific: ECS tests get specialized Container Insights cleanup
• EMF-specific: EMF tests clean up their specific namespaces
• Manual: Utility functions available for custom cleanup scenarios

Benefits

1. Cost Reduction: Prevents accumulation of test log groups
2. Account Cleanliness: Keeps AWS accounts tidy after testing
3. Automated: No manual intervention required
4. Safe: Multiple safety mechanisms prevent accidental deletions
5. Configurable: Environment variables provide flexible control

Backward Compatibility

• Non-breaking: All existing tests continue to work unchanged
• Opt-in: Actual cleanup requires explicit environment variable
• Interface addition: New CleanupAfterTest() method added to ITestRunner interface

Testing

The implementation has been tested with:

• ✅ Dry-run operations (safe default)
• ✅ Pattern matching for various log group types
• ✅ Environment variable controls
• ✅ Error handling and graceful degradation
• ✅ Integration with existing test framework

Files Changed

• util/awsservice/log_cleanup.go - New cleanup utility
• test/test_runner/base_test_runner.go - Interface and default cleanup
• test/test_runner/ecs_test_runner.go - ECS-specific cleanup integration
• test/emf/emf_container_test.go - EMF-specific cleanup implementation
• docs/log-cleanup.md - Comprehensive documentation

Resolves #147

---

This implementation provides a robust, safe, and automated solution for cleaning up EMF and Container Insights logs while maintaining backward compatibility and providing extensive safety measures.