[WIP] Fix pre-signed URL bug by implementing STS credentials

[el-feo]

🚧 Work In Progress - Do Not Merge

This PR implements AWS STS temporary credentials to fix a fundamental bug in the current pre-signed URL approach.

The Problem

The current implementation has a critical bug:

# Current code in image_uploader.rb (line 191)
image_uri.path = "#{base_uri.path}page-#{index + 1}.png"

This modifies the path of a pre-signed PUT URL, which invalidates the signature because:

• Pre-signed URLs include the exact object key in signature calculation
• Changing the path after signing makes the signature invalid
• The service cannot work with pre-signed destination URLs

The Solution: STS Credentials

Instead of pre-signed URLs, clients now:

1. Assume an IAM role via STS to get temporary credentials (15-min expiration)
2. Pass credentials + bucket/key to Lambda
3. Lambda uses credentials to access S3 directly

Benefits

✅ Fixes the signature invalidation bug
✅ Works for any number of pages (no pre-signing each file)
✅ Client maintains access control (they scope IAM permissions)
✅ Time-limited credentials (15 minutes)
✅ Preflight validation ensures credentials work before processing

---

What's Implemented ✅

Phase 1: IAM Infrastructure

• scripts/setup_iam_role.rb - Creates IAM role with proper trust policy
• Role uses ExternalId for security
• Scoped S3 permissions (GetObject source, PutObject destination)

Phase 2: Request Validation

• New request format: source{bucket,key}, destination{bucket,prefix}, credentials{}
• Validates S3 bucket names (AWS naming rules)
• Validates S3 keys (length, .pdf extension)
• Validates credential format (ASIA/AKIA prefix)

Phase 3: Core Components (Partial)

• CredentialSanitizer - Prevents credential leakage in logs
• PdfDownloader refactored to use S3 SDK
  • Preflight check with HEAD object (validates access before download)
  • File size validation (max 100MB)
  • PDF magic number validation
  • Detailed error messages for permission issues

---

Still TODO 🚧

Phase 3 Completion

• Refactor ImageUploader to use S3 SDK (upload_images_to_s3 method)
• Update app.rb main handler to wire new components
• Update Gemfile to add aws-sdk-s3 gem

Phase 4: Testing Scripts

• Create scripts/generate_sts_credentials.rb
• Update existing test scripts for new format
• Remove old pre-signed URL scripts

Phase 5: Documentation

• Update README.md API specification
• Update CLAUDE.md
• Create STS setup guide
• Update Getting Started instructions

Phase 6: Testing

• Update unit tests for new request format
• Update RequestValidator tests
• Add PdfDownloader S3 tests
• Add ImageUploader S3 tests
• Integration tests with LocalStack/real AWS
• Test credential validation
• Test preflight checks

Phase 7: Cleanup

• Remove old URL-based code paths
• Remove url_validator.rb (no longer needed)
• Remove retry_handler.rb (S3 SDK has built-in retries)
• Update webhook response to use S3 URLs

---

New API Request Format

{
  "source": {
    "bucket": "my-input-bucket",
    "key": "pdfs/document.pdf"
  },
  "destination": {
    "bucket": "my-output-bucket",  
    "prefix": "converted/project-123/"
  },
  "credentials": {
    "accessKeyId": "ASIAIOSFODNN7EXAMPLE",
    "secretAccessKey": "wJalrXUtnFEMI/K7MDENG/...",
    "sessionToken": "FQoGZXIvYXdzEPT//////////..."
  },
  "unique_id": "client-123",
  "webhook": "https://example.com/webhook"
}

Client Workflow

1. Setup (one-time):

   ./scripts/setup_iam_role.rb \
     --source-bucket my-pdfs \
     --dest-bucket my-images

2. Each request:

   # Generate temporary credentials
   ./scripts/generate_sts_credentials.rb \
     --role-arn arn:aws:iam::123456789012:role/PdfConverterClientRole \
     --source-bucket my-pdfs \
     --dest-bucket my-images
   
   # Use credentials in API request
   curl -X POST https://api.../convert \
     -H "Authorization: Bearer $JWT_TOKEN" \
     -d @request.json

---

Security Enhancements

• 15-minute credential expiration enforced
• Preflight validation - credentials tested before processing
• Credential sanitization - all logs redact secrets
• ExternalId required - prevents confused deputy problem
• Session policies - clients can further scope permissions

---

Implementation Plan

See full plan: .agent-os/specs/2025-11-11-sts-credentials/plan.md

Estimated remaining time: 10-12 hours

• Phase 3 completion: 2 hours
• Phase 4 scripts: 2 hours
• Phase 5 docs: 3 hours
• Phase 6 testing: 4 hours
• Phase 7 cleanup: 1 hour

---

Review Checklist

Before marking ready for review:

• All TODO items completed
• Unit tests pass
• Integration tests pass
• Documentation updated
• Old code removed
• Security review (credential handling)
• Tested end-to-end with real AWS

🤖 Generated with Claude Code

[el-feo]

Closing for a different approach.