Skip to content

fix: make empty bucket use queue to remove underlying objects asynchronously #701

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jul 22, 2025

Conversation

itslenny
Copy link
Contributor

@itslenny itslenny commented May 30, 2025

What kind of change does this PR introduce?

Bug fix

What is the current behavior?

When a bucket is emptied it sends DeleteObjectsCommand to S3 in a tight loop resulting in rate limit errors and ultimately orphan objects.

What is the new behavior?

Use the queue to delete objects from the underlying object store asynchronously

Additional context

This adds ObjectDeleteAllBeforeEvent which recursively removes all objects before a specified time in batches using storage.backend.deleteObjects(...) within a transaction so the database objects will be rolled back if the S3 operation fails.

Each job run removes as many batches as it can in 10 seconds and then re-queues if there are remaining objects to delete.

@itslenny itslenny marked this pull request as ready for review May 30, 2025 18:06
@itslenny itslenny force-pushed the fix/use-queue-in-empty-to-delete-objects branch from 1904a35 to 1d91784 Compare May 30, 2025 18:13
@coveralls
Copy link

coveralls commented May 30, 2025

Pull Request Test Coverage Report for Build 16450960387

Details

  • 174 of 221 (78.73%) changed or added relevant lines in 9 files are covered.
  • 1 unchanged line in 1 file lost coverage.
  • Overall coverage increased (+0.02%) to 76.953%

Changes Missing Coverage Covered Lines Changed/Added Lines %
src/storage/database/knex.ts 32 33 96.97%
src/storage/events/workers.ts 2 3 66.67%
src/storage/storage.ts 23 25 92.0%
src/internal/errors/codes.ts 2 7 28.57%
src/storage/events/objects/object-admin-delete-all-before.ts 103 141 73.05%
Files with Coverage Reduction New Missed Lines %
src/internal/errors/codes.ts 1 72.02%
Totals Coverage Status
Change from base Build 16426721998: 0.02%
Covered Lines: 20551
Relevant Lines: 26475

💛 - Coveralls

@itslenny itslenny force-pushed the fix/use-queue-in-empty-to-delete-objects branch from 3c757e7 to d4c173d Compare June 5, 2025 15:55
@itslenny itslenny force-pushed the fix/use-queue-in-empty-to-delete-objects branch 2 times, most recently from 987481f to 7e303c8 Compare June 24, 2025 16:26
@itslenny itslenny force-pushed the fix/use-queue-in-empty-to-delete-objects branch from 7e303c8 to bd49271 Compare July 22, 2025 16:45
@itslenny itslenny force-pushed the fix/use-queue-in-empty-to-delete-objects branch from bd49271 to 25fda8f Compare July 22, 2025 17:11
@itslenny itslenny merged commit 332a2e1 into master Jul 22, 2025
2 checks passed
@itslenny itslenny deleted the fix/use-queue-in-empty-to-delete-objects branch July 22, 2025 18:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants