S3 Uploads Slow Over Time and Eventually Timeout with payloadcms/storage-s3

[prxssh]

Describe the Bug

We’re running Payload CMS in a Kubernetes cluster and using the payloadcms/storage-s3 adapter to handle media uploads. We recently upgraded from v3.0.0-beta.106 to v3.37.0, and since then, on pod startup uploads to S3 are fast and reliable—but after several hours of continuous use, upload latency increases dramatically and requests begin failing with 504 Gateway Timeout. Restarting the pod immediately restores normal performance.

This is the storage-s3 config that we're using:

 s3Storage({
      collections: {
        media: {
          prefix: 'cms/media/',
        },
      },
      bucket: process.env.S3_BUCKET || '',
      config: {
        credentials: {
          accessKeyId: process.env.S3_ACCESS_KEY || '',
          secretAccessKey: process.env.S3_SECRET || '',
        },
        forcePathStyle: true,
        region: process.env.S3_REGION || '',
      },
    })

Link to the code that reproduces this issue

N.A

Reproduction Steps

I was unable to reproduce the issue using a minimal template project. However, I’m attaching two videos: the first shows normal performance immediately after the server starts, and the second shows the performance regression over time.

1. Payload Working Fine
2. Payload Media Uploads/Downloads Failing

Which area(s) are affected? (Select all that apply)

plugin: cloud-storage, Not sure

Environment Info

Binaries:
  Node: 20.18.1
  npm: 10.8.2
  Yarn: 1.22.22
  pnpm: N/A
Relevant Packages:
  payload: 3.37.0
  next: 15.3.1
  @payloadcms/db-mongodb: 3.37.0
  @payloadcms/graphql: 3.37.0
  @payloadcms/next/utilities: 3.37.0
  @payloadcms/plugin-cloud-storage: 3.37.0
  @payloadcms/plugin-seo: 3.37.0
  @payloadcms/richtext-lexical: 3.37.0
  @payloadcms/richtext-slate: 3.37.0
  @payloadcms/storage-s3: 3.37.0
  @payloadcms/translations: 3.37.0
  @payloadcms/ui/shared: 3.37.0
  react: 19.1.0
  react-dom: 19.1.0
Operating System:
  Platform: linux
  Arch: arm64
  Version: #1 SMP Tue Feb 11 11:41:18 UTC 2025
  Available memory (MB): 3817
  Available CPU cores: 2

[github-actions]

Please add a reproduction in order for us to be able to investigate.

Depending on the quality of reproduction steps, this issue may be closed if no reproduction is provided.

Why was this issue marked with the invalid-reproduction label?

To be able to investigate, we need access to a reproduction to identify what triggered the issue. We prefer a link to a public GitHub repository created with create-payload-app@latest -t blank or a forked/branched version of this repository with tests added (more info in the reproduction-guide).

To make sure the issue is resolved as quickly as possible, please make sure that the reproduction is as minimal as possible. This means that you should remove unnecessary code, files, and dependencies that do not contribute to the issue. Ensure your reproduction does not depend on secrets, 3rd party registries, private dependencies, or any other data that cannot be made public. Avoid a reproduction including a whole monorepo (unless relevant to the issue). The easier it is to reproduce the issue, the quicker we can help.

Please test your reproduction against the latest version of Payload to make sure your issue has not already been fixed.

I added a link, why was it still marked?

Ensure the link is pointing to a codebase that is accessible (e.g. not a private repository). "example.com", "n/a", "will add later", etc. are not acceptable links -- we need to see a public codebase. See the above section for accepted links.

Useful Resources

• Reproduction Guide
• Contributing to Payload

[achung98]

I am facing the same issue. My s3 requests timeout but paylod still runs as expected.
Did you find a solution for this ?

[prxssh]

@achung98 unfortunately, no! We had to revert to older version.

[redmundas]

@achung98 unfortunately, no! We had to revert to older version.

having similar problem with v3.42.0. @prxssh which version did you revert to?

[utvara]

We are experiencing the same issue with v3.40.0. Timeout on the media routes. It doesn't happen right away. I'm still trying to figure out the conditions that trigger the issue. Two things that I could add to the conversation (might not be relevant):

• we are using minio S3 server
• we have the s3plugin setup multiple times for different buckets

	s3Storage({
			collections: {
				media: true,
			},
			config: getS3ClientConfig(),
			bucket: config.S3_BUCKET_MEDIA,
		}),
		s3Storage({
			collections: {
				'client-documents': true,
			},
			config: getS3ClientConfig(),
			bucket: config.S3_BUCKET_CLIENT_DOCUMENTS,
		}),

Only remedy have found so far was to restart the payload server. I can also confirm that we do not see any increase in memory or CPU usage.

[redmundas]

I was experimenting with it, and it looks like some config options helped. here's our config:

s3Storage({
    enabled: useObjectStorage,
    disableLocalStorage: useObjectStorage,
    collections: {
      media: true,
    },
    bucket: process.env.S3_BUCKET,
    config: {
      credentials: {
        accessKeyId: process.env.S3_ACCESS_KEY_ID,
        secretAccessKey: process.env.S3_SECRET_ACCESS_KEY,
      },
      endpoint: process.env.S3_ENDPOINT,
      region: process.env.S3_REGION,
      maxAttempts: 3,
    },
    signedDownloads: true,
  }),

what specifically I changed is I've added maxAttempts: 3 and signedDownloads: true

[bwcroft]

I'm experiencing the same issue. Restarting the server temporarily resolves the problem, but over time, as the system is used, performance begins to degrade. The first symptoms is usually failed image uploads and downloads, caused by timeout errors. In some cases, the CMS completely locks up. Interestingly, CPU and Memory Usage remain within normal ranges, so it doesn’t appear to be a resource exhaustion issue.

[prxssh]

@bwcroft yes, we also observed the same. CPU and Memory Usage remains normal but everything else starts failing.

[velezhanski]

try this solution from @3k3m1n1, helped for our project

import { s3Adapter } from '@payloadcms/plugin-cloud-storage/s3'

const s3 = s3Adapter({
  config: {
    requestHandler: {
      httpAgent: {
        maxSockets: 300,
        keepAlive: true,
      },
      httpsAgent: {
        maxSockets: 300,
        keepAlive: true,
      },
      connectionTimeout: 5 * 1000,
      requestTimeout: 5 * 1000,
    },
    // ... set up credentials, region, bucket as usual
  },
}

// ... pass s3 to the cloudStorage plugin in buildConfig()

[bwcroft]

@prxssh

After investigating the issue, I’ve identified a significant performance bottleneck in the @payloadcms/plugin-storage-s3 package. Specifically, in the staticHandler function, the plugin reads the S3 object stream and converts it entirely into a buffer before returning it in the response.

This approach introduces a major scalability issue: for each request, the entire file is loaded into memory, which quickly becomes problematic under moderate to high load. As concurrent requests increase—especially for large media files—memory usage can spike sharply, leading to potential Node.js event loop blocking and degraded system performance.

Additionally, if a client disconnects before the stream resolves, the buffer operation may still complete, leaving a dangling operation in memory and potentially contributing to memory leaks over time.

A recommended workaround is to avoid proxying media downloads through the server entirely. Instead, use signed URLs that allow clients to download files directly from your S3 bucket. Alternatively, you can configure a CDN (such as CloudFront) to serve the assets directly, bypassing the API route (/api/media/file) and offloading the memory-intensive work from your Node.js server.

We’ve updated our system to use signed S3 URLs for asset delivery. Since making this change, our memory utilization has decreased significantly, and we’ve not experienced any further server lockups.

---

The proposed fix is to stream the S3 object directly to the response using pipe(), rather than buffering the entire file in memory. This not only reduces memory usage significantly under load but also ensures proper cleanup if the client disconnects before the stream completes.

[prxssh]

As the comments above have already suggested, minor tweaks in the S3 config does work. What fixed the issue for me was directly serving the media from S3 instead of proxying through the server. Thank you @bwcroft @redmundas for your suggestions.

[rezaarkan]

@prxssh can you give some pointers on how you're serving directly from S3?