Skip to content

Conversation

@patsevanton
Copy link
Contributor

@patsevanton patsevanton commented Nov 14, 2025

This commit updates the Helm chart's appVersion from 25.9.0 to 25.10.0. The changes in this release have been reviewed by comparing the differences between versions 25.9.0 and 25.10.0 in the following key files:

Get error #1937

@patsevanton patsevanton marked this pull request as draft November 14, 2025 14:37
@patsevanton
Copy link
Contributor Author

fyi @aldy505 @blunt1973

@patsevanton
Copy link
Contributor Author

k logs sentry-worker-6b664ff4cc-kxmlp -n test
14:56:52 [INFO] sentry.taskworker.client: taskworker.client.start (hosts=['127.0.0.1:50051'] options=[('grpc.max_receive_message_length', 10485760), ('grpc.service_config', '{"loadBalancingConfig": [{"round_robin": {}}]}')])
14:56:52 [INFO] sentry.taskworker.client: taskworker.client.connect (host='127.0.0.1:50051')
14:56:52 [INFO] sentry.taskworker.worker: taskworker.fetch_task.failed (error=<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "connections to all backends failing; last error: UNKNOWN: ipv4:127.0.0.1:50051: Failed to connect to remote host: connect: Connection refused (111)"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"connections to all backends failing; last error: UNKNOWN: ipv4:127.0.0.1:50051: Failed to connect to remote host: connect: Connection refused (111)", grpc_status:14, created_time:"2025-11-14T14:56:52.934628927+00:00"}"
> processing_pool='unknown')
14:56:52 [INFO] sentry.taskworker.worker: taskworker.spawn_child (pid=19 processing_pool='unknown')
14:56:56 [INFO] sentry.taskworker.worker: taskworker.fetch_task.failed (error=<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "connections to all backends failing; last error: UNKNOWN: ipv4:127.0.0.1:50051: Failed to connect to remote host: connect: Connection refused (111)"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"connections to all backends failing; last error: UNKNOWN: ipv4:127.0.0.1:50051: Failed to connect to remote host: connect: Connection refused (111)", grpc_status:14, created_time:"2025-11-14T14:56:56.936255643+00:00"}"
> processing_pool='unknown')
14:57:04 [INFO] sentry.taskworker.worker: taskworker.fetch_task.failed (error=<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "connections to all backends failing; last error: UNKNOWN: ipv4:127.0.0.1:50051: Failed to connect to remote host: connect: Connection refused (111)"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"connections to all backends failing; last error: UNKNOWN: ipv4:127.0.0.1:50051: Failed to connect to remote host: connect: Connection refused (111)", grpc_status:14, created_time:"2025-11-14T14:57:04.937282121+00:00"}"
> processing_pool='unknown')
14:57:16 [INFO] sentry.taskworker.client: taskworker.client.no_available_hosts (sleeping for=7.998152017593384)
14:57:24 [INFO] sentry.taskworker.worker: taskworker.fetch_task.failed (error=<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "connections to all backends failing; last error: UNKNOWN: ipv4:127.0.0.1:50051: Failed to connect to remote host: connect: Connection refused (111)"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"connections to all backends failing; last error: UNKNOWN: ipv4:127.0.0.1:50051: Failed to connect to remote host: connect: Connection refused (111)", grpc_status:14, created_time:"2025-11-14T14:57:24.939765365+00:00"}"
> processing_pool='unknown')
14:57:40 [INFO] sentry.taskworker.client: taskworker.client.no_available_hosts (sleeping for=3.9923126697540283)
14:57:44 [INFO] sentry.taskworker.worker: taskworker.fetch_task.failed (error=<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "connections to all backends failing; last error: UNKNOWN: ipv4:127.0.0.1:50051: Failed to connect to remote host: connect: Connection refused (111)"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"connections to all backends failing; last error: UNKNOWN: ipv4:127.0.0.1:50051: Failed to connect to remote host: connect: Connection refused (111)", grpc_status:14, created_time:"2025-11-14T14:57:44.942379951+00:00"}"
> processing_pool='unknown')
14:58:04 [INFO] sentry.taskworker.worker: taskworker.fetch_task.failed (error=<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "connections to all backends failing; last error: UNKNOWN: ipv4:127.0.0.1:50051: Failed to connect to remote host: connect: Connection refused (111)"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"connections to all backends failing; last error: UNKNOWN: ipv4:127.0.0.1:50051: Failed to connect to remote host: connect: Connection refused (111)", grpc_status:14, created_time:"2025-11-14T14:58:04.945210704+00:00"}"
> processing_pool='unknown')
14:58:24 [INFO] sentry.taskworker.worker: taskworker.fetch_task.failed (error=<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "connections to all backends failing; last error: UNKNOWN: ipv4:127.0.0.1:50051: Failed to connect to remote host: connect: Connection refused (111)"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"connections to all backends failing; last error: UNKNOWN: ipv4:127.0.0.1:50051: Failed to connect to remote host: connect: Connection refused (111)", grpc_status:14, created_time:"2025-11-14T14:58:24.947473627+00:00"}"
> processing_pool='unknown')
14:58:44 [INFO] sentry.taskworker.worker: taskworker.fetch_task.failed (error=<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "connections to all backends failing; last error: UNKNOWN: ipv4:127.0.0.1:50051: Failed to connect to remote host: connect: Connection refused (111)"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2025-11-14T14:58:44.949717805+00:00", grpc_status:14, grpc_message:"connections to all backends failing; last error: UNKNOWN: ipv4:127.0.0.1:50051: Failed to connect to remote host: connect: Connection refused (111)"}"
> processing_pool='unknown')
14:59:04 [INFO] sentry.taskworker.worker: taskworker.fetch_task.failed (error=<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "connections to all backends failing; last error: UNKNOWN: ipv4:127.0.0.1:50051: Failed to connect to remote host: connect: Connection refused (111)"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2025-11-14T14:59:04.952236575+00:00", grpc_status:14, grpc_message:"connections to all backends failing; last error: UNKNOWN: ipv4:127.0.0.1:50051: Failed to connect to remote host: connect: Connection refused (111)"}"
> processing_pool='unknown')
14:59:24 [INFO] sentry.taskworker.worker: taskworker.fetch_task.failed (error=<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "connections to all backends failing; last error: UNKNOWN: ipv4:127.0.0.1:50051: Failed to connect to remote host: connect: Connection refused (111)"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2025-11-14T14:59:24.954616134+00:00", grpc_status:14, grpc_message:"connections to all backends failing; last error: UNKNOWN: ipv4:127.0.0.1:50051: Failed to connect to remote host: connect: Connection refused (111)"}"
> processing_pool='unknown')
14:59:44 [INFO] sentry.taskworker.worker: taskworker.fetch_task.failed (error=<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "connections to all backends failing; last error: UNKNOWN: ipv4:127.0.0.1:50051: Failed to connect to remote host: connect: Connection refused (111)"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2025-11-14T14:59:44.957058575+00:00", grpc_status:14, grpc_message:"connections to all backends failing; last error: UNKNOWN: ipv4:127.0.0.1:50051: Failed to connect to remote host: connect: Connection refused (111)"}"
> processing_pool='unknown')
14:59:45 [INFO] sentry.taskworker.worker: taskworker.worker.shutdown.start
14:59:45 [INFO] sentry.taskworker.worker: taskworker.worker.shutdown.spawn_children
14:59:45 [INFO] sentry.taskworker.worker: taskworker.worker.shutdown.children
14:59:45 [INFO] sentry.taskworker.worker: taskworker.worker.shutdown.result
14:59:45 [INFO] sentry.taskworker.worker: taskworker.worker.shutdown.complete

Aborted!

@patsevanton
Copy link
Contributor Author

patsevanton commented Nov 14, 2025

k describe pod sentry-worker-6b664ff4cc-kxmlp -n test

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Warning  Unhealthy  7m54s (x19 over 30m)  kubelet            Liveness probe failed: Traceback (most recent call last):
  File "<string>", line 7, in <module>
ModuleNotFoundError: No module named 'sentry.celery'
Traceback (most recent call last):
  File "<string>", line 7, in <module>
ModuleNotFoundError: No module named 'sentry.celery'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/.venv/bin/sentry", line 4, in <module>
    raise SystemExit(main())
                     ~~~~^^
  File "/usr/src/sentry/src/sentry/runner/main.py", line 145, in main
    func(**kwargs)
    ~~~~^^^^^^^^^^
  File "/.venv/lib/python3.13/site-packages/click/core.py", line 1442, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.13/site-packages/click/core.py", line 1363, in main
    rv = self.invoke(ctx)
  File "/.venv/lib/python3.13/site-packages/click/core.py", line 1830, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/.venv/lib/python3.13/site-packages/click/core.py", line 1226, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.13/site-packages/click/core.py", line 794, in invoke
    return callback(*args, **kwargs)
  File "/usr/src/sentry/src/sentry/runner/commands/exec.py", line 118, in exec_
    exec(compile(script, file, "exec"), g, g)
    ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 11, in <module>
ScriptError: Failed to execute script '<string>'

@aldy505
Copy link

aldy505 commented Nov 15, 2025

@patsevanton I sent you an email regarding some tips for performance, please take a look

@patsevanton
Copy link
Contributor Author

@aldy505 I still don't understand and don't know how to fix these errors.

@aldy505
Copy link

aldy505 commented Nov 16, 2025

@aldy505 I still don't understand and don't know how to fix these errors.

@patsevanton You set the wrong command for taskworker and taskworker-scheduler. Both are not drop-in replacement for sentry cron and sentry worker containers. Please refer to here: https://github.com/getsentry/self-hosted/blob/fe477b41d937b838146f83a48bbbe705ec959f49/docker-compose.yml#L739-L746

@patsevanton
Copy link
Contributor Author

@aldy505
how to do it right?

  1. adjust the settings for taksworker in the current yaml files
  2. refine the files from mr https://github.com/sentry-kubernetes/charts/pull/1831/files ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants