Skip to content

Conversation

@benbz
Copy link
Member

@benbz benbz commented Nov 28, 2025

ESS CI is sometimes seeing issues where MAS (with multiple replicas) refuses to initially startup

The Postgres logs show

2025-11-28 14:42:46.989 UTC [127] ERROR:  deadlock detected
2025-11-28 14:42:46.989 UTC [127] DETAIL:  Process 127 waits for ShareLock on virtual transaction 19/5; blocked by process 128.
	Process 128 waits for ExclusiveLock on advisory lock [16385,438033630,3765090638,1]; blocked by process 127.
	Process 127: -- no-transaction
	-- Copyright 2025 New Vector Ltd.
	--
	-- SPDX-License-Identifier: AGPL-3.0-only
	-- Please see LICENSE in the repository root for full details.
	
	CREATE INDEX CONCURRENTLY
	  compat_access_tokens_session_fk
	  ON compat_access_tokens (compat_session_id);
	
	Process 128: SELECT pg_advisory_lock($1)

DB migration jobs then fail with relation "compat_access_tokens_session_fk" already exists.

On discussion this is the first migration that isn't in a transaction / uses CREATE INDEX CONCURRENTLY. It is assumed that removing CONCURRENTLY isn't desirable as will cause slowness on large MAS DBs upgrading, so use IF NOT EXISTS to subsequent migration attempts to progress. Unclear whether this will stop the deadlock in the first place but this shouldn't make the problem any worse

@benbz benbz requested a review from a team as a code owner November 28, 2025 15:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants