Skip to content

Conversation

@osmaczko
Copy link
Contributor

@osmaczko osmaczko commented Oct 13, 2025

  • make each messaging submodule define its own persistence interface and
    SQLite implementation reference
  • separate migration tables for messaging submodules
  • move migrations from protocol to relevant messaging submodules

Explanation of how to move existing migrations to submodules: https://miro.com/app/board/uXjVJ3l1qdo=/?moveToWidget=3458764645112517555&cot=14

closes: #6792

@status-im-auto
Copy link
Member

status-im-auto commented Oct 13, 2025

Jenkins Builds

Click to see older builds (85)
Commit #️⃣ Finished (UTC) Duration Platform Result
✔️ 2a376aa #1 2025-10-13 17:54:35 ~4 min linux/status-go 📦zip
✖️ 2a376aa #1 2025-10-13 17:55:31 ~5 min tests 📄log
✔️ 2a376aa #1 2025-10-13 17:56:37 ~6 min windows/status-go 📦zip
✔️ 2a376aa #1 2025-10-13 17:57:19 ~7 min macos/status-go 📦zip
✔️ 2a376aa #1 2025-10-13 18:03:05 ~13 min linux/nwaku 📦zip
✖️ 2a376aa #1 2025-10-13 18:07:20 ~17 min tests-rpc 📄log
✔️ da7328e #2 2025-10-15 19:07:01 ~4 min linux/status-go 📦zip
✔️ da7328e #2 2025-10-15 19:09:03 ~6 min windows/status-go 📦zip
✖️ da7328e #2 2025-10-15 19:11:57 ~9 min tests 📄log
✔️ da7328e #2 2025-10-15 19:16:15 ~13 min linux/nwaku 📦zip
✔️ da7328e #2 2025-10-15 19:17:20 ~14 min macos/status-go 📦zip
✖️ da7328e #2 2025-10-15 19:22:15 ~19 min tests-rpc 📄log
✔️ b2ed375 #3 2025-10-16 07:52:24 ~7 min windows/status-go 📦zip
✔️ ad98019 #4 2025-10-16 08:01:06 ~9 min linux/status-go 📦zip
✔️ ad98019 #4 2025-10-16 08:01:09 ~8 min windows/status-go 📦zip
✖️ ad98019 #4 2025-10-16 08:07:25 ~15 min tests 📄log
✔️ ad98019 #4 2025-10-16 08:10:05 ~18 min linux/nwaku 📦zip
✔️ ad98019 #4 2025-10-16 08:11:26 ~19 min tests-rpc 📄log
3576e19 #5 2025-10-16 08:57:09 ~4 min windows/status-go 📄log
3576e19 #5 2025-10-16 09:03:42 ~11 min linux/status-go 📄log
✖️ 3576e19 #5 2025-10-16 09:07:09 ~14 min tests 📄log
✖️ 3576e19 #5 2025-10-16 09:09:03 ~16 min tests-rpc 📄log
3576e19 #5 2025-10-16 09:11:32 ~19 min linux/nwaku 📄log
✔️ 53eb2e7 #6 2025-10-16 09:38:06 ~4 min linux/status-go 📦zip
✔️ 53eb2e7 #6 2025-10-16 09:40:11 ~6 min windows/status-go 📦zip
✖️ 53eb2e7 #6 2025-10-16 09:41:02 ~7 min tests 📄log
✔️ 53eb2e7 #6 2025-10-16 09:46:21 ~12 min linux/nwaku 📦zip
✖️ 53eb2e7 #6 2025-10-16 09:48:00 ~14 min tests-rpc 📄log
✔️ 0ee8ad6 #7 2025-10-16 10:54:40 ~8 min windows/status-go 📦zip
✔️ 0ee8ad6 #7 2025-10-16 10:57:20 ~11 min linux/status-go 📦zip
✖️ 0ee8ad6 #7 2025-10-16 11:03:42 ~17 min tests 📄log
✔️ 0ee8ad6 #7 2025-10-16 11:06:23 ~20 min linux/nwaku 📦zip
✔️ 0ee8ad6 #7 2025-10-16 11:08:52 ~22 min tests-rpc 📄log
0ee8ad6 #7 2025-10-16 11:58:50 ~49 min macos/status-go 📄log
459e2ec #8 2025-10-16 12:32:00 ~2 min macos/status-go 📄log
✔️ 459e2ec #8 2025-10-16 12:35:22 ~5 min linux/status-go 📦zip
✔️ 459e2ec #8 2025-10-16 12:36:18 ~6 min windows/status-go 📦zip
✔️ 459e2ec #8 2025-10-16 12:44:09 ~14 min linux/nwaku 📦zip
✔️ 459e2ec #8 2025-10-16 12:48:58 ~18 min tests-rpc 📄log
✔️ 459e2ec #8 2025-10-16 13:08:07 ~38 min tests 📄log
459e2ec #9 2025-10-16 13:10:54 ~11 min macos/status-go 📄log
459e2ec #10 2025-10-16 13:50:48 ~1 min macos/status-go 📄log
✔️ 78d6e84 #9 2025-10-16 13:58:57 ~4 min linux/status-go 📦zip
✔️ 78d6e84 #9 2025-10-16 14:02:15 ~7 min windows/status-go 📦zip
✔️ 78d6e84 #11 2025-10-16 14:04:27 ~9 min macos/status-go 📦zip
✔️ 78d6e84 #9 2025-10-16 14:09:48 ~14 min linux/nwaku 📦zip
✖️ 78d6e84 #9 2025-10-16 14:10:50 ~15 min tests-rpc 📄log
✔️ 78d6e84 #9 2025-10-16 14:26:51 ~31 min tests 📄log
✔️ 78d6e84 #10 2025-10-16 18:57:07 ~10 min tests-rpc 📄log
✔️ 557faaa #12 2025-10-20 17:46:19 ~3 min macos/status-go 📦zip
✔️ 557faaa #10 2025-10-20 17:46:39 ~4 min linux/status-go 📦zip
✔️ 557faaa #10 2025-10-20 17:49:30 ~6 min windows/status-go 📦zip
✔️ 557faaa #10 2025-10-20 17:53:53 ~11 min linux/nwaku 📦zip
✔️ a03ea66 #13 2025-10-20 17:58:28 ~3 min macos/status-go 📦zip
✔️ a03ea66 #11 2025-10-20 17:58:44 ~3 min linux/status-go 📦zip
✔️ a03ea66 #11 2025-10-20 18:01:49 ~6 min windows/status-go 📦zip
✔️ a03ea66 #11 2025-10-20 18:06:12 ~11 min linux/nwaku 📦zip
✔️ 764bce8 #14 2025-10-20 18:02:21 ~3 min macos/status-go 📦zip
✔️ 764bce8 #12 2025-10-20 18:02:54 ~4 min linux/status-go 📦zip
✔️ 9e0e0d7 #15 2025-10-20 18:06:11 ~3 min macos/status-go 📦zip
✔️ 9e0e0d7 #13 2025-10-20 18:07:06 ~4 min linux/status-go 📦zip
✔️ 9e0e0d7 #13 2025-10-20 18:08:40 ~6 min windows/status-go 📦zip
✔️ 9e0e0d7 #12 2025-10-20 18:17:04 ~10 min linux/nwaku 📦zip
✔️ 58bf218 #14 2025-10-20 19:00:04 ~3 min linux/status-go 📦zip
✔️ 58bf218 #14 2025-10-20 19:02:44 ~6 min windows/status-go 📦zip
✔️ 58bf218 #16 2025-10-20 19:03:11 ~7 min macos/status-go 📦zip
✔️ 58bf218 #13 2025-10-20 19:07:16 ~11 min linux/nwaku 📦zip
✔️ 7ba960b #17 2025-10-20 19:10:42 ~3 min macos/status-go 📦zip
✔️ 7ba960b #15 2025-10-20 19:10:46 ~4 min linux/status-go 📦zip
✔️ 7ba960b #15 2025-10-20 19:13:42 ~6 min windows/status-go 📦zip
✔️ 7ba960b #14 2025-10-20 19:18:27 ~11 min linux/nwaku 📦zip
cfc1044 #18 2025-10-20 19:24:59 ~1 min macos/status-go 📄log
cfc1044 #16 2025-10-20 19:25:07 ~2 min linux/status-go 📄log
cfc1044 #16 2025-10-20 19:27:26 ~4 min windows/status-go 📄log
cfc1044 #15 2025-10-20 19:32:02 ~9 min linux/nwaku 📄log
✔️ 803b74f #19 2025-10-20 19:38:55 ~3 min macos/status-go 📦zip
✔️ 803b74f #17 2025-10-20 19:39:24 ~4 min linux/status-go 📦zip
✔️ 803b74f #17 2025-10-20 19:41:58 ~6 min windows/status-go 📦zip
✔️ 803b74f #16 2025-10-20 19:46:25 ~11 min linux/nwaku 📦zip
✔️ 99abe84 #18 2025-10-21 13:16:35 ~4 min linux/status-go 📦zip
✔️ 99abe84 #18 2025-10-21 13:19:42 ~7 min windows/status-go 📦zip
✔️ 99abe84 #20 2025-10-21 13:22:14 ~9 min macos/status-go 📦zip
✔️ 99abe84 #20 2025-10-21 13:24:24 ~11 min tests-rpc 📄log
✔️ 99abe84 #17 2025-10-21 13:26:27 ~13 min linux/nwaku 📦zip
✔️ 99abe84 #19 2025-10-21 13:39:34 ~26 min tests 📄log
Commit #️⃣ Finished (UTC) Duration Platform Result
✔️ 6445824 #19 2025-10-21 20:20:38 ~3 min linux/status-go 📦zip
✔️ 6445824 #19 2025-10-21 20:23:32 ~6 min windows/status-go 📦zip
✔️ 6445824 #18 2025-10-21 20:27:37 ~10 min linux/nwaku 📦zip
✔️ 6445824 #21 2025-10-21 20:29:09 ~12 min tests-rpc 📄log
✔️ 6445824 #20 2025-10-21 20:45:15 ~28 min tests 📄log
✔️ 6445824 #21 2025-10-21 23:04:03 ~2 hr 47 min macos/status-go 📦zip
✔️ 18059a8 #20 2025-10-22 16:08:00 ~4 min linux/status-go 📦zip
✖️ 18059a8 #21 2025-10-22 16:10:05 ~6 min tests 📄log
✔️ 18059a8 #20 2025-10-22 16:11:38 ~7 min windows/status-go 📦zip
✔️ 18059a8 #19 2025-10-22 16:15:20 ~11 min linux/nwaku 📦zip
✔️ 18059a8 #22 2025-10-22 16:16:22 ~12 min tests-rpc 📄log
✖️ 18059a8 #22 2025-10-22 17:18:04 ~29 min tests 📄log
✔️ 18059a8 #22 2025-10-22 18:11:45 ~2 hr 7 min macos/status-go 📦zip
✔️ 18059a8 #23 2025-10-22 18:31:16 ~28 min tests 📄log

@osmaczko osmaczko marked this pull request as draft October 13, 2025 17:54
@osmaczko osmaczko force-pushed the refactor/messaging-persistence-interfaces branch 7 times, most recently from 0ee8ad6 to 459e2ec Compare October 16, 2025 12:29
@osmaczko osmaczko marked this pull request as ready for review October 16, 2025 12:59
@codecov
Copy link

codecov bot commented Oct 16, 2025

Codecov Report

❌ Patch coverage is 74.14005% with 421 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.50%. Comparing base (770ce11) to head (18059a8).
⚠️ Report is 1 commits behind head on develop.

Files with missing lines Patch % Lines
messaging/layers/encryption/sqlite_persistence.go 78.53% 119 Missing and 42 partials ⚠️
...ayers/encryption/multidevice/sqlite_persistence.go 70.79% 48 Missing and 18 partials ⚠️
...yers/encryption/sharedsecret/sqlite_persistence.go 64.36% 22 Missing and 9 partials ⚠️
messaging/layers/transport/sqlite_persistence.go 72.38% 21 Missing and 8 partials ⚠️
...essaging/layers/segmentation/sqlite_persistence.go 64.55% 21 Missing and 7 partials ⚠️
sqlite/migrate.go 60.65% 17 Missing and 7 partials ⚠️
messaging/common/sqlite_persistence.go 75.82% 16 Missing and 6 partials ⚠️
...ayers/encryption/hash_ratchet_key_compatibility.go 70.73% 8 Missing and 4 partials ⚠️
messaging/sqlite_persistence.go 70.73% 8 Missing and 4 partials ⚠️
messaging/waku/sqlite_persistence.go 79.16% 7 Missing and 3 partials ⚠️
... and 7 more
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #6998      +/-   ##
===========================================
- Coverage    59.55%   59.50%   -0.05%     
===========================================
  Files          801      805       +4     
  Lines       114386   114504     +118     
===========================================
+ Hits         68119    68133      +14     
- Misses       39276    39363      +87     
- Partials      6991     7008      +17     
Flag Coverage Δ
functional 35.51% <52.94%> (-0.03%) ⬇️
unit 55.15% <73.40%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
appdatabase/database.go 73.23% <100.00%> (ø)
cmd/push-notification-server/main.go 51.66% <100.00%> (-0.27%) ⬇️
messaging/common/message_sender.go 66.22% <100.00%> (+0.48%) ⬆️
messaging/layers/encryption/encryptor.go 72.25% <100.00%> (ø)
...aging/layers/encryption/multidevice/multidevice.go 86.44% <100.00%> (ø)
messaging/layers/encryption/protocol.go 69.98% <100.00%> (-0.51%) ⬇️
messaging/layers/transport/filters_manager.go 76.31% <ø> (ø)
messaging/waku/gowaku.go 65.41% <100.00%> (-0.72%) ⬇️
protocol/messenger.go 61.28% <ø> (+0.16%) ⬆️
protocol/messenger_peersyncing.go 18.18% <100.00%> (ø)
... and 19 more

... and 27 files with indirect coverage changes

@osmaczko osmaczko force-pushed the refactor/messaging-persistence-interfaces branch from 459e2ec to 78d6e84 Compare October 16, 2025 13:54
@osmaczko
Copy link
Contributor Author

@igor-sirotin @dlipicar it is now ready for review 🙏

Copy link
Collaborator

@igor-sirotin igor-sirotin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't look the whole code yet, this is the first part of comments

if _, err := db.Exec(createIndex); err != nil {
return err
}
insertVersion := fmt.Sprintf(`INSERT INTO %s (version, dirty) VALUES (?, ?)`, name) // nolint:gosec
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about this to avoid // nolint?

Suggested change
insertVersion := fmt.Sprintf(`INSERT INTO %s (version, dirty) VALUES (?, ?)`, name) // nolint:gosec
insertVersion := fmt.Sprintf(`INSERT INTO %s (version, dirty)`, name) + "VALUES (?, ?)"

@osmaczko osmaczko force-pushed the refactor/messaging-persistence-interfaces branch from 78d6e84 to 557faaa Compare October 20, 2025 17:42
@osmaczko osmaczko changed the title refactor: introduce encryption persistence interface refactor: modularize messaging persistence Oct 20, 2025
@osmaczko osmaczko force-pushed the refactor/messaging-persistence-interfaces branch 9 times, most recently from 99abe84 to 6445824 Compare October 21, 2025 20:16
Copy link
Collaborator

@igor-sirotin igor-sirotin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing critical, just 19 little comments 😂

Great job! 👏

Would be amazing if you can write down the approach in some README in the repo. So that further we can follow it with other modules.

Comment on lines +30 to +34
db, err := helpers.SetupTestMemorySQLDB(helpers.NewTestDBInitializer([]*bindata.AssetSource{
{
Names: migrations.AssetNames(),
AssetFunc: migrations.Asset,
},
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can implement

package migrations

func AssetSources() []*bindata.AssetSource {
	return []*bindata.AssetSource{
		{
			Names:     AssetNames(),
			AssetFunc: Asset,
		},
	}
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, we see these two lines repeated in a lot of places, would be nice to make it a single one

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

migrations is a generated package. I’d have to create a separate file for each module containing migrations and define an identical AssetSources() function, which is also far from ideal. For now, I prefer duplication for the sake of simplicity, wdyt?

- make each messaging submodule define its own persistence interface and
  SQLite implementation reference
- separate migration tables for messaging submodules
- move migrations from protocol to relevant messaging submodules

closes: #6792
@osmaczko osmaczko force-pushed the refactor/messaging-persistence-interfaces branch from 6445824 to 18059a8 Compare October 22, 2025 16:03
osmaczko added a commit to status-im/status-desktop that referenced this pull request Oct 22, 2025
@osmaczko osmaczko merged commit 09819b6 into develop Oct 22, 2025
21 checks passed
@osmaczko osmaczko deleted the refactor/messaging-persistence-interfaces branch October 22, 2025 18:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Messaging module should not consume sqlite db

4 participants