feat: pre-messages / next version of download on demand #7371

Simon-Laux · 2025-10-29T20:53:40Z

feat: Remove "Download maximum available until" and remove stock string DC_STR_DOWNLOAD_AVAILABILITY #7369
remove: partial downloads (remove creation of the stub messages) #7373
feat: pre messages can now get read receipts #7433
feat: send pre-message on messages with large attachments #7410
feat: receive pre-messages and adapt download on demand #7431
... for now read the issue for more info on todo items and what this is about

This is the branch for #7367
closes #7367

Currently removed tests

test_something.py::test_reaction_to_partially_fetched_msg
test_something.py::test_download_limit_chat_assignment
python/tests/test_1_online.py::test_webxdc_download_on_demand
src/calls/calls_tests.py::test_no_partial_calls
src/message/message_tests.rs::test_markseen_not_downloaded_msg
src/message/message_tests.rs::test_msg_seen_on_imap_when_downloaded
src/receive_imf/receive_imf_tests.rs::test_download_later
src/receive_imf/receive_imf_tests.rs::test_create_group_with_big_msg
src/receive_imf/receive_imf_tests.rs::test_partial_group_consistency
src/receive_imf/receive_imf_tests.rs::test_prefer_references_to_downloaded_msgs
src/receive_imf/receive_imf_tests.rs::test_partial_download_key_contact_lookup
src/webxdc/webxdc_tests.rs::test_webxdc_update_for_not_downloaded_instance
src/download.rs::test_download_limit
src/download.rs::test_partial_receive_imf
src/download.rs::test_partial_download_and_ephemeral
src/download.rs::test_status_update_expands_to_nothing
src/download.rs::test_mdn_expands_to_nothing
src/download.rs::test_partial_download_trashed
src/reaction.rs::test_partial_download_and_reaction

Progress of the tests

Overview about the recycled(♻️) and dropped(🗑️) tests of the tests that I removed in #7373

🗑️test_reaction_to_partially_fetched_msg
🗑️ test_partial_download_and_ephemeral
🗑️ test_status_update_expands_to_nothing
♻️ test_partial_download_trashed
♻️ test_partial_download_and_reaction
♻️ test_partial_group_consistency
🗑️ test_partial_download_key_contact_lookup
♻️ test_webxdc_update_for_not_downloaded_instance
🗑️test_no_partial_calls
♻️ test_markseen_not_downloaded_msg
♻️ test_msg_seen_on_imap_when_downloaded
♻️ test_create_group_with_big_msg
♻️ test_download_later
🗑️ test_prefer_references_to_downloaded_msgs
♻️test_download_on_demand
♻️test_download_limit_chat_assignment
🗑️test_webxdc_download_on_demand

All the recycled tests were already recycled/re-made except for some which are still to do:

test_partial_group_consistency
test_msg_seen_on_imap_when_downloaded
test_download_limit_chat_assignment

Furthermore there need to be new tests to test the downloading/scheduling changes.

TODO Tests

postponed to do tests

rust: test_msg_seen_on_imap_when_downloaded (I guess this is for syncing seen state between devices)
rust: test_partial_group_consistency

discarded/ignored test ideas

maybe we should reconsider / discuss those?

test that full message, replacing pre-message does not work if the message encryption state is not the same for both messages (both full- and pre-message)
- unencrypted pre- and full-message are not sent

src/sql/migrations.rs

- Remove partial downloads (remove creation of the stub messages) (#7373) - Remove "Download maximum available until" and remove stock string `DC_STR_DOWNLOAD_AVAILABILITY` (#7369) - Send pre-message on messages with large attachments (#7410) - Pre messages can now get read receipts (#7433) Co-authored-by: Hocuri <[email protected]>

…ty summaries

…e optimization

Hocuri · 2025-12-18T15:24:35Z

src/sql/migrations.rs

+            "CREATE TABLE download_new (
+                rfc724_mid TEXT NOT NULL DEFAULT '',
+                msg_id INTEGER NOT NULL DEFAULT 0
+            ) STRICT;


@link2xt I'm wondering whether we should add UNIQUE(rfc724_mid) or UNIQUE(msg_id)? Or even both? I guess it doesn't really make sense to have the same message in there twice?

Do we need both msg_id and rfc724_mid in this table? rfc724_mid can be obtained from msg_id and there's even INDEX msgs_index1 ON msgs (rfc724_mid) if we need to do a reverse lookup

Previously, the uid_next wasn't advanced, which didn't create any problems, but it also was inefficient, because another loop was done trying to fetch the message again (and then finally skipping it, because it's already known).

iequidoo · 2025-12-22T06:47:30Z

deltachat-ffi/deltachat.h

 * - `download_limit` = Messages up to this number of bytes are downloaded automatically.
- *                    For larger messages, only the header is downloaded and a placeholder is shown.
+ *                    For messages with large attachments, two messages are sent:
+ *                    a Pre-Message containing metadata and a Post-Message containing the attachment.


"Full message" was a better term, "post-message" is literally "smth going after the message". And the full message should contain metadata as well, but the documentation doesn't say that

iequidoo · 2025-12-22T06:47:41Z

deltachat-ffi/deltachat.h

 * Get the size of the file. Returns the size of the file associated with a
- * message, if applicable.
+ * message, if applicable. 
+ * If message is a pre-message, then this returns size of the to be downloaded file.


Suggested change

* If message is a pre-message, then this returns size of the to be downloaded file.

* If message is a pre-message, then this returns the size of the file to be downloaded.

iequidoo · 2025-12-22T06:47:48Z

deltachat-jsonrpc/src/api/types/message.rs

    file_mime: Option<String>,
+
+    /// The size of the file in bytes, if applicable.
+    /// If message is a pre-message, then this is the size of the to be downloaded file.


Suggested change

/// If message is a pre-message, then this is the size of the to be downloaded file.

/// If message is a pre-message, then this is the size of the file to be downloaded.

iequidoo · 2025-12-22T06:47:56Z

src/chat.rs

+/// Pre-Message is a small message with metadata which announces a larger Post-Message.
+/// Post-Messages are not downloaded in the background.
+///
+/// If pre-message is not nessesary this returns a normal message instead.


Suggested change

/// If pre-message is not nessesary this returns a normal message instead.

/// If pre-message is not nessesary this returns `None` as the second value.

But i'd return the pre-message as the first value, this looks more natural to me.

iequidoo · 2025-12-22T06:48:02Z

src/chat.rs

+    if needs_pre_message {
+        info!(
+            context,
+            "Message is large and will be split into a pre- and a post-message.",


Suggested change

"Message is large and will be split into a pre- and a post-message.",

"Message {} is large and will be split into a pre- and a post-message.", msg.id,

iequidoo · 2025-12-22T06:48:56Z

src/download.rs

+}

-        Ok(())
+async fn remove_from_download_table(context: &Context, rfc724_mid: &str) -> Result<()> {


Suggested change

async fn remove_from_download_table(context: &Context, rfc724_mid: &str) -> Result<()> {

async fn delete_download_row(context: &Context, rfc724_mid: &str) -> Result<()> {

iequidoo · 2025-12-22T06:49:01Z

src/download.rs

+// this is a dedicated method because it is used in multiple places.
+pub(crate) async fn premessage_is_downloaded_for(


Suggested change

// this is a dedicated method because it is used in multiple places.

pub(crate) async fn premessage_is_downloaded_for(

pub(crate) async fn pre_message_is_downloaded_for(

for consistent naming. The comment isn't necessary, one can easily grep and see that

iequidoo · 2025-12-22T06:49:08Z

src/download.rs

+                // This is probably a classical email that vanished before we could download it
+                warn!(
+                    context,
+                    "{rfc724_mid} is probably a classical email that vanished before we could download it"


Maybe say what happened exactly? There's already a comment explaining the likely scenario. Logs are for developers mainly anyway

iequidoo · 2025-12-22T06:49:27Z

src/download.rs

+    Ok(())
+}
+
+/// Download known post messages without pre_message


Suggested change

/// Download known post messages without pre_message

/// Download known post-messages without pre-message

iequidoo · 2025-12-22T06:49:31Z

src/download.rs

+        .await?;
+    for rfc724_mid in &rfc724_mids {
+        if !premessage_is_downloaded_for(context, rfc724_mid).await? {
+            // Download the Post-Message unconditionally,


Do we the same for non-deltachat messages, encrypted or unencrypted? They don't have pre-messages, but they are important as well, i.e. should be displayed in chats

iequidoo · 2025-12-24T09:59:41Z

src/chat.rs

+/// Renders the Message or splits it into Post-Message and Pre-Message.
+///
+/// Pre-Message is a small message with metadata which announces a larger Post-Message.
+/// Post-Messages are not downloaded in the background.


If the pre-message is missing, shouldn't the "post-message" be downloaded in background (like usual emails)?

iequidoo · 2025-12-24T09:59:50Z

src/download/pre_msg_metadata.rs

+
+/// Metadata contained in Pre-Message that describes the Post-Message.
+#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
+pub struct PreMsgMetadata {


Actually this is PreMsgData, it's metadata for the "post-message" (and even not metadata, but a part thereof)

iequidoo · 2025-12-24T09:59:53Z

src/download/pre_msg_metadata.rs

+    pub(crate) filename: String,
+    /// Dimensions: width and height of image or video
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub(crate) dimensions: Option<(i32, i32)>,


Can be renamed to wh for more consistency with Param::{Width,Height} and to state that the width goes first

we could also think about using u32 instead of i32, because dimensions can't be negative, but that's probably sth. for another pr, because it would also touch other existing code.

iequidoo · 2025-12-24T09:59:56Z

src/download/pre_msg_metadata.rs

+                _ => {
+                    warn!(context, "Message misses either width or height.");


Suggested change

_ => {

warn!(context, "Message misses either width or height.");

wh => {

warn!(context, "Message {} misses width or height: {:?}.", message.id, wh);

iequidoo · 2025-12-24T09:59:59Z

src/download/pre_msg_metadata.rs

+
+impl Params {
+    /// Applies data from pre_msg_metadata to Params
+    pub(crate) fn apply_from_pre_msg_metadata(


Suggested change

pub(crate) fn apply_from_pre_msg_metadata(

pub(crate) fn apply_pre_msg_metadata(

iequidoo · 2025-12-24T10:01:19Z

src/scheduler.rs

        .log_err(ctx)
        .ok();

+    download_msgs(ctx, &mut session)


I think it's better to move this right after download_known_post_messages_without_pre_message() so that it goes before scan_folders(). Scanning folders isn't that important, it's even debounced to once per minute

iequidoo · 2025-12-24T10:01:22Z

src/scheduler.rs

    Ok(session)
 }

+/// The simplified IMAP IDLE loop to watch non primary folders (non-inbox folders)


Suggested change

/// The simplified IMAP IDLE loop to watch non primary folders (non-inbox folders)

/// The simplified IMAP loop to watch non-inbox folders.

iequidoo · 2025-12-24T10:01:25Z

src/sql/migrations.rs

+             JOIN msgs m ON d.msg_id = m.id
+             WHERE m.rfc724_mid IS NOT NULL AND m.rfc724_mid != '';
+            DROP TABLE download;
+            ALTER TABLE download_new RENAME TO download;


Consider renaming to downloads, we usually use plurals for tables

iequidoo · 2025-12-24T10:01:28Z

src/stock_str.rs

+            Viewtype::File => file(context).await,
+            Viewtype::Webxdc => "Mini App".to_owned(),
+            Viewtype::Vcard => "👤".to_string(),
+            Viewtype::Unknown | Viewtype::Text | Viewtype::Call => self.to_string(),


Maybe add a comment that for these an untranslated string shouldn't normally appear, otherwise may look as a lack of implementation

iequidoo · 2025-12-24T10:01:30Z

src/tests.rs

@@ -1,3 +1,4 @@
 mod account_events;
 mod aeap;
+mod pre_messages;


Btw, i think this is a better layout than e.g. mimeparser/mimeparser_tests.rs. Reminds me of Filesystem Hierarchy Standard which is a more familiar thing

Simon-Laux changed the title ~~[Branch] Pre-messages / next version of download on demand~~ [Branch] pre-messages / next version of download on demand Oct 29, 2025

Simon-Laux force-pushed the pre-messages branch from 6fe6723 to 8140ebc Compare November 6, 2025 13:13

link2xt force-pushed the pre-messages branch 3 times, most recently from 2f1c383 to 6bcc795 Compare December 11, 2025 12:00

link2xt mentioned this pull request Dec 11, 2025

feat: receive pre-messages and adapt download on demand #7431

Merged

26 tasks

link2xt force-pushed the pre-messages branch 6 times, most recently from 8443b17 to 702771f Compare December 11, 2025 21:24

Hocuri reviewed Dec 11, 2025

View reviewed changes

src/sql/migrations.rs Show resolved Hide resolved

link2xt force-pushed the pre-messages branch from 126c64f to bb0fd7c Compare December 14, 2025 17:21

link2xt force-pushed the pre-messages branch from bb0fd7c to a98fe05 Compare December 14, 2025 17:54

remove "Contact" stock string again, use emoji like in summaries

eb489bc

link2xt changed the title ~~[Branch] pre-messages / next version of download on demand~~ feat: pre-messages / next version of download on demand Dec 14, 2025

link2xt force-pushed the main branch from e0ae864 to 1ea8ed6 Compare December 14, 2025 19:44

Hocuri and others added 8 commits December 14, 2025 21:00

fix: Add additional_text also to the summary, in order to prevent emp…

78f884f

…ty summaries

refactor: Rename download_when_normal_starts->download_later

411126f

refactor: use more idiomatic way of creating Vec, and remove prematur…

96fa64d

…e optimization

improve error message

a8f0a6d

move download_msgs call

4458663

log request to download

60b28cb

rustfmt

53dc1c7

test: add pre-message test

73a2c94

Hocuri reviewed Dec 18, 2025

View reviewed changes

Hocuri added 2 commits December 18, 2025 18:28

Merge branch 'main' into pre-messages

3f5fc45

iequidoo reviewed Dec 22, 2025

View reviewed changes

iequidoo reviewed Dec 24, 2025

View reviewed changes

	* If message is a pre-message, then this returns size of the to be downloaded file.
	* If message is a pre-message, then this returns the size of the file to be downloaded.

	/// If message is a pre-message, then this is the size of the to be downloaded file.
	/// If message is a pre-message, then this is the size of the file to be downloaded.

	/// If pre-message is not nessesary this returns a normal message instead.
	/// If pre-message is not nessesary this returns `None` as the second value.

	"Message is large and will be split into a pre- and a post-message.",
	"Message {} is large and will be split into a pre- and a post-message.", msg.id,

	async fn remove_from_download_table(context: &Context, rfc724_mid: &str) -> Result<()> {
	async fn delete_download_row(context: &Context, rfc724_mid: &str) -> Result<()> {

		// this is a dedicated method because it is used in multiple places.
		pub(crate) async fn premessage_is_downloaded_for(

	// this is a dedicated method because it is used in multiple places.
	pub(crate) async fn premessage_is_downloaded_for(
	pub(crate) async fn pre_message_is_downloaded_for(

	/// Download known post messages without pre_message
	/// Download known post-messages without pre-message

		_ => {
		warn!(context, "Message misses either width or height.");

	pub(crate) fn apply_from_pre_msg_metadata(
	pub(crate) fn apply_pre_msg_metadata(

	/// The simplified IMAP IDLE loop to watch non primary folders (non-inbox folders)
	/// The simplified IMAP loop to watch non-inbox folders.

Uh oh!

feat: pre-messages / next version of download on demand #7371

Are you sure you want to change the base?

feat: pre-messages / next version of download on demand #7371

Uh oh!

Conversation

Simon-Laux commented Oct 29, 2025 • edited by link2xt Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Currently removed tests

Progress of the tests

TODO Tests

postponed to do tests

discarded/ignored test ideas

Uh oh!

Uh oh!

Hocuri Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iequidoo Dec 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Simon-Laux commented Oct 29, 2025 •

edited by link2xt

Loading

Hocuri Dec 18, 2025 •

edited

Loading

iequidoo Dec 20, 2025 •

edited

Loading