Snapshot Migration

[JBenda]

https://github.com/inkle/ink implements a save/load functionality which can adapt to small story changes.
Can we implement this too? At least, we should throw an error if the snapshot does not match the story.

See #111

[JBenda]

I'm not sure that ink C# has a "migration" algorithm. It seems more that they do create resources not present in the snapshot on the fly, which more seems like an oversight to me than an actual feature.

@harryr0se

[harryr0se]

Heya, thanks for looking into this!

It seems more that they do create resources not present in the snapshot on the fly, which more seems like an oversight to me than an actual feature.

My understanding is that actually is a key part of their save migration mechanism, their strategy is quite well defined in their official guide in the following section:
Running Your Ink > Production Features > Migrating Saves Between Versions - Pages 176 - 178

I can't find a version of the text online, I'll summarise here but I definitely think the guide should be referenced when implementing this in InkCpp (Which I would love 🙏)

---

Text changes
Text changes that don't add additional gathers or choices cause no issue with migration and are completely safe
This includes fixing typos, new words, paragraphs, new conditional blocks and inline variations e.g {1|2|3}

Variables
Variables in the snapshot that no longer exist in the story are discarded
Any new variables added to the story are created as required and initialised to the default values defined in the VAR statement

Read Counts
Basically the same as variables
Any existing read counts for knots and stitches that no longer exist are discarded
Read counts for any new knows and stitches are created with a value of zero

Current story position
They mention that there's some logic that is executed if the saved current story position is no longer available which tries to find the nearest matching address, but the logic is not detailed in the guide and they mention that it's unpredictable

Other details
They give implementation details about how read counts are implemented for sections with aren't knots and stitches which are hand named.
A good example is once only choices, These are implemented by knot + index ids
Which means that care is needed if you change the order of choices in a story with existing saves

They then generally give suggestions on ways to save the game and to update your ink that are less likely to result in lost data

---

For me, this would be an amazing feature. I've been working on a game for the playdate that is making heavy use of the library and without migration functionality the following scenario has been worrying me a fair bit.

1. I ship the game (frankly, a miracle 😄)
2. Players start the game and snapshots are created as part of the game saving
3. I get reports of issues with the story logic, or perhaps I want to add new content
4. I wouldn't be able to make any changes to the story as part of an update without invalidating all existing player saves

I don't currently have the capacity to make large contributions to the code but if it's helpful I'd be very willing to create some before and after ink files with the expected behaviour so we could lock in the functionality with tests? Let me know 😊

[JBenda]

Thanks for summarizing this.
Snapshot migration is indeed an important feature.

Since Inkcpp snaps "compact" it could be that the old story bin is needed for the migration (which could be optional stored inside the snapshot for convenience)

I will put some thoughts into it in the next few days on how to go about the implementation, (local variables and external functions will probably cause some headaches)

It would be nice if you could provide some example Inkcpp files with expected behavior.

Looking forward to implement this

[JBenda]

Probably a tooling to Check the migratability between two story files would also be nice

[JBenda]

Bucket list:

• Variables: Dummy execute current knot/stitch to get default values for new local variables
• Variables: execute global init to get default values for global values
• Variables: remove variables not touched above
• Read counts: load new container list and transfer existing values
• Variables: list of (address, hash, path) to updated stored diverts
• Text changes: store story pointer relative to last knot/stitch/choice in lines
  Only allow migratable snapshots directly after a choice (jump)

Potential drawbacks:

• no snapshots in tunnels/functions because of diverts stored at the stack (potentially difficult to translate)

[harryr0se]

That all sounds great, thanks! 😊

Probably a tooling to Check the migratability between two story files would also be nice

I think that’d be really valuable

no snapshots in tunnels/functions because of diverts stored at the stack (potential difficult to translate)

I think those are reasonable exceptions, Inkle mention in their guide that saving in tunnels is dangerous in their implementation too

On the above, it’d be great if there could be a bool can_create_snapshot or similar on the runner which would return false when the story is currently in a tunnel/function or any other state that’s deemed to be unsafe.

I’ll try to provide some test data for you soon

[harryr0se]

Some example data! :)
This test would cover the following

1. Adding new stitches/tunnels
2. Maintaining the read count of once only choices
3. Adding to a existing list
4. Adding a new choice to the knot that the snapshot is part of

I'm not sure if the use of tunnels here makes a difference, does inkcpp treat those as different from normal stitches?
I'll make some test data which is just simple knot additions as that's a more simple use case

Before

LIST activities = Swimming, SandCastle
VAR completed = ()

-> holiday
=== holiday
We're going to the seaside!
{completed: So far we've done the following: {completed}}
+ [Make a sand castle] -> sand_castle -> holiday
* [Go swimming] -> swimming -> holiday
* Time to go home -> home

= sand_castle
We made a great sand castle, it even has a moat!
~ completed += SandCastle
->->

= swimming
We swim and swam, it was delightful!
~ completed += Swimming
->->

= home
What a nice holiday that was
-> END

After

LIST activities = Swimming, SandCastle, IceCream
VAR completed = ()

-> holiday
=== holiday
We're going to the seaside!
{completed: So far we've done the following: {completed}}
+ [Make a sand castle] -> sand_castle -> holiday
* [Go swimming] -> swimming -> holiday
+ [Get Ice Cream] -> ice_cream -> holiday
* Time to go home -> home

= sand_castle
We made a great sand castle, it even has a moat!
~ completed += SandCastle
->->

= swimming
We swim and swam, it was delightful!
~ completed += Swimming
->->

= ice_cream
We got ice cream, mine was raspberry!
~ completed += IceCream
->->

= home
What a nice holiday that was
-> END

Pseudo code for test:

1. Load before.bin
2. Call getall()
3. Choice 1 // Sand castle
4. Call getall()
5. Choice 2 // Swimming
6. Call getall()
7. Expect line So far we've done the following: Swimming, SandCastle
8. Expect that swimming choice is no longer available
9. Save snapshot at holiday hub knot
10. Load after.bin
11. Restore snapshot
12. Assert completed contains Swimming, SandCastle but not IceCream
13. Expect that swimming choice is still no longer available
14. Choice 3 // IceCream
15. Expect So far we've done the following: Swimming, SandCastle, IceCream

I've attached pre-compiled versions of the before and after stories here (using inkcpp_cl from main)
BeforeAndAfter.zip

[JBenda]

Thanks for providing example data, I'm still thinking about some architecture thinks, but it is going forward.

A tunnel stores a return pointer on the stack to go back after leaving the tunnel.
As long as you do not call the same tunnel multiple times in one context block (continues text without choice/knot/stitch) this point can be easily restored.

The primary idea to translate story position is to count newlines since the last knot/stitch/choice,
Which would allow extending lines and adding calculations without breaking the position.
Adding new lines on top, on the other hand, will break it.
So you will probably have to insert a tunnel if you want to add multiple new lines inside a continuous part without invalidating potential old saves.
My current idea is that you can specify a strategy how to handle a position lost: like go to begin of current block, go to x regadles of current position if you inside x go to y (can be specified multiple times) and if you unable to deduce position go to x (maybe because of a deleted knot)

[JBenda]

I think using tunnels could be a good way to ensure that you can change a story without too many troubles, since if you're in a tunnel you can find the entry point (easy?) and have a deterministic point even iff you add lines before them.

Or avoid huge context chunks anyhow.

[JBenda]

This will take a bit longer, I just discovered that also list must be migrated, since they are stored based on their numeric value, but when you shuffle a list it should probably also migrate.

[JBenda]

Please excuse the long stall; I now have an idea, and the implementation starts to look promising.

Currently it is only possible to create a migratable snapshot directly after a runner->choose(x), because this is the cleanest state we get.

Global variable migration, and visit counts seem to work.

The current problem: how to handle global tags.
A.ink # sun ...
B.ink # rain ...
Load A.snap in B:
Currently, the global tags from the old story are kept. (# sun)
Alternataive:

• tags from the new story (# rain)
• both (#sun rain)
• both with delimiter (#sun ## rain)

Any preferences? I think I like the delimiter approach ...

[harryr0se]

@JBenda that’s great news, thanks for the update and work on this feature! 🙏

I don’t have a strong opinion on the global tags issue, do you know how it’s handled when migrating ink versions in the Inkle C# implementation?

[JBenda]

They use the current tags and allow only static tags for Knot/Global tags.
I find these restrictions sensible and will adapt to them.

Local variables are currently quite annoying, since inklecate does not check if you produce invalid local variable configurations

This story, for example, throws warnings if you enter 1 or 2 1.
So it feels like any temporary variable handling is doomed to produce unwanted behavior.
My offer: Scan all commands from the last named knot/start of story and define all temporary variables found until the current point.
Caviates:

1. If the calculation calls a function, it will result in an assertion. Because if the function has side effects, the Ink story starts to be dependent on the runner implementation details.

Does this sound sensible?

~temp A = 1
+ A1
+ A2
++ A1.1
++ A1.2
~ temp C = 3
--
~ temp B = 1
-
{A}{B}{C}

1: A1
2: A2
?> A1
1-0-0
RUNTIME WARNING: 'test.ink' line 10: Variable not found: 'B'. Using default value of 0 (false). This can happen with temporary variables if the declaration hasn't yet been hit. Globals are always given a default value on load if a value doesn't exist in the save state.
RUNTIME WARNING: 'test.ink' line 10: Variable not found: 'C'. Using default value of 0 (false). This can happen with temporary variables if the declaration hasn't yet been hit. Globals are always given a default value on load if a value doesn't exist in the save state.

===
1: A1
2: A2
?> A2
1: A1.1
2: A1.2
?> A1.1
1-1-0
RUNTIME WARNING: 'test.ink' line 10: Variable not found: 'C'. Using default value of 0 (false). This can happen with temporary variables if the declaration hasn't yet been hit. Globals are always given a default value on load if a value doesn't exist in the save state.

===
1: A1
2: A2
?> A2
1: A1.1
2: A1.2
?> A1.2
1-1-3