Allow loading of custom PCKs

[Xananax]

Related to #688

Translations and fonts add weight to the package.

For this reason, mainly, we want to split the additional assets that will not serve everyone in a different bundle.

There's a recommended method, but it isn't clear how to manage this.

Requirements:

1. Additional packages should be in the main project's code, so contributors can test them and keep them maintained. They can be in a git sub-repo, but they need to be accessible in the main code with minimal fuss
2. Additional packages should not be part of the published app. This shouldn't require any manual adjustments and must be doable from a CI environment
3. Additional packages should be exportable individually, without repeating the main project's files. This must be doable in a CI, unattended environment

These are all completely unanswered questions.

Before answering them, however, there is the question of packaging and distributing packages in a cross-platform way.

Therefore, here's the plan

• build a small MVP where the app loads a (manually exported) package on Desktop & Web platforms and reaches a similar result in both cases.
• package the update in improvement: Add Chinese font #688 manually and load that. Document the process.

Answers to automation can be found along the way or later (but this issue should not be closed before they are, or another issue is opened)

Note: if loading packages on Web and Desktop is sufficiently divergent, it might be easier to build a version with all packages for Desktop, and use a specific mechanism on the web. This would be also considered a valid resolution, provided the build system doesn't get overly complex as a result.

[NathanLovato]

Suggestions by @Razoric480:

• Using an HTTPRequest to download the .pck to res:// then load_resource() (smart!)
• downloading the files on JS side

Details: #688 (comment)

[Xananax]

I'll continue the conversation here:

I'm not too concerned about how to load the files on web, my main concern is how to do it in a way that is consistent across platforms without too much divergent code or branches.

What's the Simplest Version of This? (TLDR)

I think the simplest and most restrained version that isn't future blocking is:

1. Put the PCKs on a public URL, and load them conditionally in the web app through preload(). For example, load gdquest.github.com/lang.cn.pck if the URL contains &lang=en (also sets internal language, on load, from the URL). Uses the browser's natural caching mechanisms for deciding whether to load a new version or not.
2. Embed the resources with the app for Desktop

Longer Discussion

Three possible ways of handling additional resources on web:

1. User can download the resources and unpack them anywhere (user://, more likely than res://).
2. Loading them through the browser
3. Loading the font through the browser

Parallel to those two mechanisms, on a Desktop app, that'd mean:

1. downloading resources through the app's UI and storing in user://
2. telling people to download the resources manually and putting them in <user dir> at the right location
3. Have the font embedded

I'm going to discuss the pros and cons of those implementations below. Bear in mind we don't have to bear the full weight of the cons as of right now (or the full benefits of the pros) but if the intent is to support multiple languages and localization (not just translation), which might entail more than just a font file, then these are still considerations for the long term.

---

Pro/Cons

IndexedDB/Load at Runtime

Pros:

1. Both Desktop and Web download from the same location.
2. Both unpack and load the files in the exact same manner.
3. More flexibility in how we process the files or how we propose them to the user (we can propose to download multiple language files, all, a bundle, ...).

Cons:

1. Building a UI for loading/progress/failure/retry.
2. We need to check integrity, for erroneous/interrupted downloads
3. Building a UI/popup to warn that the package is updated, and to trigger a new download/check validity/replace either automatically or manually.
4. Minor: We can theoretically hit the storage limit (which the app will share with anything else it needs to store -- 5MB for localStorage, 250MB for IndexedDB, I think Godot uses IndexedDB).

note: We might need custom handling of changed assets if they affect loaded scenes (At worst, we can ask people to reset the entire app).

Cache/Load using natural Godot mechanism

Pros:

1. Both Desktop and Web work the same.
2. Loading is not their concern, it is done externally (specific URLs for web, packages in user dir for Desktop). The app's code goes through minimal changes
3. Allows for natural caching mechanisms and no space limits.

Cons:

1. We need a top-mechanism to determine loaded packages, such as a URL parameter or something of the sort, to trigger the preload() behavior with the right URLs
2. downloading manually packages for Desktop is not UX-friendly

Here are some mitigations for con 2, based on easiest to do to hardest:

• Proposing direct links to the assets downloads, and a button that opens the user dir in the operating system file browser, with instructions about how to move them there
• proposing pre-compiled bundles with the assets baked in (that would be a behavior similar to loading different URLs for different languages on the web).
• providing a command-line switch that does the work, then relaunches itself
• making a prelauncher/installer (can still be a single binary, we can use command-line switches for running the app in different modes).

---

Additional Notes

In both cases, we need to build a caching and versioning logic to check for presence of a package, invalidate it (if an update is out), and tell the user.

---

Loading Font/Font Embedded

(note that we could load the font also on Desktop, it just doesn't make a lot of sense)

Pros:

1. Not a generic solution (which means it's simpler to implement)
2. Font will not get updated, there's probably no need for invalidation and versioning

Cons:

1. Possibly divergent code between Desktop and Web (unless we decide to download also on Desktop)
2. Same problems as either solution 1 or 2: we either need to load at runtime, show an UI to the user, etc, or, load beforehand somehow. One major difference is that we don't need to build any versioning (probably).
3. Minor?: if we need to update something else than fonts for localisation (images, text, entire exercise?) this is not a solution that works for anything else.
4. Minor: Maybe creating a font resource at runtime from a TTF, in the browser, is expensive (or even doesn't work?)

[Xananax]

However it seems in principle, exporting with/ without assets seems doable through filtering. We only have to ensure the assets are always conditionally loaded

[NathanLovato]

I'll have to re-read the above to really get it, but at least to me, we could have a desktop export with all fonts included. My concern's mostly the html5 one, that it's relatively quick to launch (on slow ADSL each 5mb font would add 5+s of initial loading of the page for everyone).

Some good insights on generating PCK files here: https://twitter.com/Lauson1ex/status/1577455482410631168

This person uses system fonts, that could be an option too if the browser gives us access to that? That would save the need for downloading any fonts at least.

[rsubtil]

I'm currently working on an app heavily focused on user generated content, and the best way I found to do it was through loading exported PCK projects. I don't know how well it works for Web platforms, but here are a few tips I've gathered when working with them:

• This mechanism is not only undocumented, but also very barebones at the moment. There is a proposal at Treat loaded resource packs separately from each other godotengine/godot-proposals#2689 with many great ideas to bring more functionality to it.
  • I've implemented unloading resource packs at Implement unload of PCK files godotengine/godot#61286 for master, and backported it for 3.x in a custom engine fork.
  • When I got the time I'm also looking to implement a way for inspecting individual files without loading the full PCK content.
• While there are likely changes to behavior between Web and Desktop, PCK files are indeed platform-independent (confirmed from Cannot export PCK without templates installed/configured godotengine/godot#64748), so if you're only adding new resource files, hopefully you can implement your functionality without too much hassle.
• Exporting PCK projects is a bit cumbersome as Godot forces you to have a valid export template and config for a given platform. This behavior (Cannot export PCK without templates installed/configured godotengine/godot#64748) has been improved on master and cherry-picked for 3.6, but I think this doesn't entirely get rid of all issues (for example, you still need to setup a platform to be able to set resource export filters and/or feature tags). I'm looking into opening a proposal for improving this behavior in the near future (if there isn't any yet)
• Automating PCK exporting is easy. Godot can export PCK files from CLI with godot --export-pack "Export Name" exported_project.pck, and it can easily be integrated in CI workflows. This was something I was worrying during development, but thankfully it's been smooth so far (you still need an export template installed, but this requirement should disappear in the near future, see above point).

Hope these help you in development 🙂

[NathanLovato]

Thanks much for the insights,they're most appreciated! It sounds a little cumbersome for our needs then. Someone mentioned that we can just zip the resources and download and load them at runtime like a PCK file.

If it's this simple it's something to consider as all we need is downloading fonts.

[Xananax]

Wether zip or pck doesn't really change much.

The principle is the same for whatever format we want to use. Either we load:

a) at runtime, which implies a UI, error reporting (to the user), integrity check, invalidation
b) before loading the game, which requires loading assets in JS following some logic, and custom builds for Desktop and Web, respectively with and without the assets

In both cases we need the assets packaged and stored somewhere on a canonical URL

(also, I don't think you can unzip in Godot natively)

[NathanLovato]

(also, I don't think you can godotengine/godot#34444)

Godot can, pck are zip files as far as I know, but you don't have an API to do more with zips. Just something like load_resource().

integrity check, invalidation

Can't we rely on Godot's load() to error out and return null?

I'm leaning towards a) right now because of the ability to keep as possible the app's logic in gdscript, and I feel it's straightforward, but what do you think?

[Xananax]

Tentative plan:

1. Allow languages directories in learn-gdscript-translations to contain a font
2. For each language directory, build the directory as a PCK containing the translations and the font
3. Host the packages on https://GDQuest.github.io/learn-gdscript-translations/<file>.pck. Export also a json file listing them all, and each language's coverage
4. In the app, in the settings page, load that JSON, and display only the languages with 100% coverage, unless some flag is true (show the % coverage for all languages in the interface)
5. Upon selecting a language, download PCK, unpack the files, move the translations and fonts to the correct place, then switch the language.
6. Optional: download all the packages as part of the Desktop CI build and store them all in the exported build so no additional download is necessary

For 5, some research is required regarding how Godot handles translations. Can we load translations at runtime, or do we need to process them somehow?

[fire]

godotengine/godot#65281 is merged (Expose minizip API to allow creating zips using scripts).