Add rendering of the web into a directory

[Kobzol]

This is an experimental implementation of turning the Rust website from a dynamic website to a static website (a bunch of assets in a directory on disk).

I sketched out a simple API for rendering the individual pages/routes, and got the index and governance pages to work, including translations.

If you agree with the approach, I can continue :)

[senekor]

As long as the output is good, I won't be pedantic about the code. So, feel free to go ahead and thanks for working on this!

[Manishearth]

Yeah, we've wanted this!

[Manishearth]

We should start to think about what this means for the team and rust version bits, which are dynamic. Probably just regen a couple times a day and have a manual trigger. Probably via GitHub actions.

[Kobzol]

Yes, deploying from GH actions e.g. once per day, plus triggering the deploy when a release happens, is IMO enough.

What about the approach to land this? Are you fine with me completely ripping out Rocket and "atomically" switching to the static web? We could do that using other approaches than Heroku, I suppose.

Or do you want the static and dynamic approach to live side-by-side for some time?

[Kobzol]

@senekor Any thoughts about how to approach the change? Are you fine with landing a PR that completely switches the website over to a static web (that would be ofc the simplest)?

[senekor]

Yeah, I'm fine with switching over all at once.

[Manishearth]

I think it should be fine as long as you can generate redirects.

cc @rust-lang/infra for the Heroku part

[Kobzol]

What do you mean by redirects?

[senekor]

There is some redirect logic in src/redirect.rs. I'm guessing we should reproduce that behavior by generating redirect pages. Something with:

<meta http-equiv="refresh" content="0; url={{ url }}">

[Kobzol]

I see. Hmm, that is a bit hacky, but as a workaround why not, I guess. Depending on the method for static deployment that we choose, we might want to use something better on the webserver layer.

[pietroalbini]

I would do both the meta refresh and a <script>location.href = "{{ url }}";</script>: when I did redirects like that in the past I noticed that JS ones are way faster to execute than the meta refresh ones. We should still include the meta refresh for clients not able to execute JS though.

[Kobzol]

I pushed an MVP, together with GitHub Pages integration. A demo is available here: https://kobzol.github.io/www.rust-lang.org/

Some things missing/unclear:

1. Is Pontoon still needed? It doesn't seem to be enabled on the live site.
2. Redirects are only partially working. I can do simple HTML page redirects, and also try to replicate the logic for redirecting the pre-2018 language redirects (is that still needed?), but I don't know how to do non-HTML redirects, in particular things /pdfs/foo.pdf to /static/pdfs/foo.pdf.
3. The JS language switcher can't deal with a non / base URL yet.

[jieyouxu]

Is Pontoon still needed? It doesn't seem to be enabled on the live site.

Probably not? The instance is dead for a while, and no one had the bandwidth/motivation to fix it. (It's for translation.)

[jieyouxu]

Once this PR is ready, we should also update team repo's README

Service	Synchronized every
Governance section on the website	2 minutes	Integration source

to daily. I can do that in parallel, just let me know when this is close to ready (or send a PR :D)

[senekor]

I didn't review the code in too much detail. I'm more interested in thoroughly testing the built website once the remaining open questions are resolved.

[senekor]

Currently the deploy branch is the one triggering a deployment. I would be fine with deploying from master. @Manishearth what do you think?

[senekor]

I presume this is about the pre-2018 locales? @Manishearth, what do you think, can we get rid of the pre-2018 redirects? Seems like it would save us some complexity.

[Kobzol]

I didn't review the code in too much detail. I'm more interested in thoroughly testing the built website once the remaining open questions are resolved.

That's indeed a great point. I would like to run some web checker tool to compare the old and new websites and display any differences in content. Looking for some software to do it (also trying Claude to generate such a tool xD).

[senekor]

I had similar ideas when I was thinking about making the website static. Some kind of web scraper tool that you can point at an URL and it spits out everything it found into a directory. Then you can just compare that with the site you built. That would be cool and give us way more confidence before landing this. But IDK if it's too much work and not worth it just for this.

[Kobzol]

Ah, that actually seems much better than comparing the websites directly, just download the old website to a directory and then compare the files on disk. I'll try that.

[Kobzol]

Ok I fixed JS language redirection with a non-empty base URL. It should be now possible to host the website under a custom path, e.g. https://kobzol.github.io/www.rust-lang.org, which is useful for hosting the web on GitHub Pages without a custom CNAME.

Apart from the old language redirects and maybe some CI changes, everything is ready from my side. I checked the website with some Python download script generated by Claude, apart from whitespace everything looked fine, but more eyes should look at it later, and test the website manually.