Support optimized builds with source maps

[kripken]

Right now e.g. -O3 -gsource-map is not fully optimized: we do optimize the wasm code fully, but it has some properties of a debug build, like the JS is not minified, and there is a names section. emcc.py sets the debug level to 3 when it sees -gsource-map which causes the rest of the effects.

It would be useful to optionally fully optimize. Then the source map could be used like a symbol map for crashes, to symbolize them, and not just with function names but locations as well. (edit: fully optimized builds can be symbolized using emcc --emit-symbol-map, which gives proper function names, but not line:column positions inside functions)

I'm not sure if we should do this as a breaking change. On the one hand it is on a debug build, so shrinking size wouldn't regress people's production builds, but it could interfere with debug workflows people are used to. On the other hand, adding a new flag for this is annoying.

[kripken]

cc @dschuff @aheejin

[dschuff]

+1 on fixing this, IMO "fully optimized but still symbolizable" is an important use case. I think breaking debug workflows is less bad than some other kinds of breaking changes (e.g. those that affect how an application runs), so this kind of breaking change isn't necessarily out of the question, but yes we should have a plan for what the final flags and workflows would be. Maybe thinking through the workflows first would help decide how.

With ELF, you can take a -O3 -g binary and run it through strip to get a binary that you can ship, but which is otherwise identical to the original and can be symbolized using the original. And of course when the binary is separate from the debug info like on Windows, you have 2 files but avoid the problem in the first place. Things are made slightly complicated for us by the fact that we sort of have both: 2 files that run (which are both different when debugging), plus the source map itself. It's of course possible to strip the wasm file; is it possible to similarly transform the output we get today with -gsource-map into something identical to what we'd generate without it?
Or more generally, what would we expect the developer workflow to look like for this use case?

[sbc100]

+1 sounds reasonable to me to make this change without adding a new setting.

We can add a ChangeLog entry and explain how to avoid minifying and/or include a name section, which I believe are already covered by existing options.

[dschuff]

Bringing the discussion back here (instead of splitting it between #24936 and #24626; /cc @jmcclellan-figma and @adamvleggett):

I think before we mark this bug as fixed we should also have a solution that better handles name sections (in addition to fixing #24626), and I think we want to add more comprehensive documentation (presumably somewhere in https://emscripten.org/docs/porting/Debugging.html) that covers the types of debug info available and the flags used all in one place. As the test_dwarf_sourcemap_names and test_binaryen_debug demonstrate, there are a lot of combinations of behavior and a lot of non-obvious ways to control it all.

Currently we have several classes of flags:

• Amount of debug information:-gN (where N is a number), -gmlt/-gline-tables-only (all inherited from clang)
• Type of debug info (DWARF) in the binary: -gdwarf-5 and a few other clang flags
  • Emission of name section in the binary (--profiling-funcs, --profiling)
• Where the DWARF is written: -gsplit-dwarf (from clang) and -gseparate-dwarf (emscripten-specific)
• Emission of non-DWARF external debug info: -gsource-maps and --emit-symbol-map
• JS minification (--profiling, --minify=N)

Some of these are use-case-driven so they are not all orthogonal/composable. e.g. --profiling sets a name section and disables minification; also adding to the complication, some of them are link-time-only vs compile-time-only and/or have different effects at link time vs compile time, e.g. -g3 at compile time adds DWARF to object files, but (unlike other platforms), that DWARF will be stripped out unless you add -g3 at link time. Adding -g at link time also affects JS minification (and -gsource-map as we discovered). Some I'm not even sure about myself (e.g. how exactly do --minify, --profiling, and -gsource-map interact?)

There's some value in having everything be orthogonal and easily composable, but there's also value in having commonly-used flags have the "best" behavior by default (i.e. the behavior users will expect, or best by some other measure such as ease of debugging). In particular I wonder whether -gN and/or -gsource-maps should include name sections or not (and whether they should affect minification). It seems likely that if someone is doing basically anything with their binary other than shipping it to users, the name section is likely useful (e.g. for standalone stack traces) and has little downside (it's not huge enough that it's likely to cause problems in a development environment). Perhaps that's also true of JS minification. OTOH source maps are valuable for production binaries, so you could argue you don't want names or minification by default there.

The most common(?) use cases:

1. Edit/compile/test cycle (you want fast builds, but enough symbols to have useful stack traces. For some devs this is probably just the live-debugger build)
2. Live debugger (DWARF in the wasm file) ... I think for this use case you probably want names so you don't need to invoke the debugger (actually I can't recall whether the DWARF extension will symbolize your stack traces based on DWARF automatically nor not. probably should check that. But maybe in this case names at least are probably harmless given how large DWARF is)
3. Profiling
4. Production build with separate symbols for stack trace symbolization (for most users I expect this is probably source maps)

Less common:
5. Split-dwarf or separate-dwarf for live debugging. This takes more setup than regular DWARF and uses non-default flags, but is useful/needed for debugging large programs. These users might also always need -O1 to keep code size/startup time under control.
6. Production build with DWARF, with the DWARF stripped and archived (could be for stack traces, but maybe also even postmortem debugging?)
7. Live debugging with source maps

You can probably debate the relative frequency or importance of these.

Here a couple of possible policies:

1. Minimum number of flags for the most common use cases
2. Flags are all orthogonal; e.g. -gN etc only affect DWARF, others only affect sourcemaps/names, others affect JS.
3. Flags that result in DWARF also affect names and minification by default. Flags that do not result in DWARF do not (in both cases we'd want the ability to override the default). This isn't the simplest possible factoring of flags, but might be simple enough.

For policy 1: -g at compile/link for live debugging and you get names and JS lines for your stack traces, which you want for use case 1 and 2 (for use case 1 you could also use -gmlt or -g1 instead); --profiling gets you names and JS lines and doesn't limit optimization, case 3; -gsource-maps by itself serves case 4. Case 5 is fine with -gseparate-dwarf or dwp and you get names too. For case 6 you'd do -g followed by strip which would remove everything. Case 7 is more flags because if you prioritize case 4 then you need -gsource-maps -gnames --minify=0 (and -gnames doesn't currently exist). The downside is that there's less rhyme and reason to the flag semantics. -g gives you DWARF, names, and JS whitespace. -gsource-maps gives you just source maps.

For policy 2: For case 1 and 2 you'd probably want -gnames and/or --minify=0 even if you also use -g. For case 3, you'd need --minify=0 -gnames (but we could probably keep --profiling as an alias for those for backward compat). For case 4, you'd just do -gsource-maps. For 5 you'd probably also want -gnames. For 6 I think you wouldn't really need -gnames; compared to policy 1 you could probably just use the same flags and you'd get different results but it wouldn't matter much. Case 7 would be the same as policy 1.

Comparing those 2, I'm not sure the value of totally orthogonal flags is worth it given the need for extra flags in the most common 2 use cases.

I think policy 3 might actually be the same as policy 1. I think possibly the only difference between the current state and those is that we'd need to change the behavior of -gsource-maps to not emit names and to not block minification. This seems less of a breaking change than some other options. I want to think about that a little more.

[jmcclellan-figma]

Not sure if this is muddying the waters too much but for this use case:

Production build with separate symbols for stack trace symbolization (for most users I expect this is probably source maps)

DWARF is still a bit better than source maps in principle, since it better represents inlining (e.g., if a single instruction came from an inlined function call, ideally we'd include both the caller and callee in the symbolized trace, but source maps can only encode one source location for the instruction). In emscripten right now, enabling dwarf will disable some binaryen optimizations - I think to make sure dwarf variable information is usable for the "live-debugging" use case. In an ideal world, there would be some combination of settings that would turn on most binaryen passes (at least as much as gets turned on for source maps right now), and generate line-table-only dwarf for the symbolization use case.

[dschuff]

No worries, this is just the kind of feedback on use cases I was hoping to get.
You are right that enabling DWARF in Binaryen will disable some optimizations, but our currently-recommended workflow for live debugging is to build with -O1 (or no optimizations) and then (with the most common options) Binaryen will actually not run at all.
It's a good point about debug info quality though; you are also right that for the symbolized-optimized-build workflow there is still an advantage of DWARF over source maps (and emscripten's emsymbolizer tool includes support for it along with the rest of the way to symbolize wasm files).

There are 2 possible use cases here: The use case Binaryen's DWARF was originally written for was the live-debugging of optimized code use case: some users really do want to debug the fully-optimized code (game programmers are notable for this in particular). But some Binaryen optimizations trash variable location info and it would take a lot of work and improvements to fully track this in the IR, keep the info up-to-date, and regenerate all the DWARF on output, so we compromise and update the debug info more-or-less in-place at the cost of disabling a few optimizations. Maybe we can at this to our list as use case # 8. Improving support for debugging with -O1 made this case a lot less important, because O1 optimizes enough that you can debug even large programs without resorting to the full production level of optimization. Also to further muddy the water, our recommended "live-debugging" workflow also uses DWARF 5 because it scales better to larger binaries, but Binaryen only supports DWARF 4. Overall it would be quite a bit of work to update to DWARF 5 (let alone doing all the the other variable tracking work), so we haven't found it worthwhile.

The other use case is the symbolized-production-build discussed here (case 6), where you really mostly care about line info: In that case you want Binaryen to optimize as much as it can including all passes, and you only want a "small" subset of the debug info (compile unit, subprogram, inlined_subroutine, call_site descriptors, but not types, variables, etc). Binaryen already tracks the location info and could maybe do some simple tracking of inlining info as well (which it would have to add to when it does its own inlining), and then generate that DWARF subset on output (we'd probably need to explicitly handle inlining better). This would still be some work but probably less than what you'd need for case 8, and this is probably a more important case.