micromark: render shouldn't freeze the page if jump source and destination are equal

[OEvgeny]

Initial checklist

• I read the support docs
• I read the contributing guide
• I agree to follow the code of conduct
• I searched issues and couldn’t find anything (or linked relevant results below)

Affected packages and versions

micromark@4.0.1 and its dep micromark-util-subtokenize@2.0.2

Link to runnable example

https://bolt.new/~/sb1-7ektru or https://stackblitz.com/edit/sb1-7ektru

Steps to reproduce

1. Open the link provided and wait until Preview appears
2. Click Clear button
3. Right click on the frame page, and select Inpsect (so you could stop execution)
4. Type the following

\[
\
\]

5. Notice the page freezes.
6. Go to Devtools sources pane and hit pause button.
7. Inspect the jumps object
   

Environment details provided directly in the repro. Please use built-in terminal to verify versions.

❯ npm -v
10.2.3
❯ node -v
v18.20.3

Expected behavior

While the extension could have mistakes in token placement, I think it's still expected that parser won't hang during such error. EIther such jumps should be omitted or bail.

Actual behavior

Page hangs in an infinite while loop (line 2698 on the screenshot).

Runtime

Other (please specify in steps to reproduce)

Package manager

Other (please specify in steps to reproduce)

OS

Other (please specify in steps to reproduce)

Build and bundle tools

Vite

[ChristianMurphy]

Welcome @OEvgeny! 👋
Sorry you ran into a spot of trouble.
It looks like you are trying to overload [] to mean either a reference link to to mean math?
I'd recommend not doing that, it can cause crashes due to conflicts in the tokenizer as you are running into.
That isn't a micromark bug, but a but in the custom tokenizer.

---

While not stated here, that looks a lot like ChatGPT generated math.
While not finalized, I'd highly recommend reading unifiedjs/unifiedjs.github.io#46

[ChristianMurphy]

Adding a line like

- **LaTeX Formatting for Clarity**: When presenting math, use LaTeX with "`$...$`" for inline math (e.g., `$x^2 + y^2 = z^2$`) and "`$$...$$`" for display math (e.g., `$$\int x^2 \, dx$$`).

to the LLM prompt resolves the issue without needing a customer tokenizer.

[ChristianMurphy]

sigh I see this is related to microsoft/BotFramework-WebChat#5353
Please, use the markdown flavor your renderer uses instead of creating a custom one, it doesn't "improve rendering and error handling".
It makes yet another custom markdown flavor https://github.com/micromark/micromark?tab=readme-ov-file#extending-markdown

[github-actions]

Hi! Thanks for reaching out! Because we treat issues as our backlog, we close issues that are questions since they don’t represent a task to be completed.

See our support docs for how and where to ask questions.

Thanks,
— bb

[OEvgeny]

It looks like you are trying to overload [] to mean either a reference link to to mean math?

Correct, we need \[ and \] to be the open and closing tags for math blocks. But it isn't related to the issue really.

That isn't a micromark bug, but a but in the custom tokenizer.

I saw some assertions for tokens are already in place. I don't see why such simple assertion is not welcomed, even if it's not met in micromarks code, rather a result of buggy tokenizer extension.

sigh I see this is related to microsoft/BotFramework-WebChat#5353
(...)
it doesn't "improve rendering and error handling".

It doesn't, another part of the PR does though.

sigh I see this is related to microsoft/BotFramework-WebChat#5353

Unfortunately we're forced to use the different flavor of markdown. I'm not happy about it as well.

[ChristianMurphy]

I saw some assertions for tokens are already in place. I don't see why such simple assertion is not welcomed, even if it's not met in micromarks code, rather a result of buggy tokenizer extension.

I'd defer some to @wooorm on this.
Most assertions are used in develop mode.
More of those are probably fine, but changing the built/production output would have performance implications and would take more consideration.

[wooorm]

Assertions are indeed only in development mode. They are compiled away. Are you running in development mode? If you are running in production mode, it is very likely that you are missing all the assertions.

another part of the PR does though.

Which PR?

---

Especially backslashes, and especially square brackets, are already highly used in markdown. Better to use something else.

---

Markdown is not a good grammar. How it all hangs together is complex and everything interacts with each other. It is a goal to allow extensions, but: not everything is possible.

[OEvgeny]

Assertions are indeed only in development mode. They are compiled away. Are you running in development mode? If you are running in production mode, it is very likely that you are missing all the assertions.

Not sure which build I'm using, but from the screenshot above:

7. Inspect the jumps object
   

On the line 2702 you can spot ok(...) call to check tokenizer validity. I'm pretty sure that something should be done, so jumps don't contain entries where source and destination is the same index (5 => 5) in the example.

Could add the check into the function calculating jumps array. Or at least don't emit such entries.

Which PR?

Updated the original quote to include the context.

Especially backslashes, and especially square brackets, are already highly used in markdown. Better to use something else.

We'll keep this in mind, thank you.

Markdown is not a good grammar. How it all hangs together is complex and everything interacts with each other. It is a goal to allow extensions, but: not everything is possible.

I feel the same, that's why I opened the issue, this seems like an easy improvement to make both DX and UX better.