in VE use Positron's statementRangeProvider to execute statements

[vezwork]

New in Positron: pressing ctrl+enter with your cursor in a multiline R or Python statement executes the whole statement.

Before it would only execute the current line.

Note: this fix relies on Positron's vscode.executeStatementRangeProvider command and does not work in VSCode. I was attempting to get this working in VSCode by creating a virtual doc for the code block, setting your cursor in the virtual doc, and running a cell executor on the virtual doc. That did not work out because I don't know how to set your cursor's position in a virtual doc and cell execution commands don't take positions as arguments.

addresses posit-dev/positron#7709

[cscheid]

I think it's ok to prioritize Positron fixes and leave the VSC fix open for a community PR if there's an approach there that would work.

[juliasilge]

This is really exciting to see! 🎉

I think there is a bit of improvement to do in terms of where the cursor goes in the visual editor after using these. I see that you said there is some difficulty about setting the cursor position in the visual editor 😬 so it's possible this is a big challenge. Take a look at what I mean:

---
title: "Diamond sizes"
format: html
---

```{r}
library(tidyverse)

smaller <- diamonds |>
  filter(carat <= 2.5)
```

• In the source editor, put your cursor at library
• Use Cmd+Enter to step through the lines, including the multiline R statement (notice where your cursor goes after each statement execution
• Switch to the visual editor and put your cursor back at library
• Use Cmd+Enter to try to step through the lines (notice that the cursor doesn't advance enough, which feels especially weird on the empty line)

I also wanted to ask about Python, because I don't see the Python statement range provider being applied, I think? Here is an example to look at:

---
title: "Quarto Basics"
format: html
---

## Matplotlib

For a demonstration of a line plot on a polar axis, see @fig-polar.

```{python}
#| label: fig-polar
#| fig-cap: "A line plot on a polar axis"

import numpy as np
import matplotlib.pyplot as plt

r = np.arange(0, 2, 0.01)
theta = 2 * np.pi * r
fig, ax = plt.subplots(
    subplot_kw = {'projection': 'polar'}
    )
ax.plot(theta, r)
ax.set_rticks([0.5, 1, 1.5, 2])
ax.grid(True)
plt.show()
```

• In the source editor, put your cursor at import numpy
• Use Cmd+Enter to step through the lines, including the multiline Python statement (notice where your cursor goes after each statement execution
• Switch to the visual editor and pur your cursor back at import numpy
• Use Cmd+Enter to try to step through the lines (notice it executes the whole cell, not line by line)

[vezwork]

I think there is a bit of improvement to do in terms of where the cursor goes in the visual editor after using these. I see that you said there is some difficulty about setting the cursor position in the visual editor 😬 [..]

I should be able to set the cursor in the visual editor! The cursor placement problem I was having wasn't visual editor specific, it was virtual doc specific.

I will try to improve where the cursor goes after running ctrl+enter.

Aside, to elaborate on the virtual doc + cursor problem I was facing: it would be nice to unify execution in the visual editor and the source editor by creating a virtual doc for the current code block, placing a cursor in the virtual doc, and running the relevent execution-at-cursor command in that virtual doc. Unfortunately, I think virtual docs are just files and are not "open" in an editor, so they don't have a cursor associated with them.

[vezwork]

I also wanted to ask about Python, because I don't see the Python statement range provider being applied, I think? Here is an example to look at:

Heard and seen! Thanks for the great example. Indeed the python statement range provider does not seem to be being applied. Also looking at this.

[vezwork]

Q: In the source editor, when you Cmd+Enter to step through executing the statements in a cell, what logic does it use to find the index of the start of the next statement?

A: @sharon-wang found it for me: https://github.com/posit-dev/positron/blob/main/src/vs/workbench/contrib/positronConsole/browser/positronConsoleActions.ts#L428 in the advanceStatement function.

I will copy this logic into the Quarto LSP. Maybe ideally Positron would provide a command like "vscode.executeStatementRangeProviderAndGetNextStatementPosition" that returns both a StatementRange and a Position of the subsequent statement so the logic wouldn't have to be duplicated?

[juliasilge]

I think this was only applying to R documents because of this here:

quarto/apps/vscode/src/providers/cell/commands.ts

Line 344 in 2dede3f

if (selection.length <= 0 && !isKnitrDocument(editor.document, this.engine_)) {

---

What happens in the source editor is something like this:

• Positron says, "I have a quarto document and I need a statement range provider for it"
• Quarto extension says, "I have one of those":

quarto/apps/vscode/src/host/hooks.ts

Line 97 in 012d457

return hooks.languages.registerStatementRangeProvider('quarto',

• Positron says, "OK, I want you to give me the statement range at position X"
• Quarto extension say, "I will do that":

quarto/apps/vscode/src/host/hooks.ts

Line 161 in 012d457

async provideStatementRange(

• Quarto extension fulfills getStatementRange() by executing vscode.executeStatementRangeProvider for the virtual doc (i.e. it asks Positron for the statement range for the underlying virtual doc, since Positron has statement range providers for Python and R):

quarto/apps/vscode/src/host/hooks.ts

Line 186 in 012d457

"vscode.executeStatementRangeProvider",

So it goes back and forth a fair amount!

I think you will want to set up request forwarding in the visualEditorServer function in connection.ts, in codeViewServerMethods; I think what needs to happen for a bunch of the LSP methods (for the statement range provider, F1 help, signature help, etc) is request forwarding. My understanding is that piping through this LSP functionality new to the visual editor will be similar to how completions are piped through.

You can see what LSP functionality the source editor has with all the "providers" in here:

quarto/apps/vscode/src/lsp/client.ts

Line 72 in 012d457

export async function activateLsp(

[vezwork]

Statement execution (with cursor advancement to the next statement) should now be working for both R and Python cells in the Visual Editor in Positron.

A caveat:

• there is a hard-coded assumption in index calculation code about what the first line of a code cell looks like, its assumed it looks like {r} or {python} depending on the language.
  • right now the first line is not passed around with the code block data, which is why I didn't use the actual contents
  • would the first line ever not look like this?

[cscheid]

A caveat:

* there is a hard-coded assumption in index calculation code about what the first line of a code cell looks like, its assumed it looks like `{r}` or `{python}` depending on the language.
  
  * right now the first line is not passed around with the code block data, which is why I didn't use the actual contents
  * would the first line ever not look like this?

I think this is a safe assumption. Someone might one day want to run code cells with extra braces, like {{r}}. But those are not actually executed by Quarto, so we're ok.

[cscheid]

I wonder if you want to add a field to that object that you can explicitly use as the tag for a tagged union.

If you needed to expand the type today, then you've accepted that migrations are necessary. But you've now made migration a little bit harder in the case where a new value also has line: number, character: number.

export type CodeViewSelectionAction = "nextline" | "nextblock" | "prevblock" | { line: number, character: number } 
| {some_other_field: string, line: number, character: number} // testing for this is not annoying.

[cscheid]

One minor comment on type design, but otherwise LGTM.

[juliasilge]

Can you resolve these missing semicolons?

[juliasilge]

Suggested change

	if (hasHooks()) {
	if (isPositron) {

Instead of adding a new use of hasHooks(), can we use the better support in the positron npm package, like this:

quarto/apps/vscode/src/core/quarto.ts

Line 20 in 681e682

import { tryAcquirePositronApi } from "@posit-dev/positron";

And then something like:

const isPositron = tryAcquirePositronApi();

We don't need to do a wholesale refactor of the extension, but let's not add new uses of the old, worse approach.

[juliasilge]

Thank you so much for your persistence in working on this tough problem! 🙌

This implementation has unfortunately broken how the statement range provider works in source mode in Python. Take a file like this:

---
title: "Quarto Basics"
format: html
---

Hi!

## Matplotlib

For a demonstration of a line plot on a polar axis, see @fig-polar.

```{python}
#| label: fig-polar
#| fig-cap: "A line plot on a polar axis"

import numpy as np
import matplotlib.pyplot as plt

r = np.arange(0, 2, 0.01)
theta = 2 * np.pi * r
fig, ax = plt.subplots(
    subplot_kw = {'projection': 'polar'}
    )
ax.plot(theta, r)
ax.set_rticks([0.5, 1, 1.5, 2])
ax.grid(True)
plt.show()
```

Put your cursor in the Python code and use Cmd+Enter to step through the statements. With the current release build of the Quarto extension, you'll step through the statements one by one, including the multiline statement, moving forward to the next statement each time. With this current PR, the statements are not identified correctly and you do not move forward correctly

I still feel surprised about the approach being taken here, with putting really specific logic into the command vs. piping through the LSP request to the visual editor. Can you outline for my understanding why we are taking this approach? From my previous experience in this area,

• I would expect to see the visual mode document's executor use executeSelectionInteractive rather than executeInteractive, so it can use the language aware functionality like the statement range provider

• I would expect to see something here that I could picture extending to F1 help

I would love to understand more about why that type of approach hasn't played out. I know you have really focused on this area, and I definitely trust that you have learned a lot and we can figure out a good way forward!

[vezwork]

Hey @juliasilge, I am working on writing up a proper response to those questions.

In the meantime though, could you check if the source-mode python statement range execution is not broken on main of the extension? I have been trying to understand why it is not working and I think it may not be this PR? It seems to be broken on main for me, but I may be confused.

edit: okay, I am fairly confident now that the latest commit on main with commit message "overview documentation of the extension (#857)" 😭 broke statement range execution in source. Looking into that now

edit 2: I tracked down the problem: Embarrassingly, I did not properly fix my small refactor after being called out on it being breaking. I failed to properly inline getStatementRange. Next time I do a refactor, I need to find ways to be more systematic about it. It also motivates me to set up some infrastructure for source mode behaviour tests. For now, I will put up a small PR to fix the problem.

edit 3: I have gone ahead and merged a one line PR to fix the source mode statement execution. In that PR I describe where the issue came from.

edit 4: I have rebased this PR on main and source mode statement execution should now be working.