Using variables in _variables.yml within code chunks

[jwimberl]

Description

I am trying to make a quarto document that contains tutorials for a software package, and I'd like to make the document generic with the ability to change customer-specific settings -- like the connection URL for a database -- and evaluate and render the code with these installation specific settings interpolated in. So, I am trying to define these settings in _variables.yml. However, it seems

_variables.yml

db_hostname: db.customer.com

quarto document

conn <- mockDatabaseConnection(
  host = "{{< var db_hostname >}}",
  ...
)

I would like this to render and show the user the actual command for them to run, e.g.

conn <- mockDatabaseConnection(
  host = "db.customer.com",
  ...
)

However, I found that this variable was not substituted before the code chunk was executed, producing errors. Are the variables in _variables.yml only designed to work in Markdown text outside of code blocks? I had at first thought they were substituted by a Lua pre-processing step that occurred prior to code evaluation. However, according to a comment from a couple years ago, the order is opposite:

#6768 (comment)

The technical reason that what you're trying to do doesn't work is that the resolution of {{< var ... >}} shortcodes happens in Lua, inside Pandoc, and that happens after engine execution.
{{< include ... >}} shortcodes, on the other hand, are special in that they're resolved before engine execution, precisely so that included files can have executable code cells.

Is this still the case? If so, there is a different method recommended to have some sort of "macro" system to templatize code blocks before execution? One brute force solution would be for me to use jinja2 templates (though the syntax overlaps with Quarto variable/metadata syntax and might make it hard to use the latter at all); but is there a Quarto-native alternative?

[mcanouil]

Nothing has changed.

You can use environment variables instead with the corresponding file and shortcode.

• https://quarto.org/docs/projects/environment.html
• https://quarto.org/docs/authoring/variables.html#env

Alternatively, parameters are designed for computations: https://quarto.org/docs/computations/parameters.html

[mcanouil]

How does it not solve exactly? it's the same approach you were trying to use.

The following code cell is using {{< env db_hostname >}} as host name from `_environment`.

```{r}
conn <- mockDatabaseConnection(
  host = Sys.getenv("db_hostname"),
  ...
)
```

And since _environment files work with profiles you can make as many as you want:

• _environment-customerA
• _environment-customerB

quarto render --profile customerA

[dvg-p4]

We don't just want to run the code with the correct hostname -- we want to render the code with the correct hostname as well. (And other various customer-specific object names and such in the rest of the notebook.)

[mcanouil]

Then you need an extra step to display the code, for example (there are many more ways to do this):

---
format: html
---


The following code cell is using {{< env db_hostname >}} as host name from `_environment`.


```r
conn <- mockDatabaseConnection(
  host = "{{< env db_hostname >}}",
  ...
)
```

```{r}
#| echo: false
conn <- mockDatabaseConnection(
  host = Sys.getenv("db_hostname"),
  ...
)
```

[dvg-p4]

Ah I see. That'd work fine if it was just the initial connection block, but we're going to need to have similar customer-specific string literals throughout; in order to avoid duplicating every single code block it seems like we may need to go with the jinja2 templates that render to qmd.

[mcanouil]

If you are going to use engine: knitr there are possible ways using code cells hooks.

• https://bookdown.org/yihui/rmarkdown-cookbook/output-hooks.html

---
format: html
---

```{r}
#| include: false
local({
  hook_source <- knitr::knit_hooks[["get"]]("source")
  knitr::knit_hooks[["set"]](source = function(x, options) {
    # Replace Sys.getenv("variable") with "{{< env variable >}}"
    x <- gsub('Sys\\.getenv\\("([^"]+)"\\)', '"{{< env \\1 >}}"', x)
    
    # Apply default source hook
    hook_source(x, options)
  })
})
```

The following code cell is using {{< env db_hostname >}} as host name from `_environment`.

```{r}
cat(Sys.getenv("db_hostname"))

grepl("customerA", Sys.getenv("db_hostname")) # to illustrate that it's really the environment variable that is used
```

[cscheid]

The {{< var >}} shortcode is resolved long after code cells are executed, during Lua filter execution.

The {{< include >}} shortcode is the only shortcode that is resolved before the engines run.

The order is:

• include shortcodes are resolved
• engines execute code
• Lua filters process all other shortcodes