[proposal] Expose ScriptParser as an extension point

[bentsherman]

The ScriptParser class defines how a Nextflow script is executed. It essentially takes a Nextflow script and produces a Groovy class (i.e. BaseScript) which, when executed, builds the workflow DAG.

Exposing this interface as an extension point would serve a number of purposes:

• allow Nextflow to support multiple parser implementations / DSL versions in a modular way (could also be done with a regular interface)

• allow me to develop the formal grammar (Script parser #4613) in a plugin, where I can iterate much faster because the build time is so much lower (few seconds vs 30-60 seconds)

• allow community developers to experiment with Nextflow syntax and propose changes in a way that can be easily tested

• in the long term, allow developers from other domains to develop their own DSLs atop the Nextflow runtime (see BEAM and JS runtimes, which can be targeted by many different languages from different domains)

Lots of potential use cases from having this extension point. But for now I mostly just want to have faster build times 😄

[stevekm]

would this enable the development of methods to use CWL to define Nextflow tasks? just wondering, there's a bunch of old issues related to that
https://github.com/nextflow-io/nextflow/issues?q=cwl
in particular this one #545

[bentsherman]

That is a potential use case, supporting other workflow languages like CWL and WDL. Though I would push for something that simply translates them to Nextflow rather than run them directly, so that the user has a chance to make manual refinements and it's clear what is being executed

[rcannood]

Hi all!

I came across this page because #4613 unfortunately breaks my codebase because it relies on nextflow.script.ScriptParser 😅 No worries, I can adapt my code, and I've also noticed the benefits of the new script parser in having really good syntax highlighting in my VSCode.

However, I would be very interested in exposing the ScriptParser/ScriptLoader as an extension point. Is this still on the roadmap? Is there something I can do to help with this?

Kind regards,
Robrecht

[bentsherman]

Well, it's not as urgent anymore. I found other ways to keep the build time low, and at this point it's probably easier to just use Seqera AI to convert CWL/WDL pipelines to Nextflow.

Being able to support multiple Nextflow language versions in a single release would still be useful, similar to how the JDK can supposedly still run ancient Java code 😅