Finalize workflow output definition

[bentsherman]

Continuation of #4670

Enumerating the proposed changes that we've collected so far:

• Support additional index file formats (json, yaml)

• Generate output schema. Essentially a list of index file schemas. Should be generated on each run or via some separate command. Should eventually be used with parameter schema for chaining pipelines. See DSL2+ nf-core/fetchngs#312 for a concrete example (schema_outputs.yml). -> Generate output schema from output definition #5213

• Dynamic path mapping. Allow the path option in target definition to be a closure:

  workflow {
    main:
    ch_foo = Channel.of( [ [id: 1], file('foo.fastq') ] )
  
    publish:
    ch_foo >> 'foo'
  }
  
  output {
    'foo' {
      path { meta, fastq -> "foo/${meta.id}" }
    }
  }

  Note that the dynamic path need only define the directory, not the full filename. Since a channel value may contain multiple files, an alternative syntax could be to provide the file and the channel value to the closure, so that it's clear which file is being published:

      path { file, value -> "foo/${value[0].id}" }

• Move publish options to config. Publish options like directory and mode typically need to be configurable by the user, which currently would require you to define a special param for each option. Therefore it makes more sense for them to be config settings rather than pipeline code:

  // nextflow.config
  workflow {
    output {
      directory = 'results'
      mode = 'copy'
  
      withTarget:'foo' {
        mode = 'link'
      }
    }
  }
  
  // main.nf
  output {
    'foo' {
      index {
        path 'index.csv'
        // ...
      }
    }
  }

  The output block should be used only to define index files (i.e. the output schema). In other words, the pipeline code should define what is published and the config should define how it is published.

  For the output directory, it has also been proposed to provide a CLI option for it e.g. -output-dir and shorter config option e.g. outputDir. The output directory would be available in the pipeline code as part of workflow metadata i.e. workflow.outputDir.

• Remove publish section from process definition. Still under discussion. The rationale for the process publish section was to provide some sensible defaults which can be overridden, however I've come to think that it only makes it harder to determine what is being published. Instead of enumerating the publish targets in one place, they are scattered throughout the pipeline code. Also, process definitions are abstracted from any particular pipeline, so it doesn't make much sense to put pipeline-specific details like params and publishing in the process definition.

  A better way to give some sensible defaults for a process would be to write an entry workflow in the process definition that gives an example usage:

  process foo {
    // ...
  }
  
  workflow {
    main:
    params.input = '...'
    foo( params.input )
  
    publish:
    foo.out >> 'foo'
  }

  This workflow will be ignored when importing the process as a module, but it provides a concrete example and can even be used to run the process directly. In fact it could even replace the custom nf-test DSL eventually.

• Allow publish section only in entry workflow. I am less certain about this one but wanted to lay out the case for it. Building on the previous item, having publish sections potentially spread across several workflows in a pipeline makes it hard to see what all is being published. Instead, maybe named workflows should only be allowed to emit outputs, and only the entry workflow be able to publish outputs. As with the previous point, you could write an entry workflow for each named workflow which gives some example publishing (and allow you to run the workflow as a standalone pipeline).

  This idea is inspired by a principle in software engineering that side effects (a.k.a. I/O, publishing) should be pushed to the "boundaries" of the code, to make it more readable and testable, and to make it easier to swap out different I/O strategies (file copy, database insert, API call, etc).

  At the same time, I appreciate that publishing from a named workflow is a convenient shorthand, especially when you considering having to propagate outputs back up through potentially several nested workflows. But I wonder if being more strict here would be better in the long run. The example entry workflow is something that will be written anyway, both for testing and to run workflows as standalone pipelines.

• Runtime enhancements

  • report target names in the output block that weren't defined in a process or workflow
  • prevent leading or trailing slashes
  • detect published file collisions at runtime
  • detect outputs that aren't published or used by downstream processes

• Include output targets in inspect command. Similar to how inspect lists all processes with some resolved directives, etc, it could also show the resolved list of output targets. Not as essential if we implement some of the above points, but still useful for things like resolving params.

[robsyme]

Great suggestions Ben. Do you expect that there will be any changes to the observable events in this new channel-focused model?

A common use-case for plugins is to register the outputs with a third-party service (LIMS, for example). It would be helpful if plugins could hook into a publish event that has both the source, destination, and other channel objects (i.e. meta) in a single event. So that the outputs could be appropriately tagged/saved.

If we are now "publishing channels", it would be helpful to have a way to include everything, including non-file data in that publishing event.

[pinin4fjords]

Great to see this coming together after previous discussions Ben!

Really like the output-dir. It always felt odd to me that work-dir was something native while the workflow builders had to deal with the output directory themselves.

The distinction between 'output' and 'publishing' never sat well with me, it made much more sense to me if everything was an 'output', with things marked temporary/ retained as appropriate. Plus it is a nightmare searching across multiple config files for the publishing logic for a specific set of files (see the current dev branch of nf-core/rnaseq). So I really like all the places where you're removing publishing logic and it makes a LOT of sense to me that the publishing/ retaining part happens only at the outermost layer- I hope you keep that.

Really like having entry workflows as well, having all components be more easily runnable without as much surrounding boilerplate would be awesome.

Would also be interested to hear more on the non-file output aspects, speaking to Rob's point, and how it could assist with daisy chaining workflows, but overall love the way this is going!

[pditommaso]

Points I'd like to discuss:

• Support additional index file formats: for which file(s)?
• Dynamic path mapping: syntax clashes with block definition
• Move publish options to config:
  1. why is requires withTarget?
  2. still should be possible to have in the script
• Remove publish section from process definition.
• Allow publish section only in entry workflow.

[bentsherman]

From @Midnighter in #5130 :

In the last podcast episode, you were debating a bit, whether allowing to define output publishing from a module/process-level.

My clear opinion is that this is a bad idea:

• It's a mix of concerns. It should not be a process' (function's) concern where its output is stored. You are giving it too much responsibility.
• Additionally, allowing publishing to only be set at the workflow level ensures the modularity of processes is optimal. The same actually applies to sub-workflows in my mind. I would only consider the highest-level workflow publishing instructions and ignore publishing set in sub-workflows. That way, modules/sub-workflows can easily be used in different pipelines without requiring overrides.
• In my view, processes should be as close as possible to pure functions, such that you have a reproducible output when given a deterministic environment (container hash), same input (hash of data), and same operations (hash of code/git commit). Changing publishing behavior requires code changes, even though the operation itself is not changing.
• Flexibility in storage backends: If a module assumes certain output path capabilities that are not supported by my storage solution, I have to redefine all those publishing options.

[bentsherman]

Thanks @Midnighter, I have basically reached the same conclusions. After discussing with Paolo, I think we will encourage publish: to be used only in the entry workflow, but still allow it in sub-workflows as a convenience, and not allow it in processes.

[FriederikeHanssen]

Hey! If I can add something from my wishlist:

it would be great to get logging on what dynamic output paths like "foo/${meta.id}" are resolved to. Either just for WARN/Errors (i.e.publishDir path is null ) and have it show up somewhere in the error logs. Or if it doesn't get to verbose in the regular nextflow log.

[nvnieuwk]

Any news on when the dynamic path mapping will be in a nextflow edge release? I'm currently trying to convert my file publishing to the new output definitions but need this feature to be able to copy the old way

[bentsherman]

@nvnieuwk Still in progress. I have an implementation in the linked PR but it needs to be reviewed and tested. You're welcome to try it out if you want. I doubt it will make the next edge release, maybe 24.09.0-edge.

[bentsherman]

I'd like to get some opinions on some of the nuances of the dynamic path mapping. If I have something like this:

workflow {
  main:
  ch_fastq = Channel.of( [ [id: 1], [ file('foo.fastq') ] ] )

  publish:
  ch_fastq >> 'fastq'
}

output {
  directory 'results'

  'fastq' {
    path { /* ... */ }
  }
}

I see a few different ways to design the path closure:

1. The path closure defines the entire file path relative to the output directory:

   // val == [ meta, fastqs ]
   path { file, val -> "fastq/${val[0].id}/${file.name}" }

   The final published path would be results/fastq/${meta.id}/${file.name}. Since a channel value can contain multiple files, the path closure is invoked for each individual file, passing both the file and the containing channel value.

2. The path closure only defines the subdirectory:

   path { val -> "fastq/${val[0].id}" }
   // could also do { meta, fastqs -> "fastq/${meta.id}" }

   The final published path is the same as (1), but now the file name is implied and can't be customized in the path closure. All files in a channel value are published to the same subdirectory.

I like (2) because it's simpler and retains the meaning of the path setting, but it's also not as flexible as (1). If by "dynamic paths" we only mean things like "organize samples into subdirectories by id", then (2) is all we need. But if people also want to be able to change the base name of each file (which is supported by saveAs), then (2) is not flexible enough.

Option (1) is essentially the same as saveAs but with the context explicitly given as a second parameter (the channel value), rather than the task inputs being implicitly available.

Let me know what you think. This is the essential question we need to resolve in order to move this feature forward.

[nvnieuwk]

I personally like option 1 because that way I don't have to make sure the files emitted by processes have the correct name as I will set the correct output name here. It's also a nice option to be able to specify some kind of optional nesting which option 1 would enable.

[nvnieuwk]

During my testing I also noticed a weird error. So when an input file that hasn't been staged is present in the channel specified in the publish section, the next error happens:

  ERROR ~ assert path
         |
         null

It started working again if I filtered out these paths