Skip to content

Conversation

@jorgee
Copy link
Contributor

@jorgee jorgee commented Nov 6, 2025

close #5077

This PR retrieves the platform metada during the onWorkflowCreate and stores in the seqeraPlatform property in the WorkflowMetadata class. The lineage module has been extended adding a metadata field in the WorkflowRun record including the platform metadata.

This PR includes the changes of #6069.

To tests the whole integration, it is required to generate a nf-launcher image with the nextflow changes as in e2e tests. Then , use tw launch to run a workflow including the options to use a modified launcher image (--launcher-image) and a nextflow.config file (--config) with the following lineage configuration

lineage.enabled=true
lineage.store.location='s3://<bucket>/lineage'

Use a <bucket> associated to the CE running the pipeline to ensure you have the proper permissions.

Once finished, you can use the nextflow lineage commands to inspect the workflow lineage records.

Example of the generated metadata extracted form a test execution:

"metadata" : {
    ...
    "seqeraPlatform": {
        "workflowId": "4mjnvcrG15bCMp",
        "user": {
          "id": "13898",
          "userName": "jorge-ejarque",
          "email": "[email protected]",
          "firstName": "Jorge",
          "lastName": "Ejarque",
          "organization": null
        },
        "workspace": {
          "id": "20121123931769",
          "name": "jorge-test-workspace",
          "fullName": "Jorge test workspace",
          "organization": "test-org-jorge"
        },
        "computeEnv": {
          "id": "3xjr2I8RofsG5Kx0pjZQoQ",
          "name": "test-ce",
          "platform": "aws-cloud"
        },
        "pipeline": {
          "id": "51175491284294",
          "name": "https://github.com/nextflow-io/hello",
          "revision": "master",
          "commitId": null
        },
        "labels": []
      },
}

TODO:

  • Check if version upgrade is required

@netlify
Copy link

netlify bot commented Nov 6, 2025

Deploy Preview for nextflow-docs-staging canceled.

Name Link
🔨 Latest commit b4a524b
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/690dd6261c68290008d57f2c

@jorgee
Copy link
Contributor Author

jorgee commented Nov 7, 2025

Updates:

Regarding increasing the version, I have tested reading WorkflowRun generated with this PR with 25.10.0 and olded versions and viceversa.

  • Records generated wih 25.04.x can be read with this PR metadata appear as null as expected.
  • Records generated with this PR can be read since 25.07.0-edge, but metadata field is not shown.
  • We can't read in 25.04.x because of the spec change. They could be read if we backport commit 6328613 where we added the spec management.

Apart form that, a minor comment is that there are some duplicated data such as run name, session id, repository, script files or revision. They are already Workflow or WorkflowRun properties and they are also inside metadata. Should these properties be removed form metadata ?

void setCommonApi( TowerCommonApi commonApi ){
this.commonApi = commonApi
}
/**
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would not make sense to pass both TowerCommonApi and HxClient via the constructor?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

New introspection workflow.seqeraPlatform scope for data from Seqera Platform

3 participants