Use Managed Identities with Azure to download and upload files #6344
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… in Azure Batch Previously, Azure Batch required SAS tokens to download and upload the resource files (.command.sh and .command.run). This PR adds support for using managed identities to download these files. We use the managed identity specified in the Azure Batch configuration where available, if it's not found it falls back to using a SAS token. - Add getPoolManagedIdentityResourceId() method to retrieve managed identity from pool - Modify resourceFileUrls() to use managed identity authentication when available - Support 'auto' mode to use first available identity or specific client ID - Fall back to SAS token authentication when managed identity is not available This allows Azure Batch tasks to download resource files using pool-assigned managed identities instead of SAS tokens, improving security and eliminating the need to manage token expiration. Signed-off-by: adamrtalbot <[email protected]>
…atch Signed-off-by: adamrtalbot <[email protected]>
✅ Deploy Preview for nextflow-docs-staging canceled.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
feat: Use Azure Managed identities to download and upload files
When using Managed Identities on Azure Batch, Nextflow still needed to generate a SAS token for the auxiliary files such as the .command.* files. This PR removes that requirement and means Nextflow only uses the managed identity attached to the machine.
It does this by changing the resource file and output properties of the task from a normal URL including a SAS to a URL without a SAS, and configuring the Managed Identity resource ID (ARM) as the authentication.
To do this, we have to perform an additional step where we fetch the resource ID of the managed identity by querying the node pool for available identities and matching them to the client ID. If one does not match, we use the first one as a best guess. Most users should specify a single managed ID or a specific one by client ID, so I think this should be sufficient.
This improves security for Azure Batch and allows users to switch off the ability to create SAS tokens, opening up the use of Nextflow to environments with increased security.