Skip to content
349 changes: 227 additions & 122 deletions platform-cloud/docs/compute-envs/aws-batch.md

Large diffs are not rendered by default.

7 changes: 3 additions & 4 deletions platform-cloud/docs/compute-envs/eks.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ After you have prepared your Kubernetes cluster and assigned a service account r
1. Add the IAM user **Access key** and **Secret key**. This is the IAM user with the service account role detailed in the previous section.
1. (Optional) Under **Assume role**, specify the IAM role to be assumed by the Seqera IAM user to access the compute environment AWS resources.
:::note
When using AWS keys without an assumed role, the associated AWS user account must have Seqera [Launch](https://github.com/seqeralabs/nf-tower-aws/tree/master/launch) and [Forge](https://github.com/seqeralabs/nf-tower-aws/tree/master/forge) permissions. When an assumed role is provided, the keys are only used to retrieve temporary credentials impersonating the role specified. In this case, Seqera [Launch](https://github.com/seqeralabs/nf-tower-aws/tree/master/launch) and [Forge](https://github.com/seqeralabs/nf-tower-aws/tree/master/forge) permissions must be granted to the role instead of the user account.
When using AWS keys without an assumed role, the associated AWS user account must have all the appropriate [IAM permissions](./aws-batch.md#required-platform-iam-permissions). When an assumed role is provided, the keys are only used to retrieve temporary credentials impersonating the role specified: in this case, the permissions must be granted to the role instead of the user account, and the user must have the `sts:AssumeRole` permission for the role (see [AWS Batch IAM role creation (optional)](./aws-batch.md#iam-role-creation-optional)).
:::
1. Select a **Region**, e.g., _eu-west-1 - Europe (Ireland)_.
1. Select a **Cluster name** from the list of available EKS clusters in the selected region.
Expand All @@ -85,9 +85,9 @@ After you have prepared your Kubernetes cluster and assigned a service account r
1. Apply [**Resource labels**](../resource-labels/overview) to the cloud resources consumed by this compute environment. Workspace default resource labels are prefilled.
1. Expand **Staging options** to include:
- Optional [pre- or post-run Bash scripts](../launch/advanced#pre-and-post-run-scripts) that execute before or after the Nextflow pipeline execution in your environment.
- Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Values defined here are pre-filled in the **Nextflow config file** field in the pipeline launch form. These values can be overridden during pipeline launch.
- Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Values defined here are pre-filled in the **Nextflow config file** field in the pipeline launch form. These values can be overridden during pipeline launch.
:::info
Configuration settings in this field override the same values in the pipeline repository `nextflow.config` file. See [Nextflow config file](../launch/advanced#nextflow-config-file) for more information on configuration priority.
Configuration settings in this field override the same values in the pipeline repository `nextflow.config` file. See [Nextflow config file](../launch/advanced#nextflow-config-file) for more information on configuration priority.
:::
1. Specify custom **Environment variables** for the **Head job** and/or **Compute jobs**.
1. Configure any advanced options described in the next section, as needed.
Expand Down Expand Up @@ -191,4 +191,3 @@ To use [Fusion v2](https://docs.seqera.io/fusion) in your Seqera EKS compute env
See the [AWS documentation](https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html) for further details.

</details>

2 changes: 1 addition & 1 deletion platform-cloud/docs/data/data-explorer.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Data Explorer lists public and private data repositories. Repositories accessibl

- **Retrieve data repositories with workspace credentials**

Private data repositories accessible to the credentials defined in your workspace are listed in Data Explorer automatically. The permissions required for your [AWS](../compute-envs/aws-batch#iam), [Google Cloud](../compute-envs/google-cloud-batch#iam), [Azure Batch](../compute-envs/azure-batch#storage-account), or Amazon S3-compatible API storage: credentials allow full Data Explorer functionality.
Private data repositories accessible to the credentials defined in your workspace are listed in Data Explorer automatically. The permissions required for your [AWS](../compute-envs/aws-batch#required-platform-iam-permissions), [Google Cloud](../compute-envs/google-cloud-batch#iam), [Azure Batch](../compute-envs/azure-batch#storage-account), or Amazon S3-compatible API storage: credentials allow full Data Explorer functionality.

- **Configure individual data repositories manually**

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,11 @@ import TabItem from '@theme/TabItem';
This page describes how to set up AWS roles and Batch queues manually for the deployment of Nextflow workloads with Seqera Platform.

:::tip
Manual AWS Batch configuration is only necessary if you don't use Batch Forge.

Batch Forge _automatically creates_ the AWS Batch queues required for your workflow executions.
Manual AWS Batch configuration is only necessary if you don't want to let Seqera Platform create the required AWS Batch resources in your AWS account automatically, done using the internal tool called Batch Forge.
:::

Complete the following procedures to configure AWS Batch manually:
Complete the following steps to configure the AWS Batch resources needed by Seqera Platform:

1. Create a user policy.
2. Create the instance role policy.
3. Create the AWS Batch service role.
4. Create an EC2 Instance role.
Expand All @@ -27,39 +24,6 @@ Complete the following procedures to configure AWS Batch manually:
7. Create the AWS Batch compute environments.
8. Create the AWS Batch queue.

### Create a user policy

Create the policy for the user launching Nextflow jobs:

1. In the [IAM Console](https://console.aws.amazon.com/iam/home), select **Create policy** from the Policies page.
1. Create a new policy with the following content:

```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1530313170000",
"Effect": "Allow",
"Action": [
"batch:CancelJob",
"batch:RegisterJobDefinition",
"batch:DescribeComputeEnvironments",
"batch:DescribeJobDefinitions",
"batch:DescribeJobQueues",
"batch:DescribeJobs",
"batch:ListJobs",
"batch:SubmitJob",
"batch:TerminateJob"
],
"Resource": ["*"]
}
]
}
```

1. Save with it the name `seqera-user`.

### Create the instance role policy

Create the policy with a role that allows Seqera to submit Batch jobs on your EC2 instances:
Expand Down Expand Up @@ -174,7 +138,7 @@ Create a launch template to configure the EC2 instances deployed by Batch jobs:
-a fetch-config \
-m ec2 \
-s \
-c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json
-c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json
mkdir -p /scratch/fusion
NVME_DISKS=($(nvme list | grep 'Amazon EC2 NVMe Instance Storage' | awk '{ print $1 }'))
NUM_DISKS=${#NVME_DISKS[@]}
Expand Down Expand Up @@ -276,8 +240,8 @@ Create a launch template to configure the EC2 instances deployed by Batch jobs:

### Create the Batch compute environments

:::caution
AWS Graviton instances (ARM64 CPU architecture) are not supported in manual compute environments. To use Graviton instances, create your AWS Batch compute environment with [Batch Forge](../../compute-envs/aws-batch#batch-forge-compute-environment).
:::caution
AWS Graviton instances (ARM64 CPU architecture) are not supported in manual compute environments. To use Graviton instances, create your AWS Batch compute environment with [Batch Forge](../../compute-envs/aws-batch#create-a-seqera-aws-batch-compute-environment).
:::

Nextflow makes use of two job queues during workflow execution:
Expand All @@ -301,7 +265,7 @@ The head queue requires an on-demand compute environment. Do not select **Use Sp
1. In the [Batch Console](https://eu-west-1.console.aws.amazon.com/batch/home), select **Create** on the Compute environments page.
1. Select **Amazon EC2** as the compute environment configuration.
:::note
Seqera AWS Batch compute environments created with [Batch Forge](../../compute-envs/aws-batch#batch-forge-compute-environment) support using Fargate for the head job, but manual compute environments must use EC2.
Seqera AWS Batch compute environments created with [Batch Forge](../../compute-envs/aws-batch#create-a-seqera-aws-batch-compute-environment) support using Fargate for the head job, but manual compute environments must use EC2.
:::
1. Enter a name of your choice, and apply the `seqera-servicerole` and `seqera-instancerole`.
1. Enter vCPU limits and instance types, if needed.
Expand All @@ -310,7 +274,7 @@ The head queue requires an on-demand compute environment. Do not select **Use Sp
:::
1. Expand **Additional configuration** and select the `seqera-launchtemplate` from the Launch template dropdown.
1. Configure VPCs, subnets, and security groups on the next page as needed.
1. Review your configuration and select **Create compute environment**.
1. Review your configuration and select **Create compute environment**.

</TabItem>
<TabItem value="Compute queue with Spot instances" label="Compute queue with Spot instances" default>
Expand All @@ -320,11 +284,11 @@ Create this compute environment to use Spot instances for your workflow compute
1. In the [Batch Console](https://eu-west-1.console.aws.amazon.com/batch/home), select **Create** on the Compute environments page.
1. Select **Amazon EC2** as the compute environment configuration.
1. Enter a name of your choice, and apply the `seqera-servicerole` and `seqera-instancerole`.
1. Select **Enable using Spot instances** to use Spot instances and save computing costs.
1. Select **Enable using Spot instances** to use Spot instances and save computing costs.
1. Select the `seqera-fleetrole` and enter vCPU limits and instance types, if needed.
1. Expand **Additional configuration** and select the `seqera-launchtemplate` from the Launch template dropdown.
1. Configure VPCs, subnets, and security groups on the next page as needed.
1. Review your configuration and select **Create compute environment**.
1. Review your configuration and select **Create compute environment**.

</TabItem>
</Tabs>
Expand Down
2 changes: 1 addition & 1 deletion platform-cloud/docs/getting-started/proteinfold.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ You will need the following to get started:

- [Admin](../orgs-and-teams/roles) permissions in an existing organization workspace. See [Set up your workspace](./workspace-setup) to create an organization and workspace from scratch.
- An existing AWS cloud account with access to the AWS Batch service.
- Existing access credentials with permissions to create and manage resources in your AWS account. See [IAM](../compute-envs/aws-batch#iam) for guidance to set up IAM permissions for Platform.
- Existing access credentials with permissions to create and manage resources in your AWS account. See [IAM](../compute-envs/aws-batch#required-platform-iam-permissions) for guidance to set up IAM permissions for Platform.
:::

## Compute environment
Expand Down
2 changes: 1 addition & 1 deletion platform-cloud/docs/getting-started/rnaseq.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ You will need the following to get started:

- [Admin](../orgs-and-teams/roles) permissions in an existing organization workspace. See [Set up your workspace](./workspace-setup) to create an organization and workspace from scratch.
- An existing AWS cloud account with access to the AWS Batch service.
- Existing access credentials with permissions to create and manage resources in your AWS account. See [IAM](../compute-envs/aws-batch#iam) for guidance to set up IAM permissions for Platform.
- Existing access credentials with permissions to create and manage resources in your AWS account. See [IAM](../compute-envs/aws-batch#required-platform-iam-permissions) for guidance to set up IAM permissions for Platform.
:::

## Compute environment
Expand Down
10 changes: 5 additions & 5 deletions platform-cloud/docs/getting-started/studios.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ This guide explores how Studios integrates with your existing workflows, bridgin
You will need the following to get started:

- At least the **Maintain** workspace [user role](../orgs-and-teams/roles) to create and configure Studios.
- An [AWS Batch compute environment](../compute-envs/aws-batch#batch-forge-compute-environment) (**without Fargate**) with sufficient resources (minimum: 2 CPUs, 8192 MB RAM).
- An [AWS Batch compute environment](../compute-envs/aws-batch#create-a-seqera-aws-batch-compute-environment) (**without Fargate**) with sufficient resources (minimum: 2 CPUs, 8192 MB RAM).
- Valid [credentials](../credentials/overview) for your cloud storage account and compute environment.
- [Data Explorer](../data/data-explorer) enabled in your workspace.
:::
Expand All @@ -32,7 +32,7 @@ This script and instructions can also be used to visualize the structures from *

#### Create an AWS Batch compute environment

Studios require an AWS Batch compute environment. If you do not have an existing compute environment available, [create one](../compute-envs/aws-batch#batch-forge-compute-environment) with the following attributes:
Studios require an AWS Batch compute environment. If you do not have an existing compute environment available, [create one](../compute-envs/aws-batch#create-a-seqera-aws-batch-compute-environment) with the following attributes:

- **Region**: To minimize costs, your compute environment should be in the same region as your data. To browse the nf-core AWS megatests public data optimally, select **eu-west-1**.
- **Provisioning model**: Use **On-demand** EC2 instances.
Expand Down Expand Up @@ -366,7 +366,7 @@ An R-IDE enables interactive analysis using R libraries and tools. For example,

#### Create an AWS Batch compute environment

Studios require an AWS Batch compute environment. If you do not have an existing compute environment available, [create one](../compute-envs/aws-batch#batch-forge-compute-environment) with the following attributes:
Studios require an AWS Batch compute environment. If you do not have an existing compute environment available, [create one](../compute-envs/aws-batch#create-a-seqera-aws-batch-compute-environment) with the following attributes:

- **Region**: To minimize costs, your compute environment should be in the same region as your data. To browse the nf-core AWS megatests public data optimally, select **eu-west-1**.
- **Provisioning model**: Use **On-demand** EC2 instances.
Expand Down Expand Up @@ -463,7 +463,7 @@ Xpra provides remote desktop functionality that enables many interactive analysi

#### Create an AWS Batch compute environment

Studios require an AWS Batch compute environment. If you do not have an existing compute environment available, [create one](../compute-envs/aws-batch#batch-forge-compute-environment) with the following attributes:
Studios require an AWS Batch compute environment. If you do not have an existing compute environment available, [create one](../compute-envs/aws-batch#create-a-seqera-aws-batch-compute-environment) with the following attributes:

- **Region**: To minimize costs, your compute environment should be in the same region as your data. To browse the 1000 Genomes public data optimally, select **us-east-1**.
- **Provisioning model**: Use **On-demand** EC2 instances.
Expand Down Expand Up @@ -531,7 +531,7 @@ Using Studios and Visual Studio Code allows you to create a portable and interac

#### Create an AWS Batch compute environment

Studios require an AWS Batch compute environment. If you do not have an existing compute environment available, [create one](../compute-envs/aws-batch#batch-forge-compute-environment) with the following attributes:
Studios require an AWS Batch compute environment. If you do not have an existing compute environment available, [create one](../compute-envs/aws-batch#create-a-seqera-aws-batch-compute-environment) with the following attributes:

- **Region**: To minimize costs, your compute environment should be in the same region as your data. To use the iGenomes public data bucket that contains the *nf-core/fetchngs* `test` profile data, select **eu-west-1**.
- **Provisioning model**: Use **On-demand** EC2 instances.
Expand Down
Loading