-
Notifications
You must be signed in to change notification settings - Fork 405
Guide: migrating from OSS to Cloud using BACKUP/RESTORE commands #4663
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
23ab174
c5500de
df734d8
b2743ea
77a328d
f463387
a31cd21
55ff192
794d22d
bf436e6
4e354e0
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -20,13 +20,16 @@ import s3_h from '@site/static/images/_snippets/s3/s3-h.png'; | |||||||||
| <details> | ||||||||||
| <summary>Create S3 buckets and an IAM user</summary> | ||||||||||
|
|
||||||||||
| This article demonstrates the basics of how to configure an AWS IAM user, create an S3 bucket and configure ClickHouse to use the bucket as an S3 disk. You should work with your security team to determine the permissions to be used, and consider these as a starting point. | ||||||||||
| This article demonstrates the basics of how to configure an AWS IAM user, create an S3 bucket and configure ClickHouse to use the bucket as an S3 disk. | ||||||||||
| You should work with your security team to determine the permissions to be used, and consider these as a starting point. | ||||||||||
|
|
||||||||||
| ### Create an AWS IAM user {#create-an-aws-iam-user} | ||||||||||
|
|
||||||||||
| In this procedure, we'll be creating a service account user, not a login user. | ||||||||||
|
|
||||||||||
| 1. Log into the AWS IAM Management Console. | ||||||||||
|
|
||||||||||
| 2. In "users", select **Add users** | ||||||||||
| 2. In the `Users`, select `Create user` | ||||||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
or
Suggested change
|
||||||||||
|
|
||||||||||
| <Image size="md" img={s3_1} alt="AWS IAM Management Console - Adding a new user" border force/> | ||||||||||
|
|
||||||||||
|
|
||||||||||
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -8,47 +8,56 @@ | |||||||||||||||||||||||
| --- | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| import Image from '@theme/IdealImage'; | ||||||||||||||||||||||||
| import secure_s3 from '@site/static/images/cloud/security/secures3.jpg'; | ||||||||||||||||||||||||
| import secure_s3 from '@site/static/images/cloud/security/secures3.png'; | ||||||||||||||||||||||||
| import s3_info from '@site/static/images/cloud/security/secures3_arn.png'; | ||||||||||||||||||||||||
| import s3_output from '@site/static/images/cloud/security/secures3_output.jpg'; | ||||||||||||||||||||||||
| import s3_output from '@site/static/images/cloud/security/secures3_output.png'; | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| This article demonstrates how ClickHouse Cloud customers can leverage role-based access to authenticate with Amazon Simple Storage Service (S3) and access their data securely. | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| ## Introduction {#introduction} | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| Before diving into the setup for secure S3 access, it is important to understand how this works. Below is an overview of how ClickHouse services can access private S3 buckets by assuming into a role within customers' AWS account. | ||||||||||||||||||||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| <br/> | ||||||||||||||||||||||||
| <Image img={secure_s3} size="md" alt="Overview of Secure S3 Access with ClickHouse"/> | ||||||||||||||||||||||||
| <br/> | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| This approach allows customers to manage all access to their S3 buckets in a single place (the IAM policy of the assumed-role) without having to go through all of their bucket policies to add or remove access. | ||||||||||||||||||||||||
| In the section below, you will learn how to set this up. | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| ## Setup {#setup} | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| ### Obtaining the ClickHouse service IAM role ARN {#obtaining-the-clickhouse-service-iam-role-arn} | ||||||||||||||||||||||||
| ## Obtain the IAM role ARN of your ClickHouse service {#obtaining-the-clickhouse-service-iam-role-arn} | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| 1 - Login to your ClickHouse cloud account. | ||||||||||||||||||||||||
| 1. Login to your ClickHouse cloud account. | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| 2 - Select the ClickHouse service you want to create the integration | ||||||||||||||||||||||||
| 2. Select the ClickHouse service you want to create the integration | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| 3 - Select the **Settings** tab | ||||||||||||||||||||||||
| 3. Select the **Settings** tab | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| 4 - Scroll down to the **Network security information** section at the bottom of the page | ||||||||||||||||||||||||
| 4. Scroll down to the **Network security information** section at the bottom of the page | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| 5 - Copy the **Service role ID (IAM)** value belong to the service as shown below. | ||||||||||||||||||||||||
| 5. Copy the **Service role ID (IAM)** value belong to the service as shown below. | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| <Image img={s3_info} size="lg" alt="Obtaining ClickHouse service IAM Role ARN" border /> | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| ### Setting up IAM assume role {#setting-up-iam-assume-role} | ||||||||||||||||||||||||
| ## Set up IAM assume role {#setting-up-iam-assume-role} | ||||||||||||||||||||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| #### Option 1: Deploying with CloudFormation stack {#option-1-deploying-with-cloudformation-stack} | ||||||||||||||||||||||||
| The IAM assume role can be setup in one of two ways: | ||||||||||||||||||||||||
| - [Using CloudFormation stack](#option-1-deploying-with-cloudformation-stack) | ||||||||||||||||||||||||
| - [Manually creating an IAM role](#option-2-manually-create-iam-role) | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| 1 - Login to your AWS Account in the web browser with an IAM user that has permission to create & manage IAM role. | ||||||||||||||||||||||||
| ### Deploying with CloudFormation stack {#option-1-deploying-with-cloudformation-stack} | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| 2 - Visit [this url](https://us-west-2.console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/quickcreate?templateURL=https://s3.us-east-2.amazonaws.com/clickhouse-public-resources.clickhouse.cloud/cf-templates/secure-s3.yaml&stackName=ClickHouseSecureS3) to populate the CloudFormation stack. | ||||||||||||||||||||||||
| 1. Login to your AWS Account in the web browser with an IAM user that has permission to create & manage IAM role. | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| 3 - Enter (or paste) the **IAM Role** belong to the ClickHouse service | ||||||||||||||||||||||||
| 2. Visit the following [CloudFormation URL](https://us-west-2.console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/quickcreate?templateURL=https://s3.us-east-2.amazonaws.com/clickhouse-public-resources.clickhouse.cloud/cf-templates/secure-s3.yaml&stackName=ClickHouseSecureS3) to populate the CloudFormation stack. | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| 4 - Configure the CloudFormation stack. Below is additional information about these parameters. | ||||||||||||||||||||||||
| 3. Enter (or paste) the **service role ID (IAM)** for your service that you obtained earlier into the input titled "ClickHouse Instance Roles" | ||||||||||||||||||||||||
| You can paste the service role ID exactly as it appears in Cloud console. | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| 4. Enter your bucket name in the input titled "Bucket Names". If your bucket URL is `https://ch-docs-s3-bucket.s3.eu-central-1.amazonaws.com/clickhouseS3/` then the bucket name is `ch-docs-s3-bucket`. | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| :::note | ||||||||||||||||||||||||
| Do not put the full bucket ARN but instead just the bucket name only. | ||||||||||||||||||||||||
| ::: | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
|
Comment on lines
+54
to
+59
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||||
| 5. Configure the CloudFormation stack. Below is additional information about these parameters. | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| | Parameter | Default Value | Description | | ||||||||||||||||||||||||
| | :--- | :----: | :---- | | ||||||||||||||||||||||||
|
|
@@ -58,29 +67,27 @@ | |||||||||||||||||||||||
| | Bucket Access | Read | Sets the level of access for the provided buckets. | | ||||||||||||||||||||||||
| | Bucket Names | | Comma separated list of **bucket names** that this role will have access to. | | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| *Note*: Do not put the full bucket Arn but instead just the bucket name only. | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| 5 - Select the **I acknowledge that AWS CloudFormation might create IAM resources with custom names.** checkbox | ||||||||||||||||||||||||
| 6. Select the **I acknowledge that AWS CloudFormation might create IAM resources with custom names.** checkbox | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| 6 - Click **Create stack** button at bottom right | ||||||||||||||||||||||||
| 7. Click the **Create stack** button at the bottom right | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| 7 - Make sure the CloudFormation stack completes with no error. | ||||||||||||||||||||||||
| 8. Make sure the CloudFormation stack completes with no error. | ||||||||||||||||||||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| 8 - Select the **Outputs** of the CloudFormation stack | ||||||||||||||||||||||||
| 9. Select the newly created Stack then select the **Outputs** tab of the CloudFormation stack | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| 9 - Copy the **RoleArn** value for this integration. This is what needed to access your S3 bucket. | ||||||||||||||||||||||||
| 10. Copy the **RoleArn** value for this integration, which is what you need to access your S3 bucket. | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| <Image img={s3_output} size="lg" alt="CloudFormation stack output showing IAM Role ARN" border /> | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| #### Option 2: Manually create IAM role {#option-2-manually-create-iam-role} | ||||||||||||||||||||||||
| ### Manually create IAM role {#option-2-manually-create-iam-role} | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| 1 - Login to your AWS Account in the web browser with an IAM user that has permission to create & manage IAM role. | ||||||||||||||||||||||||
| 1. Login to your AWS Account in the web browser with an IAM user that has permission to create & manage IAM role. | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| 2 - Browse to IAM Service Console | ||||||||||||||||||||||||
| 2. Browse to the IAM Service Console | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| 3 - Create a new IAM role with the following IAM & Trust policy. | ||||||||||||||||||||||||
| 3. Create a new IAM role with the following IAM & Trust policy | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| Trust policy (Please replace `{ClickHouse_IAM_ARN}` with the IAM Role arn belong to your ClickHouse instance): | ||||||||||||||||||||||||
| Trust policy (Please replace `{ClickHouse_IAM_ARN}` with the IAM Role arn belong to your ClickHouse instance): | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| ```json | ||||||||||||||||||||||||
| { | ||||||||||||||||||||||||
|
|
@@ -127,22 +134,25 @@ | |||||||||||||||||||||||
| } | ||||||||||||||||||||||||
| ``` | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| 4 - Copy the new **IAM Role Arn** after creation. This is what needed to access your S3 bucket. | ||||||||||||||||||||||||
| 4. Copy the new **IAM Role Arn** after creation, which is what is needed to access your S3 bucket. | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| ## Access your S3 bucket with the ClickHouseAccess role {#access-your-s3-bucket-with-the-clickhouseaccess-role} | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| ClickHouse Cloud has a new feature that allows you to specify `extra_credentials` as part of the S3 table function. Below is an example of how to run a query using the newly created role copied from above. | ||||||||||||||||||||||||
| ClickHouse Cloud allows you to specify `extra_credentials` as part of the S3 table function. | ||||||||||||||||||||||||
| Below is an example of how to run a query using the newly created role copied from above. | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| ```sql | ||||||||||||||||||||||||
| DESCRIBE TABLE s3('https://s3.amazonaws.com/BUCKETNAME/BUCKETOBJECT.csv','CSVWithNames',extra_credentials(role_arn = 'arn:aws:iam::111111111111:role/ClickHouseAccessRole-001')) | ||||||||||||||||||||||||
| ``` | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| Below is an example query that uses the `role_session_name` as a shared secret to query data from a bucket. If the `role_session_name` is not correct, this operation will fail. | ||||||||||||||||||||||||
| Below is an example query that uses the `role_session_name` as a shared secret to query data from a bucket. | ||||||||||||||||||||||||
| If the `role_session_name` is not correct, this operation will fail. | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| ```sql | ||||||||||||||||||||||||
| DESCRIBE TABLE s3('https://s3.amazonaws.com/BUCKETNAME/BUCKETOBJECT.csv','CSVWithNames',extra_credentials(role_arn = 'arn:aws:iam::111111111111:role/ClickHouseAccessRole-001', role_session_name = 'secret-role-name')) | ||||||||||||||||||||||||
| ``` | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| :::note | ||||||||||||||||||||||||
| We recommend that your source S3 is in the same region as your ClickHouse Cloud Service to reduce on data transfer costs. For more information, refer to [S3 pricing]( https://aws.amazon.com/s3/pricing/) | ||||||||||||||||||||||||
| We recommend that your source S3 is in the same region as your ClickHouse Cloud Service to reduce on data transfer costs. | ||||||||||||||||||||||||
| For more information, refer to [S3 pricing]( https://aws.amazon.com/s3/pricing/) | ||||||||||||||||||||||||
| ::: | ||||||||||||||||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit - procedure wording sounds awkward here.