From df3f74366323f635ce1a913d454d0cbcd32a4558 Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Sun, 29 Jun 2025 12:04:21 -0500 Subject: [PATCH 01/62] Restructured "Is Migration Assistant Right for you?" page. Signed-off-by: Brian Presley --- .../getting-started-data-migration.md | 17 +------- _migration-assistant/overview/index.md | 6 +-- .../is-migration-assistant-right-for-you.md | 43 ++++++++++--------- 3 files changed, 27 insertions(+), 39 deletions(-) diff --git a/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md b/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md index 856d2e53aa5..5e0ac5dcf59 100644 --- a/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md +++ b/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md @@ -12,22 +12,7 @@ redirect_from: This quickstart outlines how to deploy Migration Assistant for OpenSearch and execute an existing data migration using `Reindex-from-Snapshot` (RFS). It uses AWS for illustrative purposes. However, the steps can be modified for use with other cloud providers. -## Prerequisites and assumptions - -Before using this quickstart, make sure you fulfill the following prerequisites: - -* Verify that your migration path [is supported]({{site.url}}{{site.baseurl}}/migration-assistant/overview/is-migration-assistant-right-for-you/#supported-migration-paths). Note that we test with the exact versions specified, but you should be able to migrate data on alternative minor versions as long as the major version is supported. -* The source cluster must be deployed Amazon Simple Storage Service (Amazon S3) plugin. -* The target cluster must be deployed. -* Verify that the `CDKToolkit` stack exists and is set to `CREATE_COMPLETE`. For more information about how to bootstrap your AWS account in the required AWS Region, see [the CDKToolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html). - -The steps in this guide assume the following: - -* In this guide, a snapshot will be taken and stored in Amazon S3; the following assumptions are made about this snapshot: - * The `_source` flag is enabled on all indexes to be migrated. - * The snapshot includes the global cluster state (`include_global_state` is `true`). - * Shard sizes of up to approximately 80 GB are supported. Larger shards cannot be migrated. If this presents challenges for your migration, contact the [migration team](https://opensearch.slack.com/archives/C054JQ6UJFK). -* Migration Assistant will be installed in the same AWS Region and have access to both the source snapshot and target cluster. +Before using this quickstart, make sure you review ["Is Migration Assistant right for you?"]({{site.url}}{{site.baseurl}}/migration-assistant/overview/is-migration-assistant-right-for-you/#supported-migration-paths). --- diff --git a/_migration-assistant/overview/index.md b/_migration-assistant/overview/index.md index 7962fe5e85d..11c7d716ef5 100644 --- a/_migration-assistant/overview/index.md +++ b/_migration-assistant/overview/index.md @@ -6,15 +6,15 @@ has_children: true has_toc: false permalink: /migration-assistant/overview/ items: + - heading: "Is Migration Assistant right for you?" + description: "Evaluate whether Migration Assistant is right for your use case." + link: "/migration-assistant/overview/is-migration-assistant-right-for-you/" - heading: "Key components" description: "Get familiar with the key components of Migration Assistant." link: "/migration-assistant/overview/key-components/" - heading: "Architecture" description: "Understand how Migration Assistant integrates into your infrastructure." link: "/migration-assistant/overview/architecture/" - - heading: "Is Migration Assistant right for you?" - description: "Evaluate whether Migration Assistant is right for your use case." - link: "/migration-assistant/overview/is-migration-assistant-right-for-you/" --- # Migration Assistant overview diff --git a/_migration-assistant/overview/is-migration-assistant-right-for-you.md b/_migration-assistant/overview/is-migration-assistant-right-for-you.md index 2d9a4a20508..308e461de3c 100644 --- a/_migration-assistant/overview/is-migration-assistant-right-for-you.md +++ b/_migration-assistant/overview/is-migration-assistant-right-for-you.md @@ -9,13 +9,26 @@ redirect_from: # Is Migration Assistant right for you? -Deciding whether to use Migration Assistant depends on your specific upgrade path, infrastructure complexity, and operational goals. This page will help you evaluate whether Migration Assistant is right for your use case—or whether another tool might be a better fit. +Deciding whether to use Migration Assistant depends on your specific upgrade path, infrastructure complexity, and operational goals. This page will help you evaluate whether Migration Assistant is right for your case. -Migration Assistant was built to fill important gaps in common migration strategies. For example, if you're upgrading across multiple major versions—such as from Elasticsearch 6.8 to OpenSearch 2.19—Migration Assistant lets you do this in a single step. Other methods, like rolling upgrades or snapshot restores, require you to upgrade through each major version, often reindexing your data at every step. +Migration Assistant was built to address limitations in common migration strategies. For example, if you're upgrading across multiple major versions—such as from Elasticsearch 6.8 to OpenSearch 2.19—Migration Assistant lets you do this in a single step. Other methods, like rolling upgrades or snapshot restores, require you to upgrade through each major version, often reindexing your data at every step. -Migration Assistant also supports live traffic replication, allowing for zero-downtime migrations. This makes it a strong choice for production environments, where minimizing service disruption is critical. +Migration Assistant also supports live traffic replication, allowing for zero-downtime migrations. This makes it a strong choice for environments where minimizing service disruption is critical. -If your migration is limited to a static cluster configuration (like index templates and aliases), or if you're not concerned about downtime, simpler tools may be sufficient. But for complex migrations involving real-time traffic or major version jumps, Migration Assistant offers robust, flexible capabilities. +## Migration Assistant Assumptions and Limitations + +Before using Migration Assistant, review the following assumptions and limitations. If a limitation only, applies to a specific component (i.e., Capture-and-Replay or Reindex-from-Snapshot) it is specified. + +* If deploying to AWS, the Idendity used to deploy Migration Assistant must have permissions to install all resources used by Migration Assistant. For a full list of resource refer to the AWS documentation, [AWS Services in this Solution](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/architecture-details.html#aws-services-in-this-solution). +* *Reindex-from-Snapshot only.* The source cluster must be deployed Amazon Simple Storage Service (Amazon S3) plugin for Reindex-from-Snapshot. +* The target cluster must be deployed and reachable by Migration Assistant by Reindex-from-Snapshot resources for backfill and the Traffic Replayer for live capture. +* *Capture-and-Replay.* The Traffic Capture Proxy must be deployed to intercept client traffic relevant for migration. The proxy must. +* *Reindex-from-Snapshot.* The snapshot includes the global cluster state (`include_global_state` is `true`). +* By default shards up to 80GB are supported but larger shards are permitted if configured. There is, however, a hard limitation of 80GB shards if the source is in an AWS GovCloud region. +* *Capture-and-Replay.* Live capture should be liitated to 4TB/day of network traffic. +* *Capture-and-Replay.* Currently, for index requests, automatically generated document IDs will not be preserved during the migration. Clients must specify document IDs when writing or updating indecing if using Capture-and-Replay. +* The `_source` flag is enabled on all indexes to be migrated. Refer to [Source documentation]({{site.url}}{{site.baseurl}}/field-types/metadata-fields/source/) for details. +* If deploying to AWS, verify `CDKToolkit` stack exists and is set to `CREATE_COMPLETE`. For more information about how to bootstrap your AWS account in the required AWS Region, see [the CDKToolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html). ## Supported migration paths @@ -34,7 +47,7 @@ The following matrix shows which source versions can be directly migrated to whi - + {% for target in unique_targets %} {% endfor %} @@ -59,28 +72,18 @@ The following matrix shows which source versions can be directly migrated to whi **Source and target platforms** - Self-managed (on premises or hosted by a cloud provider) -- Amazon OpenSearch Service - - -**AWS Regions** - -Migration Assistant is supported in the following AWS Regions: - -- US East (N. Virginia, Ohio) -- US West (Oregon, N. California) -- Europe (Frankfurt, Ireland, London) -- Asia Pacific (Tokyo, Singapore, Sydney) -- AWS GovCloud (US-East, US-West)[^1] +- Amazon OpenSearch Service. Amazon OpenSearch Serverless Collections are not supported. -[^1]: In AWS GovCloud (US), `reindex-from-snapshot` (RFS) is limited to shard sizes of 80 GiB or smaller. +**Supported AWS Regions** +Refer to [AWS Supported Regions](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/plan-your-deployment.html#supported-aws-regions) for the official list of supported regions. -## Supported components +## Supported features Before starting a migration, consider the scope of the components involved. The following table outlines components that should potentially be migrated, indicates whether they are supported by Migration Assistant, and provides recommendations. -| Component | Supported | Recommendations | +| Feature | Supported | Recommendations | | :--- |:--- | :--- | | **Documents** | Yes | Migrate existing data with RFS and live traffic with capture and replay. | | **Index settings** | Yes | Migrate with the `Metadata-Migration-Tool`. | From c62887a1e3bb9d11a0f687de68c1f39ab26ab460 Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Sun, 29 Jun 2025 12:38:06 -0500 Subject: [PATCH 02/62] Broke down limitation by component. Minor editorial changes. Signed-off-by: Brian Presley --- .../is-migration-assistant-right-for-you.md | 50 +++++++++++-------- 1 file changed, 30 insertions(+), 20 deletions(-) diff --git a/_migration-assistant/overview/is-migration-assistant-right-for-you.md b/_migration-assistant/overview/is-migration-assistant-right-for-you.md index 308e461de3c..6e530a9dfc9 100644 --- a/_migration-assistant/overview/is-migration-assistant-right-for-you.md +++ b/_migration-assistant/overview/is-migration-assistant-right-for-you.md @@ -9,30 +9,41 @@ redirect_from: # Is Migration Assistant right for you? -Deciding whether to use Migration Assistant depends on your specific upgrade path, infrastructure complexity, and operational goals. This page will help you evaluate whether Migration Assistant is right for your case. +Whether Migration Assistant is right for you depends on your upgrade path, infrastructure complexity, and operational goals. This page will help you evaluate whether Migration Assistant fits your use case. -Migration Assistant was built to address limitations in common migration strategies. For example, if you're upgrading across multiple major versions—such as from Elasticsearch 6.8 to OpenSearch 2.19—Migration Assistant lets you do this in a single step. Other methods, like rolling upgrades or snapshot restores, require you to upgrade through each major version, often reindexing your data at every step. +Migration Assistant addresses key limitations in traditional migration approaches. For example, if you're upgrading across multiple major versions—such as from Elasticsearch 6.8 to OpenSearch 2.19—Migration Assistant enables you to complete the process in a single step. Other methods, like rolling upgrades or snapshot restores, require upgrading through each major version, often with reindexing at every stage. -Migration Assistant also supports live traffic replication, allowing for zero-downtime migrations. This makes it a strong choice for environments where minimizing service disruption is critical. +Migration Assistant also supports live traffic replication, enabling zero-downtime migrations. This makes it a strong fit for environments where minimizing service disruption is critical. -## Migration Assistant Assumptions and Limitations +## Migration Assistant assumptions and limitations -Before using Migration Assistant, review the following assumptions and limitations. If a limitation only, applies to a specific component (i.e., Capture-and-Replay or Reindex-from-Snapshot) it is specified. +Before using Migration Assistant, review the following assumptions and limitations. They are grouped by scope to clarify which apply generally and which are specific to components. -* If deploying to AWS, the Idendity used to deploy Migration Assistant must have permissions to install all resources used by Migration Assistant. For a full list of resource refer to the AWS documentation, [AWS Services in this Solution](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/architecture-details.html#aws-services-in-this-solution). -* *Reindex-from-Snapshot only.* The source cluster must be deployed Amazon Simple Storage Service (Amazon S3) plugin for Reindex-from-Snapshot. -* The target cluster must be deployed and reachable by Migration Assistant by Reindex-from-Snapshot resources for backfill and the Traffic Replayer for live capture. -* *Capture-and-Replay.* The Traffic Capture Proxy must be deployed to intercept client traffic relevant for migration. The proxy must. -* *Reindex-from-Snapshot.* The snapshot includes the global cluster state (`include_global_state` is `true`). -* By default shards up to 80GB are supported but larger shards are permitted if configured. There is, however, a hard limitation of 80GB shards if the source is in an AWS GovCloud region. -* *Capture-and-Replay.* Live capture should be liitated to 4TB/day of network traffic. -* *Capture-and-Replay.* Currently, for index requests, automatically generated document IDs will not be preserved during the migration. Clients must specify document IDs when writing or updating indecing if using Capture-and-Replay. -* The `_source` flag is enabled on all indexes to be migrated. Refer to [Source documentation]({{site.url}}{{site.baseurl}}/field-types/metadata-fields/source/) for details. -* If deploying to AWS, verify `CDKToolkit` stack exists and is set to `CREATE_COMPLETE`. For more information about how to bootstrap your AWS account in the required AWS Region, see [the CDKToolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html). +### General + +- If deploying on AWS, the identity used to deploy Migration Assistant must have permission to install all required resources. For a full list of services, see [AWS Services in this Solution](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/architecture-details.html#aws-services-in-this-solution). +- The target cluster must be deployed and reachable by Migration Assistant components, including the Migration Console, Reindex-from-Snapshot, and Traffic Replayer, depending on which features are used. +- The `_source` field must be enabled on all indexes to be migrated. See [Source documentation]({{site.url}}{{site.baseurl}}/field-types/metadata-fields/source/) for more information. +- If deploying to AWS, ensure the `CDKToolkit` stack exists and is in the `CREATE_COMPLETE` state. For setup instructions, see the [CDK Toolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html). + +### Reindex-from-Snapshot + +- The source cluster must have the Amazon S3 plugin installed. +- Snapshots must include global cluster state (`include_global_state` is `true`). +- Shards up to 80 GiB are supported by default. Larger shards can be configured, except in AWS GovCloud, where 80 GiB is the maximum supported size. + +### Capture-and-Replay + +- The Traffic Capture Proxy must be deployed to intercept relevant client traffic. +- Live capture is recommended only for environments with less than 4 TB/day of incoming traffic to the source cluster. +- Automatically generated document IDs are not preserved for index requests. Clients must explicitly provide document IDs when indexing or updating documents. + +--- ## Supported migration paths -The following matrix shows which source versions can be directly migrated to which OpenSearch target versions: +The matrix below shows which source versions can be directly migrated to which OpenSearch target versions: + {% comment %}First, collect all unique target versions{% endcomment %} @@ -72,16 +83,15 @@ The following matrix shows which source versions can be directly migrated to whi **Source and target platforms** - Self-managed (on premises or hosted by a cloud provider) -- Amazon OpenSearch Service. Amazon OpenSearch Serverless Collections are not supported. - +- Amazon OpenSearch Service (OpenSearch Serverless Collections are not supported) **Supported AWS Regions** -Refer to [AWS Supported Regions](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/plan-your-deployment.html#supported-aws-regions) for the official list of supported regions. +Refer to [AWS Supported Regions](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/plan-your-deployment.html#supported-aws-regions) for the for the full list of supported regions. ## Supported features -Before starting a migration, consider the scope of the components involved. The following table outlines components that should potentially be migrated, indicates whether they are supported by Migration Assistant, and provides recommendations. +Before starting an upgrade or migration, consider the cluster feature to be included. The table below lists what can be migrated using Migration Assistant, whether it is currently supported, and recommendations for how to handle each component. | Feature | Supported | Recommendations | | :--- |:--- | :--- | From b58adc45081075915648f9f8f18f4434a7f30e14 Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Sun, 29 Jun 2025 12:58:32 -0500 Subject: [PATCH 03/62] Removed "Chossing migration approach" since it will be revised and included in a seperate migration patterns page. Updated checklist. Signed-off-by: Brian Presley --- .../is-migration-assistant-right-for-you.md | 58 +++---------------- 1 file changed, 9 insertions(+), 49 deletions(-) diff --git a/_migration-assistant/overview/is-migration-assistant-right-for-you.md b/_migration-assistant/overview/is-migration-assistant-right-for-you.md index 6e530a9dfc9..11d2a3cf10e 100644 --- a/_migration-assistant/overview/is-migration-assistant-right-for-you.md +++ b/_migration-assistant/overview/is-migration-assistant-right-for-you.md @@ -108,59 +108,19 @@ Before starting an upgrade or migration, consider the cluster feature to be incl [^2]: Support for Kibana 5.0 through 7.10.2 migration paths to OpenSearch Dashboards will be added in a future version. Kibana 8 and later are not supported. For more information, see [issue #944](https://github.com/opensearch-project/opensearch-migrations/issues/944). -## Choosing your migration approach - -Use the following checklist to determine which Migration Assistant components best fit your use case. - -### Metadata migration - -Use [metadata migration]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrating-metadata/) if: - -- You need to migrate while mitigating breaking changes between the source and target clusters, such as differences in mappings, settings, aliases, or component templates. -- You want a relatively consistent configuration between the source and target clusters. - -### Backfill migration - -Use [backfill migration]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/backfill/) if: - -- You need to move historical data without disrupting live traffic. -- You want to backfill indexes from a specific point in time without impacting the source cluster. -- You want to verify historical data in the target cluster before switching over. -- You want to backfill using an existing or incremental snapshot. -- You need the fastest backfill option that includes reindexing. -- You want the ability to pause and resume migration. - -### RFS - -Use [RFS]({{site.url}}{{site.baseurl}}/migration-assistant/deploying-migration-assistant/getting-started-data-migration/) if: - -- You already use OpenSearch snapshots for backups. -- You need to migrate documents at scale in parallel, such as with Amazon Elastic Container Service (Amazon ECS). -- You require a data migration path as part of a zero-downtime migration. -- Your AWS Region supports RFS and your shard sizes are within supported limits. - -### Combination of all three - -Use a combination of all three migration types if: - -- You're performing a complex, multi-version migration. -- You require zero downtime and full validation of the target environment. -- You want end-to-end tooling for metadata, data movement, and cluster behavior comparison. -- You're cloning an existing cluster and changing the source's configuration. -- You're setting up disaster recovery. ## Checklist -Use this checklist to decide whether Migration Assistant is right for you: - -- Are you migrating across one or more major versions? - -- Do you need to maintain service availability with zero downtime? - -- Do you need to validate a new OpenSearch cluster before switching over? +Use this checklist to determine whether Migration Assistant is the right fit for your migration: +- Are you migrating across one or more major versions—for example, from Elasticsearch 5 to OpenSearch 3—in a single step? +- Are you upgrading but want the ability to safely back out, reducing the risk of data loss or service disruption? +- Do you need to maintain high service availability with minimal or zero downtime? +- Do you need to validate a new OpenSearch cluster before switching over, with rollback capabilities? - Is your environment self-managed or running on Amazon OpenSearch Service? - -- Are you looking for tooling that can automate metadata migration and performance comparison? +- Are you looking for tooling to migrate index settings and other metadata? +- Do you need to reconfigure your target cluster—for example, by changing the sharding strategy and reindexing? +- Are you migrating across regions, from on-premises, or from another cloud provider? +- Do you need a high-performance backfill solution that can reliably reindex documents—with support for pause, resume, or checkpoint recovery? If you answered "yes" to most of these questions, Migration Assistant is likely the right solution for your migration. From 84a25c1a6b883cf8367ec8c05f6a6223e4baa448 Mon Sep 17 00:00:00 2001 From: Archer Date: Tue, 1 Jul 2025 06:58:03 -0500 Subject: [PATCH 04/62] Rename Migration Assistant section Signed-off-by: Archer --- _config.yml | 8 +- _migration-assistant/index.md | 44 ------ .../assets/css/breaking-changes-selector.css | 0 .../assets/js/breaking-changes-data.js | 0 .../assets/js/breaking-changes-filter.js | 0 .../assets/js/breaking-changes-index.js | 0 .../assets/js/breaking-changes-module.js | 0 .../assets/js/breaking-changes-ui.js | 0 .../configuration-options.md | 1 + .../getting-started-data-migration.md | 1 + ...d-security-groups-for-existing-clusters.md | 1 + .../deploying-migration-assistant/index.md | 0 _migration-upgrade/index.md | 132 ++++++++++++++++++ .../accessing-the-migration-console.md | 1 + .../migration-console/index.md | 0 .../migration-console-commands-references.md | 1 + .../migration-phases/backfill.md | 1 + .../migration-phases/index.md | 0 .../live-traffic-migration/index.md | 1 + ...itching-traffic-from-the-source-cluster.md | 1 + .../using-traffic-replayer.md | 1 + .../migration-phases/migrating-metadata.md | 1 + .../assessing-your-cluster-for-migration.md | 1 + .../handling-field-type-breaking-changes.md | 1 + .../handling-type-mapping-deprecation.md | 1 + .../planning-your-migration/index.md | 1 + .../verifying-migration-tools.md | 1 + .../removing-migration-infrastructure.md | 1 + .../overview/architecture.md | 1 + .../overview/index.md | 0 .../is-migration-assistant-right-for-you.md | 1 + .../overview/key-components.md | 1 + 32 files changed, 155 insertions(+), 48 deletions(-) delete mode 100644 _migration-assistant/index.md rename {_migration-assistant => _migration-upgrade}/assets/css/breaking-changes-selector.css (100%) rename {_migration-assistant => _migration-upgrade}/assets/js/breaking-changes-data.js (100%) rename {_migration-assistant => _migration-upgrade}/assets/js/breaking-changes-filter.js (100%) rename {_migration-assistant => _migration-upgrade}/assets/js/breaking-changes-index.js (100%) rename {_migration-assistant => _migration-upgrade}/assets/js/breaking-changes-module.js (100%) rename {_migration-assistant => _migration-upgrade}/assets/js/breaking-changes-ui.js (100%) rename {_migration-assistant => _migration-upgrade}/deploying-migration-assistant/configuration-options.md (99%) rename {_migration-assistant => _migration-upgrade}/deploying-migration-assistant/getting-started-data-migration.md (99%) rename {_migration-assistant => _migration-upgrade}/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md (97%) rename {_migration-assistant => _migration-upgrade}/deploying-migration-assistant/index.md (100%) create mode 100644 _migration-upgrade/index.md rename {_migration-assistant => _migration-upgrade}/migration-console/accessing-the-migration-console.md (95%) rename {_migration-assistant => _migration-upgrade}/migration-console/index.md (100%) rename {_migration-assistant => _migration-upgrade}/migration-console/migration-console-commands-references.md (97%) rename {_migration-assistant => _migration-upgrade}/migration-phases/backfill.md (99%) rename {_migration-assistant => _migration-upgrade}/migration-phases/index.md (100%) rename {_migration-assistant => _migration-upgrade}/migration-phases/live-traffic-migration/index.md (96%) rename {_migration-assistant => _migration-upgrade}/migration-phases/live-traffic-migration/switching-traffic-from-the-source-cluster.md (96%) rename {_migration-assistant => _migration-upgrade}/migration-phases/live-traffic-migration/using-traffic-replayer.md (99%) rename {_migration-assistant => _migration-upgrade}/migration-phases/migrating-metadata.md (99%) rename {_migration-assistant => _migration-upgrade}/migration-phases/planning-your-migration/assessing-your-cluster-for-migration.md (97%) rename {_migration-assistant => _migration-upgrade}/migration-phases/planning-your-migration/handling-field-type-breaking-changes.md (97%) rename {_migration-assistant => _migration-upgrade}/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md (99%) rename {_migration-assistant => _migration-upgrade}/migration-phases/planning-your-migration/index.md (93%) rename {_migration-assistant => _migration-upgrade}/migration-phases/planning-your-migration/verifying-migration-tools.md (99%) rename {_migration-assistant => _migration-upgrade}/migration-phases/removing-migration-infrastructure.md (95%) rename {_migration-assistant => _migration-upgrade}/overview/architecture.md (96%) rename {_migration-assistant => _migration-upgrade}/overview/index.md (100%) rename {_migration-assistant => _migration-upgrade}/overview/is-migration-assistant-right-for-you.md (99%) rename {_migration-assistant => _migration-upgrade}/overview/key-components.md (97%) diff --git a/_config.yml b/_config.yml index 5c2ef7212c6..baac4a1c7c6 100644 --- a/_config.yml +++ b/_config.yml @@ -89,7 +89,7 @@ collections: data-prepper: permalink: /:collection/:path/ output: true - migration-assistant: + migration-upgrade: permalink: /:collection/:path/ output: true tools: @@ -215,10 +215,10 @@ clients_collection: name: Clients nav_fold: true -migration_assistant_collection: +migration_upgrade_collection: collections: - migration-assistant: - name: Migration Assistant + migration-upgrade: + name: Migration and upgrade nav_fold: true benchmark_collection: diff --git a/_migration-assistant/index.md b/_migration-assistant/index.md deleted file mode 100644 index 2ce5f0d8d74..00000000000 --- a/_migration-assistant/index.md +++ /dev/null @@ -1,44 +0,0 @@ ---- -layout: default -title: Migration Assistant for OpenSearch -nav_order: 1 -has_children: false -nav_exclude: true -has_toc: false -permalink: /migration-assistant/ -redirect_from: - - /migration-assistant/index/ - - /upgrade-to/index/ - - /upgrade-to/ - - /upgrade-to/upgrade-to/ -tutorial_cards: - - heading: "Overview" - description: "Get familiar with the key components of Migration Assistant and evaluate your use case." - link: "/migration-assistant/overview/" - - heading: "Deploying Migration Assistant" - description: "Follow step-by-step instructions to deploy Migration Assistant and prepare data for migration." - link: "/deploying-migration-assistant/" - - heading: "Migration phases" - description: "Execute your migration in phases—metadata, backfill, and traffic replay—for a controlled and validated transition." - link: "/migration-phases/" - - heading: "Migration console" - description: "Use CLI commands provided by the migration console to orchestrate and monitor your migration process." - link: "/migration-console/" ---- - -# Migration Assistant for OpenSearch - -Migration Assistant for OpenSearch aids you in successfully performing an end-to-end, zero-downtime migration to OpenSearch from other search providers. It helps with the following scenarios: - -- **Metadata migration**: Migrating cluster metadata, such as index settings, aliases, and templates. -- **Backfill migration**: Migrating existing or historical data from a source to a target cluster. -- **Live traffic migration**: Replicating live ongoing traffic from a source to a target cluster. -- **Comparative tooling**: Comparing the performance and behaviors of an existing cluster with a prospective new one. - -This user guide focuses on conducting a comprehensive migration involving both existing and live data with zero downtime and the option to back out of a migration. - -It's crucial to note that migration strategies are not universally applicable. This guide provides a detailed methodology, based on certain assumptions detailed throughout, emphasizing the importance of robust engineering practices to ensure a successful migration. -{: .tip } - -{% include cards.html cards=page.tutorial_cards %} - diff --git a/_migration-assistant/assets/css/breaking-changes-selector.css b/_migration-upgrade/assets/css/breaking-changes-selector.css similarity index 100% rename from _migration-assistant/assets/css/breaking-changes-selector.css rename to _migration-upgrade/assets/css/breaking-changes-selector.css diff --git a/_migration-assistant/assets/js/breaking-changes-data.js b/_migration-upgrade/assets/js/breaking-changes-data.js similarity index 100% rename from _migration-assistant/assets/js/breaking-changes-data.js rename to _migration-upgrade/assets/js/breaking-changes-data.js diff --git a/_migration-assistant/assets/js/breaking-changes-filter.js b/_migration-upgrade/assets/js/breaking-changes-filter.js similarity index 100% rename from _migration-assistant/assets/js/breaking-changes-filter.js rename to _migration-upgrade/assets/js/breaking-changes-filter.js diff --git a/_migration-assistant/assets/js/breaking-changes-index.js b/_migration-upgrade/assets/js/breaking-changes-index.js similarity index 100% rename from _migration-assistant/assets/js/breaking-changes-index.js rename to _migration-upgrade/assets/js/breaking-changes-index.js diff --git a/_migration-assistant/assets/js/breaking-changes-module.js b/_migration-upgrade/assets/js/breaking-changes-module.js similarity index 100% rename from _migration-assistant/assets/js/breaking-changes-module.js rename to _migration-upgrade/assets/js/breaking-changes-module.js diff --git a/_migration-assistant/assets/js/breaking-changes-ui.js b/_migration-upgrade/assets/js/breaking-changes-ui.js similarity index 100% rename from _migration-assistant/assets/js/breaking-changes-ui.js rename to _migration-upgrade/assets/js/breaking-changes-ui.js diff --git a/_migration-assistant/deploying-migration-assistant/configuration-options.md b/_migration-upgrade/deploying-migration-assistant/configuration-options.md similarity index 99% rename from _migration-assistant/deploying-migration-assistant/configuration-options.md rename to _migration-upgrade/deploying-migration-assistant/configuration-options.md index 5fa28e44c43..5032dcb757e 100644 --- a/_migration-assistant/deploying-migration-assistant/configuration-options.md +++ b/_migration-upgrade/deploying-migration-assistant/configuration-options.md @@ -3,6 +3,7 @@ layout: default title: Configuration options nav_order: 15 parent: Deploying Migration Assistant +permalink: /migration-assistant/deploying-migration-assistant/configuration-options/ --- # Configuration options diff --git a/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md b/_migration-upgrade/deploying-migration-assistant/getting-started-data-migration.md similarity index 99% rename from _migration-assistant/deploying-migration-assistant/getting-started-data-migration.md rename to _migration-upgrade/deploying-migration-assistant/getting-started-data-migration.md index 856d2e53aa5..6fb28102840 100644 --- a/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md +++ b/_migration-upgrade/deploying-migration-assistant/getting-started-data-migration.md @@ -3,6 +3,7 @@ layout: default title: Getting started with data migration parent: Deploying Migration Assistant nav_order: 10 +permalink: /migration-assistant/deploying-migration-assistant/getting-started-with-data-migration/ redirect_from: - /upgrade-to/snapshot-migrate/ - /migration-assistant/getting-started-with-data-migration/ diff --git a/_migration-assistant/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md b/_migration-upgrade/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md similarity index 97% rename from _migration-assistant/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md rename to _migration-upgrade/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md index 291c2cba6a5..5ed62887838 100644 --- a/_migration-assistant/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md +++ b/_migration-upgrade/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md @@ -3,6 +3,7 @@ layout: default title: IAM and security groups for existing clusters nav_order: 20 parent: Deploying Migration Assistant +permalink: /migration-assistant/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters/ --- # IAM and security groups for existing clusters diff --git a/_migration-assistant/deploying-migration-assistant/index.md b/_migration-upgrade/deploying-migration-assistant/index.md similarity index 100% rename from _migration-assistant/deploying-migration-assistant/index.md rename to _migration-upgrade/deploying-migration-assistant/index.md diff --git a/_migration-upgrade/index.md b/_migration-upgrade/index.md new file mode 100644 index 00000000000..c44a1b5855d --- /dev/null +++ b/_migration-upgrade/index.md @@ -0,0 +1,132 @@ +--- +layout: default +title: Migration Assistant for OpenSearch +nav_order: 1 +has_children: false +nav_exclude: true +has_toc: false +permalink: /migration-assistant/ +redirect_from: + - /migration-assistant/index/ + - /upgrade-to/index/ + - /upgrade-to/ + - /upgrade-to/upgrade-to/ +tutorial_cards: + - heading: "Overview" + description: "Get familiar with the key components of Migration Assistant and evaluate your use case." + link: "/migration-assistant/overview/" + - heading: "Deploying Migration Assistant" + description: "Follow step-by-step instructions to deploy Migration Assistant and prepare data for migration." + link: "/deploying-migration-assistant/" + - heading: "Migration phases" + description: "Execute your migration in phases—metadata, backfill, and traffic replay—for a controlled and validated transition." + link: "/migration-phases/" + - heading: "Migration console" + description: "Use CLI commands provided by the migration console to orchestrate and monitor your migration process." + link: "/migration-console/" +--- + +# Migration and upgrade options + +Upgrading or migrating OpenSearch is essential for maintaining optimal performance, security, and access to the latest features. Whether you're upgrading an existing OpenSearch deployment or migrating from another system such as Elasticsearch OSS, choosing the right approach is critical to a successful transition. + +This page outlines four primary methods for upgrading or migrating OpenSearch clusters—each with distinct benefits and trade-offs. These methods include rolling upgrades, snapshot and restore, Migration Assistant, and remote reindexing. Use this guide to evaluate which option best fits your data size, infrastructure, and operational requirements. + +## Migration Assistant + +The Migration Assistant is a comprehensive tool designed to simplify upgrades by automating the process and supporting live data capture. + +**Pros:** +- Supports multi-version upgrades in a single migration. +- Enables near-zero downtime using live data capture. +- Includes rollback support to revert changes if issues occur. +- Integrates with existing OpenSearch tooling for a streamlined experience. + +**Cons:** +- Requires additional setup and infrastructure. + +### Why choose Migration Assistant? + +Migration Assistant offers the most automated and resilient path for OpenSearch upgrades, especially for complex environments and large version gaps. + +- **Multi-version hops:** Avoids sequential upgrades by migrating across multiple versions in one step. +- **Live data capture:** Maintains service continuity by syncing changes during the migration. +- **Reversion support:** Provides a fallback option in case of errors or issues. +- **Integrated automation:** Designed to fit into OpenSearch workflows for a guided upgrade experience. + +### Getting started with Migration Assistant + +To get started with Migration Assistant, determine which of the following scenarios best suits your needs: + +- **Metadata migration**: Migrating cluster metadata, such as index settings, aliases, and templates. +- **Backfill migration**: Migrating existing or historical data from a source to a target cluster. +- **Live traffic migration**: Replicating live ongoing traffic from a source to a target cluster. +- **Comparative tooling**: Comparing the performance and behaviors of an existing cluster with a prospective new one. + +It's crucial to note that migration strategies are not universally applicable. This guide provides a detailed methodology, based on certain assumptions detailed throughout, emphasizing the importance of robust engineering practices to ensure a successful migration. +{: .tip } + +{% include cards.html cards=page.tutorial_cards %} + +## Rolling upgrades + +Rolling upgrades allow you to upgrade one node at a time, keeping the cluster operational throughout the process. + +**Pros:** +- Minimal downtime; the cluster remains available during the upgrade. +- No new infrastructure is required. + +**Cons:** +- Only supports adjacent major version upgrades. +- Incompatibilities may still arise, requiring a snapshot restore that can be complex and risky. +- Multiple upgrade cycles are needed for large version jumps. +- Manual reindexing may be required to enable newer features. + + +## Snapshot and restore + +This method involves taking a snapshot of your existing OpenSearch or Elasticsearch OSS cluster and restoring it to a new cluster running the target version. + +**Pros:** +- Original cluster remains untouched, allowing rollback if needed (note: may result in data loss without a change data capture (CDC) solution). +- Scales well for large data volumes, including cold storage and datasets up to 1 PB. + +**Cons:** +- Requires downtime or an external CDC solution. +- Requires provisioning a new cluster. +- Manual reindexing may be necessary for full feature support. + +## Remote reindexing + +This method reindexes data from your current cluster into a new OpenSearch cluster, typically running in parallel. + +**Pros:** +- No downtime; clusters operate concurrently. +- Supports upgrades across multiple versions. + +**Cons:** +- Slow and resource-intensive, not recommended for data sets over 1 GB. +- May impact performance of the source cluster. +- Requires careful configuration and a compatible target cluster. + +## Why choose Migration Assistant? + +Migration Assistant offers the most automated and resilient path for OpenSearch upgrades, especially for complex environments and large version gaps. + +- **Multi-version hops:** Avoids sequential upgrades by migrating across multiple versions in one step. +- **Live data capture:** Maintains service continuity by syncing changes during the migration. +- **Reversion support:** Provides a fallback option in case of errors or issues. +- **Integrated automation:** Designed to fit into OpenSearch workflows for a guided upgrade experience. + +## Before you begin + +Before choosing a method, make sure that your OpenSearch clients and plugins are compatible with the target version. For example, tools like Logstash OSS and Filebeat OSS may enforce version checks that impact upgrade paths. + + +## Next steps + +After you've determined which upgrade or migration path works best for you, take one of the following next steps: + +- Review the [Migration Assistant Overview]({{site.url}}{{site.baseurl}}/migration-assistant/overview/) +- Perform a [rolling upgrade]({{site.url}}{{site.baseurl}}/install-and-configure/upgrade-opensearch/rolling-upgrade/) + diff --git a/_migration-assistant/migration-console/accessing-the-migration-console.md b/_migration-upgrade/migration-console/accessing-the-migration-console.md similarity index 95% rename from _migration-assistant/migration-console/accessing-the-migration-console.md rename to _migration-upgrade/migration-console/accessing-the-migration-console.md index ea66f5c04c9..cf530e3ccd4 100644 --- a/_migration-assistant/migration-console/accessing-the-migration-console.md +++ b/_migration-upgrade/migration-console/accessing-the-migration-console.md @@ -3,6 +3,7 @@ layout: default title: Accessing the migration console nav_order: 35 parent: Migration console +permalink: /migration-assistant/migration-console/accessing-the-migration-console/ --- # Accessing the migration console diff --git a/_migration-assistant/migration-console/index.md b/_migration-upgrade/migration-console/index.md similarity index 100% rename from _migration-assistant/migration-console/index.md rename to _migration-upgrade/migration-console/index.md diff --git a/_migration-assistant/migration-console/migration-console-commands-references.md b/_migration-upgrade/migration-console/migration-console-commands-references.md similarity index 97% rename from _migration-assistant/migration-console/migration-console-commands-references.md rename to _migration-upgrade/migration-console/migration-console-commands-references.md index 21d793b3f3f..a07439e32b8 100644 --- a/_migration-assistant/migration-console/migration-console-commands-references.md +++ b/_migration-upgrade/migration-console/migration-console-commands-references.md @@ -3,6 +3,7 @@ layout: default title: Command reference nav_order: 40 parent: Migration console +permalink: /migration-assistant/migration-console/migration-console-commands-reference/ --- # Migration console command reference diff --git a/_migration-assistant/migration-phases/backfill.md b/_migration-upgrade/migration-phases/backfill.md similarity index 99% rename from _migration-assistant/migration-phases/backfill.md rename to _migration-upgrade/migration-phases/backfill.md index e4b621c16ec..8a98412675b 100644 --- a/_migration-assistant/migration-phases/backfill.md +++ b/_migration-upgrade/migration-phases/backfill.md @@ -3,6 +3,7 @@ layout: default title: Backfill nav_order: 90 parent: Migration phases +permalink: /migration-assistant/migration-phases/backfill/ --- # Backfill diff --git a/_migration-assistant/migration-phases/index.md b/_migration-upgrade/migration-phases/index.md similarity index 100% rename from _migration-assistant/migration-phases/index.md rename to _migration-upgrade/migration-phases/index.md diff --git a/_migration-assistant/migration-phases/live-traffic-migration/index.md b/_migration-upgrade/migration-phases/live-traffic-migration/index.md similarity index 96% rename from _migration-assistant/migration-phases/live-traffic-migration/index.md rename to _migration-upgrade/migration-phases/live-traffic-migration/index.md index f9e3d9f71f0..7cdf7c466d1 100644 --- a/_migration-assistant/migration-phases/live-traffic-migration/index.md +++ b/_migration-upgrade/migration-phases/live-traffic-migration/index.md @@ -3,6 +3,7 @@ layout: default title: Live traffic migration nav_order: 99 parent: Migration phases +permalink: /live-traffic-migration/ has_toc: false has_children: true --- diff --git a/_migration-assistant/migration-phases/live-traffic-migration/switching-traffic-from-the-source-cluster.md b/_migration-upgrade/migration-phases/live-traffic-migration/switching-traffic-from-the-source-cluster.md similarity index 96% rename from _migration-assistant/migration-phases/live-traffic-migration/switching-traffic-from-the-source-cluster.md rename to _migration-upgrade/migration-phases/live-traffic-migration/switching-traffic-from-the-source-cluster.md index e9e477654ab..8ec7f1d929b 100644 --- a/_migration-assistant/migration-phases/live-traffic-migration/switching-traffic-from-the-source-cluster.md +++ b/_migration-upgrade/migration-phases/live-traffic-migration/switching-traffic-from-the-source-cluster.md @@ -4,6 +4,7 @@ title: Switching traffic from the source cluster nav_order: 110 grand_parent: Migration phases parent: Live traffic migration +permalink: /migration-assistant/migration-phases/live-traffic-migration/switching-traffic-from-the-source-cluster/ redirect_from: - /migration-assistant/migration-phases/switching-traffic-from-the-source-cluster/ --- diff --git a/_migration-assistant/migration-phases/live-traffic-migration/using-traffic-replayer.md b/_migration-upgrade/migration-phases/live-traffic-migration/using-traffic-replayer.md similarity index 99% rename from _migration-assistant/migration-phases/live-traffic-migration/using-traffic-replayer.md rename to _migration-upgrade/migration-phases/live-traffic-migration/using-traffic-replayer.md index a1b54d49417..6ebedf0d91a 100644 --- a/_migration-assistant/migration-phases/live-traffic-migration/using-traffic-replayer.md +++ b/_migration-upgrade/migration-phases/live-traffic-migration/using-traffic-replayer.md @@ -4,6 +4,7 @@ title: Using Traffic Replayer nav_order: 100 grand_parent: Migration phases parent: Live traffic migration +permalink: /migration-assistant/migration-phases/live-traffic-migration/switching-traffic-from-the-source-cluster/ redirect_from: - /migration-assistant/migration-phases/using-traffic-replayer/ --- diff --git a/_migration-assistant/migration-phases/migrating-metadata.md b/_migration-upgrade/migration-phases/migrating-metadata.md similarity index 99% rename from _migration-assistant/migration-phases/migrating-metadata.md rename to _migration-upgrade/migration-phases/migrating-metadata.md index 249a2ca4d0c..71e3da9776d 100644 --- a/_migration-assistant/migration-phases/migrating-metadata.md +++ b/_migration-upgrade/migration-phases/migrating-metadata.md @@ -3,6 +3,7 @@ layout: default title: Migrating metadata nav_order: 85 parent: Migration phases +permalink: /migration-assistant/migration-phases/migrating-metadata/ --- # Migrating metadata diff --git a/_migration-assistant/migration-phases/planning-your-migration/assessing-your-cluster-for-migration.md b/_migration-upgrade/migration-phases/planning-your-migration/assessing-your-cluster-for-migration.md similarity index 97% rename from _migration-assistant/migration-phases/planning-your-migration/assessing-your-cluster-for-migration.md rename to _migration-upgrade/migration-phases/planning-your-migration/assessing-your-cluster-for-migration.md index 1687da15893..a86f9f5db5e 100644 --- a/_migration-assistant/migration-phases/planning-your-migration/assessing-your-cluster-for-migration.md +++ b/_migration-upgrade/migration-phases/planning-your-migration/assessing-your-cluster-for-migration.md @@ -4,6 +4,7 @@ title: Assessing your cluster for migration nav_order: 60 parent: Planning your migration grand_parent: Migration phases +permalink: /migration-assistant/migration-phases/planning-your-migration/assessing-your-cluster-for-migration/ redirect_from: - /migration-assistant/migration-phases/assessing-your-cluster-for-migration/ --- diff --git a/_migration-assistant/migration-phases/planning-your-migration/handling-field-type-breaking-changes.md b/_migration-upgrade/migration-phases/planning-your-migration/handling-field-type-breaking-changes.md similarity index 97% rename from _migration-assistant/migration-phases/planning-your-migration/handling-field-type-breaking-changes.md rename to _migration-upgrade/migration-phases/planning-your-migration/handling-field-type-breaking-changes.md index b6a653f96ae..e09da59fd00 100644 --- a/_migration-assistant/migration-phases/planning-your-migration/handling-field-type-breaking-changes.md +++ b/_migration-upgrade/migration-phases/planning-your-migration/handling-field-type-breaking-changes.md @@ -3,6 +3,7 @@ layout: default title: Handling breaking changes in field types nav_order: 60 parent: Planning your migration +permalink: /migration-assistant/migration-phases/planning-your-migration/handling-breaking-changes-in-field-types/ grand_parent: Migration phases --- diff --git a/_migration-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md b/_migration-upgrade/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md similarity index 99% rename from _migration-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md rename to _migration-upgrade/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md index edd81612dac..a39ac1fdda9 100644 --- a/_migration-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md +++ b/_migration-upgrade/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md @@ -4,6 +4,7 @@ title: Managing type mapping deprecation nav_order: 60 parent: Planning your migration grand_parent: Migration phases +permalink: /migration-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation/ --- # Managing type mapping deprecation diff --git a/_migration-assistant/migration-phases/planning-your-migration/index.md b/_migration-upgrade/migration-phases/planning-your-migration/index.md similarity index 93% rename from _migration-assistant/migration-phases/planning-your-migration/index.md rename to _migration-upgrade/migration-phases/planning-your-migration/index.md index 4f0d264e93e..9c6e7fee0bc 100644 --- a/_migration-assistant/migration-phases/planning-your-migration/index.md +++ b/_migration-upgrade/migration-phases/planning-your-migration/index.md @@ -3,6 +3,7 @@ layout: default title: Planning your migration nav_order: 59 parent: Migration phases +permalink: /planning-your-migration/ has_toc: false has_children: true --- diff --git a/_migration-assistant/migration-phases/planning-your-migration/verifying-migration-tools.md b/_migration-upgrade/migration-phases/planning-your-migration/verifying-migration-tools.md similarity index 99% rename from _migration-assistant/migration-phases/planning-your-migration/verifying-migration-tools.md rename to _migration-upgrade/migration-phases/planning-your-migration/verifying-migration-tools.md index 6cfe762c85b..42bde888a1a 100644 --- a/_migration-assistant/migration-phases/planning-your-migration/verifying-migration-tools.md +++ b/_migration-upgrade/migration-phases/planning-your-migration/verifying-migration-tools.md @@ -4,6 +4,7 @@ title: Verifying migration tools nav_order: 70 parent: Planning your migration grand_parent: Migration phases +permalink: /migration-assistant/migration-phases/planning-your-migration/verifying-migration-tools/ redirect_from: - /migration-assistant/migration-phases/verifying-migration-tools/ --- diff --git a/_migration-assistant/migration-phases/removing-migration-infrastructure.md b/_migration-upgrade/migration-phases/removing-migration-infrastructure.md similarity index 95% rename from _migration-assistant/migration-phases/removing-migration-infrastructure.md rename to _migration-upgrade/migration-phases/removing-migration-infrastructure.md index adc2a4b7776..7d677908822 100644 --- a/_migration-assistant/migration-phases/removing-migration-infrastructure.md +++ b/_migration-upgrade/migration-phases/removing-migration-infrastructure.md @@ -3,6 +3,7 @@ layout: default title: Removing migration infrastructure nav_order: 120 parent: Migration phases +permalink: /migration-assistant/migration-phases/removing-migration-infrastructure/ --- # Removing migration infrastructure diff --git a/_migration-assistant/overview/architecture.md b/_migration-upgrade/overview/architecture.md similarity index 96% rename from _migration-assistant/overview/architecture.md rename to _migration-upgrade/overview/architecture.md index 309261e4915..48b3c8fbcfc 100644 --- a/_migration-assistant/overview/architecture.md +++ b/_migration-upgrade/overview/architecture.md @@ -3,6 +3,7 @@ layout: default title: Architecture nav_order: 15 parent: Overview +permalink: /migration-assistant/overview/architecture/ --- # Architecture diff --git a/_migration-assistant/overview/index.md b/_migration-upgrade/overview/index.md similarity index 100% rename from _migration-assistant/overview/index.md rename to _migration-upgrade/overview/index.md diff --git a/_migration-assistant/overview/is-migration-assistant-right-for-you.md b/_migration-upgrade/overview/is-migration-assistant-right-for-you.md similarity index 99% rename from _migration-assistant/overview/is-migration-assistant-right-for-you.md rename to _migration-upgrade/overview/is-migration-assistant-right-for-you.md index 2d9a4a20508..1e7fc287974 100644 --- a/_migration-assistant/overview/is-migration-assistant-right-for-you.md +++ b/_migration-upgrade/overview/is-migration-assistant-right-for-you.md @@ -3,6 +3,7 @@ layout: default title: Is Migration Assistant right for you? nav_order: 10 parent: Overview +permalink: /migration-assistant/overview/is-migration-assistant-right-for-you/ redirect_from: - /migration-assistant/is-migration-assistant-right-for-you/ --- diff --git a/_migration-assistant/overview/key-components.md b/_migration-upgrade/overview/key-components.md similarity index 97% rename from _migration-assistant/overview/key-components.md rename to _migration-upgrade/overview/key-components.md index ca09c7f29e8..cfa69e70311 100644 --- a/_migration-assistant/overview/key-components.md +++ b/_migration-upgrade/overview/key-components.md @@ -3,6 +3,7 @@ layout: default title: Key components nav_order: 10 parent: Overview +permalink: /migration-assistant/overview/key-components/ --- # Key components From a28ad150fb517547b21fc5f3fb4d0f4f8709f098 Mon Sep 17 00:00:00 2001 From: Archer Date: Tue, 1 Jul 2025 07:35:30 -0500 Subject: [PATCH 05/62] Fix broken links Signed-off-by: Archer --- .../getting-started-data-migration.md | 2 +- _migration-upgrade/migration-phases/index.md | 2 +- .../live-traffic-migration/using-traffic-replayer.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/_migration-upgrade/deploying-migration-assistant/getting-started-data-migration.md b/_migration-upgrade/deploying-migration-assistant/getting-started-data-migration.md index 6fb28102840..d0592e2dd24 100644 --- a/_migration-upgrade/deploying-migration-assistant/getting-started-data-migration.md +++ b/_migration-upgrade/deploying-migration-assistant/getting-started-data-migration.md @@ -3,7 +3,7 @@ layout: default title: Getting started with data migration parent: Deploying Migration Assistant nav_order: 10 -permalink: /migration-assistant/deploying-migration-assistant/getting-started-with-data-migration/ +permalink: /migration-assistant/deploying-migration-assistant/getting-started-data-migration/ redirect_from: - /upgrade-to/snapshot-migrate/ - /migration-assistant/getting-started-with-data-migration/ diff --git a/_migration-upgrade/migration-phases/index.md b/_migration-upgrade/migration-phases/index.md index 37d835135fd..a330b9e598e 100644 --- a/_migration-upgrade/migration-phases/index.md +++ b/_migration-upgrade/migration-phases/index.md @@ -11,7 +11,7 @@ redirect_from: # Migration phases -This page details how to conduct a migration with Migration Assistant. It shows you how to [plan for your migration]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/planning-your-migration/index/) and encompasses a variety of migration scenarios, including: +This page details how to conduct a migration with Migration Assistant. It shows you how to [plan for your migration]({{site.url}}{{site.baseurl}}/planning-your-migration/) and encompasses a variety of migration scenarios, including: - [**Metadata migration**]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrating-metadata/): Migrating cluster metadata, such as index settings, aliases, and templates. - [**Backfill migration**]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/backfill/): Migrating existing or historical data from a source to a target cluster. diff --git a/_migration-upgrade/migration-phases/live-traffic-migration/using-traffic-replayer.md b/_migration-upgrade/migration-phases/live-traffic-migration/using-traffic-replayer.md index 6ebedf0d91a..1454f7f04a3 100644 --- a/_migration-upgrade/migration-phases/live-traffic-migration/using-traffic-replayer.md +++ b/_migration-upgrade/migration-phases/live-traffic-migration/using-traffic-replayer.md @@ -4,7 +4,7 @@ title: Using Traffic Replayer nav_order: 100 grand_parent: Migration phases parent: Live traffic migration -permalink: /migration-assistant/migration-phases/live-traffic-migration/switching-traffic-from-the-source-cluster/ +permalink: /migration-assistant/migration-phases/live-traffic-migration/using-traffic-replayer/ redirect_from: - /migration-assistant/migration-phases/using-traffic-replayer/ --- From 9d68c5a221bfccd590bc20025a14f34735429567 Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Tue, 1 Jul 2025 21:27:26 -0500 Subject: [PATCH 06/62] Add addition upgrade options section Signed-off-by: Brian Presley --- _migration-upgrade/index.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/_migration-upgrade/index.md b/_migration-upgrade/index.md index c44a1b5855d..5d17cba894c 100644 --- a/_migration-upgrade/index.md +++ b/_migration-upgrade/index.md @@ -70,7 +70,7 @@ It's crucial to note that migration strategies are not universally applicable. T ## Rolling upgrades -Rolling upgrades allow you to upgrade one node at a time, keeping the cluster operational throughout the process. +Rolling upgrades allow you to upgrade one node at a time, keeping the cluster operational throughout the process. Note that this is only an upgrade option. **Pros:** - Minimal downtime; the cluster remains available during the upgrade. @@ -100,6 +100,10 @@ This method involves taking a snapshot of your existing OpenSearch or Elasticsea This method reindexes data from your current cluster into a new OpenSearch cluster, typically running in parallel. +## Other migration options + +Additional migration strategies not covered in detail in this documentation include rebuilding your target cluster from source systems and using traffic replication to mirror production traffic during migration. + **Pros:** - No downtime; clusters operate concurrently. - Supports upgrades across multiple versions. From b6b1220ce7ee34da2843dcd69331f049337f697a Mon Sep 17 00:00:00 2001 From: Archer Date: Wed, 2 Jul 2025 09:24:05 -0500 Subject: [PATCH 07/62] Fix home cards page Signed-off-by: Archer --- _includes/home_cards.html | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/_includes/home_cards.html b/_includes/home_cards.html index 098472d4c85..1509d5402ea 100644 --- a/_includes/home_cards.html +++ b/_includes/home_cards.html @@ -61,9 +61,9 @@
- -

Migration Assistant

-

Migrate to OpenSearch.

+ +

Migrate and upgrade

+

Migrate or upgrade to OpenSearch.

Documentation →

From 3e364faf9eb3fab5221ab06b9e53b1d7cc1e9fe0 Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Wed, 2 Jul 2025 18:22:32 -0500 Subject: [PATCH 08/62] Updated requirements around VPC connectivity. Signed-off-by: Brian Presley --- .../is-migration-assistant-right-for-you.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/_migration-assistant/overview/is-migration-assistant-right-for-you.md b/_migration-assistant/overview/is-migration-assistant-right-for-you.md index 11d2a3cf10e..0cda220dc7b 100644 --- a/_migration-assistant/overview/is-migration-assistant-right-for-you.md +++ b/_migration-assistant/overview/is-migration-assistant-right-for-you.md @@ -21,10 +21,25 @@ Before using Migration Assistant, review the following assumptions and limitatio ### General + - If deploying on AWS, the identity used to deploy Migration Assistant must have permission to install all required resources. For a full list of services, see [AWS Services in this Solution](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/architecture-details.html#aws-services-in-this-solution). - The target cluster must be deployed and reachable by Migration Assistant components, including the Migration Console, Reindex-from-Snapshot, and Traffic Replayer, depending on which features are used. - The `_source` field must be enabled on all indexes to be migrated. See [Source documentation]({{site.url}}{{site.baseurl}}/field-types/metadata-fields/source/) for more information. - If deploying to AWS, ensure the `CDKToolkit` stack exists and is in the `CREATE_COMPLETE` state. For setup instructions, see the [CDK Toolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html). +- If deploying Migration Assistant into an existing AWS VPC, you must configure VPC interface endpoints and IAM permissions for the following AWS services: + - **Amazon CloudWatch Logs** – for log ingestion from ECS tasks. + - **Amazon CloudWatch** – for metrics published during migration + - **Amazon Elastic Container Registry (ECR)** – for pulling container images. + - **Amazon Elastic Container Service (ECS)** – for task orchestration and migration. + - **Elastic Load Balancing (ELB)** – for routing traffic to Capture Proxy. + - **AWS Secrets Manager** – for storing credentials. + - **AWS Systems Manager Parameter Store** – for storing and accessing parameter values. + - **AWS Systems Manager Session Manager** – for secure shell access to EC2 (i.e., bootstrap instance). + - **Amazon EC2** – for launching the bootstrap instance responsible for build and deployment. + - **Amazon Elastic Block Store (EBS)** – for disk storage during migration. + - **Amazon Virtual Private Cloud (VPC)** – for private networking and VPC endpoints. + - **AWS X-Ray** – for distributed tracing across components. + - **Amazon Elastic File System (EFS)** – for persistent logging. ### Reindex-from-Snapshot From abdfa960c1945ac11fc3fbfefb6c49389882a045 Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Wed, 2 Jul 2025 21:29:36 -0500 Subject: [PATCH 09/62] Restructured migration journey and added migration phases. Signed-off-by: Brian Presley --- _config.yml | 18 +- _includes/home_cards.html | 6 +- .../assets/css/breaking-changes-selector.css | 0 .../assets/js/breaking-changes-data.js | 0 .../assets/js/breaking-changes-filter.js | 0 .../assets/js/breaking-changes-index.js | 0 .../assets/js/breaking-changes-module.js | 0 .../assets/js/breaking-changes-ui.js | 0 _migration-assistant/index.md | 35 ++ .../accessing-the-migration-console.md | 42 ++ .../migration-phases/assessment.md | 76 ++++ .../migration-phases/create-snapshot.md | 244 ++++++++++++ .../configuration-options.md | 233 ++++++++++++ .../getting-started-data-migration.md | 360 ++++++++++++++++++ ...d-security-groups-for-existing-clusters.md | 93 +++++ .../deploying-migration-assistant/index.md | 13 + .../migration-phases/deployment.md | 341 +++++++++++++++++ .../migration-phases/index.md | 87 +++++ .../migration-phases/migrate-metadata.md | 249 ++++++++++++ .../replay-captured-traffic.md | 319 ++++++++++++++++ .../reroute-source-to-proxy.md | 102 +++++ ...te-traffic-from-capture-proxy-to-target.md | 54 +++ .../migration-phases/teardown.md | 30 ++ .../verify-backfill-components.md | 191 ++++++++++ .../verify-live-capture-components.md | 157 ++++++++ _migration-assistant/overview/architecture.md | 25 ++ _migration-assistant/overview/index.md | 25 ++ .../is-migration-assistant-right-for-you.md | 154 ++++++++ .../overview/key-components.md | 39 ++ .../handling-field-type-breaking-changes.md | 158 ++++++++ .../handling-type-mapping-deprecation.md | 318 ++++++++++++++++ .../planning-your-migration/index.md | 15 + _migration-upgrade/index.md | 36 +- .../migration-phases/backfill.md | 97 +---- _migration-upgrade/migration-phases/index.md | 72 +++- 35 files changed, 3456 insertions(+), 133 deletions(-) rename {_migration-upgrade => _migration-assistant}/assets/css/breaking-changes-selector.css (100%) rename {_migration-upgrade => _migration-assistant}/assets/js/breaking-changes-data.js (100%) rename {_migration-upgrade => _migration-assistant}/assets/js/breaking-changes-filter.js (100%) rename {_migration-upgrade => _migration-assistant}/assets/js/breaking-changes-index.js (100%) rename {_migration-upgrade => _migration-assistant}/assets/js/breaking-changes-module.js (100%) rename {_migration-upgrade => _migration-assistant}/assets/js/breaking-changes-ui.js (100%) create mode 100644 _migration-assistant/index.md create mode 100644 _migration-assistant/migration-phases/accessing-the-migration-console.md create mode 100644 _migration-assistant/migration-phases/assessment.md create mode 100644 _migration-assistant/migration-phases/create-snapshot.md create mode 100644 _migration-assistant/migration-phases/deploying-migration-assistant/configuration-options.md create mode 100644 _migration-assistant/migration-phases/deploying-migration-assistant/getting-started-data-migration.md create mode 100644 _migration-assistant/migration-phases/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md create mode 100644 _migration-assistant/migration-phases/deploying-migration-assistant/index.md create mode 100644 _migration-assistant/migration-phases/deployment.md create mode 100644 _migration-assistant/migration-phases/index.md create mode 100644 _migration-assistant/migration-phases/migrate-metadata.md create mode 100644 _migration-assistant/migration-phases/replay-captured-traffic.md create mode 100644 _migration-assistant/migration-phases/reroute-source-to-proxy.md create mode 100644 _migration-assistant/migration-phases/reroute-traffic-from-capture-proxy-to-target.md create mode 100644 _migration-assistant/migration-phases/teardown.md create mode 100644 _migration-assistant/migration-phases/verify-backfill-components.md create mode 100644 _migration-assistant/migration-phases/verify-live-capture-components.md create mode 100644 _migration-assistant/overview/architecture.md create mode 100644 _migration-assistant/overview/index.md create mode 100644 _migration-assistant/overview/is-migration-assistant-right-for-you.md create mode 100644 _migration-assistant/overview/key-components.md create mode 100644 _migration-assistant/planning-your-migration/handling-field-type-breaking-changes.md create mode 100644 _migration-assistant/planning-your-migration/handling-type-mapping-deprecation.md create mode 100644 _migration-assistant/planning-your-migration/index.md diff --git a/_config.yml b/_config.yml index baac4a1c7c6..27f751e4010 100644 --- a/_config.yml +++ b/_config.yml @@ -92,6 +92,9 @@ collections: migration-upgrade: permalink: /:collection/:path/ output: true + migration-assistant: + permalink: /:collection/:path/ + output: true tools: permalink: /:collection/:path/ output: true @@ -221,6 +224,12 @@ migration_upgrade_collection: name: Migration and upgrade nav_fold: true +migration_assistant_collection: + collections: + migration-assistant: + name: Migration Assistant + nav_fold: true + benchmark_collection: collections: benchmark: @@ -262,10 +271,10 @@ defaults: section-name: "Benchmark" - scope: - path: "_migration-assistant" + path: "_migrations-and-upgrades" values: - section: "migration-assistant" - section-name: "Migration Assistant" + section: "migrations-and-upgrades" + section-name: "Migrations and Upgrades" # Enable or disable the site search # By default, just-the-docs enables its JSON file-based search. We also have an OpenSearch-driven search functionality. @@ -326,6 +335,7 @@ plugins: - jekyll-redirect-from - jekyll-sitemap - jekyll-spec-insert + - jekyll-include-cache # This format has to conform to RFC822 last-modified-at: @@ -352,4 +362,4 @@ exclude: - .bundle/ - _site/ - spec-insert - - release-notes \ No newline at end of file + - release-notes diff --git a/_includes/home_cards.html b/_includes/home_cards.html index 1509d5402ea..8be45489317 100644 --- a/_includes/home_cards.html +++ b/_includes/home_cards.html @@ -61,9 +61,9 @@
- -

Migrate and upgrade

-

Migrate or upgrade to OpenSearch.

+ +

Modernize Workloads

+

Migration Assistant for OpenSearch and other upgrade and migration options

Documentation →

diff --git a/_migration-upgrade/assets/css/breaking-changes-selector.css b/_migration-assistant/assets/css/breaking-changes-selector.css similarity index 100% rename from _migration-upgrade/assets/css/breaking-changes-selector.css rename to _migration-assistant/assets/css/breaking-changes-selector.css diff --git a/_migration-upgrade/assets/js/breaking-changes-data.js b/_migration-assistant/assets/js/breaking-changes-data.js similarity index 100% rename from _migration-upgrade/assets/js/breaking-changes-data.js rename to _migration-assistant/assets/js/breaking-changes-data.js diff --git a/_migration-upgrade/assets/js/breaking-changes-filter.js b/_migration-assistant/assets/js/breaking-changes-filter.js similarity index 100% rename from _migration-upgrade/assets/js/breaking-changes-filter.js rename to _migration-assistant/assets/js/breaking-changes-filter.js diff --git a/_migration-upgrade/assets/js/breaking-changes-index.js b/_migration-assistant/assets/js/breaking-changes-index.js similarity index 100% rename from _migration-upgrade/assets/js/breaking-changes-index.js rename to _migration-assistant/assets/js/breaking-changes-index.js diff --git a/_migration-upgrade/assets/js/breaking-changes-module.js b/_migration-assistant/assets/js/breaking-changes-module.js similarity index 100% rename from _migration-upgrade/assets/js/breaking-changes-module.js rename to _migration-assistant/assets/js/breaking-changes-module.js diff --git a/_migration-upgrade/assets/js/breaking-changes-ui.js b/_migration-assistant/assets/js/breaking-changes-ui.js similarity index 100% rename from _migration-upgrade/assets/js/breaking-changes-ui.js rename to _migration-assistant/assets/js/breaking-changes-ui.js diff --git a/_migration-assistant/index.md b/_migration-assistant/index.md new file mode 100644 index 00000000000..1d7f8bb1db5 --- /dev/null +++ b/_migration-assistant/index.md @@ -0,0 +1,35 @@ +--- +layout: default +title: Migration Assistant for OpenSearch +nav_order: 1 +has_children: false +nav_exclude: true +has_toc: false +permalink: /migration-assistant/ +redirect_from: + - /migration-assistant/index/ + - /upgrade-to/index/ + - /upgrade-to/ + - /upgrade-to/upgrade-to/ +tutorial_cards: + - heading: "Overview" + description: "Get familiar with the key components of Migration Assistant and evaluate your use case." + link: "/migration-assistant/overview/" + - heading: "Migration phases" + description: "Execute your migration in phases—metadata, backfill, and traffic replay—for a controlled and validated transition." + link: "/migration-assistant/migration-phases/" +--- + +# Migration Assistant for OpenSearch + +Migration Assistant for OpenSearch aids you in successfully performing an end-to-end, zero-downtime migration to OpenSearch from other search providers. It helps with the following scenarios: + +- **Metadata migration**: Migrating cluster metadata, such as index settings, aliases, and templates. +- **Backfill migration**: Migrating existing or historical data from a source to a target cluster. +- **Live traffic migration**: Replicating live ongoing traffic from a source to a target cluster. +- **Comparative tooling**: Comparing the performance and behaviors of an existing cluster with a prospective new one. + +This user guide focuses on conducting a comprehensive migration involving both existing and live data with zero downtime and the option to back out of a migration. + +{% include cards.html cards=page.tutorial_cards %} + diff --git a/_migration-assistant/migration-phases/accessing-the-migration-console.md b/_migration-assistant/migration-phases/accessing-the-migration-console.md new file mode 100644 index 00000000000..fc86d61ca2b --- /dev/null +++ b/_migration-assistant/migration-phases/accessing-the-migration-console.md @@ -0,0 +1,42 @@ +--- +layout: default +title: Accessing the Migration Console +parent: Deploying Migration Assistant +nav_order: 10 +redirect_from: +--- +# Accessing the Migration Console + +The Migrations Assistant deployment includes an ECS task that hosts tools to run different phases of the migration and check the progress or results of the migration. + +## SSH into the Migration Console + +Following the AWS Solutions deployment, the bootstrap box contains a script that simplifies access to the migration console through that instance. + +To access the Migration Console, use the following commands: + +```sh +export STAGE=dev +export AWS_REGION=us-west-2 +/opensearch-migrations/deployment/cdk/opensearch-service-migration/accessContainer.sh migration-console ${STAGE} ${AWS_REGION} +``` +When opening the console a message will appear above the command prompt, Welcome to the Migration Assistant Console. +
+ + +SSH from any machine into Migration Console + + +On a machine with the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) ↗ and the [AWS Session Manager Plugin](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html) ↗, you can directly connect to the migration console. Ensure you've run `aws configure` with credentials that have access to the environment. + +Use the following commands: + +```shell +export STAGE=dev +export SERVICE_NAME=migration-console +export TASK_ARN=$(aws ecs list-tasks --cluster migration-${STAGE}-ecs-cluster --family "migration-${STAGE}-${SERVICE_NAME}" | jq --raw-output '.taskArns[0]') +aws ecs execute-command --cluster "migration-${STAGE}-ecs-cluster" --task "${TASK_ARN}" --container "${SERVICE_NAME}" --interactive --command "/bin/bash" +``` +
+ +> **Note:** Typically, `STAGE` is `dev`, but this may vary based on what the user specified during deployment. \ No newline at end of file diff --git a/_migration-assistant/migration-phases/assessment.md b/_migration-assistant/migration-phases/assessment.md new file mode 100644 index 00000000000..d5210674e66 --- /dev/null +++ b/_migration-assistant/migration-phases/assessment.md @@ -0,0 +1,76 @@ +--- +layout: default +title: Assess +nav_order: 1 +parent: Migration phases +redirect_from: + - /migration-assistant/migration-phases/assessing-your-cluster-for-migration/ +--- + +# Assessing your cluster for migration + +The goal of the Migration Assistant is to streamline the process of migrating from one location or version of Elasticsearch/OpenSearch to another. However, completing a migration sometimes requires resolving client compatibility issues before they can communicate directly with the target cluster. + + +## Understanding breaking changes + +Before performing any upgrade or migration, you should review any breaking changes that might exist between version because there may be changes required in order for clients to connect to the new cluster. Use the following tool by selecting the versions in your migration path. The tool will respond with any breaking changes you should note when migrating: + + + +
+

Find a list of breaking changes for your migration path

+ +
+ + + + + +
+ +
+
+ + +
+ +
+
+ + + + + +## Impact of data transformations + +Any time you apply a transformation to your data, such as: + +- Changing index names +- Modifying field names or field mappings +- Splitting indices with type mappings + +These changes might need to be reflected in your client configurations. For example, if your clients are reliant on specific index or field names, you must ensure that their queries are updated accordingly. + +We recommend running production-like queries against the target cluster before switching over actual production traffic. This helps verify that the client can: + +- Communicate with the target cluster +- Locate the necessary indices and fields +- Retrieve the expected results + +For complex migrations involving multiple transformations or breaking changes, we highly recommend performing a trial migration with representative, non-production data (e.g., in a staging environment) to fully test client compatibility with the target cluster. + +## Supported transformations + +The following [transformations]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/live-traffic-migration/using-traffic-replayer/#transformations) are included in Migration Assistant. They can be enabled, combined, and configured to customize a migration for your use case. To request additional Migration Assistant transformations , create a GitHub issue [in the OpenSearch migrations repository](https://github.com/opensearch-project/opensearch-migrations/issues). + +- [Type mapping deprecation]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation/) diff --git a/_migration-assistant/migration-phases/create-snapshot.md b/_migration-assistant/migration-phases/create-snapshot.md new file mode 100644 index 00000000000..6b24ae78b6b --- /dev/null +++ b/_migration-assistant/migration-phases/create-snapshot.md @@ -0,0 +1,244 @@ +--- +layout: default +title: Snapshot +nav_order: 4 +parent: Migration phases +--- + +## Creating snapshot + +TODO: Add Bring your own snapshot + +Creating a snapshot of the source cluster captures all the metadata and documents to be migrated to a new target cluster. + +Create the initial snapshot of the source cluster using the following command: + +```shell +console snapshot create +``` +{% include copy.html %} + +To check the progress of the snapshot in real time, use the following command: + +```shell +console snapshot status --deep-check +``` +{% include copy.html %} + +You should receive the following response when the snapshot is created: + +```shell +SUCCESS +Snapshot is SUCCESS. +Percent completed: 100.00% +Data GiB done: 29.211/29.211 +Total shards: 40 +Successful shards: 40 +Failed shards: 0 +Start time: 2024-07-22 18:21:42 +Duration: 0h 13m 4s +Anticipated duration remaining: 0h 0m 0s +Throughput: 38.13 MiB/sec +``` + +### Managing slow snapshot speeds + +Depending on the size of the data in the source cluster and the bandwidth allocated for snapshots, the process can take some time. Adjust the maximum rate at which the source cluster's nodes create the snapshot using the `--max-snapshot-rate-mb-per-node` option. Increasing the snapshot rate will consume more node resources, which may affect the cluster's ability to handle normal traffic. + +## Command arguments + +For the following commands, to identify all valid arguments, please run with `--help`. + +```shell +console metadata evaluate --help +``` +{% include copy.html %} + +```shell +console metadata migrate --help +``` +{% include copy.html %} + +Based on the migration console deployment options, a number of commands will be pre-populated. To view them, run console with verbosity: + +```shell +console -v metadata migrate --help +``` +{% include copy.html %} + +You should receive a response similar to the following: + +```shell +(.venv) bash-5.2# console -v metadata migrate --help +INFO:console_link.cli:Logging set to INFO +. +. +. +INFO:console_link.models.metadata:Migrating metadata with command: /root/metadataMigration/bin/MetadataMigration --otel-collector-endpoint http://otel-collector:4317 migrate --snapshot-name snapshot_2023_01_01 --target-host https://opensearchtarget:9200 --min-replicas 0 --file-system-repo-path /snapshot/test-console --target-username admin --target-password ******** --target-insecure --help +. +. +. +``` + + +## Using the `evaluate` command + +By scanning the contents of the source cluster, applying filtering, and applying modifications a list of all items that will be migrated will be created. Any items not seen in this output will not be migrated onto the target cluster if the migrate command was to be run. This is a safety check before making modifications on the target cluster. + +```shell +console metadata evaluate [...] +``` +{% include copy.html %} + +You should receive a response similar to the following: + +```bash +Starting Metadata Evaluation +Clusters: + Source: + Remote Cluster: OpenSearch 1.3.16 ConnectionContext(uri=http://localhost:33039, protocol=HTTP, insecure=false, compressionSupported=false) + + Target: + Remote Cluster: OpenSearch 2.14.0 ConnectionContext(uri=http://localhost:33037, protocol=HTTP, insecure=false, compressionSupported=false) + + +Migration Candidates: + Index Templates: + simple_index_template + + Component Templates: + simple_component_template + + Indexes: + blog_2023, movies_2023 + + Aliases: + alias1, movies-alias + + +Results: + 0 issue(s) detected +``` + + +## Using the migrate command + +Running through the same data as the evaluate command all of the migrated items will be applied onto the target cluster. If re-run multiple times items that were previously migrated will not be recreated. If any items do need to be re-migrated, please delete them from the target cluster and then rerun the evaluate then migrate commands to ensure the desired changes are made. + +```shell +console metadata migrate [...] +``` +{% include copy.html %} + +You should receive a response similar to the following: + +```shell +Starting Metadata Migration + +Clusters: + Source: + Snapshot: OpenSearch 1.3.16 FileSystemRepo(repoRootDir=/tmp/junit10626813752669559861) + + Target: + Remote Cluster: OpenSearch 2.14.0 ConnectionContext(uri=http://localhost:33042, protocol=HTTP, insecure=false, compressionSupported=false) + + +Migrated Items: + Index Templates: + simple_index_template + + Component Templates: + simple_component_template + + Indexes: + blog_2023, movies_2023 + + Aliases: + alias1, movies-alias + + +Results: + 0 issue(s) detected +``` + + +## Metadata verification process + +Before moving on to additional migration steps, it is recommended to confirm details of your cluster. Depending on your configuration, this could be checking the sharding strategy or making sure index mappings are correctly defined by ingesting a test document. + +## Troubleshooting + +Use these instructions to help troubleshoot the following issues. + +### Accessing detailed logs + +Metadata migration creates a detailed log file that includes low level tracing information for troubleshooting. For each execution of the program a log file is created inside a shared volume on the migration console named `shared-logs-output` the following command will list all log files, one for each run of the command. + +```shell +ls -al /shared-logs-output/migration-console-default/*/metadata/ +``` +{% include copy.html %} + +To inspect the file within the console `cat`, `tail` and `grep` commands line tools. By looking for warnings, errors and exceptions in this log file can help understand the source of failures, or at the very least be useful for creating issues in this project. + +```shell +tail /shared-logs-output/migration-console-default/*/metadata/*.log +``` +{% include copy.html %} + +### Warnings and errors + +When encountering `WARN` or `ERROR` elements in the response, they will be accompanied by a short message, such as `WARN - my_index already exists`. More information can be found in the detailed logs associated with the warning or error. + +### OpenSearch running in compatibility mode + +There might be an error about being unable to update an ES 7.10.2 cluster, this can occur when compatibility mode has been enabled on an OpenSearch cluster disable it to continue, see [Enable compatibility mode](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/rename.html#rename-upgrade). + + +### Breaking change compatibility + +Metadata migration requires modifying data from the source to the target versions to recreate items. Sometimes these features are no longer supported and have been removed from the target version. Sometimes these features are not available in the target version, which is especially true when downgrading. While this tool is meant to make this process easier, it is not exhaustive in its support. When encountering a compatibility issue or an important feature gap for your migration, [search the issues and comment on the existing issue](https://github.com/opensearch-project/opensearch-migrations/issues) or [create a new](https://github.com/opensearch-project/opensearch-migrations/issues/new/choose) issue if one cannot be found. + +#### Deprecation of Mapping Types + +In Elasticsearch 6.8 the mapping types feature was discontinued in Elasticsearch 7.0+ which has created complexity in migrating to newer versions of Elasticsearch and OpenSearch, [learn more](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/removal-of-types.html) ↗. + +As Metadata migration supports migrating from ES 6.8 on to the latest versions of OpenSearch this scenario is handled by removing the type mapping types and restructuring the template or index properties. Note that, at the time of this writing multiple type mappings are not supported, [tracking task](https://opensearch.atlassian.net/browse/MIGRATIONS-1778) ↗. + + +**Example starting state with mapping type foo (ES 6):** + +```json +{ + "mappings": [ + { + "foo": { + "properties": { + "field1": { "type": "text" }, + "field2": { "type": "keyword" } + } + } + } + ] +} +``` +{% include copy.html %} + +**Example ending state with foo removed (ES 7):** + +```json +{ + "mappings": { + "properties": { + "field1": { "type": "text" }, + "field2": { "type": "keyword" }, + } + } +} +``` +{% include copy.html %} + +For additional technical details, [view the mapping type removal source code](https://github.com/opensearch-project/opensearch-migrations/blob/main/transformation/src/main/java/org/opensearch/migrations/transformation/rules/IndexMappingTypeRemoval.java). + +TODO: Add troublshooting +
  • Verify Backfill Components
    • \ No newline at end of file diff --git a/_migration-assistant/migration-phases/deploying-migration-assistant/configuration-options.md b/_migration-assistant/migration-phases/deploying-migration-assistant/configuration-options.md new file mode 100644 index 00000000000..5032dcb757e --- /dev/null +++ b/_migration-assistant/migration-phases/deploying-migration-assistant/configuration-options.md @@ -0,0 +1,233 @@ +--- +layout: default +title: Configuration options +nav_order: 15 +parent: Deploying Migration Assistant +permalink: /migration-assistant/deploying-migration-assistant/configuration-options/ +--- + +# Configuration options + +This page outlines the configuration options for three key migrations scenarios: + +1. **Metadata migration** +2. **Backfill migration with `Reindex-from-Snapshot` (RFS)** +3. **Live capture migration with Capture and Replay (C&R)** + +Each of these migrations depends on either a snapshot or a capture proxy. The following example `cdk.context.json` configurations are used by AWS Cloud Development Kit (AWS CDK) to deploy and configure Migration Assistant for OpenSearch, shown as separate blocks for each migration type. If you are performing a migration applicable to multiple scenarios, these options can be combined. + + +For a complete list of configuration options, see [opensearch-migrations-options.md](https://github.com/opensearch-project/opensearch-migrations/blob/main/deployment/cdk/opensearch-service-migration/options.md). If you need a configuration option that is not found on this page, create an issue in the [OpenSearch Migrations repository](https://github.com/opensearch-project/opensearch-migrations/issues). +{: .tip } + +Options for the source cluster endpoint, target cluster endpoint, and existing virtual private cloud (VPC) should be configured in order for the migration tools to function effectively. + +## Shared configuration options + +Each migration configuration shares the following options. + + +| Name | Example | Description | +| :--- | :--- | :--- | +| `sourceClusterEndpoint` | `"https://source-cluster.elb.us-east-1.endpoint.com"` | The endpoint for the source cluster. | +| `targetClusterEndpoint` | `"https://vpc-demo-opensearch-cluster-cv6hggdb66ybpk4kxssqt6zdhu.us-west-2.es.amazonaws.com:443"` | The endpoint for the target cluster. Required if using an existing target cluster for the migration instead of creating a new one. | +| `vpcId` | `"vpc-123456789abcdefgh"` | The ID of the existing VPC in which the migration resources will be stored. The VPC must have at least two private subnets that span two Availability Zones. | + + +## Backfill migration using RFS + +The following CDK performs a backfill migrations using RFS: + +```json +{ + "backfill-migration": { + "stage": "dev", + "vpcId": , + "sourceCluster": { + "endpoint": , + "version": "ES 7.10", + "auth": {"type": "none"} + }, + "targetCluster": { + "endpoint": , + "auth": { + "type": "basic", + "username": , + "passwordFromSecretArn": + } + }, + "reindexFromSnapshotServiceEnabled": true, + "reindexFromSnapshotExtraArgs": "", + "artifactBucketRemovalPolicy": "DESTROY" + } +} +``` +{% include copy.html %} + +Performing an RFS backfill migration requires an existing snapshot. + + +The RFS configuration uses the following options. All options are optional. + +| Name | Example | Description | +| :--- | :--- | :--- | +| `reindexFromSnapshotServiceEnabled` | `true` | Enables deployment and configuration of the RFS ECS service. | +| `reindexFromSnapshotExtraArgs` | `"--target-aws-region us-east-1 --target-aws-service-signing-name es"` | Extra arguments for the Document Migration command, with space separation. See [RFS Extra Arguments](https://github.com/opensearch-project/opensearch-migrations/blob/main/DocumentsFromSnapshotMigration/README.md#arguments) for more information. You can pass `--no-insecure` to remove the `--insecure` flag. | + +To view all available arguments for `reindexFromSnapshotExtraArgs`, see [Snapshot migrations README](https://github.com/opensearch-project/opensearch-migrations/blob/main/DocumentsFromSnapshotMigration/README.md#arguments). At a minimum, no extra arguments may be needed. + +## Live capture migration with C&R + +The following sample CDK performs a live capture migration with C&R: + +```json +{ + "live-capture-migration": { + "stage": "dev", + "vpcId": , + "sourceCluster": { + "endpoint": , + "version": "ES 7.10", + "auth": {"type": "none"} + }, + "targetCluster": { + "endpoint": , + "auth": { + "type": "basic", + "username": , + "passwordFromSecretArn": + } + }, + "captureProxyServiceEnabled": true, + "captureProxyExtraArgs": "", + "trafficReplayerServiceEnabled": true, + "trafficReplayerExtraArgs": "--speedup-factor 2.0", + "targetClusterProxyServiceEnabled": true + } +} +``` +{% include copy.html %} + +Performing a live capture migration requires that a Capture Proxy be configured to capture incoming traffic and send it to the target cluster using the Traffic Replayer service. For arguments available in `captureProxyExtraArgs`, refer to the `@Parameter` fields [here](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java). For `trafficReplayerExtraArgs`, refer to the `@Parameter` fields [here](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java). At a minimum, no extra arguments may be needed. + + +| Name | Example | Description | +| :--- | :--- | :--- | +| `captureProxyServiceEnabled` | `true` | Enables the Capture Proxy service deployment using an AWS CloudFormation stack. | +| `captureProxyExtraArgs` | `"--suppressCaptureForHeaderMatch user-agent .*elastic-java/7.17.0.*"` | Extra arguments for the Capture Proxy command, including options specified by the [Capture Proxy](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java). | +| `trafficReplayerServiceEnabled` | `true` | Enables the Traffic Replayer service deployment using a CloudFormation stack. | +| `trafficReplayerExtraArgs` | `"--sigv4-auth-header-service-region es,us-east-1 --speedup-factor 5"` | Extra arguments for the Traffic Replayer command, including options for auth headers and other parameters specified by the [Traffic Replayer](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java). | +| `targetClusterProxyServiceEnabled` | `true` | Enables the target cluster proxy service deployment using a CloudFormation stack. | + +For arguments available in `captureProxyExtraArgs`, see the `@Parameter` fields in [`CaptureProxy.java`](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java). For `trafficReplayerExtraArgs`, see the `@Parameter` fields in [`TrafficReplayer.java`](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java). + + +## Cluster authentication options + +Both the source and target cluster can use no authentication, authentication limited to VPC, basic authentication with a username and password, or AWS Signature Version 4 scoped to a user or role. + +### No authentication + +```json + "sourceCluster": { + "endpoint": , + "version": "ES 7.10", + "auth": {"type": "none"} + } +``` +{% include copy.html %} + +### Basic authentication + +```json + "sourceCluster": { + "endpoint": , + "version": "ES 7.10", + "auth": { + "type": "basic", + "username": , + "passwordFromSecretArn": + } + } +``` +{% include copy.html %} + +### AWS Signature Version 4 authentication + +```json + "sourceCluster": { + "endpoint": , + "version": "ES 7.10", + "auth": { + "type": "sigv4", + "region": "us-east-1", + "serviceSigningName": "es" + } + } +``` +{% include copy.html %} + +The `serviceSigningName` can be `es` for an Elasticsearch or OpenSearch domain. + +All of these authentication options apply to both source and target clusters. + +## Snapshot options + +The following configuration options customize the process of migrating from snapshots. + +### Snapshot of a managed service source + +If your source cluster is on Amazon OpenSearch Service, you need to set up an additional AWS Identity and Access Management (IAM) role and pass it with the snapshot creation call, as described in the [AWS documentation](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-snapshots.html). Migration Assistant can automatically manage this process. OpenSearch Service snapshots are only compatible with AWS Signature Version 4 authentication. The following parameter ensures that the additional IAM role is created and passed. + +| Name | Example | Description | +| :--- | :--- | :--- | +| `managedServiceSourceSnapshotEnabled` | `true` | Creates the necessary roles and trust relationships for taking a snapshot of an OpenSearch Service source cluster. This is only compatible with AWS Signature Version 4 authentication.| + +### Bring your own snapshot + +You can use an existing Amazon Simple Storage Service (Amazon S3) snapshot to perform [metadata]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrating-metadata/) and [backfill]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/backfill/) migrations instead of using Migration Assistant to create a snapshot: + +```json + "snapshot": { + "snapshotName": "my-snapshot-name", + "snapshotRepoName": "my-snapshot-repo", + "s3Uri": "s3://my-s3-bucket-name/my-bucket-path-to-snapshot-repo", + "s3Region": "us-east-2" + } +``` +{% include copy.html %} + +By default, Amazon S3 buckets automatically allow roles in the same AWS account (with the appropriate `s3:*` permissions) to access the S3 bucket, regardless of the bucket's AWS Region. If the external S3 bucket is in the same AWS account as the Migration Assistant deployment, no further IAM configuration is required to access the bucket. + +If you use a custom permission model with Amazon S3, any access control list (ACL) or custom bucket policy should allow the Migration Assistant task roles for RFS and the migration console to read from the S3 bucket. + +If the S3 bucket is in a separate AWS account from the Migration Assistant deployment, you need a custom bucket policy similar to the following to allow access to Migration Assistant: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "AllowExternalAccountReadAccessToBucket", + "Effect": "Allow", + "Principal": { + "AWS": "arn:aws:iam:::root" + }, + "Action": [ + "s3:GetObject", + "s3:ListBucket", + "s3:GetBucketLocation" + ], + "Resource": [ + "arn:aws:s3:::my-s3-bucket-name", + "arn:aws:s3:::my-s3-bucket-name/*" + ] + } + ] +} +``` +{% include copy.html %} + +## Network configuration + +The migration tooling expects the source cluster, target cluster, and migration resources to exist in the same VPC. If this is not the case, manual networking setup outside of this documentation is likely required. diff --git a/_migration-assistant/migration-phases/deploying-migration-assistant/getting-started-data-migration.md b/_migration-assistant/migration-phases/deploying-migration-assistant/getting-started-data-migration.md new file mode 100644 index 00000000000..d0592e2dd24 --- /dev/null +++ b/_migration-assistant/migration-phases/deploying-migration-assistant/getting-started-data-migration.md @@ -0,0 +1,360 @@ +--- +layout: default +title: Getting started with data migration +parent: Deploying Migration Assistant +nav_order: 10 +permalink: /migration-assistant/deploying-migration-assistant/getting-started-data-migration/ +redirect_from: + - /upgrade-to/snapshot-migrate/ + - /migration-assistant/getting-started-with-data-migration/ +--- + +# Getting started with data migration + +This quickstart outlines how to deploy Migration Assistant for OpenSearch and execute an existing data migration using `Reindex-from-Snapshot` (RFS). It uses AWS for illustrative purposes. However, the steps can be modified for use with other cloud providers. + +## Prerequisites and assumptions + +Before using this quickstart, make sure you fulfill the following prerequisites: + +* Verify that your migration path [is supported]({{site.url}}{{site.baseurl}}/migration-assistant/overview/is-migration-assistant-right-for-you/#supported-migration-paths). Note that we test with the exact versions specified, but you should be able to migrate data on alternative minor versions as long as the major version is supported. +* The source cluster must be deployed Amazon Simple Storage Service (Amazon S3) plugin. +* The target cluster must be deployed. +* Verify that the `CDKToolkit` stack exists and is set to `CREATE_COMPLETE`. For more information about how to bootstrap your AWS account in the required AWS Region, see [the CDKToolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html). + +The steps in this guide assume the following: + +* In this guide, a snapshot will be taken and stored in Amazon S3; the following assumptions are made about this snapshot: + * The `_source` flag is enabled on all indexes to be migrated. + * The snapshot includes the global cluster state (`include_global_state` is `true`). + * Shard sizes of up to approximately 80 GB are supported. Larger shards cannot be migrated. If this presents challenges for your migration, contact the [migration team](https://opensearch.slack.com/archives/C054JQ6UJFK). +* Migration Assistant will be installed in the same AWS Region and have access to both the source snapshot and target cluster. + +--- + +## Step 1: Install Bootstrap on an Amazon EC2 instance (~10 minutes) + +To begin your migration, use the following steps to install a `bootstrap` box on an Amazon Elastic Compute Cloud (Amazon EC2) instance. The instance uses AWS CloudFormation to create and manage the stack. + +1. Log in to the target AWS account in which you want to deploy Migration Assistant. +2. From the browser where you are logged in to your target AWS account, right-click [here](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/migration-assistant-for-amazon-opensearch-service/latest/migration-assistant-for-amazon-opensearch-service.template&redirectId=SolutionWeb) to load the CloudFormation template from a new browser tab. +3. Follow the CloudFormation stack wizard: + * **Stack Name:** `MigrationBootstrap` + * **Stage Name:** `dev` + * Choose **Next** after each step > **Acknowledge** > **Submit**. +4. Verify that the Bootstrap stack exists and is set to `CREATE_COMPLETE`. This process takes around 10 minutes to complete. + +--- + +## Step 2: Set up Bootstrap instance access (~5 minutes) + +Use the following steps to set up Bootstrap instance access: + +1. After deployment, find the EC2 instance ID for the `bootstrap-dev-instance`. +2. Create an AWS Identity and Access Management (IAM) policy using the following snippet, replacing ``, ``, ``, and `` with your information: + + ```json + { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": "ssm:StartSession", + "Resource": [ + "arn:aws:ec2:::instance/", + "arn:aws:ssm:::document/BootstrapShellDoc--" + ] + } + ] + } + ``` + {% include copy.html %} + +3. Name the policy, for example, `SSM-OSMigrationBootstrapAccess`, and then create the policy by selecting **Create policy**. +4. Attach the newly created policy to your EC2 instance's IAM role. + +--- + +## Step 3: Log in to Bootstrap and building Migration Assistant (~15 minutes) + +Next, log in to Bootstrap and build Migration Assistant using the following steps. + +### Prerequisites + +To use these steps, make sure you fulfill the following prerequisites: + +* The AWS Command Line Interface (AWS CLI) and AWS Session Manager plugin are installed on your instance. +* The AWS credentials are configured (`aws configure`) for your instance. + +### Steps + +1. Load AWS credentials into your terminal. +2. Log in to the instance using the following command, replacing `` and `` with your instance ID and Region: + + ```bash + aws ssm start-session --document-name BootstrapShellDoc-- --target --region [--profile ] + ``` + {% include copy.html %} + +3. Once logged in, run the following command from the shell of the Bootstrap instance in the `/opensearch-migrations` directory: + + ```bash + ./initBootstrap.sh && cd deployment/cdk/opensearch-service-migration + ``` + {% include copy.html %} + +4. After a successful build, note the path for infrastructure deployment, which will be used in the next step. + +--- + +## Step 4: Configure and deploy RFS (~20 minutes) + +To deploy Migration Assistant with RFS, the following stacks must be deployed: + +These commands deploy the following stacks: + +* `Migration Assistant network` stack +* `RFS` stack +* `Migration console` stack + +Use the following steps to configure and deploy RFS, deploy Migration Assistant, and verify installation of the required stacks: + +1. Add the source and target cluster password as separate **Secrets** in [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) as an unstructured string. Be sure to copy the secret Amazon Resource Name (ARN) for use during deployment. +2. From the same shell as the Bootstrap instance, modify the `cdk.context.json` file located in the `/opensearch-migrations/deployment/cdk/opensearch-service-migration` directory and configure the following settings: + + ```json + { + "default": { + "stage": "dev", + "vpcId": "", + "targetCluster": { + "endpoint": "", + "auth": { + "type": "basic", + "username": "", + "passwordFromSecretArn": "" + } + }, + "sourceCluster": { + "endpoint": "", + "version": "", + "auth": { + "type": "basic", + "username": "", + "passwordFromSecretArn": "" + } + }, + "reindexFromSnapshotExtraArgs": "", + "reindexFromSnapshotMaxShardSizeGiB": 80, + "otelCollectorEnabled": true, + "migrationConsoleServiceEnabled": true + } + } + ``` + {% include copy.html %} + + The source and target cluster authorization can be configured to have no authorization, `basic` with a username and password, or `sigv4`. + +3. After the `cdk.context.json` file is fully configured, bootstrap the account and deploy the required stacks using the following command: + + ```bash + cdk bootstrap --c contextId=default --require-approval never + ``` + {% include copy.html %} + +4. Deploy Migration Assistant using the following command: + + ```bash + cdk deploy "*" --c contextId=default --require-approval never --concurrency 5 + ``` + {% include copy.html %} + +5. From the same Bootstrap instance shell, verify that all CloudFormation stacks were installed successfully: + + ```bash + aws cloudformation list-stacks --query "StackSummaries[?StackStatus!='DELETE_COMPLETE'].[StackName,StackStatus]" --output table + ``` + {% include copy.html %} + +You should receive a similar output for your Region: + +```bash +------------------------------------------------------------------------ +| ListStacks | ++--------------------------------------------------+-------------------+ +| OSMigrations-dev-us-east-1-MigrationConsole | CREATE_COMPLETE | +| OSMigrations-dev-us-east-1-ReindexFromSnapshot | CREATE_COMPLETE | +| OSMigrations-dev-us-east-1-MigrationInfra | CREATE_COMPLETE | +| OSMigrations-dev-us-east-1-default-NetworkInfra | CREATE_COMPLETE | +| MigrationBootstrap | CREATE_COMPLETE | +| CDKToolkit | CREATE_COMPLETE | ++--------------------------------------------------+-------------------+ +``` + +### RFS parameters + +If you're creating a snapshot using migration tooling, these parameters are automatically configured. If you're using an existing snapshot, modify the `reindexFromSnapshotExtraArgs` setting with the following values: + +```bash + "reindexFromSnapshotExtraArgs": "--s3-repo-uri s3:/// --s3-region --snapshot-name " +``` + +You will also need to give the `migrationconsole` and `reindexFromSnapshot` TaskRoles permissions to the S3 bucket. + +--- + +## Step 5: Access the migration console + +Run the following command to access the migration console: + +```bash +./accessContainer.sh migration-console dev +``` +{% include copy.html %} + + +`accessContainer.sh` is located in `/opensearch-migrations/deployment/cdk/opensearch-service-migration/` on the Bootstrap instance. To learn more, see [Accessing the migration console]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrating-metadata/). +{: .note} + +--- + +## Step 6: Verify the connection to the source and target clusters + +To verify the connection to the clusters, run the following command: + +```bash +console clusters connection-check +``` +{% include copy.html %} + +You should receive the following output: + +```bash +SOURCE CLUSTER +ConnectionResult(connection_message='Successfully connected!', connection_established=True, cluster_version='') +TARGET CLUSTER +ConnectionResult(connection_message='Successfully connected!', connection_established=True, cluster_version='') +``` + +To learn more about migration console commands, see [Migration commands]. + +--- + +## Step 7: Create a snapshot + +Run the following command to initiate snapshot creation from the source cluster: + +```bash +console snapshot create [...] +``` +{% include copy.html %} + +To check the snapshot creation status, run the following command: + +```bash +console snapshot status [...] +``` +{% include copy.html %} + +To learn more information about the snapshot, run the following command: + +```bash +console snapshot status --deep-check [...] +``` +{% include copy.html %} + +Wait for snapshot creation to complete before moving to step 9. + +To learn more about snapshot creation, see [Snapshot Creation]. + +--- + +## Step 8: Migrate metadata + +Run the following command to migrate metadata: + +```bash +console metadata migrate [...] +``` +{% include copy.html %} + +For more information, see [Migrating metadata]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrating-metadata/). + +--- + +## Step 9: Migrate documents with RFS + +You can now use RFS to migrate documents from your original cluster: + +1. To start the migration from RFS, start a `backfill` using the following command: + + ```bash + console backfill start + ``` + {% include copy.html %} + +2. _(Optional)_ To speed up the migration, increase the number of documents processed at a simultaneously by using the following command: + + ```bash + console backfill scale + ``` + {% include copy.html %} + +3. To check the status of the documentation backfill, use the following command: + + ```bash + console backfill status + ``` + {% include copy.html %} + +4. If you need to stop the backfill process, use the following command: + + ```bash + console backfill stop + ``` + {% include copy.html %} + +For more information, see [Backfill]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/backfill/). + +--- + +## Step 10: Backfill monitoring + +Use the following command for detailed monitoring of the backfill process: + +```bash +console backfill status --deep-check +``` +{% include copy.html %} + +You should receive the following output: + +```json +BackfillStatus.RUNNING +Running=9 +Pending=1 +Desired=10 +Shards total: 62 +Shards completed: 46 +Shards incomplete: 16 +Shards in progress: 11 +Shards unclaimed: 5 +``` + +Logs and metrics are available in Amazon CloudWatch in the `OpenSearchMigrations` log group. + +--- + +## Step 11: Verify that all documents were migrated + +Use the following query in CloudWatch Logs Insights to identify failed documents: + +```bash +fields @message +| filter @message like "Bulk request succeeded, but some operations failed." +| sort @timestamp desc +| limit 10000 +``` +{% include copy.html %} + +If any failed documents are identified, you can index the failed documents directly as opposed to using RFS. diff --git a/_migration-assistant/migration-phases/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md b/_migration-assistant/migration-phases/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md new file mode 100644 index 00000000000..5ed62887838 --- /dev/null +++ b/_migration-assistant/migration-phases/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md @@ -0,0 +1,93 @@ +--- +layout: default +title: IAM and security groups for existing clusters +nav_order: 20 +parent: Deploying Migration Assistant +permalink: /migration-assistant/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters/ +--- + +# IAM and security groups for existing clusters + +This page outlines security scenarios for using the migration tools with existing clusters, including any necessary configuration changes to ensure proper communication between them. + +## Importing an Amazon OpenSearch Service or Amazon OpenSearch Serverless target cluster + +Use the following scenarios for Amazon OpenSearch Service or Amazon OpenSearch Serverless target clusters. + +### OpenSearch Service + +For an OpenSearch Domain, two main configurations are typically required to ensure proper functioning of the migration solution: + +1. **Security Group Configuration** + + The domain should have a security group that allows communication from the applicable migration services (Traffic Replayer, Migration Console, `Reindex-from-Snapshot`). The CDK automatically creates an `osClusterAccessSG` security group, which is applied to the migration services. The user should then add this security group to their existing domain to allow access. + +2. **Access Policy Configuration** should be one of the following: + - An open access policy that allows all access. + - Configured to allow at least the AWS Identity and Access Management (IAM) task roles for the applicable migration services (Traffic Replayer, Migration Console, `Reindex-from-Snapshot`) to access the domain. + +### Managed service role mapping (Cross-managed-cluster migrations) + +When migrating between two managed clusters, for example, when both domains were created using Amazon OpenSearch Service, provide Migration Assistant components with sufficient permissions to modify both the source and target clusters. + +Use the following steps to grant the required permissions: + +1. In the AWS Management Console, navigate to **CloudFormation** > **Stacks**. +2. Locate the stack that starts with `OSMigrations--` (created during CDK deployment). +3. Go to the **Resources** tab and locate the following IAM roles: + + ```bash + arn:aws:iam::****:role/OSMigrations---MigrationServiceTaskRoleC- + arn:aws:iam::****:role/OSMigrations---reindexfromsnapshotTaskRo- + arn:aws:iam::****:role/OSMigrations---trafficreplayerdefaultTas- + ``` + +4. In both the source and target clusters, map users to each Amazon Resource Name (ARN) using the following steps: + A. Access OpenSearch Dashboards. If you're using Elasticsearch, access Kibana. + B. Navigate to **Security -> Roles -> all_access**. + C. In the "Mapped users" section, add each ARN as a backend role. + D. Save your changes. + +### OpenSearch Serverless + +For an OpenSearch Serverless Collection, you will need to configure both network and data access policies: + +1. **Network Policy Configuration**: + The Collection should have a network policy that uses the `VPC` access type. This requires creating a VPC endpoint on the VPC used for the solution. The VPC endpoint should be configured for the private subnets of the VPC and should attach the `osClusterAccessSG` security group. + +2. **Data Access Policy Configuration**: + The data access policy should grant permission to perform all [index operations](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-data-access.html#serverless-data-supported-permissions) (`aoss:*`) for all indexes in the Collection. The IAM task roles of the applicable Migration services (Traffic Replayer, migration console, `Reindex-from-Snapshot`) should be used as the principals for this data access policy. + +## Capture Proxy on Coordinator Nodes of Source Cluster + +Although the CDK does not automatically set up the Capture Proxy on source cluster nodes (except in the demo solution), the Capture Proxy instances must communicate with the resources deployed by the CDK, such as Kafka. This section outlines the necessary steps to set up communication. + +Before [setting up Capture Proxy instances](https://github.com/opensearch-project/opensearch-migrations/tree/main/TrafficCapture/trafficCaptureProxyServer#installing-capture-proxy-on-coordinator-nodes) on the source cluster, ensure the following configurations are in place: + +1. **Security Group Configuration**: + The coordinator nodes should add the `trafficStreamSourceSG` security group to allow sending captured traffic to Kafka. + +2. **IAM Policy Configuration**: + The IAM role used by the coordinator nodes should have permissions to publish captured traffic to Kafka. You can add the following template policy through the AWS Console (IAM Role → Add permissions → Create inline policy → JSON view): + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Action": "kafka-cluster:Connect", + "Resource": "arn:aws:kafka:::cluster/migration-msk-cluster-/*", + "Effect": "Allow" + }, + { + "Action": [ + "kafka-cluster:CreateTopic", + "kafka-cluster:DescribeTopic", + "kafka-cluster:WriteData" + ], + "Resource": "arn:aws:kafka:::topic/migration-msk-cluster-/*", + "Effect": "Allow" + } + ] +} +``` diff --git a/_migration-assistant/migration-phases/deploying-migration-assistant/index.md b/_migration-assistant/migration-phases/deploying-migration-assistant/index.md new file mode 100644 index 00000000000..1c559a81b1c --- /dev/null +++ b/_migration-assistant/migration-phases/deploying-migration-assistant/index.md @@ -0,0 +1,13 @@ +--- +layout: default +title: Deploying Migration Assistant +nav_order: 15 +has_children: true +permalink: /deploying-migration-assistant/ +redirect-from: + - /deploying-migration-assistant/index/ +--- + +# Deploying Migration Assistant + +This section provides information about the available options for deploying Migration Assistant. diff --git a/_migration-assistant/migration-phases/deployment.md b/_migration-assistant/migration-phases/deployment.md new file mode 100644 index 00000000000..2c4da875663 --- /dev/null +++ b/_migration-assistant/migration-phases/deployment.md @@ -0,0 +1,341 @@ +--- +layout: default +title: Deploy +parent: Migration phases +nav_order: 2 +redirect_from: + - /upgrade-to/snapshot-migrate/ + - /migration-assistant/getting-started-with-data-migration/ +--- + +# Deploying Migration Assistant +This document assumes you have performed assessment. + +--- + +## Step 1: Install Bootstrap on an Amazon EC2 instance (~10 minutes) + +To begin your migration, use the following steps to install a `bootstrap` box on an Amazon Elastic Compute Cloud (Amazon EC2) instance. The instance uses AWS CloudFormation to create and manage the stack. + +1. Log in to the target AWS account in which you want to deploy Migration Assistant. +2. From the browser where you are logged in to your target AWS account, right-click [here](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/migration-assistant-for-amazon-opensearch-service/latest/migration-assistant-for-amazon-opensearch-service.template&redirectId=SolutionWeb) to load the CloudFormation template from a new browser tab. +3. Follow the CloudFormation stack wizard: + * **Stack Name:** `MigrationBootstrap` + * **Stage Name:** `dev` + * Choose **Next** after each step > **Acknowledge** > **Submit**. +4. Verify that the Bootstrap stack exists and is set to `CREATE_COMPLETE`. This process takes around 10 minutes to complete. + +--- + +## Step 2: Set up Bootstrap instance access (~5 minutes) + +Use the following steps to set up Bootstrap instance access: + +1. After deployment, find the EC2 instance ID for the `bootstrap-dev-instance`. +2. Create an AWS Identity and Access Management (IAM) policy using the following snippet, replacing ``, ``, ``, and `` with your information: + + ```json + { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": "ssm:StartSession", + "Resource": [ + "arn:aws:ec2:::instance/", + "arn:aws:ssm:::document/BootstrapShellDoc--" + ] + } + ] + } + ``` + {% include copy.html %} + +3. Name the policy, for example, `SSM-OSMigrationBootstrapAccess`, and then create the policy by selecting **Create policy**. +4. Attach the newly created policy to your EC2 instance's IAM role. + +--- + +## Step 3: Log in to Bootstrap and building Migration Assistant (~15 minutes) + +Next, log in to Bootstrap and build Migration Assistant using the following steps. + +### Prerequisites + +To use these steps, make sure you fulfill the following prerequisites: + +* The AWS Command Line Interface (AWS CLI) and AWS Session Manager plugin are installed on your instance. +* The AWS credentials are configured (`aws configure`) for your instance. + +### Steps + +1. Load AWS credentials into your terminal. +2. Log in to the instance using the following command, replacing `` and `` with your instance ID and Region: + + ```bash + aws ssm start-session --document-name BootstrapShellDoc-- --target --region [--profile ] + ``` + {% include copy.html %} + +3. Once logged in, run the following command from the shell of the Bootstrap instance in the `/opensearch-migrations` directory: + + ```bash + ./initBootstrap.sh && cd deployment/cdk/opensearch-service-migration + ``` + {% include copy.html %} + +4. After a successful build, note the path for infrastructure deployment, which will be used in the next step. + +--- + +## Step 4: Configure and deploy RFS (~20 minutes) + +To deploy Migration Assistant with RFS, the following stacks must be deployed: + +These commands deploy the following stacks: + +* `Migration Assistant network` stack +* `RFS` stack +* `Migration console` stack + +Use the following steps to configure and deploy RFS, deploy Migration Assistant, and verify installation of the required stacks: + +1. Add the source and target cluster password as separate **Secrets** in [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) as an unstructured string. Be sure to copy the secret Amazon Resource Name (ARN) for use during deployment. +2. From the same shell as the Bootstrap instance, modify the `cdk.context.json` file located in the `/opensearch-migrations/deployment/cdk/opensearch-service-migration` directory and configure the following settings: + + ```json + { + "default": { + "stage": "dev", + "vpcId": "", + "targetCluster": { + "endpoint": "", + "auth": { + "type": "basic", + "username": "", + "passwordFromSecretArn": "" + } + }, + "sourceCluster": { + "endpoint": "", + "version": "", + "auth": { + "type": "basic", + "username": "", + "passwordFromSecretArn": "" + } + }, + "reindexFromSnapshotExtraArgs": "", + "reindexFromSnapshotMaxShardSizeGiB": 80, + "otelCollectorEnabled": true, + "migrationConsoleServiceEnabled": true + } + } + ``` + {% include copy.html %} + + The source and target cluster authorization can be configured to have no authorization, `basic` with a username and password, or `sigv4`. + +3. After the `cdk.context.json` file is fully configured, bootstrap the account and deploy the required stacks using the following command: + + ```bash + cdk bootstrap --c contextId=default --require-approval never + ``` + {% include copy.html %} + +4. Deploy Migration Assistant using the following command: + + ```bash + cdk deploy "*" --c contextId=default --require-approval never --concurrency 5 + ``` + {% include copy.html %} + +5. From the same Bootstrap instance shell, verify that all CloudFormation stacks were installed successfully: + + ```bash + aws cloudformation list-stacks --query "StackSummaries[?StackStatus!='DELETE_COMPLETE'].[StackName,StackStatus]" --output table + ``` + {% include copy.html %} + +You should receive a similar output for your Region: + +```bash +------------------------------------------------------------------------ +| ListStacks | ++--------------------------------------------------+-------------------+ +| OSMigrations-dev-us-east-1-MigrationConsole | CREATE_COMPLETE | +| OSMigrations-dev-us-east-1-ReindexFromSnapshot | CREATE_COMPLETE | +| OSMigrations-dev-us-east-1-MigrationInfra | CREATE_COMPLETE | +| OSMigrations-dev-us-east-1-default-NetworkInfra | CREATE_COMPLETE | +| MigrationBootstrap | CREATE_COMPLETE | +| CDKToolkit | CREATE_COMPLETE | ++--------------------------------------------------+-------------------+ +``` + +### RFS parameters + +If you're creating a snapshot using migration tooling, these parameters are automatically configured. If you're using an existing snapshot, modify the `reindexFromSnapshotExtraArgs` setting with the following values: + +```bash + "reindexFromSnapshotExtraArgs": "--s3-repo-uri s3:/// --s3-region --snapshot-name " +``` + +You will also need to give the `migrationconsole` and `reindexFromSnapshot` TaskRoles permissions to the S3 bucket. + +--- + +## Step 5: Access the migration console + +Run the following command to access the migration console: + +```bash +./accessContainer.sh migration-console dev +``` +{% include copy.html %} + + +`accessContainer.sh` is located in `/opensearch-migrations/deployment/cdk/opensearch-service-migration/` on the Bootstrap instance. To learn more, see [Accessing the migration console]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrating-metadata/). +{: .note} + +--- + +## Step 6: Verify the connection to the source and target clusters + +To verify the connection to the clusters, run the following command: + +```bash +console clusters connection-check +``` +{% include copy.html %} + +You should receive the following output: + +```bash +SOURCE CLUSTER +ConnectionResult(connection_message='Successfully connected!', connection_established=True, cluster_version='') +TARGET CLUSTER +ConnectionResult(connection_message='Successfully connected!', connection_established=True, cluster_version='') +``` + +To learn more about migration console commands, see [Migration console command reference](https://docs.opensearch.org/docs/latest/migration-assistant/migration-console/migration-console-commands-references/). + +--- + +## Step 7: Create a snapshot + +Run the following command to initiate snapshot creation from the source cluster: + +```bash +console snapshot create [...] +``` +{% include copy.html %} + +To check the snapshot creation status, run the following command: + +```bash +console snapshot status [...] +``` +{% include copy.html %} + +To learn more information about the snapshot, run the following command: + +```bash +console snapshot status --deep-check [...] +``` +{% include copy.html %} + +Wait for snapshot creation to complete before moving to step 9. + +To learn more about snapshot creation, see [Snapshot Creation]. + +--- + +## Step 8: Migrate metadata + +Run the following command to migrate metadata: + +```bash +console metadata migrate [...] +``` +{% include copy.html %} + +For more information, see [Migrating metadata]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrating-metadata/). + +--- + +## Step 9: Migrate documents with RFS + +You can now use RFS to migrate documents from your original cluster: + +1. To start the migration from RFS, start a `backfill` using the following command: + + ```bash + console backfill start + ``` + {% include copy.html %} + +2. _(Optional)_ To speed up the migration, increase the number of documents processed at a simultaneously by using the following command: + + ```bash + console backfill scale + ``` + {% include copy.html %} + +3. To check the status of the documentation backfill, use the following command: + + ```bash + console backfill status + ``` + {% include copy.html %} + +4. If you need to stop the backfill process, use the following command: + + ```bash + console backfill stop + ``` + {% include copy.html %} + +For more information, see [Backfill]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/backfill/). + +--- + +## Step 10: Backfill monitoring + +Use the following command for detailed monitoring of the backfill process: + +```bash +console backfill status --deep-check +``` +{% include copy.html %} + +You should receive the following output: + +```json +BackfillStatus.RUNNING +Running=9 +Pending=1 +Desired=10 +Shards total: 62 +Shards completed: 46 +Shards incomplete: 16 +Shards in progress: 11 +Shards unclaimed: 5 +``` + +Logs and metrics are available in Amazon CloudWatch in the `OpenSearchMigrations` log group. + +--- + +## Step 11: Verify that all documents were migrated + +Use the following query in CloudWatch Logs Insights to identify failed documents: + +```bash +fields @message +| filter @message like "Bulk request succeeded, but some operations failed." +| sort @timestamp desc +| limit 10000 +``` +{% include copy.html %} + +If any failed documents are identified, you can index the failed documents directly as opposed to using RFS. diff --git a/_migration-assistant/migration-phases/index.md b/_migration-assistant/migration-phases/index.md new file mode 100644 index 00000000000..8a7dcc95ab7 --- /dev/null +++ b/_migration-assistant/migration-phases/index.md @@ -0,0 +1,87 @@ +--- +layout: default +title: Migration phases +parent: Migration Assistant for OpenSearch +nav_order: 30 +has_children: false +has_toc: true +permalink: /migration-assistant/overview/ +redirect-from: /migration-assistant/overview/index/ +--- + +# Migration phases + +This page outlines the core phases of migrating with Migration Assistant. Each scenario below consists of a sequence of common steps. Expand a scenario to explore the detailed flow. + +--- + + + +
    +Scenario 1 – Backfill Only + +
      +
    1. Assessment
      2. +
    2. Deployment
      3. + +
    3. Create Snapshot
      4. +
    4. Migrate Metadata
      5. +
    5. Migrate Data
      6. +
    6. Teardown
      7. +
    + +
    + +
    +Scenario 2 – Live Capture Only + +
      +
    1. Assessment
      2. +
    2. Deployment
      3. +
    3. Verify Backfill Components
      4. +
    4. Reroute Traffic from Source to Capture Proxy
      5. +
    5. Migrate Metadata
      6. +
    6. Verify Live Capture Components
      7. +
    7. Replay Captured Traffic
      8. +
    8. Reroute Traffic from Capture Proxy to Target
      9. +
    9. Teardown
      10. +
    + +
    + +
    +Scenario 3 – Live Capture with Backfill + +
      +
    1. Assessment
      2. +
    2. Deployment
      3. +
    3. Reroute Traffic from Source to Capture Proxy
      4. +
    4. Create Snapshot
      5. +
    5. Migrate Metadata
      6. +
    6. Migrate Data
      7. +
    7. Replay Traffic
      8. +
    8. Reroute Traffic from Capture Proxy to Target
      9. +
    9. Teardown
      10. +
    + +
    + +--- + +💡 *Need help getting started? Begin with [Planning your migration]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/planning-your-migration/index/).* diff --git a/_migration-assistant/migration-phases/migrate-metadata.md b/_migration-assistant/migration-phases/migrate-metadata.md new file mode 100644 index 00000000000..39c1d7d1149 --- /dev/null +++ b/_migration-assistant/migration-phases/migrate-metadata.md @@ -0,0 +1,249 @@ +--- +layout: default +title: Migrate Metadata +nav_order: 5 +parent: Migration phases +redirect_from: + - /migration-assistant/migration-phases/migrating-metadata +--- + +# Migrate metadata + +Metadata migration involves creating a snapshot of your cluster and then migrating the metadata from the snapshot using the migration console. + +This tool gathers information from a source cluster through a snapshot or through HTTP requests against the source cluster. These snapshots are fully compatible with the backfill process for `Reindex-From-Snapshot` (RFS) scenarios. + +After collecting information on the source cluster, comparisons are made against the target cluster. If running a migration, any metadata items that do not already exist will be created on the target cluster. + +## Creating the snapshot + +Creating a snapshot of the source cluster captures all the metadata and documents to be migrated to a new target cluster. + +Create the initial snapshot of the source cluster using the following command: + +```shell +console snapshot create +``` +{% include copy.html %} + +To check the progress of the snapshot in real time, use the following command: + +```shell +console snapshot status --deep-check +``` +{% include copy.html %} + +You should receive the following response when the snapshot is created: + +```shell +SUCCESS +Snapshot is SUCCESS. +Percent completed: 100.00% +Data GiB done: 29.211/29.211 +Total shards: 40 +Successful shards: 40 +Failed shards: 0 +Start time: 2024-07-22 18:21:42 +Duration: 0h 13m 4s +Anticipated duration remaining: 0h 0m 0s +Throughput: 38.13 MiB/sec +``` + +### Managing slow snapshot speeds + +Depending on the size of the data in the source cluster and the bandwidth allocated for snapshots, the process can take some time. Adjust the maximum rate at which the source cluster's nodes create the snapshot using the `--max-snapshot-rate-mb-per-node` option. Increasing the snapshot rate will consume more node resources, which may affect the cluster's ability to handle normal traffic. + +## Command arguments + +For the following commands, to identify all valid arguments, please run with `--help`. + +```shell +console metadata evaluate --help +``` +{% include copy.html %} + +```shell +console metadata migrate --help +``` +{% include copy.html %} + +Based on the migration console deployment options, a number of commands will be pre-populated. To view them, run console with verbosity: + +```shell +console -v metadata migrate --help +``` +{% include copy.html %} + +You should receive a response similar to the following: + +```shell +(.venv) bash-5.2# console -v metadata migrate --help +INFO:console_link.cli:Logging set to INFO +. +. +. +INFO:console_link.models.metadata:Migrating metadata with command: /root/metadataMigration/bin/MetadataMigration --otel-collector-endpoint http://otel-collector:4317 migrate --snapshot-name snapshot_2023_01_01 --target-host https://opensearchtarget:9200 --min-replicas 0 --file-system-repo-path /snapshot/test-console --target-username admin --target-password ******** --target-insecure --help +. +. +. +``` + + +## Using the `evaluate` command + +By scanning the contents of the source cluster, applying filtering, and applying modifications a list of all items that will be migrated will be created. Any items not seen in this output will not be migrated onto the target cluster if the migrate command was to be run. This is a safety check before making modifications on the target cluster. + +```shell +console metadata evaluate [...] +``` +{% include copy.html %} + +You should receive a response similar to the following: + +```bash +Starting Metadata Evaluation +Clusters: + Source: + Remote Cluster: OpenSearch 1.3.16 ConnectionContext(uri=http://localhost:33039, protocol=HTTP, insecure=false, compressionSupported=false) + + Target: + Remote Cluster: OpenSearch 2.14.0 ConnectionContext(uri=http://localhost:33037, protocol=HTTP, insecure=false, compressionSupported=false) + + +Migration Candidates: + Index Templates: + simple_index_template + + Component Templates: + simple_component_template + + Indexes: + blog_2023, movies_2023 + + Aliases: + alias1, movies-alias + + +Results: + 0 issue(s) detected +``` + + +## Using the migrate command + +Running through the same data as the evaluate command all of the migrated items will be applied onto the target cluster. If re-run multiple times items that were previously migrated will not be recreated. If any items do need to be re-migrated, please delete them from the target cluster and then rerun the evaluate then migrate commands to ensure the desired changes are made. + +```shell +console metadata migrate [...] +``` +{% include copy.html %} + +You should receive a response similar to the following: + +```shell +Starting Metadata Migration + +Clusters: + Source: + Snapshot: OpenSearch 1.3.16 FileSystemRepo(repoRootDir=/tmp/junit10626813752669559861) + + Target: + Remote Cluster: OpenSearch 2.14.0 ConnectionContext(uri=http://localhost:33042, protocol=HTTP, insecure=false, compressionSupported=false) + + +Migrated Items: + Index Templates: + simple_index_template + + Component Templates: + simple_component_template + + Indexes: + blog_2023, movies_2023 + + Aliases: + alias1, movies-alias + + +Results: + 0 issue(s) detected +``` + + +## Metadata verification process + +Before moving on to additional migration steps, it is recommended to confirm details of your cluster. Depending on your configuration, this could be checking the sharding strategy or making sure index mappings are correctly defined by ingesting a test document. + +## Troubleshooting + +Use these instructions to help troubleshoot the following issues. + +### Accessing detailed logs + +Metadata migration creates a detailed log file that includes low level tracing information for troubleshooting. For each execution of the program a log file is created inside a shared volume on the migration console named `shared-logs-output` the following command will list all log files, one for each run of the command. + +```shell +ls -al /shared-logs-output/migration-console-default/*/metadata/ +``` +{% include copy.html %} + +To inspect the file within the console `cat`, `tail` and `grep` commands line tools. By looking for warnings, errors and exceptions in this log file can help understand the source of failures, or at the very least be useful for creating issues in this project. + +```shell +tail /shared-logs-output/migration-console-default/*/metadata/*.log +``` +{% include copy.html %} + +### Warnings and errors + +When encountering `WARN` or `ERROR` elements in the response, they will be accompanied by a short message, such as `WARN - my_index already exists`. More information can be found in the detailed logs associated with the warning or error. + +### OpenSearch running in compatibility mode + +There might be an error about being unable to update an ES 7.10.2 cluster, this can occur when compatibility mode has been enabled on an OpenSearch cluster disable it to continue, see [Enable compatibility mode](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/rename.html#rename-upgrade). + + +### Breaking change compatibility + +Metadata migration requires modifying data from the source to the target versions to recreate items. Sometimes these features are no longer supported and have been removed from the target version. Sometimes these features are not available in the target version, which is especially true when downgrading. While this tool is meant to make this process easier, it is not exhaustive in its support. When encountering a compatibility issue or an important feature gap for your migration, [search the issues and comment on the existing issue](https://github.com/opensearch-project/opensearch-migrations/issues) or [create a new](https://github.com/opensearch-project/opensearch-migrations/issues/new/choose) issue if one cannot be found. + +#### Deprecation of Mapping Types + +In Elasticsearch 6.8 the mapping types feature was discontinued in Elasticsearch 7.0+ which has created complexity in migrating to newer versions of Elasticsearch and OpenSearch, [learn more](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/removal-of-types.html) ↗. + +As Metadata migration supports migrating from ES 6.8 on to the latest versions of OpenSearch this scenario is handled by removing the type mapping types and restructuring the template or index properties. Note that, at the time of this writing multiple type mappings are not supported, [tracking task](https://opensearch.atlassian.net/browse/MIGRATIONS-1778) ↗. + + +**Example starting state with mapping type foo (ES 6):** + +```json +{ + "mappings": [ + { + "foo": { + "properties": { + "field1": { "type": "text" }, + "field2": { "type": "keyword" } + } + } + } + ] +} +``` +{% include copy.html %} + +**Example ending state with foo removed (ES 7):** + +```json +{ + "mappings": { + "properties": { + "field1": { "type": "text" }, + "field2": { "type": "keyword" }, + } + } +} +``` +{% include copy.html %} + +For additional technical details, [view the mapping type removal source code](https://github.com/opensearch-project/opensearch-migrations/blob/main/transformation/src/main/java/org/opensearch/migrations/transformation/rules/IndexMappingTypeRemoval.java). diff --git a/_migration-assistant/migration-phases/replay-captured-traffic.md b/_migration-assistant/migration-phases/replay-captured-traffic.md new file mode 100644 index 00000000000..0f4d1d6ea2f --- /dev/null +++ b/_migration-assistant/migration-phases/replay-captured-traffic.md @@ -0,0 +1,319 @@ +--- +layout: default +title: Replay +nav_order: 7 +grand_parent: Migration phases +parent: Live traffic migration +redirect_from: + - /migration-assistant/migration-phases/using-traffic-replayer/ +--- + +# Using Traffic Replayer + +This guide covers how to use Traffic Replayer to replay captured traffic from a source cluster to a target cluster during the migration process. Traffic Replayer allows you to verify that the target cluster can handle requests in the same way as the source cluster and catch up to real-time traffic for a smooth migration. + +## When to run Traffic Replayer + +After deploying Migration Assistant, Traffic Replayer does not run by default. It should be started only after all metadata and documents have been migrated to ensure that recent changes to the source cluster are properly reflected in the target cluster. + +For example, if a document was deleted after a snapshot was taken, starting Traffic Replayer before the document migration is complete may cause the deletion request to execute before the document is added to the target. Running Traffic Replayer after all other migration processes ensures that the target cluster will be consistent with the source cluster. + +## Configuration options + +[Traffic Replayer settings]({{site.url}}{{site.baseurl}}/migration-assistant/deploying-migration-assistant/configuration-options/) are configured during the deployment of Migration Assistant. Make sure to set the authentication mode for Traffic Replayer so that it can properly communicate with the target cluster. + +## Using Traffic Replayer + +To manage Traffic Replayer, use the `console replay` command. The following examples show the available commands. + +### Start Traffic Replayer + +The following command starts Traffic Replayer with the options specified at deployment: + +```bash +console replay start +``` + +When starting Traffic Replayer, you should receive an output similar to the following: + +```bash +root@ip-10-0-2-66:~# console replay start +Replayer started successfully. +Service migration-dev-traffic-replayer-default set to 1 desired count. Currently 0 running and 0 pending. +``` + +## Check the status of Traffic Replayer + +Use the following command to show the status of Traffic Replayer: + +```bash +console replay status +``` + +Replay will return one of the following statuses: + +- `Running` shows how many container instances are actively running. +- `Pending` indicates how many instances are being provisione.d +- `Desired` shows the total number of instances that should be running. + +You should receive an output similar to the following: + +```bash +root@ip-10-0-2-66:~# console replay status +(, 'Running=0\nPending=0\nDesired=0') +``` + +## Stop Traffic Replayer + +The following command stops Traffic Replayer: + +```bash +console replay stop +``` + +You should receive an output similar to the following: + +```bash +root@ip-10-0-2-66:~# console replay stop +Replayer stopped successfully. +Service migration-dev-traffic-replayer-default set to 0 desired count. Currently 0 running and 0 pending. +``` + + + +### Delivery guarantees + +Traffic Replayer retrieves traffic from Kafka and updates its commit cursor after sending requests to the target cluster. This provides an "at least once" delivery guarantee; however, success isn't always guaranteed. Therefore, you should monitor metrics and tuple outputs or perform external validation to ensure that the target cluster is functioning as expected. + +## Time scaling + +Traffic Replayer sends requests in the same order that they were received from each connection to the source. However, relative timing between different connections is not guaranteed. For example: + +- **Scenario**: Two connections exist:one sends a PUT request every minute, and the other sends a GET request every second. +- **Behavior**: Traffic Replayer will maintain the sequence within each connection, but the relative timing between the connections (PUTs and GETs) is not preserved. + +Assume that a source cluster responds to requests (GETs and PUTs) within 100 ms: + +- With a **speedup factor of 1**, the target will experience the same request rates and idle periods as the source. +- With a **speedup factor of 2**, requests will be sent twice as fast, with GETs sent every 500 ms and PUTs every 30 seconds. +- With a **speedup factor of 10**, requests will be sent 10x faster, and as long as the target responds quickly, Traffic Replayer can maintain the pace. + +If the target cannot respond fast enough, Traffic Replayer will wait for the previous request to complete before sending the next one. This may cause delays and affect global relative ordering. + +## Transformations + +During migrations, some requests may need to be transformed between versions. For example, Elasticsearch previously supported multiple type mappings in indexes, but this is no longer the case in OpenSearch. Clients may need to be adjusted accordingly by splitting documents into multiple indexes or transforming request data. + +Traffic Replayer automatically rewrites host and authentication headers, but for more complex transformations, custom transformation rules can be specified using the `--transformer-config` option. For more information, see the [Traffic Replayer README](https://github.com/opensearch-project/opensearch-migrations/blob/c3d25958a44ec2e7505892b4ea30e5fbfad4c71b/TrafficCapture/trafficReplayer/README.md#transformations). + +### Example transformation + +Suppose that a source request contains a `tagToExcise` element that needs to be removed and its children promoted and that the URI path includes `extraThingToRemove`, which should also be removed. The following Jolt script handles this transformation: + +```json +[{ "JsonJoltTransformerProvider": +[ + { + "script": { + "operation": "shift", + "spec": { + "payload": { + "inlinedJsonBody": { + "top": { + "tagToExcise": { + "*": "payload.inlinedJsonBody.top.&" + }, + "*": "payload.inlinedJsonBody.top.&" + }, + "*": "payload.inlinedJsonBody.&" + }, + "*": "payload.&" + }, + "*": "&" + } + } + }, + { + "script": { + "operation": "modify-overwrite-beta", + "spec": { + "URI": "=split('/extraThingToRemove',@(1,&))" + } + } + }, + { + "script": { + "operation": "modify-overwrite-beta", + "spec": { + "URI": "=join('',@(1,&))" + } + } + } +] +}] +``` + +The resulting request sent to the target will appear similar to the following: + +```bash +PUT /oldStyleIndex/moreStuff HTTP/1.0 +host: testhostname + +{"top":{"properties":{"field1":{"type":"text"},"field2":{"type":"keyword"}}}} +``` +{% include copy.html %} + +You can pass Base64-encoded transformation scripts using `--transformer-config-base64`. + +## Result logs + +HTTP transactions from the source capture and those resent to the target cluster are logged in files located at `/shared-logs-output/traffic-replayer-default/*/tuples/tuples.log`. The `/shared-logs-output` directory is shared across containers, including the migration console. You can access these files from the migration console using the same path. Previous runs are also available in a `gzipped` format. + +Each log entry is a newline-delimited JSON object, containing information about the source and target requests/responses along with other transaction details, such as response times. + +These logs contain the contents of all requests, including authorization headers and the contents of all HTTP messages. Ensure that access to the migration environment is restricted, as these logs serve as a source of truth for determining what happened in both the source and target clusters. Response times for the source refer to the amount of time between the proxy sending the end of a request and receiving the response. While response times for the target are recorded in the same manner, keep in mind that the locations of the capture proxy, Traffic Replayer, and target may differ and that these logs do not account for the client's location. +{: .note} + + +### Example log entry + +The following example log entry shows a `/_cat/indices?v` request sent to both the source and target clusters: + +```json +{ + "sourceRequest": { + "Request-URI": "/_cat/indices?v", + "Method": "GET", + "HTTP-Version": "HTTP/1.1", + "Host": "capture-proxy:9200", + "Authorization": "Basic YWRtaW46YWRtaW4=", + "User-Agent": "curl/8.5.0", + "Accept": "*/*", + "body": "" + }, + "sourceResponse": { + "HTTP-Version": {"keepAliveDefault": true}, + "Status-Code": 200, + "Reason-Phrase": "OK", + "response_time_ms": 59, + "content-type": "text/plain; charset=UTF-8", + "content-length": "214", + "body": "aGVhbHRoIHN0YXR1cyBpbmRleCAgICAgICB..." + }, + "targetRequest": { + "Request-URI": "/_cat/indices?v", + "Method": "GET", + "HTTP-Version": "HTTP/1.1", + "Host": "opensearchtarget", + "Authorization": "Basic YWRtaW46bXlTdHJvbmdQYXNzd29yZDEyMyE=", + "User-Agent": "curl/8.5.0", + "Accept": "*/*", + "body": "" + }, + "targetResponses": [{ + "HTTP-Version": {"keepAliveDefault": true}, + "Status-Code": 200, + "Reason-Phrase": "OK", + "response_time_ms": 721, + "content-type": "text/plain; charset=UTF-8", + "content-length": "484", + "body": "aGVhbHRoIHN0YXR1cyBpbmRleCAgICAgICB..." + }], + "connectionId": "0242acfffe13000a-0000000a-00000005-1eb087a9beb83f3e-a32794b4.0", + "numRequests": 1, + "numErrors": 0 +} +``` +{% include copy.html %} + + +### Decoding log content + +The contents of HTTP message bodies are Base64 encoded in order to handle various types of traffic, including compressed data. To view the logs in a more human-readable format, use the console library `tuples show`. Running the script as follows will produce a `readable-tuples.log` in the home directory: + +```shell +console tuples show --in /shared-logs-output/traffic-replayer-default/d3a4b31e1af4/tuples/tuples.log > readable-tuples.log +``` + +The `readable-tuples.log` should appear similar to the following: + +```json +{ + "sourceRequest": { + "Request-URI": "/_cat/indices?v", + "Method": "GET", + "HTTP-Version": "HTTP/1.1", + "Host": "capture-proxy:9200", + "Authorization": "Basic YWRtaW46YWRtaW4=", + "User-Agent": "curl/8.5.0", + "Accept": "*/*", + "body": "" + }, + "sourceResponse": { + "HTTP-Version": {"keepAliveDefault": true}, + "Status-Code": 200, + "Reason-Phrase": "OK", + "response_time_ms": 59, + "content-type": "text/plain; charset=UTF-8", + "content-length": "214", + "body": "health status index uuid ..." + }, + "targetRequest": { + "Request-URI": "/_cat/indices?v", + "Method": "GET", + "HTTP-Version": "HTTP/1.1", + "Host": "opensearchtarget", + "Authorization": "Basic YWRtaW46bXlTdHJvbmdQYXNzd29yZDEyMyE=", + "User-Agent": "curl/8.5.0", + "Accept": "*/*", + "body": "" + }, + "targetResponses": [{ + "HTTP-Version": {"keepAliveDefault": true}, + "Status-Code": 200, + "Reason-Phrase": "OK", + "response_time_ms": 721, + "content-type": "text/plain; charset=UTF-8", + "content-length": "484", + "body": "health status index uuid ..." + }], + "connectionId": "0242acfffe13000a-0000000a-00000005-1eb087a9beb83f3e-a32794b4.0", + "numRequests": 1, + "numErrors": 0 +} +``` + + +## Amazon CloudWatch metrics and dashboard +Migration Assistant creates an Amazon CloudWatch dashboard named `MigrationAssistant_ReindexFromSnapshot_Dashboard` to visualize the health and performance of the backfill process. This dashboard combines metrics for the backfill workers and migration to Amazon OpenSearch Service, providing insights into the performance and health of the Capture Proxy and Traffic Replayer components, including metrics such as: + +- The number of bytes read and written. +- The number of active connections. +- The replay speed multiplier. + +You can find the Capture and Replay dashboard in the AWS Management Console for CloudWatch Dashboards in the AWS Region where you deployed Migration Assistant. + +Traffic Replayer emits various OpenTelemetry metrics to Amazon CloudWatch, and traces are sent through AWS X-Ray. The following are some useful metrics that can help evaluate migration performance. + +### `sourceStatusCode` + +This metric tracks the HTTP status codes for both the source and target clusters, with dimensions for the HTTP verb, such as `GET` or `POST`, and the status code families (200--299). These dimensions can help quickly identify discrepancies between the source and target, such as when `DELETE 200s` becomes `4xx` or `GET 4xx` errors turn into `5xx` errors. + +### `lagBetweenSourceAndTargetRequests` + +This metric shows the delay between requests hitting the source and target clusters. With a speedup factor greater than 1 and a target cluster that can handle requests efficiently, this value should decrease as the replay progresses, indicating a reduction in replay lag. + +### Additional metrics + +The following metrics are also reported: + +- **Throughput**: `bytesWrittenToTarget` and `bytesReadFromTarget` indicate the throughput to and from the cluster. +- **Retries**: `numRetriedRequests` tracks the number of requests retried due to status code mismatches between the source and target. +- **Event counts**: Various `(*)Count` metrics track the number of completed events. +- **Durations**: `(*)Duration` metrics measure the duration of each step in the process. +- **Exceptions**: `(*)ExceptionCount` shows the number of exceptions encountered during each processing phase. + + +## CloudWatch considerations + +Metrics and dashboards pushed to CloudWatch may experience a visibility lag of around 5 minutes. CloudWatch also retains higher-resolution data for a shorter period than lower-resolution data. For more information, see [Amazon CloudWatch concepts](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html). \ No newline at end of file diff --git a/_migration-assistant/migration-phases/reroute-source-to-proxy.md b/_migration-assistant/migration-phases/reroute-source-to-proxy.md new file mode 100644 index 00000000000..9bfa70bfd2b --- /dev/null +++ b/_migration-assistant/migration-phases/reroute-source-to-proxy.md @@ -0,0 +1,102 @@ +--- +layout: default +title: Reroute Client Traffic +nav_order: 3 +parent: Migration phases +redirect_from: + - /migration-assistant/migration-phases/resource-source-to-proxy +--- + +## Capture proxy data replication + +If you're interested in capturing live traffic during your migration, Migration Assistant includes an Application Load Balancer for routing traffic to the capture proxy and the target cluster. Upstream client traffic must be routed through the capture proxy in order to replay the requests later. Before using the capture proxy, remember the following: + +* The layer upstream from the Application Load Balancer is compatible with the certificate on the Application Load Balancer listener, whether it's for clients or a Network Load Balancer. The `albAcmCertArn` in the `cdk.context.json` may need to be provided to ensure that clients trust the Application Load Balancer certificate. +* If a Network Load Balancer is used directly upstream of the Application Load Balancer, it must use a TLS listener. +* Upstream resources and security groups must allow network access to the Migration Assistant Application Load Balancer. + +To set up the capture proxy, go to the AWS Management Console and navigate to **EC2 > Load Balancers > Migration Assistant Application Load Balancer**. Copy the Application Load Balancer URL. With the URL copied, you can use one of the following options. + + + +### If you are using **Network Load Balancer → Application Load Balancer → Cluster** + +1. Ensure that ingress is provided directly to the Application Load Balancer for the capture proxy. +2. Create a target group for the Migration Assistant Application Load Balancer on port `9200`, and set the health check to `HTTPS`. +3. Associate this target group with your existing Network Load Balancer on a new listener for testing. +4. Verify that the health check is successful, and perform smoke testing with some clients through the new listener port. +5. Once you are ready to migrate all clients, detach the Migration Assistant Application Load Balancer target group from the testing Network Load Balancer listener and modify the existing Network Load Balancer listener to direct traffic to this target group. +6. Now client requests will be routed through the proxy (once they establish a new connection). Verify the application metrics. + +### If you are using **Network Load Balancer → Cluster** + +If you do not want to modify application logic, add an Application Load Balancer in front of your cluster and follow the **Network Load Balancer → Application Load Balancer → Cluster** steps. Otherwise: + +1. Create a target group for the Application Load Balancer on port `9200` and set the health check to `HTTPS`. +2. Associate this target group with your existing Network Load Balancer on a new listener. +3. Verify that the health check is successful, and perform smoke testing with some clients through the new listener port. +4. Once you are ready to migrate all clients, deploy a change so that clients hit the new listener. + + +### If you are **not using an Network Load Balancer** + +If you're only using backfill as your migration technique, make a client/DNS change to route clients to the Migration Assistant Application Load Balancer on port `9200`. + + +### Kafka connection + +After you have routed the client based on your use case, test adding records against HTTP requests using the following steps: + +In the migration console, run the following command: + +```bash +console kafka describe-topic-records +``` +{% include copy.html %} + +Note the records in the logging topic. + +After a short period, execute the same command again and compare the increased number of records against the expected HTTP requests. + +## Creating a snapshot + +Create a snapshot for your backfill using the following command: + +```bash +console snapshot create +``` +{% include copy.html %} + +To check the progress of your snapshot, use the following command: + +```bash +console snapshot status --deep-check +``` +{% include copy.html %} + +Depending on the size of the data in the source cluster and the bandwidth allocated for snapshots, the process can take some time. Adjust the maximum rate at which the source cluster's nodes create the snapshot using the `--max-snapshot-rate-mb-per-node` option. Increasing the snapshot rate will consume more node resources, which may affect the cluster's ability to handle normal traffic. + +## Backfilling documents to the source cluster + +From the snapshot you created of your source cluster, you can begin backfilling documents into the target cluster. Once you have started this process, a fleet of workers will spin up to read the snapshot and reindex documents into the target cluster. This fleet of workers can be scaled to increased the speed at which documents are reindexed into the target cluster. + +### Checking the starting state of the clusters + +You can check the indexes and document counts of the source and target clusters by running the `cat-indices` command. This can be used to monitor the difference between the source and target for any migration scenario. Check the indexes of both clusters using the following command: + +```shell +console clusters cat-indices +``` +{% include copy.html %} + +You should receive the following response: + +```shell +SOURCE CLUSTER +health status index uuid pri rep docs.count docs.deleted store.size pri.store.size +green open my-index WJPVdHNyQ1KMKol84Cy72Q 1 0 8 0 44.7kb 44.7kb + +TARGET CLUSTER +health status index uuid pri rep docs.count docs.deleted store.size pri.store.size +green open .opendistro_security N3uy88FGT9eAO7FTbLqqqA 1 0 10 0 78.3kb 78.3kb +``` diff --git a/_migration-assistant/migration-phases/reroute-traffic-from-capture-proxy-to-target.md b/_migration-assistant/migration-phases/reroute-traffic-from-capture-proxy-to-target.md new file mode 100644 index 00000000000..314cc9a8647 --- /dev/null +++ b/_migration-assistant/migration-phases/reroute-traffic-from-capture-proxy-to-target.md @@ -0,0 +1,54 @@ +--- +layout: default +title: Cutover +nav_order: 7 +parent: Migration phases +redirect_from: + - /migration-assistant/migration-phases/switching-traffic-from-the-source-cluster/ +--- + +# Switching traffic from the source cluster + +After the source and target clusters are synchronized, traffic needs to be switched to the target cluster so that the source cluster can be taken offline. + +## Assumptions + +This page assumes that the following has occurred before making the switch: + +- All client traffic is being routed through a switchover listener in the [MigrationAssistant Application Load Balancer]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/backfill/). +- Client traffic has been verified as compatible with the target cluster. +- The target cluster is in a good state to accept client traffic. +- The target proxy service is deployed. + +## Switching traffic + +Use the following steps to switch traffic from the source cluster to the target cluster: + +1. In the AWS Management Console, navigate to **ECS** > **Migration Assistant Cluster**. Note the desired count of the capture proxy, which should be greater than 1. + +2. Update the **ECS Service** of the target proxy to be at least as large as the traffic capture proxy. Wait for tasks to start up, and verify that all targets are healthy in the target proxy service's **Load balancer health** section. + +3. Navigate to **EC2** > **Load Balancers** > **Migration Assistant ALB**. + +4. Navigate to **ALB Metrics** and examine any useful information, specifically looking at **Active Connection Count** and **New Connection Count**. Note any large discrepancies, which can indicate reused connections affecting traffic switchover. + +5. Navigate to **Capture Proxy Target Group** (`ALBSourceProxy--TG`) > **Monitoring**. + +6. Examine the **Metrics Requests**, **Target (2XX, 3XX, 4XX, 5XX)**, and **Target Response Time** metrics. Verify that this appears as expected and includes all traffic expected to be included in the switchover. Note details that could help identify anomalies during the switchover, including the expected response time and response code rate. + +7. Navigate back to **ALB Metrics** and choose **Target Proxy Target Group** (`ALBTargetProxy--TG`). Verify that all expected targets are healthy and that none are in a draining state. + +8. Navigate back to **ALB Metrics** and to the **Listener** on port `9200`. + +9. Choose the **Default rule** and **Edit**. + +10. Modify the weights of the targets to switch the desired traffic to the target proxy. To perform a full switchover, modify the **Target Proxy** weight to `1` and the **Source Proxy** weight to `0`. + +11. Choose **Save Changes**. + +12. Navigate to both **SourceProxy** and **TargetProxy TG Monitoring** metrics and verify that traffic is switching over as expected. If connections are being reused by clients, perform any necessary actions to terminate them. Monitor these metrics until **SourceProxy TG** shows 0 requests when all clients have switched over. + + +## Fallback + +If you need to fall back to the source cluster at any point during the switchover, revert the **Default rule** so that the Application Load Balancer routes to the **SourceProxy Target Group**. \ No newline at end of file diff --git a/_migration-assistant/migration-phases/teardown.md b/_migration-assistant/migration-phases/teardown.md new file mode 100644 index 00000000000..99e975c93b6 --- /dev/null +++ b/_migration-assistant/migration-phases/teardown.md @@ -0,0 +1,30 @@ +--- +layout: default +title: Remove Migration Infrastructure +nav_order: 8 +parent: Migration phases +redirect_from: + - /migration-assistant/migration-phases/removing-migration-infrastructure +--- + +# Removing migration infrastructure + +After a migration is complete, you should remove all resources except for the target cluster and, optionally, your Amazon CloudWatch logs and Traffic Replayer logs. + +To remove the AWS Cloud Development Kit (AWS CDK) stack(s) created during a deployment, run the following command within the CDK directory: + +```bash +cd deployment/cdk/opensearch-service-migration +cdk destroy "*" --c contextId= +``` +{% include copy.html %} + +Follow the instructions on the command line to remove the deployed resources from your AWS account. + +You can also use the AWS Management Console to remove Migration Assistant resources and confirm that they are no longer present in the account. + +## Uninstalling Migration Assistant for Amazon OpenSearch Service + +You can uninstall Migration Assistant for Amazon OpenSearch Service from the AWS Management Console or by using the AWS Command Line Interface (AWS CLI). Manually remove the contents of the Amazon Simple Storage Service (Amazon S3) bucket that matches the syntax `cdk--assets--`, the bucket created by Migration Assistant. Migration Assistant for Amazon OpenSearch Service does not automatically delete Amazon S3 buckets. + +To delete the stored data and the AWS CloudFormation stacks created by Migration Assistant, see [Uninstall the solution](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/uninstall-the-solution.html) in the Amazon OpenSearch Service documentation. diff --git a/_migration-assistant/migration-phases/verify-backfill-components.md b/_migration-assistant/migration-phases/verify-backfill-components.md new file mode 100644 index 00000000000..e8f99d7d88b --- /dev/null +++ b/_migration-assistant/migration-phases/verify-backfill-components.md @@ -0,0 +1,191 @@ +--- +layout: default +title: Backfill Setup Verification +nav_order: 3 +redirect_from: + - /migration-assistant/migration-phases/verifying-migration-tools/ +--- + +# Verifying Migration Assistant Backfill Components + +Before using the Migration Assistant, take the following steps to verify that your cluster is ready for migration. + +## Verifying snapshot creation + +Verify that a snapshot can be created of your source cluster and used for metadata and backfill scenarios. + +### Installing the Elasticsearch S3 Repository plugin + +The snapshot needs to be stored in a location that Migration Assistant can access. This guide uses Amazon Simple Storage Service (Amazon S3). By default, Migration Assistant creates an S3 bucket for storage. Therefore, it is necessary to install the [Elasticsearch S3 repository plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/7.10/repository-s3.html) on your source nodes (https://www.elastic.co/guide/en/elasticsearch/plugins/7.10/repository-s3.html). + +Additionally, make sure that the plugin has been configured with AWS credentials that allow it to read and write to Amazon S3. If your Elasticsearch cluster is running on Amazon Elastic Compute Cloud (Amazon EC2) or Amazon Elastic Container Service (Amazon ECS) instances with an AWS Identity and Access Management (IAM) execution role, include the necessary S3 permissions. Alternatively, you can store the credentials in the [Elasticsearch keystore](https://www.elastic.co/guide/en/elasticsearch/plugins/7.10/repository-s3-client.html). + +### Verifying the S3 repository plugin configuration + +You can verify that the S3 repository plugin is configured correctly by creating a test snapshot. + +Create an S3 bucket for the snapshot using the following AWS Command Line Interface (AWS CLI) command: + +```shell +aws s3api create-bucket --bucket --region +``` +{% include copy.html %} + +Register a new S3 snapshot repository on your source cluster using the following cURL command: + +```shell +curl -X PUT "http://:9200/_snapshot/test_s3_repository" -H "Content-Type: application/json" -d '{ + "type": "s3", + "settings": { + "bucket": "", + "region": "" + } +}' +``` +{% include copy.html %} + +Next, create a test snapshot that captures only the cluster's metadata: + +```shell +curl -X PUT "http://:9200/_snapshot/test_s3_repository/test_snapshot_1" -H "Content-Type: application/json" -d '{ + "indices": "", + "ignore_unavailable": true, + "include_global_state": true +}' +``` +{% include copy.html %} + +Check the AWS Management Console to confirm that your bucket contains the snapshot. + +### Removing test snapshots after verification + +To remove the resources created during verification, you can use the following deletion commands: + +**Test snapshot** + +```shell +curl -X DELETE "http://:9200/_snapshot/test_s3_repository/test_snapshot_1?pretty" +``` +{% include copy.html %} + +**Test snapshot repository** + +```shell +curl -X DELETE "http://:9200/_snapshot/test_s3_repository?pretty" +``` +{% include copy.html %} + +**S3 bucket** + +```shell +aws s3 rm s3:// --recursive +aws s3api delete-bucket --bucket --region +``` +{% include copy.html %} + +### Troubleshooting + +Use this guidance to troubleshoot any of the following snapshot verification issues. + +#### Access denied error (403) + +If you encounter an error like `AccessDenied (Service: Amazon S3; Status Code: 403)`, verify the following: + +- Make sure you're using the S3 bucket created by Migration Assistant. +- If you're using a custom S3 bucket, verify that: + - The IAM role assigned to your Elasticsearch cluster has the necessary S3 permissions. + - The bucket name and AWS Region provided in the snapshot configuration match the actual S3 bucket you created. + +#### Older versions of Elasticsearch + +Older versions of the Elasticsearch S3 repository plugin may have trouble reading IAM role credentials embedded in Amazon EC2 and Amazon ECS instances. This is because the copy of the AWS SDK shipped with them is too old to read the new standard way of retrieving those credentials, as shown in [the Instance Metadata Service v2 (IMDSv2) specification](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html). This can result in snapshot creation failures, with an error message similar to the following: + +```json +{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[migration_assistant_repo] path [rfs-snapshot-repo] is not accessible on master node"}],"type":"repository_verification_exception","reason":"[migration_assistant_repo] path [rfs-snapshot-repo] is not accessible on master node","caused_by":{"type":"i_o_exception","reason":"Unable to upload object [rfs-snapshot-repo/tests-s8TvZ3CcRoO8bvyXcyV2Yg/master.dat] using a single upload","caused_by":{"type":"amazon_service_exception","reason":"Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null)"}}},"status":500} +``` + +If you encounter this issue, you can resolve it by temporarily enabling IMDSv1 on the instances in your source cluster for the duration of the snapshot. There is a toggle for this available in the AWS Management Console as well as in the AWS CLI. Switching this toggle will turn on the older access model and enable the Elasticsearch S3 repository plugin to work as normal. For more information about IMDSv1, see [Modify instance metadata options for existing instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-IMDS-existing-instances.html). + +## Switching over client traffic + +The Migration Assistant Application Load Balancer is deployed with a listener that shifts traffic between the source and target clusters through proxy services. The Application Load Balancer should start in **Source Passthrough** mode. + +### Verifying that the traffic switchover is complete + +Use the following steps to verify that the traffic switchover is complete: + +1. In the AWS Management Console, navigate to **EC2 > Load Balancers**. +2. Select the **MigrationAssistant ALB**. +3. Examine the listener on port `9200` and verify that 100% of the traffic is directed to the **Source Proxy**. +4. Navigate to the **Migration ECS Cluster** in the AWS Management Console. +5. Select the **Target Proxy Service**. +6. Verify that the desired count for the service is running: + * If the desired count is not met, update the service to increase it to at least 1 and wait for the service to start. +7. On the **Health and Metrics** tab under **Load balancer health**, verify that all targets are reporting as healthy: + * This confirms that the Application Load Balancer can connect to the target cluster through the target proxy. +8. (Reset) Update the desired count for the **Target Proxy Service** back to its original value in Amazon ECS. + +### Fixing unidentified traffic patterns + +When switching over traffic to the target cluster, you might encounter unidentified traffic patterns. To help identify the cause of these patterns, use the following steps: +* Verify that the target cluster allows traffic ingress from the **Target Proxy Security Group**. +* Navigate to **Target Proxy ECS Tasks** to investigate any failing tasks. +Set the **Filter desired status** to **Any desired status** to view all tasks, then navigate to the logs for any stopped tasks. + + +## Verifying replication + +Use the following steps to verify that replication is working once the traffic capture proxy is deployed: + + +1. Navigate to the **Migration ECS Cluster** in the AWS Management Console. +2. Navigate to **Capture Proxy Service**. +3. Verify that the capture proxy is running with the desired proxy count. If it is not, update the service to increase it to at least 1 and wait for startup. +4. Under **Health and Metrics** > **Load balancer health**, verify that all targets are healthy. This means that the Application Load Balancer is able to connect to the source cluster through the capture proxy. +5. Navigate to the **Migration Console Terminal**. +6. Run `console kafka describe-topic-records`. Wait 30 seconds for another Application Load Balancer health check. +7. Run `console kafka describe-topic-records` again and verify that the number of RECORDS increased between runs. +8. Run `console replay start` to start Traffic Replayer. +9. Run `tail -f /shared-logs-output/traffic-replayer-default/*/tuples/tuples.log | jq '.targetResponses[]."Status-Code"'` to confirm that the Kafka requests were sent to the target and that it responded as expected. If the responses don't appear: + * Check that the migration console can access the target cluster by running `./catIndices.sh`, which should show the indexes in the source and target. + * Confirm that messages are still being recorded to Kafka. + * Check for errors in the Traffic Replayer logs (`/migration/STAGE/default/traffic-replayer-default`) using CloudWatch. +10. (Reset) Update the desired count for the **Capture Proxy Service** back to its original value in Amazon ECS. + +### Troubleshooting + +Use this guidance to troubleshoot any of the following replication verification issues. + +### Health check responses with 401/403 status code + +If the source cluster is configured to require authentication, the capture proxy will not be able to verify replication beyond receiving a 401/403 status code for Application Load Balancer health checks. For more information, see [Failure Modes](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/README.md#failure-modes). + +### Traffic does not reach the source cluster + +Verify that the source cluster allows traffic ingress from the Capture Proxy Security Group. + +Look for failing tasks by navigating to **Traffic Capture Proxy ECS**. Change **Filter desired status** to **Any desired status** in order to see all tasks and navigate to the logs for stopped tasks. + +### Snapshot and S3 bucket issues + +When using the CDK deployment for Migration Assistant, you might encounter the following errors during snapshot creation and deletion. + +#### Bucket permissions + +To make sure that you can delete snapshots as well as create them during the CDK deployment process, confirm that the `OSMigrations-dev--CustomS3AutoDeleteObjects` stack has S3 object deletion rights. Then, verify that `OSMigrations-dev--default-SnapshotRole` has the following S3 permissions: + + - List bucket contents + - Read/Write/Delete objects + +#### Snapshot conflicts + +To prevent snapshot conflicts, use the `console snapshot delete` command from the migration console. If you delete snapshots or snapshot repositories in a location other than the migration console, you might encounter "already exists" errors. + +## Resetting before migration + +After all verifications are complete, reset all resources before using Migration Assistant for an actual migration. + +The following steps outline how to reset resources with Migration Assistant before executing the actual migration. At this point all verifications are expected to have been completed. These steps can be performed after [Accessing the Migration Console]({{site.url}}{{site.baseurl}}/migration-assistant/migration-console/accessing-the-migration-console/). + + +{% include copy.html %} diff --git a/_migration-assistant/migration-phases/verify-live-capture-components.md b/_migration-assistant/migration-phases/verify-live-capture-components.md new file mode 100644 index 00000000000..90666b80bc8 --- /dev/null +++ b/_migration-assistant/migration-phases/verify-live-capture-components.md @@ -0,0 +1,157 @@ +--- +layout: default +title: Live Capture Setup Verification +nav_order: 9 +redirect_from: + - /migration-assistant/migration-phases/verifying-migration-tools/ +--- + +# Verifying migration tools + +Before using the Migration Assistant, take the following steps to verify that your cluster is ready for migration. + +### Traffic Replayer + +To stop running Traffic Replayer, use the following command: + +```bash +console replay stop +``` +{% include copy.html %} + +### Kafka + +To clear all captured traffic from the Kafka topic, you can run the following command. + +This command will result in the loss of any traffic data captured by the capture proxy up to this point and thus should be used with caution. +{: .warning} + +```bash +console kafka delete-topic +``` +{% include copy.html %} + +### Target cluster + +To clear non-system indexes from the target cluster that may have been created as a result of testing, you can run the following command: + +This command will result in the loss of all data in the target cluster and should be used with caution. +{: .warning} + +```bash +console clusters clear-indices --cluster target +``` + +## Switching over client traffic + +The Migration Assistant Application Load Balancer is deployed with a listener that shifts traffic between the source and target clusters through proxy services. The Application Load Balancer should start in **Source Passthrough** mode. + +### Verifying that the traffic switchover is complete + +Use the following steps to verify that the traffic switchover is complete: + +1. In the AWS Management Console, navigate to **EC2 > Load Balancers**. +2. Select the **MigrationAssistant ALB**. +3. Examine the listener on port `9200` and verify that 100% of the traffic is directed to the **Source Proxy**. +4. Navigate to the **Migration ECS Cluster** in the AWS Management Console. +5. Select the **Target Proxy Service**. +6. Verify that the desired count for the service is running: + * If the desired count is not met, update the service to increase it to at least 1 and wait for the service to start. +7. On the **Health and Metrics** tab under **Load balancer health**, verify that all targets are reporting as healthy: + * This confirms that the Application Load Balancer can connect to the target cluster through the target proxy. +8. (Reset) Update the desired count for the **Target Proxy Service** back to its original value in Amazon ECS. + +### Fixing unidentified traffic patterns + +When switching over traffic to the target cluster, you might encounter unidentified traffic patterns. To help identify the cause of these patterns, use the following steps: +* Verify that the target cluster allows traffic ingress from the **Target Proxy Security Group**. +* Navigate to **Target Proxy ECS Tasks** to investigate any failing tasks. +Set the **Filter desired status** to **Any desired status** to view all tasks, then navigate to the logs for any stopped tasks. + + +## Verifying replication + +Use the following steps to verify that replication is working once the traffic capture proxy is deployed: + + +1. Navigate to the **Migration ECS Cluster** in the AWS Management Console. +2. Navigate to **Capture Proxy Service**. +3. Verify that the capture proxy is running with the desired proxy count. If it is not, update the service to increase it to at least 1 and wait for startup. +4. Under **Health and Metrics** > **Load balancer health**, verify that all targets are healthy. This means that the Application Load Balancer is able to connect to the source cluster through the capture proxy. +5. Navigate to the **Migration Console Terminal**. +6. Run `console kafka describe-topic-records`. Wait 30 seconds for another Application Load Balancer health check. +7. Run `console kafka describe-topic-records` again and verify that the number of RECORDS increased between runs. +8. Run `console replay start` to start Traffic Replayer. +9. Run `tail -f /shared-logs-output/traffic-replayer-default/*/tuples/tuples.log | jq '.targetResponses[]."Status-Code"'` to confirm that the Kafka requests were sent to the target and that it responded as expected. If the responses don't appear: + * Check that the migration console can access the target cluster by running `./catIndices.sh`, which should show the indexes in the source and target. + * Confirm that messages are still being recorded to Kafka. + * Check for errors in the Traffic Replayer logs (`/migration/STAGE/default/traffic-replayer-default`) using CloudWatch. +10. (Reset) Update the desired count for the **Capture Proxy Service** back to its original value in Amazon ECS. + +### Troubleshooting + +Use this guidance to troubleshoot any of the following replication verification issues. + +### Health check responses with 401/403 status code + +If the source cluster is configured to require authentication, the capture proxy will not be able to verify replication beyond receiving a 401/403 status code for Application Load Balancer health checks. For more information, see [Failure Modes](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/README.md#failure-modes). + +### Traffic does not reach the source cluster + +Verify that the source cluster allows traffic ingress from the Capture Proxy Security Group. + +Look for failing tasks by navigating to **Traffic Capture Proxy ECS**. Change **Filter desired status** to **Any desired status** in order to see all tasks and navigate to the logs for stopped tasks. + +### Snapshot and S3 bucket issues + +When using the CDK deployment for Migration Assistant, you might encounter the following errors during snapshot creation and deletion. + +#### Bucket permissions + +To make sure that you can delete snapshots as well as create them during the CDK deployment process, confirm that the `OSMigrations-dev--CustomS3AutoDeleteObjects` stack has S3 object deletion rights. Then, verify that `OSMigrations-dev--default-SnapshotRole` has the following S3 permissions: + + - List bucket contents + - Read/Write/Delete objects + +#### Snapshot conflicts + +To prevent snapshot conflicts, use the `console snapshot delete` command from the migration console. If you delete snapshots or snapshot repositories in a location other than the migration console, you might encounter "already exists" errors. + +## Resetting before migration + +After all verifications are complete, reset all resources before using Migration Assistant for an actual migration. + +The following steps outline how to reset resources with Migration Assistant before executing the actual migration. At this point all verifications are expected to have been completed. These steps can be performed after [Accessing the Migration Console]({{site.url}}{{site.baseurl}}/migration-assistant/migration-console/accessing-the-migration-console/). + +### Traffic Replayer + +To stop running Traffic Replayer, use the following command: + +```bash +console replay stop +``` +{% include copy.html %} + +### Kafka + +To clear all captured traffic from the Kafka topic, you can run the following command. + +This command will result in the loss of any traffic data captured by the capture proxy up to this point and thus should be used with caution. +{: .warning} + +```bash +console kafka delete-topic +``` +{% include copy.html %} + +### Target cluster + +To clear non-system indexes from the target cluster that may have been created as a result of testing, you can run the following command: + +This command will result in the loss of all data in the target cluster and should be used with caution. +{: .warning} + +```bash +console clusters clear-indices --cluster target +``` +{% include copy.html %} diff --git a/_migration-assistant/overview/architecture.md b/_migration-assistant/overview/architecture.md new file mode 100644 index 00000000000..48b3c8fbcfc --- /dev/null +++ b/_migration-assistant/overview/architecture.md @@ -0,0 +1,25 @@ +--- +layout: default +title: Architecture +nav_order: 15 +parent: Overview +permalink: /migration-assistant/overview/architecture/ +--- + +# Architecture + +The Migration Assistant architecture is based on the use of an AWS Cloud infrastructure, but most tools are designed to be cloud independent. A local containerized version of this solution is also available. + +The design deployed on AWS uses the following architecture. + +![Migration architecture overview]({{site.url}}{{site.baseurl}}/images/migrations/migrations-architecture-overview.png) + +Each node in the diagram correlates to the following steps in the migration process: + +1. Client traffic is directed to the existing cluster. +2. An Application Load Balancer with capture proxies relays traffic to a source while replicating data to Amazon Managed Streaming for Apache Kafka (Amazon MSK). +3. Using the migration console, you can initiate metadata migration to establish indexes, templates, component templates, and aliases on the target cluster. +4. With continuous traffic capture in place, you can use a `reindex-from-snapshot` process to capture data from your current index. +5. Once `Reindex-from-Snapshot` is complete, captured traffic is replayed from Amazon MSK to the target cluster by [Traffic Replayer](https://docs.opensearch.org/docs/latest/migration-assistant/migration-phases/live-traffic-migration/using-traffic-replayer/). +6. Performance and behavior of traffic sent to the source and target clusters are compared by reviewing logs and metrics. +7. After confirming that the target cluster's functionality meets expectations, clients are redirected to the new target. \ No newline at end of file diff --git a/_migration-assistant/overview/index.md b/_migration-assistant/overview/index.md new file mode 100644 index 00000000000..7962fe5e85d --- /dev/null +++ b/_migration-assistant/overview/index.md @@ -0,0 +1,25 @@ +--- +layout: default +title: Overview +nav_order: 2 +has_children: true +has_toc: false +permalink: /migration-assistant/overview/ +items: + - heading: "Key components" + description: "Get familiar with the key components of Migration Assistant." + link: "/migration-assistant/overview/key-components/" + - heading: "Architecture" + description: "Understand how Migration Assistant integrates into your infrastructure." + link: "/migration-assistant/overview/architecture/" + - heading: "Is Migration Assistant right for you?" + description: "Evaluate whether Migration Assistant is right for your use case." + link: "/migration-assistant/overview/is-migration-assistant-right-for-you/" +--- + +# Migration Assistant overview + +Use this section to get familiar with the key concepts and structure of Migration Assistant before diving into setup or execution. These pages provide the architecture, key components, and guidance to help you determine whether Migration Assistant is right for your use case. + +{% include list.html list_items=page.items%} + diff --git a/_migration-assistant/overview/is-migration-assistant-right-for-you.md b/_migration-assistant/overview/is-migration-assistant-right-for-you.md new file mode 100644 index 00000000000..1e7fc287974 --- /dev/null +++ b/_migration-assistant/overview/is-migration-assistant-right-for-you.md @@ -0,0 +1,154 @@ +--- +layout: default +title: Is Migration Assistant right for you? +nav_order: 10 +parent: Overview +permalink: /migration-assistant/overview/is-migration-assistant-right-for-you/ +redirect_from: + - /migration-assistant/is-migration-assistant-right-for-you/ +--- + +# Is Migration Assistant right for you? + +Deciding whether to use Migration Assistant depends on your specific upgrade path, infrastructure complexity, and operational goals. This page will help you evaluate whether Migration Assistant is right for your use case—or whether another tool might be a better fit. + +Migration Assistant was built to fill important gaps in common migration strategies. For example, if you're upgrading across multiple major versions—such as from Elasticsearch 6.8 to OpenSearch 2.19—Migration Assistant lets you do this in a single step. Other methods, like rolling upgrades or snapshot restores, require you to upgrade through each major version, often reindexing your data at every step. + +Migration Assistant also supports live traffic replication, allowing for zero-downtime migrations. This makes it a strong choice for production environments, where minimizing service disruption is critical. + +If your migration is limited to a static cluster configuration (like index templates and aliases), or if you're not concerned about downtime, simpler tools may be sufficient. But for complex migrations involving real-time traffic or major version jumps, Migration Assistant offers robust, flexible capabilities. + +## Supported migration paths + +The following matrix shows which source versions can be directly migrated to which OpenSearch target versions: + + +{% comment %}First, collect all unique target versions{% endcomment %} +{% assign all_targets = "" | split: "" %} +{% for path in site.data.migration-assistant.valid_migrations.migration_paths %} + {% for target in path.targets %} + {% assign all_targets = all_targets | push: target %} + {% endfor %} +{% endfor %} +{% assign unique_targets = all_targets | uniq | sort %} + +
    Source Version{{ target }}
    + + + + {% for target in unique_targets %} + + {% endfor %} + + + + {% for path in site.data.migration-assistant.valid_migrations.migration_paths %} + + + {% for target_version in unique_targets %} + + {% endfor %} + + {% endfor %} + +
    {{ target }}
    {{ path.source }} + {% if path.targets contains target_version %}✓{% endif %} +
    + +## Supported platforms + +**Source and target platforms** + +- Self-managed (on premises or hosted by a cloud provider) +- Amazon OpenSearch Service + + +**AWS Regions** + +Migration Assistant is supported in the following AWS Regions: + +- US East (N. Virginia, Ohio) +- US West (Oregon, N. California) +- Europe (Frankfurt, Ireland, London) +- Asia Pacific (Tokyo, Singapore, Sydney) +- AWS GovCloud (US-East, US-West)[^1] + +[^1]: In AWS GovCloud (US), `reindex-from-snapshot` (RFS) is limited to shard sizes of 80 GiB or smaller. + + + +## Supported components + +Before starting a migration, consider the scope of the components involved. The following table outlines components that should potentially be migrated, indicates whether they are supported by Migration Assistant, and provides recommendations. + +| Component | Supported | Recommendations | +| :--- |:--- | :--- | +| **Documents** | Yes | Migrate existing data with RFS and live traffic with capture and replay. | +| **Index settings** | Yes | Migrate with the `Metadata-Migration-Tool`. | +| **Index mappings** | Yes | Migrate with the `Metadata-Migration-Tool`. | +| **Index templates** | Yes | Migrate with the `Metadata-Migration-Tool`. | +| **Component templates** | Yes | Migrate with the `Metadata-Migration-Tool`. | +| **Aliases** | Yes | Migrate with the `Metadata-Migration-Tool`. | +| **Index State Management (ISM) policies** | Expected in 2025 | Manually migrate using an API. For more information about ISM support, see [issue #944](https://github.com/opensearch-project/opensearch-migrations/issues/944). | +| **Elasticsearch Kibana[^2] dashboards** | Expected in 2025 | This tool is only needed when migrating from Elasticsearch Kibana dashboards to OpenSearch Dashboards. Start by exporting JSON files from Kibana and importing them into OpenSearch Dashboards. For Elasticsearch versions 7.10.2 to 7.17, use the [`dashboardsSanitizer`](https://github.com/opensearch-project/opensearch-migrations/tree/main/dashboardsSanitizer) tool before importing X-Pack visualizations like Canvas and Lens in Kibana dashboards, as they may require recreation for compatibility with OpenSearch.| +| **Security constructs** | No | Configure roles and permissions based on cloud provider recommendations. For example, if using AWS, leverage AWS Identity and Access Management (IAM) for enhanced security management. | +| **Plugins** | No | Check plugin compatibility; some Elasticsearch plugins may not have direct OpenSearch equivalents. | + +[^2]: Support for Kibana 5.0 through 7.10.2 migration paths to OpenSearch Dashboards will be added in a future version. Kibana 8 and later are not supported. For more information, see [issue #944](https://github.com/opensearch-project/opensearch-migrations/issues/944). + +## Choosing your migration approach + +Use the following checklist to determine which Migration Assistant components best fit your use case. + +### Metadata migration + +Use [metadata migration]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrating-metadata/) if: + +- You need to migrate while mitigating breaking changes between the source and target clusters, such as differences in mappings, settings, aliases, or component templates. +- You want a relatively consistent configuration between the source and target clusters. + +### Backfill migration + +Use [backfill migration]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/backfill/) if: + +- You need to move historical data without disrupting live traffic. +- You want to backfill indexes from a specific point in time without impacting the source cluster. +- You want to verify historical data in the target cluster before switching over. +- You want to backfill using an existing or incremental snapshot. +- You need the fastest backfill option that includes reindexing. +- You want the ability to pause and resume migration. + +### RFS + +Use [RFS]({{site.url}}{{site.baseurl}}/migration-assistant/deploying-migration-assistant/getting-started-data-migration/) if: + +- You already use OpenSearch snapshots for backups. +- You need to migrate documents at scale in parallel, such as with Amazon Elastic Container Service (Amazon ECS). +- You require a data migration path as part of a zero-downtime migration. +- Your AWS Region supports RFS and your shard sizes are within supported limits. + +### Combination of all three + +Use a combination of all three migration types if: + +- You're performing a complex, multi-version migration. +- You require zero downtime and full validation of the target environment. +- You want end-to-end tooling for metadata, data movement, and cluster behavior comparison. +- You're cloning an existing cluster and changing the source's configuration. +- You're setting up disaster recovery. + +## Checklist + +Use this checklist to decide whether Migration Assistant is right for you: + +- Are you migrating across one or more major versions? + +- Do you need to maintain service availability with zero downtime? + +- Do you need to validate a new OpenSearch cluster before switching over? + +- Is your environment self-managed or running on Amazon OpenSearch Service? + +- Are you looking for tooling that can automate metadata migration and performance comparison? + +If you answered "yes" to most of these questions, Migration Assistant is likely the right solution for your migration. diff --git a/_migration-assistant/overview/key-components.md b/_migration-assistant/overview/key-components.md new file mode 100644 index 00000000000..cfa69e70311 --- /dev/null +++ b/_migration-assistant/overview/key-components.md @@ -0,0 +1,39 @@ +--- +layout: default +title: Key components +nav_order: 10 +parent: Overview +permalink: /migration-assistant/overview/key-components/ +--- + +# Key components + +The following are the key components of Migration Assistant. + +## Elasticsearch/OpenSearch source + +In this solution, your source cluster operates on either Elasticsearch or OpenSearch and is hosted on Amazon Elastic Compute Cloud (Amazon EC2) instances or in a similar computing environment. A proxy is set up to interact with this source cluster, either positioned in front of or directly on the coordinating nodes of the cluster. + +## Migration console + +The migration console provides a migration-specific CLI and offers a variety of tools for streamlining the migration process. Everything necessary for completing a migration, other than cleaning up the migration resources, can be performed through this console. + +## Traffic capture proxy + +This component is designed for HTTP RESTful traffic. It forwards traffic to the source cluster and also splits and channels this traffic to a stream processing service for later playback. + +## Traffic Replayer + +Acting as a traffic simulation tool, [Traffic Replayer](https://docs.opensearch.org/docs/latest/migration-assistant/migration-phases/live-traffic-migration/using-traffic-replayer/) replays recorded request traffic to a target cluster, mirroring source traffic patterns. It links original requests and their responses to those directed at the target cluster, facilitating comparative analysis. + +## Metadata migration tool + +The metadata migration tool integrated into the Migration CLI can be used independently to migrate cluster metadata, including index mappings, index configuration settings, templates, component templates, and aliases. + +## Reindex-from-Snapshot + +`Reindex-from-Snapshot` (RFS) reindexes data from an existing snapshot. Amazon Elastic Container Service (Amazon ECS) workers coordinate the migration of documents from an existing snapshot, reindexing the documents in parallel to a target cluster. + +## Target cluster + +The target cluster is the destination cluster for migration or comparison in an A/B test. \ No newline at end of file diff --git a/_migration-assistant/planning-your-migration/handling-field-type-breaking-changes.md b/_migration-assistant/planning-your-migration/handling-field-type-breaking-changes.md new file mode 100644 index 00000000000..b6a653f96ae --- /dev/null +++ b/_migration-assistant/planning-your-migration/handling-field-type-breaking-changes.md @@ -0,0 +1,158 @@ +--- +layout: default +title: Handling breaking changes in field types +nav_order: 60 +parent: Planning your migration +grand_parent: Migration phases +--- + +# Handling breaking changes in field types + +This guide explains how to use Migration Assistant to transform field types that are deprecated or incompatible during a migration to OpenSearch. + +Field types define how data is stored and queried in an index. Each field in a document is mapped to a data type, which determines how it is indexed and what operations can be performed on it. + +For example, the following index mapping for a library's book collection defines three fields, each with a different type: + +```json +GET /library-books/_mappings +{ + "library-books": { + "mappings": { + "properties": { + "title": { "type": "text" }, + "publishedDate": { "type": "date" }, + "pageCount": { "type": "integer" } + } + } + } +} +``` + +For more information, see [Mappings and field types]({{site.url}}{{site.baseurl}}/field-types/). + +## Configure item transformations + +You can customize how field types are transformed during metadata and data migrations by supplying a transformation configuration file using the following steps: + +1. Open the Migration Assistant console. +2. Create a JavaScript file to define your transformation logic using the following command: + + ```bash + vim /shared-logs-output/field-type-converter.js + ``` + {% include copy.html %} + +3. Write any JavaScript rules that perform the desired field type conversions. For an example of how the rules can be implemented, see the [example `field-type-converter.js` implementation](#example-field-type-converterjs-implementation). +4. Create a transformation descriptor file using the following command: + + ```bash + vim /shared-logs-output/transformation.json + ``` + {% include copy.html %} + +5. Add a reference to your JavaScript file in `transformation.json`. +6. Run the metadata migration and supply the transformation configuration using a command similar to the following: + + ```bash + console metadata migrate \ + --transformer-config-file /shared-logs-output/transformation.json + ``` + {% include copy.html %} + +### Example `field-type-converter.js` implementation + +The following script demonstrates how to perform common field type conversions, including: + +* Replacing the deprecated `string` type with `text`. +* Converting `flattened` to `flat_object` and removing the `index` property if present. + +```javascript +function main(context) { + const rules = [ + { + when: { type: "string" }, + set: { type: "text" } + }, + { + when: { type: "flattened" }, + set: { type: "flat_object" }, + remove: ["index"] + } + ]; + + function applyRules(node, rules) { + if (Array.isArray(node)) { + node.forEach((child) => applyRules(child, rules)); + } else if (node instanceof Map) { + for (const { when, set, remove = [] } of rules) { + const matches = Object.entries(when).every(([k, v]) => node.get(k) === v); + if (matches) { + Object.entries(set).every(([k, v]) => node.set(k, v)); + remove.forEach((key) => node.delete(key)); + } + } + for (const child of node.values()) { + applyRules(child, rules); + } + } else if (node && typeof node === "object") { + for (const { when, set, remove = [] } of rules) { + const matches = Object.entries(when).every(([k, v]) => node[k] === v); + if (matches) { + Object.assign(node, set); + remove.forEach((key) => delete node[key]); + } + } + Object.values(node).forEach((child) => applyRules(child, rules)); + } + } + + return (doc) => { + if (doc && doc.type && doc.name && doc.body) { + applyRules(doc, rules); + } + return doc; + }; +} +(() => main)(); +``` +{% include copy.html %} + +The script contains the following elements: + +1. The `rules` array defines transformation logic: + + * `when`: Key-value conditions to match on a node + * `set`: Key-value pairs to apply when the `when` clause matches + * `remove` (optional): Keys to delete from the node when matched + +2. The `applyRules` function recursively traverses the input: + + * Arrays are recursively processed element by element. + * `Map` objects are matched and mutated using the defined rules. + * Plain objects are checked for matches and transformed accordingly. + +3. The `main` function returns a transformation function that: + + * Applies the rules to each document. + * Returns the modified document for migration or replay. + +### Example `transformation.json` + +The following JSON file references your transformation script and initializes the JavaScript engine with your custom rules: + +```json +[ + { + "JsonJSTransformerProvider": { + "initializationScriptFile": "/shared-logs-output/field-type-converter.js", + "bindingsObject": "{}" + } + } +] +``` +{% include copy.html %} + +## Summary + +By using a transformation configuration, you can rewrite deprecated or incompatible field types during metadata migration or data replay. This ensures that your target OpenSearch cluster only receives compatible mappings—even if the source cluster includes outdated types like `string` or features like `flattened` that need conversion. diff --git a/_migration-assistant/planning-your-migration/handling-type-mapping-deprecation.md b/_migration-assistant/planning-your-migration/handling-type-mapping-deprecation.md new file mode 100644 index 00000000000..edd81612dac --- /dev/null +++ b/_migration-assistant/planning-your-migration/handling-type-mapping-deprecation.md @@ -0,0 +1,318 @@ +--- +layout: default +title: Managing type mapping deprecation +nav_order: 60 +parent: Planning your migration +grand_parent: Migration phases +--- + +# Managing type mapping deprecation + +This guide provides solutions for managing the deprecation of the type mapping functionality when migrating from Elasticsearch 6.x or earlier to OpenSearch. + +In versions of Elasticsearch prior to 6.x, an index could contain multiple types, each with its own mapping. These types allowed you to store and query different kinds of documents—such as books and movies—in a single index. For example, both `book` and `movie` types could have a shared field like `title`, while each had additional fields specific to that type. + +Newer versions of Elasticsearch and OpenSearch no longer support multiple mapping types. Each index now supports only a single mapping type. During migration, you must define how to transform or restructure data that used multiple types. The following example shows multiple mapping types: + + +```JSON +GET /library/_mappings +{ + "library": { + "mappings": { + "book": { + "properties": { + "title": { "type": "text" }, + "pageCount": { "type": "integer" } + } + }, + "movie": { + "properties": { + "title": { "type": "text" }, + "runTime": { "type": "integer" } + } + } + } + } +} +``` + +For more information, see the [official Elasticsearch documentation on the removal of mapping types](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/removal-of-types.html). + +## Using the type mapping transformer + +To address type mapping deprecation, use the `TypeMappingsSanitizationTransformer`. This transformer can modify data, including metadata, documents, and requests, so that the previously mapped data can be used in OpenSearch. To use the mapping transformer: + +1. Navigate to the bootstrap box and open the `cdk.context.json` file with Vim. +2. Add or update the key `reindexFromSnapshotExtraArgs` to include `--doc-transformer-config-file /shared-logs-output/transformation.json`. +3. Add or update the key `trafficReplayerExtraArgs` to include `--transformer-config-file /shared-logs-output/transformation.json`. +4. Deploy Migration Assistant. +5. Navigate to the Migration Assistant console. +6. Create a file named `/shared-logs-output/transformation.json`. +7. Add your transformation configuration to the file. For configuration options, see [Configuration options](#configuration-options). +8. When running the metadata migration, run the configuration with the transformer using the command `console metadata migrate --transformer-config-file /shared-logs-output/transformation.json`. + +Whenever the transformation configuration is updated, the backfill and replayer tools need to be stopped and restarted in order to apply the changes. Any previously migrated data and metadata may need to be cleared in order to avoid an inconsistent state. + +### Configuration options + +The `TypeMappingsSanitizationTransformer` supports several strategies for managing type mappings: + +1. **Route different types to separate indexes**: Split different types into their own indexes. +2. **Merge all types into one index**: Combine multiple types into a single index. +3. **Drop specific types**: Selectively migrate only specific types. +4. **Keep the original structure**: Maintain the same index name while conforming to the new type standards. + +### Type mapping transformer configuration schema + +The type mapping transformer uses the following configuration options. + +| **Field** | **Type** | **Required** | **Description** | +| :--- | :--- | :--- | :--- | +| `staticMappings` | `object` | No | A map of `{ indexName: { typeName: targetIndex } }` used to **statically** route specific types.

    For any **index** listed on this page, types **not** included in its object are **dropped** (no data or requests are migrated for those omitted types). | +| `regexMappings` | `array` | No | A list of **regex-based** rules for **dynamic** routing of source index/type names to a target index.

    Each element in this array is itself an object with `sourceIndexPattern`, `sourceTypePattern`, and `targetIndexPattern` fields.

    For information about the **default value**, see [Defaults](#Defaults). | +| `sourceProperties` | `object` | Yes | Additional **metadata** about the source (for example, its Elasticsearch/OpenSearch version). Must include at least `"version"` with `"major"` and `"minor"` fields. | + +The following example JSON configuration provides a transformation schema: + +
    +Example JSON Configuration + +```JSON +{ + "TypeMappingSanitizationTransformerProvider": { + "staticMappings": { + "{index-name-1}": { + "{type-name-1}": "{target-index-name-1}", + "{type-name-2}": "{target-index-name-2}" + } + }, + "regexMappings": [ + { + "sourceIndexPattern": "{source-index-pattern}", + "sourceTypePattern": "{source-type-pattern}", + "targetIndexPattern": "{target-index-pattern}" + } + ], + "sourceProperties": { + "version": { + "major": "NUMBER", + "minor": "NUMBER" + } + } + } +} +``` +{% include copy.html %} + +
    + +## Example configurations + +The following example configurations show you how to use the transformer for different mapping type scenarios. + +### Route different types to separate indexes + +If you have an index `activity` with types `user` and `post` that you want to split into separate indexes, use the following configuration: + +```json +[ + { + "TypeMappingSanitizationTransformerProvider": { + "staticMappings": { + "activity": { + "user": "new_users", + "post": "new_posts" + } + }, + "sourceProperties": { + "version": { + "major": 6, + "minor": 8 + } + } + } + } +] +{% include copy.html %} +``` + +This transformer will perform the following: + +- Route documents with type `user` to the `new_users` index. +- Route documents with type `post` to the `new_posts` index. + +### Merge all types into one index + +To merge all types into one index, use the following configuration: + +```json +[ + { + "TypeMappingSanitizationTransformerProvider": { + "staticMappings": { + "activity": { + "user": "activity", + "post": "activity" + } + }, + "sourceProperties": { + "version": { + "major": 6, + "minor": 8 + } + } + } + } +] +``` +{% include copy.html %} + +### Drop specific types + +To migrate only the `user` type within the `activity` index and drop all documents/requests with types not directly specified, use the following configuration: + +```json +[ + { + "TypeMappingSanitizationTransformerProvider": { + "staticMappings": { + "activity": { + "user": "users_only" + } + }, + "sourceProperties": { + "version": { + "major": 6, + "minor": 8 + } + } + } + } +] +``` +{% include copy.html %} + +This configuration only migrates documents of type `user` and ignores other document types in the `activity` index. + +### Keep the original structure + +To migrate only specific types and keep the original structure, use the following configuration: + +```JSON +[ + { + "TypeMappingSanitizationTransformerProvider": { + "regexMappings": [ + { + "sourceIndexPattern": "(.*)", + "sourceTypePattern": ".*", + "targetIndexPattern": "$1" + } + ], + "sourceProperties": { + "version": { + "major": 6, + "minor": 8 + } + } + } + } +] +``` +{% include copy.html %} + +This is equivalent to the strategy of merging all types into one index but also uses a pattern-based routing strategy. + +### Combining multiple strategies + +You can combine both static and regex-based mappings to manage different indexes or patterns in a single migration. For example, you might have one index that must use `staticMappings` and another that uses `regexMappings` to route all types by pattern. + +For each document, request, or metadata item (processed individually for bulk requests), the following steps are performed: + +1. The index is checked to determine whether it matches an entry in the static mappings. + - If matched, the type is checked against the index component of the static mappings entry. + - If the type matches, the mapping is applied, and the resulting index includes the value of the type key. + - If the type doesn't match, the request/document/metadata is dropped and not migrated. +2. If the index is not matched in the static mappings, the index-type combination is checked against each item in the regex mappings list, in order from first to last. If a match is found, the mapping is applied, the resulting index includes the value of the type key, and no further regex matching is performed. +3. Any request, document, or metadata that doesn't match the preceding cases is dropped, and the documents they contain are not migrated. + +The following example demonstrates how to combine static and regex-based mappings for different indexes: + +```json +[ + { + "TypeMappingSanitizationTransformerProvider": { + "staticMappings": { + "activity": { + "user": "users_activity", + "post": "posts_activity" + }, + "logs": { + "error": "logs_error", + "info": "logs_info" + } + }, + "regexMappings": [ + { + "sourceIndexPattern": "orders.*", + "sourceTypePattern": ".*", + "targetIndexPattern": "all_orders" + } + ], + "sourceProperties": { + "version": { + "major": 6, + "minor": 8 + } + } + } + } +] +``` +{% include copy.html %} + +### Defaults + +When the `regexMappings` key is missing from the transformation configuration, `regexMappings` will default to the following: + +```JSON +{ + "regexMappings": [ + { + "sourceIndexPattern": "(.+)", + "sourceTypePattern": "_doc", + "targetIndexPattern": "$1" + }, + { + "sourceIndexPattern": "(.+)", + "sourceTypePattern": "(.+)", + "targetIndexPattern": "$1_$2" + } + ] +} +``` +{% include copy.html %} + +This has the effect of retaining the index name for indexes created in Elasticsearch 6.x or later while combining the type and index name for indexes created in Elasticsearch 5.x. If you want to retain the index name for indexes created in Elasticsearch 5.x, use the `staticMappings` option or override the type mappings using the `regexMappings` option. + +## Limitations + +When using the transformer, remember the following limitations. +When using the transformer, remember the following limitations. +### Traffic Replayer + +For the Traffic Replayer, **only a subset** of requests that include types is supported. These requests are listed in the following table. + +| **Operation** | **HTTP method(s)** | **Endpoint** | **Description** | +| :--- | :--- | :--- | :--- | +| **Index (by ID)** | PUT/POST | `/{index}/{type}/{id}` | Create or update a single document with an explicit ID. | +| **Index (auto ID)** | PUT/POST | `/{index}/{type}/` | Create a single document for which the ID is automatically generated. | +| **Get Document** | GET | `/{index}/{type}/{id}` | Retrieve a document by ID. | +| **Bulk Index/Update/Delete** | PUT/POST | `/_bulk` | Perform multiple create/update/delete operations in a single request. | +| **Bulk Index/Update/Delete** | PUT/POST | `/{index}/_bulk` | Perform multiple create/update/delete operations in a single request with default index assignment. | +| **Bulk Index/Update/Delete** | PUT/POST | `/{index}/{type}/_bulk` | Perform multiple create/update/delete operations in a single request with default index and type assignment. | +| **Create/Update Index** | PUT/POST | `/{index}` | Create or update an index.

    **Split** behavior is not supported in the Traffic Replayer. See [this GitHub issue](https://github.com/opensearch-project/opensearch-migrations/issues/1305) to provide feedback or to vote on this feature. | + +### Reindex-From_Shapshot +For `Reindex-From-Snapshot,` indexes created in Elasticsearch 6.x or later will use `_doc` as the type for all documents, even if a different type was specified in Elasticsearch 6. diff --git a/_migration-assistant/planning-your-migration/index.md b/_migration-assistant/planning-your-migration/index.md new file mode 100644 index 00000000000..4f0d264e93e --- /dev/null +++ b/_migration-assistant/planning-your-migration/index.md @@ -0,0 +1,15 @@ +--- +layout: default +title: Planning your migration +nav_order: 59 +parent: Migration phases +has_toc: false +has_children: true +--- + +# Planning your migration + +This section describes how to plan for your migration to OpenSearch by: + +- [Assessing your current cluster for migration]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/planning-your-migration/assessing-your-cluster-for-migration/). +- [Verifying that you have the tools for migration]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/planning-your-migration/verifying-migration-tools/). \ No newline at end of file diff --git a/_migration-upgrade/index.md b/_migration-upgrade/index.md index 5d17cba894c..7aba907619d 100644 --- a/_migration-upgrade/index.md +++ b/_migration-upgrade/index.md @@ -5,9 +5,8 @@ nav_order: 1 has_children: false nav_exclude: true has_toc: false -permalink: /migration-assistant/ +permalink: /migrations-and-upgrades/ redirect_from: - - /migration-assistant/index/ - /upgrade-to/index/ - /upgrade-to/ - /upgrade-to/upgrade-to/ @@ -15,15 +14,9 @@ tutorial_cards: - heading: "Overview" description: "Get familiar with the key components of Migration Assistant and evaluate your use case." link: "/migration-assistant/overview/" - - heading: "Deploying Migration Assistant" - description: "Follow step-by-step instructions to deploy Migration Assistant and prepare data for migration." - link: "/deploying-migration-assistant/" - heading: "Migration phases" description: "Execute your migration in phases—metadata, backfill, and traffic replay—for a controlled and validated transition." - link: "/migration-phases/" - - heading: "Migration console" - description: "Use CLI commands provided by the migration console to orchestrate and monitor your migration process." - link: "/migration-console/" + link: "/migration-assistant/migration-phases/" --- # Migration and upgrade options @@ -54,19 +47,10 @@ Migration Assistant offers the most automated and resilient path for OpenSearch - **Reversion support:** Provides a fallback option in case of errors or issues. - **Integrated automation:** Designed to fit into OpenSearch workflows for a guided upgrade experience. -### Getting started with Migration Assistant - -To get started with Migration Assistant, determine which of the following scenarios best suits your needs: - -- **Metadata migration**: Migrating cluster metadata, such as index settings, aliases, and templates. -- **Backfill migration**: Migrating existing or historical data from a source to a target cluster. -- **Live traffic migration**: Replicating live ongoing traffic from a source to a target cluster. -- **Comparative tooling**: Comparing the performance and behaviors of an existing cluster with a prospective new one. -It's crucial to note that migration strategies are not universally applicable. This guide provides a detailed methodology, based on certain assumptions detailed throughout, emphasizing the importance of robust engineering practices to ensure a successful migration. -{: .tip } +### Getting started with Migration Assistant -{% include cards.html cards=page.tutorial_cards %} +- Review the [Migration Assistant]({{site.url}}{{site.baseurl}}/migration-assistant/) ## Rolling upgrades @@ -82,6 +66,11 @@ Rolling upgrades allow you to upgrade one node at a time, keeping the cluster op - Multiple upgrade cycles are needed for large version jumps. - Manual reindexing may be required to enable newer features. +### Getting started with rolling upgrades + +- Perform a [rolling upgrade]({{site.url}}{{site.baseurl}}/install-and-configure/upgrade-opensearch/rolling-upgrade/) + + ## Snapshot and restore @@ -127,10 +116,3 @@ Migration Assistant offers the most automated and resilient path for OpenSearch Before choosing a method, make sure that your OpenSearch clients and plugins are compatible with the target version. For example, tools like Logstash OSS and Filebeat OSS may enforce version checks that impact upgrade paths. -## Next steps - -After you've determined which upgrade or migration path works best for you, take one of the following next steps: - -- Review the [Migration Assistant Overview]({{site.url}}{{site.baseurl}}/migration-assistant/overview/) -- Perform a [rolling upgrade]({{site.url}}{{site.baseurl}}/install-and-configure/upgrade-opensearch/rolling-upgrade/) - diff --git a/_migration-upgrade/migration-phases/backfill.md b/_migration-upgrade/migration-phases/backfill.md index 8a98412675b..f2d879c0f7a 100644 --- a/_migration-upgrade/migration-phases/backfill.md +++ b/_migration-upgrade/migration-phases/backfill.md @@ -1,7 +1,7 @@ --- layout: default -title: Backfill -nav_order: 90 +title: Migrate Data +nav_order: 6 parent: Migration phases permalink: /migration-assistant/migration-phases/backfill/ --- @@ -10,99 +10,6 @@ permalink: /migration-assistant/migration-phases/backfill/ After the [metadata]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrating-metadata/) for your cluster has been migrated, you can use capture proxy data replication and snapshots to backfill your data into the next cluster. -## Capture proxy data replication - -If you're interested in capturing live traffic during your migration, Migration Assistant includes an Application Load Balancer for routing traffic to the capture proxy and the target cluster. Upstream client traffic must be routed through the capture proxy in order to replay the requests later. Before using the capture proxy, remember the following: - -* The layer upstream from the Application Load Balancer is compatible with the certificate on the Application Load Balancer listener, whether it's for clients or a Network Load Balancer. The `albAcmCertArn` in the `cdk.context.json` may need to be provided to ensure that clients trust the Application Load Balancer certificate. -* If a Network Load Balancer is used directly upstream of the Application Load Balancer, it must use a TLS listener. -* Upstream resources and security groups must allow network access to the Migration Assistant Application Load Balancer. - -To set up the capture proxy, go to the AWS Management Console and navigate to **EC2 > Load Balancers > Migration Assistant Application Load Balancer**. Copy the Application Load Balancer URL. With the URL copied, you can use one of the following options. - - -### If you are using **Network Load Balancer → Application Load Balancer → Cluster** - -1. Ensure that ingress is provided directly to the Application Load Balancer for the capture proxy. -2. Create a target group for the Migration Assistant Application Load Balancer on port `9200`, and set the health check to `HTTPS`. -3. Associate this target group with your existing Network Load Balancer on a new listener for testing. -4. Verify that the health check is successful, and perform smoke testing with some clients through the new listener port. -5. Once you are ready to migrate all clients, detach the Migration Assistant Application Load Balancer target group from the testing Network Load Balancer listener and modify the existing Network Load Balancer listener to direct traffic to this target group. -6. Now client requests will be routed through the proxy (once they establish a new connection). Verify the application metrics. - -### If you are using **Network Load Balancer → Cluster** - -If you do not want to modify application logic, add an Application Load Balancer in front of your cluster and follow the **Network Load Balancer → Application Load Balancer → Cluster** steps. Otherwise: - -1. Create a target group for the Application Load Balancer on port `9200` and set the health check to `HTTPS`. -2. Associate this target group with your existing Network Load Balancer on a new listener. -3. Verify that the health check is successful, and perform smoke testing with some clients through the new listener port. -4. Once you are ready to migrate all clients, deploy a change so that clients hit the new listener. - - -### If you are **not using an Network Load Balancer** - -If you're only using backfill as your migration technique, make a client/DNS change to route clients to the Migration Assistant Application Load Balancer on port `9200`. - - -### Kafka connection - -After you have routed the client based on your use case, test adding records against HTTP requests using the following steps: - -In the migration console, run the following command: - -```bash -console kafka describe-topic-records -``` -{% include copy.html %} - -Note the records in the logging topic. - -After a short period, execute the same command again and compare the increased number of records against the expected HTTP requests. - -## Creating a snapshot - -Create a snapshot for your backfill using the following command: - -```bash -console snapshot create -``` -{% include copy.html %} - -To check the progress of your snapshot, use the following command: - -```bash -console snapshot status --deep-check -``` -{% include copy.html %} - -Depending on the size of the data in the source cluster and the bandwidth allocated for snapshots, the process can take some time. Adjust the maximum rate at which the source cluster's nodes create the snapshot using the `--max-snapshot-rate-mb-per-node` option. Increasing the snapshot rate will consume more node resources, which may affect the cluster's ability to handle normal traffic. - -## Backfilling documents to the source cluster - -From the snapshot you created of your source cluster, you can begin backfilling documents into the target cluster. Once you have started this process, a fleet of workers will spin up to read the snapshot and reindex documents into the target cluster. This fleet of workers can be scaled to increased the speed at which documents are reindexed into the target cluster. - -### Checking the starting state of the clusters - -You can check the indexes and document counts of the source and target clusters by running the `cat-indices` command. This can be used to monitor the difference between the source and target for any migration scenario. Check the indexes of both clusters using the following command: - -```shell -console clusters cat-indices -``` -{% include copy.html %} - -You should receive the following response: - -```shell -SOURCE CLUSTER -health status index uuid pri rep docs.count docs.deleted store.size pri.store.size -green open my-index WJPVdHNyQ1KMKol84Cy72Q 1 0 8 0 44.7kb 44.7kb - -TARGET CLUSTER -health status index uuid pri rep docs.count docs.deleted store.size pri.store.size -green open .opendistro_security N3uy88FGT9eAO7FTbLqqqA 1 0 10 0 78.3kb 78.3kb -``` - ### Starting the backfill Use the following command to start the backfill and deploy the workers: diff --git a/_migration-upgrade/migration-phases/index.md b/_migration-upgrade/migration-phases/index.md index a330b9e598e..d8f314d6b75 100644 --- a/_migration-upgrade/migration-phases/index.md +++ b/_migration-upgrade/migration-phases/index.md @@ -11,13 +11,77 @@ redirect_from: # Migration phases -This page details how to conduct a migration with Migration Assistant. It shows you how to [plan for your migration]({{site.url}}{{site.baseurl}}/planning-your-migration/) and encompasses a variety of migration scenarios, including: +This page outlines the core phases of migrating with Migration Assistant. Each scenario below consists of a sequence of common steps. Expand a scenario to explore the detailed flow. -- [**Metadata migration**]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrating-metadata/): Migrating cluster metadata, such as index settings, aliases, and templates. -- [**Backfill migration**]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/backfill/): Migrating existing or historical data from a source to a target cluster. -- [**Live traffic migration**]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/using-traffic-replayer/): Replicating live ongoing traffic from [a source cluster]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/switching-traffic-from-the-source-cluster/) to a target cluster. +--- + + + +
    +Scenario 1 – Backfill Only + +
      +
    1. Assessment
      2. +
    2. Deployment
      3. + +
    3. Create Snapshot
      4. +
    4. Migrate Metadata
      5. +
    5. Migrate Data
      6. +
    6. Teardown
      7. +
    +
    +
    +Scenario 2 – Live Capture Only +
      +
    1. Assessment
      2. +
    2. Deployment
      3. +
    3. Verify Backfill Components
      4. +
    4. Reroute Traffic from Source to Capture Proxy
      5. +
    5. Migrate Metadata
      6. +
    6. Verify Live Capture Components
      7. +
    7. Replay Captured Traffic
      8. +
    8. Reroute Traffic from Capture Proxy to Target
      9. +
    9. Teardown
      10. +
    +
    + +
    +Scenario 3 – Live Capture with Backfill + +
      +
    1. Assessment
      2. +
    2. Deployment
      3. +
    3. Reroute Traffic from Source to Capture Proxy
      4. +
    4. Create Snapshot
      5. +
    5. Migrate Metadata
      6. +
    6. Migrate Data
      7. +
    7. Replay Traffic
      8. +
    8. Reroute Traffic from Capture Proxy to Target
      9. +
    9. Teardown
      10. +
    + +
    + +--- +💡 *Need help getting started? Begin with [Planning your migration]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/planning-your-migration/index/).* From 49e03abc6e5aa35570838c6834a3fe1426fb3ded Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Wed, 2 Jul 2025 21:50:01 -0500 Subject: [PATCH 10/62] Remove 3.x source support Signed-off-by: Brian Presley --- _data/migration-assistant/valid_migrations.yml | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/_data/migration-assistant/valid_migrations.yml b/_data/migration-assistant/valid_migrations.yml index 772c3bc981a..60031815539 100644 --- a/_data/migration-assistant/valid_migrations.yml +++ b/_data/migration-assistant/valid_migrations.yml @@ -26,7 +26,4 @@ migration_paths: - source: "OpenSearch 2.x" targets: - "OpenSearch 2.x" - - "OpenSearch 3.x" - - source: "OpenSearch 3.x" - targets: - - "OpenSearch 3.x" + - "OpenSearch 3.x" \ No newline at end of file From a4a178caccf43621181adca3c2cd556dc76c4b4d Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Wed, 2 Jul 2025 23:14:52 -0500 Subject: [PATCH 11/62] Modifications, fixes, and clarity for migration phases. Signed-off-by: Brian Presley --- _migration-assistant/index.md | 23 ++++-- .../migration-phases/index.md | 8 +- _migration-assistant/overview/architecture.md | 9 +- _migration-assistant/overview/index.md | 25 ------ .../is-migration-assistant-right-for-you.md | 82 +++++++++---------- .../overview/key-components.md | 10 +-- _migration-upgrade/index.md | 11 ++- 7 files changed, 75 insertions(+), 93 deletions(-) delete mode 100644 _migration-assistant/overview/index.md diff --git a/_migration-assistant/index.md b/_migration-assistant/index.md index 1d7f8bb1db5..e4147b7dcff 100644 --- a/_migration-assistant/index.md +++ b/_migration-assistant/index.md @@ -8,16 +8,23 @@ has_toc: false permalink: /migration-assistant/ redirect_from: - /migration-assistant/index/ + - /migration-assistant/overview/ - /upgrade-to/index/ - /upgrade-to/ - /upgrade-to/upgrade-to/ -tutorial_cards: - - heading: "Overview" - description: "Get familiar with the key components of Migration Assistant and evaluate your use case." - link: "/migration-assistant/overview/" - - heading: "Migration phases" - description: "Execute your migration in phases—metadata, backfill, and traffic replay—for a controlled and validated transition." - link: "/migration-assistant/migration-phases/" +items: + - heading: "Is Migration Assistant right for you?" + description: "Evaluate whether Migration Assistant is right for your use case." + link: "/migration-assistant/overview/is-migration-assistant-right-for-you/" + - heading: "Key components" + description: "Get familiar with the key components of Migration Assistant." + link: "/migration-assistant/overview/key-components/" + - heading: "Architecture" + description: "Understand how Migration Assistant integrates into your infrastructure." + link: "/migration-assistant/overview/architecture/" + - heading: "Execute your migration in phases" + description: "A step-by-step guide for performing a migration." + link: "/migration-assistant/migration-phases" --- # Migration Assistant for OpenSearch @@ -31,5 +38,5 @@ Migration Assistant for OpenSearch aids you in successfully performing an end-to This user guide focuses on conducting a comprehensive migration involving both existing and live data with zero downtime and the option to back out of a migration. -{% include cards.html cards=page.tutorial_cards %} +{% include list.html list_items=page.items%} diff --git a/_migration-assistant/migration-phases/index.md b/_migration-assistant/migration-phases/index.md index 8a7dcc95ab7..6db56d90d2e 100644 --- a/_migration-assistant/migration-phases/index.md +++ b/_migration-assistant/migration-phases/index.md @@ -1,12 +1,12 @@ --- layout: default title: Migration phases -parent: Migration Assistant for OpenSearch +parent: Migration Assistant nav_order: 30 has_children: false -has_toc: true -permalink: /migration-assistant/overview/ -redirect-from: /migration-assistant/overview/index/ +has_toc: false +permalink: /migration-assistant/migration-phases/ +redirect-from: /migration-assistant/migration-phases/index/ --- # Migration phases diff --git a/_migration-assistant/overview/architecture.md b/_migration-assistant/overview/architecture.md index 48b3c8fbcfc..f768d3fa646 100644 --- a/_migration-assistant/overview/architecture.md +++ b/_migration-assistant/overview/architecture.md @@ -18,8 +18,7 @@ Each node in the diagram correlates to the following steps in the migration proc 1. Client traffic is directed to the existing cluster. 2. An Application Load Balancer with capture proxies relays traffic to a source while replicating data to Amazon Managed Streaming for Apache Kafka (Amazon MSK). -3. Using the migration console, you can initiate metadata migration to establish indexes, templates, component templates, and aliases on the target cluster. -4. With continuous traffic capture in place, you can use a `reindex-from-snapshot` process to capture data from your current index. -5. Once `Reindex-from-Snapshot` is complete, captured traffic is replayed from Amazon MSK to the target cluster by [Traffic Replayer](https://docs.opensearch.org/docs/latest/migration-assistant/migration-phases/live-traffic-migration/using-traffic-replayer/). -6. Performance and behavior of traffic sent to the source and target clusters are compared by reviewing logs and metrics. -7. After confirming that the target cluster's functionality meets expectations, clients are redirected to the new target. \ No newline at end of file +3. Using the migration console, a point-in-time snapshot is taken. Once the snapshot completes, `Metadata-Migration-Tool` is used to establish indexes, templates, component templates, and aliases on the target cluster. With continuous traffic capture in place, `Reindex-from-Snapshot` process to migrate data from source. +4. Once `Reindex-from-Snapshot` is complete, captured traffic is replayed from Amazon MSK to the target cluster by `Traffic-Capture-Replayer`. +5. Performance and behavior of traffic sent to the source and target clusters are compared by reviewing logs and metrics. +6. After confirming that the target cluster's functionality meets expectations, clients are redirected to the new target. \ No newline at end of file diff --git a/_migration-assistant/overview/index.md b/_migration-assistant/overview/index.md deleted file mode 100644 index 11c7d716ef5..00000000000 --- a/_migration-assistant/overview/index.md +++ /dev/null @@ -1,25 +0,0 @@ ---- -layout: default -title: Overview -nav_order: 2 -has_children: true -has_toc: false -permalink: /migration-assistant/overview/ -items: - - heading: "Is Migration Assistant right for you?" - description: "Evaluate whether Migration Assistant is right for your use case." - link: "/migration-assistant/overview/is-migration-assistant-right-for-you/" - - heading: "Key components" - description: "Get familiar with the key components of Migration Assistant." - link: "/migration-assistant/overview/key-components/" - - heading: "Architecture" - description: "Understand how Migration Assistant integrates into your infrastructure." - link: "/migration-assistant/overview/architecture/" ---- - -# Migration Assistant overview - -Use this section to get familiar with the key concepts and structure of Migration Assistant before diving into setup or execution. These pages provide the architecture, key components, and guidance to help you determine whether Migration Assistant is right for your use case. - -{% include list.html list_items=page.items%} - diff --git a/_migration-assistant/overview/is-migration-assistant-right-for-you.md b/_migration-assistant/overview/is-migration-assistant-right-for-you.md index 9a5580d54ed..dcca20f553f 100644 --- a/_migration-assistant/overview/is-migration-assistant-right-for-you.md +++ b/_migration-assistant/overview/is-migration-assistant-right-for-you.md @@ -16,45 +16,6 @@ Migration Assistant addresses key limitations in traditional migration approache Migration Assistant also supports live traffic replication, enabling zero-downtime migrations. This makes it a strong fit for environments where minimizing service disruption is critical. -## Migration Assistant assumptions and limitations - -Before using Migration Assistant, review the following assumptions and limitations. They are grouped by scope to clarify which apply generally and which are specific to components. - -### General - - -- If deploying on AWS, the identity used to deploy Migration Assistant must have permission to install all required resources. For a full list of services, see [AWS Services in this Solution](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/architecture-details.html#aws-services-in-this-solution). -- The target cluster must be deployed and reachable by Migration Assistant components, including the Migration Console, Reindex-from-Snapshot, and Traffic Replayer, depending on which features are used. -- The `_source` field must be enabled on all indexes to be migrated. See [Source documentation]({{site.url}}{{site.baseurl}}/field-types/metadata-fields/source/) for more information. -- If deploying to AWS, ensure the `CDKToolkit` stack exists and is in the `CREATE_COMPLETE` state. For setup instructions, see the [CDK Toolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html). -- If deploying Migration Assistant into an existing AWS VPC, you must configure VPC interface endpoints and IAM permissions for the following AWS services: - - **Amazon CloudWatch Logs** – for log ingestion from ECS tasks. - - **Amazon CloudWatch** – for metrics published during migration - - **Amazon Elastic Container Registry (ECR)** – for pulling container images. - - **Amazon Elastic Container Service (ECS)** – for task orchestration and migration. - - **Elastic Load Balancing (ELB)** – for routing traffic to Capture Proxy. - - **AWS Secrets Manager** – for storing credentials. - - **AWS Systems Manager Parameter Store** – for storing and accessing parameter values. - - **AWS Systems Manager Session Manager** – for secure shell access to EC2 (i.e., bootstrap instance). - - **Amazon EC2** – for launching the bootstrap instance responsible for build and deployment. - - **Amazon Elastic Block Store (EBS)** – for disk storage during migration. - - **Amazon Virtual Private Cloud (VPC)** – for private networking and VPC endpoints. - - **AWS X-Ray** – for distributed tracing across components. - - **Amazon Elastic File System (EFS)** – for persistent logging. - -### Reindex-from-Snapshot - -- The source cluster must have the Amazon S3 plugin installed. -- Snapshots must include global cluster state (`include_global_state` is `true`). -- Shards up to 80 GiB are supported by default. Larger shards can be configured, except in AWS GovCloud, where 80 GiB is the maximum supported size. - -### Capture-and-Replay - -- The Traffic Capture Proxy must be deployed to intercept relevant client traffic. -- Live capture is recommended only for environments with less than 4 TB/day of incoming traffic to the source cluster. -- Automatically generated document IDs are not preserved for index requests. Clients must explicitly provide document IDs when indexing or updating documents. - ---- ## Supported migration paths @@ -122,9 +83,6 @@ Before starting an upgrade or migration, consider the cluster feature to be incl | **Security constructs** | No | Configure roles and permissions based on cloud provider recommendations. For example, if using AWS, leverage AWS Identity and Access Management (IAM) for enhanced security management. | | **Plugins** | No | Check plugin compatibility; some Elasticsearch plugins may not have direct OpenSearch equivalents. | -[^2]: Support for Kibana 5.0 through 7.10.2 migration paths to OpenSearch Dashboards will be added in a future version. Kibana 8 and later are not supported. For more information, see [issue #944](https://github.com/opensearch-project/opensearch-migrations/issues/944). - - ## Checklist Use this checklist to determine whether Migration Assistant is the right fit for your migration: @@ -140,3 +98,43 @@ Use this checklist to determine whether Migration Assistant is the right fit for - Do you need a high-performance backfill solution that can reliably reindex documents—with support for pause, resume, or checkpoint recovery? If you answered "yes" to most of these questions, Migration Assistant is likely the right solution for your migration. + +## Migration Assistant assumptions and limitations + +Before using Migration Assistant, review the following assumptions and limitations. They are grouped by scope to clarify which apply generally and which are specific to components. + +### General + + +- If deploying on AWS, the identity used to deploy Migration Assistant must have permission to install all required resources. For a full list of services, see [AWS Services in this Solution](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/architecture-details.html#aws-services-in-this-solution). +- The target cluster must be deployed and reachable by Migration Assistant components, including the Migration Console, Reindex-from-Snapshot, and Traffic Replayer, depending on which features are used. +- The `_source` field must be enabled on all indexes to be migrated. See [Source documentation]({{site.url}}{{site.baseurl}}/field-types/metadata-fields/source/) for more information. +- If deploying to AWS, ensure the `CDKToolkit` stack exists and is in the `CREATE_COMPLETE` state. For setup instructions, see the [CDK Toolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html). +- If deploying Migration Assistant into an existing AWS VPC, you must configure VPC interface endpoints and IAM permissions for the following AWS services: + - **Amazon CloudWatch Logs** – for log ingestion from ECS tasks. + - **Amazon CloudWatch** – for metrics published during migration + - **Amazon Elastic Container Registry (ECR)** – for pulling container images. + - **Amazon Elastic Container Service (ECS)** – for task orchestration and migration. + - **Elastic Load Balancing (ELB)** – for routing traffic to Capture Proxy. + - **AWS Secrets Manager** – for storing credentials. + - **AWS Systems Manager Parameter Store** – for storing and accessing parameter values. + - **AWS Systems Manager Session Manager** – for secure shell access to EC2 (i.e., bootstrap instance). + - **Amazon EC2** – for launching the bootstrap instance responsible for build and deployment. + - **Amazon Elastic Block Store (EBS)** – for disk storage during migration. + - **Amazon Virtual Private Cloud (VPC)** – for private networking and VPC endpoints. + - **AWS X-Ray** – for distributed tracing across components. + - **Amazon Elastic File System (EFS)** – for persistent logging. + +### Reindex-from-Snapshot + +- The source cluster must have the Amazon S3 plugin installed. +- Snapshots must include global cluster state (`include_global_state` is `true`). +- Shards up to 80 GiB are supported by default. Larger shards can be configured, except in AWS GovCloud, where 80 GiB is the maximum supported size. + +### Capture-and-Replay + +- The Traffic Capture Proxy must be deployed to intercept relevant client traffic. +- Live capture is recommended only for environments with less than 4 TB/day of incoming traffic to the source cluster. +- Automatically generated document IDs are not preserved for index requests. Clients must explicitly provide document IDs when indexing or updating documents. + +--- \ No newline at end of file diff --git a/_migration-assistant/overview/key-components.md b/_migration-assistant/overview/key-components.md index cfa69e70311..6e5aa34db44 100644 --- a/_migration-assistant/overview/key-components.md +++ b/_migration-assistant/overview/key-components.md @@ -12,13 +12,13 @@ The following are the key components of Migration Assistant. ## Elasticsearch/OpenSearch source -In this solution, your source cluster operates on either Elasticsearch or OpenSearch and is hosted on Amazon Elastic Compute Cloud (Amazon EC2) instances or in a similar computing environment. A proxy is set up to interact with this source cluster, either positioned in front of or directly on the coordinating nodes of the cluster. +In this solution, your source cluster operates on either Elasticsearch or OpenSearch and is hosted on Amazon Elastic Compute Cloud (Amazon EC2) instances or in a similar computing environment. Traffic is rerouted from the source cluster to a traffic capture proxy and replayed to a target typically on a later version of OpenSearch. -## Migration console +## Migration Console The migration console provides a migration-specific CLI and offers a variety of tools for streamlining the migration process. Everything necessary for completing a migration, other than cleaning up the migration resources, can be performed through this console. -## Traffic capture proxy +## Traffic Capture Proxy This component is designed for HTTP RESTful traffic. It forwards traffic to the source cluster and also splits and channels this traffic to a stream processing service for later playback. @@ -26,7 +26,7 @@ This component is designed for HTTP RESTful traffic. It forwards traffic to the Acting as a traffic simulation tool, [Traffic Replayer](https://docs.opensearch.org/docs/latest/migration-assistant/migration-phases/live-traffic-migration/using-traffic-replayer/) replays recorded request traffic to a target cluster, mirroring source traffic patterns. It links original requests and their responses to those directed at the target cluster, facilitating comparative analysis. -## Metadata migration tool +## Metadata-Migration-Tool The metadata migration tool integrated into the Migration CLI can be used independently to migrate cluster metadata, including index mappings, index configuration settings, templates, component templates, and aliases. @@ -36,4 +36,4 @@ The metadata migration tool integrated into the Migration CLI can be used indepe ## Target cluster -The target cluster is the destination cluster for migration or comparison in an A/B test. \ No newline at end of file +The target cluster is the destination cluster for migration or comparison cluster in an A/B test. \ No newline at end of file diff --git a/_migration-upgrade/index.md b/_migration-upgrade/index.md index 7aba907619d..bd0f7319ee0 100644 --- a/_migration-upgrade/index.md +++ b/_migration-upgrade/index.md @@ -50,7 +50,7 @@ Migration Assistant offers the most automated and resilient path for OpenSearch ### Getting started with Migration Assistant -- Review the [Migration Assistant]({{site.url}}{{site.baseurl}}/migration-assistant/) +- Upgrade or migrate with [Migration Assistant for OpenSearch]({{site.url}}{{site.baseurl}}/migration-assistant/) ## Rolling upgrades @@ -68,8 +68,7 @@ Rolling upgrades allow you to upgrade one node at a time, keeping the cluster op ### Getting started with rolling upgrades -- Perform a [rolling upgrade]({{site.url}}{{site.baseurl}}/install-and-configure/upgrade-opensearch/rolling-upgrade/) - +- Perform a [rolling upgrade]({{site.url}}{{site.baseurl}}/install-and-configure/upgrade-opensearch/rolling-upgrade/). ## Snapshot and restore @@ -77,7 +76,7 @@ Rolling upgrades allow you to upgrade one node at a time, keeping the cluster op This method involves taking a snapshot of your existing OpenSearch or Elasticsearch OSS cluster and restoring it to a new cluster running the target version. **Pros:** -- Original cluster remains untouched, allowing rollback if needed (note: may result in data loss without a change data capture (CDC) solution). +- Original cluster remains untouched, allowing rollback if needed (note: may result in data loss without a change data capture (CDC) solution. In some cases, one may choose to use `Capture-and-Replay` from Migration Assistant in combination with Snapshot and Restore). - Scales well for large data volumes, including cold storage and datasets up to 1 PB. **Cons:** @@ -85,6 +84,10 @@ This method involves taking a snapshot of your existing OpenSearch or Elasticsea - Requires provisioning a new cluster. - Manual reindexing may be necessary for full feature support. +### Get started with Snapshot and Restore + +Upgrade or Migration with [Snapshot and Restore](https://docs.aws.amazon.com/solutions/latest/ tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore/) + ## Remote reindexing This method reindexes data from your current cluster into a new OpenSearch cluster, typically running in parallel. From 198c4487bdbe2cdc1b560bec0976bd62cc189c09 Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Thu, 3 Jul 2025 13:07:16 -0500 Subject: [PATCH 12/62] Moved assets from _migration-upgrade to _migration-assistant Signed-off-by: Brian Presley --- .../configuration-options.md | 0 .../getting-started-data-migration.md | 0 ...d-security-groups-for-existing-clusters.md | 0 .../deploying-migration-assistant/index.md | 0 _migration-assistant/index.md | 1 + .../accessing-the-migration-console.md | 0 .../migration-console/index.md | 0 .../migration-console-commands-references.md | 0 .../migration-phases/backfill.md | 0 .../migration-phases/deployment.md | 2 +- .../migration-phases/index.md | 2 +- .../live-traffic-migration/index.md | 0 ...itching-traffic-from-the-source-cluster.md | 0 .../using-traffic-replayer.md | 0 .../migration-phases/migrating-metadata.md | 0 .../assessing-your-cluster-for-migration.md | 0 .../handling-field-type-breaking-changes.md | 0 .../handling-type-mapping-deprecation.md | 0 .../planning-your-migration/index.md | 0 .../verifying-migration-tools.md | 0 .../removing-migration-infrastructure.md | 0 .../overview/index.md | 0 .../deploying-migration-assistant/index.md | 13 -- _migration-upgrade/migration-phases/index.md | 87 ---------- _migration-upgrade/overview/architecture.md | 25 --- .../is-migration-assistant-right-for-you.md | 154 ------------------ _migration-upgrade/overview/key-components.md | 39 ----- 27 files changed, 3 insertions(+), 320 deletions(-) rename {_migration-upgrade => _migration-assistant}/deploying-migration-assistant/configuration-options.md (100%) rename {_migration-upgrade => _migration-assistant}/deploying-migration-assistant/getting-started-data-migration.md (100%) rename {_migration-upgrade => _migration-assistant}/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md (100%) rename _migration-assistant/{migration-phases => }/deploying-migration-assistant/index.md (100%) rename {_migration-upgrade => _migration-assistant}/migration-console/accessing-the-migration-console.md (100%) rename {_migration-upgrade => _migration-assistant}/migration-console/index.md (100%) rename {_migration-upgrade => _migration-assistant}/migration-console/migration-console-commands-references.md (100%) rename {_migration-upgrade => _migration-assistant}/migration-phases/backfill.md (100%) rename {_migration-upgrade => _migration-assistant}/migration-phases/live-traffic-migration/index.md (100%) rename {_migration-upgrade => _migration-assistant}/migration-phases/live-traffic-migration/switching-traffic-from-the-source-cluster.md (100%) rename {_migration-upgrade => _migration-assistant}/migration-phases/live-traffic-migration/using-traffic-replayer.md (100%) rename {_migration-upgrade => _migration-assistant}/migration-phases/migrating-metadata.md (100%) rename {_migration-upgrade => _migration-assistant}/migration-phases/planning-your-migration/assessing-your-cluster-for-migration.md (100%) rename {_migration-upgrade => _migration-assistant}/migration-phases/planning-your-migration/handling-field-type-breaking-changes.md (100%) rename {_migration-upgrade => _migration-assistant}/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md (100%) rename {_migration-upgrade => _migration-assistant}/migration-phases/planning-your-migration/index.md (100%) rename {_migration-upgrade => _migration-assistant}/migration-phases/planning-your-migration/verifying-migration-tools.md (100%) rename {_migration-upgrade => _migration-assistant}/migration-phases/removing-migration-infrastructure.md (100%) rename {_migration-upgrade => _migration-assistant}/overview/index.md (100%) delete mode 100644 _migration-upgrade/deploying-migration-assistant/index.md delete mode 100644 _migration-upgrade/migration-phases/index.md delete mode 100644 _migration-upgrade/overview/architecture.md delete mode 100644 _migration-upgrade/overview/is-migration-assistant-right-for-you.md delete mode 100644 _migration-upgrade/overview/key-components.md diff --git a/_migration-upgrade/deploying-migration-assistant/configuration-options.md b/_migration-assistant/deploying-migration-assistant/configuration-options.md similarity index 100% rename from _migration-upgrade/deploying-migration-assistant/configuration-options.md rename to _migration-assistant/deploying-migration-assistant/configuration-options.md diff --git a/_migration-upgrade/deploying-migration-assistant/getting-started-data-migration.md b/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md similarity index 100% rename from _migration-upgrade/deploying-migration-assistant/getting-started-data-migration.md rename to _migration-assistant/deploying-migration-assistant/getting-started-data-migration.md diff --git a/_migration-upgrade/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md b/_migration-assistant/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md similarity index 100% rename from _migration-upgrade/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md rename to _migration-assistant/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md diff --git a/_migration-assistant/migration-phases/deploying-migration-assistant/index.md b/_migration-assistant/deploying-migration-assistant/index.md similarity index 100% rename from _migration-assistant/migration-phases/deploying-migration-assistant/index.md rename to _migration-assistant/deploying-migration-assistant/index.md diff --git a/_migration-assistant/index.md b/_migration-assistant/index.md index e4147b7dcff..cf3a006ce32 100644 --- a/_migration-assistant/index.md +++ b/_migration-assistant/index.md @@ -12,6 +12,7 @@ redirect_from: - /upgrade-to/index/ - /upgrade-to/ - /upgrade-to/upgrade-to/ + - /upgrade-to/snapshot-migrate/ items: - heading: "Is Migration Assistant right for you?" description: "Evaluate whether Migration Assistant is right for your use case." diff --git a/_migration-upgrade/migration-console/accessing-the-migration-console.md b/_migration-assistant/migration-console/accessing-the-migration-console.md similarity index 100% rename from _migration-upgrade/migration-console/accessing-the-migration-console.md rename to _migration-assistant/migration-console/accessing-the-migration-console.md diff --git a/_migration-upgrade/migration-console/index.md b/_migration-assistant/migration-console/index.md similarity index 100% rename from _migration-upgrade/migration-console/index.md rename to _migration-assistant/migration-console/index.md diff --git a/_migration-upgrade/migration-console/migration-console-commands-references.md b/_migration-assistant/migration-console/migration-console-commands-references.md similarity index 100% rename from _migration-upgrade/migration-console/migration-console-commands-references.md rename to _migration-assistant/migration-console/migration-console-commands-references.md diff --git a/_migration-upgrade/migration-phases/backfill.md b/_migration-assistant/migration-phases/backfill.md similarity index 100% rename from _migration-upgrade/migration-phases/backfill.md rename to _migration-assistant/migration-phases/backfill.md diff --git a/_migration-assistant/migration-phases/deployment.md b/_migration-assistant/migration-phases/deployment.md index 2c4da875663..40e3e6244ee 100644 --- a/_migration-assistant/migration-phases/deployment.md +++ b/_migration-assistant/migration-phases/deployment.md @@ -4,8 +4,8 @@ title: Deploy parent: Migration phases nav_order: 2 redirect_from: - - /upgrade-to/snapshot-migrate/ - /migration-assistant/getting-started-with-data-migration/ + - /deploying-migration-assistant/ --- # Deploying Migration Assistant diff --git a/_migration-assistant/migration-phases/index.md b/_migration-assistant/migration-phases/index.md index 6db56d90d2e..c8b76a34dda 100644 --- a/_migration-assistant/migration-phases/index.md +++ b/_migration-assistant/migration-phases/index.md @@ -3,7 +3,7 @@ layout: default title: Migration phases parent: Migration Assistant nav_order: 30 -has_children: false +has_children: true has_toc: false permalink: /migration-assistant/migration-phases/ redirect-from: /migration-assistant/migration-phases/index/ diff --git a/_migration-upgrade/migration-phases/live-traffic-migration/index.md b/_migration-assistant/migration-phases/live-traffic-migration/index.md similarity index 100% rename from _migration-upgrade/migration-phases/live-traffic-migration/index.md rename to _migration-assistant/migration-phases/live-traffic-migration/index.md diff --git a/_migration-upgrade/migration-phases/live-traffic-migration/switching-traffic-from-the-source-cluster.md b/_migration-assistant/migration-phases/live-traffic-migration/switching-traffic-from-the-source-cluster.md similarity index 100% rename from _migration-upgrade/migration-phases/live-traffic-migration/switching-traffic-from-the-source-cluster.md rename to _migration-assistant/migration-phases/live-traffic-migration/switching-traffic-from-the-source-cluster.md diff --git a/_migration-upgrade/migration-phases/live-traffic-migration/using-traffic-replayer.md b/_migration-assistant/migration-phases/live-traffic-migration/using-traffic-replayer.md similarity index 100% rename from _migration-upgrade/migration-phases/live-traffic-migration/using-traffic-replayer.md rename to _migration-assistant/migration-phases/live-traffic-migration/using-traffic-replayer.md diff --git a/_migration-upgrade/migration-phases/migrating-metadata.md b/_migration-assistant/migration-phases/migrating-metadata.md similarity index 100% rename from _migration-upgrade/migration-phases/migrating-metadata.md rename to _migration-assistant/migration-phases/migrating-metadata.md diff --git a/_migration-upgrade/migration-phases/planning-your-migration/assessing-your-cluster-for-migration.md b/_migration-assistant/migration-phases/planning-your-migration/assessing-your-cluster-for-migration.md similarity index 100% rename from _migration-upgrade/migration-phases/planning-your-migration/assessing-your-cluster-for-migration.md rename to _migration-assistant/migration-phases/planning-your-migration/assessing-your-cluster-for-migration.md diff --git a/_migration-upgrade/migration-phases/planning-your-migration/handling-field-type-breaking-changes.md b/_migration-assistant/migration-phases/planning-your-migration/handling-field-type-breaking-changes.md similarity index 100% rename from _migration-upgrade/migration-phases/planning-your-migration/handling-field-type-breaking-changes.md rename to _migration-assistant/migration-phases/planning-your-migration/handling-field-type-breaking-changes.md diff --git a/_migration-upgrade/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md b/_migration-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md similarity index 100% rename from _migration-upgrade/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md rename to _migration-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md diff --git a/_migration-upgrade/migration-phases/planning-your-migration/index.md b/_migration-assistant/migration-phases/planning-your-migration/index.md similarity index 100% rename from _migration-upgrade/migration-phases/planning-your-migration/index.md rename to _migration-assistant/migration-phases/planning-your-migration/index.md diff --git a/_migration-upgrade/migration-phases/planning-your-migration/verifying-migration-tools.md b/_migration-assistant/migration-phases/planning-your-migration/verifying-migration-tools.md similarity index 100% rename from _migration-upgrade/migration-phases/planning-your-migration/verifying-migration-tools.md rename to _migration-assistant/migration-phases/planning-your-migration/verifying-migration-tools.md diff --git a/_migration-upgrade/migration-phases/removing-migration-infrastructure.md b/_migration-assistant/migration-phases/removing-migration-infrastructure.md similarity index 100% rename from _migration-upgrade/migration-phases/removing-migration-infrastructure.md rename to _migration-assistant/migration-phases/removing-migration-infrastructure.md diff --git a/_migration-upgrade/overview/index.md b/_migration-assistant/overview/index.md similarity index 100% rename from _migration-upgrade/overview/index.md rename to _migration-assistant/overview/index.md diff --git a/_migration-upgrade/deploying-migration-assistant/index.md b/_migration-upgrade/deploying-migration-assistant/index.md deleted file mode 100644 index 1c559a81b1c..00000000000 --- a/_migration-upgrade/deploying-migration-assistant/index.md +++ /dev/null @@ -1,13 +0,0 @@ ---- -layout: default -title: Deploying Migration Assistant -nav_order: 15 -has_children: true -permalink: /deploying-migration-assistant/ -redirect-from: - - /deploying-migration-assistant/index/ ---- - -# Deploying Migration Assistant - -This section provides information about the available options for deploying Migration Assistant. diff --git a/_migration-upgrade/migration-phases/index.md b/_migration-upgrade/migration-phases/index.md deleted file mode 100644 index d8f314d6b75..00000000000 --- a/_migration-upgrade/migration-phases/index.md +++ /dev/null @@ -1,87 +0,0 @@ ---- -layout: default -title: Migration phases -nav_order: 30 -has_children: true -has_toc: false -permalink: /migration-phases/ -redirect_from: - - /migration-phases/index/ ---- - -# Migration phases - -This page outlines the core phases of migrating with Migration Assistant. Each scenario below consists of a sequence of common steps. Expand a scenario to explore the detailed flow. - ---- - - - -
    -Scenario 1 – Backfill Only - -
      -
    1. Assessment
      2. -
    2. Deployment
      3. - -
    3. Create Snapshot
      4. -
    4. Migrate Metadata
      5. -
    5. Migrate Data
      6. -
    6. Teardown
      7. -
    - -
    - -
    -Scenario 2 – Live Capture Only - -
      -
    1. Assessment
      2. -
    2. Deployment
      3. -
    3. Verify Backfill Components
      4. -
    4. Reroute Traffic from Source to Capture Proxy
      5. -
    5. Migrate Metadata
      6. -
    6. Verify Live Capture Components
      7. -
    7. Replay Captured Traffic
      8. -
    8. Reroute Traffic from Capture Proxy to Target
      9. -
    9. Teardown
      10. -
    - -
    - -
    -Scenario 3 – Live Capture with Backfill - -
      -
    1. Assessment
      2. -
    2. Deployment
      3. -
    3. Reroute Traffic from Source to Capture Proxy
      4. -
    4. Create Snapshot
      5. -
    5. Migrate Metadata
      6. -
    6. Migrate Data
      7. -
    7. Replay Traffic
      8. -
    8. Reroute Traffic from Capture Proxy to Target
      9. -
    9. Teardown
      10. -
    - -
    - ---- - -💡 *Need help getting started? Begin with [Planning your migration]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/planning-your-migration/index/).* diff --git a/_migration-upgrade/overview/architecture.md b/_migration-upgrade/overview/architecture.md deleted file mode 100644 index 48b3c8fbcfc..00000000000 --- a/_migration-upgrade/overview/architecture.md +++ /dev/null @@ -1,25 +0,0 @@ ---- -layout: default -title: Architecture -nav_order: 15 -parent: Overview -permalink: /migration-assistant/overview/architecture/ ---- - -# Architecture - -The Migration Assistant architecture is based on the use of an AWS Cloud infrastructure, but most tools are designed to be cloud independent. A local containerized version of this solution is also available. - -The design deployed on AWS uses the following architecture. - -![Migration architecture overview]({{site.url}}{{site.baseurl}}/images/migrations/migrations-architecture-overview.png) - -Each node in the diagram correlates to the following steps in the migration process: - -1. Client traffic is directed to the existing cluster. -2. An Application Load Balancer with capture proxies relays traffic to a source while replicating data to Amazon Managed Streaming for Apache Kafka (Amazon MSK). -3. Using the migration console, you can initiate metadata migration to establish indexes, templates, component templates, and aliases on the target cluster. -4. With continuous traffic capture in place, you can use a `reindex-from-snapshot` process to capture data from your current index. -5. Once `Reindex-from-Snapshot` is complete, captured traffic is replayed from Amazon MSK to the target cluster by [Traffic Replayer](https://docs.opensearch.org/docs/latest/migration-assistant/migration-phases/live-traffic-migration/using-traffic-replayer/). -6. Performance and behavior of traffic sent to the source and target clusters are compared by reviewing logs and metrics. -7. After confirming that the target cluster's functionality meets expectations, clients are redirected to the new target. \ No newline at end of file diff --git a/_migration-upgrade/overview/is-migration-assistant-right-for-you.md b/_migration-upgrade/overview/is-migration-assistant-right-for-you.md deleted file mode 100644 index 1e7fc287974..00000000000 --- a/_migration-upgrade/overview/is-migration-assistant-right-for-you.md +++ /dev/null @@ -1,154 +0,0 @@ ---- -layout: default -title: Is Migration Assistant right for you? -nav_order: 10 -parent: Overview -permalink: /migration-assistant/overview/is-migration-assistant-right-for-you/ -redirect_from: - - /migration-assistant/is-migration-assistant-right-for-you/ ---- - -# Is Migration Assistant right for you? - -Deciding whether to use Migration Assistant depends on your specific upgrade path, infrastructure complexity, and operational goals. This page will help you evaluate whether Migration Assistant is right for your use case—or whether another tool might be a better fit. - -Migration Assistant was built to fill important gaps in common migration strategies. For example, if you're upgrading across multiple major versions—such as from Elasticsearch 6.8 to OpenSearch 2.19—Migration Assistant lets you do this in a single step. Other methods, like rolling upgrades or snapshot restores, require you to upgrade through each major version, often reindexing your data at every step. - -Migration Assistant also supports live traffic replication, allowing for zero-downtime migrations. This makes it a strong choice for production environments, where minimizing service disruption is critical. - -If your migration is limited to a static cluster configuration (like index templates and aliases), or if you're not concerned about downtime, simpler tools may be sufficient. But for complex migrations involving real-time traffic or major version jumps, Migration Assistant offers robust, flexible capabilities. - -## Supported migration paths - -The following matrix shows which source versions can be directly migrated to which OpenSearch target versions: - - -{% comment %}First, collect all unique target versions{% endcomment %} -{% assign all_targets = "" | split: "" %} -{% for path in site.data.migration-assistant.valid_migrations.migration_paths %} - {% for target in path.targets %} - {% assign all_targets = all_targets | push: target %} - {% endfor %} -{% endfor %} -{% assign unique_targets = all_targets | uniq | sort %} - - - - - - {% for target in unique_targets %} - - {% endfor %} - - - - {% for path in site.data.migration-assistant.valid_migrations.migration_paths %} - - - {% for target_version in unique_targets %} - - {% endfor %} - - {% endfor %} - -
    {{ target }}
    {{ path.source }} - {% if path.targets contains target_version %}✓{% endif %} -
    - -## Supported platforms - -**Source and target platforms** - -- Self-managed (on premises or hosted by a cloud provider) -- Amazon OpenSearch Service - - -**AWS Regions** - -Migration Assistant is supported in the following AWS Regions: - -- US East (N. Virginia, Ohio) -- US West (Oregon, N. California) -- Europe (Frankfurt, Ireland, London) -- Asia Pacific (Tokyo, Singapore, Sydney) -- AWS GovCloud (US-East, US-West)[^1] - -[^1]: In AWS GovCloud (US), `reindex-from-snapshot` (RFS) is limited to shard sizes of 80 GiB or smaller. - - - -## Supported components - -Before starting a migration, consider the scope of the components involved. The following table outlines components that should potentially be migrated, indicates whether they are supported by Migration Assistant, and provides recommendations. - -| Component | Supported | Recommendations | -| :--- |:--- | :--- | -| **Documents** | Yes | Migrate existing data with RFS and live traffic with capture and replay. | -| **Index settings** | Yes | Migrate with the `Metadata-Migration-Tool`. | -| **Index mappings** | Yes | Migrate with the `Metadata-Migration-Tool`. | -| **Index templates** | Yes | Migrate with the `Metadata-Migration-Tool`. | -| **Component templates** | Yes | Migrate with the `Metadata-Migration-Tool`. | -| **Aliases** | Yes | Migrate with the `Metadata-Migration-Tool`. | -| **Index State Management (ISM) policies** | Expected in 2025 | Manually migrate using an API. For more information about ISM support, see [issue #944](https://github.com/opensearch-project/opensearch-migrations/issues/944). | -| **Elasticsearch Kibana[^2] dashboards** | Expected in 2025 | This tool is only needed when migrating from Elasticsearch Kibana dashboards to OpenSearch Dashboards. Start by exporting JSON files from Kibana and importing them into OpenSearch Dashboards. For Elasticsearch versions 7.10.2 to 7.17, use the [`dashboardsSanitizer`](https://github.com/opensearch-project/opensearch-migrations/tree/main/dashboardsSanitizer) tool before importing X-Pack visualizations like Canvas and Lens in Kibana dashboards, as they may require recreation for compatibility with OpenSearch.| -| **Security constructs** | No | Configure roles and permissions based on cloud provider recommendations. For example, if using AWS, leverage AWS Identity and Access Management (IAM) for enhanced security management. | -| **Plugins** | No | Check plugin compatibility; some Elasticsearch plugins may not have direct OpenSearch equivalents. | - -[^2]: Support for Kibana 5.0 through 7.10.2 migration paths to OpenSearch Dashboards will be added in a future version. Kibana 8 and later are not supported. For more information, see [issue #944](https://github.com/opensearch-project/opensearch-migrations/issues/944). - -## Choosing your migration approach - -Use the following checklist to determine which Migration Assistant components best fit your use case. - -### Metadata migration - -Use [metadata migration]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrating-metadata/) if: - -- You need to migrate while mitigating breaking changes between the source and target clusters, such as differences in mappings, settings, aliases, or component templates. -- You want a relatively consistent configuration between the source and target clusters. - -### Backfill migration - -Use [backfill migration]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/backfill/) if: - -- You need to move historical data without disrupting live traffic. -- You want to backfill indexes from a specific point in time without impacting the source cluster. -- You want to verify historical data in the target cluster before switching over. -- You want to backfill using an existing or incremental snapshot. -- You need the fastest backfill option that includes reindexing. -- You want the ability to pause and resume migration. - -### RFS - -Use [RFS]({{site.url}}{{site.baseurl}}/migration-assistant/deploying-migration-assistant/getting-started-data-migration/) if: - -- You already use OpenSearch snapshots for backups. -- You need to migrate documents at scale in parallel, such as with Amazon Elastic Container Service (Amazon ECS). -- You require a data migration path as part of a zero-downtime migration. -- Your AWS Region supports RFS and your shard sizes are within supported limits. - -### Combination of all three - -Use a combination of all three migration types if: - -- You're performing a complex, multi-version migration. -- You require zero downtime and full validation of the target environment. -- You want end-to-end tooling for metadata, data movement, and cluster behavior comparison. -- You're cloning an existing cluster and changing the source's configuration. -- You're setting up disaster recovery. - -## Checklist - -Use this checklist to decide whether Migration Assistant is right for you: - -- Are you migrating across one or more major versions? - -- Do you need to maintain service availability with zero downtime? - -- Do you need to validate a new OpenSearch cluster before switching over? - -- Is your environment self-managed or running on Amazon OpenSearch Service? - -- Are you looking for tooling that can automate metadata migration and performance comparison? - -If you answered "yes" to most of these questions, Migration Assistant is likely the right solution for your migration. diff --git a/_migration-upgrade/overview/key-components.md b/_migration-upgrade/overview/key-components.md deleted file mode 100644 index cfa69e70311..00000000000 --- a/_migration-upgrade/overview/key-components.md +++ /dev/null @@ -1,39 +0,0 @@ ---- -layout: default -title: Key components -nav_order: 10 -parent: Overview -permalink: /migration-assistant/overview/key-components/ ---- - -# Key components - -The following are the key components of Migration Assistant. - -## Elasticsearch/OpenSearch source - -In this solution, your source cluster operates on either Elasticsearch or OpenSearch and is hosted on Amazon Elastic Compute Cloud (Amazon EC2) instances or in a similar computing environment. A proxy is set up to interact with this source cluster, either positioned in front of or directly on the coordinating nodes of the cluster. - -## Migration console - -The migration console provides a migration-specific CLI and offers a variety of tools for streamlining the migration process. Everything necessary for completing a migration, other than cleaning up the migration resources, can be performed through this console. - -## Traffic capture proxy - -This component is designed for HTTP RESTful traffic. It forwards traffic to the source cluster and also splits and channels this traffic to a stream processing service for later playback. - -## Traffic Replayer - -Acting as a traffic simulation tool, [Traffic Replayer](https://docs.opensearch.org/docs/latest/migration-assistant/migration-phases/live-traffic-migration/using-traffic-replayer/) replays recorded request traffic to a target cluster, mirroring source traffic patterns. It links original requests and their responses to those directed at the target cluster, facilitating comparative analysis. - -## Metadata migration tool - -The metadata migration tool integrated into the Migration CLI can be used independently to migrate cluster metadata, including index mappings, index configuration settings, templates, component templates, and aliases. - -## Reindex-from-Snapshot - -`Reindex-from-Snapshot` (RFS) reindexes data from an existing snapshot. Amazon Elastic Container Service (Amazon ECS) workers coordinate the migration of documents from an existing snapshot, reindexing the documents in parallel to a target cluster. - -## Target cluster - -The target cluster is the destination cluster for migration or comparison in an A/B test. \ No newline at end of file From 1f597805a63a569353344454d91e10d592ad4a9d Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Thu, 3 Jul 2025 13:09:33 -0500 Subject: [PATCH 13/62] Add Migration Assistant to _config.yml Signed-off-by: Brian Presley --- _config.yml | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/_config.yml b/_config.yml index 27f751e4010..a1d9d2a0464 100644 --- a/_config.yml +++ b/_config.yml @@ -218,10 +218,10 @@ clients_collection: name: Clients nav_fold: true -migration_upgrade_collection: +migrations_and_upgrades_collection: collections: - migration-upgrade: - name: Migration and upgrade + migrations-and-upgrades: + name: Migrations and upgrades nav_fold: true migration_assistant_collection: @@ -274,7 +274,13 @@ defaults: path: "_migrations-and-upgrades" values: section: "migrations-and-upgrades" - section-name: "Migrations and Upgrades" + section-name: "Migrations and upgrades" + - + scope: + path: "_migration-assistant" + values: + section: "migrations-assistant" + section-name: "Migration Assistant" # Enable or disable the site search # By default, just-the-docs enables its JSON file-based search. We also have an OpenSearch-driven search functionality. From 2a8b60151228eb3a213e2814dc47cbe8044c2ab1 Mon Sep 17 00:00:00 2001 From: Archer Date: Thu, 10 Jul 2025 11:32:32 -0500 Subject: [PATCH 14/62] Add additional redirects Signed-off-by: Archer --- _config.yml | 18 +- .../configuration-options.md | 233 ------------ .../getting-started-data-migration.md | 360 ------------------ ...d-security-groups-for-existing-clusters.md | 93 ----- .../deploying-migration-assistant/index.md | 13 - _migration-assistant/index.md | 4 - .../migration-console-commands-references.md | 2 +- .../accessing-the-migration-console.md | 4 +- .../migration-phases/assessment.md | 4 +- .../migration-phases/backfill.md | 4 +- .../migration-phases/create-snapshot.md | 12 +- .../configuration-options.md | 2 +- .../getting-started-data-migration.md | 2 +- ...d-security-groups-for-existing-clusters.md | 2 +- .../migration-phases/deployment.md | 3 +- .../migration-phases/index.md | 11 +- .../live-traffic-migration/index.md | 2 +- ...itching-traffic-from-the-source-cluster.md | 3 +- .../migration-phases/migrate-metadata.md | 249 ------------ .../migration-phases/migrating-metadata.md | 11 +- .../assessing-your-cluster-for-migration.md | 77 ---- .../handling-field-type-breaking-changes.md | 159 -------- .../handling-type-mapping-deprecation.md | 319 ---------------- .../planning-your-migration/index.md | 16 - .../verifying-migration-tools.md | 224 ----------- .../removing-migration-infrastructure.md | 2 +- .../replay-captured-traffic.md | 319 ---------------- .../migration-phases/teardown.md | 30 -- .../using-traffic-replayer.md | 6 +- .../verifying-backfill-components.md} | 8 +- .../verifying-live-capture-components.md} | 8 +- .../verifying-migration-tools.md | 20 + .../handling-field-type-breaking-changes.md | 1 - .../handling-type-mapping-deprecation.md | 1 - .../planning-your-migration/index.md | 5 +- 35 files changed, 69 insertions(+), 2158 deletions(-) delete mode 100644 _migration-assistant/deploying-migration-assistant/configuration-options.md delete mode 100644 _migration-assistant/deploying-migration-assistant/getting-started-data-migration.md delete mode 100644 _migration-assistant/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md delete mode 100644 _migration-assistant/deploying-migration-assistant/index.md delete mode 100644 _migration-assistant/migration-phases/migrate-metadata.md delete mode 100644 _migration-assistant/migration-phases/planning-your-migration/assessing-your-cluster-for-migration.md delete mode 100644 _migration-assistant/migration-phases/planning-your-migration/handling-field-type-breaking-changes.md delete mode 100644 _migration-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md delete mode 100644 _migration-assistant/migration-phases/planning-your-migration/index.md delete mode 100644 _migration-assistant/migration-phases/planning-your-migration/verifying-migration-tools.md delete mode 100644 _migration-assistant/migration-phases/replay-captured-traffic.md delete mode 100644 _migration-assistant/migration-phases/teardown.md rename _migration-assistant/migration-phases/{live-traffic-migration => }/using-traffic-replayer.md (98%) rename _migration-assistant/migration-phases/{verify-backfill-components.md => verifying-migration-tools/verifying-backfill-components.md} (98%) rename _migration-assistant/migration-phases/{verify-live-capture-components.md => verifying-migration-tools/verifying-live-capture-components.md} (98%) create mode 100644 _migration-assistant/migration-phases/verifying-migration-tools/verifying-migration-tools.md diff --git a/_config.yml b/_config.yml index a1d9d2a0464..1d4455bd9e3 100644 --- a/_config.yml +++ b/_config.yml @@ -145,7 +145,10 @@ opensearch_collection: name: Tutorials nav_fold: true install-and-configure: - name: Install and upgrade + name: Install and configure + nav_fold: true + migration-upgrade: + name: Migrate or upgrade nav_fold: true tuning-your-cluster: name: Creating and tuning your cluster @@ -218,12 +221,6 @@ clients_collection: name: Clients nav_fold: true -migrations_and_upgrades_collection: - collections: - migrations-and-upgrades: - name: Migrations and upgrades - nav_fold: true - migration_assistant_collection: collections: migration-assistant: @@ -269,12 +266,6 @@ defaults: values: section: "benchmark" section-name: "Benchmark" - - - scope: - path: "_migrations-and-upgrades" - values: - section: "migrations-and-upgrades" - section-name: "Migrations and upgrades" - scope: path: "_migration-assistant" @@ -341,7 +332,6 @@ plugins: - jekyll-redirect-from - jekyll-sitemap - jekyll-spec-insert - - jekyll-include-cache # This format has to conform to RFC822 last-modified-at: diff --git a/_migration-assistant/deploying-migration-assistant/configuration-options.md b/_migration-assistant/deploying-migration-assistant/configuration-options.md deleted file mode 100644 index 5032dcb757e..00000000000 --- a/_migration-assistant/deploying-migration-assistant/configuration-options.md +++ /dev/null @@ -1,233 +0,0 @@ ---- -layout: default -title: Configuration options -nav_order: 15 -parent: Deploying Migration Assistant -permalink: /migration-assistant/deploying-migration-assistant/configuration-options/ ---- - -# Configuration options - -This page outlines the configuration options for three key migrations scenarios: - -1. **Metadata migration** -2. **Backfill migration with `Reindex-from-Snapshot` (RFS)** -3. **Live capture migration with Capture and Replay (C&R)** - -Each of these migrations depends on either a snapshot or a capture proxy. The following example `cdk.context.json` configurations are used by AWS Cloud Development Kit (AWS CDK) to deploy and configure Migration Assistant for OpenSearch, shown as separate blocks for each migration type. If you are performing a migration applicable to multiple scenarios, these options can be combined. - - -For a complete list of configuration options, see [opensearch-migrations-options.md](https://github.com/opensearch-project/opensearch-migrations/blob/main/deployment/cdk/opensearch-service-migration/options.md). If you need a configuration option that is not found on this page, create an issue in the [OpenSearch Migrations repository](https://github.com/opensearch-project/opensearch-migrations/issues). -{: .tip } - -Options for the source cluster endpoint, target cluster endpoint, and existing virtual private cloud (VPC) should be configured in order for the migration tools to function effectively. - -## Shared configuration options - -Each migration configuration shares the following options. - - -| Name | Example | Description | -| :--- | :--- | :--- | -| `sourceClusterEndpoint` | `"https://source-cluster.elb.us-east-1.endpoint.com"` | The endpoint for the source cluster. | -| `targetClusterEndpoint` | `"https://vpc-demo-opensearch-cluster-cv6hggdb66ybpk4kxssqt6zdhu.us-west-2.es.amazonaws.com:443"` | The endpoint for the target cluster. Required if using an existing target cluster for the migration instead of creating a new one. | -| `vpcId` | `"vpc-123456789abcdefgh"` | The ID of the existing VPC in which the migration resources will be stored. The VPC must have at least two private subnets that span two Availability Zones. | - - -## Backfill migration using RFS - -The following CDK performs a backfill migrations using RFS: - -```json -{ - "backfill-migration": { - "stage": "dev", - "vpcId": , - "sourceCluster": { - "endpoint": , - "version": "ES 7.10", - "auth": {"type": "none"} - }, - "targetCluster": { - "endpoint": , - "auth": { - "type": "basic", - "username": , - "passwordFromSecretArn": - } - }, - "reindexFromSnapshotServiceEnabled": true, - "reindexFromSnapshotExtraArgs": "", - "artifactBucketRemovalPolicy": "DESTROY" - } -} -``` -{% include copy.html %} - -Performing an RFS backfill migration requires an existing snapshot. - - -The RFS configuration uses the following options. All options are optional. - -| Name | Example | Description | -| :--- | :--- | :--- | -| `reindexFromSnapshotServiceEnabled` | `true` | Enables deployment and configuration of the RFS ECS service. | -| `reindexFromSnapshotExtraArgs` | `"--target-aws-region us-east-1 --target-aws-service-signing-name es"` | Extra arguments for the Document Migration command, with space separation. See [RFS Extra Arguments](https://github.com/opensearch-project/opensearch-migrations/blob/main/DocumentsFromSnapshotMigration/README.md#arguments) for more information. You can pass `--no-insecure` to remove the `--insecure` flag. | - -To view all available arguments for `reindexFromSnapshotExtraArgs`, see [Snapshot migrations README](https://github.com/opensearch-project/opensearch-migrations/blob/main/DocumentsFromSnapshotMigration/README.md#arguments). At a minimum, no extra arguments may be needed. - -## Live capture migration with C&R - -The following sample CDK performs a live capture migration with C&R: - -```json -{ - "live-capture-migration": { - "stage": "dev", - "vpcId": , - "sourceCluster": { - "endpoint": , - "version": "ES 7.10", - "auth": {"type": "none"} - }, - "targetCluster": { - "endpoint": , - "auth": { - "type": "basic", - "username": , - "passwordFromSecretArn": - } - }, - "captureProxyServiceEnabled": true, - "captureProxyExtraArgs": "", - "trafficReplayerServiceEnabled": true, - "trafficReplayerExtraArgs": "--speedup-factor 2.0", - "targetClusterProxyServiceEnabled": true - } -} -``` -{% include copy.html %} - -Performing a live capture migration requires that a Capture Proxy be configured to capture incoming traffic and send it to the target cluster using the Traffic Replayer service. For arguments available in `captureProxyExtraArgs`, refer to the `@Parameter` fields [here](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java). For `trafficReplayerExtraArgs`, refer to the `@Parameter` fields [here](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java). At a minimum, no extra arguments may be needed. - - -| Name | Example | Description | -| :--- | :--- | :--- | -| `captureProxyServiceEnabled` | `true` | Enables the Capture Proxy service deployment using an AWS CloudFormation stack. | -| `captureProxyExtraArgs` | `"--suppressCaptureForHeaderMatch user-agent .*elastic-java/7.17.0.*"` | Extra arguments for the Capture Proxy command, including options specified by the [Capture Proxy](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java). | -| `trafficReplayerServiceEnabled` | `true` | Enables the Traffic Replayer service deployment using a CloudFormation stack. | -| `trafficReplayerExtraArgs` | `"--sigv4-auth-header-service-region es,us-east-1 --speedup-factor 5"` | Extra arguments for the Traffic Replayer command, including options for auth headers and other parameters specified by the [Traffic Replayer](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java). | -| `targetClusterProxyServiceEnabled` | `true` | Enables the target cluster proxy service deployment using a CloudFormation stack. | - -For arguments available in `captureProxyExtraArgs`, see the `@Parameter` fields in [`CaptureProxy.java`](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java). For `trafficReplayerExtraArgs`, see the `@Parameter` fields in [`TrafficReplayer.java`](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java). - - -## Cluster authentication options - -Both the source and target cluster can use no authentication, authentication limited to VPC, basic authentication with a username and password, or AWS Signature Version 4 scoped to a user or role. - -### No authentication - -```json - "sourceCluster": { - "endpoint": , - "version": "ES 7.10", - "auth": {"type": "none"} - } -``` -{% include copy.html %} - -### Basic authentication - -```json - "sourceCluster": { - "endpoint": , - "version": "ES 7.10", - "auth": { - "type": "basic", - "username": , - "passwordFromSecretArn": - } - } -``` -{% include copy.html %} - -### AWS Signature Version 4 authentication - -```json - "sourceCluster": { - "endpoint": , - "version": "ES 7.10", - "auth": { - "type": "sigv4", - "region": "us-east-1", - "serviceSigningName": "es" - } - } -``` -{% include copy.html %} - -The `serviceSigningName` can be `es` for an Elasticsearch or OpenSearch domain. - -All of these authentication options apply to both source and target clusters. - -## Snapshot options - -The following configuration options customize the process of migrating from snapshots. - -### Snapshot of a managed service source - -If your source cluster is on Amazon OpenSearch Service, you need to set up an additional AWS Identity and Access Management (IAM) role and pass it with the snapshot creation call, as described in the [AWS documentation](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-snapshots.html). Migration Assistant can automatically manage this process. OpenSearch Service snapshots are only compatible with AWS Signature Version 4 authentication. The following parameter ensures that the additional IAM role is created and passed. - -| Name | Example | Description | -| :--- | :--- | :--- | -| `managedServiceSourceSnapshotEnabled` | `true` | Creates the necessary roles and trust relationships for taking a snapshot of an OpenSearch Service source cluster. This is only compatible with AWS Signature Version 4 authentication.| - -### Bring your own snapshot - -You can use an existing Amazon Simple Storage Service (Amazon S3) snapshot to perform [metadata]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrating-metadata/) and [backfill]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/backfill/) migrations instead of using Migration Assistant to create a snapshot: - -```json - "snapshot": { - "snapshotName": "my-snapshot-name", - "snapshotRepoName": "my-snapshot-repo", - "s3Uri": "s3://my-s3-bucket-name/my-bucket-path-to-snapshot-repo", - "s3Region": "us-east-2" - } -``` -{% include copy.html %} - -By default, Amazon S3 buckets automatically allow roles in the same AWS account (with the appropriate `s3:*` permissions) to access the S3 bucket, regardless of the bucket's AWS Region. If the external S3 bucket is in the same AWS account as the Migration Assistant deployment, no further IAM configuration is required to access the bucket. - -If you use a custom permission model with Amazon S3, any access control list (ACL) or custom bucket policy should allow the Migration Assistant task roles for RFS and the migration console to read from the S3 bucket. - -If the S3 bucket is in a separate AWS account from the Migration Assistant deployment, you need a custom bucket policy similar to the following to allow access to Migration Assistant: - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "AllowExternalAccountReadAccessToBucket", - "Effect": "Allow", - "Principal": { - "AWS": "arn:aws:iam:::root" - }, - "Action": [ - "s3:GetObject", - "s3:ListBucket", - "s3:GetBucketLocation" - ], - "Resource": [ - "arn:aws:s3:::my-s3-bucket-name", - "arn:aws:s3:::my-s3-bucket-name/*" - ] - } - ] -} -``` -{% include copy.html %} - -## Network configuration - -The migration tooling expects the source cluster, target cluster, and migration resources to exist in the same VPC. If this is not the case, manual networking setup outside of this documentation is likely required. diff --git a/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md b/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md deleted file mode 100644 index d0592e2dd24..00000000000 --- a/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md +++ /dev/null @@ -1,360 +0,0 @@ ---- -layout: default -title: Getting started with data migration -parent: Deploying Migration Assistant -nav_order: 10 -permalink: /migration-assistant/deploying-migration-assistant/getting-started-data-migration/ -redirect_from: - - /upgrade-to/snapshot-migrate/ - - /migration-assistant/getting-started-with-data-migration/ ---- - -# Getting started with data migration - -This quickstart outlines how to deploy Migration Assistant for OpenSearch and execute an existing data migration using `Reindex-from-Snapshot` (RFS). It uses AWS for illustrative purposes. However, the steps can be modified for use with other cloud providers. - -## Prerequisites and assumptions - -Before using this quickstart, make sure you fulfill the following prerequisites: - -* Verify that your migration path [is supported]({{site.url}}{{site.baseurl}}/migration-assistant/overview/is-migration-assistant-right-for-you/#supported-migration-paths). Note that we test with the exact versions specified, but you should be able to migrate data on alternative minor versions as long as the major version is supported. -* The source cluster must be deployed Amazon Simple Storage Service (Amazon S3) plugin. -* The target cluster must be deployed. -* Verify that the `CDKToolkit` stack exists and is set to `CREATE_COMPLETE`. For more information about how to bootstrap your AWS account in the required AWS Region, see [the CDKToolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html). - -The steps in this guide assume the following: - -* In this guide, a snapshot will be taken and stored in Amazon S3; the following assumptions are made about this snapshot: - * The `_source` flag is enabled on all indexes to be migrated. - * The snapshot includes the global cluster state (`include_global_state` is `true`). - * Shard sizes of up to approximately 80 GB are supported. Larger shards cannot be migrated. If this presents challenges for your migration, contact the [migration team](https://opensearch.slack.com/archives/C054JQ6UJFK). -* Migration Assistant will be installed in the same AWS Region and have access to both the source snapshot and target cluster. - ---- - -## Step 1: Install Bootstrap on an Amazon EC2 instance (~10 minutes) - -To begin your migration, use the following steps to install a `bootstrap` box on an Amazon Elastic Compute Cloud (Amazon EC2) instance. The instance uses AWS CloudFormation to create and manage the stack. - -1. Log in to the target AWS account in which you want to deploy Migration Assistant. -2. From the browser where you are logged in to your target AWS account, right-click [here](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/migration-assistant-for-amazon-opensearch-service/latest/migration-assistant-for-amazon-opensearch-service.template&redirectId=SolutionWeb) to load the CloudFormation template from a new browser tab. -3. Follow the CloudFormation stack wizard: - * **Stack Name:** `MigrationBootstrap` - * **Stage Name:** `dev` - * Choose **Next** after each step > **Acknowledge** > **Submit**. -4. Verify that the Bootstrap stack exists and is set to `CREATE_COMPLETE`. This process takes around 10 minutes to complete. - ---- - -## Step 2: Set up Bootstrap instance access (~5 minutes) - -Use the following steps to set up Bootstrap instance access: - -1. After deployment, find the EC2 instance ID for the `bootstrap-dev-instance`. -2. Create an AWS Identity and Access Management (IAM) policy using the following snippet, replacing ``, ``, ``, and `` with your information: - - ```json - { - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": "ssm:StartSession", - "Resource": [ - "arn:aws:ec2:::instance/", - "arn:aws:ssm:::document/BootstrapShellDoc--" - ] - } - ] - } - ``` - {% include copy.html %} - -3. Name the policy, for example, `SSM-OSMigrationBootstrapAccess`, and then create the policy by selecting **Create policy**. -4. Attach the newly created policy to your EC2 instance's IAM role. - ---- - -## Step 3: Log in to Bootstrap and building Migration Assistant (~15 minutes) - -Next, log in to Bootstrap and build Migration Assistant using the following steps. - -### Prerequisites - -To use these steps, make sure you fulfill the following prerequisites: - -* The AWS Command Line Interface (AWS CLI) and AWS Session Manager plugin are installed on your instance. -* The AWS credentials are configured (`aws configure`) for your instance. - -### Steps - -1. Load AWS credentials into your terminal. -2. Log in to the instance using the following command, replacing `` and `` with your instance ID and Region: - - ```bash - aws ssm start-session --document-name BootstrapShellDoc-- --target --region [--profile ] - ``` - {% include copy.html %} - -3. Once logged in, run the following command from the shell of the Bootstrap instance in the `/opensearch-migrations` directory: - - ```bash - ./initBootstrap.sh && cd deployment/cdk/opensearch-service-migration - ``` - {% include copy.html %} - -4. After a successful build, note the path for infrastructure deployment, which will be used in the next step. - ---- - -## Step 4: Configure and deploy RFS (~20 minutes) - -To deploy Migration Assistant with RFS, the following stacks must be deployed: - -These commands deploy the following stacks: - -* `Migration Assistant network` stack -* `RFS` stack -* `Migration console` stack - -Use the following steps to configure and deploy RFS, deploy Migration Assistant, and verify installation of the required stacks: - -1. Add the source and target cluster password as separate **Secrets** in [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) as an unstructured string. Be sure to copy the secret Amazon Resource Name (ARN) for use during deployment. -2. From the same shell as the Bootstrap instance, modify the `cdk.context.json` file located in the `/opensearch-migrations/deployment/cdk/opensearch-service-migration` directory and configure the following settings: - - ```json - { - "default": { - "stage": "dev", - "vpcId": "", - "targetCluster": { - "endpoint": "", - "auth": { - "type": "basic", - "username": "", - "passwordFromSecretArn": "" - } - }, - "sourceCluster": { - "endpoint": "", - "version": "", - "auth": { - "type": "basic", - "username": "", - "passwordFromSecretArn": "" - } - }, - "reindexFromSnapshotExtraArgs": "", - "reindexFromSnapshotMaxShardSizeGiB": 80, - "otelCollectorEnabled": true, - "migrationConsoleServiceEnabled": true - } - } - ``` - {% include copy.html %} - - The source and target cluster authorization can be configured to have no authorization, `basic` with a username and password, or `sigv4`. - -3. After the `cdk.context.json` file is fully configured, bootstrap the account and deploy the required stacks using the following command: - - ```bash - cdk bootstrap --c contextId=default --require-approval never - ``` - {% include copy.html %} - -4. Deploy Migration Assistant using the following command: - - ```bash - cdk deploy "*" --c contextId=default --require-approval never --concurrency 5 - ``` - {% include copy.html %} - -5. From the same Bootstrap instance shell, verify that all CloudFormation stacks were installed successfully: - - ```bash - aws cloudformation list-stacks --query "StackSummaries[?StackStatus!='DELETE_COMPLETE'].[StackName,StackStatus]" --output table - ``` - {% include copy.html %} - -You should receive a similar output for your Region: - -```bash ------------------------------------------------------------------------- -| ListStacks | -+--------------------------------------------------+-------------------+ -| OSMigrations-dev-us-east-1-MigrationConsole | CREATE_COMPLETE | -| OSMigrations-dev-us-east-1-ReindexFromSnapshot | CREATE_COMPLETE | -| OSMigrations-dev-us-east-1-MigrationInfra | CREATE_COMPLETE | -| OSMigrations-dev-us-east-1-default-NetworkInfra | CREATE_COMPLETE | -| MigrationBootstrap | CREATE_COMPLETE | -| CDKToolkit | CREATE_COMPLETE | -+--------------------------------------------------+-------------------+ -``` - -### RFS parameters - -If you're creating a snapshot using migration tooling, these parameters are automatically configured. If you're using an existing snapshot, modify the `reindexFromSnapshotExtraArgs` setting with the following values: - -```bash - "reindexFromSnapshotExtraArgs": "--s3-repo-uri s3:/// --s3-region --snapshot-name " -``` - -You will also need to give the `migrationconsole` and `reindexFromSnapshot` TaskRoles permissions to the S3 bucket. - ---- - -## Step 5: Access the migration console - -Run the following command to access the migration console: - -```bash -./accessContainer.sh migration-console dev -``` -{% include copy.html %} - - -`accessContainer.sh` is located in `/opensearch-migrations/deployment/cdk/opensearch-service-migration/` on the Bootstrap instance. To learn more, see [Accessing the migration console]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrating-metadata/). -{: .note} - ---- - -## Step 6: Verify the connection to the source and target clusters - -To verify the connection to the clusters, run the following command: - -```bash -console clusters connection-check -``` -{% include copy.html %} - -You should receive the following output: - -```bash -SOURCE CLUSTER -ConnectionResult(connection_message='Successfully connected!', connection_established=True, cluster_version='') -TARGET CLUSTER -ConnectionResult(connection_message='Successfully connected!', connection_established=True, cluster_version='') -``` - -To learn more about migration console commands, see [Migration commands]. - ---- - -## Step 7: Create a snapshot - -Run the following command to initiate snapshot creation from the source cluster: - -```bash -console snapshot create [...] -``` -{% include copy.html %} - -To check the snapshot creation status, run the following command: - -```bash -console snapshot status [...] -``` -{% include copy.html %} - -To learn more information about the snapshot, run the following command: - -```bash -console snapshot status --deep-check [...] -``` -{% include copy.html %} - -Wait for snapshot creation to complete before moving to step 9. - -To learn more about snapshot creation, see [Snapshot Creation]. - ---- - -## Step 8: Migrate metadata - -Run the following command to migrate metadata: - -```bash -console metadata migrate [...] -``` -{% include copy.html %} - -For more information, see [Migrating metadata]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrating-metadata/). - ---- - -## Step 9: Migrate documents with RFS - -You can now use RFS to migrate documents from your original cluster: - -1. To start the migration from RFS, start a `backfill` using the following command: - - ```bash - console backfill start - ``` - {% include copy.html %} - -2. _(Optional)_ To speed up the migration, increase the number of documents processed at a simultaneously by using the following command: - - ```bash - console backfill scale - ``` - {% include copy.html %} - -3. To check the status of the documentation backfill, use the following command: - - ```bash - console backfill status - ``` - {% include copy.html %} - -4. If you need to stop the backfill process, use the following command: - - ```bash - console backfill stop - ``` - {% include copy.html %} - -For more information, see [Backfill]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/backfill/). - ---- - -## Step 10: Backfill monitoring - -Use the following command for detailed monitoring of the backfill process: - -```bash -console backfill status --deep-check -``` -{% include copy.html %} - -You should receive the following output: - -```json -BackfillStatus.RUNNING -Running=9 -Pending=1 -Desired=10 -Shards total: 62 -Shards completed: 46 -Shards incomplete: 16 -Shards in progress: 11 -Shards unclaimed: 5 -``` - -Logs and metrics are available in Amazon CloudWatch in the `OpenSearchMigrations` log group. - ---- - -## Step 11: Verify that all documents were migrated - -Use the following query in CloudWatch Logs Insights to identify failed documents: - -```bash -fields @message -| filter @message like "Bulk request succeeded, but some operations failed." -| sort @timestamp desc -| limit 10000 -``` -{% include copy.html %} - -If any failed documents are identified, you can index the failed documents directly as opposed to using RFS. diff --git a/_migration-assistant/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md b/_migration-assistant/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md deleted file mode 100644 index 5ed62887838..00000000000 --- a/_migration-assistant/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md +++ /dev/null @@ -1,93 +0,0 @@ ---- -layout: default -title: IAM and security groups for existing clusters -nav_order: 20 -parent: Deploying Migration Assistant -permalink: /migration-assistant/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters/ ---- - -# IAM and security groups for existing clusters - -This page outlines security scenarios for using the migration tools with existing clusters, including any necessary configuration changes to ensure proper communication between them. - -## Importing an Amazon OpenSearch Service or Amazon OpenSearch Serverless target cluster - -Use the following scenarios for Amazon OpenSearch Service or Amazon OpenSearch Serverless target clusters. - -### OpenSearch Service - -For an OpenSearch Domain, two main configurations are typically required to ensure proper functioning of the migration solution: - -1. **Security Group Configuration** - - The domain should have a security group that allows communication from the applicable migration services (Traffic Replayer, Migration Console, `Reindex-from-Snapshot`). The CDK automatically creates an `osClusterAccessSG` security group, which is applied to the migration services. The user should then add this security group to their existing domain to allow access. - -2. **Access Policy Configuration** should be one of the following: - - An open access policy that allows all access. - - Configured to allow at least the AWS Identity and Access Management (IAM) task roles for the applicable migration services (Traffic Replayer, Migration Console, `Reindex-from-Snapshot`) to access the domain. - -### Managed service role mapping (Cross-managed-cluster migrations) - -When migrating between two managed clusters, for example, when both domains were created using Amazon OpenSearch Service, provide Migration Assistant components with sufficient permissions to modify both the source and target clusters. - -Use the following steps to grant the required permissions: - -1. In the AWS Management Console, navigate to **CloudFormation** > **Stacks**. -2. Locate the stack that starts with `OSMigrations--` (created during CDK deployment). -3. Go to the **Resources** tab and locate the following IAM roles: - - ```bash - arn:aws:iam::****:role/OSMigrations---MigrationServiceTaskRoleC- - arn:aws:iam::****:role/OSMigrations---reindexfromsnapshotTaskRo- - arn:aws:iam::****:role/OSMigrations---trafficreplayerdefaultTas- - ``` - -4. In both the source and target clusters, map users to each Amazon Resource Name (ARN) using the following steps: - A. Access OpenSearch Dashboards. If you're using Elasticsearch, access Kibana. - B. Navigate to **Security -> Roles -> all_access**. - C. In the "Mapped users" section, add each ARN as a backend role. - D. Save your changes. - -### OpenSearch Serverless - -For an OpenSearch Serverless Collection, you will need to configure both network and data access policies: - -1. **Network Policy Configuration**: - The Collection should have a network policy that uses the `VPC` access type. This requires creating a VPC endpoint on the VPC used for the solution. The VPC endpoint should be configured for the private subnets of the VPC and should attach the `osClusterAccessSG` security group. - -2. **Data Access Policy Configuration**: - The data access policy should grant permission to perform all [index operations](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-data-access.html#serverless-data-supported-permissions) (`aoss:*`) for all indexes in the Collection. The IAM task roles of the applicable Migration services (Traffic Replayer, migration console, `Reindex-from-Snapshot`) should be used as the principals for this data access policy. - -## Capture Proxy on Coordinator Nodes of Source Cluster - -Although the CDK does not automatically set up the Capture Proxy on source cluster nodes (except in the demo solution), the Capture Proxy instances must communicate with the resources deployed by the CDK, such as Kafka. This section outlines the necessary steps to set up communication. - -Before [setting up Capture Proxy instances](https://github.com/opensearch-project/opensearch-migrations/tree/main/TrafficCapture/trafficCaptureProxyServer#installing-capture-proxy-on-coordinator-nodes) on the source cluster, ensure the following configurations are in place: - -1. **Security Group Configuration**: - The coordinator nodes should add the `trafficStreamSourceSG` security group to allow sending captured traffic to Kafka. - -2. **IAM Policy Configuration**: - The IAM role used by the coordinator nodes should have permissions to publish captured traffic to Kafka. You can add the following template policy through the AWS Console (IAM Role → Add permissions → Create inline policy → JSON view): - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Action": "kafka-cluster:Connect", - "Resource": "arn:aws:kafka:::cluster/migration-msk-cluster-/*", - "Effect": "Allow" - }, - { - "Action": [ - "kafka-cluster:CreateTopic", - "kafka-cluster:DescribeTopic", - "kafka-cluster:WriteData" - ], - "Resource": "arn:aws:kafka:::topic/migration-msk-cluster-/*", - "Effect": "Allow" - } - ] -} -``` diff --git a/_migration-assistant/deploying-migration-assistant/index.md b/_migration-assistant/deploying-migration-assistant/index.md deleted file mode 100644 index 1c559a81b1c..00000000000 --- a/_migration-assistant/deploying-migration-assistant/index.md +++ /dev/null @@ -1,13 +0,0 @@ ---- -layout: default -title: Deploying Migration Assistant -nav_order: 15 -has_children: true -permalink: /deploying-migration-assistant/ -redirect-from: - - /deploying-migration-assistant/index/ ---- - -# Deploying Migration Assistant - -This section provides information about the available options for deploying Migration Assistant. diff --git a/_migration-assistant/index.md b/_migration-assistant/index.md index cf3a006ce32..78d8b743ddd 100644 --- a/_migration-assistant/index.md +++ b/_migration-assistant/index.md @@ -8,10 +8,6 @@ has_toc: false permalink: /migration-assistant/ redirect_from: - /migration-assistant/index/ - - /migration-assistant/overview/ - - /upgrade-to/index/ - - /upgrade-to/ - - /upgrade-to/upgrade-to/ - /upgrade-to/snapshot-migrate/ items: - heading: "Is Migration Assistant right for you?" diff --git a/_migration-assistant/migration-console/migration-console-commands-references.md b/_migration-assistant/migration-console/migration-console-commands-references.md index a07439e32b8..c1aa7c37ad1 100644 --- a/_migration-assistant/migration-console/migration-console-commands-references.md +++ b/_migration-assistant/migration-console/migration-console-commands-references.md @@ -3,7 +3,7 @@ layout: default title: Command reference nav_order: 40 parent: Migration console -permalink: /migration-assistant/migration-console/migration-console-commands-reference/ +permalink: /migration-assistant/migration-console/migration-console-command-reference/ --- # Migration console command reference diff --git a/_migration-assistant/migration-phases/accessing-the-migration-console.md b/_migration-assistant/migration-phases/accessing-the-migration-console.md index fc86d61ca2b..44dc350af10 100644 --- a/_migration-assistant/migration-phases/accessing-the-migration-console.md +++ b/_migration-assistant/migration-phases/accessing-the-migration-console.md @@ -1,10 +1,10 @@ --- layout: default title: Accessing the Migration Console -parent: Deploying Migration Assistant +parent: Migration phases nav_order: 10 -redirect_from: --- + # Accessing the Migration Console The Migrations Assistant deployment includes an ECS task that hosts tools to run different phases of the migration and check the progress or results of the migration. diff --git a/_migration-assistant/migration-phases/assessment.md b/_migration-assistant/migration-phases/assessment.md index d5210674e66..2d2ee5f872b 100644 --- a/_migration-assistant/migration-phases/assessment.md +++ b/_migration-assistant/migration-phases/assessment.md @@ -1,10 +1,10 @@ --- layout: default -title: Assess +title: Assessing your cluster for migration nav_order: 1 parent: Migration phases redirect_from: - - /migration-assistant/migration-phases/assessing-your-cluster-for-migration/ + - /migration-assistant/migration-phases/planning-your-migration/assessing-your-cluster-for-migration/ --- # Assessing your cluster for migration diff --git a/_migration-assistant/migration-phases/backfill.md b/_migration-assistant/migration-phases/backfill.md index f2d879c0f7a..da1df4282c5 100644 --- a/_migration-assistant/migration-phases/backfill.md +++ b/_migration-assistant/migration-phases/backfill.md @@ -1,12 +1,12 @@ --- layout: default -title: Migrate Data +title: Using backfill nav_order: 6 parent: Migration phases permalink: /migration-assistant/migration-phases/backfill/ --- -# Backfill +# Using backfill After the [metadata]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrating-metadata/) for your cluster has been migrated, you can use capture proxy data replication and snapshots to backfill your data into the next cluster. diff --git a/_migration-assistant/migration-phases/create-snapshot.md b/_migration-assistant/migration-phases/create-snapshot.md index 6b24ae78b6b..3e8c5ddcf1c 100644 --- a/_migration-assistant/migration-phases/create-snapshot.md +++ b/_migration-assistant/migration-phases/create-snapshot.md @@ -5,9 +5,9 @@ nav_order: 4 parent: Migration phases --- -## Creating snapshot +# Creating a snapshot -TODO: Add Bring your own snapshot +<----!TODO: Add Bring your own snapshot-----!> Creating a snapshot of the source cluster captures all the metadata and documents to be migrated to a new target cluster. @@ -41,11 +41,11 @@ Anticipated duration remaining: 0h 0m 0s Throughput: 38.13 MiB/sec ``` -### Managing slow snapshot speeds +## Managing slow snapshot speeds Depending on the size of the data in the source cluster and the bandwidth allocated for snapshots, the process can take some time. Adjust the maximum rate at which the source cluster's nodes create the snapshot using the `--max-snapshot-rate-mb-per-node` option. Increasing the snapshot rate will consume more node resources, which may affect the cluster's ability to handle normal traffic. -## Command arguments +### Command arguments For the following commands, to identify all valid arguments, please run with `--help`. @@ -121,7 +121,7 @@ Results: ``` -## Using the migrate command +## Using the `migrate` command Running through the same data as the evaluate command all of the migrated items will be applied onto the target cluster. If re-run multiple times items that were previously migrated will not be recreated. If any items do need to be re-migrated, please delete them from the target cluster and then rerun the evaluate then migrate commands to ensure the desired changes are made. @@ -241,4 +241,4 @@ As Metadata migration supports migrating from ES 6.8 on to the latest versions o For additional technical details, [view the mapping type removal source code](https://github.com/opensearch-project/opensearch-migrations/blob/main/transformation/src/main/java/org/opensearch/migrations/transformation/rules/IndexMappingTypeRemoval.java). TODO: Add troublshooting -
  • Verify Backfill Components
    • \ No newline at end of file +
  • Verify Backfill Components
    • \ No newline at end of file diff --git a/_migration-assistant/migration-phases/deploying-migration-assistant/configuration-options.md b/_migration-assistant/migration-phases/deploying-migration-assistant/configuration-options.md index 5032dcb757e..dccea3a74d2 100644 --- a/_migration-assistant/migration-phases/deploying-migration-assistant/configuration-options.md +++ b/_migration-assistant/migration-phases/deploying-migration-assistant/configuration-options.md @@ -2,7 +2,7 @@ layout: default title: Configuration options nav_order: 15 -parent: Deploying Migration Assistant +nav_exclude: true permalink: /migration-assistant/deploying-migration-assistant/configuration-options/ --- diff --git a/_migration-assistant/migration-phases/deploying-migration-assistant/getting-started-data-migration.md b/_migration-assistant/migration-phases/deploying-migration-assistant/getting-started-data-migration.md index b83d10eb495..009003d9f7c 100644 --- a/_migration-assistant/migration-phases/deploying-migration-assistant/getting-started-data-migration.md +++ b/_migration-assistant/migration-phases/deploying-migration-assistant/getting-started-data-migration.md @@ -1,7 +1,7 @@ --- layout: default title: Getting started with data migration -parent: Deploying Migration Assistant +nav_exclude: true nav_order: 10 permalink: /migration-assistant/deploying-migration-assistant/getting-started-data-migration/ redirect_from: diff --git a/_migration-assistant/migration-phases/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md b/_migration-assistant/migration-phases/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md index 5ed62887838..ee1442fd74c 100644 --- a/_migration-assistant/migration-phases/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md +++ b/_migration-assistant/migration-phases/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md @@ -2,7 +2,7 @@ layout: default title: IAM and security groups for existing clusters nav_order: 20 -parent: Deploying Migration Assistant +nav_exclude: true permalink: /migration-assistant/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters/ --- diff --git a/_migration-assistant/migration-phases/deployment.md b/_migration-assistant/migration-phases/deployment.md index 40e3e6244ee..8fa097afd6a 100644 --- a/_migration-assistant/migration-phases/deployment.md +++ b/_migration-assistant/migration-phases/deployment.md @@ -1,6 +1,6 @@ --- layout: default -title: Deploy +title: Deploying Migration Assistant parent: Migration phases nav_order: 2 redirect_from: @@ -9,6 +9,7 @@ redirect_from: --- # Deploying Migration Assistant + This document assumes you have performed assessment. --- diff --git a/_migration-assistant/migration-phases/index.md b/_migration-assistant/migration-phases/index.md index c8b76a34dda..6c3a8a8d688 100644 --- a/_migration-assistant/migration-phases/index.md +++ b/_migration-assistant/migration-phases/index.md @@ -6,7 +6,6 @@ nav_order: 30 has_children: true has_toc: false permalink: /migration-assistant/migration-phases/ -redirect-from: /migration-assistant/migration-phases/index/ --- # Migration phases @@ -41,9 +40,9 @@ details[open] {
  • Deployment
  • Create Snapshot
    • -
  • Migrate Metadata
    • +
  • Migrate Metadata
  • Migrate Data
    • -
  • Teardown
    • +
  • Teardown
    • @@ -54,10 +53,10 @@ details[open] {
    1. Assessment
    2. Deployment
      3. -
    3. Verify Backfill Components
      4. +
    4. Verify Backfill Components
    5. Reroute Traffic from Source to Capture Proxy
      6. -
    6. Migrate Metadata
      7. -
    7. Verify Live Capture Components
      8. +
    8. Migrate Metadata
      9. +
    9. Verify Live Capture Components
    10. Replay Captured Traffic
    11. Reroute Traffic from Capture Proxy to Target
    12. Teardown
      13. diff --git a/_migration-assistant/migration-phases/live-traffic-migration/index.md b/_migration-assistant/migration-phases/live-traffic-migration/index.md index 7cdf7c466d1..255df68528d 100644 --- a/_migration-assistant/migration-phases/live-traffic-migration/index.md +++ b/_migration-assistant/migration-phases/live-traffic-migration/index.md @@ -2,7 +2,7 @@ layout: default title: Live traffic migration nav_order: 99 -parent: Migration phases +nav_exclude: true permalink: /live-traffic-migration/ has_toc: false has_children: true diff --git a/_migration-assistant/migration-phases/live-traffic-migration/switching-traffic-from-the-source-cluster.md b/_migration-assistant/migration-phases/live-traffic-migration/switching-traffic-from-the-source-cluster.md index 8ec7f1d929b..b5e2894c0f2 100644 --- a/_migration-assistant/migration-phases/live-traffic-migration/switching-traffic-from-the-source-cluster.md +++ b/_migration-assistant/migration-phases/live-traffic-migration/switching-traffic-from-the-source-cluster.md @@ -2,8 +2,7 @@ layout: default title: Switching traffic from the source cluster nav_order: 110 -grand_parent: Migration phases -parent: Live traffic migration +nav_exclude: true permalink: /migration-assistant/migration-phases/live-traffic-migration/switching-traffic-from-the-source-cluster/ redirect_from: - /migration-assistant/migration-phases/switching-traffic-from-the-source-cluster/ diff --git a/_migration-assistant/migration-phases/migrate-metadata.md b/_migration-assistant/migration-phases/migrate-metadata.md deleted file mode 100644 index 39c1d7d1149..00000000000 --- a/_migration-assistant/migration-phases/migrate-metadata.md +++ /dev/null @@ -1,249 +0,0 @@ ---- -layout: default -title: Migrate Metadata -nav_order: 5 -parent: Migration phases -redirect_from: - - /migration-assistant/migration-phases/migrating-metadata ---- - -# Migrate metadata - -Metadata migration involves creating a snapshot of your cluster and then migrating the metadata from the snapshot using the migration console. - -This tool gathers information from a source cluster through a snapshot or through HTTP requests against the source cluster. These snapshots are fully compatible with the backfill process for `Reindex-From-Snapshot` (RFS) scenarios. - -After collecting information on the source cluster, comparisons are made against the target cluster. If running a migration, any metadata items that do not already exist will be created on the target cluster. - -## Creating the snapshot - -Creating a snapshot of the source cluster captures all the metadata and documents to be migrated to a new target cluster. - -Create the initial snapshot of the source cluster using the following command: - -```shell -console snapshot create -``` -{% include copy.html %} - -To check the progress of the snapshot in real time, use the following command: - -```shell -console snapshot status --deep-check -``` -{% include copy.html %} - -You should receive the following response when the snapshot is created: - -```shell -SUCCESS -Snapshot is SUCCESS. -Percent completed: 100.00% -Data GiB done: 29.211/29.211 -Total shards: 40 -Successful shards: 40 -Failed shards: 0 -Start time: 2024-07-22 18:21:42 -Duration: 0h 13m 4s -Anticipated duration remaining: 0h 0m 0s -Throughput: 38.13 MiB/sec -``` - -### Managing slow snapshot speeds - -Depending on the size of the data in the source cluster and the bandwidth allocated for snapshots, the process can take some time. Adjust the maximum rate at which the source cluster's nodes create the snapshot using the `--max-snapshot-rate-mb-per-node` option. Increasing the snapshot rate will consume more node resources, which may affect the cluster's ability to handle normal traffic. - -## Command arguments - -For the following commands, to identify all valid arguments, please run with `--help`. - -```shell -console metadata evaluate --help -``` -{% include copy.html %} - -```shell -console metadata migrate --help -``` -{% include copy.html %} - -Based on the migration console deployment options, a number of commands will be pre-populated. To view them, run console with verbosity: - -```shell -console -v metadata migrate --help -``` -{% include copy.html %} - -You should receive a response similar to the following: - -```shell -(.venv) bash-5.2# console -v metadata migrate --help -INFO:console_link.cli:Logging set to INFO -. -. -. -INFO:console_link.models.metadata:Migrating metadata with command: /root/metadataMigration/bin/MetadataMigration --otel-collector-endpoint http://otel-collector:4317 migrate --snapshot-name snapshot_2023_01_01 --target-host https://opensearchtarget:9200 --min-replicas 0 --file-system-repo-path /snapshot/test-console --target-username admin --target-password ******** --target-insecure --help -. -. -. -``` - - -## Using the `evaluate` command - -By scanning the contents of the source cluster, applying filtering, and applying modifications a list of all items that will be migrated will be created. Any items not seen in this output will not be migrated onto the target cluster if the migrate command was to be run. This is a safety check before making modifications on the target cluster. - -```shell -console metadata evaluate [...] -``` -{% include copy.html %} - -You should receive a response similar to the following: - -```bash -Starting Metadata Evaluation -Clusters: - Source: - Remote Cluster: OpenSearch 1.3.16 ConnectionContext(uri=http://localhost:33039, protocol=HTTP, insecure=false, compressionSupported=false) - - Target: - Remote Cluster: OpenSearch 2.14.0 ConnectionContext(uri=http://localhost:33037, protocol=HTTP, insecure=false, compressionSupported=false) - - -Migration Candidates: - Index Templates: - simple_index_template - - Component Templates: - simple_component_template - - Indexes: - blog_2023, movies_2023 - - Aliases: - alias1, movies-alias - - -Results: - 0 issue(s) detected -``` - - -## Using the migrate command - -Running through the same data as the evaluate command all of the migrated items will be applied onto the target cluster. If re-run multiple times items that were previously migrated will not be recreated. If any items do need to be re-migrated, please delete them from the target cluster and then rerun the evaluate then migrate commands to ensure the desired changes are made. - -```shell -console metadata migrate [...] -``` -{% include copy.html %} - -You should receive a response similar to the following: - -```shell -Starting Metadata Migration - -Clusters: - Source: - Snapshot: OpenSearch 1.3.16 FileSystemRepo(repoRootDir=/tmp/junit10626813752669559861) - - Target: - Remote Cluster: OpenSearch 2.14.0 ConnectionContext(uri=http://localhost:33042, protocol=HTTP, insecure=false, compressionSupported=false) - - -Migrated Items: - Index Templates: - simple_index_template - - Component Templates: - simple_component_template - - Indexes: - blog_2023, movies_2023 - - Aliases: - alias1, movies-alias - - -Results: - 0 issue(s) detected -``` - - -## Metadata verification process - -Before moving on to additional migration steps, it is recommended to confirm details of your cluster. Depending on your configuration, this could be checking the sharding strategy or making sure index mappings are correctly defined by ingesting a test document. - -## Troubleshooting - -Use these instructions to help troubleshoot the following issues. - -### Accessing detailed logs - -Metadata migration creates a detailed log file that includes low level tracing information for troubleshooting. For each execution of the program a log file is created inside a shared volume on the migration console named `shared-logs-output` the following command will list all log files, one for each run of the command. - -```shell -ls -al /shared-logs-output/migration-console-default/*/metadata/ -``` -{% include copy.html %} - -To inspect the file within the console `cat`, `tail` and `grep` commands line tools. By looking for warnings, errors and exceptions in this log file can help understand the source of failures, or at the very least be useful for creating issues in this project. - -```shell -tail /shared-logs-output/migration-console-default/*/metadata/*.log -``` -{% include copy.html %} - -### Warnings and errors - -When encountering `WARN` or `ERROR` elements in the response, they will be accompanied by a short message, such as `WARN - my_index already exists`. More information can be found in the detailed logs associated with the warning or error. - -### OpenSearch running in compatibility mode - -There might be an error about being unable to update an ES 7.10.2 cluster, this can occur when compatibility mode has been enabled on an OpenSearch cluster disable it to continue, see [Enable compatibility mode](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/rename.html#rename-upgrade). - - -### Breaking change compatibility - -Metadata migration requires modifying data from the source to the target versions to recreate items. Sometimes these features are no longer supported and have been removed from the target version. Sometimes these features are not available in the target version, which is especially true when downgrading. While this tool is meant to make this process easier, it is not exhaustive in its support. When encountering a compatibility issue or an important feature gap for your migration, [search the issues and comment on the existing issue](https://github.com/opensearch-project/opensearch-migrations/issues) or [create a new](https://github.com/opensearch-project/opensearch-migrations/issues/new/choose) issue if one cannot be found. - -#### Deprecation of Mapping Types - -In Elasticsearch 6.8 the mapping types feature was discontinued in Elasticsearch 7.0+ which has created complexity in migrating to newer versions of Elasticsearch and OpenSearch, [learn more](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/removal-of-types.html) ↗. - -As Metadata migration supports migrating from ES 6.8 on to the latest versions of OpenSearch this scenario is handled by removing the type mapping types and restructuring the template or index properties. Note that, at the time of this writing multiple type mappings are not supported, [tracking task](https://opensearch.atlassian.net/browse/MIGRATIONS-1778) ↗. - - -**Example starting state with mapping type foo (ES 6):** - -```json -{ - "mappings": [ - { - "foo": { - "properties": { - "field1": { "type": "text" }, - "field2": { "type": "keyword" } - } - } - } - ] -} -``` -{% include copy.html %} - -**Example ending state with foo removed (ES 7):** - -```json -{ - "mappings": { - "properties": { - "field1": { "type": "text" }, - "field2": { "type": "keyword" }, - } - } -} -``` -{% include copy.html %} - -For additional technical details, [view the mapping type removal source code](https://github.com/opensearch-project/opensearch-migrations/blob/main/transformation/src/main/java/org/opensearch/migrations/transformation/rules/IndexMappingTypeRemoval.java). diff --git a/_migration-assistant/migration-phases/migrating-metadata.md b/_migration-assistant/migration-phases/migrating-metadata.md index 71e3da9776d..3c9bf8f3c45 100644 --- a/_migration-assistant/migration-phases/migrating-metadata.md +++ b/_migration-assistant/migration-phases/migrating-metadata.md @@ -1,12 +1,13 @@ --- layout: default -title: Migrating metadata -nav_order: 85 +title: Migrate Metadata +nav_order: 5 parent: Migration phases -permalink: /migration-assistant/migration-phases/migrating-metadata/ +redirect_from: + - /migration-assistant/migration-phases/migrating-metadata --- -# Migrating metadata +# Migrate metadata Metadata migration involves creating a snapshot of your cluster and then migrating the metadata from the snapshot using the migration console. @@ -128,7 +129,7 @@ Results: ``` -## Using the migrate command +## Using the `migrate` command Running through the same data as the evaluate command all of the migrated items will be applied onto the target cluster. If re-run multiple times items that were previously migrated will not be recreated. If any items do need to be re-migrated, please delete them from the target cluster and then rerun the evaluate then migrate commands to ensure the desired changes are made. diff --git a/_migration-assistant/migration-phases/planning-your-migration/assessing-your-cluster-for-migration.md b/_migration-assistant/migration-phases/planning-your-migration/assessing-your-cluster-for-migration.md deleted file mode 100644 index a86f9f5db5e..00000000000 --- a/_migration-assistant/migration-phases/planning-your-migration/assessing-your-cluster-for-migration.md +++ /dev/null @@ -1,77 +0,0 @@ ---- -layout: default -title: Assessing your cluster for migration -nav_order: 60 -parent: Planning your migration -grand_parent: Migration phases -permalink: /migration-assistant/migration-phases/planning-your-migration/assessing-your-cluster-for-migration/ -redirect_from: - - /migration-assistant/migration-phases/assessing-your-cluster-for-migration/ ---- - -# Assessing your cluster for migration - -The goal of the Migration Assistant is to streamline the process of migrating from one location or version of Elasticsearch/OpenSearch to another. However, completing a migration sometimes requires resolving client compatibility issues before they can communicate directly with the target cluster. - -## Understanding breaking changes - -Before performing any upgrade or migration, you should review any breaking changes that might exist between version because there may be changes required in order for clients to connect to the new cluster. Use the following tool by selecting the versions in your migration path. The tool will respond with any breaking changes you should note when migrating: - - - -
      -

      Find a list of breaking changes for your migration path

      - -
      - - - - - -
      - -
      -
      - - -
      - -
      -
      - - - - - -## Impact of data transformations - -Any time you apply a transformation to your data, such as: - -- Changing index names -- Modifying field names or field mappings -- Splitting indices with type mappings - -These changes might need to be reflected in your client configurations. For example, if your clients are reliant on specific index or field names, you must ensure that their queries are updated accordingly. - -We recommend running production-like queries against the target cluster before switching over actual production traffic. This helps verify that the client can: - -- Communicate with the target cluster -- Locate the necessary indices and fields -- Retrieve the expected results - -For complex migrations involving multiple transformations or breaking changes, we highly recommend performing a trial migration with representative, non-production data (e.g., in a staging environment) to fully test client compatibility with the target cluster. - -## Supported transformations - -The following [transformations]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/live-traffic-migration/using-traffic-replayer/#transformations) are included in Migration Assistant. They can be enabled, combined, and configured to customize a migration for your use case. To request additional Migration Assistant transformations , create a GitHub issue [in the OpenSearch migrations repository](https://github.com/opensearch-project/opensearch-migrations/issues). - -- [Type mapping deprecation]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation/) diff --git a/_migration-assistant/migration-phases/planning-your-migration/handling-field-type-breaking-changes.md b/_migration-assistant/migration-phases/planning-your-migration/handling-field-type-breaking-changes.md deleted file mode 100644 index e09da59fd00..00000000000 --- a/_migration-assistant/migration-phases/planning-your-migration/handling-field-type-breaking-changes.md +++ /dev/null @@ -1,159 +0,0 @@ ---- -layout: default -title: Handling breaking changes in field types -nav_order: 60 -parent: Planning your migration -permalink: /migration-assistant/migration-phases/planning-your-migration/handling-breaking-changes-in-field-types/ -grand_parent: Migration phases ---- - -# Handling breaking changes in field types - -This guide explains how to use Migration Assistant to transform field types that are deprecated or incompatible during a migration to OpenSearch. - -Field types define how data is stored and queried in an index. Each field in a document is mapped to a data type, which determines how it is indexed and what operations can be performed on it. - -For example, the following index mapping for a library's book collection defines three fields, each with a different type: - -```json -GET /library-books/_mappings -{ - "library-books": { - "mappings": { - "properties": { - "title": { "type": "text" }, - "publishedDate": { "type": "date" }, - "pageCount": { "type": "integer" } - } - } - } -} -``` - -For more information, see [Mappings and field types]({{site.url}}{{site.baseurl}}/field-types/). - -## Configure item transformations - -You can customize how field types are transformed during metadata and data migrations by supplying a transformation configuration file using the following steps: - -1. Open the Migration Assistant console. -2. Create a JavaScript file to define your transformation logic using the following command: - - ```bash - vim /shared-logs-output/field-type-converter.js - ``` - {% include copy.html %} - -3. Write any JavaScript rules that perform the desired field type conversions. For an example of how the rules can be implemented, see the [example `field-type-converter.js` implementation](#example-field-type-converterjs-implementation). -4. Create a transformation descriptor file using the following command: - - ```bash - vim /shared-logs-output/transformation.json - ``` - {% include copy.html %} - -5. Add a reference to your JavaScript file in `transformation.json`. -6. Run the metadata migration and supply the transformation configuration using a command similar to the following: - - ```bash - console metadata migrate \ - --transformer-config-file /shared-logs-output/transformation.json - ``` - {% include copy.html %} - -### Example `field-type-converter.js` implementation - -The following script demonstrates how to perform common field type conversions, including: - -* Replacing the deprecated `string` type with `text`. -* Converting `flattened` to `flat_object` and removing the `index` property if present. - -```javascript -function main(context) { - const rules = [ - { - when: { type: "string" }, - set: { type: "text" } - }, - { - when: { type: "flattened" }, - set: { type: "flat_object" }, - remove: ["index"] - } - ]; - - function applyRules(node, rules) { - if (Array.isArray(node)) { - node.forEach((child) => applyRules(child, rules)); - } else if (node instanceof Map) { - for (const { when, set, remove = [] } of rules) { - const matches = Object.entries(when).every(([k, v]) => node.get(k) === v); - if (matches) { - Object.entries(set).every(([k, v]) => node.set(k, v)); - remove.forEach((key) => node.delete(key)); - } - } - for (const child of node.values()) { - applyRules(child, rules); - } - } else if (node && typeof node === "object") { - for (const { when, set, remove = [] } of rules) { - const matches = Object.entries(when).every(([k, v]) => node[k] === v); - if (matches) { - Object.assign(node, set); - remove.forEach((key) => delete node[key]); - } - } - Object.values(node).forEach((child) => applyRules(child, rules)); - } - } - - return (doc) => { - if (doc && doc.type && doc.name && doc.body) { - applyRules(doc, rules); - } - return doc; - }; -} -(() => main)(); -``` -{% include copy.html %} - -The script contains the following elements: - -1. The `rules` array defines transformation logic: - - * `when`: Key-value conditions to match on a node - * `set`: Key-value pairs to apply when the `when` clause matches - * `remove` (optional): Keys to delete from the node when matched - -2. The `applyRules` function recursively traverses the input: - - * Arrays are recursively processed element by element. - * `Map` objects are matched and mutated using the defined rules. - * Plain objects are checked for matches and transformed accordingly. - -3. The `main` function returns a transformation function that: - - * Applies the rules to each document. - * Returns the modified document for migration or replay. - -### Example `transformation.json` - -The following JSON file references your transformation script and initializes the JavaScript engine with your custom rules: - -```json -[ - { - "JsonJSTransformerProvider": { - "initializationScriptFile": "/shared-logs-output/field-type-converter.js", - "bindingsObject": "{}" - } - } -] -``` -{% include copy.html %} - -## Summary - -By using a transformation configuration, you can rewrite deprecated or incompatible field types during metadata migration or data replay. This ensures that your target OpenSearch cluster only receives compatible mappings—even if the source cluster includes outdated types like `string` or features like `flattened` that need conversion. diff --git a/_migration-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md b/_migration-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md deleted file mode 100644 index a39ac1fdda9..00000000000 --- a/_migration-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md +++ /dev/null @@ -1,319 +0,0 @@ ---- -layout: default -title: Managing type mapping deprecation -nav_order: 60 -parent: Planning your migration -grand_parent: Migration phases -permalink: /migration-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation/ ---- - -# Managing type mapping deprecation - -This guide provides solutions for managing the deprecation of the type mapping functionality when migrating from Elasticsearch 6.x or earlier to OpenSearch. - -In versions of Elasticsearch prior to 6.x, an index could contain multiple types, each with its own mapping. These types allowed you to store and query different kinds of documents—such as books and movies—in a single index. For example, both `book` and `movie` types could have a shared field like `title`, while each had additional fields specific to that type. - -Newer versions of Elasticsearch and OpenSearch no longer support multiple mapping types. Each index now supports only a single mapping type. During migration, you must define how to transform or restructure data that used multiple types. The following example shows multiple mapping types: - - -```JSON -GET /library/_mappings -{ - "library": { - "mappings": { - "book": { - "properties": { - "title": { "type": "text" }, - "pageCount": { "type": "integer" } - } - }, - "movie": { - "properties": { - "title": { "type": "text" }, - "runTime": { "type": "integer" } - } - } - } - } -} -``` - -For more information, see the [official Elasticsearch documentation on the removal of mapping types](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/removal-of-types.html). - -## Using the type mapping transformer - -To address type mapping deprecation, use the `TypeMappingsSanitizationTransformer`. This transformer can modify data, including metadata, documents, and requests, so that the previously mapped data can be used in OpenSearch. To use the mapping transformer: - -1. Navigate to the bootstrap box and open the `cdk.context.json` file with Vim. -2. Add or update the key `reindexFromSnapshotExtraArgs` to include `--doc-transformer-config-file /shared-logs-output/transformation.json`. -3. Add or update the key `trafficReplayerExtraArgs` to include `--transformer-config-file /shared-logs-output/transformation.json`. -4. Deploy Migration Assistant. -5. Navigate to the Migration Assistant console. -6. Create a file named `/shared-logs-output/transformation.json`. -7. Add your transformation configuration to the file. For configuration options, see [Configuration options](#configuration-options). -8. When running the metadata migration, run the configuration with the transformer using the command `console metadata migrate --transformer-config-file /shared-logs-output/transformation.json`. - -Whenever the transformation configuration is updated, the backfill and replayer tools need to be stopped and restarted in order to apply the changes. Any previously migrated data and metadata may need to be cleared in order to avoid an inconsistent state. - -### Configuration options - -The `TypeMappingsSanitizationTransformer` supports several strategies for managing type mappings: - -1. **Route different types to separate indexes**: Split different types into their own indexes. -2. **Merge all types into one index**: Combine multiple types into a single index. -3. **Drop specific types**: Selectively migrate only specific types. -4. **Keep the original structure**: Maintain the same index name while conforming to the new type standards. - -### Type mapping transformer configuration schema - -The type mapping transformer uses the following configuration options. - -| **Field** | **Type** | **Required** | **Description** | -| :--- | :--- | :--- | :--- | -| `staticMappings` | `object` | No | A map of `{ indexName: { typeName: targetIndex } }` used to **statically** route specific types.

      For any **index** listed on this page, types **not** included in its object are **dropped** (no data or requests are migrated for those omitted types). | -| `regexMappings` | `array` | No | A list of **regex-based** rules for **dynamic** routing of source index/type names to a target index.

      Each element in this array is itself an object with `sourceIndexPattern`, `sourceTypePattern`, and `targetIndexPattern` fields.

      For information about the **default value**, see [Defaults](#Defaults). | -| `sourceProperties` | `object` | Yes | Additional **metadata** about the source (for example, its Elasticsearch/OpenSearch version). Must include at least `"version"` with `"major"` and `"minor"` fields. | - -The following example JSON configuration provides a transformation schema: - -
      -Example JSON Configuration - -```JSON -{ - "TypeMappingSanitizationTransformerProvider": { - "staticMappings": { - "{index-name-1}": { - "{type-name-1}": "{target-index-name-1}", - "{type-name-2}": "{target-index-name-2}" - } - }, - "regexMappings": [ - { - "sourceIndexPattern": "{source-index-pattern}", - "sourceTypePattern": "{source-type-pattern}", - "targetIndexPattern": "{target-index-pattern}" - } - ], - "sourceProperties": { - "version": { - "major": "NUMBER", - "minor": "NUMBER" - } - } - } -} -``` -{% include copy.html %} - -
      - -## Example configurations - -The following example configurations show you how to use the transformer for different mapping type scenarios. - -### Route different types to separate indexes - -If you have an index `activity` with types `user` and `post` that you want to split into separate indexes, use the following configuration: - -```json -[ - { - "TypeMappingSanitizationTransformerProvider": { - "staticMappings": { - "activity": { - "user": "new_users", - "post": "new_posts" - } - }, - "sourceProperties": { - "version": { - "major": 6, - "minor": 8 - } - } - } - } -] -{% include copy.html %} -``` - -This transformer will perform the following: - -- Route documents with type `user` to the `new_users` index. -- Route documents with type `post` to the `new_posts` index. - -### Merge all types into one index - -To merge all types into one index, use the following configuration: - -```json -[ - { - "TypeMappingSanitizationTransformerProvider": { - "staticMappings": { - "activity": { - "user": "activity", - "post": "activity" - } - }, - "sourceProperties": { - "version": { - "major": 6, - "minor": 8 - } - } - } - } -] -``` -{% include copy.html %} - -### Drop specific types - -To migrate only the `user` type within the `activity` index and drop all documents/requests with types not directly specified, use the following configuration: - -```json -[ - { - "TypeMappingSanitizationTransformerProvider": { - "staticMappings": { - "activity": { - "user": "users_only" - } - }, - "sourceProperties": { - "version": { - "major": 6, - "minor": 8 - } - } - } - } -] -``` -{% include copy.html %} - -This configuration only migrates documents of type `user` and ignores other document types in the `activity` index. - -### Keep the original structure - -To migrate only specific types and keep the original structure, use the following configuration: - -```JSON -[ - { - "TypeMappingSanitizationTransformerProvider": { - "regexMappings": [ - { - "sourceIndexPattern": "(.*)", - "sourceTypePattern": ".*", - "targetIndexPattern": "$1" - } - ], - "sourceProperties": { - "version": { - "major": 6, - "minor": 8 - } - } - } - } -] -``` -{% include copy.html %} - -This is equivalent to the strategy of merging all types into one index but also uses a pattern-based routing strategy. - -### Combining multiple strategies - -You can combine both static and regex-based mappings to manage different indexes or patterns in a single migration. For example, you might have one index that must use `staticMappings` and another that uses `regexMappings` to route all types by pattern. - -For each document, request, or metadata item (processed individually for bulk requests), the following steps are performed: - -1. The index is checked to determine whether it matches an entry in the static mappings. - - If matched, the type is checked against the index component of the static mappings entry. - - If the type matches, the mapping is applied, and the resulting index includes the value of the type key. - - If the type doesn't match, the request/document/metadata is dropped and not migrated. -2. If the index is not matched in the static mappings, the index-type combination is checked against each item in the regex mappings list, in order from first to last. If a match is found, the mapping is applied, the resulting index includes the value of the type key, and no further regex matching is performed. -3. Any request, document, or metadata that doesn't match the preceding cases is dropped, and the documents they contain are not migrated. - -The following example demonstrates how to combine static and regex-based mappings for different indexes: - -```json -[ - { - "TypeMappingSanitizationTransformerProvider": { - "staticMappings": { - "activity": { - "user": "users_activity", - "post": "posts_activity" - }, - "logs": { - "error": "logs_error", - "info": "logs_info" - } - }, - "regexMappings": [ - { - "sourceIndexPattern": "orders.*", - "sourceTypePattern": ".*", - "targetIndexPattern": "all_orders" - } - ], - "sourceProperties": { - "version": { - "major": 6, - "minor": 8 - } - } - } - } -] -``` -{% include copy.html %} - -### Defaults - -When the `regexMappings` key is missing from the transformation configuration, `regexMappings` will default to the following: - -```JSON -{ - "regexMappings": [ - { - "sourceIndexPattern": "(.+)", - "sourceTypePattern": "_doc", - "targetIndexPattern": "$1" - }, - { - "sourceIndexPattern": "(.+)", - "sourceTypePattern": "(.+)", - "targetIndexPattern": "$1_$2" - } - ] -} -``` -{% include copy.html %} - -This has the effect of retaining the index name for indexes created in Elasticsearch 6.x or later while combining the type and index name for indexes created in Elasticsearch 5.x. If you want to retain the index name for indexes created in Elasticsearch 5.x, use the `staticMappings` option or override the type mappings using the `regexMappings` option. - -## Limitations - -When using the transformer, remember the following limitations. -When using the transformer, remember the following limitations. -### Traffic Replayer - -For the Traffic Replayer, **only a subset** of requests that include types is supported. These requests are listed in the following table. - -| **Operation** | **HTTP method(s)** | **Endpoint** | **Description** | -| :--- | :--- | :--- | :--- | -| **Index (by ID)** | PUT/POST | `/{index}/{type}/{id}` | Create or update a single document with an explicit ID. | -| **Index (auto ID)** | PUT/POST | `/{index}/{type}/` | Create a single document for which the ID is automatically generated. | -| **Get Document** | GET | `/{index}/{type}/{id}` | Retrieve a document by ID. | -| **Bulk Index/Update/Delete** | PUT/POST | `/_bulk` | Perform multiple create/update/delete operations in a single request. | -| **Bulk Index/Update/Delete** | PUT/POST | `/{index}/_bulk` | Perform multiple create/update/delete operations in a single request with default index assignment. | -| **Bulk Index/Update/Delete** | PUT/POST | `/{index}/{type}/_bulk` | Perform multiple create/update/delete operations in a single request with default index and type assignment. | -| **Create/Update Index** | PUT/POST | `/{index}` | Create or update an index.

      **Split** behavior is not supported in the Traffic Replayer. See [this GitHub issue](https://github.com/opensearch-project/opensearch-migrations/issues/1305) to provide feedback or to vote on this feature. | - -### Reindex-From_Shapshot -For `Reindex-From-Snapshot,` indexes created in Elasticsearch 6.x or later will use `_doc` as the type for all documents, even if a different type was specified in Elasticsearch 6. diff --git a/_migration-assistant/migration-phases/planning-your-migration/index.md b/_migration-assistant/migration-phases/planning-your-migration/index.md deleted file mode 100644 index 9c6e7fee0bc..00000000000 --- a/_migration-assistant/migration-phases/planning-your-migration/index.md +++ /dev/null @@ -1,16 +0,0 @@ ---- -layout: default -title: Planning your migration -nav_order: 59 -parent: Migration phases -permalink: /planning-your-migration/ -has_toc: false -has_children: true ---- - -# Planning your migration - -This section describes how to plan for your migration to OpenSearch by: - -- [Assessing your current cluster for migration]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/planning-your-migration/assessing-your-cluster-for-migration/). -- [Verifying that you have the tools for migration]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/planning-your-migration/verifying-migration-tools/). \ No newline at end of file diff --git a/_migration-assistant/migration-phases/planning-your-migration/verifying-migration-tools.md b/_migration-assistant/migration-phases/planning-your-migration/verifying-migration-tools.md deleted file mode 100644 index 42bde888a1a..00000000000 --- a/_migration-assistant/migration-phases/planning-your-migration/verifying-migration-tools.md +++ /dev/null @@ -1,224 +0,0 @@ ---- -layout: default -title: Verifying migration tools -nav_order: 70 -parent: Planning your migration -grand_parent: Migration phases -permalink: /migration-assistant/migration-phases/planning-your-migration/verifying-migration-tools/ -redirect_from: - - /migration-assistant/migration-phases/verifying-migration-tools/ ---- - -# Verifying migration tools - -Before using the Migration Assistant, take the following steps to verify that your cluster is ready for migration. - -## Verifying snapshot creation - -Verify that a snapshot can be created of your source cluster and used for metadata and backfill scenarios. - -### Installing the Elasticsearch S3 Repository plugin - -The snapshot needs to be stored in a location that Migration Assistant can access. This guide uses Amazon Simple Storage Service (Amazon S3). By default, Migration Assistant creates an S3 bucket for storage. Therefore, it is necessary to install the [Elasticsearch S3 repository plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/7.10/repository-s3.html) on your source nodes (https://www.elastic.co/guide/en/elasticsearch/plugins/7.10/repository-s3.html). - -Additionally, make sure that the plugin has been configured with AWS credentials that allow it to read and write to Amazon S3. If your Elasticsearch cluster is running on Amazon Elastic Compute Cloud (Amazon EC2) or Amazon Elastic Container Service (Amazon ECS) instances with an AWS Identity and Access Management (IAM) execution role, include the necessary S3 permissions. Alternatively, you can store the credentials in the [Elasticsearch keystore](https://www.elastic.co/guide/en/elasticsearch/plugins/7.10/repository-s3-client.html). - -### Verifying the S3 repository plugin configuration - -You can verify that the S3 repository plugin is configured correctly by creating a test snapshot. - -Create an S3 bucket for the snapshot using the following AWS Command Line Interface (AWS CLI) command: - -```shell -aws s3api create-bucket --bucket --region -``` -{% include copy.html %} - -Register a new S3 snapshot repository on your source cluster using the following cURL command: - -```shell -curl -X PUT "http://:9200/_snapshot/test_s3_repository" -H "Content-Type: application/json" -d '{ - "type": "s3", - "settings": { - "bucket": "", - "region": "" - } -}' -``` -{% include copy.html %} - -Next, create a test snapshot that captures only the cluster's metadata: - -```shell -curl -X PUT "http://:9200/_snapshot/test_s3_repository/test_snapshot_1" -H "Content-Type: application/json" -d '{ - "indices": "", - "ignore_unavailable": true, - "include_global_state": true -}' -``` -{% include copy.html %} - -Check the AWS Management Console to confirm that your bucket contains the snapshot. - -### Removing test snapshots after verification - -To remove the resources created during verification, you can use the following deletion commands: - -**Test snapshot** - -```shell -curl -X DELETE "http://:9200/_snapshot/test_s3_repository/test_snapshot_1?pretty" -``` -{% include copy.html %} - -**Test snapshot repository** - -```shell -curl -X DELETE "http://:9200/_snapshot/test_s3_repository?pretty" -``` -{% include copy.html %} - -**S3 bucket** - -```shell -aws s3 rm s3:// --recursive -aws s3api delete-bucket --bucket --region -``` -{% include copy.html %} - -### Troubleshooting - -Use this guidance to troubleshoot any of the following snapshot verification issues. - -#### Access denied error (403) - -If you encounter an error like `AccessDenied (Service: Amazon S3; Status Code: 403)`, verify the following: - -- Make sure you're using the S3 bucket created by Migration Assistant. -- If you're using a custom S3 bucket, verify that: - - The IAM role assigned to your Elasticsearch cluster has the necessary S3 permissions. - - The bucket name and AWS Region provided in the snapshot configuration match the actual S3 bucket you created. - -#### Older versions of Elasticsearch - -Older versions of the Elasticsearch S3 repository plugin may have trouble reading IAM role credentials embedded in Amazon EC2 and Amazon ECS instances. This is because the copy of the AWS SDK shipped with them is too old to read the new standard way of retrieving those credentials, as shown in [the Instance Metadata Service v2 (IMDSv2) specification](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html). This can result in snapshot creation failures, with an error message similar to the following: - -```json -{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[migration_assistant_repo] path [rfs-snapshot-repo] is not accessible on master node"}],"type":"repository_verification_exception","reason":"[migration_assistant_repo] path [rfs-snapshot-repo] is not accessible on master node","caused_by":{"type":"i_o_exception","reason":"Unable to upload object [rfs-snapshot-repo/tests-s8TvZ3CcRoO8bvyXcyV2Yg/master.dat] using a single upload","caused_by":{"type":"amazon_service_exception","reason":"Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null)"}}},"status":500} -``` - -If you encounter this issue, you can resolve it by temporarily enabling IMDSv1 on the instances in your source cluster for the duration of the snapshot. There is a toggle for this available in the AWS Management Console as well as in the AWS CLI. Switching this toggle will turn on the older access model and enable the Elasticsearch S3 repository plugin to work as normal. For more information about IMDSv1, see [Modify instance metadata options for existing instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-IMDS-existing-instances.html). - -## Switching over client traffic - -The Migration Assistant Application Load Balancer is deployed with a listener that shifts traffic between the source and target clusters through proxy services. The Application Load Balancer should start in **Source Passthrough** mode. - -### Verifying that the traffic switchover is complete - -Use the following steps to verify that the traffic switchover is complete: - -1. In the AWS Management Console, navigate to **EC2 > Load Balancers**. -2. Select the **MigrationAssistant ALB**. -3. Examine the listener on port `9200` and verify that 100% of the traffic is directed to the **Source Proxy**. -4. Navigate to the **Migration ECS Cluster** in the AWS Management Console. -5. Select the **Target Proxy Service**. -6. Verify that the desired count for the service is running: - * If the desired count is not met, update the service to increase it to at least 1 and wait for the service to start. -7. On the **Health and Metrics** tab under **Load balancer health**, verify that all targets are reporting as healthy: - * This confirms that the Application Load Balancer can connect to the target cluster through the target proxy. -8. (Reset) Update the desired count for the **Target Proxy Service** back to its original value in Amazon ECS. - -### Fixing unidentified traffic patterns - -When switching over traffic to the target cluster, you might encounter unidentified traffic patterns. To help identify the cause of these patterns, use the following steps: -* Verify that the target cluster allows traffic ingress from the **Target Proxy Security Group**. -* Navigate to **Target Proxy ECS Tasks** to investigate any failing tasks. -Set the **Filter desired status** to **Any desired status** to view all tasks, then navigate to the logs for any stopped tasks. - - -## Verifying replication - -Use the following steps to verify that replication is working once the traffic capture proxy is deployed: - - -1. Navigate to the **Migration ECS Cluster** in the AWS Management Console. -2. Navigate to **Capture Proxy Service**. -3. Verify that the capture proxy is running with the desired proxy count. If it is not, update the service to increase it to at least 1 and wait for startup. -4. Under **Health and Metrics** > **Load balancer health**, verify that all targets are healthy. This means that the Application Load Balancer is able to connect to the source cluster through the capture proxy. -5. Navigate to the **Migration Console Terminal**. -6. Run `console kafka describe-topic-records`. Wait 30 seconds for another Application Load Balancer health check. -7. Run `console kafka describe-topic-records` again and verify that the number of RECORDS increased between runs. -8. Run `console replay start` to start Traffic Replayer. -9. Run `tail -f /shared-logs-output/traffic-replayer-default/*/tuples/tuples.log | jq '.targetResponses[]."Status-Code"'` to confirm that the Kafka requests were sent to the target and that it responded as expected. If the responses don't appear: - * Check that the migration console can access the target cluster by running `./catIndices.sh`, which should show the indexes in the source and target. - * Confirm that messages are still being recorded to Kafka. - * Check for errors in the Traffic Replayer logs (`/migration/STAGE/default/traffic-replayer-default`) using CloudWatch. -10. (Reset) Update the desired count for the **Capture Proxy Service** back to its original value in Amazon ECS. - -### Troubleshooting - -Use this guidance to troubleshoot any of the following replication verification issues. - -### Health check responses with 401/403 status code - -If the source cluster is configured to require authentication, the capture proxy will not be able to verify replication beyond receiving a 401/403 status code for Application Load Balancer health checks. For more information, see [Failure Modes](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/README.md#failure-modes). - -### Traffic does not reach the source cluster - -Verify that the source cluster allows traffic ingress from the Capture Proxy Security Group. - -Look for failing tasks by navigating to **Traffic Capture Proxy ECS**. Change **Filter desired status** to **Any desired status** in order to see all tasks and navigate to the logs for stopped tasks. - -### Snapshot and S3 bucket issues - -When using the CDK deployment for Migration Assistant, you might encounter the following errors during snapshot creation and deletion. - -#### Bucket permissions - -To make sure that you can delete snapshots as well as create them during the CDK deployment process, confirm that the `OSMigrations-dev--CustomS3AutoDeleteObjects` stack has S3 object deletion rights. Then, verify that `OSMigrations-dev--default-SnapshotRole` has the following S3 permissions: - - - List bucket contents - - Read/Write/Delete objects - -#### Snapshot conflicts - -To prevent snapshot conflicts, use the `console snapshot delete` command from the migration console. If you delete snapshots or snapshot repositories in a location other than the migration console, you might encounter "already exists" errors. - -## Resetting before migration - -After all verifications are complete, reset all resources before using Migration Assistant for an actual migration. - -The following steps outline how to reset resources with Migration Assistant before executing the actual migration. At this point all verifications are expected to have been completed. These steps can be performed after [Accessing the Migration Console]({{site.url}}{{site.baseurl}}/migration-assistant/migration-console/accessing-the-migration-console/). - -### Traffic Replayer - -To stop running Traffic Replayer, use the following command: - -```bash -console replay stop -``` -{% include copy.html %} - -### Kafka - -To clear all captured traffic from the Kafka topic, you can run the following command. - -This command will result in the loss of any traffic data captured by the capture proxy up to this point and thus should be used with caution. -{: .warning} - -```bash -console kafka delete-topic -``` -{% include copy.html %} - -### Target cluster - -To clear non-system indexes from the target cluster that may have been created as a result of testing, you can run the following command: - -This command will result in the loss of all data in the target cluster and should be used with caution. -{: .warning} - -```bash -console clusters clear-indices --cluster target -``` -{% include copy.html %} diff --git a/_migration-assistant/migration-phases/removing-migration-infrastructure.md b/_migration-assistant/migration-phases/removing-migration-infrastructure.md index 7d677908822..bb535a55d59 100644 --- a/_migration-assistant/migration-phases/removing-migration-infrastructure.md +++ b/_migration-assistant/migration-phases/removing-migration-infrastructure.md @@ -1,7 +1,7 @@ --- layout: default title: Removing migration infrastructure -nav_order: 120 +nav_order: 8 parent: Migration phases permalink: /migration-assistant/migration-phases/removing-migration-infrastructure/ --- diff --git a/_migration-assistant/migration-phases/replay-captured-traffic.md b/_migration-assistant/migration-phases/replay-captured-traffic.md deleted file mode 100644 index 0f4d1d6ea2f..00000000000 --- a/_migration-assistant/migration-phases/replay-captured-traffic.md +++ /dev/null @@ -1,319 +0,0 @@ ---- -layout: default -title: Replay -nav_order: 7 -grand_parent: Migration phases -parent: Live traffic migration -redirect_from: - - /migration-assistant/migration-phases/using-traffic-replayer/ ---- - -# Using Traffic Replayer - -This guide covers how to use Traffic Replayer to replay captured traffic from a source cluster to a target cluster during the migration process. Traffic Replayer allows you to verify that the target cluster can handle requests in the same way as the source cluster and catch up to real-time traffic for a smooth migration. - -## When to run Traffic Replayer - -After deploying Migration Assistant, Traffic Replayer does not run by default. It should be started only after all metadata and documents have been migrated to ensure that recent changes to the source cluster are properly reflected in the target cluster. - -For example, if a document was deleted after a snapshot was taken, starting Traffic Replayer before the document migration is complete may cause the deletion request to execute before the document is added to the target. Running Traffic Replayer after all other migration processes ensures that the target cluster will be consistent with the source cluster. - -## Configuration options - -[Traffic Replayer settings]({{site.url}}{{site.baseurl}}/migration-assistant/deploying-migration-assistant/configuration-options/) are configured during the deployment of Migration Assistant. Make sure to set the authentication mode for Traffic Replayer so that it can properly communicate with the target cluster. - -## Using Traffic Replayer - -To manage Traffic Replayer, use the `console replay` command. The following examples show the available commands. - -### Start Traffic Replayer - -The following command starts Traffic Replayer with the options specified at deployment: - -```bash -console replay start -``` - -When starting Traffic Replayer, you should receive an output similar to the following: - -```bash -root@ip-10-0-2-66:~# console replay start -Replayer started successfully. -Service migration-dev-traffic-replayer-default set to 1 desired count. Currently 0 running and 0 pending. -``` - -## Check the status of Traffic Replayer - -Use the following command to show the status of Traffic Replayer: - -```bash -console replay status -``` - -Replay will return one of the following statuses: - -- `Running` shows how many container instances are actively running. -- `Pending` indicates how many instances are being provisione.d -- `Desired` shows the total number of instances that should be running. - -You should receive an output similar to the following: - -```bash -root@ip-10-0-2-66:~# console replay status -(, 'Running=0\nPending=0\nDesired=0') -``` - -## Stop Traffic Replayer - -The following command stops Traffic Replayer: - -```bash -console replay stop -``` - -You should receive an output similar to the following: - -```bash -root@ip-10-0-2-66:~# console replay stop -Replayer stopped successfully. -Service migration-dev-traffic-replayer-default set to 0 desired count. Currently 0 running and 0 pending. -``` - - - -### Delivery guarantees - -Traffic Replayer retrieves traffic from Kafka and updates its commit cursor after sending requests to the target cluster. This provides an "at least once" delivery guarantee; however, success isn't always guaranteed. Therefore, you should monitor metrics and tuple outputs or perform external validation to ensure that the target cluster is functioning as expected. - -## Time scaling - -Traffic Replayer sends requests in the same order that they were received from each connection to the source. However, relative timing between different connections is not guaranteed. For example: - -- **Scenario**: Two connections exist:one sends a PUT request every minute, and the other sends a GET request every second. -- **Behavior**: Traffic Replayer will maintain the sequence within each connection, but the relative timing between the connections (PUTs and GETs) is not preserved. - -Assume that a source cluster responds to requests (GETs and PUTs) within 100 ms: - -- With a **speedup factor of 1**, the target will experience the same request rates and idle periods as the source. -- With a **speedup factor of 2**, requests will be sent twice as fast, with GETs sent every 500 ms and PUTs every 30 seconds. -- With a **speedup factor of 10**, requests will be sent 10x faster, and as long as the target responds quickly, Traffic Replayer can maintain the pace. - -If the target cannot respond fast enough, Traffic Replayer will wait for the previous request to complete before sending the next one. This may cause delays and affect global relative ordering. - -## Transformations - -During migrations, some requests may need to be transformed between versions. For example, Elasticsearch previously supported multiple type mappings in indexes, but this is no longer the case in OpenSearch. Clients may need to be adjusted accordingly by splitting documents into multiple indexes or transforming request data. - -Traffic Replayer automatically rewrites host and authentication headers, but for more complex transformations, custom transformation rules can be specified using the `--transformer-config` option. For more information, see the [Traffic Replayer README](https://github.com/opensearch-project/opensearch-migrations/blob/c3d25958a44ec2e7505892b4ea30e5fbfad4c71b/TrafficCapture/trafficReplayer/README.md#transformations). - -### Example transformation - -Suppose that a source request contains a `tagToExcise` element that needs to be removed and its children promoted and that the URI path includes `extraThingToRemove`, which should also be removed. The following Jolt script handles this transformation: - -```json -[{ "JsonJoltTransformerProvider": -[ - { - "script": { - "operation": "shift", - "spec": { - "payload": { - "inlinedJsonBody": { - "top": { - "tagToExcise": { - "*": "payload.inlinedJsonBody.top.&" - }, - "*": "payload.inlinedJsonBody.top.&" - }, - "*": "payload.inlinedJsonBody.&" - }, - "*": "payload.&" - }, - "*": "&" - } - } - }, - { - "script": { - "operation": "modify-overwrite-beta", - "spec": { - "URI": "=split('/extraThingToRemove',@(1,&))" - } - } - }, - { - "script": { - "operation": "modify-overwrite-beta", - "spec": { - "URI": "=join('',@(1,&))" - } - } - } -] -}] -``` - -The resulting request sent to the target will appear similar to the following: - -```bash -PUT /oldStyleIndex/moreStuff HTTP/1.0 -host: testhostname - -{"top":{"properties":{"field1":{"type":"text"},"field2":{"type":"keyword"}}}} -``` -{% include copy.html %} - -You can pass Base64-encoded transformation scripts using `--transformer-config-base64`. - -## Result logs - -HTTP transactions from the source capture and those resent to the target cluster are logged in files located at `/shared-logs-output/traffic-replayer-default/*/tuples/tuples.log`. The `/shared-logs-output` directory is shared across containers, including the migration console. You can access these files from the migration console using the same path. Previous runs are also available in a `gzipped` format. - -Each log entry is a newline-delimited JSON object, containing information about the source and target requests/responses along with other transaction details, such as response times. - -These logs contain the contents of all requests, including authorization headers and the contents of all HTTP messages. Ensure that access to the migration environment is restricted, as these logs serve as a source of truth for determining what happened in both the source and target clusters. Response times for the source refer to the amount of time between the proxy sending the end of a request and receiving the response. While response times for the target are recorded in the same manner, keep in mind that the locations of the capture proxy, Traffic Replayer, and target may differ and that these logs do not account for the client's location. -{: .note} - - -### Example log entry - -The following example log entry shows a `/_cat/indices?v` request sent to both the source and target clusters: - -```json -{ - "sourceRequest": { - "Request-URI": "/_cat/indices?v", - "Method": "GET", - "HTTP-Version": "HTTP/1.1", - "Host": "capture-proxy:9200", - "Authorization": "Basic YWRtaW46YWRtaW4=", - "User-Agent": "curl/8.5.0", - "Accept": "*/*", - "body": "" - }, - "sourceResponse": { - "HTTP-Version": {"keepAliveDefault": true}, - "Status-Code": 200, - "Reason-Phrase": "OK", - "response_time_ms": 59, - "content-type": "text/plain; charset=UTF-8", - "content-length": "214", - "body": "aGVhbHRoIHN0YXR1cyBpbmRleCAgICAgICB..." - }, - "targetRequest": { - "Request-URI": "/_cat/indices?v", - "Method": "GET", - "HTTP-Version": "HTTP/1.1", - "Host": "opensearchtarget", - "Authorization": "Basic YWRtaW46bXlTdHJvbmdQYXNzd29yZDEyMyE=", - "User-Agent": "curl/8.5.0", - "Accept": "*/*", - "body": "" - }, - "targetResponses": [{ - "HTTP-Version": {"keepAliveDefault": true}, - "Status-Code": 200, - "Reason-Phrase": "OK", - "response_time_ms": 721, - "content-type": "text/plain; charset=UTF-8", - "content-length": "484", - "body": "aGVhbHRoIHN0YXR1cyBpbmRleCAgICAgICB..." - }], - "connectionId": "0242acfffe13000a-0000000a-00000005-1eb087a9beb83f3e-a32794b4.0", - "numRequests": 1, - "numErrors": 0 -} -``` -{% include copy.html %} - - -### Decoding log content - -The contents of HTTP message bodies are Base64 encoded in order to handle various types of traffic, including compressed data. To view the logs in a more human-readable format, use the console library `tuples show`. Running the script as follows will produce a `readable-tuples.log` in the home directory: - -```shell -console tuples show --in /shared-logs-output/traffic-replayer-default/d3a4b31e1af4/tuples/tuples.log > readable-tuples.log -``` - -The `readable-tuples.log` should appear similar to the following: - -```json -{ - "sourceRequest": { - "Request-URI": "/_cat/indices?v", - "Method": "GET", - "HTTP-Version": "HTTP/1.1", - "Host": "capture-proxy:9200", - "Authorization": "Basic YWRtaW46YWRtaW4=", - "User-Agent": "curl/8.5.0", - "Accept": "*/*", - "body": "" - }, - "sourceResponse": { - "HTTP-Version": {"keepAliveDefault": true}, - "Status-Code": 200, - "Reason-Phrase": "OK", - "response_time_ms": 59, - "content-type": "text/plain; charset=UTF-8", - "content-length": "214", - "body": "health status index uuid ..." - }, - "targetRequest": { - "Request-URI": "/_cat/indices?v", - "Method": "GET", - "HTTP-Version": "HTTP/1.1", - "Host": "opensearchtarget", - "Authorization": "Basic YWRtaW46bXlTdHJvbmdQYXNzd29yZDEyMyE=", - "User-Agent": "curl/8.5.0", - "Accept": "*/*", - "body": "" - }, - "targetResponses": [{ - "HTTP-Version": {"keepAliveDefault": true}, - "Status-Code": 200, - "Reason-Phrase": "OK", - "response_time_ms": 721, - "content-type": "text/plain; charset=UTF-8", - "content-length": "484", - "body": "health status index uuid ..." - }], - "connectionId": "0242acfffe13000a-0000000a-00000005-1eb087a9beb83f3e-a32794b4.0", - "numRequests": 1, - "numErrors": 0 -} -``` - - -## Amazon CloudWatch metrics and dashboard -Migration Assistant creates an Amazon CloudWatch dashboard named `MigrationAssistant_ReindexFromSnapshot_Dashboard` to visualize the health and performance of the backfill process. This dashboard combines metrics for the backfill workers and migration to Amazon OpenSearch Service, providing insights into the performance and health of the Capture Proxy and Traffic Replayer components, including metrics such as: - -- The number of bytes read and written. -- The number of active connections. -- The replay speed multiplier. - -You can find the Capture and Replay dashboard in the AWS Management Console for CloudWatch Dashboards in the AWS Region where you deployed Migration Assistant. - -Traffic Replayer emits various OpenTelemetry metrics to Amazon CloudWatch, and traces are sent through AWS X-Ray. The following are some useful metrics that can help evaluate migration performance. - -### `sourceStatusCode` - -This metric tracks the HTTP status codes for both the source and target clusters, with dimensions for the HTTP verb, such as `GET` or `POST`, and the status code families (200--299). These dimensions can help quickly identify discrepancies between the source and target, such as when `DELETE 200s` becomes `4xx` or `GET 4xx` errors turn into `5xx` errors. - -### `lagBetweenSourceAndTargetRequests` - -This metric shows the delay between requests hitting the source and target clusters. With a speedup factor greater than 1 and a target cluster that can handle requests efficiently, this value should decrease as the replay progresses, indicating a reduction in replay lag. - -### Additional metrics - -The following metrics are also reported: - -- **Throughput**: `bytesWrittenToTarget` and `bytesReadFromTarget` indicate the throughput to and from the cluster. -- **Retries**: `numRetriedRequests` tracks the number of requests retried due to status code mismatches between the source and target. -- **Event counts**: Various `(*)Count` metrics track the number of completed events. -- **Durations**: `(*)Duration` metrics measure the duration of each step in the process. -- **Exceptions**: `(*)ExceptionCount` shows the number of exceptions encountered during each processing phase. - - -## CloudWatch considerations - -Metrics and dashboards pushed to CloudWatch may experience a visibility lag of around 5 minutes. CloudWatch also retains higher-resolution data for a shorter period than lower-resolution data. For more information, see [Amazon CloudWatch concepts](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html). \ No newline at end of file diff --git a/_migration-assistant/migration-phases/teardown.md b/_migration-assistant/migration-phases/teardown.md deleted file mode 100644 index 99e975c93b6..00000000000 --- a/_migration-assistant/migration-phases/teardown.md +++ /dev/null @@ -1,30 +0,0 @@ ---- -layout: default -title: Remove Migration Infrastructure -nav_order: 8 -parent: Migration phases -redirect_from: - - /migration-assistant/migration-phases/removing-migration-infrastructure ---- - -# Removing migration infrastructure - -After a migration is complete, you should remove all resources except for the target cluster and, optionally, your Amazon CloudWatch logs and Traffic Replayer logs. - -To remove the AWS Cloud Development Kit (AWS CDK) stack(s) created during a deployment, run the following command within the CDK directory: - -```bash -cd deployment/cdk/opensearch-service-migration -cdk destroy "*" --c contextId= -``` -{% include copy.html %} - -Follow the instructions on the command line to remove the deployed resources from your AWS account. - -You can also use the AWS Management Console to remove Migration Assistant resources and confirm that they are no longer present in the account. - -## Uninstalling Migration Assistant for Amazon OpenSearch Service - -You can uninstall Migration Assistant for Amazon OpenSearch Service from the AWS Management Console or by using the AWS Command Line Interface (AWS CLI). Manually remove the contents of the Amazon Simple Storage Service (Amazon S3) bucket that matches the syntax `cdk--assets--`, the bucket created by Migration Assistant. Migration Assistant for Amazon OpenSearch Service does not automatically delete Amazon S3 buckets. - -To delete the stored data and the AWS CloudFormation stacks created by Migration Assistant, see [Uninstall the solution](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/uninstall-the-solution.html) in the Amazon OpenSearch Service documentation. diff --git a/_migration-assistant/migration-phases/live-traffic-migration/using-traffic-replayer.md b/_migration-assistant/migration-phases/using-traffic-replayer.md similarity index 98% rename from _migration-assistant/migration-phases/live-traffic-migration/using-traffic-replayer.md rename to _migration-assistant/migration-phases/using-traffic-replayer.md index 1454f7f04a3..a1b892ec752 100644 --- a/_migration-assistant/migration-phases/live-traffic-migration/using-traffic-replayer.md +++ b/_migration-assistant/migration-phases/using-traffic-replayer.md @@ -1,10 +1,8 @@ --- layout: default title: Using Traffic Replayer -nav_order: 100 -grand_parent: Migration phases -parent: Live traffic migration -permalink: /migration-assistant/migration-phases/live-traffic-migration/using-traffic-replayer/ +nav_order: 7 +parent: Migration phases redirect_from: - /migration-assistant/migration-phases/using-traffic-replayer/ --- diff --git a/_migration-assistant/migration-phases/verify-backfill-components.md b/_migration-assistant/migration-phases/verifying-migration-tools/verifying-backfill-components.md similarity index 98% rename from _migration-assistant/migration-phases/verify-backfill-components.md rename to _migration-assistant/migration-phases/verifying-migration-tools/verifying-backfill-components.md index e8f99d7d88b..8103ff51b0e 100644 --- a/_migration-assistant/migration-phases/verify-backfill-components.md +++ b/_migration-assistant/migration-phases/verifying-migration-tools/verifying-backfill-components.md @@ -1,12 +1,14 @@ --- layout: default -title: Backfill Setup Verification +title: Verifying backfill components +grand_parent: Migrations phases nav_order: 3 +parent: Verifying migration tools redirect_from: - - /migration-assistant/migration-phases/verifying-migration-tools/ + - /migration-assistant/planning-your-migration/migration-phases/verifying-migration-tools/ --- -# Verifying Migration Assistant Backfill Components +# Verifying backfill components Before using the Migration Assistant, take the following steps to verify that your cluster is ready for migration. diff --git a/_migration-assistant/migration-phases/verify-live-capture-components.md b/_migration-assistant/migration-phases/verifying-migration-tools/verifying-live-capture-components.md similarity index 98% rename from _migration-assistant/migration-phases/verify-live-capture-components.md rename to _migration-assistant/migration-phases/verifying-migration-tools/verifying-live-capture-components.md index 90666b80bc8..d138d446df6 100644 --- a/_migration-assistant/migration-phases/verify-live-capture-components.md +++ b/_migration-assistant/migration-phases/verifying-migration-tools/verifying-live-capture-components.md @@ -1,12 +1,12 @@ --- layout: default -title: Live Capture Setup Verification +title: Verifying live capture components nav_order: 9 -redirect_from: - - /migration-assistant/migration-phases/verifying-migration-tools/ +grand_parent: Migration phases +parent: Verifying migration tools --- -# Verifying migration tools +# Verifying live capture components Before using the Migration Assistant, take the following steps to verify that your cluster is ready for migration. diff --git a/_migration-assistant/migration-phases/verifying-migration-tools/verifying-migration-tools.md b/_migration-assistant/migration-phases/verifying-migration-tools/verifying-migration-tools.md new file mode 100644 index 00000000000..0c320f3bac3 --- /dev/null +++ b/_migration-assistant/migration-phases/verifying-migration-tools/verifying-migration-tools.md @@ -0,0 +1,20 @@ +--- +layout: default +title: Verifying migration tools +parent: Migrations phases +nav_order: 3 +has_children: true +redirect_from: + - /migration-assistant/planning-your-migration/migration-phases/verifying-migration-tools/ +--- + +# Verifying migration tools + +Before you begin migrating data, it is important to verify that each component of your migration workflow is correctly deployed and functioning as expected. This step reduces the risk of disruptions during cutover and helps identify misconfigurations early in the process. + +Use the following pages to verify key components: + +- [Verifying backfill components]({{site.url}}{{site.baseurl}}/migration-assistant/planning-your-migration/migration-phases/verifying-migration-tools/verifying-backfill-components/) +- [Verifying live capture components]({{site.url}}{{site.baseurl}}/migration-assistant/planning-your-migration/migration-phases/verifying-migration-tools/verifying-live-capture-components/) + + diff --git a/_migration-assistant/planning-your-migration/handling-field-type-breaking-changes.md b/_migration-assistant/planning-your-migration/handling-field-type-breaking-changes.md index b6a653f96ae..4fff3cda1b3 100644 --- a/_migration-assistant/planning-your-migration/handling-field-type-breaking-changes.md +++ b/_migration-assistant/planning-your-migration/handling-field-type-breaking-changes.md @@ -3,7 +3,6 @@ layout: default title: Handling breaking changes in field types nav_order: 60 parent: Planning your migration -grand_parent: Migration phases --- # Handling breaking changes in field types diff --git a/_migration-assistant/planning-your-migration/handling-type-mapping-deprecation.md b/_migration-assistant/planning-your-migration/handling-type-mapping-deprecation.md index edd81612dac..060942591e5 100644 --- a/_migration-assistant/planning-your-migration/handling-type-mapping-deprecation.md +++ b/_migration-assistant/planning-your-migration/handling-type-mapping-deprecation.md @@ -3,7 +3,6 @@ layout: default title: Managing type mapping deprecation nav_order: 60 parent: Planning your migration -grand_parent: Migration phases --- # Managing type mapping deprecation diff --git a/_migration-assistant/planning-your-migration/index.md b/_migration-assistant/planning-your-migration/index.md index 4f0d264e93e..54b6456fc97 100644 --- a/_migration-assistant/planning-your-migration/index.md +++ b/_migration-assistant/planning-your-migration/index.md @@ -2,7 +2,6 @@ layout: default title: Planning your migration nav_order: 59 -parent: Migration phases has_toc: false has_children: true --- @@ -11,5 +10,5 @@ has_children: true This section describes how to plan for your migration to OpenSearch by: -- [Assessing your current cluster for migration]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/planning-your-migration/assessing-your-cluster-for-migration/). -- [Verifying that you have the tools for migration]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/planning-your-migration/verifying-migration-tools/). \ No newline at end of file +- [Assessing your current cluster for migration]({{site.url}}{{site.baseurl}}/migration-assistant/planning-your-migration/assessing-your-cluster-for-migration/). +- [Verifying that you have the tools for migration]({{site.url}}{{site.baseurl}}/migration-assistant/planning-your-migration/verifying-migration-tools/). \ No newline at end of file From a8ac2c97c3d4793d7c21885d80e4a61d282ce8f2 Mon Sep 17 00:00:00 2001 From: Archer Date: Thu, 10 Jul 2025 11:47:18 -0500 Subject: [PATCH 15/62] More structure fixes and redirects Signed-off-by: Archer --- _data/migration-assistant/breaking-changes.yml | 4 ++-- ...nt.md => assessing-your-cluster-for-migration.md} | 4 +--- _migration-assistant/migration-phases/index.md | 12 ++++++------ .../verifying-backfill-components.md | 2 +- .../verifying-migration-tools.md | 7 +++---- .../planning-your-migration/index.md | 2 ++ 6 files changed, 15 insertions(+), 16 deletions(-) rename _migration-assistant/migration-phases/{assessment.md => assessing-your-cluster-for-migration.md} (94%) diff --git a/_data/migration-assistant/breaking-changes.yml b/_data/migration-assistant/breaking-changes.yml index 193d1f129b1..f41e9219041 100644 --- a/_data/migration-assistant/breaking-changes.yml +++ b/_data/migration-assistant/breaking-changes.yml @@ -27,7 +27,7 @@ breaking_changes: comp: [] transformation: title: "Type Mapping Deprecation" - url: "https://docs.opensearch.org/docs/latest/migration-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation/" + url: "https://docs.opensearch.org/docs/latest/migration-assistant/planning-your-migration/handling-type-mapping-deprecation/" - title: "OpenSearch 3.x: Breaking Changes" url: "/docs/latest/breaking-changes/#300" introducedIn: "OpenSearch 3.x" @@ -46,7 +46,7 @@ breaking_changes: comp: [] transformation: title: "Type Mapping Deprecation" - url: "https://docs.opensearch.org/docs/latest/migration-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation/" + url: "https://docs.opensearch.org/docs/latest/migration-assistant/planning-your-migration/handling-type-mapping-deprecation/" - title: "Elasticsearch 6.0 - 6.6 Breaking Changes" url: "https://www.elastic.co/guide/en/elasticsearch/reference/6.8/breaking-changes.html" introducedIn: "Elasticsearch 6.x" diff --git a/_migration-assistant/migration-phases/assessment.md b/_migration-assistant/migration-phases/assessing-your-cluster-for-migration.md similarity index 94% rename from _migration-assistant/migration-phases/assessment.md rename to _migration-assistant/migration-phases/assessing-your-cluster-for-migration.md index 2d2ee5f872b..c42029fc6ec 100644 --- a/_migration-assistant/migration-phases/assessment.md +++ b/_migration-assistant/migration-phases/assessing-your-cluster-for-migration.md @@ -3,8 +3,6 @@ layout: default title: Assessing your cluster for migration nav_order: 1 parent: Migration phases -redirect_from: - - /migration-assistant/migration-phases/planning-your-migration/assessing-your-cluster-for-migration/ --- # Assessing your cluster for migration @@ -73,4 +71,4 @@ For complex migrations involving multiple transformations or breaking changes, w The following [transformations]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/live-traffic-migration/using-traffic-replayer/#transformations) are included in Migration Assistant. They can be enabled, combined, and configured to customize a migration for your use case. To request additional Migration Assistant transformations , create a GitHub issue [in the OpenSearch migrations repository](https://github.com/opensearch-project/opensearch-migrations/issues). -- [Type mapping deprecation]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation/) +- [Type mapping deprecation]({{site.url}}{{site.baseurl}}/migration-assistant/planning-your-migration/handling-type-mapping-deprecation/) diff --git a/_migration-assistant/migration-phases/index.md b/_migration-assistant/migration-phases/index.md index 6c3a8a8d688..937e40e5d1c 100644 --- a/_migration-assistant/migration-phases/index.md +++ b/_migration-assistant/migration-phases/index.md @@ -36,7 +36,7 @@ details[open] { Scenario 1 – Backfill Only
        -
      1. Assessment
        2. +
      2. Assessment
      3. Deployment
      4. Create Snapshot
        5. @@ -51,15 +51,15 @@ details[open] { Scenario 2 – Live Capture Only
          -
        1. Assessment
          2. +
        2. Assessment
        3. Deployment
        4. Verify Backfill Components
        5. Reroute Traffic from Source to Capture Proxy
        6. Migrate Metadata
        7. Verify Live Capture Components
          8. -
        8. Replay Captured Traffic
          9. +
        9. Replay Captured Traffic
        10. Reroute Traffic from Capture Proxy to Target
          11. -
        11. Teardown
          12. +
        12. Teardown
        @@ -68,7 +68,7 @@ details[open] { Scenario 3 – Live Capture with Backfill
          -
        1. Assessment
          2. +
        2. Assessment
        3. Deployment
        4. Reroute Traffic from Source to Capture Proxy
        5. Create Snapshot
          6. @@ -76,7 +76,7 @@ details[open] {
        6. Migrate Data
        7. Replay Traffic
        8. Reroute Traffic from Capture Proxy to Target
          9. -
        9. Teardown
          10. +
        10. Teardown
        diff --git a/_migration-assistant/migration-phases/verifying-migration-tools/verifying-backfill-components.md b/_migration-assistant/migration-phases/verifying-migration-tools/verifying-backfill-components.md index 8103ff51b0e..233f098a546 100644 --- a/_migration-assistant/migration-phases/verifying-migration-tools/verifying-backfill-components.md +++ b/_migration-assistant/migration-phases/verifying-migration-tools/verifying-backfill-components.md @@ -5,7 +5,7 @@ grand_parent: Migrations phases nav_order: 3 parent: Verifying migration tools redirect_from: - - /migration-assistant/planning-your-migration/migration-phases/verifying-migration-tools/ + - /migration-assistant/planning-your-migration/migration-phases/verifying-backfill-components/ --- # Verifying backfill components diff --git a/_migration-assistant/migration-phases/verifying-migration-tools/verifying-migration-tools.md b/_migration-assistant/migration-phases/verifying-migration-tools/verifying-migration-tools.md index 0c320f3bac3..c57505cc139 100644 --- a/_migration-assistant/migration-phases/verifying-migration-tools/verifying-migration-tools.md +++ b/_migration-assistant/migration-phases/verifying-migration-tools/verifying-migration-tools.md @@ -4,8 +4,7 @@ title: Verifying migration tools parent: Migrations phases nav_order: 3 has_children: true -redirect_from: - - /migration-assistant/planning-your-migration/migration-phases/verifying-migration-tools/ +permalink: /migration-assistant/planning-your-migration/migration-phases/verifying-migration-tools/ --- # Verifying migration tools @@ -14,7 +13,7 @@ Before you begin migrating data, it is important to verify that each component o Use the following pages to verify key components: -- [Verifying backfill components]({{site.url}}{{site.baseurl}}/migration-assistant/planning-your-migration/migration-phases/verifying-migration-tools/verifying-backfill-components/) -- [Verifying live capture components]({{site.url}}{{site.baseurl}}/migration-assistant/planning-your-migration/migration-phases/verifying-migration-tools/verifying-live-capture-components/) +- [Verifying backfill components]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/verifying-migration-tools/verifying-backfill-components/) +- [Verifying live capture components]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/verifying-migration-tools/verifying-live-capture-components/) diff --git a/_migration-assistant/planning-your-migration/index.md b/_migration-assistant/planning-your-migration/index.md index 54b6456fc97..20a1ab749b3 100644 --- a/_migration-assistant/planning-your-migration/index.md +++ b/_migration-assistant/planning-your-migration/index.md @@ -4,6 +4,8 @@ title: Planning your migration nav_order: 59 has_toc: false has_children: true +redirect_from: + - /migration-assistant/migration-phases/planning-your-migration/index/ --- # Planning your migration From ca73bb04fe2ccbef853d992ae0ad624b8ca89d69 Mon Sep 17 00:00:00 2001 From: Archer Date: Thu, 10 Jul 2025 12:03:12 -0500 Subject: [PATCH 16/62] Fix merge conflict Signed-off-by: Archer --- .../configuration-options.md | 38 ++++++++++++------- 1 file changed, 24 insertions(+), 14 deletions(-) diff --git a/_migration-assistant/migration-phases/deploying-migration-assistant/configuration-options.md b/_migration-assistant/migration-phases/deploying-migration-assistant/configuration-options.md index dccea3a74d2..826d3e7ed49 100644 --- a/_migration-assistant/migration-phases/deploying-migration-assistant/configuration-options.md +++ b/_migration-assistant/migration-phases/deploying-migration-assistant/configuration-options.md @@ -2,8 +2,7 @@ layout: default title: Configuration options nav_order: 15 -nav_exclude: true -permalink: /migration-assistant/deploying-migration-assistant/configuration-options/ +parent: Deploying Migration Assistant --- # Configuration options @@ -98,11 +97,20 @@ The following sample CDK performs a live capture migration with C&R: "passwordFromSecretArn": } }, - "captureProxyServiceEnabled": true, - "captureProxyExtraArgs": "", + + "// settingsForCaptureAndReplay": "Enable the following services for live traffic capture and replay:", "trafficReplayerServiceEnabled": true, - "trafficReplayerExtraArgs": "--speedup-factor 2.0", - "targetClusterProxyServiceEnabled": true + + "// help trafficReplayerExtraArgs": "Increase the speedup factor to replay requests at a faster rate in order to catch up.", + "trafficReplayerExtraArgs": "--speedup-factor 1.5", + + "// help capture/target proxy pt. 1 of 2": "captureProxyService and targetClusterProxyService deployment will fail without network access to clusters.", + "// help capture/target proxy pt. 2 of 2": "In most cases, keep the desired count setting at `0` until you verify connectivity in the migration console. After verifying connectivity, you can redeploy with a higher desired count.", + "captureProxyServiceEnabled": true, + "captureProxyDesiredCount": 3, + "targetClusterProxyServiceEnabled": true, + "targetClusterProxyDesiredCount": 3 + } } ``` @@ -111,13 +119,15 @@ The following sample CDK performs a live capture migration with C&R: Performing a live capture migration requires that a Capture Proxy be configured to capture incoming traffic and send it to the target cluster using the Traffic Replayer service. For arguments available in `captureProxyExtraArgs`, refer to the `@Parameter` fields [here](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java). For `trafficReplayerExtraArgs`, refer to the `@Parameter` fields [here](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java). At a minimum, no extra arguments may be needed. -| Name | Example | Description | -| :--- | :--- | :--- | -| `captureProxyServiceEnabled` | `true` | Enables the Capture Proxy service deployment using an AWS CloudFormation stack. | -| `captureProxyExtraArgs` | `"--suppressCaptureForHeaderMatch user-agent .*elastic-java/7.17.0.*"` | Extra arguments for the Capture Proxy command, including options specified by the [Capture Proxy](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java). | -| `trafficReplayerServiceEnabled` | `true` | Enables the Traffic Replayer service deployment using a CloudFormation stack. | -| `trafficReplayerExtraArgs` | `"--sigv4-auth-header-service-region es,us-east-1 --speedup-factor 5"` | Extra arguments for the Traffic Replayer command, including options for auth headers and other parameters specified by the [Traffic Replayer](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java). | -| `targetClusterProxyServiceEnabled` | `true` | Enables the target cluster proxy service deployment using a CloudFormation stack. | +| Name | Example | Description | +| :--- |:-----------------------------------------------------------------------| :--- | +| `captureProxyServiceEnabled` | `true` | Enables the Capture Proxy service deployment using an AWS CloudFormation stack. | +| `captureProxyExtraArgs` | `"--suppressCaptureForHeaderMatch user-agent .*elastic-java/7.17.0.*"` | Extra arguments for the Capture Proxy command, including options specified by the [Capture Proxy](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java). | +| `captureProxyDesiredCount` | `0` | Sets the number of Capture Proxy Amazon Elastic Container Service (Amazon ECS) tasks. In most cases, keep this setting at `0` until you verify connectivity between the source and target clusters in the migration console. After deployment, you can modify the networking setup to allow ingress from the migration security groups into the existing cluster security groups. | +| `trafficReplayerServiceEnabled` | `true` | Enables the Traffic Replayer service deployment using a CloudFormation stack. | +| `trafficReplayerExtraArgs` | `"--sigv4-auth-header-service-region es,us-east-1 --speedup-factor 5"` | Extra arguments for the Traffic Replayer command, including options for auth headers and other parameters specified by the [Traffic Replayer](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java). | +| `targetClusterProxyServiceEnabled` | `true` | Enables the target cluster proxy service deployment using a CloudFormation stack. | +| `targetClusterProxyDesiredCount` | `0` | Sets the number of target cluster proxy Amazon ECS tasks. In most cases, keep this setting at `0` until you verify connectivity between the source and target clusters in the migration console. After deployment, you can modify the networking setup to allow ingress from the migration security groups into the existing cluster security groups. | For arguments available in `captureProxyExtraArgs`, see the `@Parameter` fields in [`CaptureProxy.java`](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java). For `trafficReplayerExtraArgs`, see the `@Parameter` fields in [`TrafficReplayer.java`](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java). @@ -230,4 +240,4 @@ If the S3 bucket is in a separate AWS account from the Migration Assistant deplo ## Network configuration -The migration tooling expects the source cluster, target cluster, and migration resources to exist in the same VPC. If this is not the case, manual networking setup outside of this documentation is likely required. +The migration tooling expects the source cluster, target cluster, and migration resources to exist in the same VPC. If this is not the case, manual networking setup outside of this documentation is likely required. \ No newline at end of file From 5566d1055f1e07a0eda00477149f3c40471233dd Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Sat, 12 Jul 2025 15:34:29 -0500 Subject: [PATCH 17/62] Revered MA card to point to MA, renamed path to migrate-and-upgrade Signed-off-by: Brian Presley --- _config.yml | 4 ++-- _includes/home_cards.html | 6 +++--- {_migration-upgrade => _migrate-or-upgrade}/index.md | 4 ++-- 3 files changed, 7 insertions(+), 7 deletions(-) rename {_migration-upgrade => _migrate-or-upgrade}/index.md (98%) diff --git a/_config.yml b/_config.yml index 1d4455bd9e3..e69c593e1c0 100644 --- a/_config.yml +++ b/_config.yml @@ -89,7 +89,7 @@ collections: data-prepper: permalink: /:collection/:path/ output: true - migration-upgrade: + migrate-or-upgrade: permalink: /:collection/:path/ output: true migration-assistant: @@ -147,7 +147,7 @@ opensearch_collection: install-and-configure: name: Install and configure nav_fold: true - migration-upgrade: + migrate-or-upgrade: name: Migrate or upgrade nav_fold: true tuning-your-cluster: diff --git a/_includes/home_cards.html b/_includes/home_cards.html index 8be45489317..4065cf27e0c 100644 --- a/_includes/home_cards.html +++ b/_includes/home_cards.html @@ -61,9 +61,9 @@
        - -

        Modernize Workloads

        -

        Migration Assistant for OpenSearch and other upgrade and migration options

        + +

        Migration Asssistant

        +

        Simplify your upgrade or migration to OpenSearch

        Documentation →

        diff --git a/_migration-upgrade/index.md b/_migrate-or-upgrade/index.md similarity index 98% rename from _migration-upgrade/index.md rename to _migrate-or-upgrade/index.md index bd0f7319ee0..4c98ea50a38 100644 --- a/_migration-upgrade/index.md +++ b/_migrate-or-upgrade/index.md @@ -1,11 +1,11 @@ --- layout: default -title: Migration Assistant for OpenSearch +title: Migrate or upgrade nav_order: 1 has_children: false nav_exclude: true has_toc: false -permalink: /migrations-and-upgrades/ +permalink: /migrate-or-upgrade/ redirect_from: - /upgrade-to/index/ - /upgrade-to/ From 3725f610c14e607e6187cadc1697ae512666f511 Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Mon, 14 Jul 2025 21:49:12 -0500 Subject: [PATCH 18/62] Restructure checkpoint Signed-off-by: Brian Presley --- _config.yml | 8 +- _includes/home_cards.html | 2 +- .../configuring-opensearch/index.md | 3 +- _install-and-configure/index.md | 4 - .../upgrade-opensearch/index.md | 271 -------------- _migrate-or-upgrade/index.md | 218 +++++++---- .../migration-assistant}/architecture.md | 4 +- .../assets/css/breaking-changes-selector.css | 0 .../assets/js/breaking-changes-data.js | 0 .../assets/js/breaking-changes-filter.js | 0 .../assets/js/breaking-changes-index.js | 0 .../assets/js/breaking-changes-module.js | 0 .../assets/js/breaking-changes-ui.js | 0 .../migration-assistant}/index.md | 14 +- .../is-migration-assistant-right-for-you.md | 7 +- .../migration-assistant}/key-components.md | 6 +- .../accessing-the-migration-console.md | 0 .../migration-console/index.md | 5 +- .../migration-console-commands-references.md | 0 .../accessing-the-migration-console.md | 0 .../assessing-your-cluster-for-migration.md | 1 + .../migration-phases/backfill.md | 0 .../migration-phases/create-snapshot.md | 8 +- .../migration-phases/deploy.md | 2 +- .../configuration-options.md | 0 ...d-security-groups-for-existing-clusters.md | 0 .../migration-phases/index.md | 35 +- .../live-traffic-migration/index.md | 4 +- ...itching-traffic-from-the-source-cluster.md | 0 .../migration-phases/migrate-metadata.md | 3 +- .../remove-migration-infrastructure.md | 6 +- .../reroute-source-to-proxy.md | 3 +- ...te-traffic-from-capture-proxy-to-target.md | 3 +- .../using-traffic-replayer.md | 0 .../verifying-backfill-components.md | 3 +- .../verifying-live-capture-components.md | 2 +- .../verifying-migration-tools.md | 3 +- .../handling-field-type-breaking-changes.md | 0 .../handling-type-mapping-deprecation.md | 0 .../rolling-upgrade}/appendix/index.md | 0 .../appendix/rolling-upgrade-lab.md | 2 +- .../rolling-upgrade/index.md | 11 +- .../getting-started-data-migration.md | 345 ------------------ _migration-assistant/overview/index.md | 25 -- .../planning-your-migration/index.md | 16 - .../remote-store/migrating-to-remote.md | 2 +- 46 files changed, 218 insertions(+), 798 deletions(-) delete mode 100644 _install-and-configure/upgrade-opensearch/index.md rename {_migration-assistant/overview => _migrate-or-upgrade/migration-assistant}/architecture.md (94%) rename {_migration-assistant => _migrate-or-upgrade/migration-assistant}/assets/css/breaking-changes-selector.css (100%) rename {_migration-assistant => _migrate-or-upgrade/migration-assistant}/assets/js/breaking-changes-data.js (100%) rename {_migration-assistant => _migrate-or-upgrade/migration-assistant}/assets/js/breaking-changes-filter.js (100%) rename {_migration-assistant => _migrate-or-upgrade/migration-assistant}/assets/js/breaking-changes-index.js (100%) rename {_migration-assistant => _migrate-or-upgrade/migration-assistant}/assets/js/breaking-changes-module.js (100%) rename {_migration-assistant => _migrate-or-upgrade/migration-assistant}/assets/js/breaking-changes-ui.js (100%) rename {_migration-assistant => _migrate-or-upgrade/migration-assistant}/index.md (84%) rename {_migration-assistant/overview => _migrate-or-upgrade/migration-assistant}/is-migration-assistant-right-for-you.md (98%) rename {_migration-assistant/overview => _migrate-or-upgrade/migration-assistant}/key-components.md (95%) rename {_migration-assistant => _migrate-or-upgrade/migration-assistant}/migration-console/accessing-the-migration-console.md (100%) rename {_migration-assistant => _migrate-or-upgrade/migration-assistant}/migration-console/index.md (80%) rename {_migration-assistant => _migrate-or-upgrade/migration-assistant}/migration-console/migration-console-commands-references.md (100%) rename {_migration-assistant => _migrate-or-upgrade/migration-assistant}/migration-phases/accessing-the-migration-console.md (100%) rename {_migration-assistant => _migrate-or-upgrade/migration-assistant}/migration-phases/assessing-your-cluster-for-migration.md (98%) rename {_migration-assistant => _migrate-or-upgrade/migration-assistant}/migration-phases/backfill.md (100%) rename {_migration-assistant => _migrate-or-upgrade/migration-assistant}/migration-phases/create-snapshot.md (97%) rename _migration-assistant/migration-phases/deployment.md => _migrate-or-upgrade/migration-assistant/migration-phases/deploy.md (99%) rename {_migration-assistant => _migrate-or-upgrade/migration-assistant}/migration-phases/deploying-migration-assistant/configuration-options.md (100%) rename {_migration-assistant => _migrate-or-upgrade/migration-assistant}/migration-phases/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md (100%) rename {_migration-assistant => _migrate-or-upgrade/migration-assistant}/migration-phases/index.md (79%) rename {_migration-assistant => _migrate-or-upgrade/migration-assistant}/migration-phases/live-traffic-migration/index.md (84%) rename {_migration-assistant => _migrate-or-upgrade/migration-assistant}/migration-phases/live-traffic-migration/switching-traffic-from-the-source-cluster.md (100%) rename _migration-assistant/migration-phases/migrating-metadata.md => _migrate-or-upgrade/migration-assistant/migration-phases/migrate-metadata.md (99%) rename _migration-assistant/migration-phases/removing-migration-infrastructure.md => _migrate-or-upgrade/migration-assistant/migration-phases/remove-migration-infrastructure.md (64%) rename {_migration-assistant => _migrate-or-upgrade/migration-assistant}/migration-phases/reroute-source-to-proxy.md (98%) rename {_migration-assistant => _migrate-or-upgrade/migration-assistant}/migration-phases/reroute-traffic-from-capture-proxy-to-target.md (96%) rename {_migration-assistant => _migrate-or-upgrade/migration-assistant}/migration-phases/using-traffic-replayer.md (100%) rename {_migration-assistant => _migrate-or-upgrade/migration-assistant}/migration-phases/verifying-migration-tools/verifying-backfill-components.md (99%) rename {_migration-assistant => _migrate-or-upgrade/migration-assistant}/migration-phases/verifying-migration-tools/verifying-live-capture-components.md (98%) rename {_migration-assistant => _migrate-or-upgrade/migration-assistant}/migration-phases/verifying-migration-tools/verifying-migration-tools.md (85%) rename {_migration-assistant => _migrate-or-upgrade/migration-assistant}/planning-your-migration/handling-field-type-breaking-changes.md (100%) rename {_migration-assistant => _migrate-or-upgrade/migration-assistant}/planning-your-migration/handling-type-mapping-deprecation.md (100%) rename {_install-and-configure/upgrade-opensearch => _migrate-or-upgrade/rolling-upgrade}/appendix/index.md (100%) rename {_install-and-configure/upgrade-opensearch => _migrate-or-upgrade/rolling-upgrade}/appendix/rolling-upgrade-lab.md (99%) rename _install-and-configure/upgrade-opensearch/rolling-upgrade.md => _migrate-or-upgrade/rolling-upgrade/index.md (95%) delete mode 100644 _migration-assistant/migration-phases/deploying-migration-assistant/getting-started-data-migration.md delete mode 100644 _migration-assistant/overview/index.md delete mode 100644 _migration-assistant/planning-your-migration/index.md diff --git a/_config.yml b/_config.yml index e69c593e1c0..7c81dbcc3d1 100644 --- a/_config.yml +++ b/_config.yml @@ -224,7 +224,7 @@ clients_collection: migration_assistant_collection: collections: migration-assistant: - name: Migration Assistant + name: Migration Assistant for OpenSearch nav_fold: true benchmark_collection: @@ -268,10 +268,10 @@ defaults: section-name: "Benchmark" - scope: - path: "_migration-assistant" + path: "_migrate-or-upgrade/migration-assistant" values: - section: "migrations-assistant" - section-name: "Migration Assistant" + section: "migration-assistant" + section-name: "Migration Assistant for OpenSearch" # Enable or disable the site search # By default, just-the-docs enables its JSON file-based search. We also have an OpenSearch-driven search functionality. diff --git a/_includes/home_cards.html b/_includes/home_cards.html index 4065cf27e0c..ebd4eecd802 100644 --- a/_includes/home_cards.html +++ b/_includes/home_cards.html @@ -62,7 +62,7 @@
        -

        Migration Asssistant

        +

        Migration Assistant

        Simplify your upgrade or migration to OpenSearch

        Documentation →

        diff --git a/_install-and-configure/configuring-opensearch/index.md b/_install-and-configure/configuring-opensearch/index.md index c2ffbf571ba..f23c1d24e1d 100755 --- a/_install-and-configure/configuring-opensearch/index.md +++ b/_install-and-configure/configuring-opensearch/index.md @@ -3,9 +3,10 @@ layout: default title: Configuring OpenSearch nav_order: 10 has_children: true +permalink: /install-and-configure/configuring-opensearch/ redirect_from: - - /install-and-configure/configuring-opensearch/ - /opensearch/configuration/ + - /install-and-configure/configuring-opensearch/index/ --- # Configuring OpenSearch diff --git a/_install-and-configure/index.md b/_install-and-configure/index.md index 4fedacfdf16..2c707520ef9 100644 --- a/_install-and-configure/index.md +++ b/_install-and-configure/index.md @@ -28,8 +28,4 @@ OpenSearch and OpenSearch Dashboards are available on any compatible host that s After you've installed OpenSearch, learn about [configuring]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/) it for your deployment. -For more information about upgrading your OpenSearch cluster, see the [upgrade guide]({{site.url}}{{site.baseurl}}/install-and-configure/upgrade-opensearch/index/). - -For information about upgrade tools, see [OpenSearch upgrade, migration, and comparison tools]({{site.url}}{{site.baseurl}}/tools/index/#opensearch-upgrade-migration-and-comparison-tools). - For plugin installation, see [Installing plugins]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/). \ No newline at end of file diff --git a/_install-and-configure/upgrade-opensearch/index.md b/_install-and-configure/upgrade-opensearch/index.md deleted file mode 100644 index 8f13ea18422..00000000000 --- a/_install-and-configure/upgrade-opensearch/index.md +++ /dev/null @@ -1,271 +0,0 @@ ---- -layout: default -title: Upgrading OpenSearch -nav_order: 20 -has_children: true -redirect_from: - - /upgrade-opensearch/index/ ---- - -# Upgrading OpenSearch - -The OpenSearch Project releases regular updates that include new features, enhancements, and bug fixes. OpenSearch uses [Semantic Versioning](https://semver.org/), which means that breaking changes are only introduced between major version releases. To learn about upcoming features and fixes, review the [OpenSearch Project Roadmap](https://github.com/orgs/opensearch-project/projects/206) on GitHub. To view a list of previous releases or to learn more about how OpenSearch uses versioning, see [Release Schedule and Maintenance Policy]({{site.url}}/releases.html). - -We recognize that users are excited about upgrading OpenSearch in order to enjoy the latest features, and we will continue to expand on these upgrade and migration documents to cover additional topics, such as upgrading OpenSearch Dashboards and preserving custom configurations, such as for plugins. - -If you would like a specific process to be added or would like to contribute, [create an issue](https://github.com/opensearch-project/documentation-website/issues) on GitHub. See the [Contributor Guidelines](https://github.com/opensearch-project/documentation-website/blob/main/CONTRIBUTING.md) to learn how you can help. -{: .tip} - -## Workflow considerations - -Take time to plan the process before making any changes to your cluster. For example, consider the following questions: - -- How long will the upgrade process take? -- If your cluster is being used in production, how impactful is downtime? -- Do you have infrastructure in place to stand up the new cluster in a testing or development environment before you move it into production, or do you need to upgrade the production hosts directly? - -The answers to questions like these will help you determine which upgrade path will work best in your environment. - -At a minimum, you should be: - -- [Reviewing breaking changes](#reviewing-breaking-changes). -- [Reviewing the OpenSearch tools compatibility matrices](#reviewing-the-opensearch-tools-compatibility-matrices). -- [Reviewing plugin compatibility](#reviewing-plugin-compatibility). -- [Backing up configuration files](#backing-up-configuration-files). -- [Creating a snapshot](#creating-a-snapshot). - -Stop any nonessential indexing before you begin the upgrade procedure to eliminate unnecessary resource demands on the cluster while you perform the upgrade. -{: .tip} - -### Reviewing breaking changes - -It's important to determine how the new version of OpenSearch will integrate with your environment. Review [Breaking changes]({{site.url}}{{site.baseurl}}/breaking-changes/) before beginning any upgrade procedures to determine whether you will need to make adjustments to your workflow. For example, upstream or downstream components might need to be modified to be compatible with an API change (see meta issue [#2589](https://github.com/opensearch-project/OpenSearch/issues/2589)). - -### Reviewing the OpenSearch tools compatibility matrices - -If your OpenSearch cluster interacts with other services in your environment, like Logstash or Beats, then you should check the [OpenSearch tools compatibility matrices]({{site.url}}{{site.baseurl}}/tools/index/#compatibility-matrices) to determine whether other components will need to be upgraded. - -### Reviewing plugin compatibility - -Review the plugins you use to determine compatibility with the target version of OpenSearch. Official OpenSearch Project plugins can be found in the [OpenSearch Project](https://github.com/opensearch-project) repository on GitHub. If you use any third-party plugins, then you should check the documentation for those plugins to determine whether they are compatible. - -Go to [Available plugins]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/#available-plugins) to see a reference table that highlights version compatibility for bundled OpenSearch plugins. - -Major, minor, and patch plugin versions must match OpenSearch major, minor, and patch versions in order to be compatible. For example, plugin versions 2.3.0.x work only with OpenSearch 2.3.0. -{: .important} - -### Backing up configuration files - -Mitigate the risk of data loss by backing up any important files before you start an upgrade. Generally, these files will be located in either of two directories: - -- `opensearch/config` -- `opensearch-dashboards/config` - -Some examples include `opensearch.yml`, `opensearch_dashboards.yml`, plugin configuration files, and TLS certificates. Once you identify which files you want to back up, copy them to remote storage for safety. - -If you use security features, make sure to read [A word of caution]({{site.url}}{{site.baseurl}}/security-plugin/configuration/security-admin/#a-word-of-caution) for information about backing up and restoring your security settings. - -### Creating a snapshot - -We recommend that you back up your cluster state and indexes using [snapshots]({{site.url}}{{site.baseurl}}/opensearch/snapshots/index/). Snapshots you take before an upgrade can be used as restore points if you need to roll back the cluster to its original version. - -You can further reduce the risk of data loss by storing your snapshots on external storage, such as a mounted Network File System (NFS) or a cloud storage solution like those listed in the following table. - -| Snapshot repository location | Required OpenSearch plugin | -| :--- | :--- | -| [Amazon Simple Storage Service (Amazon S3)](https://aws.amazon.com/s3/) | [repository-s3](https://github.com/opensearch-project/OpenSearch/tree/{{site.opensearch_version}}/plugins/repository-s3) | -| [Google Cloud Storage (GCS)](https://cloud.google.com/storage) | [repository-gcs](https://github.com/opensearch-project/OpenSearch/tree/{{site.opensearch_version}}/plugins/repository-gcs) | -| [Apache Hadoop Distributed File System (HDFS)](https://hadoop.apache.org/) | [repository-hdfs](https://github.com/opensearch-project/OpenSearch/tree/{{site.opensearch_version}}/plugins/repository-hdfs) | -| [Microsoft Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs) | [repository-azure](https://github.com/opensearch-project/OpenSearch/tree/{{site.opensearch_version}}/plugins/repository-azure) | - -## Upgrade methods - -Choose an appropriate method for upgrading your cluster to a new version of OpenSearch based on your requirements: - -- A [rolling upgrade](#rolling-upgrade) upgrades nodes one at a time without stopping the cluster. -- A [cluster restart upgrade](#cluster-restart-upgrade) upgrades services while the cluster is stopped. - -Upgrades spanning more than a single major version of OpenSearch will require additional effort due to the need for reindexing. For more information, refer to the [Reindex]({{site.url}}{{site.baseurl}}/api-reference/document-apis/reindex/) API. See the [Index compatibility reference](#index-compatibility-reference) table included later in this guide for help planning your data migration. - -### Rolling upgrade - -A rolling upgrade is a great option if you want to keep your cluster operational throughout the process. Data may continue to be ingested, analyzed, and queried as nodes are individually stopped, upgraded, and restarted. A variation of the rolling upgrade referred to as "node replacement" follows exactly the same process except that hosts and containers are not reused for the new node. You might perform node replacement if you are upgrading the underlying host(s) as well. - -OpenSearch nodes cannot join a cluster if the cluster manager is running a newer version of OpenSearch than the node requesting membership. To avoid this issue, upgrade the cluster-manager-eligible nodes last. - - -See [Rolling Upgrade]({{site.url}}{{site.baseurl}}/install-and-configure/upgrade-opensearch/rolling-upgrade/) for more information about the process. - -### Cluster restart upgrade - -OpenSearch administrators might choose to perform a cluster restart upgrade for several reasons, such as if the administrator doesn't want to perform maintenance on a running cluster or if the cluster is being migrated to a different environment. - -Unlike a rolling upgrade, where only one node is offline at a time, a cluster restart upgrade requires you to stop OpenSearch and OpenSearch Dashboards on all nodes in the cluster before proceeding. After the nodes are stopped, a new version of OpenSearch is installed. Then OpenSearch is started and the cluster bootstraps to the new version. - -## Compatibility - -OpenSearch nodes are compatible with other OpenSearch nodes running any other *minor* version within the same *major* version release. For example, 1.1.0 is compatible with 1.3.7 because they are part of the same *major* version (1.x). Additionally, OpenSearch nodes and indexes are backward compatible with the previous major version. That means, for example, that an index created by an OpenSearch node running any 1.x version can be restored from a snapshot to an OpenSearch cluster running any 2.x version. - -OpenSearch 1.x nodes are compatible with nodes running Elasticsearch 7.x, but the longevity of a mixed-version environment should not extend beyond cluster upgrade activities. -{: .tip} - -Index compatibility is determined by the version of [Apache Lucene](https://lucene.apache.org/) that created the index. If an index was created by an OpenSearch cluster running version 1.0.0, then the index can be used by any other OpenSearch cluster running up to the latest 1.x or 2.x release. See the [Index compatibility reference](#index-compatibility-reference) table for Lucene versions running in OpenSearch 1.0.0 and later and [Elasticsearch](https://www.elastic.co/) 6.8 and later. - -If your upgrade path spans more than a single major version and you want to retain any existing indexes, then you can use the [Reindex]({{site.url}}{{site.baseurl}}/api-reference/document-apis/reindex/) API to make your indexes compatible with the target version of OpenSearch before upgrading. For example, if your cluster is currently running Elasticsearch 6.8 and you want to upgrade to OpenSearch 2.x, then you must first upgrade to OpenSearch 1.x, recreate your indexes using the [Reindex]({{site.url}}{{site.baseurl}}/api-reference/document-apis/reindex/) API, and finally upgrade to 2.x. One alternative to reindexing is to reingest data from the origin, such as by replaying a data stream or ingesting data from a database. - -### Index compatibility reference - -If you plan to retain old indexes after the OpenSearch version upgrade, then you might need to reindex or reingest the data. Refer to the following table for Lucene versions across recent OpenSearch and Elasticsearch releases. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
        Lucene VersionOpenSearch VersionElasticsearch Version
        9.10.02.14.0
        2.13.0        		8.13
        9.9.22.12.0—
        9.7.02.11.1
        2.9.0        		8.9.0
        9.6.02.8.08.8.0
        9.5.02.7.0
        2.6.0        		8.7.0
        9.4.22.5.0
        2.4.1        		8.6
        9.4.12.4.0—
        9.4.0—8.5
        9.3.02.3.0
        2.2.x        		8.4
        9.2.02.1.08.3
        9.1.02.0.x8.2
        9.0.0—8.1
        8.0
        8.11.1—7.17
        8.10.11.3.x
        1.2.x        		7.16
        8.9.01.1.07.15
        7.14
        8.8.21.0.07.13
        8.8.0—7.12
        8.7.0—7.11
        7.10
        8.6.2—7.9
        8.5.1—7.8
        7.7
        8.4.0—7.6
        8.3.0—7.5
        8.2.0—7.4
        8.1.0—7.3
        8.0.0—7.2
        7.1
        7.7.3—6.8
        -

        A dash (—) indicates that there is no product version containing the specified version of Apache Lucene.

        diff --git a/_migrate-or-upgrade/index.md b/_migrate-or-upgrade/index.md index 4c98ea50a38..d08a8844588 100644 --- a/_migrate-or-upgrade/index.md +++ b/_migrate-or-upgrade/index.md @@ -1,121 +1,201 @@ --- layout: default -title: Migrate or upgrade -nav_order: 1 -has_children: false -nav_exclude: true -has_toc: false -permalink: /migrate-or-upgrade/ +title: Upgrade or Migrate OpenSearch +nav_order: 20 +has_children: true +permalink: /upgrade-or-migrate/ redirect_from: + - /migrate-or-upgrade/index/ + - /upgrade-opensearch/index/ + - /migrate-or-upgrade/ - /upgrade-to/index/ - /upgrade-to/ - /upgrade-to/upgrade-to/ -tutorial_cards: - - heading: "Overview" - description: "Get familiar with the key components of Migration Assistant and evaluate your use case." - link: "/migration-assistant/overview/" - - heading: "Migration phases" - description: "Execute your migration in phases—metadata, backfill, and traffic replay—for a controlled and validated transition." - link: "/migration-assistant/migration-phases/" +has_toc: true --- +# Upgrade or Migrate OpenSearch -# Migration and upgrade options +The OpenSearch Project releases regular updates that include new features, enhancements, and bug fixes. OpenSearch uses [Semantic Versioning](https://semver.org/), which means that breaking changes are only introduced between major version releases. To learn about upcoming features and fixes, review the [OpenSearch Project Roadmap](https://github.com/orgs/opensearch-project/projects/206) on GitHub. To view a list of previous releases or to learn more about how OpenSearch uses versioning, see [Release Schedule and Maintenance Policy]({{site.url}}/releases.html). Upgrading or migrating OpenSearch is essential for maintaining optimal performance, security, and access to the latest features. Whether you're upgrading an existing OpenSearch deployment or migrating from another system such as Elasticsearch OSS, choosing the right approach is critical to a successful transition. -This page outlines four primary methods for upgrading or migrating OpenSearch clusters—each with distinct benefits and trade-offs. These methods include rolling upgrades, snapshot and restore, Migration Assistant, and remote reindexing. Use this guide to evaluate which option best fits your data size, infrastructure, and operational requirements. +This page outlines upgrade planning guidance and four supported methods: rolling upgrades, snapshot and restore, remote reindexing, and using the OpenSearch Migration Assistant. -## Migration Assistant +--- + +## Before You Begin + +Take time to plan the process before making any changes to your cluster: + +- How long will the upgrade process take? +- Can your system tolerate downtime? +- Do you have the ability to test in a staging environment? -The Migration Assistant is a comprehensive tool designed to simplify upgrades by automating the process and supporting live data capture. +Make sure to: + +- Review [breaking changes]({{site.url}}{{site.baseurl}}/breaking-changes/) +- Check [plugin compatibility]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/#available-plugins) +- Review [OpenSearch tools compatibility matrices]({{site.url}}{{site.baseurl}}/tools/index/#compatibility-matrices) +- Back up [configuration files](#backing-up-configuration-files) +- Take a [snapshot](#creating-a-snapshot) + +Stop nonessential indexing before upgrading. +{: .tip} + +--- + +## Migration and Upgrade Methods + +### Migration Assistant + +The Migration Assistant provides the most automated and resilient upgrade path. **Pros:** -- Supports multi-version upgrades in a single migration. -- Enables near-zero downtime using live data capture. -- Includes rollback support to revert changes if issues occur. -- Integrates with existing OpenSearch tooling for a streamlined experience. +- Supports multi-version hops +- Enables near-zero downtime via live data capture +- Includes rollback support +- Designed to integrate with OpenSearch workflows **Cons:** -- Requires additional setup and infrastructure. +- Requires setup and infrastructure -### Why choose Migration Assistant? +[Get started with Migration Assistant]({{site.url}}{{site.baseurl}}/migration-assistant/) -Migration Assistant offers the most automated and resilient path for OpenSearch upgrades, especially for complex environments and large version gaps. +--- -- **Multi-version hops:** Avoids sequential upgrades by migrating across multiple versions in one step. -- **Live data capture:** Maintains service continuity by syncing changes during the migration. -- **Reversion support:** Provides a fallback option in case of errors or issues. -- **Integrated automation:** Designed to fit into OpenSearch workflows for a guided upgrade experience. +### Rolling Upgrade + +Upgrade one node at a time while keeping the cluster operational. + +**Pros:** +- Minimal downtime +- No new infrastructure needed +**Cons:** +- Supports only adjacent major versions +- Multiple upgrade cycles for larger version gaps +- Reindexing may be required -### Getting started with Migration Assistant +[Perform a rolling upgrade]({{site.url}}{{site.baseurl}}/install-and-configure/migrate-or-upgrade/rolling-upgrade/) -- Upgrade or migrate with [Migration Assistant for OpenSearch]({{site.url}}{{site.baseurl}}/migration-assistant/) +--- -## Rolling upgrades +### Snapshot and Restore -Rolling upgrades allow you to upgrade one node at a time, keeping the cluster operational throughout the process. Note that this is only an upgrade option. +Take a snapshot of your current cluster and restore to a new OpenSearch version. **Pros:** -- Minimal downtime; the cluster remains available during the upgrade. -- No new infrastructure is required. +- Leaves source cluster unchanged +- Supports large datasets and cold storage **Cons:** -- Only supports adjacent major version upgrades. -- Incompatibilities may still arise, requiring a snapshot restore that can be complex and risky. -- Multiple upgrade cycles are needed for large version jumps. -- Manual reindexing may be required to enable newer features. +- Requires downtime or a change data capture (CDC) solution +- Requires provisioning a new cluster -### Getting started with rolling upgrades - -- Perform a [rolling upgrade]({{site.url}}{{site.baseurl}}/install-and-configure/upgrade-opensearch/rolling-upgrade/). +[Get started with Snapshot and Restore](https://docs.aws.amazon.com/solutions/latest/tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore/) +--- -## Snapshot and restore +### Remote Reindexing -This method involves taking a snapshot of your existing OpenSearch or Elasticsearch OSS cluster and restoring it to a new cluster running the target version. +Reindex data from the old cluster into the new OpenSearch cluster. **Pros:** -- Original cluster remains untouched, allowing rollback if needed (note: may result in data loss without a change data capture (CDC) solution. In some cases, one may choose to use `Capture-and-Replay` from Migration Assistant in combination with Snapshot and Restore). -- Scales well for large data volumes, including cold storage and datasets up to 1 PB. +- No downtime +- Supports large version jumps **Cons:** -- Requires downtime or an external CDC solution. -- Requires provisioning a new cluster. -- Manual reindexing may be necessary for full feature support. +- Slower and more resource-intensive +- Can degrade source cluster performance -### Get started with Snapshot and Restore +Learn more in the [Reindex API docs]({{site.url}}{{site.baseurl}}/api-reference/document-apis/reindex/) -Upgrade or Migration with [Snapshot and Restore](https://docs.aws.amazon.com/solutions/latest/ tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore/) +--- +### Reviewing the OpenSearch tools compatibility matrices -## Remote reindexing +If your OpenSearch cluster interacts with other services in your environment, like Logstash or Beats, then you should check the [OpenSearch tools compatibility matrices]({{site.url}}{{site.baseurl}}/tools/index/#compatibility-matrices) to determine whether other components will need to be upgraded. -This method reindexes data from your current cluster into a new OpenSearch cluster, typically running in parallel. +### Reviewing plugin compatibility -## Other migration options +Review the plugins you use to determine compatibility with the target version of OpenSearch. Official OpenSearch Project plugins can be found in the [OpenSearch Project](https://github.com/opensearch-project) repository on GitHub. If you use any third-party plugins, then you should check the documentation for those plugins to determine whether they are compatible. -Additional migration strategies not covered in detail in this documentation include rebuilding your target cluster from source systems and using traffic replication to mirror production traffic during migration. +Go to [Available plugins]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/#available-plugins) to see a reference table that highlights version compatibility for bundled OpenSearch plugins. -**Pros:** -- No downtime; clusters operate concurrently. -- Supports upgrades across multiple versions. +Major, minor, and patch plugin versions must match OpenSearch major, minor, and patch versions in order to be compatible. For example, plugin versions 2.3.0.x work only with OpenSearch 2.3.0. +{: .important} -**Cons:** -- Slow and resource-intensive, not recommended for data sets over 1 GB. -- May impact performance of the source cluster. -- Requires careful configuration and a compatible target cluster. -## Why choose Migration Assistant? +--- -Migration Assistant offers the most automated and resilient path for OpenSearch upgrades, especially for complex environments and large version gaps. +## Backing Up Configuration Files -- **Multi-version hops:** Avoids sequential upgrades by migrating across multiple versions in one step. -- **Live data capture:** Maintains service continuity by syncing changes during the migration. -- **Reversion support:** Provides a fallback option in case of errors or issues. -- **Integrated automation:** Designed to fit into OpenSearch workflows for a guided upgrade experience. +Back up important files such as `opensearch.yml`, plugin configs, and TLS certs from these paths: -## Before you begin +- `opensearch/config` +- `opensearch-dashboards/config` -Before choosing a method, make sure that your OpenSearch clients and plugins are compatible with the target version. For example, tools like Logstash OSS and Filebeat OSS may enforce version checks that impact upgrade paths. +See [security configuration guidance]({{site.url}}{{site.baseurl}}/security-plugin/configuration/security-admin/#a-word-of-caution). +--- + +## Creating a Snapshot + +We recommend that you back up your cluster state and indexes using [snapshots]({{site.url}}{{site.baseurl}}/opensearch/snapshots/index/). Snapshots you take before an upgrade can be used as restore points if you need to roll back the cluster to its original version. + +You can further reduce the risk of data loss by storing your snapshots on external storage, such as a mounted Network File System (NFS) or a cloud storage solution like those listed in the following table. + +| Repository | Plugin | +| --- | --- | +| [Amazon S3](https://aws.amazon.com/s3/) | [repository-s3](https://github.com/opensearch-project/OpenSearch/tree/{{site.opensearch_version}}/plugins/repository-s3) | +| [Google Cloud Storage](https://cloud.google.com/storage) | [repository-gcs](https://github.com/opensearch-project/OpenSearch/tree/{{site.opensearch_version}}/plugins/repository-gcs) | +| [HDFS](https://hadoop.apache.org/) | [repository-hdfs](https://github.com/opensearch-project/OpenSearch/tree/{{site.opensearch_version}}/plugins/repository-hdfs) | +| [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs) | [repository-azure](https://github.com/opensearch-project/OpenSearch/tree/{{site.opensearch_version}}/plugins/repository-azure) | + +--- +## Index Compatibility Reference + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
        Lucene VersionOpenSearch VersionElasticsearch Version
        9.10.02.14.0
        2.13.0        		8.13
        9.9.22.12.0—
        9.7.02.11.1
        2.9.0        		8.9.0
        9.6.02.8.08.8.0
        9.5.02.7.0
        2.6.0        		8.7.0
        9.4.22.5.0
        2.4.1        		8.6
        9.4.12.4.0—
        9.4.0—8.5
        9.3.02.3.0
        2.2.x        		8.4
        9.2.02.1.08.3
        9.1.02.0.x8.2
        9.0.0—8.1
        8.0
        8.11.1—7.17
        8.10.11.3.x
        1.2.x        		7.16
        8.9.01.1.07.15
        7.14
        8.8.21.0.07.13
        8.8.0—7.12
        8.7.0—7.11
        7.10
        8.6.2—7.9
        8.5.1—7.8
        7.7
        8.4.0—7.6
        8.3.0—7.5
        8.2.0—7.4
        8.1.0—7.3
        8.0.0—7.2
        7.1
        7.7.3—6.8
        +

        A dash (—) indicates no release contains the listed Lucene version.

        \ No newline at end of file diff --git a/_migration-assistant/overview/architecture.md b/_migrate-or-upgrade/migration-assistant/architecture.md similarity index 94% rename from _migration-assistant/overview/architecture.md rename to _migrate-or-upgrade/migration-assistant/architecture.md index f768d3fa646..ac81532cad9 100644 --- a/_migration-assistant/overview/architecture.md +++ b/_migrate-or-upgrade/migration-assistant/architecture.md @@ -2,8 +2,8 @@ layout: default title: Architecture nav_order: 15 -parent: Overview -permalink: /migration-assistant/overview/architecture/ +parent: Migration Assistant for OpenSearch +permalink: /migration-assistant/architecture/ --- # Architecture diff --git a/_migration-assistant/assets/css/breaking-changes-selector.css b/_migrate-or-upgrade/migration-assistant/assets/css/breaking-changes-selector.css similarity index 100% rename from _migration-assistant/assets/css/breaking-changes-selector.css rename to _migrate-or-upgrade/migration-assistant/assets/css/breaking-changes-selector.css diff --git a/_migration-assistant/assets/js/breaking-changes-data.js b/_migrate-or-upgrade/migration-assistant/assets/js/breaking-changes-data.js similarity index 100% rename from _migration-assistant/assets/js/breaking-changes-data.js rename to _migrate-or-upgrade/migration-assistant/assets/js/breaking-changes-data.js diff --git a/_migration-assistant/assets/js/breaking-changes-filter.js b/_migrate-or-upgrade/migration-assistant/assets/js/breaking-changes-filter.js similarity index 100% rename from _migration-assistant/assets/js/breaking-changes-filter.js rename to _migrate-or-upgrade/migration-assistant/assets/js/breaking-changes-filter.js diff --git a/_migration-assistant/assets/js/breaking-changes-index.js b/_migrate-or-upgrade/migration-assistant/assets/js/breaking-changes-index.js similarity index 100% rename from _migration-assistant/assets/js/breaking-changes-index.js rename to _migrate-or-upgrade/migration-assistant/assets/js/breaking-changes-index.js diff --git a/_migration-assistant/assets/js/breaking-changes-module.js b/_migrate-or-upgrade/migration-assistant/assets/js/breaking-changes-module.js similarity index 100% rename from _migration-assistant/assets/js/breaking-changes-module.js rename to _migrate-or-upgrade/migration-assistant/assets/js/breaking-changes-module.js diff --git a/_migration-assistant/assets/js/breaking-changes-ui.js b/_migrate-or-upgrade/migration-assistant/assets/js/breaking-changes-ui.js similarity index 100% rename from _migration-assistant/assets/js/breaking-changes-ui.js rename to _migrate-or-upgrade/migration-assistant/assets/js/breaking-changes-ui.js diff --git a/_migration-assistant/index.md b/_migrate-or-upgrade/migration-assistant/index.md similarity index 84% rename from _migration-assistant/index.md rename to _migrate-or-upgrade/migration-assistant/index.md index 78d8b743ddd..5ab4800527b 100644 --- a/_migration-assistant/index.md +++ b/_migrate-or-upgrade/migration-assistant/index.md @@ -1,24 +1,24 @@ --- layout: default title: Migration Assistant for OpenSearch -nav_order: 1 -has_children: false -nav_exclude: true -has_toc: false +nav_order: 2 +has_toc: true permalink: /migration-assistant/ redirect_from: + - /upgrade-or-migrate/migration-assistant/ - /migration-assistant/index/ + - /migrate-or-upgrade/migration-assistant/index/ - /upgrade-to/snapshot-migrate/ items: - heading: "Is Migration Assistant right for you?" description: "Evaluate whether Migration Assistant is right for your use case." - link: "/migration-assistant/overview/is-migration-assistant-right-for-you/" + link: "/migration-assistant/is-migration-assistant-right-for-you/" - heading: "Key components" description: "Get familiar with the key components of Migration Assistant." - link: "/migration-assistant/overview/key-components/" + link: "/migration-assistant/key-components/" - heading: "Architecture" description: "Understand how Migration Assistant integrates into your infrastructure." - link: "/migration-assistant/overview/architecture/" + link: "/migration-assistant/architecture/" - heading: "Execute your migration in phases" description: "A step-by-step guide for performing a migration." link: "/migration-assistant/migration-phases" diff --git a/_migration-assistant/overview/is-migration-assistant-right-for-you.md b/_migrate-or-upgrade/migration-assistant/is-migration-assistant-right-for-you.md similarity index 98% rename from _migration-assistant/overview/is-migration-assistant-right-for-you.md rename to _migrate-or-upgrade/migration-assistant/is-migration-assistant-right-for-you.md index dcca20f553f..e315e495b6e 100644 --- a/_migration-assistant/overview/is-migration-assistant-right-for-you.md +++ b/_migrate-or-upgrade/migration-assistant/is-migration-assistant-right-for-you.md @@ -2,10 +2,9 @@ layout: default title: Is Migration Assistant right for you? nav_order: 10 -parent: Overview -permalink: /migration-assistant/overview/is-migration-assistant-right-for-you/ -redirect_from: - - /migration-assistant/is-migration-assistant-right-for-you/ +nav_exclude: false +parent: Migration Assistant for penSearch +permalink: /migration-assistant/is-migration-assistant-right-for-you/ --- # Is Migration Assistant right for you? diff --git a/_migration-assistant/overview/key-components.md b/_migrate-or-upgrade/migration-assistant/key-components.md similarity index 95% rename from _migration-assistant/overview/key-components.md rename to _migrate-or-upgrade/migration-assistant/key-components.md index 6e5aa34db44..1b7201d804d 100644 --- a/_migration-assistant/overview/key-components.md +++ b/_migrate-or-upgrade/migration-assistant/key-components.md @@ -1,9 +1,9 @@ --- layout: default +parent: Migration Assistant for OpenSearch title: Key components -nav_order: 10 -parent: Overview -permalink: /migration-assistant/overview/key-components/ +nav_order: 20 +permalink: /migration-assistant/key-components/ --- # Key components diff --git a/_migration-assistant/migration-console/accessing-the-migration-console.md b/_migrate-or-upgrade/migration-assistant/migration-console/accessing-the-migration-console.md similarity index 100% rename from _migration-assistant/migration-console/accessing-the-migration-console.md rename to _migrate-or-upgrade/migration-assistant/migration-console/accessing-the-migration-console.md diff --git a/_migration-assistant/migration-console/index.md b/_migrate-or-upgrade/migration-assistant/migration-console/index.md similarity index 80% rename from _migration-assistant/migration-console/index.md rename to _migrate-or-upgrade/migration-assistant/migration-console/index.md index 64f87bc0fc5..2d200c56fa2 100644 --- a/_migration-assistant/migration-console/index.md +++ b/_migrate-or-upgrade/migration-assistant/migration-console/index.md @@ -1,11 +1,12 @@ --- layout: default +parent: Migration Assistant for OpenSearch title: Migration console nav_order: 50 has_children: true -permalink: /migration-console/ +permalink: /migration-assistant/migration-console/ redirect_from: - - /migration-console/index/ + - /migration-assistant/migration-console/index/ --- # Migration console diff --git a/_migration-assistant/migration-console/migration-console-commands-references.md b/_migrate-or-upgrade/migration-assistant/migration-console/migration-console-commands-references.md similarity index 100% rename from _migration-assistant/migration-console/migration-console-commands-references.md rename to _migrate-or-upgrade/migration-assistant/migration-console/migration-console-commands-references.md diff --git a/_migration-assistant/migration-phases/accessing-the-migration-console.md b/_migrate-or-upgrade/migration-assistant/migration-phases/accessing-the-migration-console.md similarity index 100% rename from _migration-assistant/migration-phases/accessing-the-migration-console.md rename to _migrate-or-upgrade/migration-assistant/migration-phases/accessing-the-migration-console.md diff --git a/_migration-assistant/migration-phases/assessing-your-cluster-for-migration.md b/_migrate-or-upgrade/migration-assistant/migration-phases/assessing-your-cluster-for-migration.md similarity index 98% rename from _migration-assistant/migration-phases/assessing-your-cluster-for-migration.md rename to _migrate-or-upgrade/migration-assistant/migration-phases/assessing-your-cluster-for-migration.md index c42029fc6ec..625481fe18f 100644 --- a/_migration-assistant/migration-phases/assessing-your-cluster-for-migration.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/assessing-your-cluster-for-migration.md @@ -3,6 +3,7 @@ layout: default title: Assessing your cluster for migration nav_order: 1 parent: Migration phases +permalink: /migration-assistant/migration-phases/assessment/ --- # Assessing your cluster for migration diff --git a/_migration-assistant/migration-phases/backfill.md b/_migrate-or-upgrade/migration-assistant/migration-phases/backfill.md similarity index 100% rename from _migration-assistant/migration-phases/backfill.md rename to _migrate-or-upgrade/migration-assistant/migration-phases/backfill.md diff --git a/_migration-assistant/migration-phases/create-snapshot.md b/_migrate-or-upgrade/migration-assistant/migration-phases/create-snapshot.md similarity index 97% rename from _migration-assistant/migration-phases/create-snapshot.md rename to _migrate-or-upgrade/migration-assistant/migration-phases/create-snapshot.md index 3e8c5ddcf1c..2ea1f77142a 100644 --- a/_migration-assistant/migration-phases/create-snapshot.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/create-snapshot.md @@ -1,8 +1,9 @@ --- layout: default -title: Snapshot -nav_order: 4 +title: Creating a snapshot parent: Migration phases +nav_order: 4 +permalink: /migration-assistant/migration-phases/create-snapshot/ --- # Creating a snapshot @@ -240,5 +241,4 @@ As Metadata migration supports migrating from ES 6.8 on to the latest versions o For additional technical details, [view the mapping type removal source code](https://github.com/opensearch-project/opensearch-migrations/blob/main/transformation/src/main/java/org/opensearch/migrations/transformation/rules/IndexMappingTypeRemoval.java). -TODO: Add troublshooting -
      5. Verify Backfill Components
        6. \ No newline at end of file + \ No newline at end of file diff --git a/_migration-assistant/migration-phases/deployment.md b/_migrate-or-upgrade/migration-assistant/migration-phases/deploy.md similarity index 99% rename from _migration-assistant/migration-phases/deployment.md rename to _migrate-or-upgrade/migration-assistant/migration-phases/deploy.md index 8fa097afd6a..2058db7d228 100644 --- a/_migration-assistant/migration-phases/deployment.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/deploy.md @@ -3,9 +3,9 @@ layout: default title: Deploying Migration Assistant parent: Migration phases nav_order: 2 +permalink: /migration-assistant/migration-phases/deploy/ redirect_from: - /migration-assistant/getting-started-with-data-migration/ - - /deploying-migration-assistant/ --- # Deploying Migration Assistant diff --git a/_migration-assistant/migration-phases/deploying-migration-assistant/configuration-options.md b/_migrate-or-upgrade/migration-assistant/migration-phases/deploying-migration-assistant/configuration-options.md similarity index 100% rename from _migration-assistant/migration-phases/deploying-migration-assistant/configuration-options.md rename to _migrate-or-upgrade/migration-assistant/migration-phases/deploying-migration-assistant/configuration-options.md diff --git a/_migration-assistant/migration-phases/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md b/_migrate-or-upgrade/migration-assistant/migration-phases/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md similarity index 100% rename from _migration-assistant/migration-phases/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md rename to _migrate-or-upgrade/migration-assistant/migration-phases/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md diff --git a/_migration-assistant/migration-phases/index.md b/_migrate-or-upgrade/migration-assistant/migration-phases/index.md similarity index 79% rename from _migration-assistant/migration-phases/index.md rename to _migrate-or-upgrade/migration-assistant/migration-phases/index.md index 937e40e5d1c..ab2957b6ec4 100644 --- a/_migration-assistant/migration-phases/index.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/index.md @@ -1,10 +1,11 @@ --- layout: default title: Migration phases -parent: Migration Assistant -nav_order: 30 +parent: Migration Assistant for OpenSearch +nav_order: 40 +nav_exclude: false has_children: true -has_toc: false +has_toc: true permalink: /migration-assistant/migration-phases/ --- @@ -12,8 +13,6 @@ permalink: /migration-assistant/migration-phases/ This page outlines the core phases of migrating with Migration Assistant. Each scenario below consists of a sequence of common steps. Expand a scenario to explore the detailed flow. ---- - -
        +
        Scenario 1 – Backfill Only
          @@ -40,42 +40,95 @@ details[open] {
        1. Create Snapshot
        2. Migrate Metadata
          3. -
        3. Migrate Data
          4. +
        4. Backfill
        5. Teardown
        -
        +
        Scenario 2 – Live Capture Only
        1. Assessment
          2. -
        2. Deployment
          3. +
        3. Deploy
        4. Verify Backfill Components
        5. Reroute Traffic from Source to Capture Proxy
          6. -
        6. Migrate Metadata
          7. +
        7. Migrate Metadata
        8. Verify Live Capture Components
          9. -
        9. Replay Captured Traffic
          10. -
        10. Reroute Traffic from Capture Proxy to Target
          11. +
        11. Replay Captured Traffic
          12. +
        12. Reroute Traffic from Capture Proxy to Target
        13. Teardown
        -
        +
        Scenario 3 – Live Capture with Backfill
        1. Assessment
          2. -
        2. Deployment
          3. +
        3. Deploy
        4. Reroute Traffic from Source to Capture Proxy
        5. Create Snapshot
          6. -
        6. Migrate Metadata
          7. -
        7. Migrate Data
          8. -
        8. Replay Traffic
          9. -
        9. Reroute Traffic from Capture Proxy to Target
          10. +
        10. Migrate Metadata
          11. +
        11. Backfill
          12. +
        12. Replay Captured Traffic
          13. +
        13. Reroute Traffic from Capture Proxy to Target
        14. Teardown
        -
        \ No newline at end of file +
        + + diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/live-traffic-migration/index.md b/_migrate-or-upgrade/migration-assistant/migration-phases/live-traffic-migration/index.md index 4ba2217146a..ea255a0f0e1 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/live-traffic-migration/index.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/live-traffic-migration/index.md @@ -10,10 +10,9 @@ has_children: true # Live traffic migration -Live traffic migration intercepts HTTP requests to a source cluster and stores them in a durable stream before forwarding them to the source cluster. The stored requests are then duplicated and replayed to the target cluster. This process synchronizes the source and target clusters while highlighting behavioral and performance differences between them. Kafka is used to manage the data flow and reconstruct HTTP requests. You can monitor the replication process through CloudWatch metrics and the [migration console]({{site.url}}{{site.baseurl}}/migration-assistantmigration-console/), which provides results in JSON format for analysis. +Live traffic migration intercepts HTTP requests to a source cluster and stores them in a durable stream before forwarding them to the source cluster. The stored requests are then duplicated and replayed to the target cluster. This process synchronizes the source and target clusters while highlighting behavioral and performance differences between them. Kafka is used to manage the data flow and reconstruct HTTP requests. You can monitor the replication process through CloudWatch metrics and the [migration console]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/accessing-the-migration-console/), which provides results in JSON format for analysis. To start with live traffic migration, use the following steps: -1. [Using Traffic Replayer]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/using-traffic-replayer/) -2. [Switching traffic from the source cluster]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/switching-traffic-from-the-source-cluster/) - +1. [Using Traffic Replayer]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/replay-captured-traffic/) +2. [Switching traffic from the source cluster]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/reroute-traffic-from-capture-proxy-to-target/) diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/migrate-metadata.md b/_migrate-or-upgrade/migration-assistant/migration-phases/migrate-metadata.md index cd2f6cd053f..7554cc4bce6 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/migrate-metadata.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/migrate-metadata.md @@ -3,7 +3,10 @@ layout: default title: Migrate Metadata nav_order: 5 parent: Migration phases +grand_parent: Migration Assistant for OpenSearch permalink: /migration-assistant/migration-phases/migrate-metadata/ +redirect_from: + - /migration-assistant/migration-phases/migrating-metadata/ --- # Migrate metadata @@ -14,44 +17,6 @@ This tool gathers information from a source cluster through a snapshot or throug After collecting information on the source cluster, comparisons are made against the target cluster. If running a migration, any metadata items that do not already exist will be created on the target cluster. -## Creating the snapshot - -Creating a snapshot of the source cluster captures all the metadata and documents to be migrated to a new target cluster. - -Create the initial snapshot of the source cluster using the following command: - -```shell -console snapshot create -``` -{% include copy.html %} - -To check the progress of the snapshot in real time, use the following command: - -```shell -console snapshot status --deep-check -``` -{% include copy.html %} - -You should receive the following response when the snapshot is created: - -```shell -SUCCESS -Snapshot is SUCCESS. -Percent completed: 100.00% -Data GiB done: 29.211/29.211 -Total shards: 40 -Successful shards: 40 -Failed shards: 0 -Start time: 2024-07-22 18:21:42 -Duration: 0h 13m 4s -Anticipated duration remaining: 0h 0m 0s -Throughput: 38.13 MiB/sec -``` - -### Managing slow snapshot speeds - -Depending on the size of the data in the source cluster and the bandwidth allocated for snapshots, the process can take some time. Adjust the maximum rate at which the source cluster's nodes create the snapshot using the `--max-snapshot-rate-mb-per-node` option. Increasing the snapshot rate will consume more node resources, which may affect the cluster's ability to handle normal traffic. - ## Command arguments For the following commands, to identify all valid arguments, please run with `--help`. diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/remove-migration-infrastructure.md b/_migrate-or-upgrade/migration-assistant/migration-phases/remove-migration-infrastructure.md index d3a4f9cbbb9..a88c003609d 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/remove-migration-infrastructure.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/remove-migration-infrastructure.md @@ -3,6 +3,7 @@ layout: default title: Removing migration infrastructure nav_order: 8 parent: Migration phases +grand_parent: Migration Assistant for OpenSearch permalink: /migration-assistant/migration-phases/remove-migration-infrastructure/ --- diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/using-traffic-replayer.md b/_migrate-or-upgrade/migration-assistant/migration-phases/replay-captured-traffic.md similarity index 97% rename from _migrate-or-upgrade/migration-assistant/migration-phases/using-traffic-replayer.md rename to _migrate-or-upgrade/migration-assistant/migration-phases/replay-captured-traffic.md index a1b892ec752..27df18c487e 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/using-traffic-replayer.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/replay-captured-traffic.md @@ -3,7 +3,9 @@ layout: default title: Using Traffic Replayer nav_order: 7 parent: Migration phases -redirect_from: +grand_parent: Migration Assistant for OpenSearch +permalink: /migration-assistant/migration-phases/replay-captured-traffic/ +redirect_from: - /migration-assistant/migration-phases/using-traffic-replayer/ --- @@ -19,7 +21,7 @@ For example, if a document was deleted after a snapshot was taken, starting Traf ## Configuration options -[Traffic Replayer settings]({{site.url}}{{site.baseurl}}/migration-assistant/deploying-migration-assistant/configuration-options/) are configured during the deployment of Migration Assistant. Make sure to set the authentication mode for Traffic Replayer so that it can properly communicate with the target cluster. +[Traffic Replayer settings]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/deploying-migration-assistant/configuration-options/) are configured during the deployment of Migration Assistant. Make sure to set the authentication mode for Traffic Replayer so that it can properly communicate with the target cluster. ## Using Traffic Replayer @@ -315,4 +317,4 @@ The following metrics are also reported: ## CloudWatch considerations -Metrics and dashboards pushed to CloudWatch may experience a visibility lag of around 5 minutes. CloudWatch also retains higher-resolution data for a shorter period than lower-resolution data. For more information, see [Amazon CloudWatch concepts](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html). \ No newline at end of file +Metrics and dashboards pushed to CloudWatch may experience a visibility lag of around 5 minutes. CloudWatch also retains higher-resolution data for a shorter period than lower-resolution data. For more information, see [Amazon CloudWatch concepts](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html). diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/reroute-source-to-proxy.md b/_migrate-or-upgrade/migration-assistant/migration-phases/reroute-source-to-proxy.md index 79f362e489d..d8c01337a78 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/reroute-source-to-proxy.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/reroute-source-to-proxy.md @@ -3,6 +3,7 @@ layout: default title: Reroute Client Traffic nav_order: 3 parent: Migration phases +grand_parent: Migration Assistant for OpenSearch permalink: /migration-assistant/migration-phases/reroute-source-to-proxy/ --- @@ -57,24 +58,6 @@ Note the records in the logging topic. After a short period, execute the same command again and compare the increased number of records against the expected HTTP requests. -## Creating a snapshot - -Create a snapshot for your backfill using the following command: - -```bash -console snapshot create -``` -{% include copy.html %} - -To check the progress of your snapshot, use the following command: - -```bash -console snapshot status --deep-check -``` -{% include copy.html %} - -Depending on the size of the data in the source cluster and the bandwidth allocated for snapshots, the process can take some time. Adjust the maximum rate at which the source cluster's nodes create the snapshot using the `--max-snapshot-rate-mb-per-node` option. Increasing the snapshot rate will consume more node resources, which may affect the cluster's ability to handle normal traffic. - ## Backfilling documents to the source cluster From the snapshot you created of your source cluster, you can begin backfilling documents into the target cluster. Once you have started this process, a fleet of workers will spin up to read the snapshot and reindex documents into the target cluster. This fleet of workers can be scaled to increased the speed at which documents are reindexed into the target cluster. diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/reroute-traffic-from-capture-proxy-to-target.md b/_migrate-or-upgrade/migration-assistant/migration-phases/reroute-traffic-from-capture-proxy-to-target.md index e577ed989d6..af889aab278 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/reroute-traffic-from-capture-proxy-to-target.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/reroute-traffic-from-capture-proxy-to-target.md @@ -1,9 +1,10 @@ --- layout: default -title: Cutover +title: Reroute traffic to target nav_order: 7 parent: Migration phases -permalink: /migration-assistant/migration-phases/switching-traffic-from-the-source-cluster/ +grand_parent: Migration Assistant for OpenSearch +permalink: /migration-assistant/migration-phases/reroute-traffic-from-capture-proxy-to-target/ --- # Switching traffic from the source cluster @@ -50,4 +51,4 @@ Use the following steps to switch traffic from the source cluster to the target ## Fallback -If you need to fall back to the source cluster at any point during the switchover, revert the **Default rule** so that the Application Load Balancer routes to the **SourceProxy Target Group**. \ No newline at end of file +If you need to fall back to the source cluster at any point during the switchover, revert the **Default rule** so that the Application Load Balancer routes to the **SourceProxy Target Group**. diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/verifying-migration-tools/verifying-backfill-components.md b/_migrate-or-upgrade/migration-assistant/migration-phases/verifying-migration-tools/verifying-backfill-components.md index c8ae8472d0d..c74b2b97482 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/verifying-migration-tools/verifying-backfill-components.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/verifying-migration-tools/verifying-backfill-components.md @@ -1,11 +1,11 @@ --- layout: default title: Verifying backfill components -grand_parent: Migrations phases +grand_parent: Migration phases nav_order: 3 parent: Verifying migration tools -permalink: /migration-assistant/migration-phases/verifying-backfill-components/ -retirct-from: +permalink: /migration-assistant/migration-phases/verifying-migration-tools/verifying-backfill-components/ +redirect_from: - /migration-assistant/planning-your-migration/migration-phases/verifying-backfill-components/ --- diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/verifying-migration-tools/verifying-live-capture-components.md b/_migrate-or-upgrade/migration-assistant/migration-phases/verifying-migration-tools/verifying-live-capture-components.md index 404b2c9efda..270934af763 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/verifying-migration-tools/verifying-live-capture-components.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/verifying-migration-tools/verifying-live-capture-components.md @@ -3,7 +3,7 @@ layout: default title: Verifying live capture components nav_order: 9 parent: Verifying migration tools -permalink: /migrations-assisstant/migration-phases/verifying-livee-capture-compontents/ +permalink: /migration-assistant/migration-phases/verifying-migration-tools/verifying-live-capture-components/ --- # Verifying live capture components diff --git a/_migrate-or-upgrade/migration-assistant/planning-your-migration/handling-type-mapping-deprecation.md b/_migrate-or-upgrade/migration-assistant/planning-your-migration/handling-type-mapping-deprecation.md index 060942591e5..a0053ed523f 100644 --- a/_migrate-or-upgrade/migration-assistant/planning-your-migration/handling-type-mapping-deprecation.md +++ b/_migrate-or-upgrade/migration-assistant/planning-your-migration/handling-type-mapping-deprecation.md @@ -2,7 +2,9 @@ layout: default title: Managing type mapping deprecation nav_order: 60 +grand_parent: Migration Assistant for OpenSearch parent: Planning your migration +permalink: /migration-assistant/planning-your-migration/handling-type-mapping-deprecation/ --- # Managing type mapping deprecation diff --git a/_migrate-or-upgrade/rolling-upgrade/appendix/index.md b/_migrate-or-upgrade/rolling-upgrade/appendix/index.md index 893a76f92af..a5858292e62 100644 --- a/_migrate-or-upgrade/rolling-upgrade/appendix/index.md +++ b/_migrate-or-upgrade/rolling-upgrade/appendix/index.md @@ -1,9 +1,11 @@ --- layout: default title: Upgrades appendix -parent: Upgrading OpenSearch +grand_parent: Migrate or upgrade +parent: Rolling upgrade has_children: true nav_order: 30 +permalink: /migrate-or-upgrade/rolling-upgrade/appendix/ redirect_from: - /upgrade-opensearch/appendix/ --- diff --git a/_migrate-or-upgrade/rolling-upgrade/appendix/rolling-upgrade-lab.md b/_migrate-or-upgrade/rolling-upgrade/appendix/rolling-upgrade-lab.md index f7baeccd213..6b8afc7d79e 100644 --- a/_migrate-or-upgrade/rolling-upgrade/appendix/rolling-upgrade-lab.md +++ b/_migrate-or-upgrade/rolling-upgrade/appendix/rolling-upgrade-lab.md @@ -36,7 +36,7 @@ To use, invoke class="codeblock-label" # Rolling upgrade lab -You can follow these steps on your own compatible host to recreate the same cluster state the OpenSearch Project used for testing [rolling upgrades]({{site.url}}{{site.baseurl}}/install-and-configure/migrate-or-upgrade/rolling-upgrade/). This exercise is useful if you want to test the upgrade process in a development environment. +You can follow these steps on your own compatible host to recreate the same cluster state the OpenSearch Project used for testing [rolling upgrades]({{site.url}}{{site.baseurl}}/igrate-or-upgrade/rolling-upgrade/). This exercise is useful if you want to test the upgrade process in a development environment. The steps used in this lab were validated on an arbitrarily chosen [Amazon Elastic Compute Cloud (Amazon EC2)](https://aws.amazon.com/ec2/) `t2.large` instance using [Amazon Linux 2](https://aws.amazon.com/amazon-linux-2/) kernel version `Linux 5.10.162-141.675.amzn2.x86_64` and [Docker](https://www.docker.com/) version `20.10.17, build 100c701`. The instance was provisioned with an attached 20 GiB gp2 [Amazon EBS](https://aws.amazon.com/ebs/) root volume. These specifications are included for informational purposes and do not represent hardware requirements for OpenSearch or OpenSearch Dashboards. diff --git a/_migrate-or-upgrade/rolling-upgrade/index.md b/_migrate-or-upgrade/rolling-upgrade/index.md index 9a73d2aaa34..0a012b6da13 100644 --- a/_migrate-or-upgrade/rolling-upgrade/index.md +++ b/_migrate-or-upgrade/rolling-upgrade/index.md @@ -1,10 +1,9 @@ --- layout: default title: Rolling upgrade -parent: Migrate or upgrade nav_order: 10 -nav_exclude: false has_toc: true +permalink: /migrate-or-upgrade/rolling-upgrade/ redirect_from: - /upgrade-opensearch/appendix - /rolling-upgrade/index @@ -18,7 +17,7 @@ This document serves as a high-level, platform-agnostic overview of the rolling ## Preparing to upgrade -Review [Upgrading OpenSearch]({{site.url}}{{site.baseurl}}/migration-or-upgrade/rolling-upgrade/index/) for recommendations about backing up your configuration files and creating a snapshot of the cluster state and indexes before you make any changes to your OpenSearch cluster. +Review [Upgrading OpenSearch]({{site.url}}{{site.baseurl}}/migrate-or-upgrade/rolling-upgrade/) for recommendations about backing up your configuration files and creating a snapshot of the cluster state and indexes before you make any changes to your OpenSearch cluster. **Important:** OpenSearch nodes cannot be downgraded. If you need to revert the upgrade, then you will need to perform a fresh installation of OpenSearch and restore the cluster from a snapshot. Take a snapshot and store it in a remote repository before beginning the upgrade procedure. {: .important} diff --git a/_migrate-or-upgrade/snapshot-restore/index.md b/_migrate-or-upgrade/snapshot-restore/index.md new file mode 100644 index 00000000000..b8d21634476 --- /dev/null +++ b/_migrate-or-upgrade/snapshot-restore/index.md @@ -0,0 +1,12 @@ +--- +layout: default +title: Snapshot restore +nav_order: 20 +has_toc: false +permalink: /migrate-or-upgrade/snapshot-restore/ +redirect_from: +--- + +# Snapshot and Restore + +To be completed diff --git a/_migration-assistant b/_migration-assistant new file mode 120000 index 00000000000..7ec506f4b6b --- /dev/null +++ b/_migration-assistant @@ -0,0 +1 @@ +_migrate-or-upgrade/migration-assistant \ No newline at end of file diff --git a/_tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore.md b/_tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore.md index b45afadc862..e3ad90fe689 100644 --- a/_tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore.md +++ b/_tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore.md @@ -3,6 +3,7 @@ layout: default title: Take and restore snapshots parent: Snapshots nav_order: 10 +has_toc: false has_children: false grand_parent: Availability and recovery redirect_from: From 1b87b08acaee6433c77a04652f73715f5bf3bfd6 Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Tue, 22 Jul 2025 21:37:40 -0500 Subject: [PATCH 20/62] Working structural changes with no broken links Signed-off-by: Brian Presley --- _api-reference/snapshots/index.md | 3 +- .../migration-assistant/valid_migrations.yml | 1 + _migrate-or-upgrade/index.md | 40 +++--- .../accessing-the-migration-console.md | 43 ------- .../migration-assistant/index.md | 3 +- .../accessing-the-migration-console.md | 5 +- .../migration-console/index.md | 3 + .../migration-console-commands-references.md | 3 +- .../index.md} | 11 +- .../migration-phases/backfill.md | 2 +- .../migration-phases/create-snapshot.md | 2 - .../configuration-options.md | 8 +- ...d-security-groups-for-existing-clusters.md | 9 +- .../{deploy.md => deploy/index.md} | 7 +- .../verifying-backfill-components.md | 8 +- .../verifying-live-capture-components.md | 7 +- .../migration-phases/index.md | 52 ++++++-- .../live-traffic-migration/index.md | 2 +- .../handling-field-type-breaking-changes.md | 8 +- .../handling-type-mapping-deprecation.md | 10 +- .../index.md} | 2 + .../remove-migration-infrastructure.md | 2 +- .../replay-captured-traffic.md | 5 +- .../reroute-source-to-proxy.md | 5 + ...te-traffic-from-capture-proxy-to-target.md | 5 +- .../verifying-migration-tools.md | 20 --- .../appendix/rolling-upgrade-lab.md | 3 +- _migrate-or-upgrade/snapshot-restore/index.md | 118 +++++++++++++++++- assets/js/nav-scroll.js | 24 +++- 29 files changed, 271 insertions(+), 140 deletions(-) delete mode 100644 _migrate-or-upgrade/migration-assistant/accessing-the-migration-console.md rename _migrate-or-upgrade/migration-assistant/migration-phases/{assessing-your-cluster-for-migration.md => assessment/index.md} (77%) rename _migrate-or-upgrade/migration-assistant/migration-phases/{deploying-migration-assistant => deploy}/configuration-options.md (98%) rename _migrate-or-upgrade/migration-assistant/migration-phases/{deploying-migration-assistant => deploy}/iam-and-security-groups-for-existing-clusters.md (94%) rename _migrate-or-upgrade/migration-assistant/migration-phases/{deploy.md => deploy/index.md} (97%) rename _migrate-or-upgrade/migration-assistant/migration-phases/{verifying-migration-tools => deploy}/verifying-backfill-components.md (97%) rename _migrate-or-upgrade/migration-assistant/migration-phases/{verifying-migration-tools => deploy}/verifying-live-capture-components.md (97%) rename _migrate-or-upgrade/migration-assistant/{planning-your-migration => migration-phases/migrate-metadata}/handling-field-type-breaking-changes.md (94%) rename _migrate-or-upgrade/migration-assistant/{planning-your-migration => migration-phases/migrate-metadata}/handling-type-mapping-deprecation.md (97%) rename _migrate-or-upgrade/migration-assistant/migration-phases/{migrate-metadata.md => migrate-metadata/index.md} (99%) delete mode 100644 _migrate-or-upgrade/migration-assistant/migration-phases/verifying-migration-tools/verifying-migration-tools.md diff --git a/_api-reference/snapshots/index.md b/_api-reference/snapshots/index.md index e341905ad57..ef730f9b04e 100644 --- a/_api-reference/snapshots/index.md +++ b/_api-reference/snapshots/index.md @@ -1,6 +1,7 @@ --- layout: default title: Snapshot APIs +parent: API reference has_children: true nav_order: 80 redirect_from: @@ -12,4 +13,4 @@ redirect_from: **Introduced 1.0** {: .label .label-purple } -The snapshot APIs allow you to manage snapshots and snapshot repositories. \ No newline at end of file +The snapshot APIs allow you to manage snapshots and snapshot repositories. diff --git a/_data/migration-assistant/valid_migrations.yml b/_data/migration-assistant/valid_migrations.yml index 4e0f036248f..a2223b1dd2d 100644 --- a/_data/migration-assistant/valid_migrations.yml +++ b/_data/migration-assistant/valid_migrations.yml @@ -26,3 +26,4 @@ migration_paths: - source: "OpenSearch 2.x" targets: - "OpenSearch 2.x" + - "OpenSearch 3.x" diff --git a/_migrate-or-upgrade/index.md b/_migrate-or-upgrade/index.md index 9e498f66b68..6269595c6eb 100644 --- a/_migrate-or-upgrade/index.md +++ b/_migrate-or-upgrade/index.md @@ -1,7 +1,6 @@ --- layout: default title: Migrate or upgrade -nav_order: 20 has_children: true permalink: /upgrade-or-migrate/ redirect_from: @@ -12,7 +11,7 @@ redirect_from: - /upgrade-to/ - /upgrade-to/upgrade-to/ has_toc: true -nav_exclude: false +nav_exclude: true --- # Migrate or Upgrade OpenSearch @@ -47,22 +46,6 @@ Stop nonessential indexing before upgrading. ## Migration and Upgrade Methods -### Migration Assistant - -The Migration Assistant provides the most automated and resilient upgrade path. - -**Pros:** -- Supports multi-version hops -- Enables near-zero downtime via live data capture -- Includes rollback support -- Designed to integrate with OpenSearch workflows - -**Cons:** -- Requires setup and infrastructure - -[Get started with Migration Assistant]({{site.url}}{{site.baseurl}}/migration-assistant/) - ---- ### Rolling Upgrade @@ -76,6 +59,7 @@ Upgrade one node at a time while keeping the cluster operational. - Supports only adjacent major versions - Multiple upgrade cycles for larger version gaps - Reindexing may be required +- Manual reindex may be required for full feature compatibility [Perform a rolling upgrade]({{site.url}}{{site.baseurl}}/migrate-or-upgrade/rolling-upgrade/) @@ -86,17 +70,35 @@ Upgrade one node at a time while keeping the cluster operational. Take a snapshot of your current cluster and restore to a new OpenSearch version. **Pros:** -- Leaves source cluster unchanged - Supports large datasets and cold storage +- Reversion possible since original cluster is untouched. However, there may be a loss of data if no change data capture (CDC) solution is in place. **Cons:** - Requires downtime or a change data capture (CDC) solution - Requires provisioning a new cluster +- Manually reindex may be required for full feature compatibility. [Get started with Snapshot and Restore](https://docs.aws.amazon.com/solutions/latest/tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore/) --- +### Migration Assistant + +The Migration Assistant provides the most automated and resilient upgrade path. + +**Pros:** +- Handles multi-version hops, allowing for seamless upgrades across multiple versions +- Live data capture allows for little to zero downtime, ensuring data consistency during migrations +- Ability to revert changes if issues arise during or after the upgrade + +**Cons:** +- Requires additional setup +- Requires additional infrastructure + +[Get started with Migration Assistant]({{site.url}}{{site.baseurl}}/migration-assistant/) + +--- + ### Remote Reindexing Reindex data from the old cluster into the new OpenSearch cluster. diff --git a/_migrate-or-upgrade/migration-assistant/accessing-the-migration-console.md b/_migrate-or-upgrade/migration-assistant/accessing-the-migration-console.md deleted file mode 100644 index f01ae49febe..00000000000 --- a/_migrate-or-upgrade/migration-assistant/accessing-the-migration-console.md +++ /dev/null @@ -1,43 +0,0 @@ ---- -layout: default -title: Accessing the Migration Console -parent: Migration phases -nav_order: 10 -permlink: /migration-assistant/accessing-the-migration-console ---- - -# Accessing the Migration Console - -The Migrations Assistant deployment includes an ECS task that hosts tools to run different phases of the migration and check the progress or results of the migration. - -## SSH into the Migration Console - -Following the AWS Solutions deployment, the bootstrap box contains a script that simplifies access to the migration console through that instance. - -To access the Migration Console, use the following commands: - -```sh -export STAGE=dev -export AWS_REGION=us-west-2 -/opensearch-migrations/deployment/cdk/opensearch-service-migration/accessContainer.sh migration-console ${STAGE} ${AWS_REGION} -``` -When opening the console a message will appear above the command prompt, Welcome to the Migration Assistant Console. -
        - - -SSH from any machine into Migration Console - - -On a machine with the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) ↗ and the [AWS Session Manager Plugin](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html) ↗, you can directly connect to the migration console. Ensure you've run `aws configure` with credentials that have access to the environment. - -Use the following commands: - -```shell -export STAGE=dev -export SERVICE_NAME=migration-console -export TASK_ARN=$(aws ecs list-tasks --cluster migration-${STAGE}-ecs-cluster --family "migration-${STAGE}-${SERVICE_NAME}" | jq --raw-output '.taskArns[0]') -aws ecs execute-command --cluster "migration-${STAGE}-ecs-cluster" --task "${TASK_ARN}" --container "${SERVICE_NAME}" --interactive --command "/bin/bash" -``` -
        - -> **Note:** Typically, `STAGE` is `dev`, but this may vary based on what the user specified during deployment. \ No newline at end of file diff --git a/_migrate-or-upgrade/migration-assistant/index.md b/_migrate-or-upgrade/migration-assistant/index.md index 8ad0c1f6d5e..e59f277ac20 100644 --- a/_migrate-or-upgrade/migration-assistant/index.md +++ b/_migrate-or-upgrade/migration-assistant/index.md @@ -1,8 +1,7 @@ --- layout: default title: Migration Assistant for OpenSearch -nav_order: 2 -has_toc: false +nav_order: 21 has_children: true permalink: /migration-assistant/ redirect_from: diff --git a/_migrate-or-upgrade/migration-assistant/migration-console/accessing-the-migration-console.md b/_migrate-or-upgrade/migration-assistant/migration-console/accessing-the-migration-console.md index cf530e3ccd4..1309e221def 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-console/accessing-the-migration-console.md +++ b/_migrate-or-upgrade/migration-assistant/migration-console/accessing-the-migration-console.md @@ -1,8 +1,9 @@ --- layout: default title: Accessing the migration console -nav_order: 35 +nav_order: 1 parent: Migration console +grand_parent: Migration Assistant for OpenSearch permalink: /migration-assistant/migration-console/accessing-the-migration-console/ --- @@ -33,4 +34,4 @@ aws ecs execute-command --cluster "migration-${STAGE}-ecs-cluster" --task "${TAS ``` {% include copy.html %} -Typically, `STAGE` is equivalent to a standard `dev` environment, but this may vary based on what the user specified during deployment. \ No newline at end of file +Typically, `STAGE` is equivalent to a standard `dev` environment, but this may vary based on what the user specified during deployment. diff --git a/_migrate-or-upgrade/migration-assistant/migration-console/index.md b/_migrate-or-upgrade/migration-assistant/migration-console/index.md index 2d200c56fa2..535c5734eb8 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-console/index.md +++ b/_migrate-or-upgrade/migration-assistant/migration-console/index.md @@ -1,9 +1,12 @@ --- layout: default parent: Migration Assistant for OpenSearch +grand_parent: Migrate or upgrade title: Migration console nav_order: 50 +nav_exclude: false has_children: true +has_toc: false permalink: /migration-assistant/migration-console/ redirect_from: - /migration-assistant/migration-console/index/ diff --git a/_migrate-or-upgrade/migration-assistant/migration-console/migration-console-commands-references.md b/_migrate-or-upgrade/migration-assistant/migration-console/migration-console-commands-references.md index c1aa7c37ad1..7d3396e63bd 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-console/migration-console-commands-references.md +++ b/_migrate-or-upgrade/migration-assistant/migration-console/migration-console-commands-references.md @@ -1,8 +1,9 @@ --- layout: default title: Command reference -nav_order: 40 +nav_order: 2 parent: Migration console +grand_parent: Migration Assistant for OpenSearch permalink: /migration-assistant/migration-console/migration-console-command-reference/ --- diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/assessing-your-cluster-for-migration.md b/_migrate-or-upgrade/migration-assistant/migration-phases/assessment/index.md similarity index 77% rename from _migrate-or-upgrade/migration-assistant/migration-phases/assessing-your-cluster-for-migration.md rename to _migrate-or-upgrade/migration-assistant/migration-phases/assessment/index.md index 445f3f50cd8..8eaba3d440e 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/assessing-your-cluster-for-migration.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/assessment/index.md @@ -1,13 +1,15 @@ --- layout: default -title: Assessing your cluster for migration +title: Assessment nav_order: 1 parent: Migration phases grand_parent: Migration Assistant for OpenSearch +has_children: true +has_toc: false permalink: /migration-assistant/migration-phases/assessment/ --- -# Assessing your cluster for migration +# Assessment The goal of the Migration Assistant is to streamline the process of migrating from one location or version of Elasticsearch/OpenSearch to another. However, completing a migration sometimes requires resolving client compatibility issues before they can communicate directly with the target cluster. @@ -71,6 +73,7 @@ For complex migrations involving multiple transformations or breaking changes, w ## Supported transformations -The following [transformations]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/replay-captured-traffic/#transformations) are included in Migration Assistant. They can be enabled, combined, and configured to customize a migration for your use case. To request additional Migration Assistant transformations , create a GitHub issue [in the OpenSearch migrations repository](https://github.com/opensearch-project/opensearch-migrations/issues). +Below is a list of transformations that are included in Migration Assistant. They can be enabled, combined, and configured to customize a migration for your use case. Depending on your use case and the type of transformation, a transformation may have to be added to Capture-and-Replay, Metadata Migration Tool, or Reindex-from-Snapshot. To request additional Migration Assistant transformations, create a GitHub issue [in the OpenSearch migrations repository](https://github.com/opensearch-project/opensearch-migrations/issues). -- [Type mapping deprecation]({{site.url}}{{site.baseurl}}/migration-assistant/planning-your-migration/handling-type-mapping-deprecation/) +- [Managing type mapping deprecation]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrate-metadata/handling-type-mapping-deprecation/) +- [Handling breaking changes in field types]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrate-metadata/handling-field-type-breaking-changes/) diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/backfill.md b/_migrate-or-upgrade/migration-assistant/migration-phases/backfill.md index ef064f8fc0c..bf66bf09116 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/backfill.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/backfill.md @@ -1,6 +1,6 @@ --- layout: default -title: backfill +title: Backfill nav_order: 6 parent: Migration phases grand_parent: Migration Assistant for OpenSearch diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/create-snapshot.md b/_migrate-or-upgrade/migration-assistant/migration-phases/create-snapshot.md index fe2c36395de..3eb7ce66529 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/create-snapshot.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/create-snapshot.md @@ -9,8 +9,6 @@ permalink: /migration-assistant/migration-phases/create-snapshot/ # Creating a snapshot -<----!TODO: Add Bring your own snapshot-----!> - Creating a snapshot of the source cluster captures all the metadata and documents to be migrated to a new target cluster. Create the initial snapshot of the source cluster using the following command: diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/deploying-migration-assistant/configuration-options.md b/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/configuration-options.md similarity index 98% rename from _migrate-or-upgrade/migration-assistant/migration-phases/deploying-migration-assistant/configuration-options.md rename to _migrate-or-upgrade/migration-assistant/migration-phases/deploy/configuration-options.md index 580c4f3566b..e3f2480d585 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/deploying-migration-assistant/configuration-options.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/configuration-options.md @@ -1,10 +1,12 @@ --- layout: default title: Configuration options -nav_order: 15 +nav_order: 3 grand_parent: Migration phases -parent: Deploying Migration Assistant -permalink: /migration-assistant/migration-phases/deploying-migration-assistant/configuration-options/ +parent: Deploy +permalink: /migration-assistant/migration-phases/deploy/configuration-options/ +redirect_from: + - /migration-assistant/migration-phases/deploying-migration-assistant/configuration-options/ --- # Configuration options diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md b/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/iam-and-security-groups-for-existing-clusters.md similarity index 94% rename from _migrate-or-upgrade/migration-assistant/migration-phases/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md rename to _migrate-or-upgrade/migration-assistant/migration-phases/deploy/iam-and-security-groups-for-existing-clusters.md index ee1442fd74c..f60b6de5af0 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/iam-and-security-groups-for-existing-clusters.md @@ -1,9 +1,12 @@ --- layout: default title: IAM and security groups for existing clusters -nav_order: 20 -nav_exclude: true -permalink: /migration-assistant/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters/ +nav_order: 4 +grand_parent: Migration phases +parent: Deploy +permalink: /migration-assistant/migration-phases/deploy/iam-and-security-groups-for-existing-clusters/ +redirect_from: + - /migration-assistant/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters/ --- # IAM and security groups for existing clusters diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/deploy.md b/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/index.md similarity index 97% rename from _migrate-or-upgrade/migration-assistant/migration-phases/deploy.md rename to _migrate-or-upgrade/migration-assistant/migration-phases/deploy/index.md index 977cbbc8cf4..7767e5d7483 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/deploy.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/index.md @@ -1,17 +1,18 @@ --- layout: default -title: Deploying Migration Assistant +title: Deploy parent: Migration phases grand_parent: Migration Assistant for OpenSearch nav_order: 2 +has_children: true permalink: /migration-assistant/migration-phases/deploy/ redirect_from: - /migration-assistant/getting-started-with-data-migration/ --- -# Deploying Migration Assistant +# Deploy -This document assumes you have performed assessment. +This document assumes you have performed [assessment]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/assessment/) to understand any upgrade breaking changes and limitations before beginning. --- diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/verifying-migration-tools/verifying-backfill-components.md b/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/verifying-backfill-components.md similarity index 97% rename from _migrate-or-upgrade/migration-assistant/migration-phases/verifying-migration-tools/verifying-backfill-components.md rename to _migrate-or-upgrade/migration-assistant/migration-phases/deploy/verifying-backfill-components.md index c74b2b97482..44866ba0c96 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/verifying-migration-tools/verifying-backfill-components.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/verifying-backfill-components.md @@ -2,11 +2,11 @@ layout: default title: Verifying backfill components grand_parent: Migration phases -nav_order: 3 -parent: Verifying migration tools -permalink: /migration-assistant/migration-phases/verifying-migration-tools/verifying-backfill-components/ +nav_order: 1 +parent: Deploy +permalink: /migration-assistant/migration-phases/deploy/verifying-backfill-components/ redirect_from: - - /migration-assistant/planning-your-migration/migration-phases/verifying-backfill-components/ + - /migration-assistant/migration-phases/verifying-migration-tools/verifying-backfill-components/ --- # Verifying backfill components diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/verifying-migration-tools/verifying-live-capture-components.md b/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/verifying-live-capture-components.md similarity index 97% rename from _migrate-or-upgrade/migration-assistant/migration-phases/verifying-migration-tools/verifying-live-capture-components.md rename to _migrate-or-upgrade/migration-assistant/migration-phases/deploy/verifying-live-capture-components.md index 270934af763..49e8198b4ab 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/verifying-migration-tools/verifying-live-capture-components.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/verifying-live-capture-components.md @@ -1,9 +1,10 @@ --- layout: default title: Verifying live capture components -nav_order: 9 -parent: Verifying migration tools -permalink: /migration-assistant/migration-phases/verifying-migration-tools/verifying-live-capture-components/ +grand_parent: Migration phases +nav_order: 2 +parent: Deploy +permalink: /migration-assistant/migration-phases/deploy/verifying-live-capture-components/ --- # Verifying live capture components diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/index.md b/_migrate-or-upgrade/migration-assistant/migration-phases/index.md index 94b5ebf0c3b..29d7d1068b5 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/index.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/index.md @@ -2,6 +2,7 @@ layout: default title: Migration phases parent: Migration Assistant for OpenSearch +grand_parent: Migrate or upgrade nav_order: 40 nav_exclude: false has_children: true @@ -36,10 +37,20 @@ details[open] {
        1. Assessment
          2. -
        2. Deploy
          3. - +
        3. Deploy + +
        4. Create Snapshot
          5. -
        5. Migrate Metadata
          6. +
        6. Migrate Metadata + +
        7. Backfill
        8. Teardown
        @@ -51,11 +62,20 @@ details[open] {
        1. Assessment
          2. -
        2. Deploy
          3. -
        3. Verify Backfill Components
          4. +
        4. Deploy + +
        5. Reroute Traffic from Source to Capture Proxy
          6. -
        6. Migrate Metadata
          7. -
        7. Verify Live Capture Components
          8. +
        8. Migrate Metadata + +
        9. Replay Captured Traffic
        10. Reroute Traffic from Capture Proxy to Target
        11. Teardown
          12. @@ -67,11 +87,23 @@ details[open] { Scenario 3 – Live Capture with Backfill
            -
          1. Assessment
            2. -
          2. Deploy
            3. +
          3. Assessment
            4. +
          4. Deploy + +
          5. Reroute Traffic from Source to Capture Proxy
          6. Create Snapshot
            7. -
          7. Migrate Metadata
            8. +
          8. Migrate Metadata + +
          9. Backfill
          10. Replay Captured Traffic
          11. Reroute Traffic from Capture Proxy to Target
            12. diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/live-traffic-migration/index.md b/_migrate-or-upgrade/migration-assistant/migration-phases/live-traffic-migration/index.md index ea255a0f0e1..efa30821546 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/live-traffic-migration/index.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/live-traffic-migration/index.md @@ -10,7 +10,7 @@ has_children: true # Live traffic migration -Live traffic migration intercepts HTTP requests to a source cluster and stores them in a durable stream before forwarding them to the source cluster. The stored requests are then duplicated and replayed to the target cluster. This process synchronizes the source and target clusters while highlighting behavioral and performance differences between them. Kafka is used to manage the data flow and reconstruct HTTP requests. You can monitor the replication process through CloudWatch metrics and the [migration console]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/accessing-the-migration-console/), which provides results in JSON format for analysis. +Live traffic migration intercepts HTTP requests to a source cluster and stores them in a durable stream before forwarding them to the source cluster. The stored requests are then duplicated and replayed to the target cluster. This process synchronizes the source and target clusters while highlighting behavioral and performance differences between them. Kafka is used to manage the data flow and reconstruct HTTP requests. You can monitor the replication process through CloudWatch metrics and the [migration console]({{site.url}}{{site.baseurl}}/migration-assistant/migration-console/accessing-the-migration-console/), which provides results in JSON format for analysis. To start with live traffic migration, use the following steps: diff --git a/_migrate-or-upgrade/migration-assistant/planning-your-migration/handling-field-type-breaking-changes.md b/_migrate-or-upgrade/migration-assistant/migration-phases/migrate-metadata/handling-field-type-breaking-changes.md similarity index 94% rename from _migrate-or-upgrade/migration-assistant/planning-your-migration/handling-field-type-breaking-changes.md rename to _migrate-or-upgrade/migration-assistant/migration-phases/migrate-metadata/handling-field-type-breaking-changes.md index 4fff3cda1b3..854f82647bf 100644 --- a/_migrate-or-upgrade/migration-assistant/planning-your-migration/handling-field-type-breaking-changes.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/migrate-metadata/handling-field-type-breaking-changes.md @@ -1,8 +1,12 @@ --- layout: default title: Handling breaking changes in field types -nav_order: 60 -parent: Planning your migration +nav_order: 2 +parent: Migrate Metadata +grand_parent: Migration phases +permalink: /migration-assistant/migration-phases/migrate-metadata/handling-field-type-breaking-changes/ +redirect_from: + - /migration-assistant/migration-phases/assessment/handling-field-type-breaking-changes/ --- # Handling breaking changes in field types diff --git a/_migrate-or-upgrade/migration-assistant/planning-your-migration/handling-type-mapping-deprecation.md b/_migrate-or-upgrade/migration-assistant/migration-phases/migrate-metadata/handling-type-mapping-deprecation.md similarity index 97% rename from _migrate-or-upgrade/migration-assistant/planning-your-migration/handling-type-mapping-deprecation.md rename to _migrate-or-upgrade/migration-assistant/migration-phases/migrate-metadata/handling-type-mapping-deprecation.md index a0053ed523f..724a3413235 100644 --- a/_migrate-or-upgrade/migration-assistant/planning-your-migration/handling-type-mapping-deprecation.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/migrate-metadata/handling-type-mapping-deprecation.md @@ -1,10 +1,12 @@ --- layout: default title: Managing type mapping deprecation -nav_order: 60 -grand_parent: Migration Assistant for OpenSearch -parent: Planning your migration -permalink: /migration-assistant/planning-your-migration/handling-type-mapping-deprecation/ +nav_order: 1 +parent: Migrate Metadata +grand_parent: Migration phases +permalink: /migration-assistant/migration-phases/migrate-metadata/handling-type-mapping-deprecation/ +redirect_from: + - /migration-assistant/migration-phases/assessment/handling-type-mapping-deprecation/ --- # Managing type mapping deprecation diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/migrate-metadata.md b/_migrate-or-upgrade/migration-assistant/migration-phases/migrate-metadata/index.md similarity index 99% rename from _migrate-or-upgrade/migration-assistant/migration-phases/migrate-metadata.md rename to _migrate-or-upgrade/migration-assistant/migration-phases/migrate-metadata/index.md index 7554cc4bce6..8aae1ae655e 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/migrate-metadata.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/migrate-metadata/index.md @@ -4,6 +4,8 @@ title: Migrate Metadata nav_order: 5 parent: Migration phases grand_parent: Migration Assistant for OpenSearch +has_children: true +has_toc: false permalink: /migration-assistant/migration-phases/migrate-metadata/ redirect_from: - /migration-assistant/migration-phases/migrating-metadata/ diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/remove-migration-infrastructure.md b/_migrate-or-upgrade/migration-assistant/migration-phases/remove-migration-infrastructure.md index a88c003609d..ae5ff3560f8 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/remove-migration-infrastructure.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/remove-migration-infrastructure.md @@ -1,6 +1,6 @@ --- layout: default -title: Removing migration infrastructure +title: Removing Migration Assistant nav_order: 8 parent: Migration phases grand_parent: Migration Assistant for OpenSearch diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/replay-captured-traffic.md b/_migrate-or-upgrade/migration-assistant/migration-phases/replay-captured-traffic.md index 27df18c487e..b59f3f1278d 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/replay-captured-traffic.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/replay-captured-traffic.md @@ -11,6 +11,9 @@ redirect_from: # Using Traffic Replayer +**Note:** This page is only relevant if you are using Capture-and-Replay to avoid downtime during a migration. If you are only performing backfill migration or can tolerate downtime, you can skip this step. +{: .note} + This guide covers how to use Traffic Replayer to replay captured traffic from a source cluster to a target cluster during the migration process. Traffic Replayer allows you to verify that the target cluster can handle requests in the same way as the source cluster and catch up to real-time traffic for a smooth migration. ## When to run Traffic Replayer @@ -21,7 +24,7 @@ For example, if a document was deleted after a snapshot was taken, starting Traf ## Configuration options -[Traffic Replayer settings]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/deploying-migration-assistant/configuration-options/) are configured during the deployment of Migration Assistant. Make sure to set the authentication mode for Traffic Replayer so that it can properly communicate with the target cluster. +[Traffic Replayer settings]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/deploy/configuration-options/) are configured during the deployment of Migration Assistant. Make sure to set the authentication mode for Traffic Replayer so that it can properly communicate with the target cluster. ## Using Traffic Replayer diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/reroute-source-to-proxy.md b/_migrate-or-upgrade/migration-assistant/migration-phases/reroute-source-to-proxy.md index d8c01337a78..4ef952db9cd 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/reroute-source-to-proxy.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/reroute-source-to-proxy.md @@ -7,6 +7,11 @@ grand_parent: Migration Assistant for OpenSearch permalink: /migration-assistant/migration-phases/reroute-source-to-proxy/ --- +# Reroute Client Traffic + +**Note:** This page is only relevant if you are using Capture-and-Replay to avoid downtime during a migration. If you are only performing backfill migration or can tolerate downtime, you can skip this step. +{: .note} + ## Capture proxy data replication If you're interested in capturing live traffic during your migration, Migration Assistant includes an Application Load Balancer for routing traffic to the capture proxy and the target cluster. Upstream client traffic must be routed through the capture proxy in order to replay the requests later. Before using the capture proxy, remember the following: diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/reroute-traffic-from-capture-proxy-to-target.md b/_migrate-or-upgrade/migration-assistant/migration-phases/reroute-traffic-from-capture-proxy-to-target.md index af889aab278..ad6c8fe5dba 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/reroute-traffic-from-capture-proxy-to-target.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/reroute-traffic-from-capture-proxy-to-target.md @@ -1,6 +1,6 @@ --- layout: default -title: Reroute traffic to target +title: Reroute Traffic to Target nav_order: 7 parent: Migration phases grand_parent: Migration Assistant for OpenSearch @@ -9,6 +9,9 @@ permalink: /migration-assistant/migration-phases/reroute-traffic-from-capture-pr # Switching traffic from the source cluster +**Note:** This page is only relevant if you are using Capture-and-Replay to avoid downtime during a migration. If you are only performing backfill migration or can tolerate downtime, you can skip this step. +{: .note} + After the source and target clusters are synchronized, traffic needs to be switched to the target cluster so that the source cluster can be taken offline. ## Assumptions diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/verifying-migration-tools/verifying-migration-tools.md b/_migrate-or-upgrade/migration-assistant/migration-phases/verifying-migration-tools/verifying-migration-tools.md deleted file mode 100644 index 27f5cff31ac..00000000000 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/verifying-migration-tools/verifying-migration-tools.md +++ /dev/null @@ -1,20 +0,0 @@ ---- -layout: default -title: Verifying migration tools -parent: Migrations phases -parent: Verifying migration tools -nav_order: 3 -has_children: true -permalink: /migration-assistant/migration-phases/verifying-migration-tools/verifying-migration-tools/ ---- - -# Verifying migration tools - -Before you begin migrating data, it is important to verify that each component of your migration workflow is correctly deployed and functioning as expected. This step reduces the risk of disruptions during cutover and helps identify misconfigurations early in the process. - -Use the following pages to verify key components: - -- [Verifying backfill components]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/verifying-migration-tools/verifying-backfill-components/) -- [Verifying live capture components]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/verifying-migration-tools/verifying-live-capture-components/) - - diff --git a/_migrate-or-upgrade/rolling-upgrade/appendix/rolling-upgrade-lab.md b/_migrate-or-upgrade/rolling-upgrade/appendix/rolling-upgrade-lab.md index 6b8afc7d79e..13004e608b5 100644 --- a/_migrate-or-upgrade/rolling-upgrade/appendix/rolling-upgrade-lab.md +++ b/_migrate-or-upgrade/rolling-upgrade/appendix/rolling-upgrade-lab.md @@ -36,7 +36,7 @@ To use, invoke class="codeblock-label" # Rolling upgrade lab -You can follow these steps on your own compatible host to recreate the same cluster state the OpenSearch Project used for testing [rolling upgrades]({{site.url}}{{site.baseurl}}/igrate-or-upgrade/rolling-upgrade/). This exercise is useful if you want to test the upgrade process in a development environment. +You can follow these steps on your own compatible host to recreate the same cluster state the OpenSearch Project used for testing [rolling upgrades]({{site.url}}{{site.baseurl}}/migrate-or-upgrade/rolling-upgrade/). This exercise is useful if you want to test the upgrade process in a development environment. The steps used in this lab were validated on an arbitrarily chosen [Amazon Elastic Compute Cloud (Amazon EC2)](https://aws.amazon.com/ec2/) `t2.large` instance using [Amazon Linux 2](https://aws.amazon.com/amazon-linux-2/) kernel version `Linux 5.10.162-141.675.amzn2.x86_64` and [Docker](https://www.docker.com/) version `20.10.17, build 100c701`. The instance was provisioned with an attached 20 GiB gp2 [Amazon EBS](https://aws.amazon.com/ebs/) root volume. These specifications are included for informational purposes and do not represent hardware requirements for OpenSearch or OpenSearch Dashboards. @@ -889,4 +889,3 @@ Review the following resoures to learn more about how OpenSearch works: - [REST API reference]({{site.url}}{{site.baseurl}}/api-reference/index/) - [Quickstart guide for OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/dashboards/quickstart-dashboards/) - [About Security in OpenSearch]({{site.url}}{{site.baseurl}}/security/index/) - diff --git a/_migrate-or-upgrade/snapshot-restore/index.md b/_migrate-or-upgrade/snapshot-restore/index.md index b8d21634476..299e4527e65 100644 --- a/_migrate-or-upgrade/snapshot-restore/index.md +++ b/_migrate-or-upgrade/snapshot-restore/index.md @@ -1,12 +1,124 @@ --- layout: default title: Snapshot restore -nav_order: 20 +nav_order: 30 has_toc: false permalink: /migrate-or-upgrade/snapshot-restore/ redirect_from: --- -# Snapshot and Restore +# Snapshot and restore for migration -To be completed +Snapshots are one of the most reliable methods for migrating data between OpenSearch clusters. This approach is particularly useful when you need to move data from one environment to another, such as migrating from a proof-of-concept cluster to a production environment, or when performing major version upgrades that require a fresh cluster deployment. + +## When to use snapshot and restore for migration + +Snapshot and restore is ideal for migration scenarios when: + +- **Migrating between different OpenSearch versions** where in-place upgrades aren't supported +- **Moving to a different infrastructure** (on-premises to cloud, different cloud providers) +- **Changing cluster architecture** (different node configurations, shard strategies) +- **Zero-downtime requirements** aren't critical and you can afford some downtime +- **Large data volumes** where other migration methods might be impractical +- **Complete cluster migration** including indexes, settings, and metadata + +## Migration workflow overview + +A typical snapshot-based migration follows this workflow: + +1. **Prepare the source cluster** - Ensure cluster health and configure snapshot repository +2. **Create snapshot repository** - Set up shared storage accessible by both clusters +3. **Take comprehensive snapshots** - Capture all necessary indexes and cluster state +4. **Set up target cluster** - Deploy and configure the destination OpenSearch cluster +5. **Register repository on target** - Connect the target cluster to the snapshot repository +6. **Restore snapshots** - Selectively restore indexes and configurations +7. **Validate and test** - Verify data integrity and application functionality +8. **Switch traffic** - Update applications to use the new cluster + +## Key considerations for migration + +### Data consistency +Snapshots capture data as it existed when the snapshot was initiated, but they're not instantaneous. For migration purposes, consider: +- **Stopping writes** to ensure data consistency during the final snapshot +- **Taking incremental snapshots** to minimize the final downtime window +- **Planning for data that changes** during the migration process + +### Version compatibility +- Snapshots are **forward compatible by one major version** +- For larger version gaps, you may need to restore to an intermediate cluster, reindex, and take new snapshots +- Always verify compatibility between source and target OpenSearch versions + +### Storage requirements +- **Incremental nature** means frequent snapshots don't significantly increase storage usage +- **Plan storage capacity** for the full dataset plus incremental changes +- **Consider network bandwidth** for cloud-based repositories + +## Snapshot repository options for migration + +### Shared file systems +Best for migrations within the same infrastructure where both clusters can access shared storage. + +### Amazon S3 +Ideal for cloud migrations or when migrating between different environments. Provides durability and accessibility across regions. + +### Azure Blob Storage +Suitable for Azure-based migrations or hybrid cloud scenarios. + +### Cross-cloud considerations +When migrating between different cloud providers, consider: +- **Data transfer costs** and time requirements +- **Network connectivity** between source cluster, storage, and target cluster +- **Security and access controls** across different environments + +## Migration-specific restore options + +When restoring for migration purposes, you have several options to customize the process: + +### Selective restoration +- **Choose specific indexes** rather than restoring everything +- **Exclude system indexes** that might conflict with target cluster configuration +- **Rename indexes** to avoid conflicts or implement new naming conventions + +### Index settings modification +- **Update replica counts** to match target cluster capacity +- **Modify shard allocation** for different node configurations +- **Adjust refresh intervals** and other performance settings + +### Remote snapshot restoration +For large datasets, consider using `storage_type: remote_snapshot` to: +- **Reduce initial restore time** by keeping data in the repository +- **Save local storage** on the target cluster +- **Enable faster access** to historical data + +## Security considerations for migration + +When migrating with snapshots: + +- **Exclude security indexes** (`.opendistro_security`) from snapshots to avoid conflicts +- **Plan security configuration** separately from data migration +- **Use appropriate access controls** for snapshot repositories +- **Consider encryption** for sensitive data in transit and at rest + +## Monitoring and validation + +During migration: + +- **Monitor snapshot progress** using the snapshot status API +- **Validate data integrity** after restoration +- **Test application functionality** before switching traffic +- **Keep source cluster available** until migration is fully validated + +## Next steps + +For detailed technical implementation: + +- **Snapshot creation and management**: See [Take and restore snapshots]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore/) +- **API reference**: See [Snapshot APIs]({{site.url}}{{site.baseurl}}/api-reference/snapshots/index/) for complete API documentation +- **Automated snapshots**: See [Snapshot management]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/snapshots/snapshot-management/) for scheduling and automation +- **Alternative migration methods**: Consider [Migration Assistant]({{site.url}}{{site.baseurl}}/migration-assistant/) for more complex migration scenarios + +## Related migration approaches + +- **[Migration Assistant]({{site.url}}{{site.baseurl}}/migration-assistant/)** - For zero-downtime migrations with live traffic capture +- **[Rolling upgrades]({{site.url}}{{site.baseurl}}/migrate-or-upgrade/rolling-upgrade/)** - For in-place version upgrades +- **Remote reindex** - For selective data migration between clusters diff --git a/assets/js/nav-scroll.js b/assets/js/nav-scroll.js index df74ba807f2..6b20b08ab2c 100644 --- a/assets/js/nav-scroll.js +++ b/assets/js/nav-scroll.js @@ -41,12 +41,28 @@ document.addEventListener('DOMContentLoaded', () => { }); }; + // Get breadcrumb links to determine which navigation items should be expanded + const breadcrumbLinks = document.querySelectorAll('.breadcrumb-nav a'); + const breadcrumbUrls = Array.from(breadcrumbLinks).map(link => { + // Normalize URLs by removing trailing slashes and getting pathname + const url = new URL(link.href); + return url.pathname.replace(/\/$/, ''); + }); + const topLevelUls = Array.from(navParent.children).filter(element => element.tagName === 'UL'); topLevelUls.forEach((element) => { const listItems = Array.from(element.children).filter(element => element.tagName === 'LI'); listItems.forEach((element) => { const active = element.querySelector('a.active'); - if (active) { + + // Check if any descendant links are in the breadcrumb trail + const hasDescendantInBreadcrumb = Array.from(element.querySelectorAll('a')).some(anchor => { + const anchorUrl = new URL(anchor.href); + const normalizedAnchorUrl = anchorUrl.pathname.replace(/\/$/, ''); + return breadcrumbUrls.includes(normalizedAnchorUrl); + }); + + if (active || hasDescendantInBreadcrumb) { setSubTreeAriaExpanded(element, true); } else { setSubTreeAriaExpanded(element, false); @@ -55,13 +71,13 @@ document.addEventListener('DOMContentLoaded', () => { }); } + // Configure aria attributes for all navigation items based on active state and breadcrumbs + configureAriaAttributes(); + // Give keyboard focus to the active navigation item, and ensure // it is scrolled into view. const activeNavItem = navParent.querySelector('a.active'); if (activeNavItem) { - - configureAriaAttributes(); - // The active navigation item needs to have the tabindex="0" wheras all of the other items // are excluded from the TAB order according. activeNavItem.setAttribute('tabindex', '0'); From 5dc7b8c468405b9cc9af8b8f7cb0ead15de1f0d1 Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Wed, 23 Jul 2025 10:51:13 -0500 Subject: [PATCH 21/62] Made TOC casing titles consistent and changed the order. Signed-off-by: Brian Presley --- _migrate-or-upgrade/migration-assistant/index.md | 2 +- .../migration-assistant/migration-phases/deploy/index.md | 2 +- _migrate-or-upgrade/rolling-upgrade/index.md | 4 ++-- _migrate-or-upgrade/snapshot-restore/index.md | 4 ++-- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/_migrate-or-upgrade/migration-assistant/index.md b/_migrate-or-upgrade/migration-assistant/index.md index e59f277ac20..65a2217c704 100644 --- a/_migrate-or-upgrade/migration-assistant/index.md +++ b/_migrate-or-upgrade/migration-assistant/index.md @@ -1,7 +1,7 @@ --- layout: default title: Migration Assistant for OpenSearch -nav_order: 21 +nav_order: 30 has_children: true permalink: /migration-assistant/ redirect_from: diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/index.md b/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/index.md index 7767e5d7483..c2fed82de2a 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/index.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/index.md @@ -12,7 +12,7 @@ redirect_from: # Deploy -This document assumes you have performed [assessment]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/assessment/) to understand any upgrade breaking changes and limitations before beginning. +This document assumes you have performed [assessment]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/assessment/) to understand upgrade breaking changes and limitations before beginning. --- diff --git a/_migrate-or-upgrade/rolling-upgrade/index.md b/_migrate-or-upgrade/rolling-upgrade/index.md index 0a012b6da13..cf1d052d243 100644 --- a/_migrate-or-upgrade/rolling-upgrade/index.md +++ b/_migrate-or-upgrade/rolling-upgrade/index.md @@ -1,7 +1,7 @@ --- layout: default -title: Rolling upgrade -nav_order: 10 +title: Rolling Upgrade +nav_order: 20 has_toc: true permalink: /migrate-or-upgrade/rolling-upgrade/ redirect_from: diff --git a/_migrate-or-upgrade/snapshot-restore/index.md b/_migrate-or-upgrade/snapshot-restore/index.md index 299e4527e65..cc96fed2d97 100644 --- a/_migrate-or-upgrade/snapshot-restore/index.md +++ b/_migrate-or-upgrade/snapshot-restore/index.md @@ -1,7 +1,7 @@ --- layout: default -title: Snapshot restore -nav_order: 30 +title: Snapshot Restore +nav_order: 10 has_toc: false permalink: /migrate-or-upgrade/snapshot-restore/ redirect_from: From 5557037c43cca74f7a14370b6555bfff253a5351 Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Wed, 23 Jul 2025 18:14:00 -0500 Subject: [PATCH 22/62] Allow symlink to MA without warnings. Signed-off-by: Brian Presley --- _config.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/_config.yml b/_config.yml index 56b481c5195..4b7fc08b427 100644 --- a/_config.yml +++ b/_config.yml @@ -365,3 +365,4 @@ exclude: - _site/ - spec-insert - release-notes + - _migration-assistant/ From 27135142e82eb7d6a78cd9a4ff08e2b732f5b6ce Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Wed, 23 Jul 2025 18:17:40 -0500 Subject: [PATCH 23/62] Updated verbiage for MA zstd limitation Signed-off-by: Brian Presley --- .../migration-assistant/is-migration-assistant-right-for-you.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_migrate-or-upgrade/migration-assistant/is-migration-assistant-right-for-you.md b/_migrate-or-upgrade/migration-assistant/is-migration-assistant-right-for-you.md index ccfe022edd4..8c7a84e15d6 100644 --- a/_migrate-or-upgrade/migration-assistant/is-migration-assistant-right-for-you.md +++ b/_migrate-or-upgrade/migration-assistant/is-migration-assistant-right-for-you.md @@ -164,7 +164,7 @@ To use `Reindex-from-Snapshot` (RFS), ensure the following: - `include_global_state: true` – Ensures that global cluster state is included. - `compress: false` – Disables metadata compression, which is required for compatibility with RFS. - Shards of up to **80 GiB** are supported by default. Larger shard sizes can be configured, **except in AWS GovCloud (US)**, where 80 GiB is the maximum. -- OS 2.9+ shapshots of indexes using `zstd` or `zstd_no_dict` codecs are currently not supported. If currently required to migrate with `Reindex-from-Snapshot`, you must perform a reindex on the source cluster to `default` or `best_compression` prior to using a new snapshot with RFS. +- In OpenSearch 2.9 and later, snapshots of indexes that use the zstd or zstd_no_dict codecs are not supported. If you need to migrate these indexes using `Reindex -from-Snapshot`, you must first reindex them on the source cluster using either `default` or `best_compression` before creating a new snapshot for use with RFS. ### Capture and Replay From 07bd77a138622788678942d70955bcf472c5b867 Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Wed, 23 Jul 2025 21:22:43 -0500 Subject: [PATCH 24/62] Fix breadcrumbs and added redirects for old page paths Signed-off-by: Brian Presley --- _config.yml | 6 -- _layouts/default.html | 74 +++++++++++++-- .../migration-assistant/architecture.md | 4 +- .../migration-assistant/index.md | 2 +- .../is-migration-assistant-right-for-you.md | 5 +- .../migration-assistant/key-components.md | 4 +- .../accessing-the-migration-console.md | 2 + .../migration-console/index.md | 2 +- .../migration-console-commands-references.md | 2 + .../migration-phases/backfill.md | 2 + .../deploy/configuration-options.md | 1 + ...d-security-groups-for-existing-clusters.md | 1 + .../migration-phases/deploy/index.md | 1 + .../migration-phases/index.md | 93 +++++-------------- ...itching-traffic-from-the-source-cluster.md | 3 +- .../migrate-metadata/index.md | 1 + .../replay-captured-traffic.md | 1 + 17 files changed, 115 insertions(+), 89 deletions(-) diff --git a/_config.yml b/_config.yml index 4b7fc08b427..036b6a4ce61 100644 --- a/_config.yml +++ b/_config.yml @@ -266,12 +266,6 @@ defaults: values: section: "benchmark" section-name: "Benchmark" - - - scope: - path: "_migrate-or-upgrade/migration-assistant" - values: - section: "migration-assistant" - section-name: "Migration Assistant for OpenSearch" - scope: path: "_migration-assistant" diff --git a/_layouts/default.html b/_layouts/default.html index 36ad3787024..37ac628203d 100755 --- a/_layouts/default.html +++ b/_layouts/default.html @@ -165,8 +165,24 @@
              {% comment %} Add collection as first level {% endcomment %} {% if page.collection %} - {% assign collection_config = site.opensearch_collection.collections[page.collection] %} - {% if collection_config %} + {% assign collection_config = nil %} + {% if page.section == "opensearch" %} + {% assign collection_config = site.opensearch_collection.collections[page.collection] %} + {% elsif page.section == "data-prepper" %} + {% assign collection_config = site.data_prepper_collection.collections[page.collection] %} + {% elsif page.section == "clients" %} + {% assign collection_config = site.clients_collection.collections[page.collection] %} + {% elsif page.section == "benchmark" %} + {% assign collection_config = site.benchmark_collection.collections[page.collection] %} + {% elsif page.section == "migration-assistant" %} + {% assign collection_config = site.migration_assistant_collection.collections[page.collection] %} + {% else %} + {% comment %} Fallback: try opensearch collection first {% endcomment %} + {% assign collection_config = site.opensearch_collection.collections[page.collection] %} + {% endif %} + + {% comment %} Only show collection breadcrumb if it's different from current page title {% endcomment %} + {% if collection_config and collection_config.name != page.title %}
            1. {{ collection_config.name }}
              2. @@ -174,11 +190,57 @@ {% endif %} {% if page.parent %} - {% if page.grand_parent %} -
            2. {{ page.grand_parent }}
              3. -
            3. {{ page.parent }}
              4. + {% comment %} Find the parent page {% endcomment %} + {% assign parent_page = nil %} + {% assign grand_parent_page = nil %} + + {% comment %} Look for parent in the same collection {% endcomment %} + {% if page.collection %} + {% assign collection_pages = site[page.collection] %} + {% for p in collection_pages %} + {% if p.title == page.parent %} + {% assign parent_page = p %} + {% break %} + {% endif %} + {% endfor %} + {% else %} + {% comment %} Look for parent in site pages {% endcomment %} + {% for p in site.pages %} + {% if p.title == page.parent %} + {% assign parent_page = p %} + {% break %} + {% endif %} + {% endfor %} + {% endif %} + + {% comment %} If we have a grand parent, find that too {% endcomment %} + {% if page.grand_parent and parent_page %} + {% if page.collection %} + {% assign collection_pages = site[page.collection] %} + {% for p in collection_pages %} + {% if p.title == page.grand_parent %} + {% assign grand_parent_page = p %} + {% break %} + {% endif %} + {% endfor %} + {% else %} + {% for p in site.pages %} + {% if p.title == page.grand_parent %} + {% assign grand_parent_page = p %} + {% break %} + {% endif %} + {% endfor %} + {% endif %} + {% endif %} + + {% if page.grand_parent and grand_parent_page and parent_page %} +
            4. {{ page.grand_parent }}
              5. +
            5. {{ page.parent }}
              6. + {% elsif parent_page %} +
            6. {{ page.parent }}
              7. {% else %} -
            7. {{ page.parent }}
              8. + {% comment %} Fallback: show parent name without link if page not found {% endcomment %} +
            8. {{ page.parent }}
              9. {% endif %} {% endif %} diff --git a/_migrate-or-upgrade/migration-assistant/architecture.md b/_migrate-or-upgrade/migration-assistant/architecture.md index ac81532cad9..97ec48f684e 100644 --- a/_migrate-or-upgrade/migration-assistant/architecture.md +++ b/_migrate-or-upgrade/migration-assistant/architecture.md @@ -4,6 +4,8 @@ title: Architecture nav_order: 15 parent: Migration Assistant for OpenSearch permalink: /migration-assistant/architecture/ +redirect_from: + - /migration-assistant/overview/architecture/ --- # Architecture @@ -21,4 +23,4 @@ Each node in the diagram correlates to the following steps in the migration proc 3. Using the migration console, a point-in-time snapshot is taken. Once the snapshot completes, `Metadata-Migration-Tool` is used to establish indexes, templates, component templates, and aliases on the target cluster. With continuous traffic capture in place, `Reindex-from-Snapshot` process to migrate data from source. 4. Once `Reindex-from-Snapshot` is complete, captured traffic is replayed from Amazon MSK to the target cluster by `Traffic-Capture-Replayer`. 5. Performance and behavior of traffic sent to the source and target clusters are compared by reviewing logs and metrics. -6. After confirming that the target cluster's functionality meets expectations, clients are redirected to the new target. \ No newline at end of file +6. After confirming that the target cluster's functionality meets expectations, clients are redirected to the new target. diff --git a/_migrate-or-upgrade/migration-assistant/index.md b/_migrate-or-upgrade/migration-assistant/index.md index 65a2217c704..b9add359a27 100644 --- a/_migrate-or-upgrade/migration-assistant/index.md +++ b/_migrate-or-upgrade/migration-assistant/index.md @@ -26,7 +26,7 @@ items: # Migration Assistant for OpenSearch -Migration Assistant for OpenSearch aids you in successfully performing an end-to-end, zero-downtime migration to OpenSearch from other search providers. It helps with the following scenarios: +Migration Assistant for OpenSearch aids you in successfully performing an end-to-end, zero-downtime upgrade and migration to OpenSearch. - **Metadata migration**: Migrating cluster metadata, such as index settings, aliases, and templates. - **Backfill migration**: Migrating existing or historical data from a source to a target cluster. diff --git a/_migrate-or-upgrade/migration-assistant/is-migration-assistant-right-for-you.md b/_migrate-or-upgrade/migration-assistant/is-migration-assistant-right-for-you.md index 8c7a84e15d6..025c02dfe53 100644 --- a/_migrate-or-upgrade/migration-assistant/is-migration-assistant-right-for-you.md +++ b/_migrate-or-upgrade/migration-assistant/is-migration-assistant-right-for-you.md @@ -2,9 +2,10 @@ layout: default title: Is Migration Assistant right for you? nav_order: 10 -parent: Overview +parent: Migration Assistant for OpenSearch +permalink: /migration-assistant/is-migration-assistant-right-for-you/ redirect_from: - - /migration-assistant/is-migration-assistant-right-for-you/ + - /migration-assistant/overview/is-migration-assistant-right-for-you/ --- # Is Migration Assistant right for you? diff --git a/_migrate-or-upgrade/migration-assistant/key-components.md b/_migrate-or-upgrade/migration-assistant/key-components.md index ab0ea4dd55b..deddcc38db1 100644 --- a/_migrate-or-upgrade/migration-assistant/key-components.md +++ b/_migrate-or-upgrade/migration-assistant/key-components.md @@ -4,6 +4,8 @@ parent: Migration Assistant for OpenSearch title: Key components nav_order: 20 permalink: /migration-assistant/key-components/ +redirect_from: + - /migration-assistant/overview/key-components/ --- # Key components @@ -36,4 +38,4 @@ The metadata migration tool integrated into the Migration CLI can be used indepe ## Target cluster -The target cluster is the destination cluster for migration or comparison cluster in an A/B test. \ No newline at end of file +The target cluster is the destination cluster for migration or comparison cluster in an A/B test. diff --git a/_migrate-or-upgrade/migration-assistant/migration-console/accessing-the-migration-console.md b/_migrate-or-upgrade/migration-assistant/migration-console/accessing-the-migration-console.md index 1309e221def..151503d6f9e 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-console/accessing-the-migration-console.md +++ b/_migrate-or-upgrade/migration-assistant/migration-console/accessing-the-migration-console.md @@ -5,6 +5,8 @@ nav_order: 1 parent: Migration console grand_parent: Migration Assistant for OpenSearch permalink: /migration-assistant/migration-console/accessing-the-migration-console/ +redirect_from: + - /migration-console/accessing-the-migration-console/ --- # Accessing the migration console diff --git a/_migrate-or-upgrade/migration-assistant/migration-console/index.md b/_migrate-or-upgrade/migration-assistant/migration-console/index.md index 535c5734eb8..d45689b34d2 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-console/index.md +++ b/_migrate-or-upgrade/migration-assistant/migration-console/index.md @@ -10,6 +10,7 @@ has_toc: false permalink: /migration-assistant/migration-console/ redirect_from: - /migration-assistant/migration-console/index/ + - /migration-console/ --- # Migration console @@ -17,4 +18,3 @@ redirect_from: The Migrations Assistant deployment includes an Amazon Elastic Container Service (Amazon ECS) task that hosts tools that run different phases of the migration and check the progress or results of the migration. This ECS task is called the **migration console**. The migration console is a command line interface used to interact with the deployed components of the solution. This section provides information about how to access the migration console and what commands are supported. - diff --git a/_migrate-or-upgrade/migration-assistant/migration-console/migration-console-commands-references.md b/_migrate-or-upgrade/migration-assistant/migration-console/migration-console-commands-references.md index 7d3396e63bd..c08664c40fa 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-console/migration-console-commands-references.md +++ b/_migrate-or-upgrade/migration-assistant/migration-console/migration-console-commands-references.md @@ -5,6 +5,8 @@ nav_order: 2 parent: Migration console grand_parent: Migration Assistant for OpenSearch permalink: /migration-assistant/migration-console/migration-console-command-reference/ +redirect_from: + - /migration-console/migration-console-commands-references/ --- # Migration console command reference diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/backfill.md b/_migrate-or-upgrade/migration-assistant/migration-phases/backfill.md index bf66bf09116..bb93ff8f8a8 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/backfill.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/backfill.md @@ -5,6 +5,8 @@ nav_order: 6 parent: Migration phases grand_parent: Migration Assistant for OpenSearch permalink: /migration-assistant/migration-phases/backfill/ +redirect_from: + - /migration-phases/backfill/ --- # Using backfill diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/configuration-options.md b/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/configuration-options.md index 1e5ee24c0dd..1c3f256ee47 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/configuration-options.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/configuration-options.md @@ -7,6 +7,7 @@ parent: Deploy permalink: /migration-assistant/migration-phases/deploy/configuration-options/ redirect_from: - /migration-assistant/migration-phases/deploying-migration-assistant/configuration-options/ + - /deploying-migration-assistant/configuration-options/ --- # Configuration options diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/iam-and-security-groups-for-existing-clusters.md b/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/iam-and-security-groups-for-existing-clusters.md index e19f01a7f3c..3f4ab0ccb53 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/iam-and-security-groups-for-existing-clusters.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/iam-and-security-groups-for-existing-clusters.md @@ -7,6 +7,7 @@ parent: Deploy permalink: /migration-assistant/migration-phases/deploy/iam-and-security-groups-for-existing-clusters/ redirect_from: - /migration-assistant/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters/ + - /deploying-migration-assistant/iam-and-security-groups-for-existing-clusters/ --- # IAM and security groups for existing clusters diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/index.md b/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/index.md index 63872bbfedb..3cd800038bc 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/index.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/index.md @@ -8,6 +8,7 @@ has_children: true permalink: /migration-assistant/migration-phases/deploy/ redirect_from: - /migration-assistant/getting-started-with-data-migration/ + - /deploying-migration-assistant/ --- # Deploy diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/index.md b/_migrate-or-upgrade/migration-assistant/migration-phases/index.md index 29d7d1068b5..87139d18e1a 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/index.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/index.md @@ -8,32 +8,38 @@ nav_exclude: false has_children: true has_toc: false permalink: /migration-assistant/migration-phases/ +redirect_from: + - /migration-assistant/overview/migration-phases/ + - /migration-phases/ --- # Migration phases -This page outlines the core phases of migrating with Migration Assistant. Each scenario below consists of a sequence of common steps. Expand a scenario to explore the detailed flow. +This page outlines the core phases of migrating with Migration Assistant. There are three migration scenarios, each consisting of a sequence of common steps. -
              -Scenario 1 – Backfill Only +
              +
              Scenario 1 – Backfill Only
              1. Assessment
                2. @@ -55,10 +61,10 @@ details[open] {
              2. Teardown
              -
              + -
              -Scenario 2 – Live Capture Only +
              +
              Scenario 2 – Live Capture Only
              1. Assessment
                2. @@ -81,10 +87,10 @@ details[open] {
              2. Teardown
              -
              + -
              -Scenario 3 – Live Capture with Backfill +
              +
              Scenario 3 – Live Capture with Backfill
              1. Assessment
                2. @@ -110,57 +116,4 @@ details[open] {
              2. Teardown
              -
              - - + diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/live-traffic-migration/switching-traffic-from-the-source-cluster.md b/_migrate-or-upgrade/migration-assistant/migration-phases/live-traffic-migration/switching-traffic-from-the-source-cluster.md index b5e2894c0f2..89a36a52c65 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/live-traffic-migration/switching-traffic-from-the-source-cluster.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/live-traffic-migration/switching-traffic-from-the-source-cluster.md @@ -6,6 +6,7 @@ nav_exclude: true permalink: /migration-assistant/migration-phases/live-traffic-migration/switching-traffic-from-the-source-cluster/ redirect_from: - /migration-assistant/migration-phases/switching-traffic-from-the-source-cluster/ + - /migration-phases/switching-traffic-from-the-source-cluster/ --- # Switching traffic from the source cluster @@ -52,4 +53,4 @@ Use the following steps to switch traffic from the source cluster to the target ## Fallback -If you need to fall back to the source cluster at any point during the switchover, revert the **Default rule** so that the Application Load Balancer routes to the **SourceProxy Target Group**. \ No newline at end of file +If you need to fall back to the source cluster at any point during the switchover, revert the **Default rule** so that the Application Load Balancer routes to the **SourceProxy Target Group**. diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/migrate-metadata/index.md b/_migrate-or-upgrade/migration-assistant/migration-phases/migrate-metadata/index.md index 8aae1ae655e..aaf454a8601 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/migrate-metadata/index.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/migrate-metadata/index.md @@ -9,6 +9,7 @@ has_toc: false permalink: /migration-assistant/migration-phases/migrate-metadata/ redirect_from: - /migration-assistant/migration-phases/migrating-metadata/ + - /migration-phases/migrating-metadata/ --- # Migrate metadata diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/replay-captured-traffic.md b/_migrate-or-upgrade/migration-assistant/migration-phases/replay-captured-traffic.md index b59f3f1278d..874bac4e2ad 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/replay-captured-traffic.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/replay-captured-traffic.md @@ -7,6 +7,7 @@ grand_parent: Migration Assistant for OpenSearch permalink: /migration-assistant/migration-phases/replay-captured-traffic/ redirect_from: - /migration-assistant/migration-phases/using-traffic-replayer/ + - /migration-phases/using-traffic-replayer/ --- # Using Traffic Replayer From ad062d7325e245e5f3c089f4ac4581bd387540b8 Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Thu, 24 Jul 2025 10:19:59 -0500 Subject: [PATCH 25/62] Updated broken links Signed-off-by: Brian Presley --- _migrate-or-upgrade/migration-assistant/key-components.md | 2 +- .../migration-assistant/migration-phases/deploy/index.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/_migrate-or-upgrade/migration-assistant/key-components.md b/_migrate-or-upgrade/migration-assistant/key-components.md index deddcc38db1..fdfc9d1e7e4 100644 --- a/_migrate-or-upgrade/migration-assistant/key-components.md +++ b/_migrate-or-upgrade/migration-assistant/key-components.md @@ -26,7 +26,7 @@ This component is designed for HTTP RESTful traffic. It forwards traffic to the ## Traffic Replayer -Acting as a traffic simulation tool, [Traffic Replayer](https://docs.opensearch.org/docs/latest/migration-assistant/migration-phases/live-traffic-migration/replay-captured-traffic/) replays recorded request traffic to a target cluster, mirroring source traffic patterns. It links original requests and their responses to those directed at the target cluster, facilitating comparative analysis. +Acting as a traffic simulation tool, [Traffic Replayer]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/replay-captured-traffic/) replays recorded request traffic to a target cluster, mirroring source traffic patterns. It links original requests and their responses to those directed at the target cluster, facilitating comparative analysis. ## Metadata-Migration-Tool diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/index.md b/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/index.md index 3cd800038bc..9024bc82bbd 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/index.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/index.md @@ -232,7 +232,7 @@ TARGET CLUSTER ConnectionResult(connection_message='Successfully connected!', connection_established=True, cluster_version='') ``` -To learn more about migration console commands, see [Migration console command reference](https://docs.opensearch.org/docs/latest/migration-assistant/migration-console/migration-console-commands-references/). +To learn more about migration console commands, see [Migration console command reference]({{site.url}}{{site.baseurl}}/migration-assistant/migration-console/migration-console-command-reference/). --- From 07b5dcb7cce3dde60f870ea936367bdf36709d7d Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Thu, 24 Jul 2025 19:29:28 -0500 Subject: [PATCH 26/62] Update to confirm to style guide Signed-off-by: Brian Presley --- _migrate-or-upgrade/index.md | 2 +- .../migration-assistant/architecture.md | 4 ++-- .../migration-assistant/index.md | 10 ++++----- .../is-migration-assistant-right-for-you.md | 2 +- .../migration-assistant/key-components.md | 6 +++--- .../accessing-the-migration-console.md | 2 +- .../migration-console/index.md | 2 +- .../migration-phases/assessment/index.md | 2 +- .../migration-phases/deploy/index.md | 21 +++++++++---------- .../migrate-metadata/index.md | 2 +- _migrate-or-upgrade/rolling-upgrade/index.md | 2 +- _migrate-or-upgrade/snapshot-restore/index.md | 2 +- 12 files changed, 28 insertions(+), 29 deletions(-) diff --git a/_migrate-or-upgrade/index.md b/_migrate-or-upgrade/index.md index 6269595c6eb..55e2646fc68 100644 --- a/_migrate-or-upgrade/index.md +++ b/_migrate-or-upgrade/index.md @@ -47,7 +47,7 @@ Stop nonessential indexing before upgrading. ## Migration and Upgrade Methods -### Rolling Upgrade +### Rolling upgrade Upgrade one node at a time while keeping the cluster operational. diff --git a/_migrate-or-upgrade/migration-assistant/architecture.md b/_migrate-or-upgrade/migration-assistant/architecture.md index 97ec48f684e..063a78fa819 100644 --- a/_migrate-or-upgrade/migration-assistant/architecture.md +++ b/_migrate-or-upgrade/migration-assistant/architecture.md @@ -20,7 +20,7 @@ Each node in the diagram correlates to the following steps in the migration proc 1. Client traffic is directed to the existing cluster. 2. An Application Load Balancer with capture proxies relays traffic to a source while replicating data to Amazon Managed Streaming for Apache Kafka (Amazon MSK). -3. Using the migration console, a point-in-time snapshot is taken. Once the snapshot completes, `Metadata-Migration-Tool` is used to establish indexes, templates, component templates, and aliases on the target cluster. With continuous traffic capture in place, `Reindex-from-Snapshot` process to migrate data from source. -4. Once `Reindex-from-Snapshot` is complete, captured traffic is replayed from Amazon MSK to the target cluster by `Traffic-Capture-Replayer`. +3. Using the migration console, a point-in-time snapshot is taken. Once the snapshot completes, the Metadata Migration Tool is used to establish indexes, templates, component templates, and aliases on the target cluster. With continuous traffic capture in place, `Reindex-from-Snapshot` processes to migrate data from source. +4. Once `Reindex-from-Snapshot` is complete, captured traffic is replayed from Amazon MSK to the target cluster by Traffic Replayer. 5. Performance and behavior of traffic sent to the source and target clusters are compared by reviewing logs and metrics. 6. After confirming that the target cluster's functionality meets expectations, clients are redirected to the new target. diff --git a/_migrate-or-upgrade/migration-assistant/index.md b/_migrate-or-upgrade/migration-assistant/index.md index b9add359a27..18f1537f352 100644 --- a/_migrate-or-upgrade/migration-assistant/index.md +++ b/_migrate-or-upgrade/migration-assistant/index.md @@ -26,12 +26,12 @@ items: # Migration Assistant for OpenSearch -Migration Assistant for OpenSearch aids you in successfully performing an end-to-end, zero-downtime upgrade and migration to OpenSearch. +Migration Assistant for OpenSearch helps you successfully perform an end-to-end, zero-downtime upgrade and migration to OpenSearch. -- **Metadata migration**: Migrating cluster metadata, such as index settings, aliases, and templates. -- **Backfill migration**: Migrating existing or historical data from a source to a target cluster. -- **Live traffic migration**: Replicating live ongoing traffic from a source to a target cluster. -- **Comparative tooling**: Comparing the performance and behaviors of an existing cluster with a prospective new one. +- **Metadata migration**: Migrate cluster metadata, such as index settings, aliases, and templates. +- **Backfill migration**: Migrate existing or historical data from a source to a target cluster. +- **Live traffic migration**: Replicate live ongoing traffic from a source to a target cluster. +- **Comparative tooling**: Compare the performance and behaviors of an existing cluster with a prospective new one. This user guide focuses on conducting a comprehensive migration involving both existing and live data with zero downtime and the option to back out of a migration. diff --git a/_migrate-or-upgrade/migration-assistant/is-migration-assistant-right-for-you.md b/_migrate-or-upgrade/migration-assistant/is-migration-assistant-right-for-you.md index 025c02dfe53..f9c7bb433e9 100644 --- a/_migrate-or-upgrade/migration-assistant/is-migration-assistant-right-for-you.md +++ b/_migrate-or-upgrade/migration-assistant/is-migration-assistant-right-for-you.md @@ -12,7 +12,7 @@ redirect_from: Whether Migration Assistant is right for you depends on your upgrade path, infrastructure complexity, and operational goals. This page will help you evaluate whether Migration Assistant fits your use case. -Migration Assistant addresses key limitations in traditional migration approaches. For example, if you're upgrading across multiple major versions—such as from Elasticsearch 6.8 to OpenSearch 2.19—Migration Assistant enables you to complete the process in a single step. Other methods, like rolling upgrades or snapshot restores, require upgrading through each major version, often with reindexing at every stage. +Migration Assistant addresses key limitations in traditional migration approaches. For example, if you're upgrading across multiple major versions—such as from Elasticsearch 6.8 to OpenSearch 2.19—you can use Migration Assistant to complete the process in a single step. Other methods, like rolling upgrades or snapshot restores, require upgrading through each major version, often with reindexing at every stage. Migration Assistant also supports live traffic replication, enabling zero-downtime migrations. This makes it a strong fit for environments where minimizing service disruption is critical. diff --git a/_migrate-or-upgrade/migration-assistant/key-components.md b/_migrate-or-upgrade/migration-assistant/key-components.md index fdfc9d1e7e4..888535b7665 100644 --- a/_migrate-or-upgrade/migration-assistant/key-components.md +++ b/_migrate-or-upgrade/migration-assistant/key-components.md @@ -16,9 +16,9 @@ The following are the key components of Migration Assistant. In this solution, your source cluster operates on either Elasticsearch or OpenSearch and is hosted on Amazon Elastic Compute Cloud (Amazon EC2) instances or in a similar computing environment. Traffic is rerouted from the source cluster to a traffic capture proxy and replayed to a target typically on a later version of OpenSearch. -## Migration Console +## Migration console -The migration console provides a migration-specific CLI and offers a variety of tools for streamlining the migration process. Everything necessary for completing a migration, other than cleaning up the migration resources, can be performed through this console. +The migration console provides a migration-specific CLI and offers a variety of tools for streamlining the migration process. You can perform everything necessary for completing a migration, other than cleaning up the migration resources, through this console. ## Traffic Capture Proxy @@ -28,7 +28,7 @@ This component is designed for HTTP RESTful traffic. It forwards traffic to the Acting as a traffic simulation tool, [Traffic Replayer]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/replay-captured-traffic/) replays recorded request traffic to a target cluster, mirroring source traffic patterns. It links original requests and their responses to those directed at the target cluster, facilitating comparative analysis. -## Metadata-Migration-Tool +## Metadata Migration Tool The metadata migration tool integrated into the Migration CLI can be used independently to migrate cluster metadata, including index mappings, index configuration settings, templates, component templates, and aliases. diff --git a/_migrate-or-upgrade/migration-assistant/migration-console/accessing-the-migration-console.md b/_migrate-or-upgrade/migration-assistant/migration-console/accessing-the-migration-console.md index 151503d6f9e..9cdbbc1c344 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-console/accessing-the-migration-console.md +++ b/_migrate-or-upgrade/migration-assistant/migration-console/accessing-the-migration-console.md @@ -11,7 +11,7 @@ redirect_from: # Accessing the migration console -The Bootstrap box deployed through Migration Assistant contains a script that simplifies access to the migration console through that instance. +The bootstrap box deployed through Migration Assistant contains a script that simplifies access to the migration console through that instance. To access the migration console, use the following commands: diff --git a/_migrate-or-upgrade/migration-assistant/migration-console/index.md b/_migrate-or-upgrade/migration-assistant/migration-console/index.md index d45689b34d2..5039a2fc8f7 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-console/index.md +++ b/_migrate-or-upgrade/migration-assistant/migration-console/index.md @@ -15,6 +15,6 @@ redirect_from: # Migration console -The Migrations Assistant deployment includes an Amazon Elastic Container Service (Amazon ECS) task that hosts tools that run different phases of the migration and check the progress or results of the migration. This ECS task is called the **migration console**. The migration console is a command line interface used to interact with the deployed components of the solution. +The Migration Assistant deployment includes an Amazon Elastic Container Service (Amazon ECS) task that hosts tools that run different phases of the migration and check the progress or results of the migration. This ECS task is called the **migration console**. The migration console is a command-line interface used to interact with the deployed components of the solution. This section provides information about how to access the migration console and what commands are supported. diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/assessment/index.md b/_migrate-or-upgrade/migration-assistant/migration-phases/assessment/index.md index 8eaba3d440e..ed16525ca7e 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/assessment/index.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/assessment/index.md @@ -11,7 +11,7 @@ permalink: /migration-assistant/migration-phases/assessment/ # Assessment -The goal of the Migration Assistant is to streamline the process of migrating from one location or version of Elasticsearch/OpenSearch to another. However, completing a migration sometimes requires resolving client compatibility issues before they can communicate directly with the target cluster. +The goal of Migration Assistant is to streamline the process of migrating from one location or version of Elasticsearch/OpenSearch to another. However, completing a migration sometimes requires resolving client compatibility issues before they can communicate directly with the target cluster. ## Understanding breaking changes diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/index.md b/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/index.md index 9024bc82bbd..689c44e566f 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/index.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/index.md @@ -14,16 +14,15 @@ redirect_from: # Deploy This document assumes you have performed [assessment]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/assessment/) to understand upgrade breaking changes and limitations before beginning. -This quickstart outlines how to deploy Migration Assistant for OpenSearch and execute an existing data migration using `Reindex-from-Snapshot` (RFS). It uses AWS for illustrative purposes. However, the steps can be modified for use with other cloud providers. +This guide outlines how to deploy Migration Assistant for OpenSearch and execute an existing data migration using `Reindex-from-Snapshot` (RFS). It uses AWS for illustrative purposes. However, you can modify the steps for use with other cloud providers. -Before using this quickstart, make sure you review [Is Migration Assistant right for you?]({{site.url}}{{site.baseurl}}/migration-assistant/is-migration-assistant-right-for-you/). +Before using this quickstart, review [Is Migration Assistant right for you?]({{site.url}}{{site.baseurl}}/migration-assistant/is-migration-assistant-right-for-you/). - -Because this guide uses [AWS Cloud Development Kit (AWS CDK)](https://aws.amazon.com/cdk/), make sure that the `CDKToolkit` stack exists and is in the `CREATE_COMPLETE` state. For setup instructions, see the [CDK Toolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html). +Because this guide uses [AWS Cloud Development Kit (AWS CDK)](https://aws.amazon.com/cdk/), ensure that the `CDKToolkit` stack exists and is in the `CREATE_COMPLETE` state. For setup instructions, see the [CDK Toolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html). --- -## Step 1: Install Bootstrap on an Amazon EC2 instance (~10 minutes) +## Step 1: Install bootstrap on an Amazon EC2 instance (~10 minutes) To begin your migration, use the following steps to install a `bootstrap` box on an Amazon Elastic Compute Cloud (Amazon EC2) instance. The instance uses AWS CloudFormation to create and manage the stack. @@ -33,13 +32,13 @@ To begin your migration, use the following steps to install a `bootstrap` box on * **Stack Name:** `MigrationBootstrap` * **Stage Name:** `dev` * Choose **Next** after each step > **Acknowledge** > **Submit**. -4. Verify that the Bootstrap stack exists and is set to `CREATE_COMPLETE`. This process takes around 10 minutes to complete. +4. Verify that the bootstrap stack exists and is set to `CREATE_COMPLETE`. This process takes around 10 minutes to complete. --- -## Step 2: Set up Bootstrap instance access (~5 minutes) +## Step 2: Set up bootstrap instance access (~5 minutes) -Use the following steps to set up Bootstrap instance access: +Use the following steps to set up bootstrap instance access: 1. After deployment, find the EC2 instance ID for the `bootstrap-dev-instance`. 2. Create an AWS Identity and Access Management (IAM) policy using the following snippet, replacing ``, ``, ``, and `` with your information: @@ -66,9 +65,9 @@ Use the following steps to set up Bootstrap instance access: --- -## Step 3: Log in to Bootstrap and building Migration Assistant (~15 minutes) +## Step 3: Log in to bootstrap and build Migration Assistant (~15 minutes) -Next, log in to Bootstrap and build Migration Assistant using the following steps. +Next, log in to bootstrap and build Migration Assistant using the following steps. ### Prerequisites @@ -87,7 +86,7 @@ To use these steps, make sure you fulfill the following prerequisites: ``` {% include copy.html %} -3. Once logged in, run the following command from the shell of the Bootstrap instance in the `/opensearch-migrations` directory: +3. Once logged in, run the following command from the shell of the bootstrap instance in the `/opensearch-migrations` directory: ```bash ./initBootstrap.sh && cd deployment/cdk/opensearch-service-migration diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/migrate-metadata/index.md b/_migrate-or-upgrade/migration-assistant/migration-phases/migrate-metadata/index.md index aaf454a8601..cfdc78f441f 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/migrate-metadata/index.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/migrate-metadata/index.md @@ -1,6 +1,6 @@ --- layout: default -title: Migrate Metadata +title: Migrate metadata nav_order: 5 parent: Migration phases grand_parent: Migration Assistant for OpenSearch diff --git a/_migrate-or-upgrade/rolling-upgrade/index.md b/_migrate-or-upgrade/rolling-upgrade/index.md index cf1d052d243..64e3ca72a22 100644 --- a/_migrate-or-upgrade/rolling-upgrade/index.md +++ b/_migrate-or-upgrade/rolling-upgrade/index.md @@ -1,6 +1,6 @@ --- layout: default -title: Rolling Upgrade +title: Rolling upgrade nav_order: 20 has_toc: true permalink: /migrate-or-upgrade/rolling-upgrade/ diff --git a/_migrate-or-upgrade/snapshot-restore/index.md b/_migrate-or-upgrade/snapshot-restore/index.md index cc96fed2d97..c209b676e2d 100644 --- a/_migrate-or-upgrade/snapshot-restore/index.md +++ b/_migrate-or-upgrade/snapshot-restore/index.md @@ -1,6 +1,6 @@ --- layout: default -title: Snapshot Restore +title: Snapshot and restore nav_order: 10 has_toc: false permalink: /migrate-or-upgrade/snapshot-restore/ From b639732b688e236e3b1baf65b72361ef3e92f448 Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Thu, 24 Jul 2025 20:33:20 -0500 Subject: [PATCH 27/62] Update deploy to reference C&R Signed-off-by: Brian Presley --- .../migration-assistant/migration-phases/deploy/index.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/index.md b/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/index.md index 689c44e566f..e543a0ccede 100644 --- a/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/index.md +++ b/_migrate-or-upgrade/migration-assistant/migration-phases/deploy/index.md @@ -14,7 +14,9 @@ redirect_from: # Deploy This document assumes you have performed [assessment]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/assessment/) to understand upgrade breaking changes and limitations before beginning. -This guide outlines how to deploy Migration Assistant for OpenSearch and execute an existing data migration using `Reindex-from-Snapshot` (RFS). It uses AWS for illustrative purposes. However, you can modify the steps for use with other cloud providers. +This quickstart outlines how to deploy Migration Assistant for OpenSearch and execute an existing data migration using `Reindex-from-Snapshot` (RFS). It uses AWS for illustrative purposes. However, you can modify the steps for use with other cloud providers. + +**Note:** Although this page focuses on RFS-only deployment, you can add capture and replay functionality or replace RFS options with Capture and Replay as described in [Configuration options]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/deploy/configuration-options/#live-capture-migration-with-cr). Before using this quickstart, review [Is Migration Assistant right for you?]({{site.url}}{{site.baseurl}}/migration-assistant/is-migration-assistant-right-for-you/). From d70c8df93217372caf8001835738580138fa13e0 Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Thu, 24 Jul 2025 20:39:59 -0500 Subject: [PATCH 28/62] More convention changes Signed-off-by: Brian Presley --- _migrate-or-upgrade/index.md | 27 +++++++++---------- .../appendix/rolling-upgrade-lab.md | 2 +- 2 files changed, 14 insertions(+), 15 deletions(-) diff --git a/_migrate-or-upgrade/index.md b/_migrate-or-upgrade/index.md index 55e2646fc68..584a63e06b0 100644 --- a/_migrate-or-upgrade/index.md +++ b/_migrate-or-upgrade/index.md @@ -13,17 +13,17 @@ redirect_from: has_toc: true nav_exclude: true --- -# Migrate or Upgrade OpenSearch +# Migrate or upgrade OpenSearch -The OpenSearch Project releases regular updates that include new features, enhancements, and bug fixes. OpenSearch uses [Semantic Versioning](https://semver.org/), which means that breaking changes are only introduced between major version releases. To learn about upcoming features and fixes, review the [OpenSearch Project Roadmap](https://github.com/orgs/opensearch-project/projects/206) on GitHub. To view a list of previous releases or to learn more about how OpenSearch uses versioning, see [Release Schedule and Maintenance Policy]({{site.url}}/releases.html). +The OpenSearch Project releases regular updates that include new features, enhancements, and bug fixes. OpenSearch uses [Semantic Versioning](https://semver.org/), which means that breaking changes are only introduced between major version releases. To learn about upcoming features and fixes, review the [OpenSearch Project Roadmap](https://github.com/orgs/opensearch-project/projects/206) on GitHub. To view a list of previous releases or to learn more about how OpenSearch uses versioning, see [Release schedule and maintenance policy]({{site.url}}/releases.html). Upgrading or migrating OpenSearch is essential for maintaining optimal performance, security, and access to the latest features. Whether you're upgrading an existing OpenSearch deployment or migrating from another system such as Elasticsearch OSS, choosing the right approach is critical to a successful transition. -This page outlines upgrade planning guidance and four supported methods: rolling upgrades, snapshot and restore, remote reindexing, and using the OpenSearch Migration Assistant. +This page outlines upgrade planning guidance and four supported methods: rolling upgrades, snapshot and restore, remote reindexing, and using Migration Assistant. --- -## Before You Begin +## Before you begin Take time to plan the process before making any changes to your cluster: @@ -44,8 +44,7 @@ Stop nonessential indexing before upgrading. --- -## Migration and Upgrade Methods - +## Migration and upgrade methods ### Rolling upgrade @@ -65,7 +64,7 @@ Upgrade one node at a time while keeping the cluster operational. --- -### Snapshot and Restore +### Snapshot and restore Take a snapshot of your current cluster and restore to a new OpenSearch version. @@ -78,13 +77,13 @@ Take a snapshot of your current cluster and restore to a new OpenSearch version. - Requires provisioning a new cluster - Manually reindex may be required for full feature compatibility. -[Get started with Snapshot and Restore](https://docs.aws.amazon.com/solutions/latest/tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore/) +[Get started with snapshot and restore](https://docs.aws.amazon.com/solutions/latest/tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore/) --- ### Migration Assistant -The Migration Assistant provides the most automated and resilient upgrade path. +Migration Assistant provides the most automated and resilient upgrade path. **Pros:** - Handles multi-version hops, allowing for seamless upgrades across multiple versions @@ -99,7 +98,7 @@ The Migration Assistant provides the most automated and resilient upgrade path. --- -### Remote Reindexing +### Remote reindexing Reindex data from the old cluster into the new OpenSearch cluster. @@ -114,6 +113,7 @@ Reindex data from the old cluster into the new OpenSearch cluster. Learn more in the [Reindex API docs]({{site.url}}{{site.baseurl}}/api-reference/document-apis/reindex/) --- + ### Reviewing the OpenSearch tools compatibility matrices If your OpenSearch cluster interacts with other services in your environment, like Logstash or Beats, then you should check the [OpenSearch tools compatibility matrices]({{site.url}}{{site.baseurl}}/tools/index/#compatibility-matrices) to determine whether other components will need to be upgraded. @@ -127,10 +127,9 @@ Go to [Available plugins]({{site.url}}{{site.baseurl}}/install-and-configure/plu Major, minor, and patch plugin versions must match OpenSearch major, minor, and patch versions in order to be compatible. For example, plugin versions 2.3.0.x work only with OpenSearch 2.3.0. {: .important} - --- -## Backing Up Configuration Files +## Backing up configuration files Back up important files such as `opensearch.yml`, plugin configs, and TLS certs from these paths: @@ -141,7 +140,7 @@ See [security configuration guidance]({{site.url}}{{site.baseurl}}/security-plug --- -## Creating a Snapshot +## Creating a snapshot We recommend that you back up your cluster state and indexes using [snapshots]({{site.url}}{{site.baseurl}}/opensearch/snapshots/index/). Snapshots you take before an upgrade can be used as restore points if you need to roll back the cluster to its original version. @@ -156,7 +155,7 @@ You can further reduce the risk of data loss by storing your snapshots on extern --- -## Index Compatibility Reference +## Index compatibility reference - - - - - - +{% include migration-scenarios.html scenarios=site.data.migration-scenarios.scenarios %} diff --git a/_sass/custom/custom.scss b/_sass/custom/custom.scss index 7ed0bb36d6f..f6997cbb211 100755 --- a/_sass/custom/custom.scss +++ b/_sass/custom/custom.scss @@ -1338,6 +1338,28 @@ body { } +// Migration scenario panels +.scenario-container { + border: 1px solid #aaa; + border-radius: 0.5rem; + padding: 1rem; + margin-bottom: 1.5rem; + background-color: #f9f9f9; +} + +.scenario-title { + font-weight: bold; + font-size: 1.2rem; + margin-bottom: 1rem; + color: $grey-dk-300; + border-bottom: 2px solid #e1e4e8; + padding-bottom: 0.5rem; +} + +.scenario-container ol { + margin-top: 0.5rem; +} + @import "../font-awesome.scss"; @import "../_navigation-header.scss"; @import "../footer.scss"; From 0ff3cd07e0a9a7aed0706e1b9013231197d6357d Mon Sep 17 00:00:00 2001 From: Fanit Kolchina Date: Tue, 29 Jul 2025 09:34:30 -0400 Subject: [PATCH 32/62] Fix example and details formatting Signed-off-by: Fanit Kolchina --- .../handling-type-mapping-deprecation.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/_migration-assistant/migration-phases/migrate-metadata/handling-type-mapping-deprecation.md b/_migration-assistant/migration-phases/migrate-metadata/handling-type-mapping-deprecation.md index d142ce39b4e..aa8db4e2cdf 100644 --- a/_migration-assistant/migration-phases/migrate-metadata/handling-type-mapping-deprecation.md +++ b/_migration-assistant/migration-phases/migrate-metadata/handling-type-mapping-deprecation.md @@ -21,7 +21,7 @@ In versions of Elasticsearch prior to 6.x, an index could contain multiple types Newer versions of Elasticsearch and OpenSearch no longer support multiple mapping types. Each index now supports only a single mapping type. During migration, you must define how to transform or restructure data that used multiple types. The following example shows multiple mapping types: -```JSON +```json GET /library/_mappings { "library": { @@ -81,10 +81,10 @@ The type mapping transformer uses the following configuration options. The following example JSON configuration provides a transformation schema: -
              +
              Example JSON Configuration -```JSON +```json { "TypeMappingSanitizationTransformerProvider": { "staticMappings": { @@ -140,8 +140,8 @@ If you have an index `activity` with types `user` and `post` that you want to sp } } ] -{% include copy.html %} ``` +{% include copy.html %} This transformer will perform the following: @@ -205,7 +205,7 @@ This configuration only migrates documents of type `user` and ignores other docu To migrate only specific types and keep the original structure, use the following configuration: -```JSON +```json [ { "TypeMappingSanitizationTransformerProvider": { @@ -282,7 +282,7 @@ The following example demonstrates how to combine static and regex-based mapping When the `regexMappings` key is missing from the transformation configuration, `regexMappings` will default to the following: -```JSON +```json { "regexMappings": [ { @@ -305,7 +305,7 @@ This has the effect of retaining the index name for indexes created in Elasticse ## Limitations When using the transformer, remember the following limitations. -When using the transformer, remember the following limitations. + ### Traffic Replayer For the Traffic Replayer, **only a subset** of requests that include types is supported. These requests are listed in the following table. @@ -320,5 +320,6 @@ For the Traffic Replayer, **only a subset** of requests that include types is su | **Bulk Index/Update/Delete** | PUT/POST | `/{index}/{type}/_bulk` | Perform multiple create/update/delete operations in a single request with default index and type assignment. | | **Create/Update Index** | PUT/POST | `/{index}` | Create or update an index.

              **Split** behavior is not supported in the Traffic Replayer. See [this GitHub issue](https://github.com/opensearch-project/opensearch-migrations/issues/1305) to provide feedback or to vote on this feature. | -### Reindex-From_Shapshot +### Reindex-From-Shapshot + For `Reindex-From-Snapshot,` indexes created in Elasticsearch 6.x or later will use `_doc` as the type for all documents, even if a different type was specified in Elasticsearch 6. From d4930200a57ad357eab25e5ea98cd76960d349e3 Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Tue, 29 Jul 2025 10:39:10 -0500 Subject: [PATCH 33/62] Fix broken links Signed-off-by: Brian Presley --- _data/migration-scenarios.yml | 76 +++++----- _includes/migration-scenarios.html | 4 +- .../accessing-the-migration-console.md | 30 +++- .../migration-phases/create-snapshot.md | 12 +- .../migration-phases/deploy/index.md | 132 ++++++++++++++++++ 5 files changed, 209 insertions(+), 45 deletions(-) diff --git a/_data/migration-scenarios.yml b/_data/migration-scenarios.yml index cea6af32d9c..75262795112 100644 --- a/_data/migration-scenarios.yml +++ b/_data/migration-scenarios.yml @@ -2,90 +2,90 @@ scenarios: - title: "Scenario 1 – Backfill Only" steps: - title: "Assessment" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/assessment/" + url: "/migration-assistant/migration-phases/assessment/" - title: "Deploy" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/deploy/" + url: "/migration-assistant/migration-phases/deploy/" substeps: - title: "Verify Backfill Components" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/deploy/verifying-backfill-components/" + url: "/migration-assistant/migration-phases/deploy/verifying-backfill-components/" - title: "Configuration Options" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/deploy/configuration-options/" + url: "/migration-assistant/migration-phases/deploy/configuration-options/" - title: "IAM and Security Groups for Existing Clusters" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/deploy/iam-and-security-groups-for-existing-clusters/" + url: "/migration-assistant/migration-phases/deploy/iam-and-security-groups-for-existing-clusters/" - title: "Create Snapshot" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/create-snapshot/" + url: "/migration-assistant/migration-phases/create-snapshot/" - title: "Migrate Metadata" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrate-metadata/" + url: "/migration-assistant/migration-phases/migrate-metadata/" substeps: - title: "Managing type mapping deprecation" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrate-metadata/handling-type-mapping-deprecation/" + url: "/migration-assistant/migration-phases/migrate-metadata/handling-type-mapping-deprecation/" - title: "Handling breaking changes in field types" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrate-metadata/handling-field-type-breaking-changes/" + url: "/migration-assistant/migration-phases/migrate-metadata/handling-field-type-breaking-changes/" - title: "Backfill" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/backfill/" + url: "/migration-assistant/migration-phases/backfill/" - title: "Teardown" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/remove-migration-infrastructure/" + url: "/migration-assistant/migration-phases/remove-migration-infrastructure/" - title: "Scenario 2 – Live Capture Only" steps: - title: "Assessment" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/assessment/" + url: "/migration-assistant/migration-phases/assessment/" - title: "Deploy" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/deploy/" + url: "/migration-assistant/migration-phases/deploy/" substeps: - title: "IAM and Security Groups for Existing Clusters" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/deploy/iam-and-security-groups-for-existing-clusters/" + url: "/migration-assistant/migration-phases/deploy/iam-and-security-groups-for-existing-clusters/" - title: "Configuration Options" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/deploy/configuration-options/" + url: "/migration-assistant/migration-phases/deploy/configuration-options/" - title: "Verify Live Capture Components" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/deploy/verifying-live-capture-components/" + url: "/migration-assistant/migration-phases/deploy/verifying-live-capture-components/" - title: "Reroute traffic from source to Capture Proxy" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/reroute-source-to-proxy/" + url: "/migration-assistant/migration-phases/reroute-source-to-proxy/" - title: "Migrate Metadata" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrate-metadata/" + url: "/migration-assistant/migration-phases/migrate-metadata/" substeps: - title: "Managing type mapping deprecation" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrate-metadata/handling-type-mapping-deprecation/" + url: "/migration-assistant/migration-phases/migrate-metadata/handling-type-mapping-deprecation/" - title: "Handling breaking changes in field types" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrate-metadata/handling-field-type-breaking-changes/" + url: "/migration-assistant/migration-phases/migrate-metadata/handling-field-type-breaking-changes/" - title: "Replay Captured Traffic" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/replay-captured-traffic/" + url: "/migration-assistant/migration-phases/replay-captured-traffic/" - title: "Reroute traffic from Capture Proxy to Target" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/reroute-traffic-from-capture-proxy-to-target/" + url: "/migration-assistant/migration-phases/reroute-traffic-from-capture-proxy-to-target/" - title: "Teardown" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/remove-migration-infrastructure/" + url: "/migration-assistant/migration-phases/remove-migration-infrastructure/" - title: "Scenario 3 – Live Capture with Backfill" steps: - title: "Assessment" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/assessment/" + url: "/migration-assistant/migration-phases/assessment/" - title: "Deploy" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/deploy/" + url: "/migration-assistant/migration-phases/deploy/" substeps: - title: "IAM and Security Groups for Existing Clusters" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/deploy/iam-and-security-groups-for-existing-clusters/" + url: "/migration-assistant/migration-phases/deploy/iam-and-security-groups-for-existing-clusters/" - title: "Configuration Options" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/deploy/configuration-options/" + url: "/migration-assistant/migration-phases/deploy/configuration-options/" - title: "Verify Backfill Components" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/deploy/verifying-backfill-components/" + url: "/migration-assistant/migration-phases/deploy/verifying-backfill-components/" - title: "Verify Live Capture Components" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/deploy/verifying-live-capture-components/" + url: "/migration-assistant/migration-phases/deploy/verifying-live-capture-components/" - title: "Reroute traffic from source to Capture Proxy" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/reroute-source-to-proxy/" + url: "/migration-assistant/migration-phases/reroute-source-to-proxy/" - title: "Create Snapshot" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/create-snapshot/" + url: "/migration-assistant/migration-phases/create-snapshot/" - title: "Migrate Metadata" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrate-metadata/" + url: "/migration-assistant/migration-phases/migrate-metadata/" substeps: - title: "Managing type mapping deprecation" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrate-metadata/handling-type-mapping-deprecation/" + url: "/migration-assistant/migration-phases/migrate-metadata/handling-type-mapping-deprecation/" - title: "Handling breaking changes in field types" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrate-metadata/handling-field-type-breaking-changes/" + url: "/migration-assistant/migration-phases/migrate-metadata/handling-field-type-breaking-changes/" - title: "Backfill" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/backfill/" + url: "/migration-assistant/migration-phases/backfill/" - title: "Replay Captured Traffic" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/replay-captured-traffic/" + url: "/migration-assistant/migration-phases/replay-captured-traffic/" - title: "Reroute traffic from Capture Proxy to Target" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/reroute-traffic-from-capture-proxy-to-target/" + url: "/migration-assistant/migration-phases/reroute-traffic-from-capture-proxy-to-target/" - title: "Teardown" - url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/remove-migration-infrastructure/" + url: "/migration-assistant/migration-phases/remove-migration-infrastructure/" diff --git a/_includes/migration-scenarios.html b/_includes/migration-scenarios.html index 31c95c85dd9..cdb02587c86 100644 --- a/_includes/migration-scenarios.html +++ b/_includes/migration-scenarios.html @@ -8,7 +8,7 @@ {% for step in scenario.steps %}
            9. {% if step.url %} - {{ step.title }} + {{ step.title }} {% else %} {{ step.title }} {% endif %} @@ -18,7 +18,7 @@ {% for substep in step.substeps %}
            10. {% if substep.url %} - {{ substep.title }} + {{ substep.title }} {% else %} {{ substep.title }} {% endif %} diff --git a/_migration-assistant/migration-console/accessing-the-migration-console.md b/_migration-assistant/migration-console/accessing-the-migration-console.md index 9cdbbc1c344..9753990712c 100644 --- a/_migration-assistant/migration-console/accessing-the-migration-console.md +++ b/_migration-assistant/migration-console/accessing-the-migration-console.md @@ -16,12 +16,17 @@ The bootstrap box deployed through Migration Assistant contains a script that si To access the migration console, use the following commands: ```shell -export STAGE=dev +export STAGE=dev # Use the same stage value from your cdk.context.json deployment export AWS_REGION=us-west-2 /opensearch-migrations/deployment/cdk/opensearch-service-migration/accessContainer.sh migration-console ${STAGE} ${AWS_REGION} ``` {% include copy.html %} +**Important:** The `STAGE` value must match the `stage` parameter you used in your CDK context configuration. For example: +- If you deployed with `"stage": "test"`, use `export STAGE=test` +- If you deployed with `"stage": "prod"`, use `export STAGE=prod` +- If you deployed with `"stage": "dev"`, use `export STAGE=dev` + When opening the console a message will appear above the command prompt, `Welcome to the Migration Assistant Console`. On a machine with the [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) and the [AWS Session Manager plugin](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html), you can directly connect to the migration console. Ensure that you've run `aws configure` with credentials that have access to the environment. @@ -29,11 +34,30 @@ On a machine with the [AWS Command Line Interface (AWS CLI)](https://docs.aws.am Use the following commands: ```shell -export STAGE=dev +export STAGE=dev # Match your deployment stage export SERVICE_NAME=migration-console export TASK_ARN=$(aws ecs list-tasks --cluster migration-${STAGE}-ecs-cluster --family "migration-${STAGE}-${SERVICE_NAME}" | jq --raw-output '.taskArns[0]') aws ecs execute-command --cluster "migration-${STAGE}-ecs-cluster" --task "${TASK_ARN}" --container "${SERVICE_NAME}" --interactive --command "/bin/bash" ``` {% include copy.html %} -Typically, `STAGE` is equivalent to a standard `dev` environment, but this may vary based on what the user specified during deployment. +### Stage Configuration Examples + +For different deployment environments, adjust the stage accordingly: + +```shell +# For test environment deployment +export STAGE=test +./accessContainer.sh migration-console test us-west-2 + +# For production environment deployment +export STAGE=prod +./accessContainer.sh migration-console prod us-west-2 + +# For development environment deployment +export STAGE=dev +./accessContainer.sh migration-console dev us-west-2 +``` +{% include copy.html %} + +The `STAGE` value corresponds to the `stage` parameter you specified in your CDK context configuration during deployment. diff --git a/_migration-assistant/migration-phases/create-snapshot.md b/_migration-assistant/migration-phases/create-snapshot.md index 1c66b541979..6b4ebcf2500 100644 --- a/_migration-assistant/migration-phases/create-snapshot.md +++ b/_migration-assistant/migration-phases/create-snapshot.md @@ -20,17 +20,25 @@ console snapshot create [...] ``` {% include copy.html %} +**Note:** The Migration Assistant will automatically generate a snapshot name and configure the necessary S3 repository. You only need to specify `--snapshot-name` if you are using an existing snapshot that you created outside of Migration Assistant. + +**For existing snapshots:** +If you have an existing snapshot, you can specify it during the RFS configuration in your `cdk.context.json` file using the `reindexFromSnapshotExtraArgs` parameter: +```json +"reindexFromSnapshotExtraArgs": "--s3-repo-uri s3://your-bucket/repo --s3-region us-west-2 --snapshot-name your-existing-snapshot" +``` + To check the snapshot creation status, run the following command: ```bash -console snapshot status [...] +console snapshot status ``` {% include copy.html %} To learn more information about the snapshot, run the following command: ```bash -console snapshot status --deep-check [...] +console snapshot status --deep-check ``` {% include copy.html %} diff --git a/_migration-assistant/migration-phases/deploy/index.md b/_migration-assistant/migration-phases/deploy/index.md index c2a1e8d1b76..f731d3dc707 100644 --- a/_migration-assistant/migration-phases/deploy/index.md +++ b/_migration-assistant/migration-phases/deploy/index.md @@ -22,6 +22,34 @@ Before using this quickstart, review [Is Migration Assistant right for you?]({{s Because this guide uses [AWS Cloud Development Kit (AWS CDK)](https://aws.amazon.com/cdk/), ensure that the `CDKToolkit` stack exists and is in the `CREATE_COMPLETE` state. For setup instructions, see the [CDK Toolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html). +## Planning Your Deployment Environment + +Before beginning the deployment, consider the following environment planning steps: + +- **Choose a unique stage name**: Avoid using "dev" if you have existing deployments or potential conflicts. Consider using "test", "staging", "prod", or other descriptive names. +- **Verify domain endpoints**: Ensure your source and target cluster endpoints are accessible and properly formatted. +- **Prepare authentication**: Have your cluster credentials and AWS Secrets Manager ARNs ready. +- **Check AWS credentials**: Verify that your AWS credentials are properly configured for the target account and region. + +## Prerequisites + +Before proceeding with the deployment, ensure you have completed the following prerequisites: + +### AWS Environment Setup +1. **Configure AWS credentials**: Run `aws configure` to set up your credentials, or ensure environment variables are properly set. +2. **Verify account access**: Test your credentials with `aws sts get-caller-identity`. +3. **Check region**: Ensure you're deploying to the correct AWS region. + +### Development Environment Setup +1. **Install Docker**: Docker is required for building container images. Verify installation with `docker --version`. +2. **Install Node.js and npm**: Required for CDK operations. Verify with `node --version` and `npm --version`. +3. **Install AWS CDK CLI**: Run `npm install -g aws-cdk` if not already installed. + +### Project Setup +1. **Build Docker images**: From the deployment directory, run `./buildDockerImages.sh` to build required container images. +2. **Install dependencies**: Run `npm install` in the CDK deployment directory. +3. **Bootstrap CDK**: If this is your first CDK deployment in the region, run `cdk bootstrap --c contextId=`. + --- ## Step 1: Install bootstrap on an Amazon EC2 instance (~10 minutes) @@ -152,6 +180,52 @@ Use the following steps to configure and deploy RFS, deploy Migration Assistant, The source and target cluster authorization can be configured to have no authorization, `basic` with a username and password, or `sigv4`. + ### Environment Configuration Examples + + To avoid conflicts with existing deployments, consider using different context IDs and stage names: + + ```json + { + "test-deploy": { + "stage": "test", + "migrationAssistanceEnabled": true, + "migrationConsoleServiceEnabled": true, + "reindexFromSnapshotServiceEnabled": true, + "sourceCluster": { + "endpoint": "https://migration-source-es710.us-west-2.es.amazonaws.com", + "version": "ES_7.10", + "auth": { + "type": "basic", + "username": "admin", + "passwordFromSecretArn": "arn:aws:secretsmanager:us-west-2:123456789012:secret:migration-source-password" + } + }, + "targetCluster": { + "endpoint": "https://migration-target-os219.us-west-2.es.amazonaws.com", + "auth": { + "type": "basic", + "username": "admin", + "passwordFromSecretArn": "arn:aws:secretsmanager:us-west-2:123456789012:secret:migration-target-password" + } + } + }, + "prod-deploy": { + "stage": "prod", + "migrationAssistanceEnabled": true, + "migrationConsoleServiceEnabled": true, + "reindexFromSnapshotServiceEnabled": true, + "// ... additional production-specific configuration" + } + } + ``` + {% include copy.html %} + + **Important Notes:** + - Use unique `stage` values to prevent resource naming conflicts + - Ensure secret ARNs are complete and accessible in your deployment region + - Domain endpoints can be simplified names or full AWS URLs + - Deploy using: `./deploy.sh ` (e.g., `./deploy.sh test-deploy`) + 3. After the `cdk.context.json` file is fully configured, bootstrap the account and deploy the required stacks using the following command: ```bash @@ -237,6 +311,64 @@ To learn more about migration console commands, see [Migration console command r --- +## Troubleshooting + +### Common Deployment Issues + +**Problem: AWS credentials not configured** +``` +Unable to locate credentials. You can configure credentials by running "aws configure". +``` +**Resolution:** +1. Run `aws configure` and provide your access key, secret key, and region +2. Alternatively, set environment variables: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_DEFAULT_REGION` +3. Verify credentials with: `aws sts get-caller-identity` + +**Problem: Stack naming conflicts** +``` +Stack with id OSMigrations-dev-us-west-2-MigrationConsole already exists +``` +**Resolution:** +1. Use a different `stage` value in your context configuration (e.g., "test", "staging") +2. Or destroy existing stacks: `cdk destroy "*" --c contextId=` +3. Ensure unique context IDs for parallel deployments + +**Problem: Docker build failures** +``` +ERROR: failed to solve: public.ecr.aws/sam/build-nodejs18.x: pulling from host public.ecr.aws failed +``` +**Resolution:** +1. Run `docker logout public.ecr.aws` to clear authentication cache +2. Retry the build process: `./buildDockerImages.sh` + +**Problem: CDK bootstrap required** +``` +This stack uses assets, so the toolkit stack must be deployed to the environment +``` +**Resolution:** +1. Bootstrap CDK in your region: `cdk bootstrap --c contextId=` +2. Ensure you have the correct AWS credentials and region configured + +### Rollback Procedures + +If you need to remove a deployment: + +1. **Stop all running services:** + ```bash + console backfill stop # If backfill is running + ``` + +2. **Destroy CDK stacks:** + ```bash + cdk destroy "*" --c contextId= --force + ``` + +3. **Clean up manually if needed:** + - Remove any remaining CloudFormation stacks from the AWS console + - Delete any orphaned resources (ECS tasks, load balancers, etc.) + +--- + ## Next steps After completing the deployment, proceed with the migration phases: From f13c97bfff6454b85e2c7447a249edfd7d63310c Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Tue, 29 Jul 2025 14:44:57 -0500 Subject: [PATCH 34/62] Removed code font from header Signed-off-by: Brian Presley --- _migration-assistant/migration-phases/migrate-metadata/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_migration-assistant/migration-phases/migrate-metadata/index.md b/_migration-assistant/migration-phases/migrate-metadata/index.md index cfdc78f441f..3150ffd5288 100644 --- a/_migration-assistant/migration-phases/migrate-metadata/index.md +++ b/_migration-assistant/migration-phases/migrate-metadata/index.md @@ -96,7 +96,7 @@ Results: ``` -## Using the `migrate` command +## Using the migrate command Running through the same data as the evaluate command all of the migrated items will be applied onto the target cluster. If re-run multiple times items that were previously migrated will not be recreated. If any items do need to be re-migrated, please delete them from the target cluster and then rerun the evaluate then migrate commands to ensure the desired changes are made. From 1d8730a2f54be61f94356a4da2fe20d40809386e Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Wed, 30 Jul 2025 10:23:32 -0500 Subject: [PATCH 35/62] Removed deprecated page and moved the rolling upgrade lab. Signed-off-by: Brian Presley --- .../rolling-upgrade/appendix/index.md | 21 ------------------- _migrate-or-upgrade/rolling-upgrade/index.md | 5 +++-- .../{appendix => }/rolling-upgrade-lab.md | 8 ++++--- 3 files changed, 8 insertions(+), 26 deletions(-) delete mode 100644 _migrate-or-upgrade/rolling-upgrade/appendix/index.md rename _migrate-or-upgrade/rolling-upgrade/{appendix => }/rolling-upgrade-lab.md (99%) diff --git a/_migrate-or-upgrade/rolling-upgrade/appendix/index.md b/_migrate-or-upgrade/rolling-upgrade/appendix/index.md deleted file mode 100644 index e711fc7acf7..00000000000 --- a/_migrate-or-upgrade/rolling-upgrade/appendix/index.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -layout: default -title: Upgrades appendix -grand_parent: Migrate or upgrade -parent: Rolling upgrade -has_children: true -nav_order: 30 -permalink: /migrate-or-upgrade/rolling-upgrade/appendix/ ---- - -# Upgrades appendix - -Use the upgrades appendix to find additional supporting documentation, such as labs that include example API requests and configuration files to supplement the related process documentation. Specific procedures outlined in the appendix section can be used in a variety of ways: - -- New OpenSearch users can use the steps and example resources we provide to learn about configuring and using OpenSearch and OpenSearch Dashboards. -- System administrators who work with OpenSearch clusters can use the examples we provide to simulate cluster maintenance in a test environment before applying any changes to a production workload. - -If you would like to request a specific topic, please comment on issue [#2830](https://github.com/opensearch-project/documentation-website/issues/2830) in the [OpenSearch Project](https://github.com/opensearch-project) on GitHub. - -The specific commands included in this appendix serve as examples of interacting with the OpenSearch API, and the underlying host, in order to demonstrate the steps described in the related upgrade process documents. The intention is not to be overly prescriptive but instead to add context for users who are new to OpenSearch and want to see practical examples. -{:.note} diff --git a/_migrate-or-upgrade/rolling-upgrade/index.md b/_migrate-or-upgrade/rolling-upgrade/index.md index 543a2903bc3..129826f6943 100644 --- a/_migrate-or-upgrade/rolling-upgrade/index.md +++ b/_migrate-or-upgrade/rolling-upgrade/index.md @@ -4,8 +4,8 @@ title: Rolling upgrade nav_order: 20 has_toc: true permalink: /migrate-or-upgrade/rolling-upgrade/ +nav_exclude: false redirect_from: - - /upgrade-opensearch/appendix/ - /rolling-upgrade/index/ --- @@ -13,7 +13,7 @@ redirect_from: Rolling upgrades, sometimes referred to as "node replacement upgrades," can be performed on running clusters with virtually no downtime. Nodes are individually stopped and upgraded in place. Alternatively, nodes can be stopped and replaced, one at a time, by hosts running the new version. During this process you can continue to index and query data in your cluster. -This document serves as a high-level, platform-agnostic overview of the rolling upgrade procedure. For specific examples of commands, scripts, and configuration files, refer to the [Appendix]({{site.url}}{{site.baseurl}}/migrate-or-upgrade/rolling-upgrade/appendix/). +This document serves as a high-level, platform-agnostic overview of the rolling upgrade procedure. For specific examples of commands, scripts, and configuration files, refer to the [Rolling upgrade lab]({{site.url}}{{site.baseurl}}/migrate-or-upgrade/rolling-upgrade/rolling-upgrade-lab/). ## Preparing to upgrade @@ -257,6 +257,7 @@ By preserving quorum and restarting nodes sequentially, rolling restarts ensure ## Related articles +- [Rolling upgrade lab]({{site.url}}{{site.baseurl}}/migrate-or-upgrade/rolling-upgrade/rolling-upgrade-lab/) - A hands-on lab with step-by-step instructions for practicing rolling upgrades in a test environment - [OpenSearch configuration]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/) - [Performance analyzer]({{site.url}}{{site.baseurl}}/monitoring-plugins/pa/index/) - [Install and configure OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/install-and-configure/install-dashboards/index/) diff --git a/_migrate-or-upgrade/rolling-upgrade/appendix/rolling-upgrade-lab.md b/_migrate-or-upgrade/rolling-upgrade/rolling-upgrade-lab.md similarity index 99% rename from _migrate-or-upgrade/rolling-upgrade/appendix/rolling-upgrade-lab.md rename to _migrate-or-upgrade/rolling-upgrade/rolling-upgrade-lab.md index f39bfc76468..1fbd24dd74e 100644 --- a/_migrate-or-upgrade/rolling-upgrade/appendix/rolling-upgrade-lab.md +++ b/_migrate-or-upgrade/rolling-upgrade/rolling-upgrade-lab.md @@ -1,11 +1,13 @@ --- layout: default title: Rolling upgrade lab -parent: Upgrades appendix -grand_parent: Rolling upgrade -nav_order: 50 +parent: Rolling upgrade +nav_exclude: true redirect_from: - /upgrade-opensearch/appendix/rolling-upgrade-lab/ + - /install-and-configure/upgrade-opensearch/appendix/index/ + - /migrate-or-upgrade/rolling-upgrade/appendix/ + - /migrate-or-upgrade/rolling-upgrade/appendix/rolling-upgrade-lab/ ---