Skip to content

Add compaction options to to r/aws_glue_catalog_table_optimizer #44044

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Add compaction options to to r/aws_glue_catalog_table_optimizer #44044

wants to merge 3 commits into from
+307 −4

Conversation

sonya
Copy link
Contributor

@sonya sonya commented Aug 26, 2025

Rollback Plan

If a change needs to be reverted, we will publish an updated version of the library.

Changes to Security Controls

No changes to security controls

Description

Adds compaction_configuration as a configurable block for Glue Table optimizers.

This change also makes it mandatory for optimizers with type = compaction, which may break some existing configurations. This also adds validation rules so that optimizer configuration blocks must match the type of optimizer.

The reason for making compaction_configuration mandatory is that the Glue API now always returns a CompactionConfiguration block with strategy set to binpack if no options are passed upon creation. If compaction_configuration is optional and missing, this results in an entire block that is absent from Terraform configuration but gets returned in state. This cannot be overcome with a PlanModifier because the diff affects the block count (0 vs. 1)

The most reliable way to prevent the Terraform config from conflicting with the API response seems to be mandating that the fields always present in the response are always present in the config.

Relations

Closes #43867

References

Output from Acceptance Testing

make testacc TESTS=TestAccGlue_serial PKG=glue
make: Verifying source code with gofmt...
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go1.24.6 test ./internal/service/glue/... -v -count 1 -parallel 20 -run='TestAccGlue_serial'  -timeout 360m -vet=off
2025/08/26 13:31:26 Creating Terraform AWS Provider (SDKv2-style)...
2025/08/26 13:31:26 Initializing Terraform AWS Provider (SDKv2-style)...
=== RUN   TestAccGlue_serial
=== PAUSE TestAccGlue_serial
=== CONT  TestAccGlue_serial
=== RUN   TestAccGlue_serial/CatalogTableOptimizer
=== RUN   TestAccGlue_serial/CatalogTableOptimizer/deleteOrphanFileConfiguration
=== RUN   TestAccGlue_serial/CatalogTableOptimizer/disappears
=== RUN   TestAccGlue_serial/CatalogTableOptimizer/retentionConfiguration
=== RUN   TestAccGlue_serial/CatalogTableOptimizer/update
=== PAUSE TestAccGlue_serial/CatalogTableOptimizer/update
=== RUN   TestAccGlue_serial/CatalogTableOptimizer/basic
=== RUN   TestAccGlue_serial/CatalogTableOptimizer/compactionConfiguration
=== CONT  TestAccGlue_serial/CatalogTableOptimizer/update
=== RUN   TestAccGlue_serial/DataCatalogEncryptionSettings
=== RUN   TestAccGlue_serial/DataCatalogEncryptionSettings/basic
    data_catalog_encryption_settings_test.go:23: Skipping aws_glue_data_catalog_encryption_settings tests due to potential KMS key corruption
=== RUN   TestAccGlue_serial/DataCatalogEncryptionSettings/dataSource
    data_catalog_encryption_settings_data_source_test.go:15: Skipping aws_glue_data_catalog_encryption_settings tests due to potential KMS key corruption
=== RUN   TestAccGlue_serial/ResourcePolicy
=== RUN   TestAccGlue_serial/ResourcePolicy/hybrid
=== RUN   TestAccGlue_serial/ResourcePolicy/disappears
=== RUN   TestAccGlue_serial/ResourcePolicy/equivalent
=== RUN   TestAccGlue_serial/ResourcePolicy/Identity
=== RUN   TestAccGlue_serial/ResourcePolicy/Identity/RegionOverride
    resource_policy_identity_gen_test.go:110: Terraform CLI version 1.9.8 is below minimum version 1.12.0: skipping test
=== RUN   TestAccGlue_serial/ResourcePolicy/Identity/basic
    resource_policy_identity_gen_test.go:39: Terraform CLI version 1.9.8 is below minimum version 1.12.0: skipping test
=== RUN   TestAccGlue_serial/ResourcePolicy/Identity/ExistingResource
    resource_policy_identity_gen_test.go:220: Terraform CLI version 1.9.8 is below minimum version 1.12.0: skipping test
=== RUN   TestAccGlue_serial/ResourcePolicy/basic
=== RUN   TestAccGlue_serial/ResourcePolicy/update
--- PASS: TestAccGlue_serial (322.67s)
    --- PASS: TestAccGlue_serial/CatalogTableOptimizer (192.15s)
        --- PASS: TestAccGlue_serial/CatalogTableOptimizer/deleteOrphanFileConfiguration (45.08s)
        --- PASS: TestAccGlue_serial/CatalogTableOptimizer/disappears (29.67s)
        --- PASS: TestAccGlue_serial/CatalogTableOptimizer/retentionConfiguration (44.37s)
        --- PASS: TestAccGlue_serial/CatalogTableOptimizer/basic (30.35s)
        --- PASS: TestAccGlue_serial/CatalogTableOptimizer/compactionConfiguration (42.67s)
        --- PASS: TestAccGlue_serial/CatalogTableOptimizer/update (39.60s)
    --- PASS: TestAccGlue_serial/DataCatalogEncryptionSettings (0.00s)
        --- SKIP: TestAccGlue_serial/DataCatalogEncryptionSettings/basic (0.00s)
        --- SKIP: TestAccGlue_serial/DataCatalogEncryptionSettings/dataSource (0.00s)
    --- PASS: TestAccGlue_serial/ResourcePolicy (90.92s)
        --- PASS: TestAccGlue_serial/ResourcePolicy/hybrid (28.69s)
        --- PASS: TestAccGlue_serial/ResourcePolicy/disappears (11.18s)
        --- PASS: TestAccGlue_serial/ResourcePolicy/equivalent (16.91s)
        --- PASS: TestAccGlue_serial/ResourcePolicy/Identity (0.10s)
            --- SKIP: TestAccGlue_serial/ResourcePolicy/Identity/RegionOverride (0.05s)
            --- SKIP: TestAccGlue_serial/ResourcePolicy/Identity/basic (0.03s)
            --- SKIP: TestAccGlue_serial/ResourcePolicy/Identity/ExistingResource (0.02s)
        --- PASS: TestAccGlue_serial/ResourcePolicy/basic (12.95s)
        --- PASS: TestAccGlue_serial/ResourcePolicy/update (21.10s)
PASS
ok      github.com/hashicorp/terraform-provider-aws/internal/service/glue       327.172s

Note: The full test suite with 6 optimizer tests seems to have one test fail 2 out of 3 times with a 403 response from the Glue API. I don't know if this is a major concern or a known issue with the Glue API.

@sonya
Add compaction options to to r/aws_glue_catalog_table_optimizer
57eed77 
Adds `compaction_configuration` as a configurable block for Glue Table
optimizers.

This change also makes it mandatory for optimizers with `type =
compaction`, which may break some existing configurations. This also
adds validation rules so that optimizer configuration blocks must match
the type of optimizer.

The reason for making `compaction_configuration` mandatory is that the
Glue API now always returns a `CompactionConfiguration` block with
strategy set to `binpack` if no options are passed upon creation.
If `compaction_configuration` is optional and missing, this results in
an entire block that is absent from Terraform configuration but gets
returned in state. This cannot be overcome with a PlanModifier because
the diff affects the block count (0 vs. 1)

The most reliable way to prevent the Terraform config from conflicting
with the API response seems to be mandating that the fields always
present in the response are always present in the config.
@github-actions GitHub Actions
Copy link
Contributor

Community Guidelines

This comment is added to every new Pull Request to provide quick reference to how the Terraform AWS Provider is maintained. Please review the information below, and thank you for contributing to the community that keeps the provider thriving! 🚀

Voting for Prioritization

  • Please vote on this Pull Request by adding a 👍 reaction to the original post to help the community and maintainers prioritize it.
  • Please see our prioritization guide for additional information on how the maintainers handle prioritization.
  • Please do not leave +1 or other comments that do not add relevant new information or questions; they generate extra noise for others following the Pull Request and do not help prioritize the request.

Pull Request Authors

  • Review the contribution guide relating to the type of change you are making to ensure all of the necessary steps have been taken.
  • Whether or not the branch has been rebased will not impact prioritization, but doing so is always a welcome surprise.

@github-actions GitHub Actions
Copy link
Contributor

⚠️ Our automation has detected the following potential issues with your pull request

❌ Maintainer Edit Permissions (Click to expand)

At times, our maintainers need to make direct edits to pull requests in order to prepare it to be merged. At the time of opening this pull request, your settings do not allow maintainers to make such edits. If possible, update your settings as described in the following document. If your fork is owned by an organization that limits your ability to make this change, please let us know.

Please see the GitHub documentation on allowing changes to a pull request branch created from a fork for additional details.

@github-actions github-actions bot added needs-triage Waiting for first response or review from a maintainer. documentation Introduces or discusses updates to documentation. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. service/glue Issues and PRs that pertain to the glue service. size/XL Managed by automation to categorize the size of a PR. labels Aug 26, 2025
@sonya sonya marked this pull request as ready for review August 27, 2025 04:02
@sonya sonya requested a review from a team as a code owner August 27, 2025 04:02
@sonya
Copy link
Contributor Author

sonya commented Aug 27, 2025

I opened this PR from an organization account. Please let me know if I can make requested changes here or if I should move this PR to allow maintainer edits as described above

@justinretzolk justinretzolk added enhancement Requests to existing resources that expand the functionality or scope. and removed needs-triage Waiting for first response or review from a maintainer. labels Aug 27, 2025
@sideshow
Copy link

+1 for this, as we are also needing this functionality
Thanks for submitting

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
documentation Introduces or discusses updates to documentation. enhancement Requests to existing resources that expand the functionality or scope. service/glue Issues and PRs that pertain to the glue service. size/XL Managed by automation to categorize the size of a PR. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add the ability to set compaction configuration parameters when creating a glue table optimizer
3 participants
@sonya @sideshow @justinretzolk