Skip to content

Conversation

difin
Copy link
Contributor

@difin difin commented Oct 15, 2025

What changes were proposed in this pull request?

Added try-with-resources for a catalog that is loaded in Catalogs.loadTable().

Why are the changes needed?

Described in the description of HIVE-29268.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Manual check on the local Hive Docker instance.

@difin difin changed the title HIVE-29268: Iceberg: Close catalog in Catalogs.loadTable() to fix res… HIVE-29268: Iceberg: Close catalog in Catalogs.loadTable() to fix resource leak from open ResolvingFileIO Oct 15, 2025
@difin difin requested review from deniskuzZ and okumin October 15, 2025 00:59
} else {
// fallback if not AutoCloseable
return catalog.get().loadTable(TableIdentifier.parse(tableIdentifier));
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file is a copy of apache/iceberg/mr/.../Catalogs.java, and the upstream version does not have the equivalent of this change. Why does Apache Iceberg justify the current implementation? Or it is still a problem on upstream, but no MapReduce users are interested in the warning message.

The current implementation is likely to work. For future reference, I'm also curious about what the to-be solution is.

Copy link
Member

@deniskuzZ deniskuzZ Oct 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we avoid logical fallback and close resource when required?

private static Table loadTable(Configuration conf, String tableIdentifier, String tableLocation,
                               String catalogName) {
    Optional<Catalog> catalog = loadCatalog(conf, catalogName);

    if (catalog.isPresent()) {
        Preconditions.checkArgument(tableIdentifier != null, "Table identifier not set");
        try {
            return catalog.get().loadTable(TableIdentifier.parse(tableIdentifier));
        } finally {
            if (catalog.get() instanceof Closeable closeable) {
              IOUtils.closeQuietly(closeable);
            }
        }
    }
    ....
}

Preconditions.checkArgument(tableIdentifier != null, "Table identifier not set");
return catalog.get().loadTable(TableIdentifier.parse(tableIdentifier));

if (catalog.get() instanceof AutoCloseable) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@difin difin force-pushed the fix_resource_leak branch from fceea4b to d6b7b35 Compare October 15, 2025 17:23
@difin
Copy link
Contributor Author

difin commented Oct 15, 2025

I found that this proposed change resolves the warning for the unclosed ResolvingFileIO instance in the logs, but it break the functionality, because it closes HiveRESTCatalogClient connection to the RESTCatalog server and causes below exceptions when Iceberg tries to commit to an iceberg table.

2025-10-15 13:29:33 2025-10-15T17:29:33,844 ERROR [HiveServer2-Background-Pool: Thread-159] hive.HiveIcebergStorageHandler: Error while trying to commit job: job_17605493726650_0001, starting rollback changes for table: ice_rest.ice_orc2
2025-10-15 13:29:33 java.lang.IllegalStateException: Connection pool shut down
2025-10-15 13:29:33     at org.apache.hc.core5.util.Asserts.check(Asserts.java:38) ~[httpcore5-5.3.1.jar:5.3.1]
2025-10-15 13:29:33     at org.apache.hc.core5.pool.StrictConnPool.lease(StrictConnPool.java:175) ~[httpcore5-5.3.1.jar:5.3.1]
...

Copy link

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants