Fix failures for EXPLAIN and EXPLAIN ANALYZE on Iceberg OPTIMIZE queries #26670

yangshangqing95 · 2025-09-19T02:54:46Z

Description

This PR fixes failures when running EXPLAIN and EXPLAIN ANALYZE on Iceberg OPTIMIZE statements:

Fixed server-side failure in EXPLAIN; EXPLAIN prints an error on the server side, which is only ignored because of the IGNORE_STATS_CALCULATOR_FAILURES session property. When this property is set to false, it can still cause the query to fail.

Exception example:

java.lang.IllegalArgumentException: Unexpected scanned files recording set
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
	at io.trino.plugin.iceberg.IcebergMetadata.getTableStatistics(IcebergMetadata.java:3546)
	at io.trino.plugin.base.classloader.ClassLoaderSafeConnectorMetadata.getTableStatistics(ClassLoaderSafeConnectorMetadata.java:356)
	at io.trino.tracing.TracingConnectorMetadata.getTableStatistics(TracingConnectorMetadata.java:333)
	at io.trino.metadata.MetadataManager.getTableStatistics(MetadataManager.java:492)
	at io.trino.tracing.TracingMetadata.getTableStatistics(TracingMetadata.java:318)
	at io.trino.cost.CachingTableStatsProvider.getTableStatisticsInternal(CachingTableStatsProvider.java:53)
	at java.base/java.util.Map.computeIfAbsent(Map.java:1067)
   .......

Fixed failure in EXPLAIN ANALYZE OPTIMIZE.
- Related issue: #26598

This change ensures both EXPLAIN and EXPLAIN ANALYZE OPTIMIZE queries complete without errors.

Additional context and related issues

This would also close #26598

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text:

## Iceberg
* Fix failures when running EXPLAIN and EXPLAIN ANALYZE on OPTIMIZE statements. (#26598)

cla-bot · 2025-09-19T02:54:49Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

raunaqmorarka · 2025-10-06T14:31:18Z

@cla-bot check

cla-bot · 2025-10-06T14:31:21Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2025-10-06T14:31:23Z

The cla-bot has been summoned, and re-checked this pull request!

raunaqmorarka · 2025-10-06T14:44:40Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

        // Certain table handle attributes are not applicable to select queries (which need stats).
        // If this changes, the caching logic may here may need to be revised.
        checkArgument(!originalHandle.isRecordScannedFiles(), "Unexpected scanned files recording set");
        checkArgument(originalHandle.getMaxScannedFileSize().isEmpty(), "Unexpected max scanned file size set");


I think the fix can be to either return empty statistics here instead of the checkArgument,

if (originalHandle.isRecordScannedFiles()) { // Skip return TableStatistics.empty(); }

or just remove the checks altogether and include originalHandle.isRecordScannedFiles() in the cacheKey below.

The test for the fix would be to successfully run EXPLAIN and EXPLAIN ANALYZE on OPTIMIZE query with ignore_stats_calculator_failures session property set to false.

Thanks for the suggestion! I agree this is the simplest fix — I actually considered that approach earlier as well.

However, I decided to introduce the disableTableStatisticsCache option because it provides a more explicit and controllable behavior. It allows users (and tests) to completely turn off caching when diagnosing issues related to stale or inconsistent statistics.

This approach keeps the cache key logic clean and avoids coupling it with isRecordScannedFiles(), while also giving us better flexibility to handle similar scenarios in the future — especially if we later support outputting statistics for operations like OPTIMIZE or other similar behaviors in Iceberg.

That said, I’m open to adjusting the naming or scope of the option if you think that would make the intent clearer.

We don't want to introduce the amount of complexity added here for solving this case.
Iceberg doesn't have a problem of stale statistics because we don't cache statistics across queries and the metadata file cached for planning are immutable, so can never provide stale information.
If we want to output statistics for OPTIMIZE, then my second suggestion of "remove the checks altogether and include originalHandle.isRecordScannedFiles() in the cacheKey below" is the way to do it.

Hi @raunaqmorarka If we are just trying to solve the current issue, I agree with your suggestion. The changes have already been pushed.

cla-bot · 2025-10-06T22:53:06Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

1. Fixed server-side failure in EXPLAIN; when IGNORE_STATS_CALCULATOR_FAILURES is false, the query failed. 2. Fixed failure in EXPLAIN ANALYZE OPTIMIZE.

cla-bot · 2025-10-06T22:54:34Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

yangshangqing95 · 2025-10-07T17:21:13Z

@cla-bot check

cla-bot · 2025-10-07T17:21:16Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2025-10-07T17:21:17Z

The cla-bot has been summoned, and re-checked this pull request!

github-actions bot added the iceberg Iceberg connector label Sep 19, 2025

raunaqmorarka reviewed Oct 6, 2025

View reviewed changes

yangshangqing95 force-pushed the fix-explain-optimize branch from f5a56ac to 29c5d35 Compare October 6, 2025 22:53

Fix failures for EXPLAIN and EXPLAIN ANALYZE on Iceberg OPTIMIZE queries

ab31deb

1. Fixed server-side failure in EXPLAIN; when IGNORE_STATS_CALCULATOR_FAILURES is false, the query failed. 2. Fixed failure in EXPLAIN ANALYZE OPTIMIZE.

yangshangqing95 force-pushed the fix-explain-optimize branch from 29c5d35 to ab31deb Compare October 6, 2025 22:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix failures for EXPLAIN and EXPLAIN ANALYZE on Iceberg OPTIMIZE queries #26670

Fix failures for EXPLAIN and EXPLAIN ANALYZE on Iceberg OPTIMIZE queries #26670

yangshangqing95 commented Sep 19, 2025

Uh oh!

cla-bot bot commented Sep 19, 2025

Uh oh!

raunaqmorarka commented Oct 6, 2025

Uh oh!

cla-bot bot commented Oct 6, 2025

Uh oh!

cla-bot bot commented Oct 6, 2025

Uh oh!

raunaqmorarka Oct 6, 2025

Uh oh!

yangshangqing95 Oct 6, 2025 •

edited

Loading

Uh oh!

raunaqmorarka Oct 6, 2025

Uh oh!

yangshangqing95 Oct 6, 2025

Uh oh!

cla-bot bot commented Oct 6, 2025

Uh oh!

cla-bot bot commented Oct 6, 2025

Uh oh!

yangshangqing95 commented Oct 7, 2025

Uh oh!

cla-bot bot commented Oct 7, 2025

Uh oh!

cla-bot bot commented Oct 7, 2025

Uh oh!

Uh oh!

Fix failures for EXPLAIN and EXPLAIN ANALYZE on Iceberg OPTIMIZE queries #26670

Are you sure you want to change the base?

Fix failures for EXPLAIN and EXPLAIN ANALYZE on Iceberg OPTIMIZE queries #26670

Conversation

yangshangqing95 commented Sep 19, 2025

Description

Additional context and related issues

Release notes

Uh oh!

cla-bot bot commented Sep 19, 2025

Uh oh!

raunaqmorarka commented Oct 6, 2025

Uh oh!

cla-bot bot commented Oct 6, 2025

Uh oh!

cla-bot bot commented Oct 6, 2025

Uh oh!

raunaqmorarka Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

yangshangqing95 Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

raunaqmorarka Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

yangshangqing95 Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

cla-bot bot commented Oct 6, 2025

Uh oh!

cla-bot bot commented Oct 6, 2025

Uh oh!

yangshangqing95 commented Oct 7, 2025

Uh oh!

cla-bot bot commented Oct 7, 2025

Uh oh!

cla-bot bot commented Oct 7, 2025

Uh oh!

Uh oh!

yangshangqing95 Oct 6, 2025 •

edited

Loading