Skip to content

Conversation

@ykmr1224
Copy link
Collaborator

@ykmr1224 ykmr1224 commented Oct 21, 2025

Description

  • Add permissive mode setting, and enable only in standalone integ-test
    • When permissive mode is enabled, the fields not found in the index mapping will be collected and stored in _MAP field typed as MAP<String, Object> (dynamic fields).
  • This PR adds only most primitive support to handle dynamic fields.
    • I will raise separate PRs to provide wider support.

Related Issues

Permissive mode RFC: #4349
Dynamic fields RFC: #4433

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • New PPL command checklist all confirmed.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff or -s.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@ykmr1224 ykmr1224 added enhancement New feature or request PPL Piped processing language calcite calcite migration releated labels Oct 21, 2025
@ykmr1224 ykmr1224 requested a review from RyanL1997 as a code owner October 21, 2025 22:03
@ykmr1224 ykmr1224 self-assigned this Oct 21, 2025
final RelDataType relKey = TYPE_FACTORY.createSqlType(SqlTypeName.VARCHAR);
return TYPE_FACTORY.createMapType(
relKey, TYPE_FACTORY.createSqlType(SqlTypeName.BINARY), nullable);
relKey, TYPE_FACTORY.createSqlType(SqlTypeName.ANY), nullable);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does #3459 solved?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#3459 is not solved yet. I think #3459 is not directly related to this type mapping.

This change is needed to set _MAP as Map<String, Any>.
It seems that currently object is converted to array of map when read like: [{"attr1": 1, "attr2": "v"}], "otherValue".

schema("lastname", "string"),
schema("department", "string"),
schema("salary", "int"));
verifyDataRows(result, rows("John", "Doe", "Engineering", 75000));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is explain results?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add explain validation in the tests.

Table getTable(DataSourceSchemaName dataSourceSchemaName, String tableName);

/** Get {@link Table} which allows permissive mode or not. */
default Table getPermissiveAwareTable(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getPermissiveAwareTable is temporay change? if yes, add TODO.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be kept to avoid permissive is applied to SQL.
(Only Calcite PPL implementation will use this method to allow permissive mode)

Comment on lines +122 to +124

try {
for (Map.Entry<String, ExprValue> entry : value.tupleValue().entrySet()) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only the top level <k, v> are extracted? what if field name is field.f1.f2?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, do we want to flatten nested data?

Signed-off-by: Tomoyuki Morita <[email protected]>
Signed-off-by: Tomoyuki Morita <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

calcite calcite migration releated maintenance Improves code quality, but not the product PPL Piped processing language

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants