[Feature] Paimon Spark Extensions conflict with Iceberg

[wForget]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

The Call syntax is defined in both Iceberg and Paimon, which may cause conflicts when I introduce their SparkSessionExtensions at the same time.

Reproduce:

spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions

-- create iceberg table
CREATE TABLE iceberg_catalog.sample.iceberg_t1 (
    user_id BIGINT,
    item_id BIGINT,
    behavior STRING,
    dt STRING,
    hh STRING
) using iceberg;

-- create paimon table
CREATE TABLE paimon_catalog.sample.paimon_t1 (
    user_id BIGINT,
    item_id BIGINT,
    behavior STRING,
    dt STRING,
    hh STRING
) TBLPROPERTIES (
    'primary-key' = 'dt,hh,user_id'
);

-- Successed
CALL iceberg_catalog.system.remove_orphan_files(table => "sample.iceberg_t1");

-- Failed, use iceberg ResolveProcedures
CALL paimon_catalog.sys.remove_orphan_files(table => "sample.paimon_t1");

duplicate: apache/iceberg#10143

Solution

If currentCatalog is not Spark Session catalog and Paimon Spark Catalog, we first use delegate parser to parse sqlText in PaimonSparkSqlExtensionsParser#parsePlan.

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!