Support `CREATE EXTERNAL TABLE` backed by a Catalog with DataFusion

[CTTY]

Background

DataFusion already supports CREATE EXTERNAL TABLE ... STORED AS ICEBERG. Today, iceberg-rust integrates via IcebergTableProviderFactory, but the factory primarily supports registering a static table (e.g., created from a metadata JSON path). That works for:

-- Static table (existing, backward compatible)
CREATE EXTERNAL TABLE my_table
STORED AS ICEBERG
LOCATION '/path/to/metadata.json';

However, we also want CREATE EXTERNAL TABLE to create a normal IcebergTableProvider backed by a Catalog, so users can define the catalog via SQL OPTIONS (and then resolve tables by identifier through that catalog).

Dumping my thoughts here and feedbacks are welcome!

Option A: Build Catalog inside the ProviderFactory using OPTIONS

IcebergTableProviderFactory parses OPTIONS and uses a CatalogBuilder to construct the Catalog internally, then creates a normal IcebergTableProvider

CREATE EXTERNAL TABLE my_table
STORED AS ICEBERG
LOCATION 'ignored_or_optional' // this will be ignored if a catalog is configured
OPTIONS (
  'datafusion.iceberg.catalog.type' = 'rest', // if catalog type is not configured, it should fall back to create static table
  'datafusion.iceberg.catalog.uri' = 'http://localhost:8181',
  'datafusion.iceberg.catalog.warehouse' = 's3://bucket/warehouse'
);

Option B: Allow injecting a pre-built Catalog into the factory

Essentially we have

pub struct IcebergTableProviderFactory {
  catalog: Option<Arc<dyn Catalog>>, // when it's none, fall back to static table
}
...
IcebergTableProviderFactory::new_with_catalog(Arc<dyn Catalog>)

I prefer this as it is much more straight-forward, but one drawback I can think of is users cannot easily use multiple catalogs at the same time. A workaround would look like this:

state
        .table_factories_mut()
        .insert("ICEBERG_REST_A".to_string(), Arc::new(IcebergTableProviderFactory(rest_catalog_a)));

state
        .table_factories_mut()
        .insert("ICEBERG_REST_B".to_string(), Arc::new(IcebergTableProviderFactory(rest_catalog_b)));

and then when creating the table using sql:

CREATE EXTERNAL TABLE my_table
STORED AS ICEBERG_REST_A
...

Willingness to contribute

I can contribute to this feature independently

[CTTY]

Posted a draft based on option B(#2022). I think this can even be expanded to include CatalogBuilder in the future if needed

pub struct IcebergTableProviderFactory {
  catalog: Option<Arc<dyn Catalog>>,
  catalog_builder:: Option<Arc<dyn CatalogBuilder>>,
}

But for now, I think it's a bit over-engineering

[liurenjie1024]

Hi, @CTTY Thanks for raising this discussion, it took me some time to understand how the register external table work in datafusion. I think we would ask user to use it as sth mixing option A and option B:

let df_session: SessionState = ...;
let iceberg_catalog: Arc<Catalog>  = ...;
df_session.register_catalog("c_name", icebrg_catalog_provider);
df_session. table_factories_mut("ICECBERG", new IcebergTableProviderFactory(iceberg_catalog));
df.sql("CREAT EXTERNAL TABLE c_name.default.tbl STORED AS ICEBERG ...");

With this approach, we could provide a fully functional iceberg integration. But the caveat with this approach is that, we can't stop users from registering table to non-iceberg catalog, e.g. CREATE EXTERNAL TABLE delta_catalog.default.tbl STORE AS ICEBERG ...). This is caused by datafusion's design.

Also we need to add a check in IcebergSchemaProvider::register_table to refuse non iceberg table.

[CTTY]

Hi @liurenjie1024 , thanks for the inputs!

I think your example aligns with my PR #2022

we need to add a check in IcebergSchemaProvider::register_table to refuse non iceberg table.

I agree, we will have to allow MemTable as well since datafusion has to use MemTable for CREATE TABLE sql

I will add the change to block other table types in the CREATE EXTERNAL TABLE PR