Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Destination Improvement: Generic Iceberg Catalog Destination

Answered
Jered Little User

Request: add support for configuring custom Iceberg REST catalog endpoints in Managed Data Lake Service, allowing customers to use their own catalog infrastructure instead of only Fivetran-hosted or predefined catalog integrations.

Current Limitation

MDLS documentation states it can "update other catalog services," but there's no option to configure a generic Iceberg REST catalog endpoint. Customers are limited to:

  • Fivetran-hosted Iceberg REST Catalog (default)
  • Specific pre-built integrations (BigLake Metastore, Unity Catalog, etc.)

Requested Feature

Add "Generic Iceberg REST Catalog" as a catalog type option with these configuration fields:

  • REST Endpoint URL (e.g., https://my-catalog.example.com/v1)
  • Authentication (OAuth2, Bearer Token, API Key)
  • Credentials (token/key)
  • Warehouse name
  • Namespace (optional)
  • Custom HTTP Headers

Use Case

I have an Iceberg catalog with an Iceberg REST endpoint already deployed. I want MDLS to:

  1. Write files to my cloud storage bucket (already works)
  2. Update metadata via my REST catalog endpoint (not currently possible)

Why?

  • Flexibility: Customers can use their own catalog infrastructure (BigLake, Polaris, Unity Catalog, custom implementations)
  • Standards-based: Iceberg REST spec is stable and widely adopted
  • Aligns with MDLS goals: "Flexible and open approach...ensuring compatibility across different ecosystems"
 

Please sign in to leave a comment.

Comments

1 comment

Date Votes
  • Official comment
    Casey Karst Fivetranner

    Hi Jered, 
    Thanks for bringing this up. 
    The reason for our hosting an internal catalog is based on our core value of delivering customer data accurately and on time to the destination. Because of Iceberg write conflict resolution, there are scenarios where two writes conflict and the second write to try to commit to the catalog fails.

    In Fivetran's case, this could mean that a multiday historic sync fails. We have intentionally tried to be very careful to expose that type of failure mode to our customers. 

    This is why we use catalog integrations to expose read-only bronze layer tables to Biglake, Unity Catalog or OneLake Catalog.

    An alternative approach is to use Catalog Federation (Polaris or Snowflake Catalog Linked Database) to read from the Fivetran catalog.