Connector Improvement: Airtable Connector: Name Schemas and Columns by ID only
Not plannedHello,
For experienced data engineers who are working with data owners who are not as tech-savvy, it can be difficult to communicate how important it is not to casually rename Airtable bases, tables and columns which are being synced via Fivetran's Airtable connector. Our Snowflake views break or are filled with NULL data when these changes happen in Airtable without our awareness.
My suggestion would be an opt-in change which would name each schema by the Base ID only (e.g. AIRTABLE_APPFRX74RSSFPVA19TSF) instead of including the name of the base. Also, I'd suggest we could opt in to naming tables by the Table ID (e.g. TBL4FGT490AFR34RZ). Lastly, I'd also suggest that we could opt in to naming columns by the Column ID (e.g. fldlxBdALQpV2aGAC).
If possible, having a Snowflake schema which includes a table that maps field IDs to current field names would also be helpful. Perhaps AIRTABLE_FIELD_NAMES and it could have 6 columns:
- Base ID (e.g. APPFRX74RSSFPVA19TSF)
- Base Name (e.g. David's Projects)
- Table ID (e.g. TBL4FGT490AFR34RZ)
- Table Name (e.g. Tasks)
- Column ID (e.g. fldlxBdALQpV2aGAC)
- Column Name (e.g. Category)
This would help us as data engineers avoid downstream breakage when users do what they naturally will do - experiment with changing names for their data.
-
Official comment
Hi,
Thank you for your detailed feature request regarding schema stability in the Airtable connector.
We understand how it could help build more robust data pipelines, but it also introduces some challenges. Most customers would need to rely on an additional table to manage their downstream queries and their connectors on fivetran, which could add friction.
Supporting both approaches—one with a combination of IDs and names and one with only IDs—would require significant effort. However, we will continue to monitor demand, and if there is enough interest, we may revisit this in the future.
We appreciate your insights and will keep you updated on any developments.
Best,
Sandeep
Please sign in to leave a comment.
Comments
1 comment