PostGreSQL Target types mapping with source system types
AnsweredWhich destination?:
PostGreSQL
Additional details:
Source in my case is SAP.
Voici la version corrigée et renforcée :
Which destination?: PostgreSQL
Additional details:
My source system is SAP.
SAP provides full data type definitions through its DDIC (Data Dictionary), which contains precise field types, lengths, and domain definitions for every table and column in the system. It is therefore surprising that Fivetran does not leverage this metadata during the initial connector setup to create the target table with the correct column types and lengths in PostgreSQL.
The current behavior causes a recurring issue: Fivetran frequently issues ALTER TABLE statements against PostgreSQL when it encounters a value whose length exceeds what was observed in the initial dataset. This is not sustainable for the following reasons:
- Any
ALTER TABLEon a table that has dependent views will cause the Fivetran sync to fail, requiring manual intervention - This forces us to monitor the connector every morning, which is not acceptable in a production environment
- The root cause is avoidable — SAP DDIC already contains the exact field lengths and types
Proposed enhancement: When initializing a new SAP connector, Fivetran should query the SAP DDIC metadata to generate the target PostgreSQL table with the correct column types and maximum lengths from the start, rather than inferring them from the first batch of data.
This would eliminate the need for ALTER TABLE statements during incremental syncs and significantly improve connector reliability.
Thank you for considering this enhancement.
-
Hi,
Thank you for the detailed feedback. We completely understand the operational impact: relying on dynamic type inference instead of SAP DDIC metadata causes unnecessary
ALTER TABLEactions that break downstream PostgreSQL dependencies. Your point is well taken—leveraging DDIC upfront is a much more robust approach.We are looking into how we can improve our type-mapping and initial table creation processes to better leverage source metadata and prevent these downstream schema changes.
We will keep you updated on our progress and roadmap timeline for this improvement.
Best regards,
Please sign in to leave a comment.
Comments
1 comment