Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Destination Improvement: Create Empty Columns in Destination if header exists in Source File

Rahini chandrasekaran User
  • Edited

Dear Fivetran Team,

I recently migrated a SFTP connector to a new destination and noticed that few columns were missing. Here are some details:

The Fivetran connector successfully read the header row and displays all the columns in schema. Although some of the columns had no data the past days, they are expected to receive data in the future. Because these few columns have no data at the moment, the columns are not created in the destination.

Impact: Our data pipelines are failing because the SQLs fail due to missing column at the new destination which used to exist previously. 

I hope this clarifies the reason why it would be nice to still have the empty columns created in the destination with NULLs to prevent scripts from failing after migration.

Thank you in advance.

Please sign in to leave a comment.

Comments

2 comments

Date Votes
  • Official comment
    Parmeet Kohli Fivetranner

    Hi Rahini, thanks for submitting this request! I understand the issue with data pipelines breaking because of missing columns. What data types would you expect these columns to be? Fivetran is unable to infer the data type of a column without any data because of which we don't create empty columns as of now.

    Thank you,
    Parmeet

  • Ash Rahman User

    We also have faced the same issue. Appreciate if this issue is resolved, otherwise, we need to think of alternative option.