Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Connector Improvement: Custom connectors should allow for syncing of empty columns

Completed
Shivy Deo User

Custom connectors currently omit adding a column to the resulting table if all the values in that column are null.

This is frustrating for a developer, because in certain situations, especially early in development and the production launch of a product.  Certain columns may return no data initially, but will have data eventually. 

Omitting a column entirely just because it does not initially have data, even though I am returning that empty column to Fivetran purposefully is pretty annoying.  It creates downstream work for me because Fivetran is ignoring the data model I have given it and that I know I will need.

Please sign in to leave a comment.

Comments

6 comments

Date Votes
  • Official comment
    Alison Fivetranner

    This is now possible when using the Connector SDK. If you declare a column with a data type in your connector schema Fivetran will sync an empty column for it.

    We will continue to consider prioritizing this on our function roadmap.

    Best regards

    Alison

  • Alison Fivetranner
    • Edited

    Hi Shivy,

    Thank you for you feedback. I'm thinking you are using one of our function connectors and seeing this behavior?. It is a standard part of our managed service offering to help ensure that your warehouse isn't cluttered by tables and columns that contain nothing. We also look to enable correct inference of datatype for a column - however to do that we do need at least 1 row of example data.

    Have you considered options for sending 1 row of example data for each table? can you share reasons why that isn't possible/a viable solution in this case?

    best regards,

    Alison

  • Shivy Deo User
    • Edited

    Hi Alison,

    If I'm writing a custom connector, I am intentionally choosing the data that is being send to Fivetran.

    Please point out if I am misunderstanding your position, but I think it is illogical to suggest that Fivetran should have oversight on the data that it should make available, despite that data being intentionally sent to Fivetran.

    If I wanted to omit that data, I would simply omit that data.

  • Alison Fivetranner

    Hi Shivy,

    Thank you for asking for more clarity. The key issue if that we need to know the datatype of every column we create in the warehouse. We use the first field value to infer datatype and without that first value we'd are unable to create the column.

    I understand that this is a little counter intuitive - as you clearly own the data schema when creating custom connectors.

    Do you always know the datatype of the columns you are wanting to sync with no data?

    Best regards

    Alison

  • Shivy Deo User

    Hi Alison,

    A value being present to make inference possible makes sense to me, but I think there also needs to be support for defining a schema and passing that to Fivetran when making custom connectors.  Inference is great, but I don't think it needs to come at the cost of explicitness.

    When developing a custom connector, I can't imagine a situation where a developer could be unaware of the data types they are dealing with.  Being unaware of the data types would make developing the custom connector quite challenging.

    To answer your question, yes in my current use case (and every prior custom connector use case I've worked on), I am aware of the data types I am sending to Fivetran and could define that in a schema.

    Thanks,

    Shivy

  • Alison Fivetranner

    Hi Shivy,

    Thank you for clarifying

    Alison