Skip to main content

Community

Connector Improvement: Create Empty Columns for Function Connectors

Please sign in to leave a comment.

Comments

4 comments

  • Official comment

    Hi Daniel,

    Thank you for taking the time to submit such a detailed feature request.

    I'm trying to understand a little more about exactly how you end up with columns that exist in the dev table but don't have any dev data. Does your dev setup have all the columns present while filled with nulls or something similar - so that when they are read by the function they get converted to no data.

    I'm wondering what options you have for populating 1 row of data per table?

    Maybe I could follow better if you are able to name a couple of the sources you are working with to create functions currently.

    The ability to create empty columns has come up for our file connectors before and is something we are considering our options to address.

    Thank you once again

    Alison

     

    Hi Alison - I missed your question, sorry!

    We have Function connectors built for Infor Data Lake, Omnitracs, Avature, and Invoca. The Dev connectors should have all the same fields as production, but with very limited data so often there are fields that haven't been filled in.

    The Infor Data Lake one is our most critical. We do potentially have a better option for creating a placeholder record for this particular one since we now have a schema available in the API to know field data types, but that's not true for the other connectors. I think we've run into this with file connectors as well, but I don't have a specific example of that.

    Thanks!

    Dan

    Thank you for the additional information.

    I'm thinking though that not knowing the data columns and their types is actually going to make it hard to use this feature. Even if we made it possible to create a column in the Function response and pass in a NULL, what data type should that column be created as?

    Best regards
    Alison

    That's correct - the columns are all present in the data we retrieve & send to Fivetran, but the values for a column might all be null across the entire dataset. We could potentially populate a dummy row, but we don't always have a known schema for the data we're providing. We'd have a challenge of knowing the data types of columns that do have values and providing a valid dummy value for those, while filling in the gaps for the null columns. We'd essentially have to actively process a large portion of data (to account for sparsely-populated columns), which seems like what Fivetran already does.

    The sources I mentioned in my earlier comment are our main custom connectors at this point. Infor has a schema we can read programmatically, but Omnitracs, Avature, and Invoca don't.

    Edit: I read the comments in reverse order, so hopefully this is still useful. Even if the column can just be created as a string, the main focus is to have the column exist. If we do have a full schema including data types, if there was a way to provide that schema without creating a dummy row, that would be ideal.