Hi Alon,

Thank you for providing us feedback, can you confirm you how you are retrieving columns? - I'm think it might be through part of our API.

Thank you 

Alison

Hi Alon,

Can you share more about how you are using Magic Folders such that you need to be able to retrieve their columns using the API?

I ask because the Magic Folder connectors are intended to be used to upload Ad Hoc files - no two connectors will have the same columns - but this means that at many points in our pipeline we will not know what columns are being sync'ed for such as connector.

Regards

Alison

Hi Alison,

We are using google drive to integrate with several google sheets or csv files. I would like to be able to retrieve not only tables names (files names) but also to be able to retrieve columns names. I understand that each file could have different columns, since in general files are different. But I would like for you to support to get columns names, so that I am able to select for a column for a corresponding table.

Hi Alon,

Thank you for the additional details - can you share anything about what you would be doing with the columns if we did return them in the API?
Are you setting up queries and data use directly from the API?
I'd love to learn more about the full flow you are envisioning.

Regards

Alison

HI Alison,

Most of the connectors support to retrieve the table columns. I would like to be able to retrieve the columns of the files inside google drive folders. Similar as I can do with google sheets. If I know the possible columns, then I am able to retrieve the data for an specific column. I would be great if we could use the API for that. 

Thanks

Alon

Hi Alison,

Do you have in your roadmap to support to retrieve tables and columns for  magic folders and events connectors?

Hi Alon,

We hadn't being thinking that customers would want to retrieve the columns using our API. I'm trying to more completely understand you full end to end use for this feature so I can consider it for our roadmap.

When you say that you are "able to retrieve the data for a specific column." are you fetching the data from the original sheet or from your data warehouse? What are you doing with the data once you have fetched it? Why is the Fivetran API the logical way to get the column list?

Thank you for helping me build a complete picture

Alison

Hi Alison,

I am retrieving connector schema columns using https://fivetran.com/docs/rest-api/connectors#retrievesourcetablecolumnsconfig . However when I try for magic folders or events connectors I am not able to retrieve since it returns that it is not supported. I would like to know if you are considering in your roadmap to support to retrieve the columns for these connectors. Thank u in advance

Hi Alon,

I understand what you are doing for other connectors and I understand that you would like to get the column data for our Magic Folder connectors and our Event connectors the same way.

These two types of connector are quite different to our other application connectors because the columns could be anything - as they are defined 100% by the data you choose to send through the connector ie the exact files you put in the magic folder or the messages sent to the event bus.

Adding support for this functionality is not currently on the roadmap, however because you are asking for it I would like to consider adding it.

In order to justify adding it to the roadmap I need to understand why getting column information for these two connectors types via our API is the best solution for you.
* What is your over all end to end data flow?
* Why do you need to get the column information from Fivetran rather than the source data or the data ware house?

Thank you

Alison

Hi Alison,