Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Connector Improvement: Connector SDK: Add Select Methods to Read from Destination

Answered
Nicholas Bauer User

Request:

  • Addition of a new method in the connector SDK that allows for reading data back from the destination through the use of SQL statements.

Context:

  • Fivetran already has the ability to connect to destinations to insert/update data.
  • The SDK exposes methods for checkpointing, upserting and deleting, but not for reading data from the destination.
  • Aware that we can use packages (like the snowflake package) to read back data, but this requires set up, ongoing maintenance for package versions and custom coding to be put in place when Fivetran already has this ability to connect to a destination (and has the creds for it).

Use Cases:

  • Creating custom validations by allowing comparison of record counts or other checks to be built into the pipeline.
  • For systems without a defined key but with changes to existing data, would allow us to diff between the destination and what currently exists in the source to more efficiently call the .delete() method of the operator to flag records as deleted from source.

 

Please sign in to leave a comment.

Comments

1 comment

Date Votes
  • Official comment
    Alison Fivetranner

    Hi Nicholas,

    Thank you for sharing this idea, I have added your request to the feature improvements backlog for further consideration. To help us better understand your requirements, could you share more details given that you can include python code to connect to your destination directly from the connector.py I'm curious if you have implemented validations already and if so could you share more specific details?

    With regards to the delete use case we are working on expanding our delete support by providing functionality that will easily soft delete all records not included in a particular sync for a table - as we have received a number of requests for this when the full contents of a table can be sync'ed in every sync. I'm curious is that functionality would help meet your needs or if you can more details on the specific use case you are targeting.

    We will keep the community updated on this thread with any progress regarding this feature.

    Thanks,
    Alison